Ok, so I also have a lot of mac experience and maybe I can help here as well.
First, you need to rip the source material as has been said before, preferably without any change of format. You are working with BD, so you a need a tool that does this for you. This is actually the one step I don't have the answer for you as I have another thread with this exact same question (just the ripping step). However, L8 mentions PavTube can do this; during my own research, I also have come across makemkv (which is currently free while in beta) and will rip to a mkv file. That file will need to be broken apart using another free tool I have used a lot (iMkvExtract), and the resulting files can be fed to MPEGStreamclip. Finally, I have heard that DVDFab has a product that will also rip BDs. Again, hopefully my other thread will hash out the differences and/or at least the best choice for doing this. For now though, you can just follow L8's advice on this since he obviously has had proven success with PavTube.
Now, and this is the very important part: you MUST only transcode to an intermediate codec (that is a code which is meant for editing, as opposed to a delivery codec, like h264 which is very highly compressed), and on the mac when using FCP, that ONLY means PRORES.
There are actually four different types of the PRORES codec:
1. prores 444 - the least lossy codec, which also has an alpha channel built-in. For fan editing purposes, this is WAY overkill to use as a main codec; however, it DOES have it's uses when you need to create something where you need an alpha channel. For example, I've used this codec when working in Motion (for my Episode II edit I recreated the beginning approach to coruscant) and you want an alpha channel in the result, for example, to show the stars in the background.
At any rate, there is absolutely no need to use prores 444 for your entire movie. You really will just end up using a lot of extra hard drive space and probably slightly degrade editing performance due to the extra i/o that entails
2. prores 422 - this is apple's "main" editing codec, meant for professional editors working on original source material inside final cut pro. I will not say you cannot use this codec, and if you aren't concerned at all about hard drive space, this is probably a fine choice.
3. prores 422 (LT) - this is the "lite" version of prores 422 which ends up taking up roughly 2/3 of the space of proves 422. This is the codec I use for all my HD edits, and here's why. If you read Apple's white paper on all these codecs, you will find out that this codec is visually equivalent to prores 422 until you get to about the 10th generation. Basically, unless you plan on doing A LOT of compositing and/or going back and forth between different programs, prores should fully meet your quality needs. You always have the option to take a section of your edit and export it as proves 422 or 444 if you plan on using another tool. Also, you can set FCP to do all it's own intermediate rendering in prores 422, so you kind of get the best of both worlds since your original media gets converted a single time into proves 422 LT (saving you space and being visually equivalent to proves 422), while when working in your editing program, you are using proves 422 which alleviates worries that you might be degrading quality based on extensively edited sections (e.g., lots of compositing/special effects added in a single location).
4. prores (proxy) - DON'T use this for your main edit. This is meant as a much smaller, lightweight codec for "offline" editing use by professionals. For the purposes of this discussion, we should leave it there; there are uses for this codec during your editing process, but that's a different topic!
So to sum up, use either prores 422 or 422(LT).
Ok, so we have ripped to disc in native format using some sort of software, and possibly even transcoded directly to quicktime if you follow L8's workflow.
In my current workflow, I use MPEGStreamclip for the transcode, and (I haven't checked my other thread yet) I'm hoping there will be a way to rip to a format that I can use with it, since MPEGStreamclip has proven to be a very very reliable transcoder/encoder and is widely used even in professional circles. It's main drawback is speed; it only is single threaded which is a major bummer; but it gets the job done and done right. Personally, I've had some problems in the past using Apple's Compressor (getting confused with some 23.976 vs 24fps issues I think), but I've never had a problem with MPEG stream clip. FYI, I usually transcode the video and audio separately in MPEGStreamclip, then combine them using Quicktime Pro later (see below).
I will definitely need to check out PavTube, given L8's recommendation though. If L8 thinks PavTube transcodes to prores without any problems, then that might be the easiest best solution (on the mac, having to do less steps is better). Make sure you create a quicktime movie file (.mov) and NOT anything else though. Finally, if you are using FCPX, you will still possibly need to deal with file size (see below). (And, if you are using FCPX, you might need to play around with the audio choices to get the one that works best with FCPX...I haven't used it yet so I don't know whether this is an issue or not)
At any rate, I can definitely confirm MPEGStreamclip can transcode directly to all the prores codecs, and again, it is a great tool that is very well known and respected.
There are two more issues I want to address, audio, and file size.
First, audio. Obviously you want to have 5.1 sound. MPEG streamclip can read AC3 audio files (but not DTS files, unfortunately), and export individual uncompressed (AIFF) files for each channel. In my standard workflow, this is what I do. Then I combine the video and the 6 audio channels together using Quicktime Pro (you might not have this; not sure if apple is still selling it, but it was only about $30 at the time). Quicktime Pro lets me assign each audio an individual channel (L, R, C, LFE, LR, LS), and also add them to the file in that exact order which is the order in which FCPX requires. (I think FCP7 has a bit more flexibility here and lets you assign the channels right inside the application). I forget if you are using FCP7 or FCPX, but if it is FCP7 then you probably don't need to worry about this and can just import everything into FCP7 and assign channels there. Then you can create a big compound clip and use that as your source for everything. For FCPX, you need to get the audio packaged correctly ahead of time to not cause you headaches later.
Finally, file size. This only applies to FCPX -- if you are using FCP7, you don't need to worry about this. However, with FCPX, due to how it has been engineered (it auto-saves EVERY SINGLE edit you make immediately, thus taking away the need for a manual "save" command [which doesn't even exist in the app!), you need to work with your original material split up into sections. I use Quicktime Pro to split up my one big file into several files of about 10 minutes in length. This was absolutely necessary in the early days of FCPX; maybe they fixed the problems that caused the performance issues that would crop up once you got about 30-45 minutes into your edit. But I still play it safe and do this extra step. Again, I don't recall you mentioning FCPX so maybe this won't apply to you.
Hope some of this helps!