Hey Patrick,
I guess the standard most up to date raspbian wheezy is the best place to start.
For me what's worked best is to find a super-minimal install and then build requirements on top of that.
With the project I'm working on atm I couldn't get my pmpd patch to even run with the standard install (there is quite a lot going on though). Now the patch runs fine with a whole heap of other stuff on top of that (2 sensors, 6 channel soundcard etc). My htop reading fills me with joy-I think there's about a dozen processes including 3 shells.
As it's a rev1 board I built a system on top of the hexxeh image (same guy who's done the 'rpi-update' program for updating the firmware. My understanding though is that with rev2 boards don't do it.