Bradon, I've found that a lot of pd performance problems come from the GUI. While running your patch, open up the Windows Task Manager and keep an eye on the "pd.exe" and "wish83.exe" processes. I've found that the "wish83.exe" process corresponds to the GUI and the "pd.exe" process corresponds to the actual audio/midi/control processing. Keep an eye on the CPU% usage for these processes while using your patch and you should be able to see what the bottleneck is.
I've found that changing states (e.g. via pool or state) will cause a big spike in the GUI process because many numberboxes/sliders/etc are being updated at once. Also, I've found that GOPs use up more GUI power.
If the GUI is the problem, try putting as much of the patch as possible in a sub-patch and keep that sub-patch window closed... so you only have a minimal amount of GUI elements on the screen. This is where an external MIDI controller of some sort comes in handy, because you can tweak parameters without relying on the GUI.
I hope this helps.