I think this would then be probably done in vanilla OpenGL, not Gem. And I agree this could potentially work but is perhaps not as easily implemented as with using JUCE where a lot of APIs are already prebuilt for you. The same may be the case with Qt but I don't know for sure as I haven't messed with it very recently.
I've done it in pd-extended. it's true, the initial step to do a full gui will be a hassle, because you'll have to program the interface yourself. but if that's done carefully, it should pay up afterwards.
or just make a gui in processing, csound or supercollider through osc :)