You could draw the spectrum in an array, send it to rifft~, and send the output to another array that you could use as a wavetable - something like
[tabreceive~ spectrum] | [rifft~] | [tabsend~ wavetable]
then
[phasor~] | [*~ <arraysize>] | [tabread4~ wavetable]
Yes, this is a good idea
just keep in mind that if you want to control graphically the amplitude of 500 partials (for example), the "spectrum" array / table must have a width of 500 pixels or more