To Pd developers,
After years of worrying about it, I'm thinking it's time to add video support natively to Pd. My basic idea is to add a feature to the block~ object so that windows of DSP computation can be triggered from external messages. That way, video I/O objects could be designed to spit out frames or portions of frames on demand, and the user gets the ability to explicitly schedule how the video computations should be run.
(My favorite example of non-obvious scheduling of video computation is low-latency analysis of incoming video, where you can actually use the 1/60-second fields without having to wait the additional 1/60-th second for a complete frame to arrive.)
I think there will have to be some state added to the DSP chain mechanism so that objects explicitly designed to work on video streams can find out what part of the image they're looking at. This could be as simple as a global data structure describing the current vector's position in an image; existing tilde objects would simply ignore the information.
To begin with at least, I'm hoping to be able to do all video operations except storage using floating-point numbers, thereby re-using the usual tilde objects. The disadvantage of this approach is that, if you want to "sample" an image, to get decent cache behavior you'd want the possibility of storing it in a data-reduced way, as 8-bit integers or perhaps using YUYV packing. (for example, a 512x768 color video frame takes almost 5MB to store in floating point, but only about 0.7 MB in 8-bit YUYV.)
So there will probably have to be a suite of new objects for storing 2D arrays in fixed point formats, variously trading off memory compaction with processing time needed to get the data in and out. There will probably also want to be a choice of interpolation strategies. Probably the design can look like the table/tabwrite/tabread/tabread4/tabwrite~/tabread~/tabread4~ objects.
Also like "table" objects I want to publish an API so that pdp and gem can read and write into image storage buffers.
I'll probably base the actual video I/O objects on the way Tom Shouten did it in pdp - perhaps pdp would then be able to use pd's video objects directly if I can get the design just right.
ideas and opinions welcome!
cheers Miller