Dear list,
I need to insert around 5 sets of 40 values, per second from PD into a postgresql database (both on the same machine). I can see two options - pyext + pygresql or sqlsingle.
From a brief glance at both it looks like the Python route would be
better because sqlsingle introduces a lot of overhead opening and closing the connection, and vacuuming the db between writes.
However, perhaps sqlsingle could handle the task on a modern machine (Centrino 1.6GHz)? This would probably save me some time getting the Python stuff working? Does anyone have any advice either way before I start with this?
On Mon, Nov 07, 2005 at 07:30:28PM +0000, Jamie Bullock wrote:
Dear list,
I need to insert around 5 sets of 40 values, per second from PD into a postgresql database (both on the same machine). I can see two options - pyext + pygresql or sqlsingle.
From a brief glance at both it looks like the Python route would be
better because sqlsingle introduces a lot of overhead
and so does python. why not just open up a pipe (once per session) to the daemon and stream the commands over plaintext. ive not used postgres but mysql has something similar, you just terminate the message with a ";" same as pd..
opening and closing the connection, and vacuuming the db between writes.
However, perhaps sqlsingle could handle the task on a modern machine (Centrino 1.6GHz)? This would probably save me some time getting the Python stuff working? Does anyone have any advice either way before I start with this?
if youre generating a lot of symbols from the patch, to use as text for queries, youre going to incur ram leakage, due to the fact that PD has no 'string' type, just a never-deallocated symbol table..
depending on how simple your database needs are, maybe you can use Pool or PDContainer instead...although theyre certainly a bit less flexible in terms of select-statement-esque things they can do the basics like retrieve via ID, next/prev, all, 'namespaces' etc..
-- Regards,
Jamie
PD-list@iem.at mailing list UNSUBSCRIBE and account-management -> http://lists.puredata.info/listinfo/pd-list
Le 05-11-07, à 14:47, carmen a écrit :
On Mon, Nov 07, 2005 at 07:30:28PM +0000, Jamie Bullock wrote:
Dear list,
I need to insert around 5 sets of 40 values, per second from PD into a postgresql database (both on the same machine). I can see two options
pyext + pygresql or sqlsingle.
From a brief glance at both it looks like the Python route would be
better because sqlsingle introduces a lot of overhead
and so does python. why not just open up a pipe (once per session) to the daemon and stream the commands over plaintext. ive not used postgres but mysql has something similar, you just terminate the message with a ";" same as pd..
opening and closing the connection, and vacuuming the db between writes.
However, perhaps sqlsingle could handle the task on a modern machine (Centrino 1.6GHz)? This would probably save me some time getting the Python stuff working? Does anyone have any advice either way before I start with this?
if youre generating a lot of symbols from the patch, to use as text for queries, youre going to incur ram leakage, due to the fact that PD has no 'string' type, just a never-deallocated symbol table..
depending on how simple your database needs are, maybe you can use Pool or PDContainer instead...although theyre certainly a bit less flexible in terms of select-statement-esque things they can do the basics like retrieve via ID, next/prev, all, 'namespaces' etc..
Also, if you are interested, I just wrote a PHP class to send and receive messages to and from PD. It is very easy to code in PHP, so this may be an option for your project. Let me know if you want it. I should post it on my web site soon....
+-------------------------------+--------------------------------------+ | Alexandre Quessy @ Montréal | http://www.sourcelibre.com/puredata/ | +-------------------------------+--------------------------------------+
Thanks Carmen, and everyone else who replied so quickly to this.
I like the idea of a using a unix pipe a lot - nice and simple. Just to confirm, did you mean something like:
mknod pgpipe p cat pgpipe | psql -d dbname
Then in PD just collect the commands in a textfile buffer and send them in one go at the end?
Just tried this with dummy data and it works. If I understand PD source code correctly, textfile dynamically allocates RAM for itself as new text is added. I can't see any upper limit on this, but I know how much RAM I've got, and roughly how many chars I need to write, so I should be OK, right?
Jamie
On Mon, 2005-11-07 at 19:47 +0000, carmen wrote:
On Mon, Nov 07, 2005 at 07:30:28PM +0000, Jamie Bullock wrote:
Dear list,
I need to insert around 5 sets of 40 values, per second from PD into a postgresql database (both on the same machine). I can see two options - pyext + pygresql or sqlsingle.
From a brief glance at both it looks like the Python route would be
better because sqlsingle introduces a lot of overhead
and so does python. why not just open up a pipe (once per session) to the daemon and stream the commands over plaintext. ive not used postgres but mysql has something similar, you just terminate the message with a ";" same as pd..
opening and closing the connection, and vacuuming the db between writes.
However, perhaps sqlsingle could handle the task on a modern machine (Centrino 1.6GHz)? This would probably save me some time getting the Python stuff working? Does anyone have any advice either way before I start with this?
if youre generating a lot of symbols from the patch, to use as text for queries, youre going to incur ram leakage, due to the fact that PD has no 'string' type, just a never-deallocated symbol table..
depending on how simple your database needs are, maybe you can use Pool or PDContainer instead...although theyre certainly a bit less flexible in terms of select-statement-esque things they can do the basics like retrieve via ID, next/prev, all, 'namespaces' etc..
-- Regards,
Jamie
PD-list@iem.at mailing list UNSUBSCRIBE and account-management -> http://lists.puredata.info/listinfo/pd-list
PD-list@iem.at mailing list UNSUBSCRIBE and account-management -> http://lists.puredata.info/listinfo/pd-list
-
Alexandre Quessy -
carmen -
Jamie Bullock