I would probably do the asset loading from outside pd, most programming languages have a redis client library. For this kind of job Ruby, Python, Perl, Lua or even V8 could do it. I think this kind of script would be trivial to implement.
Something could also be achieved from within Redis with Lua, using the soon to be stable "Scripting Branch" http://antirez.com/post/an-update-on-redis-and-lua.html http://antirez.com/post/an-update-on-redis-and-lua.htmlBut as the Scripting Engine blocks Redis, I would not do anything to heavy in there, still it would probably be fast and convenient.
Also, having the pdlua external loading stuff in redis from inside of PD could also be an option to consider.
I'm sure these suggestions trigger more implementations ideas for the experienced pd users.
Having a puredis-asset loader object as part of the puredis library could also be a possibility, given a CSV, SQL or JSON file path, the object could make sure redis is preset the right way.
what do you think?
2011/7/26 Hans-Christoph Steiner hans@at.or.at
How would you dump large data sets into redis? That's what I would likely use it for: taking data from CSV or SQL files and then using it within Pd.
.hc
On Jul 26, 2011, at 9:37 AM, Louis-Philippe wrote:
Thanks for the cue about the Library template, I will look into it and will probably integrate it.
About Puredis intended usage... for my part I have in mind to integrate pd in a larger system using Redis as my data's rosetta stone. For Queue systems, Data Persistence, Pub/Sub IPC... I got used to the way you can manipulate large data sets inside Redis and tought it would match nicely with pd, and after implementating it I am even more excited about its potential.
L-P
2011/7/25 Hans-Christoph Steiner hans@at.or.at
That's a great manifesto, sounds like an interesting object. Something like [pool] but with more options. What do you have in mind to use it for?
On the exciting topic of build systems, I recommend using the Library Template for this. It'll save you the headache of making a build system for GNU/Linux, Windows, etc.
http://puredata.info/docs/developer/LibraryTemplate
.hc
"A cellphone to me is just an opportunity to be irritated wherever you are." - Linus Torvalds
I like the idea of a puredis-loader object. It could either be
blocking, and that's just a known thing to workaround, i.e. load the
file at patch startup. That's a good place to start at least, then
things like threading, background loading, etc. could be explored later.
.hc
On Jul 26, 2011, at 1:22 PM, Louis-Philippe wrote:
I would probably do the asset loading from outside pd, most
programming languages have a redis client library. For this kind of
job Ruby, Python, Perl, Lua or even V8 could do it. I think this
kind of script would be trivial to implement.Something could also be achieved from within Redis with Lua, using
the soon to be stable "Scripting Branch" http://antirez.com/post/an-update-on-redis-and-lua.html But as the Scripting Engine blocks Redis, I would not do anything to
heavy in there, still it would probably be fast and convenient.Also, having the pdlua external loading stuff in redis from inside
of PD could also be an option to consider.I'm sure these suggestions trigger more implementations ideas for
the experienced pd users.Having a puredis-asset loader object as part of the puredis library
could also be a possibility, given a CSV, SQL or JSON file path, the
object could make sure redis is preset the right way.what do you think?
2011/7/26 Hans-Christoph Steiner hans@at.or.at
How would you dump large data sets into redis? That's what I would
likely use it for: taking data from CSV or SQL files and then using
it within Pd..hc
On Jul 26, 2011, at 9:37 AM, Louis-Philippe wrote:
Thanks for the cue about the Library template, I will look into it
and will probably integrate it.About Puredis intended usage... for my part I have in mind to
integrate pd in a larger system using Redis as my data's rosetta
stone. For Queue systems, Data Persistence, Pub/Sub IPC... I got
used to the way you can manipulate large data sets inside Redis and
tought it would match nicely with pd, and after implementating it I
am even more excited about its potential.L-P
2011/7/25 Hans-Christoph Steiner hans@at.or.at
That's a great manifesto, sounds like an interesting object.
Something like [pool] but with more options. What do you have in
mind to use it for?On the exciting topic of build systems, I recommend using the
Library Template for this. It'll save you the headache of making a
build system for GNU/Linux, Windows, etc.http://puredata.info/docs/developer/LibraryTemplate
.hc
"A cellphone to me is just an opportunity to be irritated wherever
you are." - Linus Torvalds
Man has survived hitherto because he was too ignorant to know how to
realize his wishes. Now that he can realize them, he must either
change them, or perish. -William Carlos Williams
before jumping on the puredis-loader implementation, here is how I see the specs, please tell me if I get what you think: ___________________________________ Strings:
string.csv: KEY, VALUE key1, value1 key2, value2
[lpuredis string string.csv]
-> creates a String for each line ___________________________________ List and Sets:
list.csv: KEY, VALUE1, VALUE2, ... key1, item1, item2, item3 key2, item1, item2, item3
[lpuredis list list.csv]
-> creates a List for each line
[lpuredis set list.csv]
-> creates a Set for each line ___________________________________ Sorted Sets:
zset.csv: KEY, VALUE1, SCORE1, VALUE2, SCORE2 key1, item1, 1, item2, 2, item3, 3 key2, item1, 4, item2, 5, item3, 6
[lpuredis zset zset.csv]
-> creates a Sorted Set for each line ____________________________________ Hashes
hash.csv: KEY, KEY1, VALUE1, KEY2, VALUE2, KEY3, VALUE3 key1, hkey1, value1, hkey2, value2, hkey3, value3 key2, hkey1, value1, hkey2, value2, hkey3, value3
or even: KEY, FIELD1, FIELD2, FIELD3 key1, item1, item2, item3 key2, item1, item2, item3
[lpuredis hash hash.csv]
-> creates a Hash for each line _______________________________________________________
For SQL file I am clueless, but I guess it will be similar... JSON would be the easiest to translate but would it be of any use to pd users?
L-P
Where I could see it used it with large data sets like I did in this
project:
http://at.or.at/hans/terrenatale/
The data is used to generate the large scale events like the dropping
of coins for each country, the layout of static representing
population, etc. I suppose that I would need to convert the data to a
specific format for redis anyway, so just supporting CSV would be
fine. I ended up using scripts and Pd to process a lot of the data.
What would make puredis more interesting is if it meant that it would
be easy to process large sets of data into something useful for
generating sound and video.
.hc
On Jul 26, 2011, at 2:54 PM, Louis-Philippe wrote:
before jumping on the puredis-loader implementation, here is how I
see the specs, please tell me if I get what you think: ___________________________________ Strings:string.csv: KEY, VALUE key1, value1 key2, value2
[lpuredis string string.csv]
-> creates a String for each line ___________________________________ List and Sets:
list.csv: KEY, VALUE1, VALUE2, ... key1, item1, item2, item3 key2, item1, item2, item3
[lpuredis list list.csv]
-> creates a List for each line
[lpuredis set list.csv]
-> creates a Set for each line ___________________________________ Sorted Sets:
zset.csv: KEY, VALUE1, SCORE1, VALUE2, SCORE2 key1, item1, 1, item2, 2, item3, 3 key2, item1, 4, item2, 5, item3, 6
[lpuredis zset zset.csv]
-> creates a Sorted Set for each line ____________________________________ Hashes
hash.csv: KEY, KEY1, VALUE1, KEY2, VALUE2, KEY3, VALUE3 key1, hkey1, value1, hkey2, value2, hkey3, value3 key2, hkey1, value1, hkey2, value2, hkey3, value3
or even: KEY, FIELD1, FIELD2, FIELD3 key1, item1, item2, item3 key2, item1, item2, item3
[lpuredis hash hash.csv]
-> creates a Hash for each line _______________________________________________________
For SQL file I am clueless, but I guess it will be similar... JSON would be the easiest to translate but would it be of any use to
pd users?L-P
If you are not part of the solution, you are part of the problem.
Ok, I came up with a generic csv loader for Redis datatypes. it is plugged into the [puredis] object with the csv method, it receives a filepath and a datatype. Making it possible to build Strings, Lists, Hashes, Sets ans SortedSets from csv files.
I updates the pd-help files to match a well as the docs (the Readme). https://github.com/lp/puredis#readme
2011/7/26 Hans-Christoph Steiner hans@at.or.at
Where I could see it used it with large data sets like I did in this project: http://at.or.at/hans/**terrenatale/ http://at.or.at/hans/terrenatale/
The data is used to generate the large scale events like the dropping of coins for each country, the layout of static representing population, etc. I suppose that I would need to convert the data to a specific format for redis anyway, so just supporting CSV would be fine. I ended up using scripts and Pd to process a lot of the data. What would make puredis more interesting is if it meant that it would be easy to process large sets of data into something useful for generating sound and video.
.hc
On Jul 26, 2011, at 2:54 PM, Louis-Philippe wrote:
before jumping on the puredis-loader implementation, here is how I see the
specs, please tell me if I get what you think: ______________________________**_____ Strings:
string.csv: KEY, VALUE key1, value1 key2, value2
[lpuredis string string.csv]
-> creates a String for each line ______________________________**_____ List and Sets:
list.csv: KEY, VALUE1, VALUE2, ... key1, item1, item2, item3 key2, item1, item2, item3
[lpuredis list list.csv]
-> creates a List for each line
[lpuredis set list.csv]
-> creates a Set for each line ______________________________**_____ Sorted Sets:
zset.csv: KEY, VALUE1, SCORE1, VALUE2, SCORE2 key1, item1, 1, item2, 2, item3, 3 key2, item1, 4, item2, 5, item3, 6
[lpuredis zset zset.csv]
-> creates a Sorted Set for each line ______________________________**______ Hashes
hash.csv: KEY, KEY1, VALUE1, KEY2, VALUE2, KEY3, VALUE3 key1, hkey1, value1, hkey2, value2, hkey3, value3 key2, hkey1, value1, hkey2, value2, hkey3, value3
or even: KEY, FIELD1, FIELD2, FIELD3 key1, item1, item2, item3 key2, item1, item2, item3
[lpuredis hash hash.csv]
-> creates a Hash for each line ______________________________**_________________________
For SQL file I am clueless, but I guess it will be similar... JSON would be the easiest to translate but would it be of any use to pd users?
L-P
------------------------------**------------------------------**
If you are not part of the solution, you are part of the problem.
-
Hans-Christoph Steiner -
Louis-Philippe