Hello,
inspired by the great effort of Louis-Philippe with Puredis, I have started developping a library for another of the NoSQL databases out there, CouchDB.
CouchDB is a database with a RESTful interface, i.e. you connect to the database with HTTP requests and depending on the HTTP method you perform the operations. The data you send to CouchDB for storage and the data you receive back from CouchDB are JSON objects, so (more or less) as a lemma, the library also contains objects for encoding and decoding JSON data.
The current version is online at https://github.com/residuum/CouchPdb
A reason for developping the library is Ubuntu One, as it offers a free 5GB space and includes a CouchDB database.
All is in an early stage of development, so do not expect anything to work, it is still very experimental.
I mainly uploaded it to Github to discuss a few points, where I do not know how to deal with in a manner fitting Pd, especially when it comes to encoding and decoding JSON data:
- How should JSON data be dealt with in general? I am currently just
outputting one long symbol separated by spaces, so an object of {"id":1, "name": "my-name"} becomes id 1 name my-name but {"id":1, "name": "my name", "year":2011} becomes id 1 name my name year 2011 and thus breaks the pattern of "key value key value key value".
I could output this as list, which would make spaces in values not break the pattern of "key value key value key value" (really?). But is this really a way to represent data in the correct Pd way, i.e. does this data format fit the principle of least surprise?
- How should nested objects and arrays be handled? Should they be
handled at all?
- How to implement attachments (binary files) for data? Or should I
forget about this stuff and e.g. store images as base64 encoded strings? I am asking this one in particular, because one application of the object that came into my mind would be a website to upload pictures that are downloaded by a Pd patch in an installation in some gallery, or multiple gallery installations worldwide uploading snapshop images and interacting somehow.
Please do not read the code ;) I have not programmed C for ~10 years, and I have never coded anything but simple stuff for university. So actually: Read the code and give me pointers (no pun intended) to problematic code.
Best regards, Thomas
Sounds interesting, do you have an application in mind? This kind of
library is best explained via an interesting example, IMHO. I'll give
some feedback inline below:
On Aug 26, 2011, at 7:00 PM, Thomas Mayer wrote:
Hello,
inspired by the great effort of Louis-Philippe with Puredis, I have started developping a library for another of the NoSQL databases out there, CouchDB.
CouchDB is a database with a RESTful interface, i.e. you connect to
the database with HTTP requests and depending on the HTTP method you
perform the operations. The data you send to CouchDB for storage and the data you receive back from CouchDB are JSON objects, so (more or less) as a lemma, the library also contains objects for encoding and decoding
JSON data.The current version is online at https://github.com/residuum/CouchPdb
A reason for developping the library is Ubuntu One, as it offers a
free 5GB space and includes a CouchDB database.All is in an early stage of development, so do not expect anything to work, it is still very experimental.
I mainly uploaded it to Github to discuss a few points, where I do not know how to deal with in a manner fitting Pd, especially when it comes to encoding and decoding JSON data:
- How should JSON data be dealt with in general? I am currently just
outputting one long symbol separated by spaces, so an object of {"id":1, "name": "my-name"} becomes id 1 name my-name but {"id":1, "name": "my name", "year":2011} becomes id 1 name my name year 2011 and thus breaks the pattern of "key value key value key value".
I could output this as list, which would make spaces in values not
break the pattern of "key value key value key value" (really?). But is this really a way to represent data in the correct Pd way, i.e. does this data format fit the principle of least surprise?
How about a message per key/value pair? Then you just need to ensure
that the key does not have spaces in it, then the value can be a
symbol, float, list, etc. It is possible to generate symbols with
spaces in them in C, and you can work with them too. The problem
really is you can't save symbols with spaces to a .pd file (unless you
use DesireData)
- How should nested objects and arrays be handled? Should they be
handled at all?
That's pretty tricky... I can't think of any examples in Pd. To start
with, I say get something working with the most simple useful set of
features, then build from there. So ignore it for now, and give
guidelines as to how to format the json for Pd.
- How to implement attachments (binary files) for data? Or should I
forget about this stuff and e.g. store images as base64 encoded
strings? I am asking this one in particular, because one application of the object that came into my mind would be a website to upload pictures
that are downloaded by a Pd patch in an installation in some gallery, or multiple gallery installations worldwide uploading snapshop images and interacting somehow.
You should look at Martin Peach's (net,osc, mrpeach lib) and Bryan
Jurish's (moocow, bytes2any, etc) work with strings. You can send
arbitrary data as lists of floats, where each float is a byte. Using
[binfile] you can write bytes to a file, using bytes2any you can
convert them to strings, and any2bytes for vice versa.
.hc
Please do not read the code ;) I have not programmed C for ~10 years, and I have never coded anything but simple stuff for university. So actually: Read the code and give me pointers (no pun intended) to problematic code.
Best regards, Thomas -- "Prisons are needed only to provide the illusion that courts and
police are effective. They're a kind of job insurance." (Leto II. in: Frank Herbert, God Emperor of Dune) http://www.residuum.org/
Pd-list@iem.at mailing list UNSUBSCRIBE and account-management -> http://lists.puredata.info/listinfo/pd-list
I have the audacity to believe that peoples everywhere can have three
meals a day for their bodies, education and culture for their minds,
and dignity, equality and freedom for their spirits. - Martin
Luther King, Jr.
Hi,
thanks for your feedback. Here is what I came up with:
On 29.08.2011 21:31, Hans-Christoph Steiner wrote:
Sounds interesting, do you have an application in mind? This kind of library is best explained via an interesting example, IMHO.
Currently I am working on a contrived example: a sequencer which gets the data from the database. If you have a central CouchDB server, you could control the sequencer from any other source, e.g. use the sequencer in an art installation and let users control it via a website. Contrived, yes, but doable, but I am still working on it.
On Aug 26, 2011, at 7:00 PM, Thomas Mayer wrote:
- How should JSON data be dealt with in general? I am currently just
outputting one long symbol separated by spaces, so an object of {"id":1, "name": "my-name"} becomes id 1 name my-name but {"id":1, "name": "my name", "year":2011} becomes id 1 name my name year 2011 and thus breaks the pattern of "key value key value key value".
How about a message per key/value pair? Then you just need to ensure that the key does not have spaces in it, then the value can be a symbol, float, list, etc. It is possible to generate symbols with spaces in them in C, and you can work with them too. The problem really is you can't save symbols with spaces to a .pd file (unless you use DesireData)
I came up with lists: The example above outputs now on the left outlet list id 1 list name my\ name list year 2011
After a JSON object is decoded, I output a bang on a second outlet, and therefore can distinguish between two objects in short succession.
Best regards, Thomas
On Wed, 31 Aug 2011, Thomas Mayer wrote:
I came up with lists: The example above outputs now on the left outlet list id 1 list name my\ name list year 2011
Yes, that sounds like the appropriate thing to do with Pd.
After a JSON object is decoded, I output a bang on a second outlet, and therefore can distinguish between two objects in short succession.
Yes, either that or an 'end' message on the first inlet, but what you did is more usual.
| Mathieu Bouchard ---- tél: +1.514.383.3801 ---- Villeray, Montréal, QC
Hello all,
On 03.09.2011 18:36, Mathieu Bouchard wrote:
On Wed, 31 Aug 2011, Thomas Mayer wrote:
I came up with lists: The example above outputs now on the left outlet list id 1 list name my\ name list year 2011
Yes, that sounds like the appropriate thing to do with Pd.
After a JSON object is decoded, I output a bang on a second outlet, and therefore can distinguish between two objects in short succession.
Yes, either that or an 'end' message on the first inlet, but what you did is more usual.
Here is what I came up for arrays and nested objects: If they come up inside an object, they are output as the symbol that they are. The JSON string can then be parsed by another instance of [json-decode] (or send it recursively to the same object).
Arrays as symbols themselves are parsed like several seperate objects. Here is an example:
Consider the JSON string {"id": 1, "name": "Residuum","member": {"first-name": "Thomas", "last-name": "Mayer"}, "albums": [{"year": 2011, "name": "Der Diskokeller des Grauens"}, {"year": 2009, "name": "Schrei-Funk-Flaeche"}]}
When parsed with an instance of [json-decode] this will output list id 1 list name Residuum list member {"first-name": "Thomas", "last-name": "Mayer"} list albums [{"year": 2011, "name": "Der Diskokeller des Grauens"}, {"year": 2009, "name": "Schrei-Funk-Flaeche"}]
followed by a bang on the second outlet.
When parsing the array for "albums", i.e. [{"year": 2011, "name": "Der Diskokeller des Grauens"}, {"year": 2009, "name": "Schrei-Funk-Flaeche"}]
This will output: list year 2011 list name Der\ Diskokeller\ des\ Grauens bang on second outlet list year 2009 list name Schrei-Funk-Flaeche bang on second outlet
The current version is available at https://github.com/residuum/CouchPdb
Thanks for reading, Thomas
On Mon, 29 Aug 2011, Hans-Christoph Steiner wrote:
On Aug 26, 2011, at 7:00 PM, Thomas Mayer wrote:
I could output this as list, which would make spaces in values not break the pattern of "key value key value key value" (really?). But is this really a way to represent data in the correct Pd way, i.e. does this data format fit the principle of least surprise?
How about a message per key/value pair? Then you just need to ensure that the key does not have spaces in it, then the value can be a symbol, float, list, etc.
What's the «etc» standing for ?
What else could it be, other than symbol, float, list ?
If CouchPdb is outputting an object, and that object contains an object, and that object contains an object, you have two different ways of handling « getting the keys and values of an object » for the toplevel object and the first-level subobjects, but the second-level subobjects are toast. Sending separate key-pairs only push back the problem by one level of nesting.
BTW, with a real-world example : if you parse JSON output from GoogleMaps search results, the coordinates of an item are two values for keys "lat" and "lng" in an object assigned to the key "location" in an object assigned to the key "geometry" in an object that is in an array that is assigned to the key "results" in the toplevel object. (If instead you want to access a bounding-box on the map, it's one level deeper than that).
(That's my only contact with JSON.)
| Mathieu Bouchard ---- tél: +1.514.383.3801 ---- Villeray, Montréal, QC
On Sat, 27 Aug 2011, Thomas Mayer wrote:
- How should nested objects and arrays be handled? Should they be
handled at all?
There's really no standard way to do any kind of nesting in Pd. Several externals provide their own way of nesting things but none of them are really widespread in use. My proposals in that direction haven't gone anywhere (reference counting would have to be supported by all externals, with a slightly different Pd API, more or less).
If you are also limited to only floats and symbols for doing this, you can't even pass delimiters like "(" and ")" to indicate beginning of something or end of something, because they are already possible text strings in JSON. It's the same problem as trying to use "begin" and "end" as delimiters, just a little bit less common because parens rarely happen as single-character text strings.
It really wouldn't be that hard to implement a binbuf atom type, if it weren't for the issue of how (when) to deallocate it. I always assumed that this sort of thing ought to deallocate itself on its own when it stops being referenced, but Pd provides no means to make that ever happen.
a binbuf atom would be something that counts as one atom but can be expanded to a complete 'list'. Thus if a list represents an object that has 5 key-value pairs, and the 4th one has an object value, then you would have an atom-type sequence like :
SYMBOL FLOAT SYMBOL SYMBOL SYMBOL FLOAT SYMBOL BINBUF SYMBOL SYMBOL
and then when you use a «binbuf $8» or «list $8» messagebox, you still would have a single atom, but something like [list frombinbuf] or [binbuf2list] would expand it to a list of n elements (input message would need to contain a single atom whose type is binbuf).
It doesn't need to be called a binbuf, but that's what it's already called in Pd, though it's currently not an atom type, only a thing for long-term (non-stack) holding of a list.
- How to implement attachments (binary files) for data? Or should I
forget about this stuff and e.g. store images as base64 encoded strings?
Have a way to write them to disk ?
Lots of things are going to be complicated unless you decide to rely upon certain rarely-used externals.
| Mathieu Bouchard ---- tél: +1.514.383.3801 ---- Villeray, Montréal, QC
On Sat, Aug 27, 2011 at 01:00, Thomas Mayer thomas@residuum.org wrote:
[...]
- How should nested objects and arrays be handled? Should they be
handled at all?
Nested objects could be just output flat along with with top-level and they could have a property "parent" or something like that. E.g.: list id 1 list parent 0 <-- means it's top level (in case the first element is 1 not 0) list name pasta (bang on second outlet) list id 2 list parent 1 <-- means it's a child of id 1 list name flour list grams 250 (bang on second outlet) list id 3 list parent 1 list name water list grams 150 (etc...)
Arrays... I'm not sure. If it's not an array of objects, just an a simple array of symbols/floats, isn't this possible?: list id 1 list name family list members Mary Joe Carl Felicity
Andras
-
András Murányi -
Hans-Christoph Steiner -
Mathieu Bouchard -
Thomas Mayer