Dear list, For a university project we are building a melody generator based on markov chains. This includes generating a big matrix of probabilities for playing a certain note. The number of rows in the matrix depends on the markov order and the pitch range that our patch works with. We have a pitch range of 9, hence markov order 3 means writing 9^3 rows = 729. We want to work with the fifth order which means writing 9^5 = 59049 rows into the matrix. We use the coll object from the cyclone library to generate our matrix because it has the handy gui for looking at your matrix after you generated it. Our problem is that it takes more than 3 minutes to write 60000 rows into the coll object and we don't understand why. Attached we have a test patch that simulates our matrix generation. Writing 700 rows into coll takes less than a second, writing 6000 rows takes round about 1.5 seconds (simulating fourth order/9^4), writing 60000 rows takes 3 minutes and 45 seconds. Do you have any hint why it takes that long to write 60000 rows into coll? Maybe we do something wrong or have missed some smarter way to write our rows into the coll matrix.
We are on: MacBook Pro 13" early 2011, macOS Sierra 10.12.6 Pd 0.48-0
Thanks, Hajar and Jakob
Hi Jakob,
using a matrix for markov chains in your use case might not be the best choice as the matrix most likely will be pretty sparse and you quickly run out of space for higher orders. In any case: There exists a markov object which I wrote quite some time ago (originally for Max but later translated to pd). It still compiles and I can send you the sources (or a binary for linux). You have to specify the maximum order on initialization, but you can change the order while it is running to see/hear the effects. For maximum efficiency it used multidimensional rb-trees. I remember that in 1993 it inserted 2000 elements per second for a maxorder of 3 (on a 16MHz machine ;-). Let me know off list whether you're interested and I'll put them online.
-- Orm
Am Samstag, den 30. November 2019 um 14:00:50 Uhr (+0100) schrieb Jakob Laue:
Dear list, For a university project we are building a melody generator based on markov chains. This includes generating a big matrix of probabilities for playing a certain note. The number of rows in the matrix depends on the markov order and the pitch range that our patch works with. We have a pitch range of 9, hence markov order 3 means writing 9^3 rows = 729. We want to work with the fifth order which means writing 9^5 = 59049 rows into the matrix. We use the coll object from the cyclone library to generate our matrix because it has the handy gui for looking at your matrix after you generated it. Our problem is that it takes more than 3 minutes to write 60000 rows into the coll object and we don't understand why. Attached we have a test patch that simulates our matrix generation. Writing 700 rows into coll takes less than a second, writing 6000 rows takes round about 1.5 seconds (simulating fourth order/9^4), writing 60000 rows takes 3 minutes and 45 seconds. Do you have any hint why it takes that long to write 60000 rows into coll? Maybe we do something wrong or have missed some smarter way to write our rows into the coll matrix.
We are on: MacBook Pro 13" early 2011, macOS Sierra 10.12.6 Pd 0.48-0
Thanks, Hajar and Jakob
Pd-list@lists.iem.at mailing list UNSUBSCRIBE and account-management -> https://lists.puredata.info/listinfo/pd-list
Jakob Laue wrote:
Dear list, For a university project we are building a melody generator based on markov chains. This includes generating a big matrix of probabilities for playing a certain note. The number of rows in the matrix depends on the markov order and the pitch range that our patch works with. We have a pitch range of 9, hence markov order 3 means writing 9^3 rows = 729. We want to work with the fifth order which means writing 9^5 = 59049 rows into the matrix. We use the coll object from the cyclone library to generate our matrix because it has the handy gui for looking at your matrix after you generated it.
for what purpose ?
how do you want to "check" 60000 numbers ?
besides that:
1.) there's the vanilla [text] objects that IMHO can handle this sort of things way better. the advantage to [coll] is that new entries into the [text] buffer are updated immediately (i.e. you can see the changes when the [text] window is open as opposed to [coll] where you have to close and reopen the content window)
2.) even with the [text] objects, opening the content window with a lot of numbers like your 60000 WILL take a long time anyway, probably even crash PD. that's a memory thing that very likely applies to the [coll] object as well. so if you really need to check a specific part of your big matrix, you should extract a sniplet from it to another smaller buffer to do so. the help files will help you ;-)
3.) maybe you should use real matrix optimized objects for your task, like the "IEMMATRIX" library or probably even GEM buffer objects
best
oliver
Hey, thanks for your replies! Oliver, your right, the text object is actually pretty cool, I didn't know about that one. I tested it out a little bit. But writing 60000 rows with 10 columns into it takes round about 2 minutes on my machine, so that's obviously too long :P The possibility to check the completely generated matrix is actually more of a comfort thing. We could also use [print] to check if our generated probabilities are correct, but seeing everything together is handy. Anyhow, opening the gui of [coll] or [text] is not our issue. Our issue is the time that it takes to write 60000 rows into these objects. I was wondering, if one could do the process of writing into the matrix on audio~ level. Don't know if that is possible and if that makes any sense though.. I am not that experienced with pd.. Another idea for increasing speed was to use various [zl nth] objects from cyclone. One object for each column. I have not tested that yet. Thanks anyways for the hint on iemmatrix, I did not know about that. And with gem, I have not worked yet. I guess, I will take a look at iemmatrix first.
Thank you, Jakob
Gesendet: Samstag, 30. November 2019 um 15:13 Uhr Von: "oliver" oliver@klingt.org An: Pd-List pd-list@lists.iem.at Betreff: Re: [PD] writing to cyclone/coll takes a long time Jakob Laue wrote:
Dear list, For a university project we are building a melody generator based on markov chains. This includes generating a big matrix of probabilities for playing a certain note. The number of rows in the matrix depends on the markov order and the pitch range that our patch works with. We have a pitch range of 9, hence markov order 3 means writing 9^3 rows = 729. We want to work with the fifth order which means writing 9^5 = 59049 rows into the matrix. We use the coll object from the cyclone library to generate our matrix because it has the handy gui for looking at your matrix after you generated it.
for what purpose ?
how do you want to "check" 60000 numbers ?
besides that:
1.) there's the vanilla [text] objects that IMHO can handle this sort of things way better. the advantage to [coll] is that new entries into the [text] buffer are updated immediately (i.e. you can see the changes when the [text] window is open as opposed to [coll] where you have to close and reopen the content window)
2.) even with the [text] objects, opening the content window with a lot of numbers like your 60000 WILL take a long time anyway, probably even crash PD. that's a memory thing that very likely applies to the [coll] object as well. so if you really need to check a specific part of your big matrix, you should extract a sniplet from it to another smaller buffer to do so. the help files will help you ;-)
3.) maybe you should use real matrix optimized objects for your task, like the "IEMMATRIX" library or probably even GEM buffer objects
best
oliver
Pd-list@lists.iem.at mailing list UNSUBSCRIBE and account-management -> https://lists.puredata.info/listinfo/pd-list
[text] has some performance problems for appending/inserting and even indexing data. if your data only consists of numbers, you could store everything inside a single flat [array]. another possibility is to use data structure arrays where each element holds a "text" field.
Gesendet: Donnerstag, 05. Dezember 2019 um 12:59 Uhr Von: "Jakob Laue" jakkesprinter@web.de An: oliver oliver@klingt.org Cc: Pd-List pd-list@lists.iem.at Betreff: Re: [PD] writing to cyclone/coll takes a long time
Hey, thanks for your replies! Oliver, your right, the text object is actually pretty cool, I didn't know about that one. I tested it out a little bit. But writing 60000 rows with 10 columns into it takes round about 2 minutes on my machine, so that's obviously too long :P The possibility to check the completely generated matrix is actually more of a comfort thing. We could also use [print] to check if our generated probabilities are correct, but seeing everything together is handy. Anyhow, opening the gui of [coll] or [text] is not our issue. Our issue is the time that it takes to write 60000 rows into these objects. I was wondering, if one could do the process of writing into the matrix on audio~ level. Don't know if that is possible and if that makes any sense though.. I am not that experienced with pd.. Another idea for increasing speed was to use various [zl nth] objects from cyclone. One object for each column. I have not tested that yet. Thanks anyways for the hint on iemmatrix, I did not know about that. And with gem, I have not worked yet. I guess, I will take a look at iemmatrix first.
Thank you, Jakob
Gesendet: Samstag, 30. November 2019 um 15:13 Uhr Von: "oliver" oliver@klingt.org An: Pd-List pd-list@lists.iem.at Betreff: Re: [PD] writing to cyclone/coll takes a long time Jakob Laue wrote:
Dear list, For a university project we are building a melody generator based on markov chains. This includes generating a big matrix of probabilities for playing a certain note. The number of rows in the matrix depends on the markov order and the pitch range that our patch works with. We have a pitch range of 9, hence markov order 3 means writing 9^3 rows = 729. We want to work with the fifth order which means writing 9^5 = 59049 rows into the matrix. We use the coll object from the cyclone library to generate our matrix because it has the handy gui for looking at your matrix after you generated it.
for what purpose ?
how do you want to "check" 60000 numbers ?
besides that:
1.) there's the vanilla [text] objects that IMHO can handle this sort of things way better. the advantage to [coll] is that new entries into the [text] buffer are updated immediately (i.e. you can see the changes when the [text] window is open as opposed to [coll] where you have to close and reopen the content window)
2.) even with the [text] objects, opening the content window with a lot of numbers like your 60000 WILL take a long time anyway, probably even crash PD. that's a memory thing that very likely applies to the [coll] object as well. so if you really need to check a specific part of your big matrix, you should extract a sniplet from it to another smaller buffer to do so. the help files will help you ;-)
3.) maybe you should use real matrix optimized objects for your task, like the "IEMMATRIX" library or probably even GEM buffer objects
best
oliver
Pd-list@lists.iem.at mailing list UNSUBSCRIBE and account-management -> https://lists.puredata.info/listinfo/pd-list
Pd-list@lists.iem.at mailing list UNSUBSCRIBE and account-management -> https://lists.puredata.info/listinfo/pd-list
On 2019-12-05 12:59, Jakob Laue wrote:
But writing 60000 rows with 10 columns into it takes round about 2 minutes on my machine, so that's obviously too long
yeah, thought so ...
I was wondering, if one could do the process of writing into the matrix on audio~ level.
yes, that's probably a way better way to go. unfortunately you have to dig there yourself, as i have no experience in that. but it's definitely possible.
Another idea for increasing speed was to use various [zl nth] objects from cyclone. One object for each column. I have not tested that yet. Thanks anyways for the hint on iemmatrix, I did not know about that. And with gem, I have not worked yet. I guess, I will take a look at iemmatrix first.
if you want to stay on the message side, yes examine them (iemmatrix objects). they're quite powerful. since you seem to be experienced with MAX a little bit, GEM objects work similar to JITTER objects, the [pix_buffer] objects can read write matrix data like [jit.matrix], so that's possibly another way to go. just install GEM with deken and have a try at it. (i think you don't even have to turn GEM rendering on for that)
best
oliver
On 30.11.19 14:00, Jakob Laue wrote:
Do you have any hint why it takes that long to write 60000 rows into coll? Maybe we do something wrong or have missed some smarter way to write our rows into the coll matrix.
i don't know what you want to do, but if this is about "writing" (to file), you could also use zexy's [msgfile].
some numbers (on an i7-7700, running Debian/GNU linux, 64bit)
- adding 700 entries: <1ms
- adding 6000 entries: <11ms
- adding 60000 entries: <50ms
- writing 60000 entries to disk: <150ms
- reading 60000 entries from disk: <130ms
- output next entry: <1ms
- seek to entry #20000 <3ms
outputting the "next" entry is O(1). seeking to a position, resp, searching an entry is O(n).
dfmasdr IOhannes
PS: there's also a memory leak that ed.kelly discovered recently, that occurs when dealing with long lines. need to put a new version online....
-
Christof Ressi -
IOhannes m zmoelnig -
Jakob Laue -
oliver -
Orm Finnendahl