Hi Ingo
Thanks for testing!
On Thu, 2011-09-15 at 05:23 +0200, Ingo wrote:
Hi Roman,
the new version works great!
I'm glad to hear that.
I made myself some testing objects around it. Maybe that could be useful if you guys ever get around fixing the help patch.
I'll have a look. Thanks.
I still think the version using individual debyte masks is far more efficient than this one. But as you pointed out this one is more scalable and might take care of boards coming in the future (I have just seen a mega clone with 70 or 72 digital inputs).
Most people don't use incremental wheels timed to 1-2 ms - like I do - anyway. So efficiency shouldn't matter in 99.9% of the cases.
I generally think it does matter. However, i don't have any concerns that the additional table look up causes an efficiency problem. Table lookups are usually very fast.
It's probably a matter of taste, but I often find myself looking for an 'algorithmic' solution instead of copying very similar code several times around, even if the former is a bit less efficient than the latter. In this case, if using several [pd debytemask], it'd be nice to use an abstraction instead. However, if the original [mapping/debytemask] would be used, every (-1) instance would require a row of 8 [+ 8] objects, [+ 16], [+ 24], etc. respectively. So it would either end up with a lot of additional objects below all [debytemask] instances or multiple manually crafted [pd debytemask] with each containing slightly different code (as you implemented it) would be required. The additional [pd polychange] with table lookup is made of just a handful of objects.
However, if it ever turns out, that in your setup the [arduino] abstraction eats a significant amount of CPU power (what I really doubt), I'll happily replace it by your version of [pd digital messages] if it helps.
And yes, the goal should be to cover also 'edge' cases like your incremental wheel. The more use cases work well with Firmata / [arduino] the better.
Roman
-----Ursprüngliche Nachricht----- Von: pd-list-bounces@iem.at [mailto:pd-list-bounces@iem.at] Im Auftrag von Hans-Christoph Steiner Gesendet: Mittwoch, 14. September 2011 22:33 An: Roman Haefeli Cc: pd-list@iem.at Betreff: Re: [PD] pduino rewrite
As Ingo pointed out, one bug is that [mapping/debytemask] has the [change] object for each outlet. So probably the way to fix this is to make a bunch of [mapping/debytemask] objects for all the possible digital ports.
[arduino] should only output on change of digital input, and it receives the digital information one byte/port at a time, i.e. 8 pins. Another approach would be to have an array of all of the previous values which are then compared to the current before outputting.
.hc
On Wed, 2011-09-14 at 11:24 +0200, Roman Haefeli wrote:
Hi Ingo
Thanks for all your reports.
Sorry that my replies sometimes only come a few days later. I'm still willing to fix any outstanding issues, but not very often I find time to get an arduino into my hands. Since sometimes I have troubles following you and keeping your several bug reports apart from each other, I'd suggest to stick with [arduino] bugs and let the documentation aspect aside for a while.
I _think_ I finally understand your problem with the digital ins. I can't currently test or reproduce the problem, since I don't have an arduino at hand, but from reading the code, I think I see what could go wrong.
On certain incoming commands of [pd digital messages], the [pd debytemask] *) subpatch generates more than one message, but only the last one is finally sent to the outlet, because it only fires, when the left inlet of [+ ] is triggered, which is under all circumstances only triggered once after all the [pd debytemask] messages have fired. Actually, the order should be inversed, so that all messages from [pd debytemask] go the _left_ inlet of [+ ], and the summand is sent the _right_ inlet before. This is what I did in the patch you find attached.
I rather have my version going into [arduino], since it is much less code than yours. From what I can tell, they both produce similar output, but as I said, I haven't had the chance to test it in real-world circumstances with a real arduino. So, please test and report back.
I guess the main reason nobody (including me) has noticed this bug yet, is that you won't trigger it, if you only test one digital in at once. Changing the state of only one input at a time makes it seem, that all inputs work correctly. Only when changing states of several inputs at the same time, you will receive only a single digital messages, which is obviously wrong.
I'm happy now that you kept bugging about this. It took me a while to (hopefully) understand the problem. Thanks for your persistence.
*) There is no [debytemask] abstraction anymore in the git version of [arduino]. I replaced it by a subpatch.
Roman
On Sun, 2011-09-11 at 06:20 +0200, Ingo wrote:
There is another thing that I just noticed about the pduino test-
patch.
The mode buttons are suggesting that you can turn of all functions by selecting "NONE". This is not true! These buttons have absolutely NO function and should be replaced with the correct commands. While doing this the option "Input-PullUp" should be added.
The Arduino generally defaults to input. Selecting "NONE" at the
current
state leaves it at the last selected option.
The analogue ins can actually be turned off by the command "analogIns
X 0"
(where the X stands for the pin number 0-5). The digital input pins
need the
command "digitalIns X 0" (where the X stands for the pin number 0-11).
I also think that there should be a separate block for digital an
analogue
(with the available options only) as beginners might think you could
select
"analog" as an option for digital pins, and so on...
Ingo
BTW with the fix I just submitted in my last email all digital ins now
work
flawlessly after testing for several hours. I am amazed that hardly
anybody
noticed is bug for over two years!
Pd-list@iem.at mailing list UNSUBSCRIBE and account-management ->
http://lists.puredata.info/listinfo/pd-list
Pd-list@iem.at mailing list UNSUBSCRIBE and account-management -> http://lists.puredata.info/listinfo/pd-list
The reason why I didn't make an abstraction for the "debyte" is that I wanted to keep the number of files and dependencies as low as possible. I think this was the original idea of the rewrite, right?
Anyway what can be done is add a simple offset number like I did it somewhere on my testing patch. Then you can copy as many instances as needed and offset them. Maybe multiplying by 8 first. But then again it's more objects and calculations than are really necessary. I am using it like this with only two objects for the Duemilanove. Your version with the table has 59 objects while my duplicated version has 73 objects for a Duemilanove while needing a lot less calculations, a fraction of the message transfers and no table lookups or writes.
But as I had mentioned - I doubt that efficiency will play a role in just about any case for the arduino's digital pins.
Ingo
-----Ursprüngliche Nachricht----- Von: Roman Haefeli [mailto:reduzent@gmail.com] Gesendet: Donnerstag, 15. September 2011 08:44 An: Ingo Cc: 'Hans-Christoph Steiner'; pd-list@iem.at Betreff: Re: AW: [PD] pduino rewrite
Hi Ingo
Thanks for testing!
On Thu, 2011-09-15 at 05:23 +0200, Ingo wrote:
Hi Roman,
the new version works great!
I'm glad to hear that.
I made myself some testing objects around it. Maybe that could be useful if you guys ever get around fixing the help patch.
I'll have a look. Thanks.
I still think the version using individual debyte masks is far more efficient than this one. But as you pointed out this one is more
scalable
and might take care of boards coming in the future (I have just seen a
mega
clone with 70 or 72 digital inputs).
Most people don't use incremental wheels timed to 1-2 ms - like I do - anyway. So efficiency shouldn't matter in 99.9% of the cases.
I generally think it does matter. However, i don't have any concerns that the additional table look up causes an efficiency problem. Table lookups are usually very fast.
It's probably a matter of taste, but I often find myself looking for an 'algorithmic' solution instead of copying very similar code several times around, even if the former is a bit less efficient than the latter. In this case, if using several [pd debytemask], it'd be nice to use an abstraction instead. However, if the original [mapping/debytemask] would be used, every (-1) instance would require a row of 8 [+ 8] objects, [+ 16], [+ 24], etc. respectively. So it would either end up with a lot of additional objects below all [debytemask] instances or multiple manually crafted [pd debytemask] with each containing slightly different code (as you implemented it) would be required. The additional [pd polychange] with table lookup is made of just a handful of objects.
However, if it ever turns out, that in your setup the [arduino] abstraction eats a significant amount of CPU power (what I really doubt), I'll happily replace it by your version of [pd digital messages] if it helps.
And yes, the goal should be to cover also 'edge' cases like your incremental wheel. The more use cases work well with Firmata / [arduino] the better.
Roman
-----Ursprüngliche Nachricht----- Von: pd-list-bounces@iem.at [mailto:pd-list-bounces@iem.at] Im Auftrag
von
Hans-Christoph Steiner Gesendet: Mittwoch, 14. September 2011 22:33 An: Roman Haefeli Cc: pd-list@iem.at Betreff: Re: [PD] pduino rewrite
As Ingo pointed out, one bug is that [mapping/debytemask] has the [change] object for each outlet. So probably the way to fix this is
to
make a bunch of [mapping/debytemask] objects for all the possible digital ports.
[arduino] should only output on change of digital input, and it
receives
the digital information one byte/port at a time, i.e. 8 pins. Another approach would be to have an array of all of the previous values which are then compared to the current before outputting.
.hc
On Wed, 2011-09-14 at 11:24 +0200, Roman Haefeli wrote:
Hi Ingo
Thanks for all your reports.
Sorry that my replies sometimes only come a few days later. I'm
still
willing to fix any outstanding issues, but not very often I find
time to
get an arduino into my hands. Since sometimes I have troubles
following
you and keeping your several bug reports apart from each other, I'd suggest to stick with [arduino] bugs and let the documentation
aspect
aside for a while.
I _think_ I finally understand your problem with the digital ins. I can't currently test or reproduce the problem, since I don't have an arduino at hand, but from reading the code, I think I see what could
go
wrong.
On certain incoming commands of [pd digital messages], the [pd debytemask] *) subpatch generates more than one message, but only
the
last one is finally sent to the outlet, because it only fires, when
the
left inlet of [+ ] is triggered, which is under all circumstances
only
triggered once after all the [pd debytemask] messages have fired. Actually, the order should be inversed, so that all messages from
[pd
debytemask] go the _left_ inlet of [+ ], and the summand is sent the _right_ inlet before. This is what I did in the patch you find attached.
I rather have my version going into [arduino], since it is much less code than yours. From what I can tell, they both produce similar
output,
but as I said, I haven't had the chance to test it in real-world circumstances with a real arduino. So, please test and report back.
I guess the main reason nobody (including me) has noticed this bug
yet,
is that you won't trigger it, if you only test one digital in at
once.
Changing the state of only one input at a time makes it seem, that
all
inputs work correctly. Only when changing states of several inputs
at
the same time, you will receive only a single digital messages,
which is
obviously wrong.
I'm happy now that you kept bugging about this. It took me a while
to
(hopefully) understand the problem. Thanks for your persistence.
*) There is no [debytemask] abstraction anymore in the git version
of
[arduino]. I replaced it by a subpatch.
Roman
On Sun, 2011-09-11 at 06:20 +0200, Ingo wrote:
There is another thing that I just noticed about the pduino test-
patch.
The mode buttons are suggesting that you can turn of all functions
by
selecting "NONE". This is not true! These buttons have absolutely
NO
function and should be replaced with the correct commands. While doing this the option "Input-PullUp" should be added.
The Arduino generally defaults to input. Selecting "NONE" at the
current
state leaves it at the last selected option.
The analogue ins can actually be turned off by the command
"analogIns
X 0"
(where the X stands for the pin number 0-5). The digital input
pins
need the
command "digitalIns X 0" (where the X stands for the pin number 0-
11).
I also think that there should be a separate block for digital an
analogue
(with the available options only) as beginners might think you
could
select
"analog" as an option for digital pins, and so on...
Ingo
BTW with the fix I just submitted in my last email all digital ins
now
work
flawlessly after testing for several hours. I am amazed that
hardly
anybody
noticed is bug for over two years!
Pd-list@iem.at mailing list UNSUBSCRIBE and account-management ->
http://lists.puredata.info/listinfo/pd-list
Pd-list@iem.at mailing list UNSUBSCRIBE and account-management -> http://lists.puredata.info/listinfo/pd-list
On Thu, 2011-09-15 at 09:44 +0200, Ingo wrote:
The reason why I didn't make an abstraction for the "debyte" is that I wanted to keep the number of files and dependencies as low as possible. I think this was the original idea of the rewrite, right?
Yeah, exactly. I would like to be able to install [arduino] also on a plain Pd-vanilla setup with the least amount of additional effort. [comport] will always be needed, of course.
Anyway what can be done is add a simple offset number like I did it somewhere on my testing patch. Then you can copy as many instances as needed and offset them. Maybe multiplying by 8 first. But then again it's more objects and calculations than are really necessary. I am using it like this with only two objects for the Duemilanove. Your version with the table has 59 objects while my duplicated version has 73 objects for a Duemilanove while needing a lot less calculations, a fraction of the message transfers and no table lookups or writes.
Interesting. How did you quantify the amount of message transfers? What makes it differ so much, like you say?
Roman
Interesting. How did you quantify the amount of message transfers? What makes it differ so much, like you say?
I simply (roughly) counted the numbers of objects the calculation including all sub processes have to pass until you get the final result. (Unfortunately I cannot tell how heavy each of these calculations is compared to another one.)
I started this a while ago since I am running my machines always at the very limit that they can handle. Which is why I started cutting down the number of processes needed to get something done wherever possible. Saving 20% of the calculations in a machine that's at the limit can make quite a difference. Of course it's the audio processes that are heavier than the control processes.
I remember a discussion here a while ago about how heavy the actual message transfer is. So keeping calculations as simple and straight forward all of the time will keep the machines from getting overloaded earlier than necessary. Which again reminds me that I have to redo lots of old stuff for efficiency - never ending story!
Ingo
On Thu, 2011-09-15 at 10:20 +0200, Ingo wrote:
Interesting. How did you quantify the amount of message transfers? What makes it differ so much, like you say?
I simply (roughly) counted the numbers of objects the calculation including all sub processes have to pass until you get the final result. (Unfortunately I cannot tell how heavy each of these calculations is compared to another one.)
I started this a while ago since I am running my machines always at the very limit that they can handle. Which is why I started cutting down the number of processes needed to get something done wherever possible. Saving 20% of the calculations in a machine that's at the limit can make quite a difference. Of course it's the audio processes that are heavier than the control processes.
I remember a discussion here a while ago about how heavy the actual message transfer is. So keeping calculations as simple and straight forward all of the time will keep the machines from getting overloaded earlier than necessary. Which again reminds me that I have to redo lots of old stuff for efficiency - never ending story!
Ingo
If you want efficiency in this object, you should implement it in C. That should not be hard to do since the Firmata code is C++, but coded mostly in a C style. So you can get all of the parsing and message generating code from Firmata.cpp and StandardFirmata.pde, and make a compiled object out of it.
And Ingo, if you implement a fixed the [debytemask] approach, I'll included it in the Pduino arduino.pd.
.hc
Hi Hans,
unfortunately I am not really good at C or C++ so I have to stick with simplifying within Pd until I get there. But I am actually working on it so I'll be able to replace certain objects in my patches by more efficient externals. Anyway, I think in the case of simplifying the pduino patch another external would be rather contra productive.
The optimized multiple debytemasks (up to 56 input pins) as a Pd-patch are attached. I just called it differently because this was taken from an old display keypad patch that I had done before.
I am using this in my remote control unit and it's working perfectly.
Ingo
-----Ursprüngliche Nachricht----- Von: Hans-Christoph Steiner [mailto:hans@at.or.at] Gesendet: Donnerstag, 15. September 2011 17:48 An: Ingo Cc: 'Roman Haefeli'; pd-list@iem.at Betreff: Re: AW: [PD] pduino rewrite
On Thu, 2011-09-15 at 10:20 +0200, Ingo wrote:
Interesting. How did you quantify the amount of message transfers?
What
makes it differ so much, like you say?
I simply (roughly) counted the numbers of objects the calculation
including
all sub processes have to pass until you get the final result. (Unfortunately I cannot tell how heavy each of these calculations is compared to another one.)
I started this a while ago since I am running my machines always at the
very
limit that they can handle. Which is why I started cutting down the
number
of processes needed to get something done wherever possible. Saving 20%
of
the calculations in a machine that's at the limit can make quite a difference. Of course it's the audio processes that are heavier than the control processes.
I remember a discussion here a while ago about how heavy the actual
message
transfer is. So keeping calculations as simple and straight forward all
of
the time will keep the machines from getting overloaded earlier than necessary. Which again reminds me that I have to redo lots of old stuff
for
efficiency - never ending story!
Ingo
If you want efficiency in this object, you should implement it in C. That should not be hard to do since the Firmata code is C++, but coded mostly in a C style. So you can get all of the parsing and message generating code from Firmata.cpp and StandardFirmata.pde, and make a compiled object out of it.
And Ingo, if you implement a fixed the [debytemask] approach, I'll included it in the Pduino arduino.pd.
.hc
On Thu, 2011-09-15 at 18:54 +0200, Ingo wrote:
Hi Hans,
unfortunately I am not really good at C or C++ so I have to stick with simplifying within Pd until I get there. But I am actually working on it so I'll be able to replace certain objects in my patches by more efficient externals. Anyway, I think in the case of simplifying the pduino patch another external would be rather contra productive.
Makes sense, I think having it as a Pd abstraction is good too, I did write it that way rather than in C :) I was just saying that C would be more efficient.
The optimized multiple debytemasks (up to 56 input pins) as a Pd-patch are attached. I just called it differently because this was taken from an old display keypad patch that I had done before.
I am using this in my remote control unit and it's working perfectly.
The [change -1] is a great idea, I just committed that to bytemask.pd and debytemask.pd. But the [pd resolve-bits_0-7] abstractions seem quite labor-intensive, but they work. I think it would work better to use multiple instances of [debytemask].
.hc
Ingo
-----Ursprüngliche Nachricht----- Von: Hans-Christoph Steiner [mailto:hans@at.or.at] Gesendet: Donnerstag, 15. September 2011 17:48 An: Ingo Cc: 'Roman Haefeli'; pd-list@iem.at Betreff: Re: AW: [PD] pduino rewrite
On Thu, 2011-09-15 at 10:20 +0200, Ingo wrote:
Interesting. How did you quantify the amount of message transfers?
What
makes it differ so much, like you say?
I simply (roughly) counted the numbers of objects the calculation
including
all sub processes have to pass until you get the final result. (Unfortunately I cannot tell how heavy each of these calculations is compared to another one.)
I started this a while ago since I am running my machines always at the
very
limit that they can handle. Which is why I started cutting down the
number
of processes needed to get something done wherever possible. Saving 20%
of
the calculations in a machine that's at the limit can make quite a difference. Of course it's the audio processes that are heavier than the control processes.
I remember a discussion here a while ago about how heavy the actual
message
transfer is. So keeping calculations as simple and straight forward all
of
the time will keep the machines from getting overloaded earlier than necessary. Which again reminds me that I have to redo lots of old stuff
for
efficiency - never ending story!
Ingo
If you want efficiency in this object, you should implement it in C. That should not be hard to do since the Firmata code is C++, but coded mostly in a C style. So you can get all of the parsing and message generating code from Firmata.cpp and StandardFirmata.pde, and make a compiled object out of it.
And Ingo, if you implement a fixed the [debytemask] approach, I'll included it in the Pduino arduino.pd.
.hc
On Thu, 2011-09-15 at 13:29 -0400, Hans-Christoph Steiner wrote:
On Thu, 2011-09-15 at 18:54 +0200, Ingo wrote:
Hi Hans,
unfortunately I am not really good at C or C++ so I have to stick with simplifying within Pd until I get there. But I am actually working on it so I'll be able to replace certain objects in my patches by more efficient externals. Anyway, I think in the case of simplifying the pduino patch another external would be rather contra productive.
Makes sense, I think having it as a Pd abstraction is good too, I did write it that way rather than in C :) I was just saying that C would be more efficient.
The optimized multiple debytemasks (up to 56 input pins) as a Pd-patch are attached. I just called it differently because this was taken from an old display keypad patch that I had done before.
I am using this in my remote control unit and it's working perfectly.
The [change -1] is a great idea, I just committed that to bytemask.pd and debytemask.pd. But the [pd resolve-bits_0-7] abstractions seem quite labor-intensive, but they work. I think it would work better to use multiple instances of [debytemask].
But then you need a row of [+ 8] ([+ 16], [+ 24]) for each instance of debytemask. So, it's still tedious work, whether you're using an abstraction or copies of the subpatch.
Roman
The [change -1] is a great idea, I just committed that to bytemask.pd and debytemask.pd. But the [pd resolve-bits_0-7] abstractions seem quite labor-intensive, but they work. I think it would work better to use multiple instances of [debytemask].
.hc
Not sure what you mean by "labor-intensive", Hans. Are you talking about manually changing 8 numbers per object (which took me less than 1 minute for 56 channels) or are you talking about cpu processing?
Which leads me to the next question: is the Boolean approach using [& 4] and [>> 2] more cpu friendly than using [mod 8] and [div 4]? I don't know how Pd handles such calculations and how it talks to the cpu. I'd be really very interested to find out if there is a difference.
Since the pin numbers are predefined when you are using a [route] object to sort out the groups I don't see the point why the pin number should be calculated again (in this case of multiple instances). This is why I hardcoded them into the message boxes.
I put the two approaches next to each other to see how much simpler my approach is object wise and calculation wise. Still with the question mark which calculation method is more cpu friendly. Anyway changing [mod 8] and [div 4] to [& 4] and [>> 2] shouldn't take more than a minute.
The main difference to Romans approach is that it uses more fixed code to end up doing less when actually working.
BTW I think Romans approach makes generally more sense for most cases since it is scalable and does not need any different code for any number of pins (up to 128 in the current version) which makes it much simpler to use.
I have attached a patch that shows the difference between the two debyte methods.
Ingo
On Fri, 2011-09-16 at 05:57 +0200, Ingo wrote:
The [change -1] is a great idea, I just committed that to bytemask.pd and debytemask.pd. But the [pd resolve-bits_0-7] abstractions seem quite labor-intensive, but they work. I think it would work better to use multiple instances of [debytemask].
.hc
Not sure what you mean by "labor-intensive", Hans. Are you talking about manually changing 8 numbers per object (which took me less than 1 minute for 56 channels) or are you talking about cpu processing?
Which leads me to the next question: is the Boolean approach using [& 4] and [>> 2] more cpu friendly than using [mod 8] and [div 4]?
I was told that it is. Bit shifting and bit mask matching is supposed to be faster than integer division and modulo with an arbitrary (inclusive non-power-of-two integers). However, I can't tell you whether they are really faster in the real world. But you should be able to test it in your own setup with [realtime]. Start [realtime], let [mod 8]-[div 4] process 1 million numbers in 0 logical time, stop [realtime]. Do the same with a [& 4]-[>> 2] chain and compare the results.
I don't know how Pd handles such calculations and how it talks to the cpu. I'd be really very interested to find out if there is a difference.
Since the pin numbers are predefined when you are using a [route] object to sort out the groups I don't see the point why the pin number should be calculated again (in this case of multiple instances). This is why I hardcoded them into the message boxes.
I put the two approaches next to each other to see how much simpler my approach is object wise and calculation wise. Still with the question mark which calculation method is more cpu friendly. Anyway changing [mod 8] and [div 4] to [& 4] and [>> 2] shouldn't take more than a minute.
You could also test the whole [pd digital message] subpatch with the above mentioned approach.
Frankly, I'm not yet convinced that those little improvements in [arduino] will significantly improve the overall Pd performance. Using one less tilde object somewhere in your patch would save some order of magnitudes of CPU power more of what you ever will be able to squeeze out of the [arduino]. Message processing is usually so cheap compared to signal processing, that most often it's hardly worth to focus on the message processing part, unless you deal with message rates of several thousands per second. This is certainly not always true, but in my own experience it most often is.
Roman
On Fri, 2011-09-16 at 11:32 +0200, Roman Haefeli wrote:
On Fri, 2011-09-16 at 05:57 +0200, Ingo wrote:
The [change -1] is a great idea, I just committed that to bytemask.pd and debytemask.pd. But the [pd resolve-bits_0-7] abstractions seem quite labor-intensive, but they work. I think it would work better to use multiple instances of [debytemask].
.hc
Not sure what you mean by "labor-intensive", Hans. Are you talking about manually changing 8 numbers per object (which took me less than 1 minute for 56 channels) or are you talking about cpu processing?
Which leads me to the next question: is the Boolean approach using [& 4] and [>> 2] more cpu friendly than using [mod 8] and [div 4]?
I was told that it is. Bit shifting and bit mask matching is supposed to be faster than integer division and modulo with an arbitrary (inclusive non-power-of-two integers).
It turns out that difference is not significant. On my box, processing 1000000 floats takes ~160ms ([mod],[div]) vs. ~150 ([&],[>>]). Probably all the message parsing overhead is consuming more than the actual computation of the numbers.
Roman
Hi Roman,
Frankly, I'm not yet convinced that those little improvements in [arduino] will significantly improve the overall Pd performance.
Here's the reason why I started really to simplify any patch, no matter if audio or control objects:
I have been programming for about 4 years on one single patch (fulltime - only with breaks to get the hardware/OS going and sampling/editing sampled instruments). You can imagine the amount of code that is in the patch by now.
When I started I thought it was very convenient to use wireless [send/receive] objects to send midi data to the sample-voices (which it is). At a certain point (about 2 years ago) the machine was completely overloaded! Then I measured that a EWI-USB wind controller can send up to 500 midi CC messages per second. I had a function that could multiply the messages to six different midi channels. That makes it 3,000 messages floating around.
The sample voices have at least 500 [receive] objects (there are close to 500 parameters per voice). There were 16 voices which adds up to 8,000 [receive] objects.
Sending 3,000 messages to 8,000 [receive] objects adds up to 24 million times per second that the individual [receive] objects had to check whether the message was meant to be for them or not.
That should be as much data shifting around only for checking [receive] objects as it would take to move the data of several hundreds of audio channels around.
The first fix was easy: assigning the parameter to receive from midi Ch01 if voices are stacked. That cut the message transfer by 6. The second fix was to replace the wireless sends with hard wired patch chords. That took care of most of the rest. The machine was working again. Unfortunately this second fix took 3-4 full months!!!
This is when I decided to think about efficiency in running mode first. If you have a piece of code that has to check between 10 different options and in a certain case only two options are available then it is worth it copying the object and take out all unnecessary options. It's more work while programming but it saves in this particular example several hundred percent cpu time when running.
When such a programming style is used consistently I am sure you can get at least double or more of the performance of a computer. Even with messages where you would think they are not too heavy.
Ingo
-----Ursprüngliche Nachricht----- Von: Roman Haefeli [mailto:reduzent@gmail.com] Gesendet: Freitag, 16. September 2011 11:32 An: Ingo Cc: 'Hans-Christoph Steiner'; pd-list@iem.at Betreff: Re: AW: AW: AW: [PD] pduino rewrite
On Fri, 2011-09-16 at 05:57 +0200, Ingo wrote:
The [change -1] is a great idea, I just committed that to bytemask.pd and debytemask.pd. But the [pd resolve-bits_0-7] abstractions seem quite labor-intensive, but they work. I think it would work better to use multiple instances of [debytemask].
.hc
Not sure what you mean by "labor-intensive", Hans. Are you talking about manually changing 8 numbers per object (which took me less than 1 minute
for
56 channels) or are you talking about cpu processing?
Which leads me to the next question: is the Boolean approach using [& 4]
and
[>> 2] more cpu friendly than using [mod 8] and [div 4]?
I was told that it is. Bit shifting and bit mask matching is supposed to be faster than integer division and modulo with an arbitrary (inclusive non-power-of-two integers). However, I can't tell you whether they are really faster in the real world. But you should be able to test it in your own setup with [realtime]. Start [realtime], let [mod 8]-[div 4] process 1 million numbers in 0 logical time, stop [realtime]. Do the same with a [& 4]-[>> 2] chain and compare the results.
I don't know how Pd handles such calculations and how it talks to the cpu. I'd be really
very
interested to find out if there is a difference.
Since the pin numbers are predefined when you are using a [route] object
to
sort out the groups I don't see the point why the pin number should be calculated again (in this case of multiple instances). This is why I hardcoded them into the message boxes.
I put the two approaches next to each other to see how much simpler my approach is object wise and calculation wise. Still with the question
mark
which calculation method is more cpu friendly. Anyway changing [mod 8]
and
[div 4] to [& 4] and [>> 2] shouldn't take more than a minute.
You could also test the whole [pd digital message] subpatch with the above mentioned approach.
Frankly, I'm not yet convinced that those little improvements in [arduino] will significantly improve the overall Pd performance. Using one less tilde object somewhere in your patch would save some order of magnitudes of CPU power more of what you ever will be able to squeeze out of the [arduino]. Message processing is usually so cheap compared to signal processing, that most often it's hardly worth to focus on the message processing part, unless you deal with message rates of several thousands per second. This is certainly not always true, but in my own experience it most often is.
Roman
Hi Ingo,
On 16/09/11 13:02, Ingo wrote:
When I started I thought it was very convenient to use wireless [send/receive] objects to send midi data to the sample-voices (which it is).
[snip]
Sending 3,000 messages to 8,000 [receive] objects adds up to 24 million times per second that the individual [receive] objects had to check whether the message was meant to be for them or not.
[snip]
The second fix was to replace the wireless sends with hard wired patch chords.
Faced with this scenario I would probably have tried dynamic sends, so the data determines which receive gets the message.
For example:
... | [pack f f f f f f] | [ ; r-$1-$2-$3 $4 $5 $6 (
[r r-1-4-7] | [unpack f f f] | ...
[r r-27-63-49] | [unpack f f f] | ....
And using nested abstractions you could create the receives based on $args, and if you need lots of voices you could use dynamic patching to instantiate them.
Claude
Hi Claude,
When I started I thought it was very convenient to use wireless [send/receive] objects to send midi data to the sample-voices (which it
is). [snip]
Sending 3,000 messages to 8,000 [receive] objects adds up to 24 million times per second that the individual [receive] objects had to check
whether
the message was meant to be for them or not.
[snip]
The second fix was to replace the wireless sends with hard wired patch chords.
Faced with this scenario I would probably have tried dynamic sends, so the data determines which receive gets the message.
For example:
... | [pack f f f f f f] | [ ; r-$1-$2-$3 $4 $5 $6 (
[r r-1-4-7] | [unpack f f f] | ...
[r r-27-63-49] | [unpack f f f] | ....
That would imply that you know which midi CC message gets there when since the left inlet of [pack] needs to be banged. Or it would be delayed until the left inlet receives a new message (if at all). Or you would have to bang the left inlet every time another one comes in. That would even multiply the data transfer. Last solution would be a fixed send timing every couple of milliseconds. That would multiply the average data transfer and lower the timing resolution.
And using nested abstractions you could create the receives based on $args
$args have to listen to all sent messages also. You are simply expanding the name with the $arg. When you have 10 voices all [receive]s of all 10 voices will have to listen for every [send] message to determine whether it is for them or not. Doesn't matter if the name starts with "$0-".
and if you need lots of voices you could use dynamic patching to instantiate them.
To initialize sample-voices like the ones I am using Pd takes about ten seconds. If you want to play a piano chord that has one note more than current voices are present you don't really want to wait 10 seconds, do you? And afterwards are you going to erase that voice again? This would again interrupt the audio stream.
Anyway audio calculation can be turned off with the [switch~] object. [receive] objects cannot be made inactive ever. The only way to do this would be to split up the voices over several independent patches which communicate over [netsend/netreceive] or [osc]. This makes audio communication very difficult and it would be very hard to keep all of those thousands of tables updated in all patches.
It's just simply more efficient to address the data directly by wired connections only to the destination that needs the data. Looks messy but works better!
Ingo
Actually, packing an id before the actual data and using a route object to distribute all separate destinations from one single [receive] -> [route] -> parameters would do the trick. Maybe that's what you meant? I just cannot picture a [route] object with up to 500 outlets, yet.
But there might be ways to organize it using a small number of [receive] and a small number of [route] and sub-[route] objects.
However, it would take just as much time to rewrite an existing patch like this as it takes to hardwire the sends. I still think that these considerations need to be made when starting to write any kind of code because problems only start showing up when it's almost too late. Once the patch gets kinda huge fixing will become very time consuming. Optimizing any code to the least amount of parsing data/messages around is the key for doing any complex patches.
Ingo
-----Ursprüngliche Nachricht----- Von: pd-list-bounces@iem.at [mailto:pd-list-bounces@iem.at] Im Auftrag von Ingo Gesendet: Freitag, 16. September 2011 16:42 An: 'Claude Heiland-Allen'; pd-list@iem.at Betreff: Re: [PD] pduino rewrite
Hi Claude,
When I started I thought it was very convenient to use wireless [send/receive] objects to send midi data to the sample-voices (which
it
is). [snip]
Sending 3,000 messages to 8,000 [receive] objects adds up to 24
million
times per second that the individual [receive] objects had to check
whether
the message was meant to be for them or not.
[snip]
The second fix was to replace the wireless sends with hard wired patch chords.
Faced with this scenario I would probably have tried dynamic sends, so the data determines which receive gets the message.
For example:
... | [pack f f f f f f] | [ ; r-$1-$2-$3 $4 $5 $6 (
[r r-1-4-7] | [unpack f f f] | ...
[r r-27-63-49] | [unpack f f f] | ....
That would imply that you know which midi CC message gets there when since the left inlet of [pack] needs to be banged. Or it would be delayed until the left inlet receives a new message (if at all). Or you would have to bang the left inlet every time another one comes in. That would even multiply the data transfer. Last solution would be a fixed send timing every couple of milliseconds. That would multiply the average data transfer and lower the timing resolution.
And using nested abstractions you could create the receives based on $args
$args have to listen to all sent messages also. You are simply expanding the name with the $arg. When you have 10 voices all [receive]s of all 10 voices will have to listen for every [send] message to determine whether it is for them or not. Doesn't matter if the name starts with "$0-".
and if you need lots of voices you could use dynamic patching to instantiate them.
To initialize sample-voices like the ones I am using Pd takes about ten seconds. If you want to play a piano chord that has one note more than current voices are present you don't really want to wait 10 seconds, do you? And afterwards are you going to erase that voice again? This would again interrupt the audio stream.
Anyway audio calculation can be turned off with the [switch~] object. [receive] objects cannot be made inactive ever. The only way to do this would be to split up the voices over several independent patches which communicate over [netsend/netreceive] or [osc]. This makes audio communication very difficult and it would be very hard to keep all of those thousands of tables updated in all patches.
It's just simply more efficient to address the data directly by wired connections only to the destination that needs the data. Looks messy but works better!
Ingo
Pd-list@iem.at mailing list UNSUBSCRIBE and account-management -> http://lists.puredata.info/listinfo/pd-list
Wow, I just compared your version of [pd digital message] with mine and yours takes only 180ms to process 1000000 of messages, while mine uses over 8s. Frankly, I wouldn't have expected such a big difference.... Let me dig into this.
Roman
On Fri, 2011-09-16 at 05:57 +0200, Ingo wrote:
The [change -1] is a great idea, I just committed that to bytemask.pd and debytemask.pd. But the [pd resolve-bits_0-7] abstractions seem quite labor-intensive, but they work. I think it would work better to use multiple instances of [debytemask].
.hc
Not sure what you mean by "labor-intensive", Hans. Are you talking about manually changing 8 numbers per object (which took me less than 1 minute for 56 channels) or are you talking about cpu processing?
Which leads me to the next question: is the Boolean approach using [& 4] and [>> 2] more cpu friendly than using [mod 8] and [div 4]? I don't know how Pd handles such calculations and how it talks to the cpu. I'd be really very interested to find out if there is a difference.
Since the pin numbers are predefined when you are using a [route] object to sort out the groups I don't see the point why the pin number should be calculated again (in this case of multiple instances). This is why I hardcoded them into the message boxes.
I put the two approaches next to each other to see how much simpler my approach is object wise and calculation wise. Still with the question mark which calculation method is more cpu friendly. Anyway changing [mod 8] and [div 4] to [& 4] and [>> 2] shouldn't take more than a minute.
The main difference to Romans approach is that it uses more fixed code to end up doing less when actually working.
BTW I think Romans approach makes generally more sense for most cases since it is scalable and does not need any different code for any number of pins (up to 128 in the current version) which makes it much simpler to use.
I have attached a patch that shows the difference between the two debyte methods.
Ingo
Wow, I just compared your version of [pd digital message] with mine and yours takes only 180ms to process 1000000 of messages, while mine uses over 8s. Frankly, I wouldn't have expected such a big difference.... Let me dig into this.
Roman
That's more than I would have expected, too! I would have been guessing it could be up to 10x as fast but not 50x.
Ingo
On Fri, 2011-09-16 at 14:05 +0200, Ingo wrote:
Wow, I just compared your version of [pd digital message] with mine and yours takes only 180ms to process 1000000 of messages, while mine uses over 8s. Frankly, I wouldn't have expected such a big difference.... Let me dig into this.
Roman
That's more than I would have expected, too! I would have been guessing it could be up to 10x as fast but not 50x.
I think I'm going to put your much more efficient version into the git version.
Roman
To make sure boards that are larger than 56 digital in pins you should copy a couple more of these objects to go up to 128. Of course since [&] and [>>] seems to be slightly faster that would be the choice.
To be even more efficient the object [pd route digital/analog] should be bypassed by adding the addresses [144 145 146 ... 151] to the route object inside the parent patch making it [route 249 240 144 145 146 147 148 149 150 151]. The last outlet goes into [pd route digital/analog].
The [route 0 1 2 3 4 5 6 7] inside the [pd digital messages] should be replace by 8 individual inlets.
BTW you could keep going on with this forever ...
All I wanted originally was to get the correct messages coming out of the patch ...
Ingo
-----Ursprüngliche Nachricht----- Von: Roman Haefeli [mailto:reduzent@gmail.com] Gesendet: Freitag, 16. September 2011 14:44 An: Ingo Cc: 'Hans-Christoph Steiner'; pd-list@iem.at Betreff: Re: AW: AW: AW: AW: [PD] pduino rewrite
On Fri, 2011-09-16 at 14:05 +0200, Ingo wrote:
Wow, I just compared your version of [pd digital message] with mine
and
yours takes only 180ms to process 1000000 of messages, while mine uses over 8s. Frankly, I wouldn't have expected such a big difference.... Let me dig into this.
Roman
That's more than I would have expected, too! I would have been guessing it could be up to 10x as fast but not 50x.
I think I'm going to put your much more efficient version into the git version.
Roman
On Thu, 2011-09-15 at 10:01 +0200, Roman Haefeli wrote:
On Thu, 2011-09-15 at 09:44 +0200, Ingo wrote:
The reason why I didn't make an abstraction for the "debyte" is that I wanted to keep the number of files and dependencies as low as possible. I think this was the original idea of the rewrite, right?
Yeah, exactly. I would like to be able to install [arduino] also on a plain Pd-vanilla setup with the least amount of additional effort. [comport] will always be needed, of course.
Well, now you can and trivially install all but one of the dependencies for 'puredata' aka Pd vanilla using:
apt-get install pd-cyclone pd-mapping pd-zexy
Only moocow is missing. I'd bet it'll be much less work to package moocow then to rewrite and manage a fork of arduino.pd.
.hc
Anyway what can be done is add a simple offset number like I did it somewhere on my testing patch. Then you can copy as many instances as needed and offset them. Maybe multiplying by 8 first. But then again it's more objects and calculations than are really necessary. I am using it like this with only two objects for the Duemilanove. Your version with the table has 59 objects while my duplicated version has 73 objects for a Duemilanove while needing a lot less calculations, a fraction of the message transfers and no table lookups or writes.
Interesting. How did you quantify the amount of message transfers? What makes it differ so much, like you say?
Roman
On Thu, 2011-09-15 at 11:36 -0400, Hans-Christoph Steiner wrote:
On Thu, 2011-09-15 at 10:01 +0200, Roman Haefeli wrote:
On Thu, 2011-09-15 at 09:44 +0200, Ingo wrote:
The reason why I didn't make an abstraction for the "debyte" is that I wanted to keep the number of files and dependencies as low as possible. I think this was the original idea of the rewrite, right?
Yeah, exactly. I would like to be able to install [arduino] also on a plain Pd-vanilla setup with the least amount of additional effort. [comport] will always be needed, of course.
Well, now you can and trivially install all but one of the dependencies for 'puredata' aka Pd vanilla using:
apt-get install pd-cyclone pd-mapping pd-zexy
Only moocow is missing. I'd bet it'll be much less work to package moocow then to rewrite and manage a fork of arduino.pd.
I'm not sure about this, but I mostly agree with you. When packaging arduino as a pd-lib.deb, it would be trivial to add the dependencies.
However, I find I found some rather ugly stuff inside [arduino] that I definitely wanted to get rid of, such as [prepend] from cyclone.
On the long run, I don't see the point in having two different [arduino] classes to be maintained. If at some point, improvements of both can be merged to one class, I'm all for it. Even if it means re-adding some externals. But for stuff that is very trivial to do with vanilla classes, I don't see the point in using externals. And no, I really don't think that adds a lot of maintenance load to the class.
Roman
On Thu, 2011-09-15 at 18:43 +0200, Roman Haefeli wrote:
On Thu, 2011-09-15 at 11:36 -0400, Hans-Christoph Steiner wrote:
On Thu, 2011-09-15 at 10:01 +0200, Roman Haefeli wrote:
On Thu, 2011-09-15 at 09:44 +0200, Ingo wrote:
The reason why I didn't make an abstraction for the "debyte" is that I wanted to keep the number of files and dependencies as low as possible. I think this was the original idea of the rewrite, right?
Yeah, exactly. I would like to be able to install [arduino] also on a plain Pd-vanilla setup with the least amount of additional effort. [comport] will always be needed, of course.
Well, now you can and trivially install all but one of the dependencies for 'puredata' aka Pd vanilla using:
apt-get install pd-cyclone pd-mapping pd-zexy
Only moocow is missing. I'd bet it'll be much less work to package moocow then to rewrite and manage a fork of arduino.pd.
I'm not sure about this, but I mostly agree with you. When packaging arduino as a pd-lib.deb, it would be trivial to add the dependencies.
However, I find I found some rather ugly stuff inside [arduino] that I definitely wanted to get rid of, such as [prepend] from cyclone.
I think that prepend works better, that's why. No need for [list trim]. With cyclone/prepend being in Pd-extended and Debian, it doesn't seem like too hard a thing to install it when you need it. I'm open to being proven wrong on cyclone's prepend working better.
On the long run, I don't see the point in having two different [arduino] classes to be maintained. If at some point, improvements of both can be merged to one class, I'm all for it. Even if it means re-adding some externals. But for stuff that is very trivial to do with vanilla classes, I don't see the point in using externals. And no, I really don't think that adds a lot of maintenance load to the class.
Maintenance is one part of it, another is so that you don't have to copy-n-paste subpatches in cases like multiple [debytemask]s, you just make as many instances as you need.
Another good reason is that there are useful bits of code developed while writing the arduino.pd object, why not share them as objects?
.hc
On Thu, 2011-09-15 at 13:19 -0400, Hans-Christoph Steiner wrote:
On Thu, 2011-09-15 at 18:43 +0200, Roman Haefeli wrote:
On Thu, 2011-09-15 at 11:36 -0400, Hans-Christoph Steiner wrote:
On Thu, 2011-09-15 at 10:01 +0200, Roman Haefeli wrote:
On Thu, 2011-09-15 at 09:44 +0200, Ingo wrote:
The reason why I didn't make an abstraction for the "debyte" is that I wanted to keep the number of files and dependencies as low as possible. I think this was the original idea of the rewrite, right?
Yeah, exactly. I would like to be able to install [arduino] also on a plain Pd-vanilla setup with the least amount of additional effort. [comport] will always be needed, of course.
Well, now you can and trivially install all but one of the dependencies for 'puredata' aka Pd vanilla using:
apt-get install pd-cyclone pd-mapping pd-zexy
Only moocow is missing. I'd bet it'll be much less work to package moocow then to rewrite and manage a fork of arduino.pd.
I'm not sure about this, but I mostly agree with you. When packaging arduino as a pd-lib.deb, it would be trivial to add the dependencies.
However, I find I found some rather ugly stuff inside [arduino] that I definitely wanted to get rid of, such as [prepend] from cyclone.
I think that prepend works better, that's why. No need for [list trim]. With cyclone/prepend being in Pd-extended and Debian, it doesn't seem like too hard a thing to install it when you need it. I'm open to being proven wrong on cyclone's prepend working better.
It's probably again a matter of taste, but cyclone is one the libraries I dislike the most. It has ugly class names (some of them with upper case), inconsistent ordering of inlets, it's very much un-Pd-like, since this is even it's purpose (emulating max object classes). IMHO, if it can so easily avoided to be used, then I'd rather not use it at all.
On the long run, I don't see the point in having two different [arduino] classes to be maintained. If at some point, improvements of both can be merged to one class, I'm all for it. Even if it means re-adding some externals. But for stuff that is very trivial to do with vanilla classes, I don't see the point in using externals. And no, I really don't think that adds a lot of maintenance load to the class.
Maintenance is one part of it, another is so that you don't have to copy-n-paste subpatches in cases like multiple [debytemask]s, you just make as many instances as you need.
As I said previously, there is a much cleaner approach, that doesn't even need many [debytemask] instances, which I implemented in the git version.
Another good reason is that there are useful bits of code developed while writing the arduino.pd object, why not share them as objects?
Can you name one? I don't know, if [debytemask] was developed for the [arduino], but while it is generally a very useful class, it cannot be used in [arduino] as is.
Roman
-
Claude Heiland-Allen -
Hans-Christoph Steiner -
Ingo -
Roman Haefeli