Hey all
Sometimes when I load certain patches containing data structures, all data structures in all patches of an instance of Pd are redrawn. This is barely noticeable with one single patch, but if many patches are loaded with each using many scalars, the whole redrawing process becomes unbearable. In larger netpd sessions, the process can take a few dozen seconds, depending on number of loaded instruments and probably other factors (maybe CPU speed, but none of the cores is fully loaded during the redrawing process). During the process, GUI elements like slider handles or number boxes are not updated and all DS based GUIs are not usable either. [cnv] labels are not affected (why?). The redrawing seems to cause drop outs in DSP processing. I got reports that glitches are more pronounced on Windows (or on slower computers?).
I tried to figure out what exact conditions trigger such a redraw in the hope that they can be avoided. I found that the order objects of a data structure template are created matters. If [drawsymbol] was created before [struct], loading such a patch triggers the redraw. After changing the creation order, loading the patch doesn't trigger the redraw. This video illustrates it:
https://netpd.org/~roman/tmp/ds-redrawing-everything.webm
Apparently, there are other conditions that trigger a redraw. I tried to make sure that all [struct] objects are created last in one of my patches, but loading the patch still triggers the redraw.
What are the conditions that lead to a redraw? Why are all data structures redrawn and not only the ones using scalars from the changed template? If order matters, why aren't the patches saved with the correct order? Why is the process so slow if it is apparently not CPU- bound? Is it because of the "chattiness" between GUI and core?
Here another video showing how redrawing affects iemguis:
https://netpd.org/~roman/tmp/ds-redrawing-affects-iemguis.webm
Roman
On Sat, 2023-03-04 at 23:17 +0100, Roman Haefeli wrote:
Sometimes when I load certain patches containing data structures, all data structures in all patches of an instance of Pd are redrawn.
As a follow-up on the matter, I identified to conditions that trigger a redraw:
Loading a patch where some scalar objects like [draw{symbol,number,text}] or [{draw,filled}{curve,polygon}] are created before the [struct] oject of the template
Loading a patch where the [struct] uses a template name containing $0 like $0.fruit (!).
$0 seems to be treated specially, since using $1, $2, etc. is not affected. Also, it doesn't help to put [struct $1 float x] into an abstraction [mystruct $0.fruit].
Interestingly, if I load the patch without the [struct] object first, and then create [sruct $0.fruit float x] manually, it doesn't trigger the redraw. I'll try if I can work-around unwanted (and probably unnecessary) redraws by dynamically creating [struct] objects instead of loading them with the patch (as I cannot abstain from using $0).
Anyway, I hoped I could to get some insight in the matter first, but I am more and more getting a sense that this deserves to be reported as an issue.
Roman
Not much of a help, just a couple of pointers:
I just opened a patch with several jmmmp/multiarray and help patches of several of my jmmmp objects using data structures, and didn't notice something like that. But, I always use $0-names for templates. I would think having less scalars for the same template would not affect existing scalars of a given template.
Is your test patch anywhere ready to test? (preferably not "go here to download the whole library, then go there and do this, ect.")
The delay on other gui objects is normal, is just pd overloading the system with anything gui-related.
It seems to me that gui peformance has gotten worse in 0.53.1, but I haven't been using Pd enough to pinpoint it. If affects mostly GOP and loading/closing speeds, maybe not related with this.
On Sat, 2023-03-04 at 23:17 +0100, Roman Haefeli wrote:
Sometimes when I load certain patches containing data structures, all data structures in all patches of an instance of Pd are redrawn.
As a follow-up on the matter, I identified to conditions that trigger a redraw:
Loading a patch where some scalar objects like [draw{symbol,number,text}] or [{draw,filled}{curve,polygon}] are created before the [struct] oject of the template
Loading a patch where the [struct] uses a template name containing $0 like $0.fruit (!).
$0 seems to be treated specially, since using $1, $2, etc. is not affected. Also, it doesn't help to put [struct $1 float x] into an abstraction [mystruct $0.fruit].
Interestingly, if I load the patch without the [struct] object first, and then create [sruct $0.fruit float x] manually, it doesn't trigger the redraw. I'll try if I can work-around unwanted (and probably unnecessary) redraws by dynamically creating [struct] objects instead of loading them with the patch (as I cannot abstain from using $0).
Anyway, I hoped I could to get some insight in the matter first, but I am more and more getting a sense that this deserves to be reported as an issue.
Roman
Pd-list@lists.iem.at mailing list UNSUBSCRIBE and account-management
On Wed, 2023-03-08 at 14:13 +0100, João Pais wrote:
I just opened a patch with several jmmmp/multiarray and help patches of several of my jmmmp objects using data structures, and didn't notice something like that. But, I always use $0-names for templates.
When scalars are not overlapping, you don't see the redraw, but you still experience the lag. It seems the lag is proportional to the overall number of scalars in Pd.
I would think having less scalars for the same template would not affect existing scalars of a given template.
My concern is exactly that the current behavior scales badly because _all_ scalars are redrawn, not only the ones belonging to the affected template.
Is your test patch anywhere ready to test? (preferably not "go here to download the whole library, then go there and do this, ect.")
Here a set of patches that aim to illustrate my findings: https://netpd.org/~roman/tmp/illustrate-ds-redraw.zip
First open 01-illiustrate-redraw-with-many-scalars.pd. It will draw 1600 lines that are overlapping, so that the redraw is clearly visible.
Then open the other patches. Depending on how they are created, they trigger a redraw at load time. Read the comments in the patches.
It takes exactly 3 seconds to redraw 1600 scalars on my machine. I wonder if it takes longer on Windows (that's what some reports suggest).
The delay on other gui objects is normal, is just pd overloading the system with anything gui-related.
That's what I've been thinking for years and now I'm starting to believe that the redraws are unnecessary and avoiding them could be possible. What I'm struggling right now is how to create re-usable abstractions that use data structures. Using $0 in struct names seems to always force a redraw. So, I'm trying to find a way around that. If replace all instances of $0 in a ds patch with a random number, loading it doesn't trigger a redraw. What makes $0 special?
It seems to me that gui peformance has gotten worse in 0.53.1, but I haven't been using Pd enough to pinpoint it. If affects mostly GOP and loading/closing speeds, maybe not related with this.
My subjective sense is similar, but I haven't actually verified it. However, the lag caused by loading patches with data structures has been bugging my for years, way before 0.53.1, I just wasn't able to pinpoint it before.
Roman
Is your test patch anywhere ready to test? (preferably not "go here to download the whole library, then go there and do this, ect.")
Here a set of patches that aim to illustrate my findings: https://netpd.org/~roman/tmp/illustrate-ds-redraw.zip
First open 01-illiustrate-redraw-with-many-scalars.pd. It will draw 1600 lines that are overlapping, so that the redraw is clearly visible.
Then open the other patches. Depending on how they are created, they trigger a redraw at load time. Read the comments in the patches.
It takes exactly 3 seconds to redraw 1600 scalars on my machine. I wonder if it takes longer on Windows (that's what some reports suggest).
Redraw times when opening patches (counted in my head):
2 - 15s
3 - 0s
4 - also around 15s, this seems to be the reference value no matter
which other patches are opened.
This on W10, Thinkcentre, 2x2Ghz, 16Gb Ram.
Opening and closing my help patches when your patch is open also causes a redraw (same duration as above) or hangs Pd for a while. This surely because they also use $0-names in the templates.
The delay on other gui objects is normal, is just pd overloading the system with anything gui-related.
That's what I've been thinking for years and now I'm starting to believe that the redraws are unnecessary and avoiding them could be possible. What I'm struggling right now is how to create re-usable abstractions that use data structures. Using $0 in struct names seems to always force a redraw. So, I'm trying to find a way around that. If replace all instances of $0 in a ds patch with a random number, loading it doesn't trigger a redraw. What makes $0 special?
Couldn't say it, as I don't know anything below the surface of pd. Maybe some problem assigning/reading the $0 into the templates.
You could also try to use $0 as parameter for scalars or template arguments, but not in the template name, to see if that makes a difference.
It seems to me that gui peformance has gotten worse in 0.53.1, but I haven't been using Pd enough to pinpoint it. If affects mostly GOP and loading/closing speeds, maybe not related with this.
My subjective sense is similar, but I haven't actually verified it. However, the lag caused by loading patches with data structures has been bugging my for years, way before 0.53.1, I just wasn't able to pinpoint it before.
My clicktracker patch (which is one GOP object) takes ages to load now, it wasn't like that a couple of months ago. (nothing changed in it during this time)
On Thu, 2023-03-09 at 22:14 +0100, João Pais wrote:
Redraw times when opening patches (counted in my head):
2 - 15s
3 - 0s
4 - also around 15s, this seems to be the reference value no matter
which other patches are opened.
This on W10, Thinkcentre, 2x2Ghz, 16Gb Ram.
Ok, thanks for testing.
My clicktracker patch (which is one GOP object) takes ages to load now, it wasn't like that a couple of months ago. (nothing changed in it during this time)
Interestingly, only redrawing takes long time with data structures. When loading the example patch, the scalars are drawn almost instantaneously (and they are created on the fly at loadbang).
So, I suspect slower loading time is a separate issue, probably unrelated to the redrawing. You could load your patch in a previous version of Pd to check if it makes a difference.
Roman
On 3/8/23 14:13, João Pais wrote:
It seems to me that gui peformance has gotten worse in 0.53.1, but I haven't been using Pd enough to pinpoint it. If affects mostly GOP and loading/closing speeds, maybe not related with this.
i would be *highly* interested in any patches demonstrating such a performance loss.
mgdsar IOhannes
I'm not using Pd much, so I could only mention my http://jmmmp.github.io/clicktracker/ From what I "feel", it takes longer now to draw when loaded (it's just a GOP patch), and also a bit more to open the GOP to see its contents. And the same again when closing the canvas+patch.
I just clicked around in my jmmmp help folder, and here are some help files that seem to take longer to load than others. I'm not sure if this is good enough to report something interesting: multiarray, matrixctrl, jp.coll, gui-edit,
E.g. the most complex patch, Granulator, opens as quick as one of the simplest ones. So I guess the delay isn't connected with the size of the patch itself.
System: W10, Pd 0.53.1
On 3/8/23 14:13, João Pais wrote:
It seems to me that gui peformance has gotten worse in 0.53.1, but I haven't been using Pd enough to pinpoint it. If affects mostly GOP and loading/closing speeds, maybe not related with this.
i would be *highly* interested in any patches demonstrating such a performance loss.
mgdsar IOhannes
Pd-list@lists.iem.at mailing list UNSUBSCRIBE and account-management -> https://lists.puredata.info/listinfo/pd-list
On Fri, 2023-03-17 at 19:45 +0100, João Pais wrote:
I'm not using Pd much, so I could only mention my http://jmmmp.github.io/clicktracker/%C2%A0From what I "feel", it takes longer now to draw when loaded (it's just a GOP patch), and also a bit more to open the GOP to see its contents. And the same again when closing the canvas+patch.
I found the culprit regarding unnecessary scalar redraws, or at least I found a way to make it bearable when not editing:
https://github.com/pure-data/pure-data/pull/1933
I made a Windows build with the suppress-redraws patch applied:
https://netpd.org/~roman/tmp/pd-0.53.1-suppress-redraw-msw.zip
I wonder if it makes things quicker on your end. Also, I'd be interested if you notice any negative side effects.
Roman
On Sat, 2023-03-25 at 18:39 +0100, Roman Haefeli wrote:
On Fri, 2023-03-17 at 19:45 +0100, João Pais wrote:
I'm not using Pd much, so I could only mention my http://jmmmp.github.io/clicktracker/%C2%A0From what I "feel", it takes longer now to draw when loaded (it's just a GOP patch), and also a bit more to open the GOP to see its contents. And the same again when closing the canvas+patch.
I found the culprit regarding unnecessary scalar redraws, or at least I found a way to make it bearable when not editing:
https://github.com/pure-data/pure-data/pull/1933
I made a Windows build with the suppress-redraws patch applied:
https://netpd.org/~roman/tmp/pd-0.53.1-suppress-redraw-msw.zip
I wonder if it makes things quicker on your end. Also, I'd be interested if you notice any negative side effects.
Having said that, I am not even sure if your observed slow loading is related to this at all. I _did_ some observations where loading of new stuff got slower when there were many scalars visible and quicker when there were no scalars visible.
Of course, there could be other reasons for slow loading. My anecdotal experience says I/O related operations are often slower on Windows than Linux. Virus scanners tend to slow down things (I/O) even more.
Roman
On Wed, 2023-03-08 at 00:13 +0100, Roman Haefeli wrote:
As a follow-up on the matter, I identified to conditions that trigger a redraw:
* Loading a patch where some scalar objects like [draw{symbol,number,text}] or [{draw,filled}{curve,polygon}] are created before the [struct] oject of the template
* Loading a patch where the [struct] uses a template name containing $0 like $0.fruit (!).
$0 seems to be treated specially, since using $1, $2, etc. is not affected. Also, it doesn't help to put [struct $1 float x] into an abstraction [mystruct $0.fruit].
I thought I might could skip using $0 in structs by creating them as singleton and reference that (instead of the localized $0-version) in all instances of the abstraction. But then I noticed that I'm often using the outlet of [struct] to detect clicks.
Assuming I have spread scalars of the same [struct] to different canvases, is there a way to detect from the click message which canvas the click originates from? In other words: Is there some way to associate a pointer to a scalar with the containing canvas?
Roman
-
IOhannes m zmölnig -
João Pais -
Roman Haefeli