oops, re-sending....
On 30.09.2024 19:19, Miller Puckette wrote:
On 9/30/24 6:23 PM, Christof Ressi wrote:
Suppose we replace "draw X, Y,Z" with "draw button". I think this then spins out in either of two possible ways:
(1) the "button" widget on the GUI side now takes care of mouse-hit detection and making itself flash (and eventually widgets with text can grab focus, handle copy/paste, etc). In this case the GUI widget has to maintain a connection with its corresponding object in Pd. This gets very complicated when, for instance, changing text causes inlets or outlets to appear, which perhaps on the Pd side the text is getting hammered by messages from elsewhere at the same time. Distributed database management, anyone?
(2) it's just a drawing with a tag, not an active widget; perhaps mouse hit detection is done in the GUI but everything else in Pd. In this case it's not a big enough change to make much difference, except for this noisome one: "GUI externals" such as the knob now have to load dynamic libraries into two programs, instead of just one.
Depending on which scenario we want to consider (or perhaps this is somehow a false choice) we can go into more detail about this...
These are important points! I think there are two separate concerns:
who is doing the mouse-hit-detection?
who initiates the state changes (e.g. flashing, selection)
For example, it would be possible to do the hit detection on the GUI, but still do the state changes on the core. For example, in IOhannes draft PR, the "bang" object has an "activate" message that is sent whenever the flash state changes: https://urldefense.com/v3/__https://github.com/pure-data/pure-data/pull/1765...
We could move such internal state changes to the GUI, but we don't have to if it only complicates things.
However, I would love to move the hit detection to the GUI! Note that some GUI frameworks already do hit detection for canvas items. (In Qt, graphic items in a QGraphicsScene can receive mouse events!) In this case, doing the hit detection on the core is just duplicate work. It is also rather expensive since the core only stores the objects in a linked list, without spatial indexing, so every mouse movement has to potentially iterate over hundreds or even thousands gobjects and do rectangle intersection tests until it finds a hit. This is not something that you would want to do on the audio thread :)
The same thing applies to selections. The GUI typically already knows which objects are selected, so we only need to send this information to the core.
If we do the hit detection on the GUI, you are right that we need to keep an association between GUI widget and gobject. I think we just need to have a hashtable of tag -> gobject on the core; when the core receives a mouse event, it looks up the tag in the hashtable and then dispatches the event to the corresponding object. If the tag is not found, it means that the object has been destroyed in flight and we can just ignore the event.
There's one slight problem: some objects need to know *where* the user clicked. E.g. a slider with steady-on-click needs to know the position to set the new value. This means that the core side has to maintain some knowledge about the graphical representation. This does not seem to much of a problem to me, but it's something to be aware of.
One idea would be to do the hit detection in the GUI and simply pass Pd the (x,y) coords along with a list of all the GUI objects that got hit (each with a "fuzziness" quotient so that we can be smart about disambiguating multiple hits). The only reason I didn't do that already is because I didn't want to get too deeply involved with TclTk's canvases. And yes, anything we kick up to the GUI has to be done both in TCL/TK and in whatever 5 new GUIs are tried out -- that's one reason I've kept as much of the intelligence as possible down in Pd land.
I also hate the way every canvas I've seen does rectangle-selection. Maybe you know of one I won't hate :)
And... every widget set I've seen thinks widgets live in rectangles. The second most important graphical item in Pd is the patch line. Giving it a widget-style selection method? Disaster. I don't think patch lines can be TK or GTK r QT or anyone's widgets - imagine how those bounding rectangles would fill th screen :)
Now here's the big catch: what should we do with GUI externals?
Ad 1: externals could define a region that acts as the bounding rectangle and which can be filled with drawing commands. This way they can partipicate in hit detection on the GUI.
Ad 2: I think state changes for externals have to be handled on the core because the external has no access to the GUI features. It does not even know which GUI it runs in.
One idea I had was to provide the GUI features as API functions. E.g. pdgui_drawrect() to draw a rectangle or pdgui_delay() to start a timer, etc. These API functions have to be implemented by the GUI, the core would just delegate the calls to the GUI. The external would register a callback function that is called whenever the object needs to be redrawn. Also, the GUI external could register callbacks for mouse events and paintings
One problem I see is that not every GUI can export C functions. For example, this probably wouldn't work with a web browser GUI...
The easiest solution is to just keep state changes for external GUI objects on the core, i.e. the external needs to send raw drawing commands. As an upside, this would make it easier to update existing externals as they would just have to update the drawing commands. Of course, as I said, sending all these individual commands not only increases traffic, it also means that the GUI has to maintain individual graphical primitives instead of just a single widget.
Either way, we should provide a stable GUI/drawing API for externals. Currently, it's all unofficial.
Thanks to Antoine for mentioning pdlua! I will need to check it out. I have already seen people doing some very cool stuff with it. Maybe they already have the perfect solution :)
I'm also looking at pdlua, which might want to become the official way to make graphical externs AND to add scripting in a fundamental way to Pd :)
I think all of this needs quite a bit of planning and design. We already had an online meeting two or three years ago before we started the GUI refactoring process. I would suggest to have another meeting where we can talk through all the potential issues and get a feel for the overall design.
Anyway, I'm very excited that the GUI refactoring is moving forward!
Christof
cheers
M
pd-dev@lists.iem.at - the Pd developers' mailinglist https://urldefense.com/v3/__https://lists.iem.at/hyperkitty/list/pd-dev@list...
pd-dev@lists.iem.at - the Pd developers' mailinglist https://urldefense.com/v3/__https://lists.iem.at/hyperkitty/list/pd-dev@list...
I also hate the way every canvas I've seen does rectangle-selection. Maybe you know of one I won't hate :)
And... every widget set I've seen thinks widgets live in rectangles. The second most important graphical item in Pd is the patch line. Giving it a widget-style selection method? Disaster. I don't think patch lines can be TK or GTK r QT or anyone's widgets - imagine how those bounding rectangles would fill th screen :)
Qt only uses bounding rectangles for the broad phase hit detection. The actual hit detection can then be further customized for each widget. The collision shape can be anything from a circle to an arbitrary polygon or even a bitmap. For example, a line item only receives a mouse event if you click on the actual line.
In case you're talkinga about the actual graphical representation of the selection: that's totally up to you. By default, QGraphicsScene only notifies the selected items; you can then decide to how to present the selection to the user. In the case of Pd, you would only change the appearance of the selected objects (i.e. make them blue), but not draw a bounding rectangle.
Another nice feature about Qt's selection mechanism is that you can use arbitrary selection shapes. For example, you can select items by drawing an arbitary path, similar to "free selection" mode in painting programs. That's actually a feature I've always wanted in Pd :)
Thanks to Antoine for mentioning pdlua! I will need to check it out. I have already seen people doing some very cool stuff with it. Maybe they already have the perfect solution :)
I'm also looking at pdlua, which might want to become the official way to make graphical externs AND to add scripting in a fundamental way to Pd :)
+1 for an official scripting language! :) (I love Lua, BTW.) And yes, it would make lots of sense to leverage the scripting language to implement custom UIs!
Christof
-
Christof Ressi -
Miller Puckette