To pd-dev (but mostly libpd gurus) -
I've pushed a first cut at thread-enabling Pd. I think this feature will be mostly useful within the context of libpd. There are things to worry about...
For instances to work at all, it's necessary to compile Pd code with -DPDINSTANCE . This sets Pd up so that it makes separate symbol tables per Pd instance. This took some surgery, since classes now have to have separate symbol-to-method lookup tables per Pd instance. If the Pd submodule is updated in libpd, libpd won't compile unless the Pd source is compiled with -DPDINSTANCE.
Thread=safety is enabled by default (I'm not sure this is the best idea but that's the way of it for now). To disable it one could compile Pd with the additional dlag -DPDTHREADS=0 . OTOH, to embrace it, it will be necessary to protect every call into Pd with sys_lock() and sys_unlock() calls, even if there's only one thread, because of the following...
...
The thread safety mechanism of Pd has two levels, one of which is to put an exclusive lock (pthread_mutex_t) on each instance of Pd . This is obtained by calling sys_lock() after setting pd_this to the desired instance via pd_setinstance() .
THEN, there's a non-exclsive-readable, exclusive-writable lock protecting all the classes and Pd instances, which Pd itself manages. sys_lock() obtains read-only permission to this lock, and finctions like class_new() within Pd release the real-only access in order to obtain exclusive write access. If read access wasn't in effect, Pd calls "bug("pd_globallock")" to warn you, but in this case Pd will inappropriately re-obtain the lock for read access, so soe later attempt to get exclusive access will likely then deadlock.
The code is in s_inter.c; search for "PDTHREADS".
...
Yet to be done: if this model looks OK, the next thing is to fix the GUI code to be per-instance and thread-safe; this will entail moving the many static variables in g_editor.c, g_template.c and elsewhere into the t_editor structure to make them per-canvas. There are other static variables floating around, some of which are OK to remain static, but some of which will need also to be made per-canvas or per-Pd-instance. It might take a while to track them all down and there might not be any way to verify that everything has been made truly thread-safe.
In any event, as long as nobody is using Pd's GUI for more than one instance of Pd within libpd it should be safe as far as I know. I don't think the official libpd even supports sprouting the GUI yet, so this might be only of interest to me (I've been using libpd with GUI for a very strange project of my own).
Your feedback on all this would be most welcome...
cheers Miller
P.S. I should make a pull request to libpd to add the necessary -DPDTHREADS to the makefile and sys_lock(), sys_unlock() wrappers wherever libpd calls Pd - I'll try to get this together tomorrow.
1. If thread #1 loads [foo] from foo.pd_linux, and thread #2 loads [foo] from foo.pd_linux, what happens? 2. Will there be any public interfaces external developers can use to keep from making their own ad-hoc, buggy revisions to legacy externals?
-Jonathan
I'm not sure whether dynamic class loading is thread-safe - it won't cause crashes (I think) because class_new(), etc., have a global lock. There would in the end be two "classes" one of which would alias the other's name. But there might be unwanted side effects here that I haven't smoothed out yet.
In working on libpd today I've noticed two other thread-unsafe situations: dollar zero and search path manipulations, which I need to make per-Pd-instance.
as to (2)... I believe legacy externals will have to be recompiled if they use symbols like "s_float" - or, later, whatever other static external variables I end up making per-instance. In this case they'll fail to load into multi-instance environments. (They'll still load into Pd vanilla since I compile that with PDINSTANCE undefined). So I believe we'll have binary compatibilty for Pd vanilla, and for most externs when loaded into libpd, but some others will refuse to load and have to be massaged as in
&s_float ..> gensym("float")
But who knows what other horrible problems will come up. I don't expect this to be an easy process... but I think it will be worth it in the end...
cheers Miller
On Mon, Apr 17, 2017 at 06:32:31PM +0000, Jonathan Wilkes wrote:
- If thread #1 loads [foo] from foo.pd_linux, and thread #2 loads [foo] from foo.pd_linux, what happens?
- Will there be any public interfaces external developers can use to keep from making their own ad-hoc, buggy revisions to legacy externals?
-Jonathan
No, each instance has its own symbol table. Thus instances can't make calls into each other as they could before - and I think that's all to the good :)
Meanwhile, there's no assumed relationship between instances and threads. A multi-threaded app can make calls to different instances on different threads if desired, and/or even call the same instance from a different thread (but not from two threads simultanously).
cheers
M
On Mon, Apr 17, 2017 at 07:58:09PM +0000, Jonathan Wilkes via Pd-dev wrote:
[...]
This sets Pd up so that it makes separate symbol tables
per Pd instance. Still not sure I understand. Did you mean "per each thread in a given Pd instance"? -Jonathan
Pd-dev mailing list Pd-dev@lists.iem.at https://lists.puredata.info/listinfo/pd-dev
No, each instance has its own symbol table. Thus instances can't make callsinto each other as they could before - and I think that's all to the good :)
More questions: what's the use case for loading multiple instances of a shared library into the same memory space? Can this be done on all supported platforms? Also, is there a fairly commonly used, extant example of a currently maintained library out there that supports multiple instantiation that you can point me to?
-Jonathan
Also, is there a fairly commonly used, extant example of a currently maintained library out there that supports multiple instantiation that you can point me to?
Sorry, that wasn't detailed enough-- I'm curious to see a library of non-trivial size that supports multiple instantiation into the same memory space that is commonly used, which also happened to start out life as a single-threaded application.
-Jonathan
Csound is a pretty good example. Also Tcl/Tk. But there must be hundreds.
cheers M
On Mon, Apr 17, 2017 at 08:50:17PM +0000, Jonathan Wilkes via Pd-dev wrote:
Also, is there a fairly commonly used, extant example of a currently maintained library out there that supports multiple instantiation that you can point me to?
Sorry, that wasn't detailed enough-- I'm curious to see a library of non-trivial size that supports multiple instantiation into the same memory space that is commonly used, which also happened to start out life as a single-threaded application.
-Jonathan
Pd-dev mailing list Pd-dev@lists.iem.at https://lists.puredata.info/listinfo/pd-dev
Csound is a pretty good example. Also Tcl/Tk. But there must be hundreds.
Hm, I didn't realize that about tcl/tk. But I'd imagine it got refactored relatively early in its life, and that most of its current packages targetted the library and not the single-threaded application. -Jonathan
cheers M
On 2017-04-17 22:03, Miller Puckette wrote:
Meanwhile, there's no assumed relationship between instances and threads.
i was tempted to say hooray, but then...
... i discovered the use of the PERTHREAD macro which expands to "__thread" which is the keyword for TLS (thread local storage); i don't really see how this relates to "no assumed relationship between instances and threads". (though i have to admit that i haven't fully studied the code yet).
in any case, what's probably more important: it seems that at least some of the compilers shipped with OSX do not support thread-local storage at all [1].
this shows in the travis-ci builds, which currently fail for OSX: https://travis-ci.org/pure-data/pure-data/jobs/227985898
fgas,dr IOhannes
[1] e.g. https://bugs.freedesktop.org/show_bug.cgi?id=98603 this thread also suggests, that there might be workarounds if one could raise the "osx-minversion" to >=10.7
Yep, I don't think there's any way to make Pd callable from multiple threads except by making pd_this either an argument to every Pd function call (thus completely changing the API) or by making pd_this per-thread. But doing the latter by no means restricts a one-thread-per-instance or a one-instance-per- thread model.
My idea is this: Pd instances can migrate from thread to thread at the caller's will, but no two threads can refer to the same Pd instance at the same time. This is realized by having each thread have its own current instance, which the caller may set at will (from any thread). Each instance has a lock to prevent any thread from setting pd_this to an instance another thread already has set.
I think I need to find a way to detect at configuration time (?) whether the compiler supports per-thread storage, hmm....
cheers Miller
On Wed, May 03, 2017 at 01:55:31PM +0200, IOhannes m zmoelnig wrote:
On 2017-04-17 22:03, Miller Puckette wrote:
Meanwhile, there's no assumed relationship between instances and threads.
i was tempted to say hooray, but then...
... i discovered the use of the PERTHREAD macro which expands to "__thread" which is the keyword for TLS (thread local storage); i don't really see how this relates to "no assumed relationship between instances and threads". (though i have to admit that i haven't fully studied the code yet).
in any case, what's probably more important: it seems that at least some of the compilers shipped with OSX do not support thread-local storage at all [1].
this shows in the travis-ci builds, which currently fail for OSX: https://travis-ci.org/pure-data/pure-data/jobs/227985898
fgas,dr IOhannes
[1] e.g. https://bugs.freedesktop.org/show_bug.cgi?id=98603 this thread also suggests, that there might be workarounds if one could raise the "osx-minversion" to >=10.7
Pd-dev mailing list Pd-dev@lists.iem.at https://lists.puredata.info/listinfo/pd-dev
On 2017-05-03 16:24, Miller Puckette wrote:
I think I need to find a way to detect at configuration time (?) whether the compiler supports per-thread storage, hmm....
https://www.gnu.org/software/autoconf-archive/ax_tls.html
mdra IOhannes
I just saw the simpler answer... thread-local only matters if PDINSTANCE is turned on. It's off for the PD application, but may be turned on to get multi-instance support in libpd. So I fixed Pd vanilla to use __thread only when both PDTHREAD and PDINSTANCE are set.
cheers Miller
On Wed, May 03, 2017 at 06:33:27PM +0200, IOhannes m zmoelnig wrote:
On 2017-05-03 16:24, Miller Puckette wrote:
I think I need to find a way to detect at configuration time (?) whether the compiler supports per-thread storage, hmm....
https://www.gnu.org/software/autoconf-archive/ax_tls.html
mdra IOhannes
Pd-dev mailing list Pd-dev@lists.iem.at https://lists.puredata.info/listinfo/pd-dev