Oops, I just realized that I sent my reply IOhannes and didn't cc pd-dev...

On Thu, Jan 19, 2012 at 4:11 AM, IOhannes m zmoelnig zmoelnig@iem.atwrote:

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

...

...

right now, only #1 is possible at all and it takes some effort on the

...

...

"thread host" (the external) to not fuck Pd's heap. i think Pd should be more helpful in this respect: a way to make Pd thread-safe is to eliminate global variables and if they can't (or shan't) be eliminate them properly protect them against parallel access.

As far as libpd is concerned, I would prefer not to have any synchronization inside Pd itself --- libpd can be used in a wide variety

of

...

settings, with lots of different approaches to concurrency, and so it's impossible to make any assumptions about threading at this level.

what are the actual drawbacks if e.g. clock_delay() could be used from any thread (in an external)?

There's nothing wrong with accessing a Pd instance from multiple threads. All I'm saying here is that the hypothetical replacement for libpd that would form the core of this refactoring is not necessarily the place to do synchronization.

By the way, once we have multiple instances of Pd, calls like clock_delay will also need to take a context parameter because each instance of Pd will have its own sys_time.

which assumptions are made that might not hold true in all your use cases?

As I interpret it, the assumption is that synchronization at the audio library level is desirable and sufficient. There are use cases that might not need synchronization at all (non-real-time batch processing, Python applications, applications that use one Pd instance per thread, etc.), which would render locks in Pd redundant as far as libpd is concerned (support for threaded externals is a different story). In other cases, an invocation of a Pd function might be part of a larger context that also requires synchronization. The Java bindings of libpd, for example, maintain a certain amount of state on the Java side. In such a situation, additional synchronization outside of Pd is needed.

In order to find the best solution for each case, each application would have to be in charge of thread synchronization, but that would lead to a lot of repetitive and error-prone code. In libpd, I decided to handle synchronization in the language bindings (no locks in Python, class-level locks in Java and Objective-C), which seems like a reasonable compromise. The upcoming C++ bindings that Dan Wilcox and Rich Eakin are working on seem to be a different case due to portability issues, but the general strategy of handling concurrency in language bindings seems to be sound.

In either case, locks in Pd would be redundant from the point of view of libpd, although they'd also be benign because they'd always be uncontested and hence cheap.

my main reasoning is, that thread synchronisation is a re-curring problem that imho should not be re-implemented whenever it is needed. also a earlier attempts to fix this, using the great BIG kernel lock (aka sys_lock()), proved (at least for me) to be inadequate, as they slow down the entire processing significantly.

When I started working on libpd, I decided to discard sys_lock altogether. The trade-off was between dealing with the complexity of sys_lock and losing some threaded externals. I haven't yet received any complaints about broken externals, so I think the decision was okay.

however, maybe i'm asking too much and what i really want is to have a standardized possibility to send messages to a Pd-instances without having to worry about threading (i use all the clock-stuff i keep mentioning mainly for doing exactly this: implementing a thread-safe message queue to send data from a worker thread back to Pd)

I see several ways of handling this. One approach might be to have a core library that remains blissfully unaware of concurrency, along with a thread-safe wrapper that externals would operate on. Another possibility would be to think really big and rethink the way Pd processes audio (why not do a topological sort of the signal processing graph and then parallelize computations on the fly, using something like OpenMP?). That might involve a general refactoring into components that might be made thread-safe with fine-grained locks or other concurrency techniques.

...

Of course, as you point out, Pd itself requires some synchronization in

its

...

interaction with externals, so there's a bit of a conflict there. My favorite solution would be to refactor Pd so that it has an audio library much like libpd at its core. Then Pd would be able to do all the synchronization it needs without affecting other applications that use

the

...

same library.

i have to admit i'm a bit unsure where you would draw the separation. Pd uses common infrastructures for a lot of things on different system levels: e.g. the message system is used to communicate between objects, to let the gui talk to the pd-core, to open a patch (instantiate it's objects),... while i think this is one of the strengths of Pd, i also think that this is probably the biggest problem when attempting a refactor as you describe it (but again: i might totally miss the point here)

I think we're already half-way there. Just look at Chris McCormick's PdDroidParty --- he's basically rebuilding a good chunk of the GUI functionality of Pd on top of libpd. What's missing is the ability to edit patches, but I believe that could be done with a minor extension of the messaging API of libpd. Cheers, Peter