mid-March 2022: I'm finally getting back to normal and returning to projects I was working on before the pandemic. 2 years of shifting priorities... ya know
The project I was working towards is doing long-length cross-covariance and deconvolution calculations in a "fast-forward" pd instance and then passing the results back to a real-time process, with minimal risk to real-time
A pd process that is running in fast forward does not poll and so does not listen for connections of some kinds. When fast forward is turned off by the patch, it starts using the regular scheduler and starts listening again.
At the time, 2 years ago, I had gotten this kind of behavior from batch mode and communication through shmem and some signal handlers (raisestop, sigcont). With [ggee/shell] to start/stop the processes and send sigcont, it rounded out an (linux-specific, and very kludgey) solution. A batch mode process could wake up, read some shmem ID's and be told what to do, transfer data in, do some calculations, write the output to other shmem ID's, and then sleep again. To make the programming a bit easier to read, it was also possible to send lists using FUDI over shmem.
My conclusion there was that shmem can be used for asynchronous inter-process communication with minimal risk to real-time. It's very good as a fundamental object--it does not block, it does not synchronize. Notable limitations:
- Every process needs to know/use the same size for shmem ID's.
- Once allocated, a shmem cannot be re-sized.
- Writing to/from an extremely large array all at once poses a risk
to real-time.
I'd like to write a pair of management abstractions for using fast forward and shmem, then, that make it easy to stage in/out large, variable-length data
Anybody else have best practices for IPC when using fast forward? Having a listener on the 2nd process between computing sprints in the "fast forward" process completely changes how I can do things.
other: is there a good way to start/stop processes other than [ggee/shell]? cross-platform?
Chuck
Hello Chuck,
Le 16/03/2022 à 22:00, Charles Z Henry a écrit :
[...]
My conclusion there was that shmem can be used for asynchronous inter-process communication with minimal risk to real-time.
it can also be used between 2 synchronous process!
It's very
good as a fundamental object--it does not block, it does not synchronize.
that's the aim!
Notable limitations:
- Every process needs to know/use the same size for shmem ID's.
is that a real limitation? Do you have a practicable example where one need to share memory of different size?
- Once allocated, a shmem cannot be re-sized.
There is an allocate message now! It allow to change the Id and the memory size. But Antoine have problem compiling it on windows, so the last version is only available in deken for linux and osX. I'll be happy if anyone else want to gives a try at a windows compilation...
- Writing to/from an extremely large array all at once poses a risk
to real-time.
yes, obviously, moving data from an memory position to an other need time. It's far from ideal, but in this situation of "extremely large array" you can spread over time your read/write.
best Cyrille
I'd like to write a pair of management abstractions for using fast forward and shmem, then, that make it easy to stage in/out large, variable-length data
Anybody else have best practices for IPC when using fast forward? Having a listener on the 2nd process between computing sprints in the "fast forward" process completely changes how I can do things.
other: is there a good way to start/stop processes other than [ggee/shell]? cross-platform?
Chuck
Pd-list@lists.iem.at mailing list UNSUBSCRIBE and account-management -> https://lists.puredata.info/listinfo/pd-list .
On 3/17/22 08:58, cyrille henry wrote:
Notable limitations:
- Every process needs to know/use the same size for shmem ID's.
is that a real limitation? Do you have a practicable example where one need to share memory of different size?
i don't think this is the problem that chuck is referring to. afaiu, it's rather that the two processes need to have a priori knowledge of two different "thingies" in order to share some memory (without bad surprises): the ID and the size.
from a UX pov the question is, why it's not possible to only have to share a single "thingy" (the ID) and have the others be shared implicitly.
fmgdsaf IOhannes
Le 17/03/2022 à 09:24, IOhannes m zmölnig a écrit :
On 3/17/22 08:58, cyrille henry wrote:
Notable limitations:
- Every process needs to know/use the same size for shmem ID's.
is that a real limitation? Do you have a practicable example where one need to share memory of different size?
i don't think this is the problem that chuck is referring to. afaiu, it's rather that the two processes need to have a priori knowledge of two different "thingies" in order to share some memory (without bad surprises): the ID and the size.
Things are like that because I copy code from your object "pix_share_read" and "pix_share_write"!
from a UX pov the question is, why it's not possible to only have to share a single "thingy" (the ID) and have the others be shared implicitly.
Yes, automatically sharing the memory size could be possible, and can be useful in some situation.
Since I don't spend a lot's of time in pd currently, one should not expect a new version soon. But I accept patch!
cheers c
fmgdsaf IOhannes
Pd-list@lists.iem.at mailing list UNSUBSCRIBE and account-management -> https://lists.puredata.info/listinfo/pd-list
On Thu, Mar 17, 2022 at 3:26 AM IOhannes m zmölnig zmoelnig@iem.at wrote:
On 3/17/22 08:58, cyrille henry wrote:
Notable limitations:
- Every process needs to know/use the same size for shmem ID's.
is that a real limitation? Do you have a practicable example where one need to share memory of different size?
i don't think this is the problem that chuck is referring to. afaiu, it's rather that the two processes need to have a priori knowledge of two different "thingies" in order to share some memory (without bad surprises): the ID and the size.
from a UX pov the question is, why it's not possible to only have to share a single "thingy" (the ID) and have the others be shared implicitly.
fmgdsaf IOhannes
Yes, it's exactly that--there's always at least one shared piece of information that's hard-coded in both patches, if you want to communicate solely through shmem. It's trivially extended though. All processes agree to read from one chosen shmem ID of size 2 on startup and know that it contains the ID/size of a variable-length shmem that's now known. Before you know it, you're writing a whole protocol.
What's the best method for callbacks from a process that has completed its task and has data ready to be staged out? The toplevel process has to be able to be reached from multiple processes--so that seems like it should just be a udp port. Unsure on this point, though
Best, Chuck
Hello, Antoine finality found the bug in my code that prevent the windows compilation.
So shmem 1.1 is finally out!
Thanks to Antoine wizardry and some magic in github, a package was automatically upload to deken with binary for everybody!
let's wait the standard time for deken to share it and test!!!
cheers C
Le 17/03/2022 à 18:04, Charles Z Henry a écrit :
On Thu, Mar 17, 2022 at 3:26 AM IOhannes m zmölnig zmoelnig@iem.at wrote:
On 3/17/22 08:58, cyrille henry wrote:
Notable limitations:
- Every process needs to know/use the same size for shmem ID's.
is that a real limitation? Do you have a practicable example where one need to share memory of different size?
i don't think this is the problem that chuck is referring to. afaiu, it's rather that the two processes need to have a priori knowledge of two different "thingies" in order to share some memory (without bad surprises): the ID and the size.
from a UX pov the question is, why it's not possible to only have to share a single "thingy" (the ID) and have the others be shared implicitly.
fmgdsaf IOhannes
Yes, it's exactly that--there's always at least one shared piece of information that's hard-coded in both patches, if you want to communicate solely through shmem. It's trivially extended though. All processes agree to read from one chosen shmem ID of size 2 on startup and know that it contains the ID/size of a variable-length shmem that's now known. Before you know it, you're writing a whole protocol.
What's the best method for callbacks from a process that has completed its task and has data ready to be staged out? The toplevel process has to be able to be reached from multiple processes--so that seems like it should just be a udp port. Unsure on this point, though
Best, Chuck
Pd-list@lists.iem.at mailing list UNSUBSCRIBE and account-management -> https://lists.puredata.info/listinfo/pd-list
Cool!
let's wait the standard time for deken to share it and test!!!
The package is already online! Deken now updates the cache immediately, you do not have to wait 24 hours anymore.
Christof
On 19.03.2022 15:34, cyrille henry wrote:
Hello, Antoine finality found the bug in my code that prevent the windows compilation.
So shmem 1.1 is finally out!
Thanks to Antoine wizardry and some magic in github, a package was automatically upload to deken with binary for everybody!
let's wait the standard time for deken to share it and test!!!
cheers C
Le 17/03/2022 à 18:04, Charles Z Henry a écrit :
On Thu, Mar 17, 2022 at 3:26 AM IOhannes m zmölnig zmoelnig@iem.at wrote:
On 3/17/22 08:58, cyrille henry wrote:
Notable limitations:
- Every process needs to know/use the same size for shmem ID's.
is that a real limitation? Do you have a practicable example where one need to share memory of different size?
i don't think this is the problem that chuck is referring to. afaiu, it's rather that the two processes need to have a priori knowledge of two different "thingies" in order to share some memory (without bad surprises): the ID and the size.
from a UX pov the question is, why it's not possible to only have to share a single "thingy" (the ID) and have the others be shared implicitly.
fmgdsaf IOhannes
Yes, it's exactly that--there's always at least one shared piece of information that's hard-coded in both patches, if you want to communicate solely through shmem. It's trivially extended though. All processes agree to read from one chosen shmem ID of size 2 on startup and know that it contains the ID/size of a variable-length shmem that's now known. Before you know it, you're writing a whole protocol.
What's the best method for callbacks from a process that has completed its task and has data ready to be staged out? The toplevel process has to be able to be reached from multiple processes--so that seems like it should just be a udp port. Unsure on this point, though
Best, Chuck
Pd-list@lists.iem.at mailing list UNSUBSCRIBE and account-management -> https://lists.puredata.info/listinfo/pd-list
Pd-list@lists.iem.at mailing list UNSUBSCRIBE and account-management -> https://lists.puredata.info/listinfo/pd-list
On 17/03/2022 04:58, cyrille henry wrote:
so the last version is only available in deken for linux and osX. I'll be happy if anyone else want to gives a try at a windows compilation...
shmem[v1.0] is the latest?
There are Windows versions for it.
If there is a new version: where I can find the sources?
--
Mensaje telepatico asistido por maquinas.
I think, the latest code is on my github : https://github.com/ch-nry/pd_shmem it still say version 1, but work have been done since, it need to be increase.
the allocate message was add in the last version.
I did not find any binary on deken so I don't know where they are!
Antoine just told me that he manage to compile it for windows, so a proper deken release should be made.
cheers C
Le 17/03/2022 à 09:33, Lucas Cordiviola a écrit :
On 17/03/2022 04:58, cyrille henry wrote:
so the last version is only available in deken for linux and osX. I'll be happy if anyone else want to gives a try at a windows compilation...
shmem[v1.0] is the latest?
There are Windows versions for it.
If there is a new version: where I can find the sources?
--
Mensaje telepatico asistido por maquinas.
.
On Thu, Mar 17, 2022 at 3:06 AM cyrille henry ch@chnry.net wrote:
Hello Chuck,
Le 16/03/2022 à 22:00, Charles Z Henry a écrit :
[...]
My conclusion there was that shmem can be used for asynchronous inter-process communication with minimal risk to real-time.
it can also be used between 2 synchronous process!
It's very
good as a fundamental object--it does not block, it does not synchronize.
that's the aim!
Notable limitations:
- Every process needs to know/use the same size for shmem ID's.
is that a real limitation? Do you have a practicable example where one need to share memory of different size?
The basic thing I'm building is a network analyzer. Send out a signal X and record from the adc Y Then, you can recover the impulse response. I want to apply it to passive electrical circuits (and loudspeakers) for measurement and to room reverberation
To get a better estimate of the impulse response, you can increase the measurement time. The noise terms grow slower than the signal terms do. There's no way of knowing before the 1st measurement how long the measurement needs to be. So, it's always going to be adjusted at least once
- Once allocated, a shmem cannot be re-sized.
There is an allocate message now! It allow to change the Id and the memory size.
Ah, that's awesome! I'll read it today. I think I will have some questions
But Antoine have problem compiling it on windows, so the last version is only available in deken for linux and osX. I'll be happy if anyone else want to gives a try at a windows compilation...
- Writing to/from an extremely large array all at once poses a risk
to real-time.
yes, obviously, moving data from an memory position to an other need time. It's far from ideal, but in this situation of "extremely large array" you can spread over time your read/write.
best Cyrille
Yes, that's a key feature I want to implement with abstractions.
There's no risk to doing the memory management and big transfers from shmem to array in a "fast forward" or non-realtime process. The toplevel, realtime, audio process should manage the staging of data and have that ability to spread it out over time
On Thu, Mar 17, 2022 at 3:06 AM cyrille henry ch@chnry.net wrote:
Hello Chuck,
Le 16/03/2022 à 22:00, Charles Z Henry a écrit :
[...]
My conclusion there was that shmem can be used for asynchronous inter-process communication with minimal risk to real-time.
it can also be used between 2 synchronous process!
I see that in the examples now. You've used [pd~] and it's pretty cool how that works.
It's very
good as a fundamental object--it does not block, it does not synchronize.
that's the aim!
Notable limitations:
- Every process needs to know/use the same size for shmem ID's.
is that a real limitation?
I was unsure about this ordering, but I have run some tests to clarify.
P1:[allocate ID size( P2:[allocate ID size( <-- Order does not matter P1:[memset array1( <-- P2:[memdump array2(
You can write the data before the 2nd process has run allocate and it still works fine.
Do you have a practicable example where one need to share memory of different size?
- Once allocated, a shmem cannot be re-sized.
There is an allocate message now! It allow to change the Id and the memory size. But Antoine have problem compiling it on windows, so the last version is only available in deken for linux and osX. I'll be happy if anyone else want to gives a try at a windows compilation...
Re-sizing now works great too. Only thing I noticed was not to send extra allocate messages in the first process, before the 2nd process has a chance to run its allocate.
There were also some orderings of allocate on P1 and P2 with different sizes that still worked, even though that's not really how anyone should use it. None of my intentional misuse caused any crashes.
- Writing to/from an extremely large array all at once poses a risk
to real-time.
yes, obviously, moving data from an memory position to an other need time. It's far from ideal, but in this situation of "extremely large array" you can spread over time your read/write.
best Cyrille
I think I misused the word "limitations", sorry I used that word. More like obstacles I was looking at I was working out some way to hand data around, while only allocating shmem in 2^N sizes that hang around and get re-used instead of re-allocated.
I'll work my way up to testing how big a data transfer has to be to cause a problem to real time
-
Charles Z Henry -
Christof Ressi -
cyrille henry -
IOhannes m zmölnig -
Lucas Cordiviola -
Roman Haefeli