[PD] benchmarking slow startup with Pd-0.55 (was: Pd 0.55-0 test version released)

On 5/27/24 21:58, Benjamin Wesch wrote:

thanks a lot for pointing out 'hyperfine' and for providing this somewhat self-defeating plugin.

thanks for the additional data.

====== recap ======

i think the most promising paths for investigation are:

• the code
• the build environment
• the signing process

ad signing process: macOS spends some time during startup to check whether the application can be trusted. in theory this should give some performance penalty at the very first startup of an application, but from then on things should be pretty fast (as the results are hashed). in any case, i'm not aware that anything has changed in the way we sign/notarize Pd since 0.54 (apart from Apple having switched to a new signing/notarizing infrastructure,...)

ad build environment: about two months ago i switched the CI builders for macOS from an aging Mac Mini 5,2 (mid 2011) running Big Sur (within a VM on a Debian hypervisor), to a Mac Studio 13,1 M1 (2022) running Monterey (within a VM on a Ventura hypervisor). this sped up builds considerably (from about ~7 minutes to ~1 minute).

(sidenote: this change only affects the build process itself; signing/notarization is still done on the old VMs)

ad code: my gut feeling tells me that there are two promising candidates:

• the changed GUI startup protocol
• something about the audio backends

i *guess* that the changed GUI startup will only affect Pd if it is started via the core (as opposed to starting via the GUI, e.g. by clicking on the app icon). i would have assumed that the changed GUI startup would actually speed up things (at least for our benchmark tests, where we start via the core and try to quit asap)

====== more benchamrks ======

in the meantime i've learned some more about hyperfine (which can do argument variations on its own), and ran some more tests with something like the following:

hyperfine \
    --export-csv benchmark.csv  \
    -N --warmup 3 \
    -L pd 0.54-1,0.55-0test3-4-g618b6325  \
    -L audio nosound,jack,pa  \
    -L callback ,no \
    -L quit '-send "pd quit" -nogui','-send "pd quit"','-path quitter/'  \
    "Pd-{pd}.app/Contents/Resources/bin/pd -noprefs -nostdpath -{audio} 
-{callback}callback {quit}"

this tests:

• multiple versions of Pd
• different audio backends (w/wo callbacks)
• w/wo gui
• quiting from either GUI or core

(it assumes that there are a couple of Pd-*.app in the current directory, so we only need vary the actual version number. also the quitter-plugin is in the quitter/ directory of the cwd and is *explicitly* activated).

i did one run on our Mac Studio (Ventura) via VNC(!), and another on my Debian desktop PC (obviously with slightly different invocation to cater for the changed path layout and different available backends).

no check has been made to see if the audio backends are actually useable (if Pd tries to open a blocked audio device, this could stall the startup considerably!)

find the results in the attached CSV files (where i've turned the startup options in separate columns. an empty value in the "audio" column means that Pd was started with "-nosound")

my prelaminary findings are:

• the build environment is not to blame for the slow down (the startup

times range between 70ms and almost 2 seconds (on macOS) for both the old and the new binaries (on Debian it's between 2ms and 300ms))

• the most important factors for the entire startup-shutdown cycle are
  1. whether there's a GUI (slow) or not (fast)
  2. whether the shutdown is initiated from the core (fast) or from the

GUI (slow)

• as far as the audio backends are concerned:
  1. whether one uses callbacks or not, does not seem to make much of a

difference 2. using PortAudio adds a considerable overhead. (JACK timings and the PortAudio timings are clearly in two different leagues, separated by a factor of 1.5 up to 5). the separation is most prominent in the tests that do not involve the GUI (suggesting that the GUI overhead hides the overhead from the audio backend)

my numbers differ greatly from ben's. esp. when running Pd-0.54 with GUI it takes much longer (e.g. using the quit-plugin, the minimum turnaround time is ~1.4secs!, rather than 0.4secs as seen by ben) this can probably be explained in part by the different CPUs and OS version. without a GUI i'm seeing comparable numbers (~70ms), but only when using JACK.

the default audio backend for macOS is PortAudio.

gfdasr IOhannes