On 06.07.20 19:39, Martin Peach wrote:
The build farm used by Pd-extended worked very well for me. It would be nice to have such a setup integrated with the deken system, so anyone could upload their externals and they would be built and packaged online instead of each developer needing to acquire one of each machine on a 3-year rolling plan.
we have something like this [0] setup at the iem, based on gitlab-ci[1].
the system is targetting pd-lib-builder projects (though it will probably will work with most build-systems that follow a "make && make install" paradigm)
whenever someone pushes a new commit to a CI-enabled Pd-project, it will build the external for the following architectures: - Darwin/amd64 (aka "64bit macOS") - Windows/amd64 (aka "64bit windows") - Windows/i386 (aka "32bit Windows") - Linux/amd64 - Linux/i386 - Linux/armhf (aka "Raspberry") - Linux/arm64 (the future)
the build artifacts are collected into a single deken package, which can be downloaded manually. if the commit was "tagged" (which we take as a "release" indicator), the deken package will be automatically uploaded to deken.puredata.info.
there's a system to declare build dependencies (via "apt" on linux, MSYS' "pacman" on Windows, and "brew" on macOS). for systems that don't come with a proper package manager installed by default (macOS, Windows), the runtime dependencies are collected and embedded in the deken package.
some libraries cannot be built into a single multi-arch library because of file conflicts (this becomes fun quickly when dealing with dependency libraries and dependencies thereof). there's an option to create multiple single-architecture deken packages for this case.
in general, the configuration is about 3-lines (e.g. [2]) a bit more, if you want to enable automatic uploads to puredata.info.
so everythings cool. what are the caveats?
## Darwin/i386 we currently don't build for Darwin/i386 (nor Darwin/amd64). while we have older macOS builders (there's builders for both macOS/Sierra and macOS/Mojave, the default being macOS/Mojave) with *not* the newest XCode, and should therefore still be able to build i386 binaries without problems, these are currently disabled because of the automatic dependency management (on macOS we use "brew" to install dependencies; brew has decided to stop supporting i386 binaries a while ago :-()
## accountability deken is a quite insecure system, as there isn't really a way to "verify" a package before installing it. the best you can get is that you know the <username> who uploaded a given package (in theory there's also the gpg-signature; but who ever checked that?) now, with automated uploads this becomes a bit muddled, as there is no longer any direct involvement of a human (whom you can trust, or not) in creating the binaries. in theory we know the user who pushed the last commit that triggered the build process. but currently there's no easy way to make this user show up in the deken results.
## passwords to upload any package to deken.puredata.info, you currently need a username/password. if you want to upload automatically, the build-process needs to know your username/password. i really don't want to encourage people to store their credential for one web-service in any other web-service (even if both services are operated by the same entity (me!)). (the credentials would be stored in some database and *NOT* in a file in the repository, so they are not directly exposed to ordinary humans; but nevertheless the credentials need to be plain-text (as in: unhashed) which is a security breach...). for my personal use i have created a "zmoelnigbot" account on puredata.info to manage those uploads (and keep it separate from my other account). but that doesn't scale well.
## resources it's not like we have a big compile farm or so, but we take interest (and pride) in running all the builds on-premises. currently there's 3 machines: - an oldish (but dedicated) MacMini (that needs to run all the Darwin-builds for license reasons; and which also does Windows builds) - a dedicated linux box (Windows and Linux builds) - a virtual machine (Linux builds)
having dedicated machines means, that the builds will (mostly) start right away (as opposed to e.g. travis-ci where it can take considerable time (as in: half an hour) until the builds start). having very few machines available means that building for all the different architectures will take longer than just running "make" locally. it also means that if multiple projects want to build at the same time, it will take longer (and we have some in-house projects where each build takes about 1hour; if they happen to be started before your project, it will take a while).
it's probably super-easy for volunteers to add their own machines to the build farm - at least for the Linux builds (that use docker internally). it's much more complicated for Windows and Darwin builds (which currently use manually pre-configured VirtualBox images that have to be regularly updated (manually), which is always a pain in the ass). of course, the Windows and Darwin builds are those where the automatic build becomes really interesting.
## resources (legally) the service is currently run by the iem, at the iem and for the iem. i would have to talk with my boss about making it available to a wider audience. it would probably be fine (but of course, i really don't know).
mgfdasr IOhannes
[0] https://git.iem.at/pd/iem-ci/ [1] https://docs.gitlab.com/ee/ci/ [2] https://git.iem.at/pd/iemnet/-/blob/master/.git-ci/gitlab-iem.yml