hi all,

TL;DR: i suggest a new deken fileformat to be ready for double-precision Pd (once it comes) and other future vagaries. it requires you to update the search plugin and (if you use them) the cmdline tools.

(this is a (longish) discussion/draft/poll. it's implemented, but not deployed yet.)

everybody loves deken. thanks joe for the ignition and chr13m for the original implementation.

now, 3 years later, i would like to fix a few inconsistencies and make deken fit for the future.

# backstory

smaller issues and real problems we have been seeing with the current implementation are: - archive formats depend on platform - the package version cannot be parsed reliably - the architecture strings are not well-defined. leading to much confusion and as a consequence, some parts of the architecture string are simply ignored by the current deken-plugin.

unrelated to deken, there has been some work done in pd-vanilla land to get native Win64 binaries to work (for deken this means: a new architecture). and there have been rumours, that we might get double-precision builds for Pd - adding a whole new can of worms with incompatible externals.

now deken is not ready for double-precision externals. as things are, it will happily suggest to download a single-precision external to a double precision Pd and vice versa, making Pd fail to load the suggested-as-compatible externals (in the best case) or simply crash Pd (in the worst case).

the latter triggered a review of the deken packaging format, with the main task to make deken ready for the future.

the result i've come up with so far, is a new filename scheme for external packages, which allows us to git rid of the design problems of the old one while keeping the pool of existing deken packages (which have well known properties) - and still be able to use it from a user's POV.

# suggestion for a new format

## internal structure the internal structure of a deken packages stays the same as it already is: a simple archive of a directory tree containing whatever things the packager thinkgs need to be packaged.

## file format however, all deken packages should uniformly use the zip-format (instead of requiring linux&osx systems to support both zip and tgz).

## filename format the only real changes affect the format of the *filename* - as the filename contains all the meta-information of a deken-package.

so my suggestion is, that the filename must have the form:

`<libraryname>[v<version>](<arch1>)(<arch2>).dek`

Example:

helloworld[1.0.0](Linux-i386-32)(Windows-i386-32)(Darwin-amd64-32).dek

### libraryname at the beginning of the filename comes the libraryname. it may contain any characters with the exception of square brackets (`[]`) and parentheses (`()`).

### version e.g. "[v3.14]"

the version of a library comes after the libraryname and must be given in square brackets (`[]`). the actual version number must be prefixed with `v`. (using a prefix allows us to easyily extend the filename format to contain more meta-info if we ever have the need).

the version string may contain any characters with the exception of square brackets (`[]`) and parentheses (`()`).

NOTE: this is different from the current format (which used `-v<version>-`)

### architecture e.g. "(Darwin-ppc-32)(Darwin-i386-32)(Darwin-amd64-32)"

to understand whether an external is compatible with the currently running system, Pd needs to know about the architectures contained in the deken package. the full architecture specification consists of a number (zero or more) of arch sepcifiers, each surrounded by parantheses (`()`).

one of the arch specifiers may be "Source", indicating that the package contains the source code for the binaries. each (remaining) arch specifier consists of 3 elements, delimited by dash: the Operating System (OS), the machine (CPU) and the floatsize of the external (e.g. "Windows-i386-32"). each arch specifier may contain any character with the exception of square brackets (`[]`) and parentheses (`()`), and the dash (`-`) is reserved as delimiter (so it must not be used in the components)

typical values for the OS are "Linux", "Darwin" (for OSX/macOS) and "Windows".

typical values for CPU are - "i386" (for 32bit Intel or AMD processors; this is also used if you are running a 32bit OS on a processors capable to run 64bit code) - "amd64" (for 64bit processors made by AMD or Intel) - "arm" (for generic ARM) - "armv7" (for a more specific ARM processor) - "ppc" (PowerPC) the CPU field describes the *calling convention* rather than the actual hardware of the CPU (e.g. Win32 (and WoW64 for that matter) uses a 32bit calling convention, even if the actual CPU is an "x86-64 CPU", so deken uses the arch-string "i386").

typical values for the floatsize are "32" (single precision float point). once there are double-precision builds of Pd available, this can also have the value of "64".

NOTE: this is similar to the original format, but now that last value has an explicit meaning.

### extension all deken packages must use the uniform filename extension ".dek". (no more ~-externals.zip~ or ~-externals.tgz~ for you).

objectlists (which keep their format) must use the ".dek.txt" extension.

NOTE: this is different from the original format

# backward compatibility now, if the deken-server returned the new .dek packages for all queries, the deken-plugin will fail to download and extract these packages (as it doesn't know how to handle ".dek" files, just like many major OSs).

the proposal for this dilemma is: - the deken-plugin tells the server whether it is a "new" plugin (by means of setting the HTTP User-Agent field) - the deken-server will only return .dek packages if the plugin is new enough - the deken-server will automatically suggest to install the "deken" package if the plugin is too old.

# implementation implementation is complete (afaict), but not yet deployed

so far, i've implemented support for this in the [dek] branch of the deken repository. the (new) "deken" cmdline tool will (by default) create packages in the new format. (it can be tricked into creating legacy packages using the "--dekformat 0" flag). the (new) "deken-plugin" (aka "Find externals...") will happily work with both old and new formats.

since the deken-plugin sends its query to a webserver (deken.puredata.info), this webserver (aka: the glue) must be updated as well (as it pre-parses all the available deken packages to be able to quickly answer any queries). code can be found at https://git.iem.at/zmoelnig/deken-server

# discussion

for now i see two points that might require discussion.

1) katja suggested to use the arch-specific Pd-extensions (e.g. "d_ppc") instead of the more verbose "Darwin-ppc"). personally i'm not totally opposed, but i kind of like the verbose naming, as it allows for more future possibilities. e.g. "b_amd64" currently means "FreeBSD-amd64" but what about other BSD flavours, are they compatible? (not that i believe that most OSs apart from the current three will matter very much to the community in the near future; but then i'd rather be too inclusive than not)

2) the proposed user-agent string might be considered a privacy issue.

it currently is something like: "Deken/0.2.6 (Linux-amd64-32) Pd/0.48.1 Tcl/8.6.8" thus showing the Deken version, the Pd version the Tcl/Tk version and the architecture.

note, that the current plugin uses Tcl/Tks default user-agent header, which already reveals the Tcl-version (and probably the OS, because different Tcl implementations seem to set the string differently). also, an architecture identifier is practically sent whenever your ordinary webbrowser sends a request to ...google or whatever.

so i don't think that the header reveals problematic information, but then i thought it might be better to raise the awareness of others (you) first, before there are any complaints afterwards.

3) having the server add an automatic search for "deken" if you are using an outdated version.

the idea is, that if people who are using a plugin which cannot handle the new .dek file, they are nagged into upgrading deken.

the only way the server can communicate to the plugin is via the search results. so why not add a search-result for a newer (compatible) version of deken whenever the user might need it (because they are missing search results which require a newer plugin).

however, such "slight nags" can be pretty^Wvery annoying. i'd like to have feedback on this before i enable it.

that's probably it. fgmdasr IOhannes

[dek] https://github.com/pure-data/deken/tree/dek