Hi katja, Thanks for the response. A few questions and responses:
- is knob's move from flatspace to flatgui a special case, or is it representative of some trend of object classes scurrying about from directory to directory?
If it is a special case, then either it is a bug: "I, user, do declare my intention to load flatspace/knob, a library so standard that it is installed in a path relative to Pd itself, yet so nonstandard that I cannot trust libdir/objectname" or Pd is wildly misleading: "I, user, do declare my intention to load library flatspace/knob as a STDLIB[see terms]" [small print] terms: the standard library path is not responsible for items lost or stolen by the libraries residing in the standard path.
If moving objects around is general trend, well... I really hope it isn't.
- I don't path priority can solve the problem. Pd has one big class called "objectmaker" which registers a new method for each object creator in Pd, including externals. So when you type "some_object" in an object box, the first thing Pd is going to search are the methods of objectmaker. And if it finds a method named "some_object" it's going to execute the function it finds in that method entry. Thus, if "some_object" was already loaded anywhere in the running Pd instance, it's going to have a method entry in objectmaker and Pd isn't going to search any paths for a library to load. Now you're back to gambling without prefixes.
Btw-- you can explore these methods in objectmaker from within Pd-l2ork with [nbx]--[classinfo objectmaker]. You can also see everything that's gone through class_new with [classlist(---[pdinfo].
On Thursday, October 16, 2014 9:44 AM, katja katjavetter@gmail.com wrote:
Thanks for bringing up this important topic. Over the years I flip-flopped between [library/objectname] and [declare] a few times, for reasons which I'll try to explain.
The first occasion where I needed [library/objectname] was when [pow~] in pd core had it's inlets swapped between pd versions, while [cyclone/pow~] retained the old inlet order and could be used for backward compatibility.
When Pd-extended stopped loading libraries by default I started using [library/objectname] for externals from all Pd-extended libraries, which was recommended practice.
Later, I came across a few cases where externals moved from one lib to another. For example, [flatspace/knob] became [flatgui/knob]. I feared further library reorganizations and abandoned the rigid [library/objectname] format. Now I started to appreciate the flexibility provided by [declare]. I used declarations like [declare -stdlib flatspace/knob] and [declare -stdlib flatgui/knob] in top level patches to make projects compatible across Pd-extended versions. Also, I use [declare] for binaries which are included in project directories: for my homebrew binaries, or selected Pd-extended binaries in projects which should run with vanilla Pd. It is much easier to change or add a few declarations in the top level patch than changing library names in all object instances in a large project.
But [declare] only adds search paths and it doesn't protect against name clash. I came across [bsaylor/svf~] which has different number of inlets and outlets than [cyclone/svf~]. Without library name, [svf~] instantiates the cyclone version. But when [bsaylor/svf~] was instantiated earlier in the Pd session, or with [import bsaylor] in the patch, the bsaylor version may be instantiated. Without namespace specification Pd seems to use the version that was loaded first within a Pd session.
I realized that instantiating without library namespaces is a gambling. And this does not only apply to binary executables, but to abstractions as well. Imagine you share a Pd project including abstractions and binaries, and you publish a new version of the project, where bugs in externals were fixed or new features were added to abstractions. Users may want to compare versions, and load the old version first in a Pd session. Even when they close the old version of the project before loading the new version, the old executables and abstraction definitions are still in Pd's memory, and Pd will not load the new ones unless they were prefixed by a different namespace, or until Pd is restarted. So, the new project version may seem to still contain the old bugs!
The current solution for such issues is systematic use of lib names and relative paths in object or abstraction instances. For example, include abstractions in a subdirectory named 'abstractions', and instantiate as [abstractions/objectname]. Pd will then load the abstraction from the subdirectory in the project, even when many projects have an 'abstractions' subdirectory containing an abstraction with the same name. This tedious and inflexible approach will only work when directory structures don't change. A project or library structure must be well organized right from the start. If something changes in the structure, you may need to find all instances of a class in your patch(es) and redefine them. Rigid directory structures, and the prospect of minor changes causing heaps of maintenance work, can be an obstacle for future innovations in a project, or in Pd-extended as a whole.
In any case, I'm reluctant to use namespaces for each and every instance. It can make your patch today, and break it tomorrow. It would be super if [declare] would not only add a search path, but also locally enforce priority for it. Say I have my own implementation of [svf~] stored in the 'bin' subdirectory of a Pd project directory. The main patch contains [declare -path bin] and several instances of [svf~]. Pd should search the bin subdirectory and other locally declared paths for an [svf~] executable. If Pd finds the one in the bin subdirectory, it should interpret all instances of [svf~] as [bin/svf~] and instantiate them accordingly. If an object name appears in two locally declared paths, you would still have a name clash. But designing a single project directory without name clashes is not so difficult.
How hard would it be to realize such path priority, I wonder? Currently, Pd can distinguish between different abstractions with the same name and relative path prefix, if they appear in different project directories. See attached test, where different versions [abstractions/namespace-testabstraction] are instantiated from patches in two different 'project folders'. Apparently Pd identifies these files with their absolute path, otherwise it could not load two different versions of [abstractions/namespace-testabstraction], right?
Katja
On Thu, Oct 16, 2014 at 6:21 AM, Jonathan Wilkes via Pd-list pd-list@lists.iem.at wrote:
Hi list, Let's say I make a patch that uses the zexy and hcs libraries. I want my patch to be portable to Pd-extended, Pd-l2ork, and whatever hand-rolled Pd-vanilla+zexy+hcs installations are sitting out there on users machines.
If that is my goal, then I am going to use libdir/objectname syntax for all my zexy and hcs objects in my patch. I am going to do this because if a user ever reports
that there is a nameclash, it is very likely to be due to a bug somewhere, and that bug is very likely to be fixable.
In fact, I will go so far as to say there is no other way to make a portable patch that uses externals on the versions of Pd I described above...
- both [import] and [declare] will happen _after_ the the default
libs of Pd-extended/L2ork are loaded, so I can't use unprefixed object names with impunity (same with .pdrc). And even if it does work, it's not future-proof, as someone may add an object that aliases one I'm using in my patch.
command lines flags don't ship with my patch, and they get parsed _after_ the user or default prefs so again, I'm just hoping to get lucky.
even with Pd-extended's current "load-nothing" dogma, all possible object names are stored globally (in what looks like both prefixed and unprefixed forms, like "foo/bar" and "bar"). So if my patch is an abstraction and I don't use libdir/objectname syntax for the object names within it, all I can do is hope the user hasn't already loaded something that aliases one of my objects. (And again, not future-proof)
I see only two possibilities, then: either the community changes the core functionality so that there is a local object-name table for each canvas-environment*, or libdir/objectname should be the canonical way to make portable patches. Since libdir/objectname exists and "canvas-environment-local namespaces" don't, I suggest libdir/objectname as the only workable approach.
However, I'm only about 3/4 of the way through reading and comprehending the library-loading code in Pd. So if anyone has thoughts or suggestions I'm all ears.
-Jonathan
- canvas-environment = all canvases which share the same $0-- basically, a canvas and its [pd] subpatches (and graphs).
Pd-list@lists.iem.at mailing list UNSUBSCRIBE and account-management -> http://lists.puredata.info/listinfo/pd-list
Maybe Hans can chime in here, because I know he wanted Pd-extended to load nothing by default (not even internal Vanilla objects). In that case there must be some plan to do canvas-environment-local objectmakers, or some other mechanism to keep various abstractions and open patches from polluting each other's namespaces. I don't understand how that would work, and at least on the face of it this seems difficult to do. I also don't see any work done to that end, which is why I'm advocating for libdir/objectname as it already exists.
- namespace-testabstraction works because abstractions do not get registered as methods to the objectmaker. So on the one hand it's a simpler system because you only need to control path priority (and in your test case I _think_ the patch's path is searched first). But those abstractions only get loaded _after_ searching for external libs in any of the available paths. So unfortunately you're back to gambling. (Again, introspection objects like [classinfo] and [pdinfo] are extremely handy here.)
I'll post a guide to how things get loaded in the next few days. It turns out to be a lot more complex than I realized.
Anyway, my idea is:
- keep libs as they are for compatibility
- use prefixes
- develop new libs with an eye toward usability and consistency (easier than wading through old, unmaintained libs and picking and choosing stuff)
-Jonathan
On Thursday, October 16, 2014 9:44 AM, katja katjavetter@gmail.com wrote:
Thanks for bringing up this important topic. Over the years I flip-flopped between [library/objectname] and [declare] a few times, for reasons which I'll try to explain.
The first occasion where I needed [library/objectname] was when [pow~] in pd core had it's inlets swapped between pd versions, while [cyclone/pow~] retained the old inlet order and could be used for backward compatibility.
When Pd-extended stopped loading libraries by default I started using [library/objectname] for externals from all Pd-extended libraries, which was recommended practice.
Later, I came across a few cases where externals moved from one lib to another. For example, [flatspace/knob] became [flatgui/knob]. I feared further library reorganizations and abandoned the rigid [library/objectname] format. Now I started to appreciate the flexibility provided by [declare]. I used declarations like [declare -stdlib flatspace/knob] and [declare -stdlib flatgui/knob] in top level patches to make projects compatible across Pd-extended versions. Also, I use [declare] for binaries which are included in project directories: for my homebrew binaries, or selected Pd-extended binaries in projects which should run with vanilla Pd. It is much easier to change or add a few declarations in the top level patch than changing library names in all object instances in a large project.
But [declare] only adds search paths and it doesn't protect against name clash. I came across [bsaylor/svf~] which has different number of inlets and outlets than [cyclone/svf~]. Without library name, [svf~] instantiates the cyclone version. But when [bsaylor/svf~] was instantiated earlier in the Pd session, or with [import bsaylor] in the patch, the bsaylor version may be instantiated. Without namespace specification Pd seems to use the version that was loaded first within a Pd session.
I realized that instantiating without library namespaces is a gambling. And this does not only apply to binary executables, but to abstractions as well. Imagine you share a Pd project including abstractions and binaries, and you publish a new version of the project, where bugs in externals were fixed or new features were added to abstractions. Users may want to compare versions, and load the old version first in a Pd session. Even when they close the old version of the project before loading the new version, the old executables and abstraction definitions are still in Pd's memory, and Pd will not load the new ones unless they were prefixed by a different namespace, or until Pd is restarted. So, the new project version may seem to still contain the old bugs!
The current solution for such issues is systematic use of lib names and relative paths in object or abstraction instances. For example, include abstractions in a subdirectory named 'abstractions', and instantiate as [abstractions/objectname]. Pd will then load the abstraction from the subdirectory in the project, even when many projects have an 'abstractions' subdirectory containing an abstraction with the same name. This tedious and inflexible approach will only work when directory structures don't change. A project or library structure must be well organized right from the start. If something changes in the structure, you may need to find all instances of a class in your patch(es) and redefine them. Rigid directory structures, and the prospect of minor changes causing heaps of maintenance work, can be an obstacle for future innovations in a project, or in Pd-extended as a whole.
In any case, I'm reluctant to use namespaces for each and every instance. It can make your patch today, and break it tomorrow. It would be super if [declare] would not only add a search path, but also locally enforce priority for it. Say I have my own implementation of [svf~] stored in the 'bin' subdirectory of a Pd project directory. The main patch contains [declare -path bin] and several instances of [svf~]. Pd should search the bin subdirectory and other locally declared paths for an [svf~] executable. If Pd finds the one in the bin subdirectory, it should interpret all instances of [svf~] as [bin/svf~] and instantiate them accordingly. If an object name appears in two locally declared paths, you would still have a name clash. But designing a single project directory without name clashes is not so difficult.
How hard would it be to realize such path priority, I wonder? Currently, Pd can distinguish between different abstractions with the same name and relative path prefix, if they appear in different project directories. See attached test, where different versions [abstractions/namespace-testabstraction] are instantiated from patches in two different 'project folders'. Apparently Pd identifies these files with their absolute path, otherwise it could not load two different versions of [abstractions/namespace-testabstraction], right?
Katja
On Thu, Oct 16, 2014 at 6:21 AM, Jonathan Wilkes via Pd-list pd-list@lists.iem.at wrote:
Hi list, Let's say I make a patch that uses the zexy and hcs libraries. I want my patch to be portable to Pd-extended, Pd-l2ork, and whatever hand-rolled Pd-vanilla+zexy+hcs installations are sitting out there on users machines.
If that is my goal, then I am going to use libdir/objectname syntax for all my zexy and hcs objects in my patch. I am going to do this because if a user ever reports
that there is a nameclash, it is very likely to be due to a bug somewhere, and that bug is very likely to be fixable.
In fact, I will go so far as to say there is no other way to make a portable patch that uses externals on the versions of Pd I described above...
- both [import] and [declare] will happen _after_ the the default
libs of Pd-extended/L2ork are loaded, so I can't use unprefixed object names with impunity (same with .pdrc). And even if it does work, it's not future-proof, as someone may add an object that aliases one I'm using in my patch.
command lines flags don't ship with my patch, and they get parsed _after_ the user or default prefs so again, I'm just hoping to get lucky.
even with Pd-extended's current "load-nothing" dogma, all possible object names are stored globally (in what looks like both prefixed and unprefixed forms, like "foo/bar" and "bar"). So if my patch is an abstraction and I don't use libdir/objectname syntax for the object names within it, all I can do is hope the user hasn't already loaded something that aliases one of my objects. (And again, not future-proof)
I see only two possibilities, then: either the community changes the core functionality so that there is a local object-name table for each canvas-environment*, or libdir/objectname should be the canonical way to make portable patches. Since libdir/objectname exists and "canvas-environment-local namespaces" don't, I suggest libdir/objectname as the only workable approach.
However, I'm only about 3/4 of the way through reading and comprehending the library-loading code in Pd. So if anyone has thoughts or suggestions I'm all ears.
-Jonathan
- canvas-environment = all canvases which share the same $0-- basically, a canvas and its [pd] subpatches (and graphs).
Pd-list@lists.iem.at mailing list UNSUBSCRIBE and account-management -> http://lists.puredata.info/listinfo/pd-list