Hi Frank, Let's take your demos one abstraction at a time: 01_loadhere.pd: [mylib/metaload here] -- doesn't load [$1], because you didn't supply any search paths anywhere. So all the ce_path entries are empty, which means that Pd tries to load "here" in the directory of the metaload abstraction. That directory doesn't contain here.pd, and so it fails to load.

[mylib/metaload there] -- creates under the same logic outlined above. "there.pd" is in metaload's directory. Success.

[mylib/metaload mylib/there] -- The [metaload] abstraction's directory doesn't contain a subdirectory "mylib", so this fails.

[mylib/metaload nbx] and [mylib/metaload hsl] -- these are pointless. [nbx] and [hsl] are internal objects, which means Pd finds them in the objectmaker method table before it even searches for libs to load. Though I guess Hans wants them not to create by default in your situation.

Btw-- you forgot to include [mylib/metaload both] (which would create the both.pd from "mylib" since that's the abstraction's directory and it exists there) and [mylib/metaload mylib/both] (which would fail to create [$1] because [metaload]'s directory doesn't contain a subdirectory "mylib").

02_loadhere-declare-mylib.pd - [declare -path mylib] will go down and "infect" each abstraction's path. (In fact Pd just searches it first when attempting to open the abstractions.)

The [declare] relative logic-- if you feed a relative path to declare, you'll get the concatenation of the toplevel patch's directory with the relative path specified to declare. So in this patch, you'll get <path-to-this-patch>/mylib. (After that I

[mylib/metaload here] -- still doesn't create because there's no "here.pd" file inside "mylib"

[mylib/metaload there] -- creates, using the same logic

[mylib/metaload mylib/there] -- won't create because there's no mylib directory inside mylib

[mylib/metaload both] and [metaload both] -- both create <path-to-this-patch>/mylib/both.pd since it exists

Also, notice that [metaload] creates because of [declare -path mylib]. If you didn't have that it'd fail.

Then why does [mylib/metaload] create? Because after Pd _fails_ to find <path-to-this-path>/mylib/mylib/metaload, it falls back to searching inside the directory of the patch, which would be <path-to-this-patch>/ + mylib/metaload, which successfully creates.

03_loadhere-declare-dot.pd

[declare -path .] concatenates <path-to-this-patch> with "." and gives that path precedence for all its objects plus infecting all objects inside the abstractions with that as the first path to search.

[mylib/metaload here] = [<path-to-this-patch>/./mylib/metaload here]. [$1] is searched first in the path from [declare], which would be <path-to-this-patch>/./here, which creates successfully.

[mylib/metaload there] = [<path-to-this-patch>/./mylib/metaload there]. [$1] is searched first as above, which fails, _but_ then Pd tries the abstraction's directory which holds "there.pd". Success on 2nd try.

[mylib/metaload both] = [<path-to-this-patch>/./mylib/metaload]. [$1] is searched as above, and the toplevel patch's directory has a "both.pd" so that one succeeds. (Thus one in the abstraction's dir is never searched.)

[metaload both] = [<path-to-this-patch>/./metaload both]. "metaload.pd" isn't in the patch's main directory so this fails to create

Do notice that Pd-l2ork's *info objects (or iemguts and hcs externals) are indispensable to figure out exactly what is going on here. Pd Vanilla has no tools to actually see what's going on under the hood, so all you can do is pen-test and hope something meaningful comes out of that.

Also, note that [declare -path .] is not the same as my suggestion for implicit path precendence for the abstraction's directory. This is because any [declare] in a parent of the abstraction will take precedence. At least that's what it looks like is happening inside canvas_open of g_canvas.c.

Finally, you're implicitly doing your own namespacing/namemangling by choosing the subdirectory name "mylib". If you choose a subdir name that's the same as a loaded external library (or even an unloaded one that's in the global path), it will take precedence over your abstractions in the subdir.

-Jonathan

On Monday, October 20, 2014 2:58 PM, Frank Barknecht fbar@footils.org wrote:

Hi,

here's some more food for thought: How to deal with library abstractions that load other objects whose name is set at runtime? This is actually pretty common, for example in abstractions that decorate other abstraction with e.g. polyphony, or when writing soundfile/score loading objects.

I attached a test case that illustrates various combinations one may meet in this case.

Please start Pd in the directory "metaprog" and open:

01_loadhere.pd 02_loadhere-declare-mylib.pd 03_loadhere-declare-dot.pd

Ciao

Frank

On Mon, Oct 20, 2014 at 08:34:13AM -0700, Miller Puckette wrote:

Hi all,

I think it might work simply to maintain a per-patch list of "externs" currently loaded. New patches would start out with an empty list, which

would

then get filled in by that patch's own search procedures.

There might be some complicated problems to resove about such searche within abstractions (indeed, there are already severe problems the way that is set up that I want to try to fix.)

cheers Miller

Oe Sun, Oct 19, 2014 at 11:21:15AM -0700, Jonathan Wilkes via Pd-list wrote:

...

Hi list, Let me describe a typical way to use Pd:

1. patch author makes a patch

patch author abstracts out certain functions into Pd abstractions 3) patch author makes a self-contained collection of these abstractions (and maybe some helper abstractions) 4) patch author takes every _reasonable_ step to make their collection portable-- that is, to ensure that their abstractions and nested abstractions create correctly on any version of Pd and don't collide with anything in the standard library path.

Currently a patch author cannot achieve this. Pure data forces them to care about what names pre-existing externals might have (which is probably why many externals make a common name prefix part of the name of each object). The reason is that an abstraction's directory is not

the first place to be searched for externals or abstractions to be loaded. It is superceded by the searchpath set by [declare] for all of that abstraction's parent canvases.

...

If any of those parent canvases declare a path which has an external or abstraction by the same name as one of the helper abstractions, the object in that path will get created instead of the helper abstraction. Furthermore, if any of those parent canvases declare a path that happens to have the same name as the subdirectory containing the helper abstraction, then "path/objectname" will override "subdirectory/helper". This is even true if you use "./subdirectory/helper".

This means that if the patch author wants to grope toward portability

they have to do this, too:

...

5. read/search all the libnames and object names in pd-extended, and all the other widely used libs and objects scattered about the internet. Then come up with a name that is human readable, but also unique enough to prevent collisions with anything else out there.

I think #5 is unreasonable, especially because creating a little zipped directory with abstractions (or even externals) is such a common way of making ostensibly portable libraries in Pd. Essentially we're asking users to do their own name-mangling.

So why isn't an abstraction's directory the _first_ one to be searched by default in Pd? I understand katja and Matt Barber's reasons

for using [declare] to infect an entire running instance with different path priorities, but I think that's the exception rather than the rule. Can we give abstraction directories precedence in the loading scheme, and maybe use a startup flag to trigger Pd's current behavior?

...

-Jonathan

...

---

Pd-list@lists.iem.at mailing list UNSUBSCRIBE and account-management -> http://lists.puredata.info/listinfo/pd-list

---

Pd-list@lists.iem.at mailing list UNSUBSCRIBE and account-management -> http://lists.puredata.info/listinfo/pd-list

My point is that if you leave these things as an exercise to the user, then the user loses. The rules for how things load are too confusing, and Pd traditionally has too few tools for the user to inspect path order, aside from trying every single possibility.

A user shipping an abstraction library with abstraction helpers that load with impunity... is not actually possible. That means the author of a library of abstractions is subject to whatever search path the parent canvases have set for it, which will _supercede_ the abstraction's directory. "Use your own name-mangling" isn't a serious solution.

-Jonathan

On Tuesday, October 21, 2014 4:40 AM, Frank Barknecht fbar@footils.org wrote:

Hi,

maybe I didn't make clear enough, that of course some of these fail to load the abstractions they may try to load: That's intentional and the whole point of this routine.

The interesting point that in my experience many Pd users are not aware of, is just this: The directory of a calling patch (0*.pd in this case) is not part of the search path for abstractions used by this patch!

This not only applies to abstractions, but also to other material like soundfiles, textfile, textures etc. because there is only one search path.

If you compare 01_loadhere.pd and 03_loadhere-declare-dot.pd you see:

[declare -path .] is a way out here. As Jonathan has analysed most cleverly and as I have shown in a more frugal way: With the directory of the toplevel patch added to its search path it becomes possible to load here.pd from an abstraction as intended.

A side effect is a possible nameclash, illustrated in both.pd. This makes it tricky to use other objects in the library. Using a prefix or more path-mangling is necessary. Updated examples in the attachement should illustrate this as well.

It is left as an exercise for the reader to implement a mylib/meta-metaload.pd that loads "metaload both" while being called from a toplevel patch like [meta-metaload both].

Have fun,

Okay, here's a first pass to outline what's going on with Pd's library loader(s): http://www.jonathanwilkes.net/2014/10/loading-a-library-in-pure-data/

I don't cover the interaction between global and canvas-local searchpaths, nor how exactly the libdir loader from Pd-extended does its thing. If anyone finds errors let me know and I'll correct them.

-Jonathan

On Monday, October 20, 2014 5:43 PM, katja katjavetter@gmail.com wrote:

Thanks for demonstrating [declare -path .], Frank. Food for thought.

On Mon, Oct 20, 2014 at 8:54 PM, Frank Barknecht fbar@footils.org wrote:

Hi,

here's some more food for thought: How to deal with library abstractions that load other objects whose name is set at runtime? This is actually pretty common, for example in abstractions that decorate other abstraction with e.g. polyphony, or when writing soundfile/score loading objects.

I attached a test case that illustrates various combinations one may meet in this case.

Please start Pd in the directory "metaprog" and open:

01_loadhere.pd 02_loadhere-declare-mylib.pd 03_loadhere-declare-dot.pd

Ciao

Frank

On Mon, Oct 20, 2014 at 08:34:13AM -0700, Miller Puckette wrote:

...

Hi all,

I think it might work simply to maintain a per-patch list of "externs" currently loaded. New patches would start out with an empty list, which would then get filled in by that patch's own search procedures.

There might be some complicated problems to resove about such searche within abstractions (indeed, there are already severe problems the way that is set up that I want to try to fix.)

cheers Miller

Oe Sun, Oct 19, 2014 at 11:21:15AM -0700, Jonathan Wilkes via Pd-list wrote:

...

Hi list, Let me describe a typical way to use Pd:

1. patch author makes a patch

patch author abstracts out certain functions into Pd abstractions 3) patch author makes a self-contained collection of these abstractions (and maybe some helper abstractions) 4) patch author takes every _reasonable_ step to make their collection portable-- that is, to ensure that their abstractions and nested abstractions create correctly on any version of Pd and don't collide with anything in the standard library path.

Currently a patch author cannot achieve this. Pure data forces them to care about what names pre-existing externals might have (which is probably why many externals make a common name prefix part of the name of each object). The reason is that an abstraction's directory is not the first place to be searched for externals or abstractions to be loaded. It is superceded by the searchpath set by [declare] for all of that abstraction's parent canvases.

If any of those parent canvases declare a path which has an external or abstraction by the same name as one of the helper abstractions, the object in that path will get created instead of the helper abstraction. Furthermore, if any of those parent canvases declare a path that happens to have the same name as the subdirectory containing the helper abstraction, then "path/objectname" will override "subdirectory/helper". This is even true if you use "./subdirectory/helper".

This means that if the patch author wants to grope toward portability they have to do this, too: 5) read/search all the libnames and object names in pd-extended, and all the other widely used libs and objects scattered about the internet. Then come up with a name that is human readable, but also unique enough to prevent collisions with anything else out there.

I think #5 is unreasonable, especially because creating a little zipped directory with abstractions (or even externals) is such a common way of making ostensibly portable libraries in Pd. Essentially we're asking users to do their own name-mangling.

So why isn't an abstraction's directory the _first_ one to be searched by default in Pd? I understand katja and Matt Barber's reasons for using [declare] to infect an entire running instance with different path priorities, but I think that's the exception rather than the rule. Can we give abstraction directories precedence in the loading scheme, and maybe use a startup flag to trigger Pd's current behavior?

-Jonathan

...

---

Pd-list@lists.iem.at mailing list UNSUBSCRIBE and account-management -> http://lists.puredata.info/listinfo/pd-list

---

Pd-list@lists.iem.at mailing list UNSUBSCRIBE and account-management -> http://lists.puredata.info/listinfo/pd-list

-- Frank Barknecht _ ______footils.org__

---

Pd-list@lists.iem.at mailing list UNSUBSCRIBE and account-management -> http://lists.puredata.info/listinfo/pd-list

Pd-list@lists.iem.at mailing list UNSUBSCRIBE and account-management -> http://lists.puredata.info/listinfo/pd-list