hi miller et all,
the current implementation of Pd's loading mechanism works as follows:
- each registered loader is asked whether they can handle a given objectclass - each loader in turn searches the entire (canvas-enhanced) path for files matching their pattern (eg.g. "*.pd_linux", or "*.pd_lua") - if all fails, Pd searches the entire (canvas-enhanced) path for abstractions.
e.g. /A/x.pd_linux /B/x.pd_linux /C/x.pd_linux /A/x.pd_lua /B/x.pd_lua /C/x.pd_lua /A/x.pd /B/x.pd /C/x.pd
this has a number of disadvantages, mainly:
- the associated loader has a higher priority then the path. this means that it is nigh¹ impossible to override a binary external (e.g. "*.pd_darwin") installed system-wide (e.g. in extra/) with a user-provided alternative (e.g. a pd_lua implementation in the current patch path).
- abstractions (arguably the most used external classes) have an extra load time penalty, since each time a new (abstraction-provided) object is created, Pd will first search the entire path for binaries, lua-scripts and what not.
iirc, this has been discussed at length, and the proper solution for this is to change the loading mechanism, so that paths have a higher that loaders: Pd shall iterate over each search-path and ask the loaders whether there is something for them in there.
this way an abstraction in a "-path" enhanced searchpath is always found early in the search, AND it can shadow an external in a later search path.
e.g. /A/x.pd_linux /A/x.pd_lua /A/x.pd /B/x.pd_linux /B/x.pd_lua /B/x.pd /C/x.pd_linux /C/x.pd_lua /C/x.pd
so i have taken the liberty and implement that (i'll submit the patches in a second).
here's a few observations:
compatibility with legacy loaders === the new implementation needs an API that tells the loaders where they should look for. rather than create a new API, i used the existing one and enhanced the callback function with a new argument <path>, so we now have: int loader_callback(t_canvas*c, char*name, char*path);
the good thing about this is that it is actually backward compatible (at least with sane calling conventions, as on my tested linux system). legacy loaders will simply ignore the <path> argument, and continue to search their files in the entire path. this will result in an overall performance degredation (while loading objects), as the legacy loaders will call the entire search path for each element in the search path :-( the good news is that: - the legacy loaders will continue to work! - the performance degredation will only happen if legacy loaders are involved. - the performance degredation will be at *load time*, so should not be too important - there are only very few (known) loaders, so they should be fixed in no time
abstraction loading === one big change is that abstractions are now loaded via a proper "loader" (rather than via some special handling as fallback) this implies that whenever the abstraction-loader wins (that is, a given objectname can first be resolved via an abstraction), this objectname is tagged as "abstraction". when another object of the same name is created, the loader competition will not take place any more, and instead the object will be loaded immediately as "abstraction" (the current implementation then still searches the entire path for the .pd file) this has two side effects: - until now it was possible to replace "future instances" of an abstraction-object with externals. e.g. imagine that [foo] is resolved via ~/pd-externals/foo.pd. if someone installs a foo.pd_linux into ~/pd-externals/ during the Pd session, then the next time someone creates [foo] it will be the external! whether you like that behaviour or not, my implementation prevents it (however it is still possible to change the actual abstraction; e.g. if [foo] is resolved via /usr/lib/pd/extra/foo.pd and somebody saves a foo.pd into ~/pd-externals/, then the next time someone creates [foo] it will be the abstraction in ~/pd-externals/!) - since this means that we don't need to search the entire path for externals (that have not been there when we looked a second ago) whenever we create another instance of our abstraction, patches generally load faster (esp. when an abstraction is used often). the speedup is not tremendous, about 3x; but still...
canvas path iterator === to ease the implementation (and to avoid code duplication), i introduced an iterator, that will call a callback function for each searchpath of a canvas. this is now also used in canvas_open(), but might be useful elsewhere.
interactive use of declare === as a side effect (and not strictly related to the loading problem), i fixed [declare] so that creating a [declare -path foo] would allow you to *immediately* use the foo/ path. i always found it *very* annoying that you had to close/open the abstraction for [declare] to have an effect. (this functionality is disabled while the patch is currently /loading/)
not done === what i wanted to do as well, but haven't gotten around was proper canvas-local library loading, so you could do [declare -lib foo/bla]+[bla] in one abstraction and [declare -lib moo/bla]+[bla] in another abstraction and the two [bla] objectclasses would refer to different externals.
hmm, that mail is already quite long. and i have forgotten what else i wanted to say.
in any case, i would like to hear your opinion on all this.
in the meantime i'll submit the 4 patches of this implementation to the sf tracker.
gfamds IOhannes
¹ ah well, for certain use-cases one could disable the standard searchpaths altogether; not.
On 09/23/2015 10:20 PM, IOhannes m zmölnig wrote:
in the meantime i'll submit the 4 patches of this implementation to the sf tracker.
it's 5 patches, and they can be found at
https://sourceforge.net/p/pure-data/patches/560
gmsdr IOhannes
Oh man, I was absolutely dreading addressing all that loading mess, so I'm excited to see you're tackling it. Question:what does it mean to "tag" an objectname with "abstraction"? Another question:How were you planning to implement canvas-local loading? (Or did you have a plan in mind?) -Jonathan
On Wednesday, September 23, 2015 4:20 PM, IOhannes m zmölnig zmoelnig@iem.at wrote:
hi miller et all,
the current implementation of Pd's loading mechanism works as follows:
- each registered loader is asked whether they can handle a given objectclass - each loader in turn searches the entire (canvas-enhanced) path for files matching their pattern (eg.g. "*.pd_linux", or "*.pd_lua") - if all fails, Pd searches the entire (canvas-enhanced) path for abstractions.
e.g. /A/x.pd_linux /B/x.pd_linux /C/x.pd_linux /A/x.pd_lua /B/x.pd_lua /C/x.pd_lua /A/x.pd /B/x.pd /C/x.pd
this has a number of disadvantages, mainly:
- the associated loader has a higher priority then the path. this means that it is nigh¹ impossible to override a binary external (e.g. "*.pd_darwin") installed system-wide (e.g. in extra/) with a user-provided alternative (e.g. a pd_lua implementation in the current patch path).
- abstractions (arguably the most used external classes) have an extra load time penalty, since each time a new (abstraction-provided) object is created, Pd will first search the entire path for binaries, lua-scripts and what not.
iirc, this has been discussed at length, and the proper solution for this is to change the loading mechanism, so that paths have a higher that loaders: Pd shall iterate over each search-path and ask the loaders whether there is something for them in there.
this way an abstraction in a "-path" enhanced searchpath is always found early in the search, AND it can shadow an external in a later search path.
e.g. /A/x.pd_linux /A/x.pd_lua /A/x.pd /B/x.pd_linux /B/x.pd_lua /B/x.pd /C/x.pd_linux /C/x.pd_lua /C/x.pd
so i have taken the liberty and implement that (i'll submit the patches in a second).
here's a few observations:
compatibility with legacy loaders === the new implementation needs an API that tells the loaders where they should look for. rather than create a new API, i used the existing one and enhanced the callback function with a new argument <path>, so we now have: int loader_callback(t_canvas*c, char*name, char*path);
the good thing about this is that it is actually backward compatible (at least with sane calling conventions, as on my tested linux system). legacy loaders will simply ignore the <path> argument, and continue to search their files in the entire path. this will result in an overall performance degredation (while loading objects), as the legacy loaders will call the entire search path for each element in the search path :-( the good news is that: - the legacy loaders will continue to work! - the performance degredation will only happen if legacy loaders are involved. - the performance degredation will be at *load time*, so should not be too important - there are only very few (known) loaders, so they should be fixed in no time
abstraction loading === one big change is that abstractions are now loaded via a proper "loader" (rather than via some special handling as fallback) this implies that whenever the abstraction-loader wins (that is, a given objectname can first be resolved via an abstraction), this objectname is tagged as "abstraction". when another object of the same name is created, the loader competition will not take place any more, and instead the object will be loaded immediately as "abstraction" (the current implementation then still searches the entire path for the .pd file) this has two side effects: - until now it was possible to replace "future instances" of an abstraction-object with externals. e.g. imagine that [foo] is resolved via ~/pd-externals/foo.pd. if someone installs a foo.pd_linux into ~/pd-externals/ during the Pd session, then the next time someone creates [foo] it will be the external! whether you like that behaviour or not, my implementation prevents it (however it is still possible to change the actual abstraction; e.g. if [foo] is resolved via /usr/lib/pd/extra/foo.pd and somebody saves a foo.pd into ~/pd-externals/, then the next time someone creates [foo] it will be the abstraction in ~/pd-externals/!) - since this means that we don't need to search the entire path for externals (that have not been there when we looked a second ago) whenever we create another instance of our abstraction, patches generally load faster (esp. when an abstraction is used often). the speedup is not tremendous, about 3x; but still...
canvas path iterator === to ease the implementation (and to avoid code duplication), i introduced an iterator, that will call a callback function for each searchpath of a canvas. this is now also used in canvas_open(), but might be useful elsewhere.
interactive use of declare === as a side effect (and not strictly related to the loading problem), i fixed [declare] so that creating a [declare -path foo] would allow you to *immediately* use the foo/ path. i always found it *very* annoying that you had to close/open the abstraction for [declare] to have an effect. (this functionality is disabled while the patch is currently /loading/)
not done === what i wanted to do as well, but haven't gotten around was proper canvas-local library loading, so you could do [declare -lib foo/bla]+[bla] in one abstraction and [declare -lib moo/bla]+[bla] in another abstraction and the two [bla] objectclasses would refer to different externals.
hmm, that mail is already quite long. and i have forgotten what else i wanted to say.
in any case, i would like to hear your opinion on all this.
in the meantime i'll submit the 4 patches of this implementation to the sf tracker.
gfamds IOhannes
¹ ah well, for certain use-cases one could disable the standard searchpaths altogether; not.
_______________________________________________ Pd-dev mailing list Pd-dev@lists.iem.at http://lists.puredata.info/listinfo/pd-dev
And a question to make sure I understand what you're proposing: Inside "foo.pd", I have a single object [bar] "foo.pd" lives in directory /home/blah The only other file in /home/blah is "bar.pd" Under your revision of the loading mechanism, when I load "foo.pd" is [bar] guaranteed to be loaded from /home/blah/bar.pd ? -Jonathan
On Wednesday, September 23, 2015 5:05 PM, Jonathan Wilkes jancsika@yahoo.com wrote:
Oh man, I was absolutely dreading addressing all that loading mess, so I'm excited to see you're tackling it. Question:what does it mean to "tag" an objectname with "abstraction"? Another question:How were you planning to implement canvas-local loading? (Or did you have a plan in mind?) -Jonathan
On Wednesday, September 23, 2015 4:20 PM, IOhannes m zmölnig zmoelnig@iem.at wrote:
hi miller et all,
the current implementation of Pd's loading mechanism works as follows:
- each registered loader is asked whether they can handle a given objectclass - each loader in turn searches the entire (canvas-enhanced) path for files matching their pattern (eg.g. "*.pd_linux", or "*.pd_lua") - if all fails, Pd searches the entire (canvas-enhanced) path for abstractions.
e.g. /A/x.pd_linux /B/x.pd_linux /C/x.pd_linux /A/x.pd_lua /B/x.pd_lua /C/x.pd_lua /A/x.pd /B/x.pd /C/x.pd
this has a number of disadvantages, mainly:
- the associated loader has a higher priority then the path. this means that it is nigh¹ impossible to override a binary external (e.g. "*.pd_darwin") installed system-wide (e.g. in extra/) with a user-provided alternative (e.g. a pd_lua implementation in the current patch path).
- abstractions (arguably the most used external classes) have an extra load time penalty, since each time a new (abstraction-provided) object is created, Pd will first search the entire path for binaries, lua-scripts and what not.
iirc, this has been discussed at length, and the proper solution for this is to change the loading mechanism, so that paths have a higher that loaders: Pd shall iterate over each search-path and ask the loaders whether there is something for them in there.
this way an abstraction in a "-path" enhanced searchpath is always found early in the search, AND it can shadow an external in a later search path.
e.g. /A/x.pd_linux /A/x.pd_lua /A/x.pd /B/x.pd_linux /B/x.pd_lua /B/x.pd /C/x.pd_linux /C/x.pd_lua /C/x.pd
so i have taken the liberty and implement that (i'll submit the patches in a second).
here's a few observations:
compatibility with legacy loaders === the new implementation needs an API that tells the loaders where they should look for. rather than create a new API, i used the existing one and enhanced the callback function with a new argument <path>, so we now have: int loader_callback(t_canvas*c, char*name, char*path);
the good thing about this is that it is actually backward compatible (at least with sane calling conventions, as on my tested linux system). legacy loaders will simply ignore the <path> argument, and continue to search their files in the entire path. this will result in an overall performance degredation (while loading objects), as the legacy loaders will call the entire search path for each element in the search path :-( the good news is that: - the legacy loaders will continue to work! - the performance degredation will only happen if legacy loaders are involved. - the performance degredation will be at *load time*, so should not be too important - there are only very few (known) loaders, so they should be fixed in no time
abstraction loading === one big change is that abstractions are now loaded via a proper "loader" (rather than via some special handling as fallback) this implies that whenever the abstraction-loader wins (that is, a given objectname can first be resolved via an abstraction), this objectname is tagged as "abstraction". when another object of the same name is created, the loader competition will not take place any more, and instead the object will be loaded immediately as "abstraction" (the current implementation then still searches the entire path for the .pd file) this has two side effects: - until now it was possible to replace "future instances" of an abstraction-object with externals. e.g. imagine that [foo] is resolved via ~/pd-externals/foo.pd. if someone installs a foo.pd_linux into ~/pd-externals/ during the Pd session, then the next time someone creates [foo] it will be the external! whether you like that behaviour or not, my implementation prevents it (however it is still possible to change the actual abstraction; e.g. if [foo] is resolved via /usr/lib/pd/extra/foo.pd and somebody saves a foo.pd into ~/pd-externals/, then the next time someone creates [foo] it will be the abstraction in ~/pd-externals/!) - since this means that we don't need to search the entire path for externals (that have not been there when we looked a second ago) whenever we create another instance of our abstraction, patches generally load faster (esp. when an abstraction is used often). the speedup is not tremendous, about 3x; but still...
canvas path iterator === to ease the implementation (and to avoid code duplication), i introduced an iterator, that will call a callback function for each searchpath of a canvas. this is now also used in canvas_open(), but might be useful elsewhere.
interactive use of declare === as a side effect (and not strictly related to the loading problem), i fixed [declare] so that creating a [declare -path foo] would allow you to *immediately* use the foo/ path. i always found it *very* annoying that you had to close/open the abstraction for [declare] to have an effect. (this functionality is disabled while the patch is currently /loading/)
not done === what i wanted to do as well, but haven't gotten around was proper canvas-local library loading, so you could do [declare -lib foo/bla]+[bla] in one abstraction and [declare -lib moo/bla]+[bla] in another abstraction and the two [bla] objectclasses would refer to different externals.
hmm, that mail is already quite long. and i have forgotten what else i wanted to say.
in any case, i would like to hear your opinion on all this.
in the meantime i'll submit the 4 patches of this implementation to the sf tracker.
gfamds IOhannes
¹ ah well, for certain use-cases one could disable the standard searchpaths altogether; not.
_______________________________________________ Pd-dev mailing list Pd-dev@lists.iem.at http://lists.puredata.info/listinfo/pd-dev
On 09/23/2015 11:09 PM, Jonathan Wilkes via Pd-dev wrote:
And a question to make sure I understand what you're proposing: Inside "foo.pd", I have a single object [bar] "foo.pd" lives in directory /home/blah The only other file in /home/blah is "bar.pd" Under your revision of the loading mechanism, when I load "foo.pd" is [bar] guaranteed to be loaded from /home/blah/bar.pd ?
i guess so.
however, i just checked and there *is* a way to trigger a peculiarity:
additional to your specs, consider: - /home/fugl contains a file "bar.pd" - /home/blah/bar.pd contains a [declare -path /home/fugl] - /home/blah/foo.pd is saved with an instance of that (/home/blah) [bar]
THEN
-> when you load "foo.pd", the [bar] will be loaded from /home/fugl/bar.pd
that's related to the (dreaded) behaviour of [declare], that injects declare-statements into parent patches (without corresponding objects). i *believe* this is a bug in [declare] (and actually wanted to fix this on the go as well, but forgot about it).
gfmasre IOhannes
On 09/23/2015 11:05 PM, Jonathan Wilkes via Pd-dev wrote:
Oh man, I was absolutely dreading addressing all that loading mess, so I'm excited to see you're tackling it. Question:what does it mean to "tag" an objectname with "abstraction"?
it means, that Pd now knows that [objectname] is to be constructed with a special new-method that happens to always emit abstractions.
Another question:How were you planning to implement canvas-local loading? (Or did you have a plan in mind?)
i had a naïve approach in mind that would just re-direct pd_objectmaker to a per-canvas objectmaker. spending a few hours without pen and paper (and a non-booting desktop machine for diversion) i got a headache and dropped it.
a proper solution (that does not treat pd_objectmaker any different from other classes) *might* involve adding a primitive inheritance system to Pd's objectclasses. however, i don't have very much hope that such a change would ever make it into Pd, so i would rather have miller's input on how he actually would like [declare -lib] to behave. or for that matter, how [declare] should behave in abstractions at all. the help-patch still warns against using it outside of top-level patches.
gmdsar IOhannes
On 09/23/2015 11:45 PM, IOhannes m zmölnig wrote:
spending a few hours without pen and paper (and a non-booting desktop machine for diversion) i got a headache and dropped it.
just to get that right: i *did* spend those hours coding a solution, but miserably failed.
gfmdsar IOhannes
In other words, [declare] objects within any child abstractions get hoisted to thetop of the relevant Pd file (in the form of an #X declare statement). Another question because I can't remember-- where does the patch's owndirectory fall in the search path scheme? Also-- can one control the order of the loaders, esp. the default loader andthe abstraction loader? Finally-- it's too bad that laws against hacking are so stringent. If they weren'tthis would be the perfect time to send you a response from Miller's email addysaying, "I trust your judgment and will accept whatever patch-local loadingsolution you come up with." Perhaps that'd be enough to convince you todevote an unfathomable _2nd_ sitting to code up a solution to a hard problem.
And when you finally figured out it was a ruse, I'd be on a plane to the bahamaswith your code safely committed to my GUI port repo.
-Jonathan
On Wednesday, September 23, 2015 5:47 PM, IOhannes m zmölnig zmoelnig@iem.at wrote:
On 09/23/2015 11:45 PM, IOhannes m zmölnig wrote:
spending a few hours without pen and paper (and a non-booting desktop machine for diversion) i got a headache and dropped it.
just to get that right: i *did* spend those hours coding a solution, but miserably failed.
gfmdsar IOhannes
_______________________________________________ Pd-dev mailing list Pd-dev@lists.iem.at http://lists.puredata.info/listinfo/pd-dev
On 2015-09-24 03:19, Jonathan Wilkes wrote:
In other words, [declare] objects within any child abstractions get hoisted to thetop of the relevant Pd file (in the form of an #X declare statement).
yes
Another question because I can't remember-- where does the patch's owndirectory fall in the search path scheme?
without checking, i think the order is: - canvas-local paths - [declare -path] - [declare -stdpath] - abstraction path - system paths - "-path" - standard search-paths (~/pd-externals/ ... /usr/local/lib/pd/extra/)
Also-- can one control the order of the loaders, esp. the default loader and the abstraction loader?
no. the order is as follows: 1. "external" loader 2..(n-1). user-defined loaders (in the order of registration==loading) n. abstraction loader
fgnsd+ IOhannes
The whole thing's a horrible mess, and with each tweaky patch it gets worse. The only thing for it is going to be for me to stop doing anything else for a month or so and think the whole thing through.
In addition to the problems already mentioned: abstractions with 'declares' pollute their calling patches search is by ending first, then by directory "loaders" aren't treated in the same way as abtractions
here are a couple more (perhaps more minor) ones: Searching a fixed directory like "~/pd-externals" is a rotten idea - I don't know why I ever accepted it as an idea. And now the "deken" engine puts patches there! "." in a path means "the current directory" when it should have meant "the directory this patch right here is in".
I don't think there's a ghost of a chance of making this all be sane and staying back compatible - the only thing I can think to do is make a whole new parallel structure and have a "compatibility" flag that throws you back to the old regime.
Anyway, I'm off to the ICMC and will have to think abotu other things for yet another week...
cheers Miller
On Thu, Sep 24, 2015 at 01:19:51AM +0000, Jonathan Wilkes via Pd-dev wrote:
In other words, [declare] objects within any child abstractions get hoisted to thetop of the relevant Pd file (in the form of an #X declare statement). Another question because I can't remember-- where does the patch's owndirectory fall in the search path scheme? Also-- can one control the order of the loaders, esp. the default loader andthe abstraction loader? Finally-- it's too bad that laws against hacking are so stringent. If they weren'tthis would be the perfect time to send you a response from Miller's email addysaying, "I trust your judgment and will accept whatever patch-local loadingsolution you come up with." Perhaps that'd be enough to convince you todevote an unfathomable _2nd_ sitting to code up a solution to a hard problem.
And when you finally figured out it was a ruse, I'd be on a plane to the bahamaswith your code safely committed to my GUI port repo.
-Jonathan
On Wednesday, September 23, 2015 5:47 PM, IOhannes m zmölnig <zmoelnig@iem.at> wrote:On 09/23/2015 11:45 PM, IOhannes m zmölnig wrote:
spending a few hours without pen and paper (and a non-booting desktop machine for diversion) i got a headache and dropped it.
just to get that right: i *did* spend those hours coding a solution, but miserably failed.
gfmdsar IOhannes
Pd-dev mailing list Pd-dev@lists.iem.at http://lists.puredata.info/listinfo/pd-dev
Pd-dev mailing list Pd-dev@lists.iem.at http://lists.puredata.info/listinfo/pd-dev
On 09/24/2015 05:16 PM, Miller Puckette wrote:
The whole thing's a horrible mess, and with each tweaky patch it gets worse. The only thing for it is going to be for me to stop doing anything else for a month or so and think the whole thing through.
In addition to the problems already mentioned: abstractions with 'declares' pollute their calling patches
search is by ending first, then by directory
my patch set addresses exactly this problem. it now searches directories first, then by ending (or whatever the loader picks to filter files).
"loaders" aren't treated in the same way as abtractions
??
Anyway, I'm off to the ICMC and will have to think abotu other things for yet another week...
have fun.
dsamr IOhannes
On 09/24/2015 05:16 PM, Miller Puckette wrote:
I don't think there's a ghost of a chance of making this all be sane and staying back compatible - the only thing I can think to do is make a whole new parallel structure and have a "compatibility" flag that throws you back to the old regime.
nah, please don't!
i'm pretty sure that for most cases one can achieve backward compatibility by clever re-arrangement of cmdline "-path"s.
up until now, playing with paths (and loaders) was near-magic - if not plain black magic. i don't think that many things will break by changing it (in a way that people won't say: "ah, i finally understand how this is supposed to work")
supporting *both* behaviours will only lead to unmaintainable code; and either create a lot of code duplication, or lead to a re-implementation of the original code (adding new bugs; do we want to have a compat flag for that as well?)
gfmads IOhannes
Yeah - having never used a 'loader' at all, they appear like magic to me. Anyway, careful thought is needed here...
I forgot yet one more problem I want to resolve: readsf~ and writesf~ use the path in a non-thread-safe way.
The problem I mentioned that you couldn't identify from my description earlier was that, if anyone ever loads an extern named "foo" (for instance) then all the search path business will be short-circuited when anyone says "foo" in an object box, as if "foo" were a built-in object. This can happen even in the middle of loading a patch so that some "foo" objects get one thing and others another. I don't know ifthis extends to other "loaders" or not.
cheers M
On Thu, Sep 24, 2015 at 05:29:49PM +0200, IOhannes m zmölnig wrote:
On 09/24/2015 05:16 PM, Miller Puckette wrote:
I don't think there's a ghost of a chance of making this all be sane and staying back compatible - the only thing I can think to do is make a whole new parallel structure and have a "compatibility" flag that throws you back to the old regime.
nah, please don't!
i'm pretty sure that for most cases one can achieve backward compatibility by clever re-arrangement of cmdline "-path"s.
up until now, playing with paths (and loaders) was near-magic - if not plain black magic. i don't think that many things will break by changing it (in a way that people won't say: "ah, i finally understand how this is supposed to work")
supporting *both* behaviours will only lead to unmaintainable code; and either create a lot of code duplication, or lead to a re-implementation of the original code (adding new bugs; do we want to have a compat flag for that as well?)
gfmads IOhannes
Pd-dev mailing list Pd-dev@lists.iem.at http://lists.puredata.info/listinfo/pd-dev
On 09/24/2015 05:36 PM, Miller Puckette wrote:
The problem I mentioned that you couldn't identify from my description earlier was that, if anyone ever loads an extern named "foo" (for instance) then all the search path business will be short-circuited when anyone says "foo" in an object box, as if "foo" were a built-in object. This can happen even in the middle of loading a patch so that some "foo" objects get one thing and others another. I don't know if this extends to other "loaders" or not.
ah i see. the problem extends to anything that can register a class (via class_new()), and loaders do that quite often. (after all, that's their main business).
this is basically what could be solved with canvas-local pd_objectmakers, and which is the part that *does* require careful thinking (and a trip to the bahamas).
gfmdsr IOhannes
On 09/24/2015 05:16 PM, Miller Puckette wrote:
In addition to the problems already mentioned: abstractions with 'declares' pollute their calling patches
i just submitted a 1-line patch that fixes this: https://sourceforge.net/p/pure-data/patches/561/
fgsrd IOhannes
On 24/09/15 23:16, Miller Puckette wrote:
Searching a fixed directory like "~/pd-externals" is a rotten idea - I don't know why I ever accepted it as an idea. And now the "deken" engine puts patches there!
Apart from polluting the user's home directory with a weird looking folder, why do you think it's a rotten idea?
Whatever you guys figure out I hope there is still some easy way for Pd users to add libraries in a standard way without having to have root and without each user having to specify it in a config manually.
Requiring the user to add each abstraction/external to their config is onerous for systems like deken and for distribution of abstractions in general.
I don't have a problem with a "fixed directory" (and I think good defaults in general are better than making a user specify or configure everything from scratch) but I always thought the path should be ~/.config/pd-externals or ~/.local/pd-externals - something out of the way rather than clogging up the user's home directory.
I'm sure we'll update deken to support whatever the new method of externals resolution becomes.
Cheers,
Chris.
Am 25. September 2015 01:45:28 MESZ, schrieb Chris McCormick chris@mccormick.cx:
I don't have a problem with a "fixed directory" (and I think good defaults in general are better than making a user specify or configure everything from scratch) but I always thought the path should be ~/.config/pd-externals or ~/.local/pd-externals - something out of the way rather than clogging up the user's home directory.
My thinking, exactly.
I'm sure we'll update deken to support whatever the new method of externals resolution becomes.
Just to make sure there is no misunderstanding: deken does not know anything about pd's paths. Instead it asks pd for a list of "default" paths at startup and uses the first writeable one.
If pd ever changes its default user-installed search path, deken will automatically pick the new one(s)
mfg.ugd.fhj IOhannes
-- Sent from my pdp-11
I'm sure we'll update deken to support whatever the new method of externals resolution becomes.
Just to make sure there is no misunderstanding: deken does not know anything about pd's paths. Instead it asks pd for a list of "default" paths at startup and uses the first writeable one.
If pd ever changes its default user-installed search path, deken will automatically pick the new one(s)
Aha - I shold have looked nstead of assuming.
I think the best "default" default would be to put things in ~/pd/extra, assuming there's no problem writing to that (and, I suppose, assuming it isn't shared among multiple users of the same machine).
I'd like to get a better understanding of all this n part because I'd like to incorporate deken into Pd vanilla assuming its developers are cool with that.
cheers M
On 25/09/15 23:04, Miller Puckette wrote:
I think the best "default" default would be to put things in ~/pd/extra, assuming there's no problem writing to that (and, I suppose, assuming it isn't shared among multiple users of the same machine).
I guess it's customary to put per-user application specific configurations and other data in a hidden folder, so e.g. ~/.pd or ~/.pd-externals or ~/.local/share/pd/ etc.
To my mind ~/pd/extra is not that different to ~/pd-externals in that it still forces the user to have a non-hidden folder in their home directory. Whenever applications do this I find it mildly annoying.
I'd like to get a better understanding of all this n part because I'd like to incorporate deken into Pd vanilla assuming its developers are cool with that.
I don't know about IOhannes [who at this point has probably contributed more lines of code than me to deken] but I am definitely cool with that.
If this happened it would probably make sense to split the tcl plugin out to go into Pd vanilla and keep the deken packaging script as a separate project.
I mean, I would be ok with having both in Pd but I am not sure you'd want my weird hybrid Python-LISP Frankenstein code hanging around in Pd's repository (although we are getting close to Halloween so you never know).
Cheers,
Chris.
On Sat, Sep 26, 2015 at 06:44:54PM +0800, Chris McCormick wrote:
On 25/09/15 23:04, Miller Puckette wrote:
I think the best "default" default would be to put things in ~/pd/extra, assuming there's no problem writing to that (and, I suppose, assuming it isn't shared among multiple users of the same machine).
I guess it's customary to put per-user application specific configurations and other data in a hidden folder, so e.g. ~/.pd or ~/.pd-externals or ~/.local/share/pd/ etc.
To my mind ~/pd/extra is not that different to ~/pd-externals in that it still forces the user to have a non-hidden folder in their home directory. Whenever applications do this I find it mildly annoying.
I thing for "settings" a hidden file is appropriate (e.g., ".pdsettings") but for libraries you want them visible - but precisely where would depend how you organize your files so should be settable.
My reason for suggesting putting them in "pd/extra" is that you already put "pd" somewhere (and presumably chose where to put it) and if you relocate pd later the extra files will follow. Also, you can then have different versions of Pd with different libraries loaded.
cheers M
On 09/26/2015 02:25 PM, Miller Puckette wrote:
On Sat, Sep 26, 2015 at 06:44:54PM +0800, Chris McCormick wrote:
On 25/09/15 23:04, Miller Puckette wrote:
I think the best "default" default would be to put things in ~/pd/extra, assuming there's no problem writing to that (and, I suppose, assuming it isn't shared among multiple users of the same machine).
I guess it's customary to put per-user application specific configurations and other data in a hidden folder, so e.g. ~/.pd or ~/.pd-externals or ~/.local/share/pd/ etc.
+1 (see below)
To my mind ~/pd/extra is not that different to ~/pd-externals in that it still forces the user to have a non-hidden folder in their home directory. Whenever applications do this I find it mildly annoying.
+1 (see belower)
here's a few disadvantages of ~/pd/extra compared to ~/pd-externals:
- it is yet *another* Pd-related directory cluttering my home-directory.
- it is a two-level directory where the lower level (extra/) doesn't discriminate against anything. this is confusing at best (the only thing i can think of is that ~/pd/ is meant to hold other Pd-related stuff that isn't meant to be loaded as external/abstraction/footage/...)
- it breaks compatibility for no good reason (well "pd" is a slightly more pleasing directory name than "pd-externals", but that's about it)
I thing for "settings" a hidden file is appropriate (e.g., ".pdsettings") but for libraries you want them visible - but precisely where would depend how you organize your files so should be settable.
i see the point (and iirc it's the same as hc's original arguing for ~/pd-externals), though i don't follow it.
also, other applications already use hidden directories for libraries.
e.g. using `pip` to install python-packages on a per-user path will put them into ~/.local/lib/pythonXY/site-packages/
e.g. installing extensions to firefox will put them into ~/.mozilla/firefox/<ID>/extensions/
e.g. installing extensions to thunderbird will similarily install them into ~/.thunderbird/<ID>/extensions/
e.g installing entire games with Steam will install them somewhere into ~/.steam/
so i *do* think that a hidden directly is a sensible place to install libraries to.
My reason for suggesting putting them in "pd/extra" is that you already put "pd" somewhere (and presumably chose where to put it) and if you relocate pd later the extra files will follow. Also, you can then have different versions of Pd with different libraries loaded.
reading this it seems that you really mean Pd's "internal" extra folder (the one besides the "tcl/" folder containing e.g. pd-gui.tcl), rather than ${HOME}/pd/extra.
because if I have a current Pd in ~/src/pd-0.47-2/ and copy that to ~/projects/hellowien/pd, this won't touch ~/pd/extra in any way (so doesn't help with relocating).
now this scheme *only* makes sense if the user has full control over the directory containing Pd.
i honestly don't think that this is the case for most users of Pd on linux. instead I do think that the usual use of Pd is to either install Pd into /usr/local by running `sudo make install` after building or to simply install a package for my distributions (since i maintain the Pd package for Debian and derivates, i'm highly biased towards the latter). both ways take away the capability from the user to install random stuff into extra/ from within Pd.
so i still believe that it is a good idea to have a place where the user can install libraries to without any special permissions, regardless where the Pd binary is installed.
i think those people who do want run Pd from a non-standard directory (~/projects/shneider/pd/bin) and how want to be able to relocate this Pd with all libraries to a different computer (including all libraries), are somewhat "power-users". it should be easy enough to setup a symlink from ~/pd-externals to ~/projects/shneider/pd/extra, and all deken-install externals will magically end-up within the local Pd's extra folder.
an even better solution would be to add a "embed needed libraries into Pd" action (from some menu), that would iterate over all dependencies (e.g. as declared with [declare]) within the currently loaded patch(es) and copy those found in ~/pd-externals/ into pd/extra.
mdsar IOhannes
On Sun, Sep 27, 2015 at 3:30 PM, IOhannes m zmölnig zmoelnig@iem.at wrote:
so i still believe that it is a good idea to have a place where the user can install libraries to without any special permissions, regardless where the Pd binary is installed.
It would also be a good idea if the non-power user could see, from within Pd, a list of all the directories Pd will scan for externals and abstractions. Then the user could choose which one to use. The Pd I have here (Pd 0.46.6) has preferences->path which contains the single item /usr/lib/pd/extra, which is not user-writeable. I can't find any reference to the settings file. There is a 'save all settings' that may or may not actually do something, but I'd need to be a power user to figure out which file was changed.
As far as hidden directories they are (by design) hard to see...you have to list your directories in a terminal with ls -al, usually not possible with the double-click-on-an-icon approach which the non-power user tends to adopt.
For example the Arduino IDE has a settings dialog that lets you change some settings but also, in the same dialog, gives the path to the actual settings file so you can change other settings with a text editor.
Martin
On 26/09/15 20:25, Miller Puckette wrote:
On Sat, Sep 26, 2015 at 06:44:54PM +0800, Chris McCormick wrote:
To my mind ~/pd/extra is not that different to ~/pd-externals in that it still forces the user to have a non-hidden folder in their home directory. Whenever applications do this I find it mildly annoying.
I thing for "settings" a hidden file is appropriate (e.g., ".pdsettings") but for libraries you want them visible - but precisely where would depend how you organize your files so should be settable.
Good point, and Linux doesn't have anything like OSX's ~/Library folder.
My reason for suggesting putting them in "pd/extra" is that you already put "pd" somewhere (and presumably chose where to put it) and if you relocate pd later the extra files will follow. Also, you can then have different versions of Pd with different libraries loaded.
Oh I see - you mean as a default just to use the "extra" folder relative to wherever the Pd binary is.
From the perspective of the deken plugin, as long as it can ask Pd where the files are supposed to be stored and the location is write-able by the current user then it can work. I agree with IOhanne's points though, and I'd prefer ~/.pd/extra over ~/pd-externals and over nothing at all.
I just want that the user is required to do/care as little as possible about the details that aren't directly related to their making art, but they are given a simple and powerful way to change things if they want to. Also world peace.
Cheers,
Chris.
On Mon, 2015-09-28 at 09:09 +0800, Chris McCormick wrote:
On 26/09/15 20:25, Miller Puckette wrote:
On Sat, Sep 26, 2015 at 06:44:54PM +0800, Chris McCormick wrote:
To my mind ~/pd/extra is not that different to ~/pd-externals in that it still forces the user to have a non-hidden folder in their home directory. Whenever applications do this I find it mildly annoying.
I thing for "settings" a hidden file is appropriate (e.g., ".pdsettings") but for libraries you want them visible - but precisely where would depend how you organize your files so should be settable.
Good point, and Linux doesn't have anything like OSX's ~/Library folder.
My reason for suggesting putting them in "pd/extra" is that you already put "pd" somewhere (and presumably chose where to put it) and if you relocate pd later the extra files will follow. Also, you can then have different versions of Pd with different libraries loaded.
Oh I see - you mean as a default just to use the "extra" folder relative to wherever the Pd binary is.
From the perspective of the deken plugin, as long as it can ask Pd where the files are supposed to be stored and the location is write-able by the current user then it can work. I agree with IOhanne's points though, and I'd prefer ~/.pd/extra over ~/pd-externals and over nothing at all.
I second that. Reasons why I prefer the 'pd' dir (or whatever name it finally will be) to be hidden:
* It is easy to make a visible symlink to a hidden folder. There is no similarly easy way I can think of to hide a visible folder.
* It's pretty standard on Linux and I'd like Pd to adhere to some standards. There is the freedesktop.org way of a file hierarchy with ~/.config and ~/.local, while putting stuff into ~/.programname is common, too. I cannot think of many programs that force you to have a visible folder in your home directory.
* Pd already uses the OS specific standards on OS X and Windows, but not so much on Linux. '~/pd-externals' is just strange, but ~/pd/extra is not less strange. What justifies to put (non-admin) user installed libraries into an extra folder on Linux, when at the same time it's <some-prefix>/pd on Windows and OS X? Treating Linux specially adds complexity I can't really see the reasoning behind.
Roman
On Sat, 2015-09-26 at 05:25 -0700, Miller Puckette wrote:
My reason for suggesting putting them in "pd/extra" is that you already put "pd" somewhere (and presumably chose where to put it) and if you relocate pd later the extra files will follow. Also, you can then have different versions of Pd with different libraries loaded.
I see your point. Such an organization is practical for someone who works on Pd related projects, I means it's quite a developer centric view. I'd like to consider a user centric view, too: Tom wants to play with a patch created by Alice. Alice doesn't know what how the users of her patches install Pd (they might run it directly from the compilation folder or they install Pd system-wide through the official repositories of their distribution), she just specifies the list of the externals her patch uses. Tom isn't familiar with how Pd search paths work, but he manages to install all required dependencies to make Alice' patch work when given quick instructions how to do it from the Pd GUI.
Things are tremendously easier if sensible defaults that work out-of-the-box automatically are chosen. I think having an automatically created (hidden) pd directory somewhere in $HOME makes life a lot easier for quite a common use case.
Roman
On 09/26/2015 12:44 PM, Chris McCormick wrote:
On 25/09/15 23:04, Miller Puckette wrote:
I'd like to get a better understanding of all this n part because I'd like to incorporate deken into Pd vanilla assuming its developers are cool with that.
I don't know about IOhannes [who at this point has probably contributed more lines of code than me to deken] but I am definitely cool with that.
i think it's great.
i think there are still a few issues open that should be fixed before integrating the code¹
If this happened it would probably make sense to split the tcl plugin out to go into Pd vanilla and keep the deken packaging script as a separate project.
i don't think that the packaging script should be part of Pd.
gfmdsa IOhannes
¹ and then there is dekenception: it would be way easier to implement it if deken was included in Pd; but i doubt whether it would be accepted... ...but then i don't even know whether it makes sense
On 2015-09-25 17:04, Miller Puckette wrote:
I think the best "default" default would be to put things in ~/pd/extra, assuming there's no problem writing to that (and, I suppose, assuming it isn't shared among multiple users of the same machine).
i just checked the code and found that currently deken tries to create the folder if it doesn't exist yet¹.
this means that it will (try to) create a ~/pd-deken/ folder (if it's not already there) before it will try to use pd/extra (even if the latter was writable).
so if people are averse to the idea of having a "pd-externals" directory in their home, deken will give them one nevertheless :-(
we could fix that by first iterating over all existing search-paths and test for writability, and only if this didn't yield any result try to create a (missing) standard installation directory.
mgas IOhannes
¹ what it really does is: iterate over all default search paths reported by Pd (e.g. ~/pd-externals /usr/local/lib/pd-externals /usr/lib/pd/extra), checks whether this folder exists and if it doesn't tries to create it. it then tries to create a temporary file in the path, and if it succeeds this folder is chosen as the download folder. if it fails, it checks the next search path. if all fails, it fails.
On Wed, 2015-09-23 at 22:20 +0200, IOhannes m zmölnig wrote:
iirc, this has been discussed at length,
Oh, IOhannes... that is some news! I truly hope your patches make it into Pd. I fully support your proposals and I'm convinced they make class loading in Pd much more stringent than it is now.
Roman
-
Chris McCormick -
IOhannes m zmoelnig -
IOhannes m zmölnig
-
IOhannes zmölnig
-
Jonathan Wilkes -
Martin Peach -
Miller Puckette -
Roman Haefeli