Hi everyone,
I've been working on some algorithms to eventually replace the way Pd finds and loads and creates objects. Sorry for the length of this mail, but you can skip some sections if you don't care about implementation details and/or user interface details :)
Table of Contents * Specifications * Example Loaders * Clearer Error Messages * Test Inputs * Test Outputs
Specifications --------------
As agreed by everyone so far, the new load order will be roughly:
for path in paths do -- the core does this bit for loader in loaders do loader(path, libray, object)
instead of the old, problematic:
for loader in loaders do for path in paths do -- the loader does this bit loader(path, library, object)
I restructured the loader plugin mechanism to be (pseudocode):
typedef err libloadfun(symbol *dir, symbol *lib); typedef err objloadfun(symbol *dir, symbol *lib, symbol *obj); struct loader { symbol *name; float priority; libloadfun *libfun; objloadfun *objfun; }
Example Loaders ---------------
-- binary loader is super simple Loader.new("bin", 70, function(dir, lib) return loadbin(dir, lib) end, function(dir, lib, obj) return loadbin(dir, obj) end)
-- directory loader is needed if there is no main library file -- the core code checks the results to see if it succeeded Loader.new("dir", 10, function(dir, lib) if dir:sub(-1 - #lib, -1) == ("/" .. lib) then return function() end else return nil, "dir doesn't match lib" end end, nil -- directory loader can't load objects )
-- the abstraction loader is quite complicated -- if the file exists it creates a callback that -- registers the class, with a constructor that -- runs the abstraction itself -- a Context here is essentially a 'canvas' Loader.new("txt", 30, nil, -- abstractions cannot register new classes function(dir, lib, obj) local src = dir .. "/" .. obj .. ".txt.lua" local f, err = fileexists(src) if f then local ldir, lobj = dir, obj return function() register(lobj, src, function(o, arguments) Context.push(Context.new({ ldir })) runtxt(ldir, lobj) Context.pop() end, src) end else return nil, err end end)
The current implementation is entirely in Lua, in an unpublished git repository. I'll upload it somewhere if there is any interest.
Clearer Error Messages ----------------------
The main upshot so far is clearer error messages instead of random possibly-working-but-perhaps-not implementation defined behaviour when more than one library defines the same name:
OBJECT [obj4] obj4 ... couldn't create (ambiguous) (found in: 'lib4') (found in: 'lib3') (found in: 'lib1') (found in: 'lib5') (found in: 'lib2') disambiguate by [import lib2/obj4] replacing lib2 by the library you want to import from
OBJECT [obj2] obj2 ... couldn't create (ambiguous qualified import) (imported: 'lib2/obj2') (imported: 'lib1/obj2') disambiguate by removing extra qualified imports
The error messages for objects that are not found are better too:
OBJECT [obj8] obj8 ... couldn't create (no such object) (looked in: 'lib4') (looked in: 'vanilla') (looked in: 'lib3') (looked in: 'lib1') (looked in: 'lib5') (looked in: 'lib2')
OBJECT [lib1/obj8] lib1/obj8 ... couldn't create (no such object in library)
OBJECT [lib6/obj1] lib6/obj1 ... couldn't create (no such library)
Note for that last [libc6/obj1] example: if libc6 actually did exist in the path but was not imported, the error would currently be the same. I want to make it so that [somelibrary/someobject] would work even when [somelibrary] is not imported (but such "implicit imports" would be deprecated, and warnings emitted).
Test Inputs -----------
The test input directory structure (.bin.lua approximates .pd_linux etc, .txt.lua approximates .pd patches/abstractions): ----8<---- .: global prj1
./global: lib1 lib2 lib3 lib3.bin.lua lib4 lib5.bin.lua vanilla
./global/lib1: obj1.txt.lua obj3.txt.lua obj4.txt.lua obj5.txt.lua obj7.txt.lua obj2.txt.lua obj4.bin.lua obj5.bin.lua obj6.bin.lua
./global/lib2: lib2.bin.lua obj2.txt.lua obj4.bin.lua obj5.bin.lua obj6.bin.lua obj1.txt.lua obj3.txt.lua obj4.txt.lua obj5.txt.lua obj7.txt.lua
./global/lib3: obj1.txt.lua obj3.txt.lua obj4.txt.lua obj5.txt.lua obj7.txt.lua obj2.txt.lua obj4.bin.lua obj5.bin.lua obj6.bin.lua
./global/lib4: obj1.txt.lua obj3.txt.lua obj4.txt.lua obj5.txt.lua obj7.txt.lua obj2.txt.lua obj4.bin.lua obj5.bin.lua obj6.bin.lua
./global/vanilla: debug.bin.lua
./prj1: prj1.txt.lua ----8<----
The test input file (prj1.txt.lua) imports stuff and creates objects: ----8<---- import({ "lib1", "lib1/obj1", "lib1/obj2", "lib2", "lib2/obj2", "lib2/obj3", "lib3", "lib4", "lib5" }) for o = 1,8 do object("obj" .. o) end for l = 1,6 do for o = 1,8 do object("lib" .. l .. "/" .. "obj" .. o) end end ----8<----
Test Outputs ------------
The full raw output: http://claudiusmaximus.goto10.org/t/pd/pd-new-loader-test-4-verbose.txt
The output with the STAT lines filtered out: http://claudiusmaximus.goto10.org/t/pd/pd-new-loader-test-4-quiet.txt
The output with the STAT lines sorted and duplicates counted: http://claudiusmaximus.goto10.org/t/pd/pd-new-loader-test-4-statistics.txt (The counts are low because the results are cached, this will need to be changed for live-coding abstractions etc...)
Hopefully I'm heading in the right direction with this, if not then yell loudly!
Thanks,
Claude
There is a lot to grapple with here, I've started to try to make sense of the work you've done so far, and in the process I'm throwing things in the wiki:
http://puredata.info/dev/PdSearchPath http://puredata.info/dev/PdNamespaces
So far its making sense, here's one perhaps naive idea of how to structure things: There are multiple forms for storing objects on the filesystem, including abstractions, single- and multi-object library files, etc. Then there is the representation of what objects are assigned to which symbols. These don't need to be the same thing. Each patch has its own table of which symbols are attached to which objects, and this can be separate from which objects are actually loaded into memory.
One idea for handling this is to have separate tables of symbols for which object files are loaded, versus which symbols are tied to which objects for a given patch. It seems for this to be workable, there would need to be a "loaded" table, then each patch would generate its table from the loaded table. Then if an object isn't in the loaded table, the loaded table handler would go and try to load it.
.hc
On Jul 18, 2010, at 3:41 AM, Claude Heiland-Allen wrote:
Hi everyone,
I've been working on some algorithms to eventually replace the way Pd finds and loads and creates objects. Sorry for the length of this mail, but you can skip some sections if you don't care about implementation details and/or user interface details :)
Table of Contents
- Specifications
- Example Loaders
- Clearer Error Messages
- Test Inputs
- Test Outputs
Specifications
As agreed by everyone so far, the new load order will be roughly:
for path in paths do -- the core does this bit for loader in loaders do loader(path, libray, object)
instead of the old, problematic:
for loader in loaders do for path in paths do -- the loader does this bit loader(path, library, object)
I restructured the loader plugin mechanism to be (pseudocode):
typedef err libloadfun(symbol *dir, symbol *lib); typedef err objloadfun(symbol *dir, symbol *lib, symbol *obj); struct loader { symbol *name; float priority; libloadfun *libfun; objloadfun *objfun; }
Example Loaders
-- binary loader is super simple Loader.new("bin", 70, function(dir, lib) return loadbin(dir, lib) end, function(dir, lib, obj) return loadbin(dir, obj) end)
-- directory loader is needed if there is no main library file -- the core code checks the results to see if it succeeded Loader.new("dir", 10, function(dir, lib) if dir:sub(-1 - #lib, -1) == ("/" .. lib) then return function() end else return nil, "dir doesn't match lib" end end, nil -- directory loader can't load objects )
-- the abstraction loader is quite complicated -- if the file exists it creates a callback that -- registers the class, with a constructor that -- runs the abstraction itself -- a Context here is essentially a 'canvas' Loader.new("txt", 30, nil, -- abstractions cannot register new classes function(dir, lib, obj) local src = dir .. "/" .. obj .. ".txt.lua" local f, err = fileexists(src) if f then local ldir, lobj = dir, obj return function() register(lobj, src, function(o, arguments) Context.push(Context.new({ ldir })) runtxt(ldir, lobj) Context.pop() end, src) end else return nil, err end end)
The current implementation is entirely in Lua, in an unpublished git repository. I'll upload it somewhere if there is any interest.
Clearer Error Messages
The main upshot so far is clearer error messages instead of random possibly-working-but-perhaps-not implementation defined behaviour when more than one library defines the same name:
OBJECT [obj4] obj4 ... couldn't create (ambiguous) (found in: 'lib4') (found in: 'lib3') (found in: 'lib1') (found in: 'lib5') (found in: 'lib2') disambiguate by [import lib2/obj4] replacing lib2 by the library you want to import from
OBJECT [obj2] obj2 ... couldn't create (ambiguous qualified import) (imported: 'lib2/obj2') (imported: 'lib1/obj2') disambiguate by removing extra qualified imports
The error messages for objects that are not found are better too:
OBJECT [obj8] obj8 ... couldn't create (no such object) (looked in: 'lib4') (looked in: 'vanilla') (looked in: 'lib3') (looked in: 'lib1') (looked in: 'lib5') (looked in: 'lib2')
OBJECT [lib1/obj8] lib1/obj8 ... couldn't create (no such object in library)
OBJECT [lib6/obj1] lib6/obj1 ... couldn't create (no such library)
Note for that last [libc6/obj1] example: if libc6 actually did exist in the path but was not imported, the error would currently be the same. I want to make it so that [somelibrary/someobject] would work even when [somelibrary] is not imported (but such "implicit imports" would be deprecated, and warnings emitted).
Test Inputs
The test input directory structure (.bin.lua approximates .pd_linux etc, .txt.lua approximates .pd patches/abstractions): ----8<---- .: global prj1
./global: lib1 lib2 lib3 lib3.bin.lua lib4 lib5.bin.lua vanilla
./global/lib1: obj1.txt.lua obj3.txt.lua obj4.txt.lua obj5.txt.lua obj7.txt.lua obj2.txt.lua obj4.bin.lua obj5.bin.lua obj6.bin.lua
./global/lib2: lib2.bin.lua obj2.txt.lua obj4.bin.lua obj5.bin.lua obj6.bin.lua obj1.txt.lua obj3.txt.lua obj4.txt.lua obj5.txt.lua obj7.txt.lua
./global/lib3: obj1.txt.lua obj3.txt.lua obj4.txt.lua obj5.txt.lua obj7.txt.lua obj2.txt.lua obj4.bin.lua obj5.bin.lua obj6.bin.lua
./global/lib4: obj1.txt.lua obj3.txt.lua obj4.txt.lua obj5.txt.lua obj7.txt.lua obj2.txt.lua obj4.bin.lua obj5.bin.lua obj6.bin.lua
./global/vanilla: debug.bin.lua
./prj1: prj1.txt.lua ----8<----
The test input file (prj1.txt.lua) imports stuff and creates objects: ----8<---- import({ "lib1", "lib1/obj1", "lib1/obj2", "lib2", "lib2/obj2", "lib2/obj3", "lib3", "lib4", "lib5" }) for o = 1,8 do object("obj" .. o) end for l = 1,6 do for o = 1,8 do object("lib" .. l .. "/" .. "obj" .. o) end end ----8<----
Test Outputs
The full raw output: http://claudiusmaximus.goto10.org/t/pd/pd-new-loader-test-4- verbose.txt
The output with the STAT lines filtered out: http://claudiusmaximus.goto10.org/t/pd/pd-new-loader-test-4-quiet.txt
The output with the STAT lines sorted and duplicates counted: http://claudiusmaximus.goto10.org/t/pd/pd-new-loader-test-4-statistics.txt (The counts are low because the results are cached, this will need to be changed for live-coding abstractions etc...)
Hopefully I'm heading in the right direction with this, if not then yell loudly!
Thanks,
Claude
http://claudiusmaximus.goto10.org
Pd-dev mailing list Pd-dev@iem.at http://lists.puredata.info/listinfo/pd-dev
----------------------------------------------------------------------------
As we enjoy great advantages from inventions of others, we should be glad of an opportunity to serve others by any invention of ours; and this we should do freely and generously. - Benjamin Franklin
-
Claude Heiland-Allen -
Hans-Christoph Steiner