Hello,
I have just submitted patch #, which implements a new loader for Pd. This allows one to specify in a patch a "base URL" in which missing abstractions are expected to be found. The loader downloads an index at each listed URL, and if it knows about an abstraction that hasn't yet been loaded, it will download the patch file to a cache and open it from there. This is an idea I've been toying with for a while, and I've given it some consideration on several levels. I'm not even convinced it's a great idea, but I think it's something some people might be interested in. Lately I haven't had time to work on it, but I've got it working well enough that perhaps I should publish it before it starts to just stagnate in my home folder.
I thought I should just send a follow-up email explaining how to use it and expanding on some of the points listed in the patch comments:
First, to test the patch you'll need to do a few things.
1) Get it compiled and loaded.
The code is in externals/loaders/urlloader. Put it in your startup path and make sure it gets loaded. (pd should post some messages when it is loaded.) You'll need the libcurl-dev package installed.
2) Make the files accessible through http.
There is a directory under /urlloader called /remote. These files must go to a web server. For testing, I have a local Apache running on my laptop, and the files are hosted at http://localhost/pd. The /pd folder is just a symlink from /var/www/pd to the /remote folder. Note that the test patch (test.pd) points to this location, so if you decide to use a different location, you'll have to modify it.
3) Sign the patches in the remote folder.
The urlloader requires GPG to work (more on this below). So you'll need to have gpg working to test this code. If you have a gpg key, use it, otherwise make one. (Not hard.) You want files ending in .asc, so issue these two commands:
gpg -b --armor -s unknown.pd gpg -b --armor -s another.pd
4) Open test.pd
It should open the test patch, and then the [another] and [unknown] objects should be instantiated. Open them and you'll see that the actual location of the files is in a cache directory in your /home folder.
Some points:
- Requires GPG to verify downloaded patches before loading them.
Since this patch essentially allows code downloaded from a website to run locally on your machine, it can obviously be a screaming security risk. Therefore I did not want to publish this code before implementing this feature. However, it's got a few ugly user interface implications. Of course not everyone knows how to use GPG (I learned how just for this actually). But also, what if you don't have the key to verify a signature? Currently it just 'posts' this to the pd window, and you have to go download it. The person publishing the abstractions must provide their public key for this purpose. You then have to manually import it into your keyring with 'gpg --import'.
- Also verifies the files MD5 sums
I think this is probably redundant with the GPG stuff mentioned above. Originally I was just going to use the MD5 sum of a file to verify its integrity. However, I realized that trust issues go much deeper than that, since a loaded abstraction can easily specify another URL and then download patches from the new location. You wouldn't want your computer automatically following and downloading anything it sees, so I saw it as a necessity to support some kind of trust infrastructure. But I left the MD5 code in the patch for now anyways. The pd.index file that sits in /remote is just the output of the 'md5sum' command.
On the TODO list:
- Port to non-Linux operating systems.
Currently I developed this on Ubuntu. It should be portable to other systems, provided libcurl and gpg work there.
- Determine proper caching rules.
Currently patches are cached for 100 seconds, then downloaded again after the expire. This is just for testing purposes. In real-world a more intelligent caching system would be necessary. (That might be a good use of the MD5 sum in the index file, now that I think about it.)
- Aliases for URLs which could be prepended to object names. (e.g., urlA:myobject)
If more than one URL has the same object listed, the first one will be downloaded, which may as well be considered undeterministic to the user. It would be nice to support a way for the user to specify what URL an object belongs to. Using some kind of alias prepended to the object name might be a way of doing it, with the alias specified in the [baseurl] object. In fact, these should be related only to the canvas the [baseurl] objects reside in, whereas right now everything is pretty much global. (It would probably require modification of the g_list data structure, if I understand right.)
- Deal sanely with versioning. (support svn URLs?)
This brings up a problem with versions. What if a patch is designed that depends on an abstraction at some server, but then the owner of that server makes an incompatible change which breaks the person's patch. It would be nice to support some kind of versioning, so you could say not only, 'this patch depends on xyz', but 'this patch depends on version 2.5 of xyz'. I was thinking that one way to support this might be to include support for subversion URLs, or at least websvn, so that specific versions of files would be downloaded, but still easily updated to a newer version.
- Determine whether this is even a good idea.
I'm not completely convinced that downloading patches and running them directly is the best way to deal with abstractions hosted in the 'cloud'. It might be better to have a coherent and trusted repository system like APT.
Please try it and give some feedback. If you'd like me to commit this work to the svn, please feel free to grant me access, though I'll likely continue developing it on my own either way.
Steve
On Wed, Jul 9, 2008 at 8:40 PM, Stephen Sinclair radarsat1@gmail.com wrote:
Hello,
I have just submitted patch #, which implements a new loader for Pd.
Sorry, meant to say patch #2014632.
Steve
Seems like an interesting idea. Personally, I think an apt-get style of package management would work better for Pd. Sounds like you have all the right infrastructure, but instead of auto-downloading patches, it could be used for package management.
FYI: if you want to be added as a developer, you need to introduce yourself to pd-dev and request to be added. You can search the archives for examples.
.hc
On Jul 9, 2008, at 8:40 PM, Stephen Sinclair wrote:
Hello,
I have just submitted patch #, which implements a new loader for Pd. This allows one to specify in a patch a "base URL" in which missing abstractions are expected to be found. The loader downloads an index at each listed URL, and if it knows about an abstraction that hasn't yet been loaded, it will download the patch file to a cache and open it from there. This is an idea I've been toying with for a while, and I've given it some consideration on several levels. I'm not even convinced it's a great idea, but I think it's something some people might be interested in. Lately I haven't had time to work on it, but I've got it working well enough that perhaps I should publish it before it starts to just stagnate in my home folder.
I thought I should just send a follow-up email explaining how to use it and expanding on some of the points listed in the patch comments:
First, to test the patch you'll need to do a few things.
- Get it compiled and loaded.
The code is in externals/loaders/urlloader. Put it in your startup path and make sure it gets loaded. (pd should post some messages when it is loaded.) You'll need the libcurl-dev package installed.
- Make the files accessible through http.
There is a directory under /urlloader called /remote. These files must go to a web server. For testing, I have a local Apache running on my laptop, and the files are hosted at http://localhost/pd. The /pd folder is just a symlink from /var/www/pd to the /remote folder. Note that the test patch (test.pd) points to this location, so if you decide to use a different location, you'll have to modify it.
- Sign the patches in the remote folder.
The urlloader requires GPG to work (more on this below). So you'll need to have gpg working to test this code. If you have a gpg key, use it, otherwise make one. (Not hard.) You want files ending in .asc, so issue these two commands:
gpg -b --armor -s unknown.pd gpg -b --armor -s another.pd
- Open test.pd
It should open the test patch, and then the [another] and [unknown] objects should be instantiated. Open them and you'll see that the actual location of the files is in a cache directory in your /home folder.
Some points:
- Requires GPG to verify downloaded patches before loading them.
Since this patch essentially allows code downloaded from a website to run locally on your machine, it can obviously be a screaming security risk. Therefore I did not want to publish this code before implementing this feature. However, it's got a few ugly user interface implications. Of course not everyone knows how to use GPG (I learned how just for this actually). But also, what if you don't have the key to verify a signature? Currently it just 'posts' this to the pd window, and you have to go download it. The person publishing the abstractions must provide their public key for this purpose. You then have to manually import it into your keyring with 'gpg --import'.
- Also verifies the files MD5 sums
I think this is probably redundant with the GPG stuff mentioned above. Originally I was just going to use the MD5 sum of a file to verify its integrity. However, I realized that trust issues go much deeper than that, since a loaded abstraction can easily specify another URL and then download patches from the new location. You wouldn't want your computer automatically following and downloading anything it sees, so I saw it as a necessity to support some kind of trust infrastructure. But I left the MD5 code in the patch for now anyways. The pd.index file that sits in /remote is just the output of the 'md5sum' command.
On the TODO list:
- Port to non-Linux operating systems.
Currently I developed this on Ubuntu. It should be portable to other systems, provided libcurl and gpg work there.
- Determine proper caching rules.
Currently patches are cached for 100 seconds, then downloaded again after the expire. This is just for testing purposes. In real-world a more intelligent caching system would be necessary. (That might be a good use of the MD5 sum in the index file, now that I think about it.)
- Aliases for URLs which could be prepended to object names. (e.g., urlA:myobject)
If more than one URL has the same object listed, the first one will be downloaded, which may as well be considered undeterministic to the user. It would be nice to support a way for the user to specify what URL an object belongs to. Using some kind of alias prepended to the object name might be a way of doing it, with the alias specified in the [baseurl] object. In fact, these should be related only to the canvas the [baseurl] objects reside in, whereas right now everything is pretty much global. (It would probably require modification of the g_list data structure, if I understand right.)
- Deal sanely with versioning. (support svn URLs?)
This brings up a problem with versions. What if a patch is designed that depends on an abstraction at some server, but then the owner of that server makes an incompatible change which breaks the person's patch. It would be nice to support some kind of versioning, so you could say not only, 'this patch depends on xyz', but 'this patch depends on version 2.5 of xyz'. I was thinking that one way to support this might be to include support for subversion URLs, or at least websvn, so that specific versions of files would be downloaded, but still easily updated to a newer version.
- Determine whether this is even a good idea.
I'm not completely convinced that downloading patches and running them directly is the best way to deal with abstractions hosted in the 'cloud'. It might be better to have a coherent and trusted repository system like APT.
Please try it and give some feedback. If you'd like me to commit this work to the svn, please feel free to grant me access, though I'll likely continue developing it on my own either way.
Steve
Pd-dev mailing list Pd-dev@iem.at http://lists.puredata.info/listinfo/pd-dev
------------------------------------------------------------------------ ----
Terrorism is not an enemy. It cannot be defeated. It's a tactic. It's about as sensible to say we declare war on night attacks and expect we're going to win that war. We're not going to win the war on terrorism. - retired U.S. Army general, William Odom
On Thu, Jul 10, 2008 at 1:15 PM, Hans-Christoph Steiner hans@eds.org wrote:
Seems like an interesting idea. Personally, I think an apt-get style of package management would work better for Pd. Sounds like you have all the right infrastructure, but instead of auto-downloading patches, it could be used for package management.
True. I don't know though, I don't have a good concept of how that might work for Pd. It's been discussed before but no one's really come up with a concrete suggestion for it as far as I know, so this is the best I could come up with for now..
I thought it might be good to get the ball rolling with some code. I'd be extremely happy if someone wanted to fork it into a package management system. At least it demonstrates how to download files, check them for trust, and tell Pd to load them, so maybe someone will find a better way to use that. As it stands, the code is good enough that you could publish a patch on your website that would download the rest of itself on the fly, so it's a start at least.
Steve
-
Hans-Christoph Steiner -
Stephen Sinclair