On Sep 26, 2011, at 2:42 PM, IOhannes m zmölnig wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 09/26/2011 06:52 PM, Hans-Christoph Steiner wrote:
IOhannes and I were discussing how to load shared code for a libdir on IRC. The sticking point is that Mac OS X seems to require dylibs to have a hard-coded path. IOhannes posted a test lib to try to work out how to do it using DYLD_LIBRARY_PATH:
http://iem.at/~zmoelnig/OSX/test.tgz
So here are my results from Mac OS X 10.5.8:
creating a [test] object:
/private/tmp/test/tmp/test.pd_darwin: dlopen(/private/tmp/test/tmp/test.pd_darwin, 10): no suitable image found. Did find: /private/tmp/test/tmp/test.pd_darwin: unknown required load command 0x80000022 test ... couldn't create
hans@dhcp-10 tmp $ ~/code/pd-extended.git/src/pd -noprefs -lib test
./test.pd_darwin: dlopen(./test.pd_darwin, 10): no suitable image found. Did find: ./test.pd_darwin: unknown required load command 0x80000022 test: can't load library
just to clarify a bit: the binaries posted under the link above were compiled under OSX-10.6, without any precautions for backward compatibility. they don't work on 10.5, nor 10.4
i also put the same set of binaries online, but this time compiled under 10.4, and they work and function as expected under 10.4 and 10.6 (and i expect them to work on 10.5 as well)
on w32, the search paths for dll's should be modifiable via the %PATH% environment, whereas on linux (and most i think BSD and hurd) the envvar in question is LD_LIBRARY_PATH
after the technical details, let's step back and have a look at what this is all about: the main motivation is to be able to bundle shared code and 3rd party libraries with externals.
the dynamic linkers of the various OSs, will automatically search for dylinked libraries your external depends on, in various places (e.g. /usr/lib).
this is great if you have an easy way to install the needed library into this place (e.g. a dependy-informed installer system like debian's "apt"). it's not so great, if you don't have such a system, and the user is expected to find and download the given dependencies and put them into your system's search path. even if an external ships with all dependencies, they still have to be installed in the "correct" places, which makes installation more complicated than just dragging a folder onto your desktop. why?
because your OS's dynamic linker doesn't look for 3rd party libraries in weird places like a folder on your desktop.
now the idea, hans and me came up with, was to make Pd tell the OS to look into those places. e.g. if library "foo.pd_darwin" gets installed into ~/Library/Pd/foo/, then it would be nice, if the dynamic linker would search for any depending libraries in the ~/Library/Pd/foo/ directory. likewise "bar.pd_linux" (installed in ~/my-externals/pub/) might have it's dependencies searched for in ~/my-externals/pub/.
if both "foo" and "bar" depend on "libdudelsack", and both provide a "libdudelsack.dll" in their homedirectories, then of course we want both libraries to choose "their" correct one.
so the idea is, to modify the automatic search path of the dynamic linker whenever a library is loaded.
example: the user installs the "frotzel" library, by downloading it, and unzipping it into /usr/local/pd-externals/
now, if the user requests a new object [frotzel], Pd will search for a matching binary and eventually find frotzel.l_ia64 in /usr/local/pd-externals/frotzel/ before dlopen()ing this binary, it will read the LD_LIBRARY_PATH variable, push it to some stack, and then add /usr/local/pd-externals/frotzel/ at the beginning of the LD_LIBRARY_PATH. it will then dlopen() the frotzel.l_ia64 binary, and the linker discovers, that it has to load the libknurbel.so library, which it will (magically) find in /usr/local/pd-externals/frotzel/ because all dependencies are fullfilled, [frotzel] will successfully be loaded and will work. after fortzel.l_ia64 has been successfully loaded, we no longer need to search for libraries in /usr/local/pd-externals/frotzel/, and therefore the original content of LD_LIBRARY_PATH is restored (popped from the stack).
I got this building and working on Mac OS X, since that's where the discussion started. It was really just a matter of getting the build flags right. I am using tkwidgets as my test bed, then I'll integrate the changes into the library template. You can see my work here in the 'newentry' branch:
https://github.com/pd-projects/tkwidgets
the exact name of "LD_LIBRARY_PATH" is system dependent, but it seems that LD_LIBRARY_PATH, DYLD_LIBRARY_PATH and PATH cover most systems.
fasdmrt IOhannes
My guess is that it'll be something like this:
#ifdef _WIN32 PATH #elif defined(__APPLE__) DYLD_LIBRARY_PATH #else LD_LIBRARY_PATH #endif
And the IRC transcript for the record:
i don't exactly know how to do LD_LIBRARY_PATH on w32 yet
_hc IRC 1:54:17 IOhannes: the annoyances don't end there... 1:54:30 if libtest10.dylib links to something esle....
IOhannes IRC 1:54:31 i _thought_ it's a matter of %PATH% or so, but i never checked
_hc IRC 1:54:52 there seems to be some vars for dyld to handle this:
IOhannes IRC 1:54:58 i'm pretty sure there is aa mechanism on w32 that allows us to do that
_hc IRC 1:54:59 DYLD_FALLBACK_LIBRARY_PATH This is a colon separated list of directories that contain libraries. It is used as the default location for libraries not found in their install path. By default, it is set to $(HOME)/lib:/usr/local/lib:/lib:/usr/lib.
DYLD_ROOT_PATH This is a colon separated list of directories. The dynamic linker will prepend each of this directory paths to every image access until a file is found.
IOhannes IRC 1:55:20 would you mind posting to pastie? 1:55:37 it's hard to read on irssie
_hc IRC 1:55:41 http://developer.apple.com/library/mac/#documentation/Darwin/Reference/ ManPages/man1/dyld.1.html
IOhannes IRC 1:55:46 irsee, that is 1:56:11 excalibas left the room (quit: Remote host closed the connection).
IOhannes IRC 1:56:50 irssi that is
_hc IRC 1:56:50 so you are proposing that each time Pd loads a lib, it sets the *_PATH, then unsets it?
IOhannes IRC 1:57:13 i fail to see the difference between DYLD_LIBRARY_PATH and DYLD_FALLBACK_PATH 1:57:36 and yes, this is what my proposal is about
_hc IRC 1:57:47 I don't quite understand it, but it seems to be related to the "install name" thing
IOhannes IRC 1:58:39 the big pro (imho), is, that it allows devs to work as they are used to work with dylibs, and not invent some ramba zama magic dlopen() hack for a standard problem 1:59:14 but don't my DYLD_LIBRARY_PATH scripts work for you?
_hc IRC 2:00:05 some dlopen hack like externals?
IOhannes IRC 2:00:54 no, an external is a plugin, and dlopen() is designed for plugins 2:01:52 in contrast, the automatic dynamic linker is designed for dynamic sharing of code 2:02:07 which is what we want when we talk about "code-sharing" 2:02:16 and not: plugins 2:02:45 bot are closely related on a technical level, but this doesn't make them interchangeable 2:03:27 s/bot/both/ 2:03:36 is on a bad connection 2:05:12 according to http://msdn.microsoft.com/en-us/library/7d83bc18%28v=VS.100%29.aspx , %PATH% is the envvar to use to trick the w32 linker 2:15:21 pob [~pob@gar31-4-82-240-199-62.fbx.proxad.net] entered the room.
_hc IRC 3:07:09 IOhannes: what about when the libdir's .dylib is linked to other libs? 3:07:22 say /sw/bin/libspeex.dylib 3:09:59 r33p left the room (quit: Quit: ronflette mode).
IOhannes IRC 3:14:56 what about it?
_hc IRC 3:15:17 I feel like that's where there will be problems
IOhannes IRC 3:15:26 why? 3:15:37 olsen [~sesselast@xdsl-188-155-179-205.adslplus.ch] entered the room.
IOhannes IRC 3:15:51 maybe this is unclear: i'm talking about _adding_ to LD_LIBRARY_PATH, not about replacing it 3:16:12 the dynamic linker will then additionally search in yet another directory
_hc IRC 3:16:14 did you run your tests on 10.4? 3:16:17 or just build them?
IOhannes IRC 3:16:25 both
_hc IRC 3:16:47 hmm, my decision on this was informed by some experience a long time ago that i can't remember 3:17:00 so I feel like something is not going to work with this
IOhannes IRC 3:18:08 maybe it was something trivial back then
_hc IRC 3:18:48 with luck... 3:20:18 IOhannes: this project was already setup with this idea in mind, so I'm trying it: github.com/pd-projects/tkwidgets 3:20:25 it has shared code in tkwidgets.c
IOhannes IRC 3:22:16 i cannot checkout with my uplink right now...
_hc IRC 3:22:41 wow, that's slow
IOhannes IRC 3:24:26 what? my connection or your download speed?
_hc IRC 3:25:06 your uplink
IOhannes IRC 3:28:27 i know; officially i'm without internet at home
_hc IRC 3:32:09 IOhannes: the test thing doesn't build for me on 10.5: http://pastebin.com/ 3:32:24 http://pastebin.com/reTPU8MR
IOhannes IRC 3:33:32 try "make distclean & autoreconf -fiv && ./configure && make" 3:33:42 (and get the ampersands rights)
_hc IRC 3:34:57 oy, there were go: make distclean; /sw/bin/autoreconf-2.63 -fiv && ./ configure && make 3:36:18 IOhannes: so you link it normally, without DYLD_LIBRARY_PATH? 3:36:26 then set DLYD_ later? 3:36:50 lorenzosu [~lorenzosu@ppp-79-0.24-151.libero.it] entered the room.
IOhannes IRC 3:36:58 DYLD_LIBRARY_PATH is an environment variable that allows you to tweak the dynamic linker at runtime
_hc IRC 3:37:19 I know
IOhannes IRC 3:37:38 "dynamic linker at runtime" means: the piece of software that resolves dynamic libraries when you run your executable 3:37:59 so DYLD_LIBRARY_PATH is never set in the compilation/linking stage
_hc IRC 3:38:11 does it affect the linkign is my question
IOhannes IRC 3:38:14 only when you run the actual program 3:38:21 which linking? 3:39:27 (there's the gcc linking the object files into a binary and adding information on where to find external references) 3:39:58 (and theres the dynamic linker that resolves the external references when you run the binary) 3:40:11 the first thing happens on the dev machine 3:40:19 the second thing happens on the deployment machine 3:40:31 DYLIB_LIBRARY_PATH has nothing to do with the first stop 3:40:44 it's _only_ about deployment
_hc IRC 3:41:08 can you make libtool show you its command line? 3:41:22 I can't get this workign in the Makefile yet
IOhannes IRC 3:41:28 usually it print's out the cmdline
_hc IRC 3:42:25 this seems incomplete to me: libtool: link: gcc -g -O2 -o .libs/test test.o libtest/.libs/libtest.dylib 3:42:42 oh wait, wrong one 3:43:02 that's the one: /bin/sh ./libtool --tag=CC --mode=link gcc -g -O2 - module -avoid-version -shared -shrext .pd_darwin libtest/libtest.la - o test.la -rpath /usr/local/lib test_la-test.lo
IOhannes IRC 3:43:40 the line following that, is the actual gcc call 3:43:54
gcc -Wl,-undefined -Wl,dynamic_lookup -o .libs/test.pd_darwin -
bundle .libs/test_la-test.o libtest/.libs/libtest.0.0.0.dylib
_hc IRC 3:43:55 libtool: link: gcc -dynamiclib -Wl,-undefined -Wl,dynamic_lookup - o .libs/libtest.0.dylib .libs/libtest.o -O2 -install_name /usr/ local/lib/libtest.0.dylib -compatibility_version 1 -current_version 1.0 -Wl,-single_module 3:43:56 yup 3:44:15 it has to have an install_name...
IOhannes IRC 3:44:20 so it sets the install_name
_hc IRC 3:44:22 and version 3:44:23 yup
IOhannes IRC 3:44:39 but you can override that with DYLIB_LIBRARY_PATH
_hc IRC 3:44:53 and a .dylib won't work without a install_name 3:44:55 that's the pisser
IOhannes IRC 3:44:59 the libtool libraries are supposed to be as standard as can be
_hc IRC 3:45:00 even tho you can override it
IOhannes IRC 3:45:10 so what's the problem then?
_hc IRC 3:45:19 that's probably what got me in the past
IOhannes IRC 3:45:55 because our "overriding" is really only telling the linker to search for all libraries in an additional place 3:46:28 it's not forcing anything (apart from a certain search _order_) 3:46:40 i have no clue what apple want the install_name to do
_hc IRC 3:49:37 got it! 3:49:43 well, it built 3:50:11 doh! 3:50:23 the install_name is what is used in the linking 3:51:09 http://pastebin.com/Anf0BnUf
IOhannes IRC 3:51:10 in which linking?
_hc IRC 3:51:24 where would you do the DYLD_LIBRARY_PATH linking?
IOhannes IRC 3:52:07 in the pd-loader 3:52:44 oiata [~oiata@cable-86-56-113-6.cust.telecolumbus.net] entered the room.
_hc IRC 3:52:52 right, where in that process?
IOhannes IRC 3:54:55 s_loader.c:172 (push (DY)LD_LIBRARY_PATH) 3:55:13 s_loader.c:173 pop (DY)LD_LIBRARY_PATH 3:55:25 s_loader.c:183: push PATH 3:55:33 s_loader.c:184: pop PATH 3:55:54 oops, s_loader.c:185: pop PATH 3:57:35 and actually it should be done for the scheduler-loader as well (s_loader.c:246) 3:57:42 and for pd~ 3:57:58 i wish miller had created a dlopen() wrapper
_hc IRC 3:59:23 just read your email on this, I didn't think about including random other libs, that's true it should work also 3:59:30 that would be very helpful on Windows and Mac OS X
_hc IRC 4:06:23 IOhannes: strange, it seems to work without the DYLD_LIBRARY_PATH set
IOhannes IRC 4:07:10 what are you doing? 4:07:40 maybe you have the library sitting around somewhere
_hc IRC 4:07:56 to link the dylib: 4:07:57 -dynamiclib -undefined dynamic_lookup -install_name $(SHARED_LIB) - compatibility_version 1 -current_version 1.0 4:08:05 to link the objects: 4:08:15 ./$(SHARED_LI 4:08:27 cc -arch ppc -arch i386 -arch x86_64 -mmacosx-version-min=10.4 - bundle -bundle_loader /Applications/Pd-extended.app/Contents/Resources/ bin/pd -undefined dynamic_lookup -o "text.pd_darwin" "text.o" -lc -L/ sw/lib ./libtkwidgets.dylib 4:11:19 here's the actual dylib link: 4:11:19 cc -arch ppc -arch i386 -arch x86_64 -mmacosx-version-min=10.4 - dynamiclib -undefined dynamic_lookup -install_name libtkwidgets.dylib - compatibility_version 1 -current_version 1.0 -o libtkwidgets.dylib tkwidgets.o -lc -L/sw/lib 4:12:29 IOhannes: I figured it out, I was starting pd in the tkwidgets/ folder, that's how it found it
IOhannes IRC 4:15:39 the only problem i have right now, is that it is crashing on linux
_hc IRC 4:16:34 IOhannes: does the lib need a versioned name in Linux? ie. can we just have libtkwidgets.so or must it be libtkwidgets.so.1.
IOhannes IRC 4:18:52 i _think_ the latter (though i'm not 100% sure) 4:19:27 ah, i think if the library has no so_version, you might get away with .sl 4:19:31 .so, i meant