hi hans,
i just saw, that the cyclone folder under windows contains about 114 mb of externals ... is this intended?
cheers .. tim
On Sat, 21 Jan 2006, Tim Blechmann wrote:
i just saw, that the cyclone folder under windows contains about 114 mb of externals ... is this intended?
I am curious: what's the size of the smallest possible external on win32 ?
E.g. arraysize.pd_linux is 1.4k, or 3.2k in debug mode, but I notice that Framestein came with a 36k arraysize.dll (compiled june 2002). Is that the normal minimal size of externals on win32 ?
and then how much RAM do they take once loaded? e.g. on Linux the RAM it's reported (by /proc/$$/maps) to take is 8k (plus one filehandle), that is the file rounded up to multiple of 4k, plus a table of pointers for connecting the .so to the rest of the process, which is 4k for small externals. Is there a way to figure out that value on Windows?
_ _ __ ___ _____ ________ _____________ _____________________ ... | Mathieu Bouchard - tél:+1.514.383.3801 - http://artengine.ca/matju | Freelance Digital Arts Engineer, Montréal QC Canada
On Sat, 21 Jan 2006 18:13:36 -0500 (EST) Mathieu Bouchard matju@artengine.ca wrote:
On Sat, 21 Jan 2006, Tim Blechmann wrote:
i just saw, that the cyclone folder under windows contains about 114 mb of externals ... is this intended?
I am curious: what's the size of the smallest possible external on win32 ?
E.g. arraysize.pd_linux is 1.4k, or 3.2k in debug mode, but I notice that Framestein came with a 36k arraysize.dll (compiled june 2002). Is that the normal minimal size of externals on win32 ?
well, zexy's abs~ is about 7k ... so it's about the same as on linux ... in general it should be about the same code as on linux (same architecture) maybe some extra code to cover multiple architectures ...
and then how much RAM do they take once loaded? e.g. on Linux the RAM it's reported (by /proc/$$/maps) to take is 8k (plus one filehandle), that is the file rounded up to multiple of 4k, plus a table of pointers for connecting the .so to the rest of the process, which is 4k for small externals. Is there a way to figure out that value on Windows?
not sure, but i think, it should be similar ... shared objects and dynamic linked libraries cover the same concept ...
tim
Mathieu Bouchard wrote:
I am curious: what's the size of the smallest possible external on win32 ?
E.g. arraysize.pd_linux is 1.4k, or 3.2k in debug mode, but I notice that Framestein came with a 36k arraysize.dll (compiled june 2002). Is that the normal minimal size of externals on win32 ?
and then how much RAM do they take once loaded? e.g. on Linux the RAM it's reported (by /proc/$$/maps) to take is 8k (plus one filehandle), that is the file rounded up to multiple of 4k, plus a table of pointers for connecting the .so to the rest of the process, which is 4k for small externals. Is there a way to figure out that value on Windows?
In recent versions you can always do ctrl-alt-delete and select the Processes tab in the resulting dialog. Then run pd and watch its memory usage change as you add externals to a patcher. One thing I noticed while doing this is that pd (or is it the OS?) doesn't free memory when objects are deleted from a patch.
Martin
On Mon, 23 Jan 2006, Martin Peach wrote:
Mathieu Bouchard wrote:
and then how much RAM do they take once loaded? e.g. on Linux the RAM it's reported (by /proc/$$/maps) to take is 8k (plus one filehandle), that is
One thing I noticed while doing this is that pd (or is it the OS?) doesn't free memory when objects are deleted from a patch.
I was more interested in space taken by a class than by an object... About freeing memory, that's because memory allocators (in libc) are two-level for small areas. The OS only deals with 4k blocks, so a large pool of memory is allocated for all small allocations together. When small allocations get freed, none of the memory gets freed at the OS level, but that memory is still free for future small allocations.
AFAIK, that way of working is the same on all platforms; what may be different from libc to libc is what's the threshold above which allocations requests get forwarded directly to the OS, and what's the memory layout used for small allocations (and for remembering what's in use and what are the current sizes of all allocations).
_ _ __ ___ _____ ________ _____________ _____________________ ... | Mathieu Bouchard - tél:+1.514.383.3801 - http://artengine.ca/matju | Freelance Digital Arts Engineer, Montréal QC Canada
Mathieu Bouchard wrote:
On Mon, 23 Jan 2006, Martin Peach wrote:
Mathieu Bouchard wrote:
and then how much RAM do they take once loaded? e.g. on Linux the RAM it's reported (by /proc/$$/maps) to take is 8k (plus one filehandle), that is
One thing I noticed while doing this is that pd (or is it the OS?) doesn't free memory when objects are deleted from a patch.
I was more interested in space taken by a class than by an object...
Oooh my brain hurts! What namespace are we in? I assume a class is the code that is loaded to implement an external (the .dll or .pd_linux) and the object is just the struct particular to one instance of the class (or is it the other way round?). If this is the case they should both show up as extra memory allocated to the pd process, the struct being a relatively trivial amount of that memory. (of course I'm probably wrong here again ;()
Martin
On Mon, 23 Jan 2006, Martin Peach wrote:
Mathieu Bouchard wrote:
On Mon, 23 Jan 2006, Martin Peach wrote:
Mathieu Bouchard wrote:
and then how much RAM do they take once loaded? e.g. on Linux the RAM it's reported (by /proc/$$/maps) to take is 8k (plus one filehandle), that is
One thing I noticed while doing this is that pd (or is it the OS?) doesn't free memory when objects are deleted from a patch.
I was more interested in space taken by a class than by an object...
Oooh my brain hurts! What namespace are we in? I assume a class is the code that is loaded to implement an external (the .dll or .pd_linux) and the object is just the struct particular to one instance of the class (or is it the other way round?).
Yeah it is. Often in the Pd world, the words "class" and "object" get confused, but I'd rather have standard OOP terminology used in the Pd world. I wanted to know what's the amount of RAM used up by a .dll file once loaded in RAM. I ask because I know that in Linux, the amount of RAM taken by a .so file once loaded in RAM is more than the length of the file due both to paging issues and to "fixup" issues (basically the table used for dlsym() which is used for relocating pointers considering that RAM positions can't be hardcoded in a .so)
There's also the actual t_class allocation but that's very small compared to any .so ... Now that I think of it, a t_class also owns several smaller structures, and while I'm sort of curious about how much RAM it takes, chances are it doesn't vary much from OS to OS.
If this is the case they should both show up as extra memory allocated to the pd process, the struct being a relatively trivial amount of that memory. (of course I'm probably wrong here again ;()
a t_object (aka t_text is 7 words. Here by word I mean the size of an
ordinary data pointer. That's 28 bytes in 32-bit mode and 56 bytes in
64-bit mode. But that's really not all. Every t_inlet is 7 words as well.
Every t_outlet is 4 words. Each of those things is allocated separately.
Every allocation requires at least an extra word for use by malloc/free,
and sometimes several extra words.
The fact that a Pd object is broken down in all of those little structs doesn't do any good to the cache, as it must keep lots of different parts of RAM together in order for the CPU/Pd to do anything. That is likely to matter a lot more than the total usage of RAM these days, but then, I haven't benchmarked it lately.
_ _ __ ___ _____ ________ _____________ _____________________ ... | Mathieu Bouchard - tél:+1.514.383.3801 - http://artengine.ca/matju | Freelance Digital Arts Engineer, Montréal QC Canada
Mathieu Bouchard wrote:
I wanted to know what's the amount of RAM used up by a .dll file once loaded in RAM. I ask because I know that in Linux, the amount of RAM taken by a .so file once loaded in RAM is more than the length of the file due both to paging issues and to "fixup" issues (basically the table used for dlsym() which is used for relocating pointers considering that RAM positions can't be hardcoded in a .so)
From a test I just did on WinXP: ctrl-alt-delete, select processes, launch pd:
startup pd 0.39 test 7 with several libs: 8,940K create new patcher: 8,948K -> +8K put object: 8,960K -> +12K name it helloworld: 9,180K -> +220K (this helloworld.dll is 6K on disk) put object: 9,180K name it helloworld: 9,180K put bang: 9,184K -> + 4K connect it to a helloworld: 9,184K put bang: 9,184K connect it to a helloworld: 9,184K remove everything and then close the patcher: 9,184K
It looks like the smallest blocks allocated are 4K. I wonder why helloworld takes 220k. It was linked against pd.lib and kernel32.lib. My pd.lib is 85K on disk, kernel32.lib is 190K.
Second test: startup pd 0.39 test 7 with several libs: 8,916K (This time pd starts with less memory) create new patcher: 8,924K -> +8K put object: 8,936K -> +12K name it expr $1 + 1: 9,140K -> +204K (expr.dll is 152K on disk) put object: 9,140K name it expr $1 + 1: 9,140K remove everything and then close the patcher: 9,140K
Martin
============================================================================================ This is the code of helloworld: #define MSW #include "m_pd.h"
static t_class *helloworld_class;
typedef struct helloworld { t_object x_obj; } t_helloworld;
void helloworld_bang(t_helloworld *x); __declspec(dllexport) void helloworld_setup(void); void *helloworld_new(void);
void helloworld_bang(t_helloworld *x) { post("Hello World :0"); }
__declspec(dllexport) void helloworld_setup(void) { helloworld_class = class_new(gensym("helloworld"), (t_newmethod)helloworld_new, 0, sizeof(t_helloworld), CLASS_DEFAULT, 0); class_addbang(helloworld_class, helloworld_bang); }
void *helloworld_new(void) { t_helloworld *x = (t_helloworld *)pd_new(helloworld_class); return (void *)x; }
On Jan 23, 2006, at 3:21 PM, Martin Peach wrote:
Mathieu Bouchard wrote:
I am curious: what's the size of the smallest possible external on
win32 ?E.g. arraysize.pd_linux is 1.4k, or 3.2k in debug mode, but I notice
that Framestein came with a 36k arraysize.dll (compiled june 2002). Is
that the normal minimal size of externals on win32 ?and then how much RAM do they take once loaded? e.g. on Linux the RAM
it's reported (by /proc/$$/maps) to take is 8k (plus one filehandle), that
is the file rounded up to multiple of 4k, plus a table of pointers for connecting the .so to the rest of the process, which is 4k for small externals. Is there a way to figure out that value on Windows?In recent versions you can always do ctrl-alt-delete and select the
Processes tab in the resulting dialog. Then run pd and watch its memory usage change as you add externals to
a patcher. One thing I noticed while doing this is that pd (or is it
the OS?) doesn't free memory when objects are deleted from a patch.
You also can't delete or change a .dll once Pd has opened it. Pd keeps
the object in memory and the file locked until it quits.
.hc ________________________________________________________________________ ____
"I have the audacity to believe that peoples everywhere can have three
meals a day for their bodies, education and culture for their minds,
and dignity, equality and freedom for their spirits."
- Martin Luther King, Jr.
Hans-Christoph Steiner wrote:
a patcher. One thing I noticed while doing this is that pd (or is it
the OS?) doesn't free memory when objects are deleted from a patch.You also can't delete or change a .dll once Pd has opened it. Pd keeps
the object in memory and the file locked until it quits.
while this is not necessarily related to the memory usage of an external, it should rather read: "Pd keeps the object in memory and w32 locks the file until pd quits." and it should even rather read ("...and w32 attempts to lock...")
just yesterday i happily discovered that i could circumvent this locking, while having pd loaded an external and at the same time compiling this external and replacing the .dll-file (whether this makes english sense is another question which i cannot answer)
mf.ads.r IOhannes
On Jan 24, 2006, at 3:57 AM, IOhannes m zmoelnig wrote:
Hans-Christoph Steiner wrote:
a patcher. One thing I noticed while doing this is that pd (or is it
the OS?) doesn't free memory when objects are deleted from a patch.You also can't delete or change a .dll once Pd has opened it. Pd
keeps the object in memory and the file locked until it quits.while this is not necessarily related to the memory usage of an
external, it should rather read: "Pd keeps the object in memory and w32 locks the file until pd quits." and it should even rather read ("...and w32 attempts to lock...")
Yes, that is correct.
just yesterday i happily discovered that i could circumvent this
locking, while having pd loaded an external and at the same time
compiling this external and replacing the .dll-file (whether this
makes english sense is another question which i cannot answer)
Does Pd then reload the newly compiled DLL? That would be quite handy
when testing.
.hc ________________________________________________________________________ ____
"Computer science is no more related to the computer than astronomy is
related to the telescope."
-Edsger
Dykstra
On Jan 21, 2006, at 4:26 PM, Tim Blechmann wrote:
hi hans,
i just saw, that the cyclone folder under windows contains about 114 mb of externals ... is this intended?
For cyclone, I just used Kzrysztof's build system, since its quite
complex. On MinGW, its probably not getting strip'ed, that's my guess
why they are large. gcc will link in the .dlls, making the files huge.
strip will remove the unnecessary stuff.
.hc ________________________________________________________________________ ____
If you are not part of the solution, you are part of the problem.
-
Eldridge Cleaver
i just saw, that the cyclone folder under windows contains about 114 mb of externals ... is this intended?
For cyclone, I just used Kzrysztof's build system, since its quite
complex. On MinGW, its probably not getting strip'ed, that's my guess why they are large. gcc will link in the .dlls, making the files huge. strip will remove the unnecessary stuff.
i'm not familiar with kzrysztof's build system for windoze, but all the externals are about 765kb ... my linux libs hammer and sickle are about 800kb (compiled with icc), so my wild guess is, that the dlls contain the code for _all_ objects, but just export the specific loader function for the single external ...
maybe you have to tell strip, what's unnecessary ;-)
tim
On Jan 21, 2006, at 7:11 PM, Tim Blechmann wrote:
i just saw, that the cyclone folder under windows contains about 114 mb of externals ... is this intended?
For cyclone, I just used Kzrysztof's build system, since its quite complex. On MinGW, its probably not getting strip'ed, that's my guess why they are large. gcc will link in the .dlls, making the files huge. strip will remove the unnecessary stuff.
i'm not familiar with kzrysztof's build system for windoze, but all the externals are about 765kb ... my linux libs hammer and sickle are about 800kb (compiled with icc), so my wild guess is, that the dlls contain the code for _all_ objects, but just export the specific loader function for the single external ...
maybe you have to tell strip, what's unnecessary ;-)
It sounds like cyclone is being statically linked with pd.dll. It was
happening with Pd-extended until I figured out the strip thing, each
object was about that size. Ah yes, I remember now, MinGW statically
links in the DLL because of the thing about DLLs not being able to link
to each other. I think its a trick to get around some more arbitrary
Microsoft restrictions.
This is what is currently be used on Pd-extended: STRIP = strip --strip-unneeded -R .note -R .comment
.hc
¡El pueblo unido jamás será vencido!
Hallo, Hans-Christoph Steiner hat gesagt: // Hans-Christoph Steiner wrote:
It sounds like cyclone is being statically linked with pd.dll.
Cyclone single-externals are huge on Linux as well, e.g.:
$ ll prepend.pd_linux hammer.pd_linux
-rwxrwxr-x 1 root root 351265 Aug 27 14:33 hammer.pd_linux
-rwxrwxr-x 1 root root 140672 Aug 27 14:33 prepend.pd_linux
Ciao
Frank Barknecht _ ______footils.org_ __goto10.org__
On Jan 22, 2006, at 5:44 AM, Frank Barknecht wrote:
Hallo, Hans-Christoph Steiner hat gesagt: // Hans-Christoph Steiner wrote:
It sounds like cyclone is being statically linked with pd.dll.
Cyclone single-externals are huge on Linux as well, e.g.:
$ ll prepend.pd_linux hammer.pd_linux -rwxrwxr-x 1 root root 351265 Aug 27 14:33 hammer.pd_linux -rwxrwxr-x 1 root root 140672 Aug 27 14:33 prepend.pd_linux
Maybe they need to be stripped too. This is what Pd-extended uses for
GNU/Linux:
STRIP = strip --strip-unneeded -R .note -R .comment
Maybe we should add a little strip hack to Pd-extended that will strip
everything once its installed.
.hc
"I have the audacity to believe that peoples everywhere can have three
meals a day for their bodies, education and culture for their minds,
and dignity, equality and freedom for their spirits."
- Martin Luther King, Jr.
Hallo, Hans-Christoph Steiner hat gesagt: // Hans-Christoph Steiner wrote:
On Jan 22, 2006, at 5:44 AM, Frank Barknecht wrote:
Hallo, Hans-Christoph Steiner hat gesagt: // Hans-Christoph Steiner wrote:
It sounds like cyclone is being statically linked with pd.dll.
Cyclone single-externals are huge on Linux as well, e.g.:
$ ll prepend.pd_linux hammer.pd_linux -rwxrwxr-x 1 root root 351265 Aug 27 14:33 hammer.pd_linux -rwxrwxr-x 1 root root 140672 Aug 27 14:33 prepend.pd_linux
Maybe they need to be stripped too. This is what Pd-extended uses for
GNU/Linux:STRIP = strip --strip-unneeded -R .note -R .commentMaybe we should add a little strip hack to Pd-extended that will strip
everything once its installed.
Well, while my prepend.pd_linux wasn't stripped, this doesn't help much:
(snd)-$ strip --strip-unneeded -R .note -R .comment /usr/lib/pd/extra/prepend.pd_linux (snd)-$ ll /usr/lib/pd/extra/prepend.pd_linux -rwxrwxr-x 1 root root 115272 Jan 22 17:29 /usr/lib/pd/extra/prepend.pd_linux
I think, Cyclone links some additional stuff into itself. All Cyclone-externals seem to be much bigger than necessary compared to other externals that provide similar functionality:
(snd)-$ ll /usr/lib/pd/extra/iter.pd_linux /usr/lib/pd/extra/drip.pd_linux -rwxrwxr-x 1 root root 6516 Aug 27 14:42 /usr/lib/pd/extra/drip.pd_linux -rwxrwxr-x 1 root root 114836 Jan 22 17:31 /usr/lib/pd/extra/iter.pd_linux
"iter" is Cyclone's version of Zexy's "drip".
Ciao
Frank Barknecht _ ______footils.org_ __goto10.org__
On Jan 22, 2006, at 5:44 AM, Frank Barknecht wrote:
Hallo, Hans-Christoph Steiner hat gesagt: // Hans-Christoph Steiner wrote:
It sounds like cyclone is being statically linked with pd.dll.
Cyclone single-externals are huge on Linux as well, e.g.:
$ ll prepend.pd_linux hammer.pd_linux -rwxrwxr-x 1 root root 351265 Aug 27 14:33 hammer.pd_linux -rwxrwxr-x 1 root root 140672 Aug 27 14:33 prepend.pd_linux
hammer.pd_linux is actually a multi-file lib.
Mac OS X also has bigger file sizes, but not as extreme, and they
benefit from a strip:
hans@sla:cyclone > ls -l prepend.pd_darwin
-rwxr-xr-x 1 hans admin 41516 Jan 17 12:02
prepend.pd_darwin*
hans@sla:cyclone > strip -x prepend.pd_darwin
hans@sla:cyclone > ls -l prepend.pd_darwin
-rwxr-xr-x 1 hans admin 23752 Jan 22 11:34
prepend.pd_darwin*
Running "strip -x" on everything in "extra" on the Mac OS X app reduced
the "extra" folder from 43MB to 31MB.
Running "strip --strip-unneeded -R .note -R .comment" on Windows
reduced "extra" from 147MB to 24MB, but it broke loading Gem.dll,
xsample.dll, and maybe other stuff. pmpd.dll still worked tho.
I also stripped "libPdTcl.dylib" aka "pdtcl.dll", and it cut it in
half. I stripped bin/pd and it went from 2222632 bytes to 745032
bytes. Everything seems to work fine still.
Any downsides to the post-install strip? Its a kludge, yes, but its
easy.
.hc
"Looking at things from a more basic level, you can come up with a more
direct solution... It may sound small in theory, but it in practice, it
can change entire economies."
- Amy Smith
-
Frank Barknecht -
Hans-Christoph Steiner -
IOhannes m zmoelnig -
Martin Peach -
Mathieu Bouchard -
Tim Blechmann