Hi,
today I noticed a readsf~ (on vanilla) opening a 5 wave file containing 32 bits float audio fails silently. The doc says 4 bytes is unavailable for AIFF, but I use WAVE.
The file is a 5 channels WAVE file with 5 tracks 32bits float at 48kHz.
Converting the audio to 16 bits PCM works.
Is that a bug ? On vanilla 0.44.
Thanks,
Sounds like it, especially if you can reproduce it everytime. File a bug report and include as simple a patch as possible to reproduce the issue, and the soundfile.
I've definitely used lots of mono and stereo 32-bit float WAV files, so those work.
.hc
On Mar 11, 2013, at 3:52 PM, Charles Goyard wrote:
Hi,
today I noticed a readsf~ (on vanilla) opening a 5 wave file containing 32 bits float audio fails silently. The doc says 4 bytes is unavailable for AIFF, but I use WAVE.
The file is a 5 channels WAVE file with 5 tracks 32bits float at 48kHz.
Converting the audio to 16 bits PCM works.
Is that a bug ? On vanilla 0.44.
Thanks,
-- Charles
Pd-list@iem.at mailing list UNSUBSCRIBE and account-management -> http://lists.puredata.info/listinfo/pd-list
Hi,
after some research, it seems more related to the size of the file after all. I thought the problem was 32 bits files because it worked when I converted them to 16 bits. But it was just the file size dropped under 2gb.
To sum up: a file over about 2Gb fails to open. This is related to how Linux handles large files on 32 bits systems.
On my test patch I get the "done" bang as soon as I send the "start" message.
Reading the code, it looks like the s_path.c/sys_open() oflags is missing O_LARGEFILE support. pd-extended and pd-l2ork suffer from the same.
Am I right there's no Large File Support for pd on 32 bits Linux ? If so that's a major shortcoming, and something a bit heavy to fix. Does it work out of the box on 64 bits linux ?
Thanks,
Darn, I just found there's a feature request for just that from IOhannes pending since 2007.
https://sourceforge.net/tracker/index.php?func=detail&aid=1638701&gr...
Cheers,
Charles
Charles Goyard wrote:
Hi,
after some research, it seems more related to the size of the file after all. I thought the problem was 32 bits files because it worked when I converted them to 16 bits. But it was just the file size dropped under 2gb.
To sum up: a file over about 2Gb fails to open. This is related to how Linux handles large files on 32 bits systems.
On my test patch I get the "done" bang as soon as I send the "start" message.
Reading the code, it looks like the s_path.c/sys_open() oflags is missing O_LARGEFILE support. pd-extended and pd-l2ork suffer from the same.
Am I right there's no Large File Support for pd on 32 bits Linux ? If so that's a major shortcoming, and something a bit heavy to fix. Does it work out of the box on 64 bits linux ?
Thanks,
-- Charles
Hans-Christoph Steiner wrote:
Sounds like it, especially if you can reproduce it everytime. File a bug report and include as simple a patch as possible to reproduce the issue, and the soundfile.
today I noticed a readsf~ (on vanilla) opening a 5 wave file containing 32 bits float audio fails silently. The doc says 4 bytes is unavailable for AIFF, but I use WAVE.
The file is a 5 channels WAVE file with 5 tracks 32bits float at 48kHz.
Converting the audio to 16 bits PCM works.
Is that a bug ? On vanilla 0.44.
It's in the source - but Pd has to be compiled with '_LARGEFILE64_SOURCE' defined for this to work. Through 0.43 the configure script I was using was checking for this somehow (too complicted for me to understand how).
Anyhow, this is a bug all right - thanks for reporting it!
... and can you tell me what OS you're on so I can check if I'm actually fixing it?
thanks Miller
On Sat, Mar 16, 2013 at 10:57:46PM +0100, Charles Goyard wrote:
Darn, I just found there's a feature request for just that from IOhannes pending since 2007.
https://sourceforge.net/tracker/index.php?func=detail&aid=1638701&gr...
Cheers,
Charles
Charles Goyard wrote:
Hi,
after some research, it seems more related to the size of the file after all. I thought the problem was 32 bits files because it worked when I converted them to 16 bits. But it was just the file size dropped under 2gb.
To sum up: a file over about 2Gb fails to open. This is related to how Linux handles large files on 32 bits systems.
On my test patch I get the "done" bang as soon as I send the "start" message.
Reading the code, it looks like the s_path.c/sys_open() oflags is missing O_LARGEFILE support. pd-extended and pd-l2ork suffer from the same.
Am I right there's no Large File Support for pd on 32 bits Linux ? If so that's a major shortcoming, and something a bit heavy to fix. Does it work out of the box on 64 bits linux ?
Thanks,
-- Charles
Hans-Christoph Steiner wrote:
Sounds like it, especially if you can reproduce it everytime. File a bug report and include as simple a patch as possible to reproduce the issue, and the soundfile.
today I noticed a readsf~ (on vanilla) opening a 5 wave file containing 32 bits float audio fails silently. The doc says 4 bytes is unavailable for AIFF, but I use WAVE.
The file is a 5 channels WAVE file with 5 tracks 32bits float at 48kHz.
Converting the audio to 16 bits PCM works.
Is that a bug ? On vanilla 0.44.
Pd-list@iem.at mailing list UNSUBSCRIBE and account-management -> http://lists.puredata.info/listinfo/pd-list
On 03/16/2013 23:42, Miller Puckette wrote:
It's in the source - but Pd has to be compiled with '_LARGEFILE64_SOURCE' defined for this to work. Through 0.43 the configure script I was using was checking for this somehow (too complicted for me to understand how).
Anyhow, this is a bug all right - thanks for reporting it!
... and can you tell me what OS you're on so I can check if I'm actually fixing it?
it's in the source, the configure script already defines _LARGEFILE64_SOURCE (iirc) but something still fails (yea, definitely a bug)
i confirmed that behaviour a few months ago on linux.
fgamr IOhannes
fbgmasdr IOhannes
So a quick (and perhaps stupid) question - can I just use #idfef __linux__ to alias open to open64, etc? It works on my x86_64 machine and on my Raspberry Pi so I'm guessing all modern linuxes define an open64(). My main fear is that android might define __linux__ but not open64() - anyone know about that?
thanks Miller
On Sun, Mar 17, 2013 at 10:27:41AM +0100, IOhannes m zmölnig wrote:
On 03/16/2013 23:42, Miller Puckette wrote:
It's in the source - but Pd has to be compiled with '_LARGEFILE64_SOURCE' defined for this to work. Through 0.43 the configure script I was using was checking for this somehow (too complicted for me to understand how).
Anyhow, this is a bug all right - thanks for reporting it!
... and can you tell me what OS you're on so I can check if I'm actually fixing it?
it's in the source, the configure script already defines _LARGEFILE64_SOURCE (iirc) but something still fails (yea, definitely a bug)
i confirmed that behaviour a few months ago on linux.
fgamr IOhannes
fbgmasdr IOhannes
Pd-list@iem.at mailing list UNSUBSCRIBE and account-management -> http://lists.puredata.info/listinfo/pd-list
Hi,
thanks for taking that up.
On my machine (archlinux, Linux 3.7.10-1-ARCH i686 GNU/Linux, glibc 2.17), changing the LARGEFILE define to __linux__ (or __gnu_linux__ yields:
git pull
./autogen.sh
./configure --prefix=/opt/pd-vanilla/git --enable-jack
make prefix=/opt/pd-vanilla-git
[...]
gcc -DPACKAGE_NAME="pd" -DPACKAGE_TARNAME="pd"
-DPACKAGE_VERSION="0.44.0" -DPACKAGE_STRING="pd\ 0.44.0"
-DPACKAGE_BUGREPORT="" -DPACKAGE_URL="" -DPACKAGE="pd"
-DVERSION="0.44.0" -DSTDC_HEADERS=1 -DHAVE_SYS_TYPES_H=1
-DHAVE_SYS_STAT_H=1 -DHAVE_STDLIB_H=1 -DHAVE_STRING_H=1
-DHAVE_MEMORY_H=1 -DHAVE_STRINGS_H=1 -DHAVE_INTTYPES_H=1
-DHAVE_STDINT_H=1 -DHAVE_UNISTD_H=1 -DHAVE_DLFCN_H=1
-DLT_OBJDIR=".libs/" -DHAVE_LIBDL=1 -DSTDC_HEADERS=1 -DHAVE_ALLOCA_H=1
-DHAVE_ALLOCA=1 -DHAVE_FCNTL_H=1 -DHAVE_LIMITS_H=1 -DHAVE_MALLOC_H=1
-DHAVE_NETDB_H=1 -DHAVE_NETINET_IN_H=1 -DHAVE_STDDEF_H=1
-DHAVE_STDLIB_H=1 -DHAVE_STRING_H=1 -DHAVE_SYS_IOCTL_H=1
-DHAVE_SYS_PARAM_H=1 -DHAVE_SYS_SOCKET_H=1 -DHAVE_SYS_SOUNDCARD_H=1
-DHAVE_SYS_TIME_H=1 -DHAVE_SYS_TIMEB_H=1 -DHAVE_UNISTD_H=1 -DHAVE_FORK=1
-DHAVE_VFORK=1 -DHAVE_WORKING_VFORK=1 -DHAVE_WORKING_FORK=1
-DHAVE_STDLIB_H=1 -DHAVE_MALLOC=1 -DHAVE_STDLIB_H=1 -DHAVE_REALLOC=1
-DRETSIGTYPE=void -DHAVE_DUP2=1 -DHAVE_GETCWD=1 -DHAVE_GETHOSTBYNAME=1
-DHAVE_GETTIMEOFDAY=1 -DHAVE_MEMMOVE=1 -DHAVE_MEMSET=1 -DHAVE_REGCOMP=1
-DHAVE_SELECT=1 -DHAVE_SOCKET=1 -DHAVE_STRCHR=1 -DHAVE_STRERROR=1
-DHAVE_STRRCHR=1 -DHAVE_STRSTR=1 -DHAVE_STRTOL=1 -I. -DPD
-DINSTALL_PREFIX="/opt/pd-vanilla-git" -DUSEAPI_ALSA -DUSEAPI_JACK
-DJACK_XRUN -DUSEAPI_OSS -DUSEAPI_PORTAUDIO -I../portaudio/include
-O6 -funroll-loops -fomit-frame-pointer -MT pd-s_path.o -MD -MP -MF
.deps/pd-s_path.Tpo -c -o pd-s_path.o test -f 's_path.c' || echo './'s_path.c
s_path.c: In function ‘sys_trytoopenone’:
s_path.c:290:21: error: storage size of ‘statbuf’ isn’t known
[...]
And that can make sense since stat is now stat64, the structure name: struct stat statbuf is struct stat64 statbuf;
So I tried with these defines instead : #ifdef __gnu_linux__ # define open(x, ...) open64(x ,__VA_ARGS__) # define lseek(x, ...) lseek64(x ,__VA_ARGS__) # define fstat(x, ...) fstat64(x ,__VA_ARGS__) # define stat(x, ...) stat64(x ,__VA_ARGS__) #endif
but opening a large file still fails.
Miller Puckette wrote:
So a quick (and perhaps stupid) question - can I just use #idfef __linux__ to alias open to open64, etc? It works on my x86_64 machine and on my Raspberry Pi so I'm guessing all modern linuxes define an open64(). My main fear is that android might define __linux__ but not open64() - anyone know about that?
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 2013-03-18 11:18, Charles Goyard wrote:
Hi,
thanks for taking that up.
On my machine (archlinux, Linux 3.7.10-1-ARCH i686 GNU/Linux, glibc 2.17), changing the LARGEFILE define to __linux__ (or __gnu_linux__ yields:
nonono. you MUST have "_LARGEFILE64_SOURCE" defined before including unistd.h, else you won't have LFS support. there are configure checks that test whether doing defining _LARGEFILE64_SOURCE makes sense on your system.
having said that, i just noticed that these checks seem to have gone from configure(.ac) during the 0.43 rewrite of the buildsystem. so most likely this is the source of the bug.
fgasdrm IOhannes
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 2013-03-18 13:10, IOhannes m zmoelnig wrote:
On 2013-03-18 11:18, Charles Goyard wrote:
Hi,
thanks for taking that up.
On my machine (archlinux, Linux 3.7.10-1-ARCH i686 GNU/Linux, glibc 2.17), changing the LARGEFILE define to __linux__ (or __gnu_linux__ yields:
nonono. you MUST have "_LARGEFILE64_SOURCE" defined before including unistd.h, else you won't have LFS support. there are configure checks that test whether doing defining _LARGEFILE64_SOURCE makes sense on your system.
having said that, i just noticed that these checks seem to have gone from configure(.ac) during the 0.43 rewrite of the buildsystem. so most likely this is the source of the bug.
so i rechecked and things are a bit more complicated than i originally thought when it comes to large file support (LFS).
it is (well, should be) easy enough to open/read/write large (>2GB) files within Pd. simply using "_LARGEFILE64_SOURCE" or "_FILE_OFFSET_BITS=64" is the recommended way. those defines have to be set whenever unistd is included for open/close/lseek/...
now Pd uses some magic in d_soundfile.c (e.g. redefining "open()" to "open64()"), but that doesn't really help at all, since the files are not opened in d_soundfile.c but instead are opened by open_via_path() in s_path.c. open_via_path() currently does not handle LFS.
the easy way would obviously be to move LFS into s_path.c.
the problem with that is, that s_path implements an API available for externals. if open_via_path() returns a filehandle for an LFS file, and the external has been compiled without LFS, the filehande will be incompatible. see [1] for a discussion.
i guess the only clean way to solve that, is to provide a complete wrapper around the file API, so an external has functions guaranteed to work with the filehandle returned by Pd. currently there are sys_open()/sys_close() and its f*-variants. but we would need at least sys_(f)seek and sys_(f)stat, but preferrably a complete wrapper around LFS-compatible file functions [2].
all this functionality (including the handilng of UTF filenames on W32) is not really Pd-related, and could well be factored out into a separate library.
fgasd, IOhannes
[1] http://ohse.de/uwe/articles/lfs.html [2] http://opengroup.org/platform/lfs.html
the problem with that is, that s_path implements an API available for externals. if open_via_path() returns a filehandle for an LFS file, and the external has been compiled without LFS, the filehande will be incompatible. see [1] for a discussion.
i guess the only clean way to solve that, is to provide a complete wrapper around the file API, so an external has functions guaranteed to work with the filehandle returned by Pd. currently there are sys_open()/sys_close() and its f*-variants. but we would need at least sys_(f)seek and sys_(f)stat, but preferrably a complete wrapper around LFS-compatible file functions [2].
all this functionality (including the handilng of UTF filenames on W32) is not really Pd-related, and could well be factored out into a separate library.
Thanks for the clear explanation of the matter, IOhannes.
Why not simply recompile externals after fixing s_path? Pd-extended already comes with most externals recompiled for every new release and adding legacy stuff just creates more confusion in maintaining the source down the road. In other words, FWIW and IMHO I think there is way too much effort the community is trying to pour into binary compatibility for a system that clearly begs for further enhancements in the core API.
Best wishes,
Ico
To answer Ico's question, the trouble I forsee is musician A gives musician B a patch, containing compiled externs - and then musician B runs it and gets a crash instead of music. Sub-optimal in my opinion :)
I now think that it should be OK to use open64() only in the file d_soundfile.c and only for linux - so putting at the head of the file,
#ifdef __linux__ #define _LARGEFILE64_SOURCE #endif
... then, because opening soundfiles currently uses open_via_path, simply closing the file and re-opening it via open64(). There's a similar hack to open "binbufs" via paths in the function binbuf_read_via_canvas().
There are two other festering problems in open_via_path() that all should probably be fixed in one go by defining an updated function call:
- externs further down the path are chosen in front of abstractions that
should be taken instead;
- open_via_path isn't thread safe - d_soundfile.c could call it at the same
moment someone in the "Pd" thread is changing the path. This should almost never hapen but should be fixed.
This is a big enough change that I think it should wait for 0.45, whereas the hack described above could go in right now as a local bug-fix.
BTW I've got a couple of other bug-fixes underway; wil push to git now.
cheers Miller
Oe Mon, Mar 18, 2013 at 12:47:47PM -0400, Ivica Ico Bukvic wrote:
the problem with that is, that s_path implements an API available for externals. if open_via_path() returns a filehandle for an LFS file, and the external has been compiled without LFS, the filehande will be incompatible. see [1] for a discussion.
i guess the only clean way to solve that, is to provide a complete wrapper around the file API, so an external has functions guaranteed to work with the filehandle returned by Pd. currently there are sys_open()/sys_close() and its f*-variants. but we would need at least sys_(f)seek and sys_(f)stat, but preferrably a complete wrapper around LFS-compatible file functions [2].
all this functionality (including the handilng of UTF filenames on W32) is not really Pd-related, and could well be factored out into a separate library.
Thanks for the clear explanation of the matter, IOhannes.
Why not simply recompile externals after fixing s_path? Pd-extended already comes with most externals recompiled for every new release and adding legacy stuff just creates more confusion in maintaining the source down the road. In other words, FWIW and IMHO I think there is way too much effort the community is trying to pour into binary compatibility for a system that clearly begs for further enhancements in the core API.
Best wishes,
Ico
Pd-list@iem.at mailing list UNSUBSCRIBE and account-management -> http://lists.puredata.info/listinfo/pd-list
I completely agree with your example, Miller, as far as pd vanilla is concerned. OTOH pd-extended ships with all its externals compiled with each version, so this would be for the most part a non-issue unless:
- users A or B are using different versions of pd (which could be
elaborated upon in the documentation when sharing patches which is rather common in other software environments), or
- if A is using custom externals that are not found in pd-extended (in
which case B would not want to use those anyways unless both A and B are using the same platform, which is the only scenario where it would make sense to keep this as a workaround)
Either way FWIW I still feel this is too much of a workaround for little gain and potentially a lot of headache down the road.
Best wishes,
Ico
-----Original Message----- From: Miller Puckette [mailto:msp@ucsd.edu] Sent: Monday, March 18, 2013 2:47 PM To: Ivica Ico Bukvic Cc: 'IOhannes m zmoelnig'; pd-list@iem.at Subject: Re: [PD] Large File Support on Linux
To answer Ico's question, the trouble I forsee is musician A gives musician B a patch, containing compiled externs - and then musician B runs it and gets a crash instead of music. Sub-optimal in my opinion :)
I now think that it should be OK to use open64() only in the file
d_soundfile.c
and only for linux - so putting at the head of the file,
#ifdef __linux__ #define _LARGEFILE64_SOURCE #endif
... then, because opening soundfiles currently uses open_via_path, simply closing the file and re-opening it via open64(). There's a similar hack to open "binbufs" via paths in the function binbuf_read_via_canvas().
There are two other festering problems in open_via_path() that all should probably be fixed in one go by defining an updated function call:
- externs further down the path are chosen in front of abstractions that
should be taken instead;
- open_via_path isn't thread safe - d_soundfile.c could call it at the
same
moment someone in the "Pd" thread is changing the path. This should almost never hapen but should be fixed.
This is a big enough change that I think it should wait for 0.45, whereas
the
hack described above could go in right now as a local bug-fix.
BTW I've got a couple of other bug-fixes underway; wil push to git now.
cheers Miller
Oe Mon, Mar 18, 2013 at 12:47:47PM -0400, Ivica Ico Bukvic wrote:
the problem with that is, that s_path implements an API available for externals. if open_via_path() returns a filehandle for an LFS file, and the external has been compiled without LFS, the filehande will be incompatible. see [1] for a discussion.
i guess the only clean way to solve that, is to provide a complete wrapper around the file API, so an external has functions guaranteed to work with the filehandle returned by Pd. currently there are sys_open()/sys_close() and its f*-variants. but we would need at least sys_(f)seek and sys_(f)stat, but preferrably a complete wrapper around LFS-compatible file functions
[2].
all this functionality (including the handilng of UTF filenames on W32) is not really Pd-related, and could well be factored out into a separate library.
Thanks for the clear explanation of the matter, IOhannes.
Why not simply recompile externals after fixing s_path? Pd-extended
already
comes with most externals recompiled for every new release and adding
legacy
stuff just creates more confusion in maintaining the source down the
road.
In other words, FWIW and IMHO I think there is way too much effort the community is trying to pour into binary compatibility for a system that clearly begs for further enhancements in the core API.
Best wishes,
Ico
Pd-list@iem.at mailing list UNSUBSCRIBE and account-management ->
Hi,
I am in favour of breaking binary compatibility and keeping the code simple. People that compile their externals themselves can understand and cope with it. Other people mostly won't notice.
My intuition is that if the LFS project was unable to provide a transparent API in the first place, there's no reason we will be able to do anything clean.
Just communicate enough about the breakage and enjoy :).
Miller Puckette wrote:
To answer Ico's question, the trouble I forsee is musician A gives musician B a patch, containing compiled externs - and then musician B runs it and gets a crash instead of music. Sub-optimal in my opinion :)
On further thought, I don't think it's even possible to change open_via_path() to use open64() and maintain even source compatibility for externs - if one of them calls lseek or fstat we're sunk - and I don't see any robust way of aliasing those calls to lseek64() etc.
I'm now realizing too that I don't know what approaches the Mac and Microsoft pltforms are taking to large file support - I think any solution will have to somehow address all of the platforms.
For right now I'd like a way to fix 0.44's sfread~ - I'm thinking I can do that internally to d_soundfile.c so as to cause the least possible disruption in a 'bugfix' pd release (probably 0.44-3).
cheers Miller On Tue, Mar 19, 2013 at 08:55:07PM +0100, Charles Goyard wrote:
Hi,
I am in favour of breaking binary compatibility and keeping the code simple. People that compile their externals themselves can understand and cope with it. Other people mostly won't notice.
My intuition is that if the LFS project was unable to provide a transparent API in the first place, there's no reason we will be able to do anything clean.
Just communicate enough about the breakage and enjoy :).
Miller Puckette wrote:
To answer Ico's question, the trouble I forsee is musician A gives musician B a patch, containing compiled externs - and then musician B runs it and gets a crash instead of music. Sub-optimal in my opinion :)
Pd-list@iem.at mailing list UNSUBSCRIBE and account-management -> http://lists.puredata.info/listinfo/pd-list
On 03/20/2013 18:02, Miller Puckette wrote:
On further thought, I don't think it's even possible to change open_via_path() to use open64() and maintain even source compatibility for externs - if one of them calls lseek or fstat we're sunk - and I don't see any robust way of aliasing those calls to lseek64() etc.
yep. that's why i said that the only real solution would be to provide a more-or-less complete set of fileIO-functions: sys_lseek, sys_stat (including f-variants).
I'm now realizing too that I don't know what approaches the Mac and Microsoft pltforms are taking to large file support - I think any solution will have to somehow address all of the platforms.
which would be possible if we provided the full set. of file accessors. (but to repeat, this is not really a Pd-problem, and could (and maybe should) be solved in an auxiliary lib).
For right now I'd like a way to fix 0.44's sfread~ - I'm thinking I can do that internally to d_soundfile.c so as to cause the least possible disruption in a 'bugfix' pd release (probably 0.44-3).
which i think is a good enough approach. but of course there's a catch: sys_open() will gracefully handle UTF-8 filenames, whereas on some platforms open() will not. so with your solution, soundfiles with non-ascii chars will fail to open on W32.
since open_via_path() is so often used to determine the full path of a file somewhere in the searchpath, it might be a good idea to wrap that functionality into a find_via_path() function that returns the canonicalized filename. it would be great if that filename would be mangled in a way so UTF8 doesn't make problems any more, but i think this is simply not possible with the way w32 handles opening such filenames.
fgmadsr IOhannes
OK.. I think I'm sold (for 0.45 :) on trying to get sys_open, etc., to do the "right" thing. I guess on Windows this would have to be somehow both UTF8 and 64-bit safe - all the more reason to do as you suggest and hide the whole mes behind sys_this_and_that() variants.
I thought exactly the same thing about a find_via_path routine. It seems to me that open_via_path, which tries not just to verify that the file exists but also that it can actually be opened, is perhaps working too hard; if a file that proves impossible to open is in an earlier spot on the path, the best thing might be to try and fail to open it instead of going out to another, perhaps incorrect, version of the file.
Also, it could be fixed to take the path as an argument so that d_soundfile.c can call it from other threads safely (using separate copies of the path).
cheers Miller
On Wed, Mar 20, 2013 at 09:42:42PM +0100, IOhannes m zmölnig wrote:
On 03/20/2013 18:02, Miller Puckette wrote:
On further thought, I don't think it's even possible to change open_via_path() to use open64() and maintain even source compatibility for externs - if one of them calls lseek or fstat we're sunk - and I don't see any robust way of aliasing those calls to lseek64() etc.
yep. that's why i said that the only real solution would be to provide a more-or-less complete set of fileIO-functions: sys_lseek, sys_stat (including f-variants).
I'm now realizing too that I don't know what approaches the Mac and Microsoft pltforms are taking to large file support - I think any solution will have to somehow address all of the platforms.
which would be possible if we provided the full set. of file accessors. (but to repeat, this is not really a Pd-problem, and could (and maybe should) be solved in an auxiliary lib).
For right now I'd like a way to fix 0.44's sfread~ - I'm thinking I can do that internally to d_soundfile.c so as to cause the least possible disruption in a 'bugfix' pd release (probably 0.44-3).
which i think is a good enough approach. but of course there's a catch: sys_open() will gracefully handle UTF-8 filenames, whereas on some platforms open() will not. so with your solution, soundfiles with non-ascii chars will fail to open on W32.
since open_via_path() is so often used to determine the full path of a file somewhere in the searchpath, it might be a good idea to wrap that functionality into a find_via_path() function that returns the canonicalized filename. it would be great if that filename would be mangled in a way so UTF8 doesn't make problems any more, but i think this is simply not possible with the way w32 handles opening such filenames.
fgmadsr IOhannes
Pd-list@iem.at mailing list UNSUBSCRIBE and account-management -> http://lists.puredata.info/listinfo/pd-list
Miller,
Does this mean if one were not so concerned with backwards compatibility and included define you listed below in the s_path.c that this would fix the LFS issue albeit at the expense of backwards/cross-platform compatibility? Also, in Linux is open64 safe for both 32-bit and 64-bit OS variants?
-----Original Message----- From: pd-list-bounces@iem.at [mailto:pd-list-bounces@iem.at] On Behalf Of Miller Puckette Sent: Wednesday, March 20, 2013 6:46 PM To: IOhannes m zmölnig Cc: pd-list@iem.at Subject: Re: [PD] Large File Support on Linux
OK.. I think I'm sold (for 0.45 :) on trying to get sys_open, etc., to do the "right" thing. I guess on Windows this would have to be somehow both UTF8 and 64-bit safe - all the more reason to do as you suggest and hide
the
whole mes behind sys_this_and_that() variants.
I thought exactly the same thing about a find_via_path routine. It seems
to
me that open_via_path, which tries not just to verify that the file exists
but
also that it can actually be opened, is perhaps working too hard; if a
file
that proves impossible to open is in an earlier spot on the path, the best thing might be to try and fail to open it instead of going out to another, perhaps incorrect, version of the file.
Also, it could be fixed to take the path as an argument so that
d_soundfile.c
can call it from other threads safely (using separate copies of the path).
cheers Miller
On Wed, Mar 20, 2013 at 09:42:42PM +0100, IOhannes m zmölnig wrote:
On 03/20/2013 18:02, Miller Puckette wrote:
On further thought, I don't think it's even possible to change
open_via_path()
to use open64() and maintain even source compatibility for externs -
if
one of them calls lseek or fstat we're sunk - and I don't see any
robust
way of aliasing those calls to lseek64() etc.
yep. that's why i said that the only real solution would be to provide a more-or-less complete set of fileIO-functions: sys_lseek, sys_stat (including f-variants).
I'm now realizing too that I don't know what approaches the Mac and
Microsoft
pltforms are taking to large file support - I think any solution will
have to
somehow address all of the platforms.
which would be possible if we provided the full set. of file accessors. (but to repeat, this is not really a Pd-problem, and could (and maybe should) be solved in an auxiliary lib).
For right now I'd like a way to fix 0.44's sfread~ - I'm thinking I
can do
that internally to d_soundfile.c so as to cause the least possible
disruption
in a 'bugfix' pd release (probably 0.44-3).
which i think is a good enough approach. but of course there's a catch: sys_open() will gracefully handle UTF-8 filenames, whereas on some platforms open() will not. so with your solution, soundfiles with non-ascii chars will fail to open on W32.
since open_via_path() is so often used to determine the full path of a file somewhere in the searchpath, it might be a good idea to wrap that functionality into a find_via_path() function that returns the canonicalized filename. it would be great if that filename would be mangled in a way so UTF8 doesn't make problems any more, but i think this is simply not possible with the way w32 handles opening such
filenames.
fgmadsr IOhannes
Pd-list@iem.at mailing list UNSUBSCRIBE and account-management ->
http://lists.puredata.info/listinfo/pd-list
Pd-list@iem.at mailing list UNSUBSCRIBE and account-management -> http://lists.puredata.info/listinfo/pd-list
I believe not - every extern that used open_bia_path0 would have to be fixed at the source level to use lseek64(), fstat64() in place of lseek() and fstat(). I can't see any way to "fix" this source-compatibly.
I checked open64() on 32-bit Raspbian (ARM) and it worked fine - so I think it's save to say open64 exists on all modern linuxes, both 64 and 32 bit.
cheers Miller On Wed, Mar 20, 2013 at 09:14:42PM -0400, Ivica Ico Bukvic wrote:
Miller,
Does this mean if one were not so concerned with backwards compatibility and included define you listed below in the s_path.c that this would fix the LFS issue albeit at the expense of backwards/cross-platform compatibility? Also, in Linux is open64 safe for both 32-bit and 64-bit OS variants?
-----Original Message----- From: pd-list-bounces@iem.at [mailto:pd-list-bounces@iem.at] On Behalf Of Miller Puckette Sent: Wednesday, March 20, 2013 6:46 PM To: IOhannes m zmölnig Cc: pd-list@iem.at Subject: Re: [PD] Large File Support on Linux
OK.. I think I'm sold (for 0.45 :) on trying to get sys_open, etc., to do the "right" thing. I guess on Windows this would have to be somehow both UTF8 and 64-bit safe - all the more reason to do as you suggest and hide
the
whole mes behind sys_this_and_that() variants.
I thought exactly the same thing about a find_via_path routine. It seems
to
me that open_via_path, which tries not just to verify that the file exists
but
also that it can actually be opened, is perhaps working too hard; if a
file
that proves impossible to open is in an earlier spot on the path, the best thing might be to try and fail to open it instead of going out to another, perhaps incorrect, version of the file.
Also, it could be fixed to take the path as an argument so that
d_soundfile.c
can call it from other threads safely (using separate copies of the path).
cheers Miller
On Wed, Mar 20, 2013 at 09:42:42PM +0100, IOhannes m zmölnig wrote:
On 03/20/2013 18:02, Miller Puckette wrote:
On further thought, I don't think it's even possible to change
open_via_path()
to use open64() and maintain even source compatibility for externs -
if
one of them calls lseek or fstat we're sunk - and I don't see any
robust
way of aliasing those calls to lseek64() etc.
yep. that's why i said that the only real solution would be to provide a more-or-less complete set of fileIO-functions: sys_lseek, sys_stat (including f-variants).
I'm now realizing too that I don't know what approaches the Mac and
Microsoft
pltforms are taking to large file support - I think any solution will
have to
somehow address all of the platforms.
which would be possible if we provided the full set. of file accessors. (but to repeat, this is not really a Pd-problem, and could (and maybe should) be solved in an auxiliary lib).
For right now I'd like a way to fix 0.44's sfread~ - I'm thinking I
can do
that internally to d_soundfile.c so as to cause the least possible
disruption
in a 'bugfix' pd release (probably 0.44-3).
which i think is a good enough approach. but of course there's a catch: sys_open() will gracefully handle UTF-8 filenames, whereas on some platforms open() will not. so with your solution, soundfiles with non-ascii chars will fail to open on W32.
since open_via_path() is so often used to determine the full path of a file somewhere in the searchpath, it might be a good idea to wrap that functionality into a find_via_path() function that returns the canonicalized filename. it would be great if that filename would be mangled in a way so UTF8 doesn't make problems any more, but i think this is simply not possible with the way w32 handles opening such
filenames.
fgmadsr IOhannes
Pd-list@iem.at mailing list UNSUBSCRIBE and account-management ->
http://lists.puredata.info/listinfo/pd-list
Pd-list@iem.at mailing list UNSUBSCRIBE and account-management -> http://lists.puredata.info/listinfo/pd-list
Could one simply redefine lseek and fstat to use lseek64 and fstat64 instead and be done with it (again, assuming one is not concerned with backwards and/or x-platform compatibility)?
Best wishes,
Ico
-----Original Message----- From: Miller Puckette [mailto:msp@ucsd.edu] Sent: Wednesday, March 20, 2013 9:21 PM To: Ivica Ico Bukvic Cc: 'IOhannes m zmölnig'; pd-list@iem.at Subject: Re: [PD] Large File Support on Linux
I believe not - every extern that used open_bia_path0 would have to be fixed at the source level to use lseek64(), fstat64() in place of lseek() and fstat(). I can't see any way to "fix" this source-compatibly.
I checked open64() on 32-bit Raspbian (ARM) and it worked fine - so I
think
it's save to say open64 exists on all modern linuxes, both 64 and 32 bit.
cheers Miller On Wed, Mar 20, 2013 at 09:14:42PM -0400, Ivica Ico Bukvic wrote:
Miller,
Does this mean if one were not so concerned with backwards compatibility
and
included define you listed below in the s_path.c that this would fix the
LFS
issue albeit at the expense of backwards/cross-platform compatibility?
Also,
in Linux is open64 safe for both 32-bit and 64-bit OS variants?
-----Original Message----- From: pd-list-bounces@iem.at [mailto:pd-list-bounces@iem.at] On
Behalf Of
Miller Puckette Sent: Wednesday, March 20, 2013 6:46 PM To: IOhannes m zmölnig Cc: pd-list@iem.at Subject: Re: [PD] Large File Support on Linux
OK.. I think I'm sold (for 0.45 :) on trying to get sys_open, etc., to
do
the "right" thing. I guess on Windows this would have to be somehow
both
UTF8 and 64-bit safe - all the more reason to do as you suggest and
hide
the
whole mes behind sys_this_and_that() variants.
I thought exactly the same thing about a find_via_path routine. It
seems
to
me that open_via_path, which tries not just to verify that the file
exists
but
also that it can actually be opened, is perhaps working too hard; if a
file
that proves impossible to open is in an earlier spot on the path, the
best
thing might be to try and fail to open it instead of going out to
another,
perhaps incorrect, version of the file.
Also, it could be fixed to take the path as an argument so that
d_soundfile.c
can call it from other threads safely (using separate copies of the
path).
cheers Miller
On Wed, Mar 20, 2013 at 09:42:42PM +0100, IOhannes m zmölnig wrote:
On 03/20/2013 18:02, Miller Puckette wrote:
On further thought, I don't think it's even possible to change
open_via_path()
to use open64() and maintain even source compatibility for externs
if
one of them calls lseek or fstat we're sunk - and I don't see any
robust
way of aliasing those calls to lseek64() etc.
yep. that's why i said that the only real solution would be to provide a more-or-less complete set of fileIO-functions: sys_lseek, sys_stat (including f-variants).
I'm now realizing too that I don't know what approaches the Mac
and
Microsoft
pltforms are taking to large file support - I think any solution
will
have to
somehow address all of the platforms.
which would be possible if we provided the full set. of file
accessors.
(but to repeat, this is not really a Pd-problem, and could (and
maybe
should) be solved in an auxiliary lib).
For right now I'd like a way to fix 0.44's sfread~ - I'm thinking
I
can do
that internally to d_soundfile.c so as to cause the least possible
disruption
in a 'bugfix' pd release (probably 0.44-3).
which i think is a good enough approach. but of course there's a catch: sys_open() will gracefully handle
UTF-8
filenames, whereas on some platforms open() will not. so with your solution, soundfiles with non-ascii chars will fail to
open
on W32.
since open_via_path() is so often used to determine the full path of
a
file somewhere in the searchpath, it might be a good idea to wrap
that
functionality into a find_via_path() function that returns the canonicalized filename. it would be great if that filename would be mangled in a way so UTF8 doesn't make problems any more, but i think this is simply not possible with the way w32 handles opening such
filenames.
fgmadsr IOhannes
Pd-list@iem.at mailing list UNSUBSCRIBE and account-management ->
http://lists.puredata.info/listinfo/pd-list
Pd-list@iem.at mailing list UNSUBSCRIBE and account-management -> http://lists.puredata.info/listinfo/pd-list
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 2013-03-21 04:22, Ivica Ico Bukvic wrote:
Could one simply redefine lseek and fstat to use lseek64 and fstat64 instead and be done with it (again, assuming one is not concerned with backwards and/or x-platform compatibility)?
if you are talking about redefining (as in "#define open open64") in m_pd.h, than please: no NO!
that will break compilation of a number of externals immediately. one that comes to my mind is any C++-external (including Gem) that uses "open" methods in classes. e.g. foo::open(s); will expand to foo::open64(s); which will break binary compatibility when linking to any library providing foo::open() simply by including m_pd.h
gfmasdr IOhannes
Hi,
Here's a small summary of what has to be done, from http://users.suse.com/~aj/linux_lfs.html :
" In a nutshell for using LFS you can choose either of the following:
Compile your programs with "gcc -D_FILE_OFFSET_BITS=64". This forces
all file access calls to use the 64 bit variants. Several types
change also, e.g. off_t becomes off64_t. It's therefore important to
always use the correct types and to not use e.g. int instead of
off_t. For portability with other platforms you should use getconf
LFS_CFLAGS which will return -D_FILE_OFFSET_BITS=64 on Linux
platforms but might return something else on e.g. Solaris. For
linking, you should use the link flags that are reported via getconf
LFS_LDFLAGS. On Linux systems, you do not need special link flags.
Define _LARGEFILE_SOURCE and _LARGEFILE64_SOURCE. With these
defines you can use the LFS functions like open64 directly.
Use the O_LARGEFILE flag with open to operate on large
files.
A complete documentation of the feature test macros like
_FILE_OFFSET_BITS and _LARGEFILE_SOURCE is in the glibc
manual (run e.g. "info libc 'Feature Test Macros'").
The LFS API is also documented in the LFS standard which is
available at
http://ftp.sas.com/standards/large.file/x_open.20Mar96.html.
"
Note than open_via_file returns int and not off_t, so the automagic stuff will fail.
I guess Miller is right and it should be targeted to 0.45 or 0.46, that's not a small change.
Cheers, Charles
Ivica Ico Bukvic wrote:
Could one simply redefine lseek and fstat to use lseek64 and fstat64 instead and be done with it (again, assuming one is not concerned with backwards and/or x-platform compatibility)?
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 2013-03-21 09:37, Charles Goyard wrote:
Note than open_via_file returns int and not off_t, so the automagic stuff will fail.
that's wrong. the declaration of open64 is:
int open64(const char *pathname, int oflag,...);
so it returns an int, not an off_t, just like open_via_file().
iiuc, the problem is, that the file descriptor cannot be used by an ordinary lseek(), but only with lseek64().
that's what i was referring to when i said that one would need to provide sys_lseek().
if you go the way of simply defining _FILE_OFFSET_BITS=64 for compiling Pd, you must compile all externals using _FILE_OFFSET_BITS=64 as well, else they won't be able to the file-handle internally. i think this is **very** bad style, as it forces the use of some obscure and potentially harmful compiler flags in order get externals to work.
gamsdr IOhannes
IOhannes m zmoelnig wrote:
On 2013-03-21 09:37, Charles Goyard wrote:
Note than open_via_file returns int and not off_t, so the automagic stuff will fail.
that's wrong. the declaration of open64 is:
int open64(const char *pathname, int oflag,...);
so it returns an int, not an off_t, just like open_via_file().
Yes, I got mixed up somewhere between filehandles and file positions.
However, that's true of lseek(), that returns an off_t, when d_soundfile.c uses long.
Sorry for the confusion.
OK... now I'm trying to fix the bug but I can't reproduce it. I made a 5 GB soundfile (wave format, 60 channels, 4-bit floats, 450 seconds long) and opened and am reading it using readsf~. This is on a 64 bit linux box.
Question for Charles Goyard: were you on a 32-bit linux machine? Or did you do something else besides just read the file from its beginning?
thanks Miller
On Wed, Mar 20, 2013 at 06:20:59PM -0700, Miller Puckette wrote:
I believe not - every extern that used open_bia_path0 would have to be fixed at the source level to use lseek64(), fstat64() in place of lseek() and fstat(). I can't see any way to "fix" this source-compatibly.
I checked open64() on 32-bit Raspbian (ARM) and it worked fine - so I think it's save to say open64 exists on all modern linuxes, both 64 and 32 bit.
cheers Miller On Wed, Mar 20, 2013 at 09:14:42PM -0400, Ivica Ico Bukvic wrote:
Miller,
Does this mean if one were not so concerned with backwards compatibility and included define you listed below in the s_path.c that this would fix the LFS issue albeit at the expense of backwards/cross-platform compatibility? Also, in Linux is open64 safe for both 32-bit and 64-bit OS variants?
-----Original Message----- From: pd-list-bounces@iem.at [mailto:pd-list-bounces@iem.at] On Behalf Of Miller Puckette Sent: Wednesday, March 20, 2013 6:46 PM To: IOhannes m zmölnig Cc: pd-list@iem.at Subject: Re: [PD] Large File Support on Linux
OK.. I think I'm sold (for 0.45 :) on trying to get sys_open, etc., to do the "right" thing. I guess on Windows this would have to be somehow both UTF8 and 64-bit safe - all the more reason to do as you suggest and hide
the
whole mes behind sys_this_and_that() variants.
I thought exactly the same thing about a find_via_path routine. It seems
to
me that open_via_path, which tries not just to verify that the file exists
but
also that it can actually be opened, is perhaps working too hard; if a
file
that proves impossible to open is in an earlier spot on the path, the best thing might be to try and fail to open it instead of going out to another, perhaps incorrect, version of the file.
Also, it could be fixed to take the path as an argument so that
d_soundfile.c
can call it from other threads safely (using separate copies of the path).
cheers Miller
On Wed, Mar 20, 2013 at 09:42:42PM +0100, IOhannes m zmölnig wrote:
On 03/20/2013 18:02, Miller Puckette wrote:
On further thought, I don't think it's even possible to change
open_via_path()
to use open64() and maintain even source compatibility for externs -
if
one of them calls lseek or fstat we're sunk - and I don't see any
robust
way of aliasing those calls to lseek64() etc.
yep. that's why i said that the only real solution would be to provide a more-or-less complete set of fileIO-functions: sys_lseek, sys_stat (including f-variants).
I'm now realizing too that I don't know what approaches the Mac and
Microsoft
pltforms are taking to large file support - I think any solution will
have to
somehow address all of the platforms.
which would be possible if we provided the full set. of file accessors. (but to repeat, this is not really a Pd-problem, and could (and maybe should) be solved in an auxiliary lib).
For right now I'd like a way to fix 0.44's sfread~ - I'm thinking I
can do
that internally to d_soundfile.c so as to cause the least possible
disruption
in a 'bugfix' pd release (probably 0.44-3).
which i think is a good enough approach. but of course there's a catch: sys_open() will gracefully handle UTF-8 filenames, whereas on some platforms open() will not. so with your solution, soundfiles with non-ascii chars will fail to open on W32.
since open_via_path() is so often used to determine the full path of a file somewhere in the searchpath, it might be a good idea to wrap that functionality into a find_via_path() function that returns the canonicalized filename. it would be great if that filename would be mangled in a way so UTF8 doesn't make problems any more, but i think this is simply not possible with the way w32 handles opening such
filenames.
fgmadsr IOhannes
Pd-list@iem.at mailing list UNSUBSCRIBE and account-management ->
http://lists.puredata.info/listinfo/pd-list
Pd-list@iem.at mailing list UNSUBSCRIBE and account-management -> http://lists.puredata.info/listinfo/pd-list
Pd-list@iem.at mailing list UNSUBSCRIBE and account-management -> http://lists.puredata.info/listinfo/pd-list
Never mind - it stopped after 74 seconds (instead of 450) - presumably 4GB early. But I don't believe this has anything at all to do with my using open() instead of open64() - apparently writesf~ put the wrong number in the soundfile header.
cheers Miller
On Sat, Mar 23, 2013 at 08:15:47PM -0700, Miller Puckette wrote:
OK... now I'm trying to fix the bug but I can't reproduce it. I made a 5 GB soundfile (wave format, 60 channels, 4-bit floats, 450 seconds long) and opened and am reading it using readsf~. This is on a 64 bit linux box.
Question for Charles Goyard: were you on a 32-bit linux machine? Or did you do something else besides just read the file from its beginning?
thanks Miller
Hi Miller,
Yes I am on 32bit linux. For what I understand on 64 bits Linux LFS is out of the box. The problem stands only for 32 bits hosts it seems.
I can follow a branch on git and test for you if you wish.
Alternatively I recall there's debian package to host a 32 bits linux in a 64 bits linux.
Thanks Charles
Miller Puckette wrote:
OK... now I'm trying to fix the bug but I can't reproduce it. I made a 5 GB soundfile (wave format, 60 channels, 4-bit floats, 450 seconds long) and opened and am reading it using readsf~. This is on a 64 bit linux box.
Question for Charles Goyard: were you on a 32-bit linux machine? Or did you do something else besides just read the file from its beginning?
No worries I'll just go crank up an old 32 bit linux box myself :)
M
On Sun, Mar 24, 2013 at 04:07:12PM +0100, Charles Goyard wrote:
Hi Miller,
Yes I am on 32bit linux. For what I understand on 64 bits Linux LFS is out of the box. The problem stands only for 32 bits hosts it seems.
I can follow a branch on git and test for you if you wish.
Alternatively I recall there's debian package to host a 32 bits linux in a 64 bits linux.
Thanks Charles
Miller Puckette wrote:
OK... now I'm trying to fix the bug but I can't reproduce it. I made a 5 GB soundfile (wave format, 60 channels, 4-bit floats, 450 seconds long) and opened and am reading it using readsf~. This is on a 64 bit linux box.
Question for Charles Goyard: were you on a 32-bit linux machine? Or did you do something else besides just read the file from its beginning?
Pd-list@iem.at mailing list UNSUBSCRIBE and account-management -> http://lists.puredata.info/listinfo/pd-list
Ivica Ico Bukvic wrote:
Also, in Linux is open64 safe for both 32-bit and 64-bit OS variants?
Yes, you don't have to switch back to open() on 64-bits systems. So basically switching everything to the 64-bits variants works on most Linux systems installed since about 2002/2003.
-
Charles Goyard -
Hans-Christoph Steiner -
IOhannes m zmoelnig -
IOhannes m zmölnig
-
Ivica Ico Bukvic -
Miller Puckette