Hello everyone,
Would someone be able to explain to me why d_soundfile.c has the following buffer size allocation?
#define MAXBUFSIZE 16777216 /* arbitrary; just don't want to hang malloc */
I'm not sure about the 'hanging malloc' thing...does it mean that enlarging that buffer size could screw with memory allocation because it we might 'run out'?
AFAICT, 16777216 is about 6.3 minutes. If I need more like 26460000 (10m) allocation, could I just alter MAXBUFSIZE and recompile? Or is there a saner way of doing it, so that I don't end up with a 'custom' pd?
d
-- David Plans Casal Researcher, UEA Studios d.casal at uea dot ac dot uk http://www.davidcasal.com
On Tue, 28 Feb 2006, David Plans Casal wrote:
#define MAXBUFSIZE 16777216 /* arbitrary; just don't want to hang malloc */ I'm not sure about the 'hanging malloc' thing...
Maybe a better word than hanging would be flooding, as in taking too much RAM. But there is another reason not stated, so it's not that arbitrary.
16777216, which is 2^24, is the last contiguous integer representable in the float32 format. 16777217 (and following odd integers) get rounded to the closest even integer. starting at 2^25 you only get multiples of 4 (which is 2^2) and so on. note that 2^25/2^2 = 2^23 which is the degree of precision of float32.
This means that [tabread~] doesn't do what you want starting at 16777216; it also means that [tabread4~] does exactly the same as [tabread~] starting at 2^23=8388608, which is not what you want, and that appears gradually, because between 2^22 and 2^23 you only have steps of 0.5, between 2^21 and 2^22 you only have steps of 0.25, etc.
There are still some objects that can handle big arrays, so the limit should be configurable at run time... but it's not.
does it mean that enlarging that buffer size could screw with memory allocation because it we might 'run out'?
You're allowed to malloc something close to 2^30 bytes = 2^28 float32's in one chunk if you have enough RAM. In 64-bit mode you would be able to malloc more, but pd arrays don't work in 64-bit mode.
If Pd used float64 you wouldn't have problems with tabread~ because float64 contains all integers up to 2^53. However Pd doesn't support float64 yet, and once it does, you can't use float32 anymore, so all your arrays will take double the RAM. To support multiple number formats in signals would be quite a overhaul of Pd. To support them in arrays only, would be easier but still quite a bit of work. (But then you would be able to store your values in int16 format, which is half of the RAM of float32!)
AFAICT, 16777216 is about 6.3 minutes. If I need more like 26460000 (10m) allocation, could I just alter MAXBUFSIZE and recompile?
yes.
Or is there a saner way of doing it, so that I don't end up with a 'custom' pd?
no... well, do you have an hex editor? ;-)
_ _ __ ___ _____ ________ _____________ _____________________ ... | Mathieu Bouchard - tél:+1.514.383.3801 - http://artengine.ca/matju | Freelance Digital Arts Engineer, Montréal QC Canada
Hallo!
AFAICT, 16777216 is about 6.3 minutes. If I need more like 26460000 (10m) allocation, could I just alter MAXBUFSIZE and recompile? Or is there a saner way of doing it, so that I don't end up with a 'custom' pd?
you could use [sndfiler] (but it is threaded - and can also read oggs ;)
LG Georg
On 28 Feb 2006, at 13:21, Georg Holzmann wrote:
Hallo!
AFAICT, 16777216 is about 6.3 minutes. If I need more like 26460000 (10m) allocation, could I just alter MAXBUFSIZE and recompile? Or is there a saner way of doing it, so that I don't end up with a 'custom' pd?
you could use [sndfiler] (but it is threaded - and can also read oggs ;)
Do you mean [soundfiler] ? I thought it could only read wav, aiff and nexstep?
I'm already using soundfiler, but the problems are PD's buffers themselves, or am I wrong...
d
-- David Plans Casal Researcher, UEA Studios d.casal at uea dot ac dot uk http://www.davidcasal.com
Hallo!
Do you mean [soundfiler] ? I thought it could only read wav, aiff and nexstep?
I'm already using soundfiler, but the problems are PD's buffers themselves, or am I wrong...
Sorry, I meant something else: #define DEFMAXSIZE 4000000 /* default maximum 16 MB per channel */
LG Georg
On 28 Feb 2006, at 13:51, Georg Holzmann wrote:
Hallo!
Do you mean [soundfiler] ? I thought it could only read wav, aiff and nexstep? I'm already using soundfiler, but the problems are PD's buffers themselves, or am I wrong...
Sorry, I meant something else: #define DEFMAXSIZE 4000000 /* default maximum 16 MB per channel */
OH! _THere_ it is! :-)
Thanks Georg...
d
-- David Plans Casal Researcher, UEA Studios d.casal at uea dot ac dot uk http://www.davidcasal.com
On some OSes (I needn't name names:) the system can hang if one asks to malloc too much space. This was a major cause of headaches for people trying to use Pd to read large soundfiles. Anyway, as you already found, this doesn't affect the "soundfiler" object which has a settable maximum.
cheers Miller
On Tue, Feb 28, 2006 at 11:54:39AM +0000, David Plans Casal wrote:
Hello everyone,
Would someone be able to explain to me why d_soundfile.c has the following buffer size allocation?
#define MAXBUFSIZE 16777216 /* arbitrary; just don't want to hang malloc */
I'm not sure about the 'hanging malloc' thing...does it mean that enlarging that buffer size could screw with memory allocation because it we might 'run out'?
AFAICT, 16777216 is about 6.3 minutes. If I need more like 26460000 (10m) allocation, could I just alter MAXBUFSIZE and recompile? Or is there a saner way of doing it, so that I don't end up with a 'custom' pd?
d
-- David Plans Casal Researcher, UEA Studios d.casal at uea dot ac dot uk http://www.davidcasal.com
PD-dev mailing list PD-dev@iem.at http://lists.puredata.info/listinfo/pd-dev
On 28 Feb 2006, at 18:29, Miller Puckette wrote:
On some OSes (I needn't name names:) the system can hang if one asks to malloc too much space. This was a major cause of headaches for people trying to use Pd to read large soundfiles. Anyway, as you already found, this doesn't affect the "soundfiler" object which has a settable maximum.
Thank you for that clarification. I understand the original limitation now.
However, I am wondering whether it wouldn't be better to pre-empt large soundfile loading, as opposed to making people who need to use large soundfile arrays use custom compilation?
Just a thought, I realise there's history at play here.
d
-- David Plans Casal Researcher, UEA Studios d.casal at uea dot ac dot uk http://www.davidcasal.com
I think that, with "soundfiler" there's no problem (you just have to specify a bigger maximum in a Pd message).
cheers Miller
On Thu, Mar 02, 2006 at 05:24:35PM +0000, David Plans Casal wrote:
On 28 Feb 2006, at 18:29, Miller Puckette wrote:
On some OSes (I needn't name names:) the system can hang if one asks to malloc too much space. This was a major cause of headaches for people trying to use Pd to read large soundfiles. Anyway, as you already found, this doesn't affect the "soundfiler" object which has a settable maximum.
Thank you for that clarification. I understand the original limitation now.
However, I am wondering whether it wouldn't be better to pre-empt large soundfile loading, as opposed to making people who need to use large soundfile arrays use custom compilation?
Just a thought, I realise there's history at play here.
d
-- David Plans Casal Researcher, UEA Studios d.casal at uea dot ac dot uk http://www.davidcasal.com
PD-dev mailing list PD-dev@iem.at http://lists.puredata.info/listinfo/pd-dev
-
David Plans Casal -
Georg Holzmann -
Mathieu Bouchard -
Miller Puckette