Hi, I've been getting two errors involving [soundfiler], and before reporting an issue or sharing the patch to see what's ruining it, I'd like to understand and make sense of these errors, cause I just have no idea why they're showing up.
First is "*soundfiler_read: test.wav: Too many open files*".
Well, this is from my partitioned convolution abstraction. Inside the abstraction I call another abstraction in [clone]. So when I get this too many files error it also just prevents the abstraction from loading inside clone, where I get "*clone: can't create subpatch './convpartition'*".
And now things hget just crazier, cause if I try to recreate the asbstraction, it doesn't create in anyway, as if it got corrupted somehow.
And if I try to save the patch, I cant', I get an error like *"~/porres/test.pd: Bad file descriptor"*. All I have left to do is force quit Pd so I can close this file and reopen it.
Ok, so the "too many files" is probably related to the number of instances I'm loading in clone, if there are too many partitions, this happens. But I know it shouldn't, as I was being able to load way more instances and I don't know how this error suddenly came up... none of it makes sense to me, specially what happens after this "too many files" error.
probably better to mention something else separately, but I've also experienced [clone] acting weird, where it seemed to load old versions of abstractions that were already updated or deleted. This may be related to the fact that I cannot recreate the abstraction once it gets corrupted, so it keeps posting "*Bad file descriptor*"...
cheers
Hi Alexandre,
On 08/02/2019 05:57, Alexandre Torres Porres wrote:
First is "*/soundfiler_read: test.wav: Too many open files/*".
Something is almost certainly leaking file handles, when too many are
open at once, you can't open any more (neither for reading nor writing),
so all kinds of bad problems happen. The default limit is usually 1000
or so, on Linux you can check and change with the ulimit command.
It's probably not [soundfiler] or [clone] to blame, but more likely some external that isn't properly closing the files (or directories) that it opens.
On Windows I think you need to close with the right function w.r.t. DLL/EXE module boundaries. There may be something in the list archives about this.
Claude
Hi Alex
I see that your version of [conv~] uses [else/dir] internally and this external of yours is still leaking file handles. I think I already reported this, but can't find an open issue about it. Maybe you closed them, despite them not being fixed.
Anyway, last time I checked, each instance of [dir] was not leaking three file handles anymore, but only one. So there is an improvement, but the issue is not fixed. You really need to close all file handles that you opened before.
Roman
P.S.: I'm kinda happy to see that it bites you. This may sound mean, but I do think that else is a great library and thus the more danger is in such issues not being fixed. So: Thanks for the library, thanks for fixing the issue!
On Fri, 2019-02-08 at 03:57 -0200, Alexandre Torres Porres wrote:
Hi, I've been getting two errors involving [soundfiler], and before reporting an issue or sharing the patch to see what's ruining it, I'd like to understand and make sense of these errors, cause I just have no idea why they're showing up.
First is "soundfiler_read: test.wav: Too many open files".
Well, this is from my partitioned convolution abstraction. Inside the abstraction I call another abstraction in [clone]. So when I get this too many files error it also just prevents the abstraction from loading inside clone, where I get "clone: can't create subpatch './convpartition'".
And now things hget just crazier, cause if I try to recreate the asbstraction, it doesn't create in anyway, as if it got corrupted somehow.
And if I try to save the patch, I cant', I get an error like "~/porres/test.pd: Bad file descriptor". All I have left to do is force quit Pd so I can close this file and reopen it.
Ok, so the "too many files" is probably related to the number of instances I'm loading in clone, if there are too many partitions, this happens. But I know it shouldn't, as I was being able to load way more instances and I don't know how this error suddenly came up... none of it makes sense to me, specially what happens after this "too many files" error.
probably better to mention something else separately, but I've also experienced [clone] acting weird, where it seemed to load old versions of abstractions that were already updated or deleted. This may be related to the fact that I cannot recreate the abstraction once it gets corrupted, so it keeps posting "Bad file descriptor"...
cheers
Pd-list@lists.iem.at mailing list UNSUBSCRIBE and account-management -> https://lists.puredata.info/listinfo/pd-list
On 08/02/2019 09:55, Roman Haefeli wrote:
I see that your version of [conv~] uses [else/dir] internally and this external of yours is still leaking file handles. I think I already reported this, but can't find an open issue about it. Maybe you closed them, despite them not being fixed.
https://github.com/porres/pd-else/blob/9b259f24569a13069c5842ec0f598e8fe96c8...
the leak seems to be on this line. you should keep the return value of opendir and be sure to close it when no longer necessary.
Claude
Em sex, 8 de fev de 2019 às 13:37, Claude Heiland-Allen claude@mathr.co.uk escreveu:
On 08/02/2019 09:55, Roman Haefeli wrote:
I see that your version of [conv~] uses [else/dir] internally and this external of yours is still leaking file handles. I think I already reported this, but can't find an open issue about it. Maybe you closed them, despite them not being fixed.
https://github.com/porres/pd-else/blob/9b259f24569a13069c5842ec0f598e8fe96c8...
the leak seems to be on this line. you should keep the return value of opendir and be sure to close it when no longer necessary.
wow, look at that :) Thanks! I'll get on it as soon as I can.
Em sex, 8 de fev de 2019 às 13:55, Alexandre Torres Porres porres@gmail.com escreveu:
https://github.com/porres/pd-else/blob/9b259f24569a13069c5842ec0f598e8fe96c8...
the leak seems to be on this line. you should keep the return value of opendir and be sure to close it when no longer necessary.
Well, I'll take there must be a leak or something there, but I'm calling this method (dir_open) in the patch. This object opens directories so we query for filenames in it, and for that it depends on <dirent.h>. And the thing is that if there are too many opened directories, crap happens. I've tried everything already, but it seems to be an issue with the opendir() function in dirent.h
See http://pubs.opengroup.org/onlinepubs/7908799/xsh/opendir.html where it says "*applications will only be able to open up to a total of {OPEN_MAX} files and directories*", that might be it. Also, the given errirs in that page mention "descriptors" and how "Too many files are currently open in the system".
Em sex, 8 de fev de 2019 às 07:58, Roman Haefeli reduzent@gmail.com escreveu:
Maybe you closed them, despite them not being fixed.
I may have closed yours because it was a duplicate issue, also reported/discussed here https://github.com/porres/pd-else/issues/248 by Liam.
cheers
Anyway, this means the only way for me to deal with this is to close the directory EVERY time right after I open it. But it's not ideal for the design of this object, where I wanted it to leave the directory open so we could keep querying for files without the need to reopen it every time. I guess I can just deal with this by opening/closing all the time, but it kinda sucks... Unless the wizards out there have a magical way out for me :)
I could also try and see how other Pd objects deal with this... I'm using [dir] to remove my dependency of [playlist], haven't tested how it'd behave in this situation, probably fine. I have tested [hcs/folder_list] though, it does this sort of thing and doesn't seem to crash!
Well, let me go ahead and sort this out, thanks a lot for your help!
cheers
Em sex, 8 de fev de 2019 às 15:59, Alexandre Torres Porres porres@gmail.com escreveu:
Em sex, 8 de fev de 2019 às 13:55, Alexandre Torres Porres < porres@gmail.com> escreveu:
https://github.com/porres/pd-else/blob/9b259f24569a13069c5842ec0f598e8fe96c8...
the leak seems to be on this line. you should keep the return value of opendir and be sure to close it when no longer necessary.
Well, I'll take there must be a leak or something there, but I'm calling this method (dir_open) in the patch. This object opens directories so we query for filenames in it, and for that it depends on <dirent.h>. And the thing is that if there are too many opened directories, crap happens. I've tried everything already, but it seems to be an issue with the opendir() function in dirent.h
See http://pubs.opengroup.org/onlinepubs/7908799/xsh/opendir.html where it says "*applications will only be able to open up to a total of {OPEN_MAX} files and directories*", that might be it. Also, the given errirs in that page mention "descriptors" and how "Too many files are currently open in the system".
Em sex, 8 de fev de 2019 às 07:58, Roman Haefeli reduzent@gmail.com escreveu:
Maybe you closed them, despite them not being fixed.
I may have closed yours because it was a duplicate issue, also reported/discussed here https://github.com/porres/pd-else/issues/248 by Liam.
cheers
On Fri, 2019-02-08 at 16:09 -0200, Alexandre Torres Porres wrote:
Anyway, this means the only way for me to deal with this is to close the directory EVERY time right after I open it. But it's not ideal for the design of this object, where I wanted it to leave the directory open so we could keep querying for files without the need to reopen it every time.
I don't understand what your concerned about. If you worry about efficiency, I believe it does not matter. The data is most likely in the filesystem cache anyway. If you worry about the exposed interface of the object, Dan already pointed out how you could keep the design while closing the file handle internally.
Roman
Em sáb, 9 de fev de 2019 às 09:09, Roman Haefeli reduzent@gmail.com escreveu:
I don't understand what your concerned about. If you worry about efficiency, I believe it does not matter.
yeah, that was my concern
Em sex, 8 de fev de 2019 às 07:58, Roman Haefeli reduzent@gmail.com escreveu:
Hi Alex
I see that your version of [conv~] uses [else/dir] internally and this external of yours is still leaking file handles. I think I already reported this, but can't find an open issue about it. Maybe you closed them, despite them not being fixed.
nope, still open, and yeah, we already tried to fix that but it wasn't fully solved :/
thanks all for clarifying this. I guess I should get back to trying to fix it ;)
Anyway, last time I checked, each instance of [dir] was not leaking three file handles anymore, but only one. So there is an improvement, but the issue is not fixed. You really need to close all file handles that you opened before.
Yeah, that was the best I could do so far...
Roman
P.S.: I'm kinda happy to see that it bites you. This may sound mean, but I do think that else is a great library and thus the more danger is in such issues not being fixed. So: Thanks for the library, thanks for fixing the issue!
I'm actually glad too, haha... but with mixed feelings. I wasn't actually aware this could lead to potential issues.
cheers
-
Alexandre Torres Porres -
Claude Heiland-Allen -
Roman Haefeli