In 0.48.1 on Ubuntu, I'm getting a horrible scenario where PD refuses to open patches or create any more abstractions for me. I get an error message saying "too many open files". Granted I have a lot open, but this is a serious problem as it means I can't access all of my old performances. They worked fine in 0.47.
Any ideas?
Liam
hello,
On 20.05.2018 06:50, Liam Goodacre wrote:
In 0.48.1 on Ubuntu, I'm getting a horrible scenario where PD refuses to open patches or create any more abstractions for me. I get an error message saying "too many open files". Granted I have a lot open, but this is a serious problem as it means I can't access all of my old performances. They worked fine in 0.47.
Any ideas?
this is not really a problem with pd, but with your shell- or system configuration limiting the number of open file descriptors.
check your current shell-limit with ulimit -Sn -S is for soft limit (you can lower, but not raise, the hard limit). raise the limit in your shell with ulimit -n 65536 start pd from that shell and see if the situation improves.
if that doesn't help, you can check the kernel limits with cat /proc/sys/fs/file-nr which returns 3 numbers, the first of which is the number of open files, the last of which is the limit.
increase the value of "nofile" in /etc/security/limits.conf do sudo sysctl -p
additional steps are required to make these settings permanent, the ulimit -n 65536 would have to go into your .bashrc so it's executed at system startup. if you normally start pd from your GUI you'd have to restart your system (actually just xorg) so your master shell knows about the new value. there's other ways, one of which would be to start pd from a wrapper script, or probably gnome provides that sort of environmental setup for it's program shortcuts.
there's more information
https://askubuntu.com/questions/162229/how-do-i-increase-the-open-files-limi...
https://unix.stackexchange.com/questions/29577/ulimit-difference-between-har...
hope that helps ... cheers, ub
Liam
Pd-list@lists.iem.at mailing list UNSUBSCRIBE and account-management -> https://lists.puredata.info/listinfo/pd-list
Thanks for this feedback, it looks very thorough. I'll work through your suggestions later on this week or at the weekend.
So that I can get a better handle on what's causing the issue, am I right in saying that each instantiation of an abstraction constitutes one file descriptor?
Liam ________________________________ From: Pd-list pd-list-bounces@lists.iem.at on behalf of ub@xdv ub@xdv.org Sent: 21 May 2018 19:35 To: pd-list@lists.iem.at Subject: Re: [PD] "too many open files" error in 0.48.1
hello,
On 20.05.2018 06:50, Liam Goodacre wrote:
In 0.48.1 on Ubuntu, I'm getting a horrible scenario where PD refuses to open patches or create any more abstractions for me. I get an error message saying "too many open files". Granted I have a lot open, but this is a serious problem as it means I can't access all of my old performances. They worked fine in 0.47.
Any ideas?
this is not really a problem with pd, but with your shell- or system configuration limiting the number of open file descriptors.
check your current shell-limit with ulimit -Sn -S is for soft limit (you can lower, but not raise, the hard limit). raise the limit in your shell with ulimit -n 65536 start pd from that shell and see if the situation improves.
if that doesn't help, you can check the kernel limits with cat /proc/sys/fs/file-nr which returns 3 numbers, the first of which is the number of open files, the last of which is the limit.
increase the value of "nofile" in /etc/security/limits.conf do sudo sysctl -p
additional steps are required to make these settings permanent, the ulimit -n 65536 would have to go into your .bashrc so it's executed at system startup. if you normally start pd from your GUI you'd have to restart your system (actually just xorg) so your master shell knows about the new value. there's other ways, one of which would be to start pd from a wrapper script, or probably gnome provides that sort of environmental setup for it's program shortcuts.
there's more information
https://askubuntu.com/questions/162229/how-do-i-increase-the-open-files-limi...
https://unix.stackexchange.com/questions/29577/ulimit-difference-between-har...
hope that helps ... cheers, ub
Liam
Pd-list@lists.iem.at mailing list UNSUBSCRIBE and account-management -> https://lists.puredata.info/listinfo/pd-list
Pd-list@lists.iem.at mailing list UNSUBSCRIBE and account-management -> https://lists.puredata.info/listinfo/pd-list
On 21/05/18 19:35, ub@xdv wrote:
hello,
On 20.05.2018 06:50, Liam Goodacre wrote:
In 0.48.1 on Ubuntu, I'm getting a horrible scenario where PD refuses to open patches or create any more abstractions for me. I get an error message saying "too many open files". Granted I have a lot open, but this is a serious problem as it means I can't access all of my old performances. They worked fine in 0.47.
Any ideas?
this is not really a problem with pd
I disagree. The most common cause of "too many open files" is a bug in closing files properly, because 1024 simultaneously-open files should be enough for most use cases.
Claude
, but with your shell- or system configuration limiting the number of open file descriptors.
check your current shell-limit with ulimit -Sn -S is for soft limit (you can lower, but not raise, the hard limit). raise the limit in your shell with ulimit -n 65536 start pd from that shell and see if the situation improves.
if that doesn't help, you can check the kernel limits with cat /proc/sys/fs/file-nr which returns 3 numbers, the first of which is the number of open files, the last of which is the limit.
increase the value of "nofile" in /etc/security/limits.conf do sudo sysctl -p
additional steps are required to make these settings permanent, the ulimit -n 65536 would have to go into your .bashrc so it's executed at system startup. if you normally start pd from your GUI you'd have to restart your system (actually just xorg) so your master shell knows about the new value. there's other ways, one of which would be to start pd from a wrapper script, or probably gnome provides that sort of environmental setup for it's program shortcuts.
there's more information
https://askubuntu.com/questions/162229/how-do-i-increase-the-open-files-limi...
https://unix.stackexchange.com/questions/29577/ulimit-difference-between-har...
hope that helps ... cheers, ub
Liam
Pd-list@lists.iem.at mailing list UNSUBSCRIBE and account-management -> https://lists.puredata.info/listinfo/pd-list
Pd-list@lists.iem.at mailing list UNSUBSCRIBE and account-management -> https://lists.puredata.info/listinfo/pd-list
Yes, it woulds like there's a bug in Pd... is it possible to reduce your patch to a reasonably simple test case that causes the problem?
thanks Miller
On Mon, May 21, 2018 at 07:54:23PM +0100, Claude Heiland-Allen wrote:
On 21/05/18 19:35, ub@xdv wrote:
hello,
On 20.05.2018 06:50, Liam Goodacre wrote:
In 0.48.1 on Ubuntu, I'm getting a horrible scenario where PD refuses to open patches or create any more abstractions for me. I get an error message saying "too many open files". Granted I have a lot open, but this is a serious problem as it means I can't access all of my old performances. They worked fine in 0.47.
Any ideas?
this is not really a problem with pd
I disagree. The most common cause of "too many open files" is a bug in closing files properly, because 1024 simultaneously-open files should be enough for most use cases.
Claude
, but with your shell- or system configuration limiting the number of open file descriptors.
check your current shell-limit with ulimit -Sn -S is for soft limit (you can lower, but not raise, the hard limit). raise the limit in your shell with ulimit -n 65536 start pd from that shell and see if the situation improves.
if that doesn't help, you can check the kernel limits with cat /proc/sys/fs/file-nr which returns 3 numbers, the first of which is the number of open files, the last of which is the limit.
increase the value of "nofile" in /etc/security/limits.conf do sudo sysctl -p
additional steps are required to make these settings permanent, the ulimit -n 65536 would have to go into your .bashrc so it's executed at system startup. if you normally start pd from your GUI you'd have to restart your system (actually just xorg) so your master shell knows about the new value. there's other ways, one of which would be to start pd from a wrapper script, or probably gnome provides that sort of environmental setup for it's program shortcuts.
there's more information
https://askubuntu.com/questions/162229/how-do-i-increase-the-open-files-limi...
https://unix.stackexchange.com/questions/29577/ulimit-difference-between-har...
hope that helps ... cheers, ub
Liam
Pd-list@lists.iem.at mailing list UNSUBSCRIBE and account-management -> https://lists.puredata.info/listinfo/pd-list
Pd-list@lists.iem.at mailing list UNSUBSCRIBE and account-management -> https://lists.puredata.info/listinfo/pd-list
Pd-list@lists.iem.at mailing list UNSUBSCRIBE and account-management -> https://lists.puredata.info/listinfo/pd-list
On 21.05.2018 20:54, Claude Heiland-Allen wrote:
On 21/05/18 19:35, ub@xdv wrote:
hello,
On 20.05.2018 06:50, Liam Goodacre wrote:
In 0.48.1 on Ubuntu, I'm getting a horrible scenario where PD refuses to open patches or create any more abstractions for me. I get an error message saying "too many open files". Granted I have a lot open, but this is a serious problem as it means I can't access all of my old performances. They worked fine in 0.47.
Any ideas?
this is not really a problem with pd
I disagree. The most common cause of "too many open files" is a bug in closing files properly,
possible. never occured to me. is it in the issue tracker?
because 1024 simultaneously-open files should be enough for most use cases.definitely, but when someone says "Granted I have a lot open", that
could be an understatement. or a regression of some sort.
Claude
, but with your shell- or system configuration limiting the number of open file descriptors.
check your current shell-limit with ulimit -Sn -S is for soft limit (you can lower, but not raise, the hard limit). raise the limit in your shell with ulimit -n 65536 start pd from that shell and see if the situation improves.
if that doesn't help, you can check the kernel limits with cat /proc/sys/fs/file-nr which returns 3 numbers, the first of which is the number of open files, the last of which is the limit.
increase the value of "nofile" in /etc/security/limits.conf do sudo sysctl -p
additional steps are required to make these settings permanent, the ulimit -n 65536 would have to go into your .bashrc so it's executed at system startup. if you normally start pd from your GUI you'd have to restart your system (actually just xorg) so your master shell knows about the new value. there's other ways, one of which would be to start pd from a wrapper script, or probably gnome provides that sort of environmental setup for it's program shortcuts.
there's more information
https://askubuntu.com/questions/162229/how-do-i-increase-the-open-files-limi...
https://unix.stackexchange.com/questions/29577/ulimit-difference-between-har...
hope that helps ... cheers, ub
Liam
Pd-list@lists.iem.at mailing list UNSUBSCRIBE and account-management -> https://lists.puredata.info/listinfo/pd-list
Pd-list@lists.iem.at mailing list UNSUBSCRIBE and account-management -> https://lists.puredata.info/listinfo/pd-list
On 05/21/2018 09:37 PM, ub@xdv wrote:
On 21.05.2018 20:54, Claude Heiland-Allen wrote:
On 21/05/18 19:35, ub@xdv wrote:
hello,
On 20.05.2018 06:50, Liam Goodacre wrote:
In 0.48.1 on Ubuntu, I'm getting a horrible scenario where PD refuses to open patches or create any more abstractions for me. I get an error message saying "too many open files". Granted I have a lot open, but this is a serious problem as it means I can't access all of my old performances. They worked fine in 0.47.
Any ideas?
this is not really a problem with pd
I disagree. The most common cause of "too many open files" is a bug in closing files properly,
intuitively i would agree.
possible. never occured to me. is it in the issue tracker?
no (afaik). it hasn't been reported before.
because 1024 simultaneously-open files should be enough for most use cases.definitely, but when someone says "Granted I have a lot open", that
could be an understatement. or a regression of some sort.
so i did a test run, with some heavily embedded abstraction, that is loaded a total of 10000 times. $ ulimit -Sn 1024 $ valgrind --track-fds=yes pd -noprefs -oss -nosound -nomidi -nrt -stderr -open test.pd [...] ==29030== FILE DESCRIPTORS: 3 open at exit. ==29030== Open file descriptor 2: /dev/pts/12 ==29030== <inherited from parent> ==29030== ==29030== Open file descriptor 1: /dev/pts/12 ==29030== <inherited from parent> ==29030== ==29030== Open file descriptor 0: /dev/pts/12 ==29030== <inherited from parent> ==29030== [...] $
so there seems to be neither a temporary file-handle leak (because my 10000 abstractions load just fine) nor an absolute file-handle leak (since the only handles that are left open by the program are stdin, stdout, stderr)
so i guess there must be something wrong with the patch.
@liam are there any externals in use? did you upgrade the entire system or just Pd? when using an older version of Pd, does the problem persist?
gfmtdsar IOhannes
Claude
, but with your shell- or system configuration limiting the number of open file descriptors.
check your current shell-limit with ulimit -Sn -S is for soft limit (you can lower, but not raise, the hard limit). raise the limit in your shell with ulimit -n 65536 start pd from that shell and see if the situation improves.
if that doesn't help, you can check the kernel limits with cat /proc/sys/fs/file-nr which returns 3 numbers, the first of which is the number of open files, the last of which is the limit.
increase the value of "nofile" in /etc/security/limits.conf do sudo sysctl -p
additional steps are required to make these settings permanent, the ulimit -n 65536 would have to go into your .bashrc so it's executed at system startup. if you normally start pd from your GUI you'd have to restart your system (actually just xorg) so your master shell knows about the new value. there's other ways, one of which would be to start pd from a wrapper script, or probably gnome provides that sort of environmental setup for it's program shortcuts.
there's more information
https://askubuntu.com/questions/162229/how-do-i-increase-the-open-files-limi...
https://unix.stackexchange.com/questions/29577/ulimit-difference-between-har...
hope that helps ... cheers, ub
Liam
Pd-list@lists.iem.at mailing list UNSUBSCRIBE and account-management -> https://lists.puredata.info/listinfo/pd-list
Pd-list@lists.iem.at mailing list UNSUBSCRIBE and account-management -> https://lists.puredata.info/listinfo/pd-list
Pd-list@lists.iem.at mailing list UNSUBSCRIBE and account-management -> https://lists.puredata.info/listinfo/pd-list
Thank you for the input. It will be a few days before I can do any serious testing, but I can answer some of your questions now.
IOhannes: yes, I upgraded PD, my externals, and my OS, all at the same time. I'll try it out with various different versions of the same and let you know what happens. But just as a heads-up, it's quite possible that I am trying to load more than 10,000 abstractions at once. Quite difficult to count them though!
If neither the externals nor the OS turn out to be the problem, I'll try to come up with a simple test case and get back to you. ________________________________ From: Pd-list pd-list-bounces@lists.iem.at on behalf of IOhannes m zm?lnig zmoelnig@iem.at Sent: 21 May 2018 20:50 To: pd-list@lists.iem.at Subject: Re: [PD] "too many open files" error in 0.48.1
On 05/21/2018 09:37 PM, ub@xdv wrote:
On 21.05.2018 20:54, Claude Heiland-Allen wrote:
On 21/05/18 19:35, ub@xdv wrote:
hello,
On 20.05.2018 06:50, Liam Goodacre wrote:
In 0.48.1 on Ubuntu, I'm getting a horrible scenario where PD refuses to open patches or create any more abstractions for me. I get an error message saying "too many open files". Granted I have a lot open, but this is a serious problem as it means I can't access all of my old performances. They worked fine in 0.47.
Any ideas?
this is not really a problem with pd
I disagree. The most common cause of "too many open files" is a bug in closing files properly,
intuitively i would agree.
possible. never occured to me. is it in the issue tracker?
no (afaik). it hasn't been reported before.
because 1024 simultaneously-open files should be enough for most use cases.definitely, but when someone says "Granted I have a lot open", that
could be an understatement. or a regression of some sort.
so i did a test run, with some heavily embedded abstraction, that is loaded a total of 10000 times. $ ulimit -Sn 1024 $ valgrind --track-fds=yes pd -noprefs -oss -nosound -nomidi -nrt -stderr -open test.pd [...] ==29030== FILE DESCRIPTORS: 3 open at exit. ==29030== Open file descriptor 2: /dev/pts/12 ==29030== <inherited from parent> ==29030== ==29030== Open file descriptor 1: /dev/pts/12 ==29030== <inherited from parent> ==29030== ==29030== Open file descriptor 0: /dev/pts/12 ==29030== <inherited from parent> ==29030== [...] $
so there seems to be neither a temporary file-handle leak (because my 10000 abstractions load just fine) nor an absolute file-handle leak (since the only handles that are left open by the program are stdin, stdout, stderr)
so i guess there must be something wrong with the patch.
@liam are there any externals in use? did you upgrade the entire system or just Pd? when using an older version of Pd, does the problem persist?
gfmtdsar IOhannes
Claude
, but with your shell- or system configuration limiting the number of open file descriptors.
check your current shell-limit with ulimit -Sn -S is for soft limit (you can lower, but not raise, the hard limit). raise the limit in your shell with ulimit -n 65536 start pd from that shell and see if the situation improves.
if that doesn't help, you can check the kernel limits with cat /proc/sys/fs/file-nr which returns 3 numbers, the first of which is the number of open files, the last of which is the limit.
increase the value of "nofile" in /etc/security/limits.conf do sudo sysctl -p
additional steps are required to make these settings permanent, the ulimit -n 65536 would have to go into your .bashrc so it's executed at system startup. if you normally start pd from your GUI you'd have to restart your system (actually just xorg) so your master shell knows about the new value. there's other ways, one of which would be to start pd from a wrapper script, or probably gnome provides that sort of environmental setup for it's program shortcuts.
there's more information
https://askubuntu.com/questions/162229/how-do-i-increase-the-open-files-limi...
https://unix.stackexchange.com/questions/29577/ulimit-difference-between-har...
hope that helps ... cheers, ub
Liam
Pd-list@lists.iem.at mailing list UNSUBSCRIBE and account-management -> https://lists.puredata.info/listinfo/pd-list
Pd-list@lists.iem.at mailing list UNSUBSCRIBE and account-management -> https://lists.puredata.info/listinfo/pd-list
Pd-list@lists.iem.at mailing list UNSUBSCRIBE and account-management -> https://lists.puredata.info/listinfo/pd-list
OK, I've just spent several hours on this, testing upwards of 200,000 parallel abstractions and crashing my computer almost as many times. But I managed to figure it out.
(Most of) you will be relieved to hear that the problem does not originate with the PD core, but with externals, as IOhannes suggested. Alex will not be relieved to hear that the culprit is [else/dir]. It seems that the 1019th instance of this object crashes PD! How strange.
I'm filing a bug report on git and moving on with my life.
Liam
From: Pd-list pd-list-bounces@lists.iem.at on behalf of Liam Goodacre liamg_uw@hotmail.com Sent: 22 May 2018 05:36 To: pd-list@lists.iem.at Subject: Re: [PD] "too many open files" error in 0.48.1
Thank you for the input. It will be a few days before I can do any serious testing, but I can answer some of your questions now.
IOhannes: yes, I upgraded PD, my externals, and my OS, all at the same time. I'll try it out with various different versions of the same and let you know what happens. But just as a heads-up, it's quite possible that I am trying to load more than 10,000 abstractions at once. Quite difficult to count them though!
If neither the externals nor the OS turn out to be the problem, I'll try to come up with a simple test case and get back to you. ________________________________ From: Pd-list pd-list-bounces@lists.iem.at on behalf of IOhannes m zmölnig zmoelnig@iem.at Sent: 21 May 2018 20:50 To: pd-list@lists.iem.at Subject: Re: [PD] "too many open files" error in 0.48.1
On 05/21/2018 09:37 PM, ub@xdv wrote:
On 21.05.2018 20:54, Claude Heiland-Allen wrote:
On 21/05/18 19:35, ub@xdv wrote:
hello,
On 20.05.2018 06:50, Liam Goodacre wrote:
In 0.48.1 on Ubuntu, I'm getting a horrible scenario where PD refuses to open patches or create any more abstractions for me. I get an error message saying "too many open files". Granted I have a lot open, but this is a serious problem as it means I can't access all of my old performances. They worked fine in 0.47.
Any ideas?
this is not really a problem with pd
I disagree. The most common cause of "too many open files" is a bug in closing files properly,
intuitively i would agree.
possible. never occured to me. is it in the issue tracker?
no (afaik). it hasn't been reported before.
because 1024 simultaneously-open files should be enough for most use cases.definitely, but when someone says "Granted I have a lot open", that
could be an understatement. or a regression of some sort.
so i did a test run, with some heavily embedded abstraction, that is loaded a total of 10000 times. $ ulimit -Sn 1024 $ valgrind --track-fds=yes pd -noprefs -oss -nosound -nomidi -nrt -stderr -open test.pd [...] ==29030== FILE DESCRIPTORS: 3 open at exit. ==29030== Open file descriptor 2: /dev/pts/12 ==29030== <inherited from parent> ==29030== ==29030== Open file descriptor 1: /dev/pts/12 ==29030== <inherited from parent> ==29030== ==29030== Open file descriptor 0: /dev/pts/12 ==29030== <inherited from parent> ==29030== [...] $
so there seems to be neither a temporary file-handle leak (because my 10000 abstractions load just fine) nor an absolute file-handle leak (since the only handles that are left open by the program are stdin, stdout, stderr)
so i guess there must be something wrong with the patch.
@liam are there any externals in use? did you upgrade the entire system or just Pd? when using an older version of Pd, does the problem persist?
gfmtdsar IOhannes
Claude
, but with your shell- or system configuration limiting the number of open file descriptors.
check your current shell-limit with ulimit -Sn -S is for soft limit (you can lower, but not raise, the hard limit). raise the limit in your shell with ulimit -n 65536 start pd from that shell and see if the situation improves.
if that doesn't help, you can check the kernel limits with cat /proc/sys/fs/file-nr which returns 3 numbers, the first of which is the number of open files, the last of which is the limit.
increase the value of "nofile" in /etc/security/limits.conf do sudo sysctl -p
additional steps are required to make these settings permanent, the ulimit -n 65536 would have to go into your .bashrc so it's executed at system startup. if you normally start pd from your GUI you'd have to restart your system (actually just xorg) so your master shell knows about the new value. there's other ways, one of which would be to start pd from a wrapper script, or probably gnome provides that sort of environmental setup for it's program shortcuts.
there's more information
https://askubuntu.com/questions/162229/how-do-i-increase-the-open-files-limi...
https://unix.stackexchange.com/questions/29577/ulimit-difference-between-har...
hope that helps ... cheers, ub
Liam
Pd-list@lists.iem.at mailing list UNSUBSCRIBE and account-management -> https://lists.puredata.info/listinfo/pd-list
Pd-list@lists.iem.at mailing list UNSUBSCRIBE and account-management -> https://lists.puredata.info/listinfo/pd-list
Pd-list@lists.iem.at mailing list UNSUBSCRIBE and account-management -> https://lists.puredata.info/listinfo/pd-list
Hey all
Not only by creating 1020 instances of [else/dir] you can crash Pd, but also by sending it a 337 'reset, open .' messages. I guess the reason is the same: [else/dir] is leaking file handles.
It appears I tried [else/dir] for the similar reasons as you, Liam. I am also using [ggee/getdir] and [moocow/readdir] now and thought about reducing dependencies by switching to else. It turns out I can't switch yet.
Roman
On Sun, 2018-05-27 at 19:26 +0000, Liam Goodacre wrote:
OK, I've just spent several hours on this, testing upwards of 200,000 parallel abstractions and crashing my computer almost as many times. But I managed to figure it out.
(Most of) you will be relieved to hear that the problem does not originate with the PD core, but with externals, as IOhannes suggested. Alex will not be relieved to hear that the culprit is [else/dir]. It seems that the 1019th instance of this object crashes PD! How strange.
I'm filing a bug report on git and moving on with my life.
Liam
From: Pd-list pd-list-bounces@lists.iem.at on behalf of Liam Goodacre liamg_uw@hotmail.com Sent: 22 May 2018 05:36 To: pd-list@lists.iem.at Subject: Re: [PD] "too many open files" error in 0.48.1
Thank you for the input. It will be a few days before I can do any serious testing, but I can answer some of your questions now.
IOhannes: yes, I upgraded PD, my externals, and my OS, all at the same time. I'll try it out with various different versions of the same and let you know what happens. But just as a heads-up, it's quite possible that I am trying to load more than 10,000 abstractions at once. Quite difficult to count them though!
If neither the externals nor the OS turn out to be the problem, I'll try to come up with a simple test case and get back to you. From: Pd-list pd-list-bounces@lists.iem.at on behalf of IOhannes m zmölnig zmoelnig@iem.at Sent: 21 May 2018 20:50 To: pd-list@lists.iem.at Subject: Re: [PD] "too many open files" error in 0.48.1
On 05/21/2018 09:37 PM, ub@xdv wrote:
On 21.05.2018 20:54, Claude Heiland-Allen wrote:
On 21/05/18 19:35, ub@xdv wrote:
hello,
On 20.05.2018 06:50, Liam Goodacre wrote:
In 0.48.1 on Ubuntu, I'm getting a horrible scenario where PD
refuses to
open patches or create any more abstractions for me. I get an
error
message saying "too many open files". Granted I have a lot open,
but
this is a serious problem as it means I can't access all of my
old
performances. They worked fine in 0.47.
Any ideas?
this is not really a problem with pd
I disagree. The most common cause of "too many open files" is a
bug in
closing files properly,
intuitively i would agree.
possible. never occured to me. is it in the issue tracker?
no (afaik). it hasn't been reported before.
because 1024 simultaneously-open files should be enough for most use cases.definitely, but when someone says
"Granted I have a lot open", that
could be an understatement. or a regression of some sort.
so i did a test run, with some heavily embedded abstraction, that is loaded a total of 10000 times. $ ulimit -Sn 1024 $ valgrind --track-fds=yes pd -noprefs -oss -nosound -nomidi -nrt -stderr -open test.pd [...] ==29030== FILE DESCRIPTORS: 3 open at exit. ==29030== Open file descriptor 2: /dev/pts/12 ==29030== <inherited from parent> ==29030== ==29030== Open file descriptor 1: /dev/pts/12 ==29030== <inherited from parent> ==29030== ==29030== Open file descriptor 0: /dev/pts/12 ==29030== <inherited from parent> ==29030== [...] $
so there seems to be neither a temporary file-handle leak (because my 10000 abstractions load just fine) nor an absolute file-handle leak (since the only handles that are left open by the program are stdin, stdout, stderr)
so i guess there must be something wrong with the patch.
@liam are there any externals in use? did you upgrade the entire system or just Pd? when using an older version of Pd, does the problem persist?
gfmtdsar IOhannes
Claude
, but with your shell- or system configuration limiting the number of open file descriptors.
check your current shell-limit with ulimit -Sn -S is for soft limit (you can lower, but not raise, the hard
limit).
raise the limit in your shell with ulimit -n 65536 start pd from that shell and see if the situation improves.
if that doesn't help, you can check the kernel limits with cat /proc/sys/fs/file-nr which returns 3 numbers, the first of which is the number of open
files,
the last of which is the limit.
increase the value of "nofile" in /etc/security/limits.conf do sudo sysctl -p
additional steps are required to make these settings permanent,
the
ulimit -n 65536 would have to go into your .bashrc so it's
executed at
system startup. if you normally start pd from your GUI you'd have
to
restart your system (actually just xorg) so your master shell
knows
about the new value. there's other ways, one of which would be to
start
pd from a wrapper script, or probably gnome provides that sort of environmental setup for it's program shortcuts.
there's more information
https://askubuntu.com/questions/162229/how-do-i-increase-the-open
-files-limit-for-a-non-root-user
https://unix.stackexchange.com/questions/29577/ulimit-difference-
between-hard-and-soft-limits
hope that helps ... cheers, ub
Liam
Pd-list@lists.iem.at mailing list UNSUBSCRIBE and account-management -> https://lists.puredata.info/listinfo/pd-list
Pd-list@lists.iem.at mailing list UNSUBSCRIBE and account-management -> https://lists.puredata.info/listinfo/pd-list
Pd-list@lists.iem.at mailing list UNSUBSCRIBE and account-management -> https://lists.puredata.info/l
istinfo/pd-list
Pd-list@lists.iem.at mailing list UNSUBSCRIBE and account-management -> https://lists.puredata.info/lis tinfo/pd-list
Sorry, I thought I had fixed this in the last update. I swear it was working for me, but then... that reported bug came back :) The object still works fine for me in my use cases. Anyway, I'll give it another go in the next update. If I fail miserably I'll ask for help. I just have other plans for the moment though...
Cheers
2018-07-23 19:09 GMT-03:00 Roman Haefeli reduzent@gmail.com:
Hey all
Not only by creating 1020 instances of [else/dir] you can crash Pd, but also by sending it a 337 'reset, open .' messages. I guess the reason is the same: [else/dir] is leaking file handles.
It appears I tried [else/dir] for the similar reasons as you, Liam. I am also using [ggee/getdir] and [moocow/readdir] now and thought about reducing dependencies by switching to else. It turns out I can't switch yet.
Roman
On Sun, 2018-05-27 at 19:26 +0000, Liam Goodacre wrote:
OK, I've just spent several hours on this, testing upwards of 200,000 parallel abstractions and crashing my computer almost as many times. But I managed to figure it out.
(Most of) you will be relieved to hear that the problem does not originate with the PD core, but with externals, as IOhannes suggested. Alex will not be relieved to hear that the culprit is [else/dir]. It seems that the 1019th instance of this object crashes PD! How strange.
I'm filing a bug report on git and moving on with my life.
Liam
From: Pd-list pd-list-bounces@lists.iem.at on behalf of Liam Goodacre liamg_uw@hotmail.com Sent: 22 May 2018 05:36 To: pd-list@lists.iem.at Subject: Re: [PD] "too many open files" error in 0.48.1
Thank you for the input. It will be a few days before I can do any serious testing, but I can answer some of your questions now.
IOhannes: yes, I upgraded PD, my externals, and my OS, all at the same time. I'll try it out with various different versions of the same and let you know what happens. But just as a heads-up, it's quite possible that I am trying to load more than 10,000 abstractions at once. Quite difficult to count them though!
If neither the externals nor the OS turn out to be the problem, I'll try to come up with a simple test case and get back to you. From: Pd-list pd-list-bounces@lists.iem.at on behalf of IOhannes m zmölnig zmoelnig@iem.at Sent: 21 May 2018 20:50 To: pd-list@lists.iem.at Subject: Re: [PD] "too many open files" error in 0.48.1
On 05/21/2018 09:37 PM, ub@xdv wrote:
On 21.05.2018 20:54, Claude Heiland-Allen wrote:
On 21/05/18 19:35, ub@xdv wrote:
hello,
On 20.05.2018 06:50, Liam Goodacre wrote:
In 0.48.1 on Ubuntu, I'm getting a horrible scenario where PD
refuses to
open patches or create any more abstractions for me. I get an
error
message saying "too many open files". Granted I have a lot open,
but
this is a serious problem as it means I can't access all of my
old
performances. They worked fine in 0.47.
Any ideas?
this is not really a problem with pd
I disagree. The most common cause of "too many open files" is a
bug in
closing files properly,
intuitively i would agree.
possible. never occured to me. is it in the issue tracker?
no (afaik). it hasn't been reported before.
because 1024 simultaneously-open files should be enough for most use cases.definitely, but when someone says
"Granted I have a lot open", that
could be an understatement. or a regression of some sort.
so i did a test run, with some heavily embedded abstraction, that is loaded a total of 10000 times. $ ulimit -Sn 1024 $ valgrind --track-fds=yes pd -noprefs -oss -nosound -nomidi -nrt -stderr -open test.pd [...] ==29030== FILE DESCRIPTORS: 3 open at exit. ==29030== Open file descriptor 2: /dev/pts/12 ==29030== <inherited from parent> ==29030== ==29030== Open file descriptor 1: /dev/pts/12 ==29030== <inherited from parent> ==29030== ==29030== Open file descriptor 0: /dev/pts/12 ==29030== <inherited from parent> ==29030== [...] $
so there seems to be neither a temporary file-handle leak (because my 10000 abstractions load just fine) nor an absolute file-handle leak (since the only handles that are left open by the program are stdin, stdout, stderr)
so i guess there must be something wrong with the patch.
@liam are there any externals in use? did you upgrade the entire system or just Pd? when using an older version of Pd, does the problem persist?
gfmtdsar IOhannes
Claude
, but with your shell- or system configuration limiting the number of open file descriptors.
check your current shell-limit with ulimit -Sn -S is for soft limit (you can lower, but not raise, the hard
limit).
raise the limit in your shell with ulimit -n 65536 start pd from that shell and see if the situation improves.
if that doesn't help, you can check the kernel limits with cat /proc/sys/fs/file-nr which returns 3 numbers, the first of which is the number of open
files,
the last of which is the limit.
increase the value of "nofile" in /etc/security/limits.conf do sudo sysctl -p
additional steps are required to make these settings permanent,
the
ulimit -n 65536 would have to go into your .bashrc so it's
executed at
system startup. if you normally start pd from your GUI you'd have
to
restart your system (actually just xorg) so your master shell
knows
about the new value. there's other ways, one of which would be to
start
pd from a wrapper script, or probably gnome provides that sort of environmental setup for it's program shortcuts.
there's more information
https://askubuntu.com/questions/162229/how-do-i-increase-the-open
-files-limit-for-a-non-root-user
https://unix.stackexchange.com/questions/29577/ulimit-difference-
between-hard-and-soft-limits
hope that helps ... cheers, ub
Liam
Pd-list@lists.iem.at mailing list UNSUBSCRIBE and account-management -> https://lists.puredata.info/listinfo/pd-list
Pd-list@lists.iem.at mailing list UNSUBSCRIBE and account-management -> https://lists.puredata.info/listinfo/pd-list
Pd-list@lists.iem.at mailing list UNSUBSCRIBE and account-management -> https://lists.puredata.info/l
istinfo/pd-list
Pd-list@lists.iem.at mailing list UNSUBSCRIBE and account-management -> https://lists.puredata.info/lis tinfo/pd-list
Pd-list@lists.iem.at mailing list UNSUBSCRIBE and account-management -> https://lists.puredata.info/ listinfo/pd-list
Hey Alex
On Tue, 2018-07-24 at 12:44 -0300, Alexandre Torres Porres wrote:
Sorry, I thought I had fixed this in the last update. I swear it was working for me, but then... that reported bug came back :)
No need to be sorry. I'm not complaining, but simply reporting.
The object still works fine for me in my use cases.
I figured it is still easy for me to work-around the problem (actually, I don't need to open the same two directories again and again, I could just use two instances of [dir] and open each of those directories once). However, stuff like that surely is going to bite you back in the future. It's certainly better to fix it anyway regardless of any specific use case.
Anyway, I'll give it another go in the next update. If I fail miserably I'll ask for help.
I don't know if it works the same on macOS, but on Linux I can check what is going on with the lsof command, specifically 'lsof -c pd' which lists all files openend by processes named pd. Each time I send 'open /home/roman/Downloads' to [dir], I see two additional lines in lsof's output:
pd 9120 roman 23r DIR 0,54 28672 29098010 /home/roman/Downloads pd 9120 roman 24r DIR 0,54 28672 29098010 /home/roman/Downloads
I just have other plans for the moment though...
Sure. Thanks for the library anyway.
Roman
On Tue, 2018-07-24 at 12:44 -0300, Alexandre Torres Porres wrote:
Sorry, I thought I had fixed this in the last update. I swear it was working for me, but then... that reported bug came back :) The object still works fine for me in my use cases. Anyway, I'll give it another go in the next update.
My understanding is that calling opendir() opens a file descriptor that is only closed after calling closedir() on that file descriptor. As far as I can see, [dir] never calls closedir() and thus it accumulates open file descriptors. So, I guess what you need to do is to close the current file descriptor each time before processing 'reset' or 'open' methods which I believe are the two methods that call opendir().
Roman
-
Alexandre Torres Porres -
Claude Heiland-Allen -
IOhannes m zmölnig -
Liam Goodacre -
Miller Puckette -
Roman Haefeli -
ub＠xdv