Hi all
While developing on and playing with netpd2, Pd sometimes crashes. This hadn't happened so frequently until recently, so I didn't investigate it further. However, with running more and bigger patches I seem to experience those crashes more often and I started running pd inside gdb to get some more information.
I haven't yet figured a way to reliably trigger a crash. It seems to happen at random times. However, it seems to me that it only happens when I touch a GUI element (sliders, numberboxes, arrays, etc), but this doesn't necessarily is the cause, as most interactions trigger a message sent to the network.
From the many backtraces I collected, most of them look very similar:
--- Program received signal SIGSEGV, Segmentation fault. 0x080a1fe5 in clock_unset (x=0x83c03f0) at m_sched.c:70 70 while (x2->c_next != x) x2 = x2->c_next;
#0 0x080a1fe5 in clock_unset (x=0x83c03f0) at m_sched.c:70 #1 0x080a2040 in clock_set (x=0x83c03f0, setticks=5047173120) at m_sched.c:81 #2 0x080a2109 in clock_delay (x=0x83c03f0, delaytime=0) at m_sched.c:103 #3 0x080b9fb3 in bang_tilde_perform (w=0x885abd0) at d_misc.c:90 #4 0x080af8b2 in dsp_tick () at d_ugen.c:336 #5 0x080a298d in sched_tick (next_sys_time=5047173120) at m_sched.c:382 #6 0x080a2b7e in m_pollingscheduler () at m_sched.c:482 #7 0x080a2d67 in m_mainloop () at m_sched.c:568 #8 0x080a347a in sys_main (argc=9, argv=0xbffff3c4) at s_main.c:302 #9 0x080ab147 in main (argc=9, argv=0xbffff3c4) at s_entry.c:32 ---
or another very similar one:
--- Program received signal SIGSEGV, Segmentation fault. 0x080a1fe5 in clock_unset (x=0x83d1660) at m_sched.c:70 70 while (x2->c_next != x) x2 = x2->c_next;
#0 0x080a1fe5 in clock_unset (x=0x83d1660) at m_sched.c:70 #1 0x080a2040 in clock_set (x=0x83d1660, setticks=6376652800) at m_sched.c:81 #2 0x080a2109 in clock_delay (x=0x83d1660, delaytime=0) at m_sched.c:103 #3 0x080b9fb3 in bang_tilde_perform (w=0x88f6c68) at d_misc.c:90 #4 0x080af8b2 in dsp_tick () at d_ugen.c:336 #5 0x080a298d in sched_tick (next_sys_time=6376652800) at m_sched.c:382 #6 0x080a2b7e in m_pollingscheduler () at m_sched.c:482 #7 0x080a2d67 in m_mainloop () at m_sched.c:568 #8 0x080a347a in sys_main (argc=9, argv=0xbffff3c4) at s_main.c:302 #9 0x080ab147 in main (argc=9, argv=0xbffff3c4) at s_entry.c:32 ---
I have many more of this kind, but I think it doesn't help to post them here.
In a few other cases, the network transmission suddenly stops, though Pd keeps running. When I then send 'disconnect' to [iemnet/tcpclient], Pd segfaults and the backtrace looks like this:
--- Program received signal SIGSEGV, Segmentation fault. 0x080a1fe5 in clock_unset (x=0x8211148) at m_sched.c:70 70 while (x2->c_next != x) x2 = x2->c_next;
#0 0x080a1fe5 in clock_unset (x=0x8211148) at m_sched.c:70 #1 0x080a217c in clock_free (x=0x8211148) at m_sched.c:131 #2 0x00508b79 in iemnet__receiver_destroy () from /usr/local/lib/pd/extra/iemnet/libiemnet.so #3 0x08211148 in ?? () Backtrace stopped: previous frame inner to this frame (corrupt stack?) ---
(What does the last line mean?)
What can I do to help to track down the reason for those crashes? Would using valgrind reveal more useful information here? Is it possible to tell from the backtraces, whether the cause is in Pd itself or in the externals used?
This is on/with: Pd 0.43.1 iemnet (current svn) zexy (2.2.6 current svn) osc (current svn mrpeach/osc) slipdec/slipenc (current svn mrpeach)
Roman
Le 2012-03-07 à 11:39:00, Roman Haefeli a écrit :
From the many backtraces I collected, most of them look very similar:
If the crashes vary and happen at seemingly random times, then there might be a single bug with different manifestations. If you patch takes a very low percentage of CPU, you may try running it with Valgrind, which might be able to find a cause for it, usually « Invalid Write », and sometimes might even be able to find a cause to the cause.
However, it's also possible that there are several bugs at once, who knows...
#2 0x00508b79 in iemnet__receiver_destroy () from /usr/local/lib/pd/extra/iemnet/libiemnet.so #3 0x08211148 in ?? () Backtrace stopped: previous frame inner to this frame (corrupt stack?) (What does the last line mean?)
Something blew the backtrace. If it happens at #0, then it's possible that running unwanted code causes the stack to not be findable, but at #3, it's more like the stack is findable but got damaged, possibly by writing past the end of a C array on the stack (non-malloc).
What can I do to help to track down the reason for those crashes? Would using valgrind reveal more useful information here?
If the crashes are random-looking, or otherwise mysterious, Valgrind is always a good guess.
Is it possible to tell from the backtraces, whether the cause is in Pd itself or in the externals used?
From the backtrace of GDB, you can't be super sure. This points at a
well-tested part of Pd. It would be a bit weird if it crashed here because of a bug in that part of pd, except if it happened in a test version of Pd that happened to have modifs in that area of the project.
______________________________________________________________________ | Mathieu BOUCHARD ----- téléphone : +1.514.383.3801 ----- Montréal, QC
(I somehow missed to reply to the list; sorry for double-posting, Matju)
Hi Matju
Thanks a lot for your expertise.
On Sat, 2012-03-10 at 23:07 -0500, Mathieu Bouchard wrote:
Le 2012-03-07 à 11:39:00, Roman Haefeli a écrit :
From the many backtraces I collected, most of them look very similar:
If the crashes vary and happen at seemingly random times, then there might be a single bug with different manifestations. If you patch takes a very low percentage of CPU, you may try running it with Valgrind, which might be able to find a cause for it, usually « Invalid Write », and sometimes might even be able to find a cause to the cause.
However, it's also possible that there are several bugs at once, who knows...
#2 0x00508b79 in iemnet__receiver_destroy () from /usr/local/lib/pd/extra/iemnet/libiemnet.so #3 0x08211148 in ?? () Backtrace stopped: previous frame inner to this frame (corrupt stack?) (What does the last line mean?)
Something blew the backtrace. If it happens at #0, then it's possible that running unwanted code causes the stack to not be findable, but at #3, it's more like the stack is findable but got damaged, possibly by writing past the end of a C array on the stack (non-malloc).
What can I do to help to track down the reason for those crashes? Would using valgrind reveal more useful information here?
If the crashes are random-looking, or otherwise mysterious, Valgrind is always a good guess.
Is it possible to tell from the backtraces, whether the cause is in Pd itself or in the externals used?
From the backtrace of GDB, you can't be super sure. This points at a well-tested part of Pd. It would be a bit weird if it crashed here because of a bug in that part of pd, except if it happened in a test version of Pd that happened to have modifs in that area of the project.
I tested with an unmodified Pd source from git. My problem is that I cannot exclude the externals used, as I need them for running the setup that triggers the crashes. And yes, the CPU load average is usually moderate when it happens, but there might be some spikes for a few milliseconds.
I'll try to get more info with valgrind.
Roman
Le 2012-03-13 à 15:25:00, Roman Haefeli a écrit :
I tested with an unmodified Pd source from git. My problem is that I cannot exclude the externals used, as I need them for running the setup that triggers the crashes.
It's fine to be running externals, and obviously, if they're necessary for the crash, they're necessary for the debugging too. But I was just talking about m_sched.c. It's possible to customise the scheduler, but there are no externals that do that (?).
______________________________________________________________________ | Mathieu BOUCHARD ----- téléphone : +1.514.383.3801 ----- Montréal, QC
Hi again
I was able to create a patch, that does not make Pd crash reliably, but far more often. On my box it crashes roughly every second time I run the patch.
This is how I run it: $ gdb -ex run --args pd-extended -noprefs -nrt -noaudio -stderr -open crashertest.pd
When the [bng] is hit, every 10ms an OSC message is sent over TCP. After a while (on my box usually only a few seconds), no more messages are received, though they are still sent. If [iemnet/tcpclient] is disconnected at this point, Pd often crashes with the following backtrace:
--- Program received signal SIGSEGV, Segmentation fault. 0x08086e42 in clock_unset (x=0x817f5b8) at m_sched.c:70 70 m_sched.c: No such file or directory. in m_sched.c (gdb) backtrace #0 0x08086e42 in clock_unset (x=0x817f5b8) at m_sched.c:70 #1 0x08086fb2 in clock_free (x=0x817f5b8) at m_sched.c:131 #2 0x00707ce0 in iemnet__receiver_destroy (rec=0x817f4c8) at iemnet_receiver.c:303 #3 0x007022e0 in tcpclient_disconnect (x=0x81042b8) at tcpclient.c:188 #4 0x0807f3f2 in pd_typedmess (x=0x81042b8, s=0x80f11e8, argc=<value optimized out>, argv=0xbffff1fc) at m_class.c:791 #5 0x080807fa in outlet_anything (x=0x8113e70, s=0x80f11e8, argc=0, argv=0xbffff1fc) at m_obj.c:470 #6 0x0807efe6 in pd_typedmess (x=0x8113e5c, s=0x80f11e8, argc=0, argv=0xbffff1fc) at m_class.c:812 #7 0x08085782 in binbuf_eval (x=0x8113e88, target=0x8113e5c, argc=0, argv=0x0) at m_binbuf.c:767 #8 0x0805fe2b in message_bang (x=0x8113e40) at g_text.c:351 #9 0x080804d3 in outlet_bang (x=0x8115bb0) at m_obj.c:399 #10 0x007137eb in delay_tick (x=0x8115b68) at delay.c:43 #11 0x08087414 in sched_tick (next_sys_time=49233920) at m_sched.c:372 #12 0x0808780b in m_pollingscheduler () at m_sched.c:485 #13 m_mainloop () at m_sched.c:571 #14 0x0808f14b in main (argc=7, argv=0xbffff3d4) at s_entry.c:32 ---
This is with Pd-0.43.1-extended-20120307
With Pd-vanilla and the necessary externals, I get the same kind of crash, but a different (corrupt) backtrace:
--- Program received signal SIGSEGV, Segmentation fault. 0x080a1fe5 in clock_unset (x=0x8140768) at m_sched.c:70 70 while (x2->c_next != x) x2 = x2->c_next; (gdb) backtrace #0 0x080a1fe5 in clock_unset (x=0x8140768) at m_sched.c:70 #1 0x080a217c in clock_free (x=0x8140768) at m_sched.c:131 #2 0x00501b79 in iemnet__receiver_destroy () from /usr/local/lib/pd/extra/iemnet/libiemnet.so #3 0x08140768 in ?? () Backtrace stopped: previous frame inner to this frame (corrupt stack?) ---
The fact, that [iemnet/tcpclient] suddenly stops transmitting data, indicates that there is some problem there, whether the crash is caused by it or not.
Roman
On Wed, 2012-03-07 at 11:39 +0100, Roman Haefeli wrote:
Hi all
While developing on and playing with netpd2, Pd sometimes crashes. This hadn't happened so frequently until recently, so I didn't investigate it further. However, with running more and bigger patches I seem to experience those crashes more often and I started running pd inside gdb to get some more information.
I haven't yet figured a way to reliably trigger a crash. It seems to happen at random times. However, it seems to me that it only happens when I touch a GUI element (sliders, numberboxes, arrays, etc), but this doesn't necessarily is the cause, as most interactions trigger a message sent to the network.
From the many backtraces I collected, most of them look very similar:
Program received signal SIGSEGV, Segmentation fault. 0x080a1fe5 in clock_unset (x=0x83c03f0) at m_sched.c:70 70 while (x2->c_next != x) x2 = x2->c_next;
#0 0x080a1fe5 in clock_unset (x=0x83c03f0) at m_sched.c:70 #1 0x080a2040 in clock_set (x=0x83c03f0, setticks=5047173120) at m_sched.c:81 #2 0x080a2109 in clock_delay (x=0x83c03f0, delaytime=0) at m_sched.c:103 #3 0x080b9fb3 in bang_tilde_perform (w=0x885abd0) at d_misc.c:90 #4 0x080af8b2 in dsp_tick () at d_ugen.c:336 #5 0x080a298d in sched_tick (next_sys_time=5047173120) at m_sched.c:382 #6 0x080a2b7e in m_pollingscheduler () at m_sched.c:482 #7 0x080a2d67 in m_mainloop () at m_sched.c:568 #8 0x080a347a in sys_main (argc=9, argv=0xbffff3c4) at s_main.c:302
#9 0x080ab147 in main (argc=9, argv=0xbffff3c4) at s_entry.c:32
or another very similar one:
Program received signal SIGSEGV, Segmentation fault. 0x080a1fe5 in clock_unset (x=0x83d1660) at m_sched.c:70 70 while (x2->c_next != x) x2 = x2->c_next;
#0 0x080a1fe5 in clock_unset (x=0x83d1660) at m_sched.c:70 #1 0x080a2040 in clock_set (x=0x83d1660, setticks=6376652800) at m_sched.c:81 #2 0x080a2109 in clock_delay (x=0x83d1660, delaytime=0) at m_sched.c:103 #3 0x080b9fb3 in bang_tilde_perform (w=0x88f6c68) at d_misc.c:90 #4 0x080af8b2 in dsp_tick () at d_ugen.c:336 #5 0x080a298d in sched_tick (next_sys_time=6376652800) at m_sched.c:382 #6 0x080a2b7e in m_pollingscheduler () at m_sched.c:482 #7 0x080a2d67 in m_mainloop () at m_sched.c:568 #8 0x080a347a in sys_main (argc=9, argv=0xbffff3c4) at s_main.c:302
#9 0x080ab147 in main (argc=9, argv=0xbffff3c4) at s_entry.c:32
I have many more of this kind, but I think it doesn't help to post them here.
In a few other cases, the network transmission suddenly stops, though Pd keeps running. When I then send 'disconnect' to [iemnet/tcpclient], Pd segfaults and the backtrace looks like this:
Program received signal SIGSEGV, Segmentation fault. 0x080a1fe5 in clock_unset (x=0x8211148) at m_sched.c:70 70 while (x2->c_next != x) x2 = x2->c_next;
#0 0x080a1fe5 in clock_unset (x=0x8211148) at m_sched.c:70 #1 0x080a217c in clock_free (x=0x8211148) at m_sched.c:131 #2 0x00508b79 in iemnet__receiver_destroy () from /usr/local/lib/pd/extra/iemnet/libiemnet.so #3 0x08211148 in ?? () Backtrace stopped: previous frame inner to this frame (corrupt stack?)
(What does the last line mean?)
What can I do to help to track down the reason for those crashes? Would using valgrind reveal more useful information here? Is it possible to tell from the backtraces, whether the cause is in Pd itself or in the externals used?
This is on/with: Pd 0.43.1 iemnet (current svn) zexy (2.2.6 current svn) osc (current svn mrpeach/osc) slipdec/slipenc (current svn mrpeach)
Roman
Le 2012-03-13 à 14:52:00, Roman Haefeli a écrit :
$ gdb -ex run --args pd-extended -noprefs -nrt -noaudio -stderr -open crashertest.pd
Where did you find -ex ? I had been making a lot of pirouettes to get gdb to run commands that I wanted, which were not very good. I wanted to have -ex all those years.
http://sourceware.org/bugzilla/show_bug.cgi?id=12756
______________________________________________________________________ | Mathieu BOUCHARD ----- téléphone : +1.514.383.3801 ----- Montréal, QC
On Tue, 2012-03-13 at 11:17 -0400, Mathieu Bouchard wrote:
Le 2012-03-13 à 14:52:00, Roman Haefeli a écrit :
$ gdb -ex run --args pd-extended -noprefs -nrt -noaudio -stderr -open crashertest.pd
Where did you find -ex ? I had been making a lot of pirouettes to get gdb to run commands that I wanted, which were not very good. I wanted to have -ex all those years.
I read the source code.
HAHAHAHAhahaha.......
Jokes aside, iirc, I googled for something like "gdb run immediately" and found this stackoverflow article: http://stackoverflow.com/questions/2119564/gdb-how-to-automatically-run-the-...
But yeah, the man page doesn't help here.
Roman
On Tue, 2012-03-13 at 14:52 +0100, Roman Haefeli wrote:
Hi again
I was able to create a patch, that does not make Pd crash reliably, but far more often. On my box it crashes roughly every second time I run the patch.
This is how I run it: $ gdb -ex run --args pd-extended -noprefs -nrt -noaudio -stderr -open crashertest.pd
When the [bng] is hit, every 10ms an OSC message is sent over TCP. After a while (on my box usually only a few seconds), no more messages are received, though they are still sent.
Actually, the opposite is the case. [iemnet/tcpclient] stops sending, but would still receive something, if [iemnet/tcpserver] would send something. Can someone observe similar behaviour or is the patch happily running on other people's machine?
I tried running it with valgrind, but then Pd doesn't crash anymore. Could it be, that the overhead added by valgrind makes Pd so slow, that it doesn't trigger anymore?
I'm kind of clueless how to track that down further. I know of a few situations in netpd, that are likely to crash Pd, but they are never reliable. I don't even know, whether the different backtraces of those crashes are due to the same cause. When I try to isolate parts, the crashes disappear. This is quite a show stopper.
Roman
Le 2012-03-15 à 14:34:00, Roman Haefeli a écrit :
I tried running it with valgrind, but then Pd doesn't crash anymore. Could it be, that the overhead added by valgrind makes Pd so slow, that it doesn't trigger anymore?
Sometimes it's because the debugged programme becomes slower ; but sometimes it's that the debugged programme has a different thread-switching pattern that causes it to accidentally avoid the problem ; and sometimes it's because valgrind's malloc-patterns causes it to accidentally avoid the problem ; and so on. This stuff can get quite complicated.
It's not by coïncidence that computer programming language designers really have spent a lot of time designing thorough memory-protection and lots of thread-safety measures. It's because troubleshooting bugs is really long and frustrating. For various reasons, memory protection is not enforced in C/C++/ObjC, and for various reasons, thread-safety is not enforceable (every thread-safety measure is a two-edged sword).
______________________________________________________________________ | Mathieu BOUCHARD ----- téléphone : +1.514.383.3801 ----- Montréal, QC
Hi all
Just to resume what I figured out in the meanwhile:
The crashes I often experienced seem related to [iemnet/tcpclient]. Simply bypassing it makes the crashes completely disappear. The good news (at least for me) is that Pd itself runs astonishingly stable (as in 'rock-solid stable'), even if I use some externals which all run in Pd's thread. The bad news is that there seems to be a bug in [iemnet/tcpclient] which is really difficult to hunt down.
IOhannes, if there is anything I could do to help you which debugging this, please let me know. If you intend not to fix it, please let me also know.
Roman
On Wed, 2012-03-07 at 11:39 +0100, Roman Haefeli wrote:
Hi all
While developing on and playing with netpd2, Pd sometimes crashes. This hadn't happened so frequently until recently, so I didn't investigate it further. However, with running more and bigger patches I seem to experience those crashes more often and I started running pd inside gdb to get some more information.
I haven't yet figured a way to reliably trigger a crash. It seems to happen at random times. However, it seems to me that it only happens when I touch a GUI element (sliders, numberboxes, arrays, etc), but this doesn't necessarily is the cause, as most interactions trigger a message sent to the network.
From the many backtraces I collected, most of them look very similar:
Program received signal SIGSEGV, Segmentation fault. 0x080a1fe5 in clock_unset (x=0x83c03f0) at m_sched.c:70 70 while (x2->c_next != x) x2 = x2->c_next;
#0 0x080a1fe5 in clock_unset (x=0x83c03f0) at m_sched.c:70 #1 0x080a2040 in clock_set (x=0x83c03f0, setticks=5047173120) at m_sched.c:81 #2 0x080a2109 in clock_delay (x=0x83c03f0, delaytime=0) at m_sched.c:103 #3 0x080b9fb3 in bang_tilde_perform (w=0x885abd0) at d_misc.c:90 #4 0x080af8b2 in dsp_tick () at d_ugen.c:336 #5 0x080a298d in sched_tick (next_sys_time=5047173120) at m_sched.c:382 #6 0x080a2b7e in m_pollingscheduler () at m_sched.c:482 #7 0x080a2d67 in m_mainloop () at m_sched.c:568 #8 0x080a347a in sys_main (argc=9, argv=0xbffff3c4) at s_main.c:302
#9 0x080ab147 in main (argc=9, argv=0xbffff3c4) at s_entry.c:32
or another very similar one:
Program received signal SIGSEGV, Segmentation fault. 0x080a1fe5 in clock_unset (x=0x83d1660) at m_sched.c:70 70 while (x2->c_next != x) x2 = x2->c_next;
#0 0x080a1fe5 in clock_unset (x=0x83d1660) at m_sched.c:70 #1 0x080a2040 in clock_set (x=0x83d1660, setticks=6376652800) at m_sched.c:81 #2 0x080a2109 in clock_delay (x=0x83d1660, delaytime=0) at m_sched.c:103 #3 0x080b9fb3 in bang_tilde_perform (w=0x88f6c68) at d_misc.c:90 #4 0x080af8b2 in dsp_tick () at d_ugen.c:336 #5 0x080a298d in sched_tick (next_sys_time=6376652800) at m_sched.c:382 #6 0x080a2b7e in m_pollingscheduler () at m_sched.c:482 #7 0x080a2d67 in m_mainloop () at m_sched.c:568 #8 0x080a347a in sys_main (argc=9, argv=0xbffff3c4) at s_main.c:302
#9 0x080ab147 in main (argc=9, argv=0xbffff3c4) at s_entry.c:32
I have many more of this kind, but I think it doesn't help to post them here.
In a few other cases, the network transmission suddenly stops, though Pd keeps running. When I then send 'disconnect' to [iemnet/tcpclient], Pd segfaults and the backtrace looks like this:
Program received signal SIGSEGV, Segmentation fault. 0x080a1fe5 in clock_unset (x=0x8211148) at m_sched.c:70 70 while (x2->c_next != x) x2 = x2->c_next;
#0 0x080a1fe5 in clock_unset (x=0x8211148) at m_sched.c:70 #1 0x080a217c in clock_free (x=0x8211148) at m_sched.c:131 #2 0x00508b79 in iemnet__receiver_destroy () from /usr/local/lib/pd/extra/iemnet/libiemnet.so #3 0x08211148 in ?? () Backtrace stopped: previous frame inner to this frame (corrupt stack?)
(What does the last line mean?)
What can I do to help to track down the reason for those crashes? Would using valgrind reveal more useful information here? Is it possible to tell from the backtraces, whether the cause is in Pd itself or in the externals used?
This is on/with: Pd 0.43.1 iemnet (current svn) zexy (2.2.6 current svn) osc (current svn mrpeach/osc) slipdec/slipenc (current svn mrpeach)
Roman
-
Mathieu Bouchard -
Roman Haefeli