I tried delaying the dsp 1 message on loadbang by 200ms since I read somewhere this could be the problem.
Try
[loadbang] --> [del 1000] --> [; pd dsp 0, dsp1(
If the audio device has already been opened (albeit unsuccessfully), you might need to turn DSP off first, so that turning DSP on will reopen the device.
Christof