I’m going to 100% say it’s the Edirol device not the Rpi. I used another very similar device for Ms Pinky DJ system before Serato. And I literally spent months trying to get that Edirol piece of crap to work consistently on Windows/Linux/osX Is just never worked smoothly. I switched from USB to Firewire for audio interface and 99% of issues went away. Now USB speeds have caught up with Firewire and USB interfaces are actually truly class compliant. But Edirol was only really good for video mixers.
Good luck with your setup, I’d recommend a newer audio device.
Kylee