As Christof says, fine tuning the reader buffer size could be thought of as fine tuning the audio buffer length, ie. a 64 block size is faster but has less space/time if your patch does some really CPU intensive stuff. If you don't need super responsively, you can increase the block size.

Just to clarify: the buffer for readsf~ itself does not cause any delay, so larger values don't hurt (except for wasting some kilobytes). The latency is explicitly controlled by the user as the time between the "open" and "start" message.

(The reason why the buffer itself does not cause a delay is because the worker thread is "eager", i.e. it already has all the data available and so it always fills the buffer as fast as possible. This is different to an audio device where the audio input is produced at the same rate as it is consumed.)

I have never personally needed to use anything but the default readsf~ buffer size.

Christof