You probably also get clicks? In general it isn't very good to change the array read by a tabosc4~ on the fly. A better solution might be to drive a tabread4~ object by a phasor~ multiplied by the length of the array. This way you also won't have problems with the different array sizes.
I have tried to implement this. tabread4~ can be driven easily with a phasor~, unfortunately tabwrite~ can only be driven by two messages (start, stop), I mean it can't be driven by a phasor~
So I tried to use a threshold~ in order to start / stop the recording from tabwrite~ depending on the values of the phasor~ But threshold~ seems to work only with adc~ or sig~ as input ; did you manage to use it with phasor~ or osc~, for example ?