There is also another problem with using cross-correlation that you ought to be aware of, that other people did not mention.
If you are using a loudspeaker and microphone. There is a substantial amount of phase shifting involved as well. (The loudspeaker and microphone contain reactive elements). So, the location of your peak, depends on the type of signal that you send through the system.
For narrow bands of sound (over which the phase shift is nearly constant), the phase shift is incorporated into your estimation of the latency.
For broad bands of sound, the phase shift can vary dramatically between different frequencies. Because of this variability, the peak you are looking for may be more diffusely distributed, or may appear to be absent.
So, keep in mind what your transfer function looks like, and how it is different for different frequencies. Your best bet (in my opinion) is to use sweeps of maybe just an octave or two, to estimate your latency.
Chuck