I noticed that when this happens, it is because of the structure or content of the sentence. There is a trigger point that causes the drift. So when the sequence is repeated, drift occurs at virtually the same spot.

The solution is to identify the drift spot and force yourself to keep focused through it.