Machine rigidity plays a large part in accuracy, especially related to cut direction.
The forces that occur when the machine changes direction are enough to cause tolerance issues.
With the spindle turned off, and no tool in the collet, grab the collet nut, and push and pull on it. If you can move it more than a few thousandths, then I'd recommend climb cutting if you need tight tolerances, and trial and error to get the results you are looking for.
Keep in mind that small holes may require lower feedrates to get more accuracy, as the quick back and forth motions can cause a machine to flex.