I have an interesting experiment for you to try. I'll bet your Z axis is no problem with backlash, because it is gravity preloaded againt the screw and bearings in one direction. So try the same thing on the X&Y. Get a gas spring with a force greater than the cutter thrust and preload the axis in one direction. An initial experiment could be done with enough bungee cords to provide the force required and tie the ends to the wall, another piece of immovable equipment.
I bring this up because I once owned a Spindle Wizard knee mill conversion that had double nuts with bellville spring washers loading a second nut against the first so it stated against one side of the screw. That machine was not fast, 20 ipm cutting speeds, but would hold tenths with no problem, and round holes were like they were were bored!