Just a thought.
When you save the toolpath, what post processor do you use?
I was just thinking if your Mach 3 native units are set to metric, but you save the toolpath in inch, could that cause this issue?
Are your steps per calculated correctly for the Z ball screw, as you state that the Z has a different pitch/start to the other 2 screws.
I use Vcarve 6, Mach 3 and Probotix boards, drivers and motors and have done a huge amount of engraving without any problem.