I started the testing of my cnc and I want to use fist the MDI window for safety reasons

So I zero the DRO and I type G00 Y12.0 (for 12”) but the axis keeps going past the 12” by a lot

I type G00 Y1.0 (for 1 foot) and the axis stops at about 5”

My units are set for inches, so how I should type inches? Perhaps in decimals?

You have your steps per unit set incorrectly in motor tuning.
Hood

My ballscrews have 10mm Lead or Pitch and the G540 driver has 10 microstep drivers

When I follow Mach3 manual par 5.5.1.2 & 5.5.1.4 the above should give me 4000 which I put into the box “Steps per” in the motor tuning screen.

But 5.5.1.1 says for a 2mm pitch screw: (screw revs per unit) = 1 divide by 2 = 0.5 revs/mm and in my case with 10mm pitch I get 0.1. Where do I put this number? 

If the steppers are direct drive, ie no gearing the number you want is
200 x 10 /10 = 200.

Assuming of course you are set in metric units, if set in Imperial native units then the number you want is 25.4 times that, so 5080.

Hood

Thanks Hood, 5080 was the right number