There is a spindle calibration file. If the VFD is controlled via Mach, then it will most likely be via a step/dir to 0-10V converter.
VFDs themselves usually a fairly linear relationship between input voltage and commanded speed, it's the 0-10V generator that will often not be very linear. The spindle calibration file allows you to compensate for non-linearity.
I'm pretty sure the auto calibration feature in Mach 3 hasn't worked for a good few versions, but if you do a search for spindle calibration, it should throw up some posts about how to edit the file manually. You may need to also have a search over on the yahoo group.
The other option, depending on the VFD, would be to implement Modbus control of the VFD. That way you get very accurate control, however implementing Modbus can be pretty problematic, especially if you're not of a more geeky nature!