Disable Spindle Relays does just what it says. It disables the spindle when it's checked. Leave it unchecked.
Pin 17 should be set to Output 1. This turns the spindle on and off. Make sure Output 1 is assigned to Clockwise (M3).
Under Motor Control, check Use Spinle Motor Output and PWM Control.
Pin 14 controls the rpm, via PWM, and should be assigned to the Spindle Step Pin.
Male sure in Config > Spindle Pulleys, that you have Min Speed set to 0, and Max Speed set to 24,000. This allows Mach3 to send the correct voltage for a given RPM.
(There are three VFD pins on the g540 - high, common, and signal.)
The manual says Ground, Output, and +10V.
Make sure your getting +10V from the VFD.
The set the spindle speed to 24,000 and make sure your getting 10V on the output. (Pin 8?)
Then set the speed to 12,000 and see if you get 5V on the output.
I don't have a G540 and VFD, so can't help you with wiring or VFD Settings. Having all the correct VFD Settings is also very important.
You might want to post your .xml here so someone can take a look and see if anything obvious is set wrong.