From what I can see in your screen shots, you have the PWM output on pin 14 and a control signal configured for pin 17, correct? However, you do not have the Spindle Relays enabled in the last screenshot. Normally, you would have the PWM setting the spindle speed and then the configured output (Output 1 on Port 1 Pin 17 in your case) driving a relay to send the VFD a start/stop control. This would likely be a relay that connects a "Common" lead to the Start or FWD Run pin on the VFD itself. Otherwise, the VFD would have to be set to Run manually and then the PWM would control the speed and if Mach 3 is told to set the speed to 0 the spindle would stop running, but still be engaged.
I would suggest unchecking the "Disable Spindle Relays" on the Spindle Setup tab and setting BOTH outputs (M3 and M4) to the output you configured in the "Output Signals" tab - Output 1 in your setup. Make sure the output you configured controls a relay to close the path between the "Common" and FWD Run pins on the VFD.
I'm not sure why you cannot change the spindle speed, but if you're running a program you should be using the SRO (Spindle RPM Override) to adjust the actual RPM. (I see you don't have that on your screen set. You may need to upgrade to the newest release of Mach 3 to get it.)
Hope this helps,
Stephen "Highspeed" Kruse