Looks like it was a combination of both.
Max speed set in the VFD is 400hZ which is 24,000RPM. In Mach3 Max speed was set to 240,000 so Mach was telling spindle to go too fast and VFD was saying "nup, not gunna do it!" and shut down.
Min spindle speed was set to zero in Mach3, so when I ran it down bellow zero, I got the mach 3 "too slow for pulley" warning, but because mach3 would only send speed = zero and the VFD was set to min at zero as well, no spindle stop happened.
Set Speed max to 24000 and min to 8000 and all is working fine. I get the to slow/fast warnings, but it keeps spinning. I'll adjust the VFD to slowest of 8K to match as well.
Thanks for your assistance.