I would suggest that you set the rpm manualy. There is a difference of rpm which Mach is seeing and what your vfd is actualy providing. They are slightly out, not much, and may be fighting each other for Mach's adjustment of the feedrate. Don't change the pitch, but rather change the feedrate.The feedrate is set based on the rpm. If you use the wizard you can put pitch out to a lot of decimal places, the wizard will show the feedrate rounded to some number, but what you input will be used. Mach can read the rpm more accuratey than you can measure it.
Do scribing tests as that is easier and more accurate than cutting a full thread thus eliminating mechanical / cutting aspects
of threading. Thus there is no spindle slow down, but at 5HP you shouldn't have any anyway.
Do not use any diagnostics, disable them, as Hood noted.
BTW, make sure you use 3 to 5x the pitch to allow for acceleration of the Z axis, even 10x.
RICH