You need to calibrate the steps per unit for the A axis. This should be the number of steps that Mach must output for one degree of axis travel.
There is no black magic about it. When gcode calls a movement for the A axis of ten units, then Mach puts out the steps needed to move ten units. Whatever you have set in "steps per unit"
Check "A-axis is Angular" in General Config. Despite popular opinion, this does only one thing. When changing from G20 to G21 or vice versa, the axis output will remain unchanged. That's all it does.
Under "Toolpath Config" you can check "Use Radius for Feedrate" if you wish to use the rotary axis feedrate compensation. I find it works well.
Check "A-Rotations Enabled" if you want the toolpath display to show the rotary toolpath. That is all this setting effects. Only toolpath display. And set which axis the rotary is aligned with, to make the toolpath display correctly.
Greolt