I really have not tried this so please dont be too hard on me.
I am building my own cnc from scratch and I noticed this video on how to setup axis.
it seems you force the cnc do a predictable distance say 18 inches, then measure the actual distance and find the % error. (the bigger the number the better)
this guy explains it so much better and he has a marvellous accent
https://www.youtube.com/watch?v=QybY4DFFzkMI understand its not a video on rotational axis but it might help as you seem to be getting the wrong results.
One last thing are you useing inches and millimeters and mixing them somehow, I suppose you may have used a another (turning) interface and it needs calibrating?