Whilst I haven't much of a clue - I might give you some general pointers - I hope you haven't thought of all these. I have looked at your web site and if some of the intricate designs on there are anything to go by, I can see what you mean.
As far as a rotary and linear axis is concerned, I cannot see that this should be any problem - both are driven by stepper motors - I assume within the limits of the motors themselves. The only difference between a linear and rotary axis is that they do not change units when switching from metric to imperial or vica versa.
When you say you have built this machine recently - I take it you have not needed, or got switched on, the backlash feature.
Yes - when you are making many tiny moves, the computing time needed shoots up. I do not know which mode, CV or Absolute stop, uses more. In Absolute Stop the axis literally stop between each move, then accelerate again - hence the jerky movement, whereas in CV the computer has to calculate similar acceleration and deceleration patterns for the two axis as they move, and introduces the next movement as it slows the previous one. Upping the "look ahead" facility - the number of lines Mach3 computes in advance - might be some help here
There has been mention in this forum some time ago of the screen redrawing interfering with running - particularly of large files. I didn't note how to turn iit off because it didn't apply to me - but no doubt someone will be able to tell us.
On a simple note - is your computer up to the job. The speed of the processor is quite clealry fundamental, but more so the RAM memory is critical in computing speed - and is a simple way to try and improve, without costing a fortune. I am a member of a theatre organ forum - computer based - and the difference in performance between 256MB and 2GB is astounding. No doubt you have a large memory, but I would up it to the maximum and see if that gets the processor working faster.
I hope some of this helps you find where to look.