If you just jog the X axis back and forth, without cutting are you getting the extreme vibration? If so, then you need to get rid of that. It sounds like a mechanical issue, perhaps due to rack and pinion alignment.
Yes, while rigidity is very important, it's also possible to get good results with poor rigidity.
On my wood machine, with skate bearings, I can push and pull the spindle about 1/8" of an inch, and yet I can get much better cuts than you are getting.
I seriously doubt that it's a software issue, but anythings possible. For it to be software, you'd see really horrendous movement, very rough and jerky.
Also, go back and try my suggestions earlier, about climb cutting, and making a finishing pass.