I wouldn't blame it on the tuning ... until you get whatever is loose corrected, recheck everything mechanical.
The Z slide, spindle bearings/thrust end play, tool holder/drawbar. Something must be loose, that machine is capable of FAR better finishes than that.
Use a prybar and push on every slide in all directions.
I can get better finishes with a hand router.
That's rough, it should be a much better finish.