Going by your plot,
it looks to me that you have too much compensation set, if you are getting a straight line at the quadrant points the axis compensation is moving the table not taking up the backlash, reduce the amount of compensation and try again.
Graham.