The DXF is symmetrical.
Backlash is probably not the problem, it would be minor.
If it shows up with light cuts, it's probably not tool deflection.
I would double check for squareness.
With a precision square or similar to what you have already done.
Mill out a perfect square block, measure it, mill a matching square pocket, then flip the male part and see how it fits in the pocket.
It will show any misalignment.
You can also clamp a good square on the table, indicate one edge true to an axis, then check the other side with the indicator.
Russ