It seems you were right larryc.
Today I woke up, and made some little spacers out of .5mm sheet aluminium. The pulleys still aren't EXACT, but much better.
While playing around with it, I noticed that sometimes the belt would actually skip teeth over the pulley!
I cut the wires to 3 motors, leaving only 1 motor for Y, and 2 for X, and it doesn't seem to have any problems now. I am just going to leave the motors in place instead of swapping them with idler pulleys, as they are cheap and saves me having to come up with something else.
Why they wouldn't run in sync in parallel is very confusing, I don't see how they could actually run out of sync. But apparently they can.
My next dilemma is whether to put a bit of weight on the carriages, or whether to add something like runner bearings underneath. The V groove bearings stay on track better if they have a little bit of force pushing them down onto the tracks.
Torque doesn't seem to be a problem as I first thought, which is why I used 2 steppers. You can still stop it with your hand, but that is good in the case of a softlimits or limit switch failure so it doesn't rip itself apart.
Cheers,
Dan