I can see several things that are less than ideal in your wiring. First, you really MUST have a very solid single-point ground. Normally, this would be a metal plate, to which everything is mounted. At a minimum, it should be a solid buss-bar to which ALL grounds are connected. Never daisy-chain grounds - they should ALL connect directly to a single point. It appears your power supply has two separate output? If so, the grounds on those MUST be tied solidly together. Never, ever connect a signal ground point to a power ground point (e.g. - connect an encoder power- signal directly to motor power- on the drive, to save a wire). Run two separate wires all the way back to the single ground point.
What you're seeing "smells" like a bad ground, since it gets worse under load - that suggests that different parts of the system "see" different grounds at different times, which can cause an active input to be "seen" as inactive, or vice-versa. Basically, the idea is to make sure that for each signal, there is one, and only one, path back to ground, and that path must NOT include shields. Otherwise, the return current will be split among the available return paths, and things may not behave as expected. In your case, I would add a buss bar alongside your power supply, tie BOTH power supply grounds to that, along with the power line ground, and make ALL ground connections directly to that buss bar. My guess is this alone will go a long way towards resolving your problem.
Next, the shielded cables - If you don't have the shields connected, then they're effectively unshielded cables. The shields should be connected to ground at the SOURCE end only. This means the signal cables (STEP, DIR, etc.) should have their shields connected ONLY at the BOB end.
There is no reason for this stuff to be particularly sensitive, if wired up properly. As an example, my E-box has the PC, power supply, two BOBs, SmoothStepper, home/limit/relay I/F board, spindle speed I/F, VFD, Geckos, Modbus board, etc. all in a single enclosure, and works flawlessly. The mounting plates and the enclosure itself are all securely tied to ground. You can barely see in the pictures that the grounds of both power supply filters are tied directly to the mounting plate, and all ground connections branch off from there.
Regards,
Ray L.