OK, I tried the shielded wire. I shielded both the ground wire and the input wire. First I tried grounding the shield at the Smooth Stepper end and it made absolutely no difference. On a whim, I reversed the wires so the shield was grounded at the machine end. Again no difference.
As an electrical engineer, I have experience with assembly language programming of PIC and Motorola processors (not to mention many other computer languages), and circuit design of microprocessor based controllers (like the Smooth Stepper). Everything in my 20+ year career as an engineer tells me this is a programming issue in Mach 3. I don't understand what the physical difference in the electrical signal could be other than noise, but I am not having issues with any of the other I/O signals on the machine. E-stop, motor controls, and even my home made tach circuit are working fine.
I shortened the wire to 4 feet hoping that might help a bit, but no joy. The problem cannot be the Smooth Stepper because it sees the input change every time, as does Mach 3 (indicated by the digitize flag change). The only thing that makes sense to me is that there is a software noise filter on the ATZ function that sets a 'no go' flag on the routine at a certain noise level. This filter is not implemented on the digitize indicator flag, so it changes to green EVERY time. However, as I said, if I touch the input to the machine chassis at any time, regardless of whether or how many times I have successfully auto zeroed by touching directly to the ESS ground terminal, it locks up the function and will not stop the Z axis downward travel until I reset Mach. Then it works fine again as long as I touch the input directly to the ESS ground terminal.
By the way, if I disconnect the 4 foot ground wire from the machine chassis and touch the wires together, the ATZ works fine, even with unshielded wire. It's just the machine chassis that is screwing up the signal. I guess it's time to pull out the oscilloscope and look at the signal, but I will be surprised if it shows anything different at the two ends of the wires, especially the shielded ones.
This is a tough one.