How long is the cable before the connector/line driver convertor?
I'm going to guess by connecting the shielding, you've introduced a ground loop, which is causing interference on the encoder wires before they reach the line driver. First thing I'd do is move the line driver to as near the encoder as possible, as then you minimise he length of wire where the single ended signals can be affected, and get the maximum benefit from the differential pairs/line driver, which is to greatly reduce interference problems.
Alternatively, you could upgrade the encoders to line driver versions, and remove the need for any converter.