Yesterday, while doing some finish wiring on my controls, (mindless work), I was thinking about your problem. It occurred to me that everything you say is happening would happen if you connected a pot to the given Dro.
In the video of one member's new pendant, he points out using encoders for FRO and Spindle speed. So it works, it works for him, it works for me.
Since you are sure it is not a pot, I'm stuck.
I suggect that you rename the Mach 3 folder, then make a virgin installation of Mach 3. Don't modify anything in Mach 3. Install the Pokeys Mach 3 Plug-in. Disable all the Pokeys inputs except the two pins for the encoder. Then try it.
Is the encoder you are using a simple 3 pin unit, or something more sophisticated? Mine is the former. Just like the diagram in the docs. Center pin to ground, outer two pins to two Pokeys inputs. If yours is not, you might try a simple one from Radio Shack.
Ozzie