I wonder if the six wire one is single ended and there are just 5v, 0v, A, B and Index and the extra wire is the shield?
If not then it could be it does not have the Index either connected or it has no index at all. Often the code number of the encoder will give you info on its signals etc, what is it?
Really a dual trace scope is the way to find out what is what.
Hood