hello,
     Can anyone confirm if the mounting hole pcb vias in 7535N and 7737 boards should be mounted electrically to an earth grounded panel?
or should they be isolated?

cheers

The 7737 and 7535N receive their ground through the connection to the power supply. These supplies should share a ground with the controller.
If the pads around the mounting holes are actually connected to the power supply ground, connecting them to the panel creates a ground loop.
If current from other devices is flowing through the panel some of it will flow through the 7737 and 7535N ground wires making noise affecting the input to those boards.

The outgoing side of the 7535N is optically isolated so the grounding of the connected equipment will not interact with the controller grounding.
The 7737 encoder inputs and the step and direction outputs are differential. Their signals are the difference of the voltages on the plus and minus wires in the pair. Noise between the grounds of the controller and the controlled equipment is canceled out.