This is a very good question - meaning I have had to redesign my own system to solve some problems here
First, you have a brushed DC motor: that can be reversed, so that part is easy.
Next: what sort of relays to use?
I suggest that DPDT power relays such as the JQX-13F are suitable. Cheap on eBay. Mine have 24 VDC coild, but you can get them with 12 VDC coils.
I started with a single DPDT relay to do the reversing. That was a simple, but it has real problems. If I was spinning in reverse (M4) and I turned the spindle off (M5), the reversing relay dropped out, and the motor suddenly had +50 V across it rather than -50 V. That did not harm the motor (I think), but it was stressing the hell out of the spindle driver, and not doing the relays any good either.
I changed the design to have TWO relays. One relay is for M3: go forwards, the other relay is for reverse (M4). I have some interlock logic in both the M3 and M4 macros and also in the hardware to prevent me ever trying to turn both relays on at once. This is ESSENTIAL!
Now when I issue M5 both relays are turned off (even though only one of them was actually turned on). All is well.
Some enhancements: when you issue M3, M4 or M5, Mach introduces a dwell time before the program continues. This allows the motor to spin down. I have about 1.5 seconds for this dwell time.
My M3 and M4 relays have the Common terminals going to the motor. The Normally Open terminals go to the power supply. When the relays are Off, the Common terminals (and hence the motor) are connected to the Normally Closed terminals. I have put 45 ohm 25 Watt power resistors across these NC terminals. These resistors are NOT in circuit when the relay is energised, but when M5 is issued the contacts drop back to the NC terminals, putting these power resistors across the motor. They very quickly suck all the energy out of the motor (which is now acting as a generator) and stop the spin. I have two resistors, but in fact you only need one due to the wiring arrangement. They are sometimes called 'braking' or 'dump' loads.
As to the strange behaviour of the speed control pot - that is probably a safety feature. If you power the system up with the pot NOT on zero, a safety circuit prevents the supply from giving out any volts. That way you don't find the spindle suddenly doing 3,000 RPM when you turn the machine on. Either power up with the control pot on zero (or reset), or manually reset it after powering up.
Cheers