There is no such thing as an analog register in Modbus, just holding registers, input registers, output coils and input discretes.
To read an analog value such as a potentiometer, your modbus device such as a ModIO will need an analog to digital converter, putting the value into an input register (or holding). Modbus then transfers this as a 16bit Register to Mach4. You can then take this 16 bit value and apply it to whatever you want such as spindle speed override, etc.
Cheers,
Peter