First go to Configure/Plugins/RegFile and add your custom register. Then open the screen with screen editor and in the Register property of that DRO select the register you created from the dropdown.
I would probably close and reopen Mach then.
Open Diagnostic/Regfile and find the register you created. From diagnostics you can also set the value of the register and the DRO should update to the new value. Entering a new value in the DRO should update the register. Right clicking the register name from the Regfile diagnostic will also give you the full path of that register you can copy to use in your macro script. Will look something like this.........
iRegs0/MyCustomRegisterName
Use that in the API call to get the register handle then the handle to get or set the register value.
local inst, hReg, value, rc
inst = mc.mcGetInstance()
hReg, rc = mc.mcRegGetHandle(inst, "iRegs0/MyCustomRegisterName")
value, rc = mc.mcRegGetValue(hReg)
rc = mc.mcRegSetValue(hReg, 126)
Registers can be different data types (Value, Value Long, String). It's type is determined by the type data you enter in it's value when creating it. Each type has its own API calls (all covered in the API help file).