If you want the overall connection status then use the mbcntl/status reg:
local modbusStatusReg = mc.mcRegGetHandle(inst, "mbcntl/status")
To get the individual function status use the handle to the function return calls:
local modbusFunc0ErrorReg = mc.mcRegGetHandle(inst, "ModbusMPG/function0/rc") 
local modbusFunc1ErrorReg = mc.mcRegGetHandle(inst, "ModbusMPG/function1/rc")
I attached a picture for reference of heirarchy. Your modbus connection names will be different.
So, as an example, if you want to make damn sure everything is running you could do a check like this:
function modbusIsRunning()
        -- check if connection is good
	modbusRunning = mc.mcRegGetValueString(modbusStatusReg) == "RUNNING"
	
        -- check if function 1 is error free
        modbusRunning = modbusRunning and mc.mcRegGetValue(modbusFunc0ErrorReg) == mc.MERROR_NOERROR
       
        -- check if function 2 is error free
	modbusRunning = (modbusRunning and mc.mcRegGetValue(modbusFunc1ErrorReg) == mc.MERROR_NOERROR) and true or false
       return modbusRunning  -- return all checks passed
end
 -- use the function to check if modbus is good
if modbusRunning() then 
      -- do something in here 
end     
Those == signs and the 'and true or false' are just flattened if statements (ternary ops like C), so you can explode them into 'if-else' statements if they are confusing to read:
-- check if connection is good
modbusRunning = mc.mcRegGetValueString(modbusStatusReg) == "RUNNING"
-- can be written like this
if mc.mcRegGetValueString(modbusStatusReg) == "RUNNING" then
      modbusRunning = true
else
     modbusRunning = false
end
----------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------
-- check if function 2 is error free
modbusRunning = (modbusRunning and mc.mcRegGetValue(modbusFunc1ErrorReg) == mc.MERROR_NOERROR) and true or false
-- can be written like this
if modbusRunning and (mc.mcRegGetValue(modbusFunc1ErrorReg) == mc.MERROR_NOERROR) then
      modbusRunning = true
else
     modbusRunning = false
end