Several things............
1). Please dont take this in a bad way, but re-read the Brains guide...... your missing some concepts there.....
2). Ok, in your Modbus set up dialog, you will need to convert HEX modbus addresses to Decimal addresses (see guide).
Double CHECK your Modbus addresses in your VFD, to make sure you are addressing your IO correclty....
3). Once you can control your VFD (all functions that your trying to control), from the Modbus Test Window, your 1/2 there.
4). Remember this (and it is a little confusing, I admit), the Modbus OUTPUTs from the VFD are the Modbus INPUTS in the modbus dialog window in mach3, and vice versa going the other way......
5). you will need ONE config for each "Type" of modbus you use: I.e. (Input Coils/discrete inputs, Output coils/discrete outputs, holding is holding). If you get it working in the Modbus test window then you have it right.
6). In Brians, if you are using the "Plugin Enabled Serial Modbus" (which in your pics you are), then in Brians you will need to set the modbus protocol type to "Plugin enabled modbus" (you have yours set to something else).
7). NOTE!!!!!!!!!!! when you put in your "start address" for a config, the Local Var starts at "0" not "1", UNLESS you are changing this on purpose........

. Your outputs in Brian MUST also correspond to the matchine config that you choose for you outputs, and again, you must SET the Using Plugin enabled serial modbus!!
9). Critical, and often overlooked thing on Brians...... When ever you edit/modify/create a Brian, you MUST!!!! "Reload all brains", even WHEN you THINK you see your brains name in the list!!!!!, Make sure you have the Enable Brain CHECKED for that new or modded Reloaded Brian!!!!!
10). For testing Purposes, I would HIGHLY, HIGHLY recommend you get some kind of Cheap secondary Computer monitor, so you can run Mach on one screen, and your "View Brain" (from the button of same name), so you can watch the items change state as you test you system, to debug your Brain.
scott