Bartvano,
If you look in the directory you installed MACH4, normally c:\Mach4Hobby\LuaExamples
You will find various examples.
You can look under C:\Mach4Hobby\Docs
You will find a file called Mach4CoreAPI.chm which is a compiled HTML file with the various calls to the core. You will also find a scripting manual PDF file in this directory. I am by no means an expert in LUA at this point, but it is very much like C program language. Also if you look in the C;|Mach4Hobby\Wizards directory you will find some example wizards which all contain some LUA code. When you want to test this stuff you basically start up MACH4 go to Operator then click on Edit/Debug Scripts, and then point to the script you want to examine. You can step through the code or run the code, but you can learn a good deal playing with some of the examples.
The first thing you want to verify is if you can see the encoders updating in Global Registers under the POKEY plugin, just turn the encoder and you should see that encoder count change. The Global Registers update in real time and once you confirm that aspect is working you need to look at the code that reads global registers. I will be doing this shortly to read the DROs from MACH4 and write them to the LCD on my POKEYS pendant, so I can write directly to the global registers under POKEYS for the LCD. I have a long way to go before I understand how all this stuff works in MACH4.