Your steps per inch are not right.
First be sure you have set the base units to inch- thats the first menu pick on the Config menu.
On a Sherline the motor is directly connected to the screw, so it takes 200 steps for the screw to make one turn. On an Inch Sherline thats .050, so it takes 20 turns to go one inch. That makes it 200 x20 or 4000 steps per inch. If you have the Xylotex set for 8 micro steps ( you should if you dont) then you need 8x4000 or 32000 steps per inch.
You also ought to download the Mach install guide and read it. It will take you through the setup step by step. Much more likely to succeed that way than trial and error.