Glad, I’ll try to help as best I can.
>Coming from you it is all sounds easy and very optimistic. I started believing that 4th axis is not a big deal .
No big deal getting it installed and set up but depending on what you want to do with it, the sky is the limit so far as complexity goes.
> My first challenge will be to rewire Sherline stepper from 6 wires to 4 (Xylotex board).
Sounds like you have good input and a handle on that part.
> Second, is to properly setup table in Mach3.
The Sherline stepper is 1.8 deg/step or 200 steps/rev (360/1.

and the rotary table has a 72:1 gear ratio or 5 deg/step (360/72). So for single stepping that would be 200/5 = 40 steps/deg. Multiply the 40 times the micro stepping jumper setting on the Xylotex board, e.g. if it’s set to 8 then 40 X 8 = 320 steps per deg for the Mach3 entry. The velocity and accel numerical values will likely be drastically different from your X, Y and Z due to the deg/min units. Presumably your linear axes are already set-up so here is what I would do. Open the spreadsheet at this link:
http://www.machsupport.com/forum/index.php/topic,16315.0.htmlEnter the input data for both linear and rotary axes. Enter your known reliable max rapid velocity for a linear axis and read the corresponding stepper RPM. Then enter that RPM in the rotary column and read the result steps per degree for the Mach3 velocity entry. That way you are comparing apples to apples (RPM). The accel will need to be changed in the same proportion as the velocity.
>Last but not list, to produce a G-code to engrave patterns on a flat surface using Y, Z and A axis.
>The idea is to install RT flat on an X/Y table perpendicular to Z. Attach lathe type of tool bit on Z axis pointing >down perpendicular to X/Y. Z axis will go up and down driving tool bit into the engraved part installed in RT >regulating engraving depth. Spindle will not rotate. Y axis will move in longitudinal direction fore. and aft. from >outside toward center of the engraved part. So, when all three axis moving simultaneously: A rotates clockwise >holding eng. part, Z drives non rotating tool in and out of the eng. part, and Y moving RT (eng. part) fore or aft.
>I hope my explanation makes some sense.
You may have lost me here. Sounds like your rotary table is going to be mounted flat on the X-Y table with it’s rotational axis vertical, i.e. parallel with the mill Z-axis. If so, then the rotary is now the C-axis (following accepted convention; A-axis is parallel with the mill X-axis). On the other hand if this is so, i.e., you are engraving a flat pattern on a flat surface, then why use the rotary? Wouldn’t the X, Y and Z do the same thing? And BTW converting Gcode X values to equivalent C values is not a trivial exercise. Been there, but sometimes it’s worth it. I would suggest that you invest in a 90 deg mount for the rotary and mount it on the X-Y table with it’s axis parallel with the mill X-axis and practice some cylindrical engraving first. Configure as X, Z, A. Lots of free wrapping software to convert Y values to A and I think you will find it quicker to get a feel for the rotary.
Hope this helps
Al