I have a part that requires the use of a rotary. It was modeled in SolidWorks and the toolpath generated using CAMWorks. When running this code using mach 3 on the cnc machine, I notice that after completing the cuts in the first setup and rotating the a-axis, the tool moves back away from the workpiece, so far that it maxes out the y axis. How should I go about troubleshooting this and removing this unwanted move?

A couple of possibilities. Look at the g-code for the Y move in the Mach 3 editor and change it to a different value. However it is possible you won't find Y value that is doing but instead a G-code that could be say homing the Y axis or something like that.  The cam program is doing a post processing step that may be for the wrong machine, or it may have defaults that insert the big Y move.