Serial port connection is no use for Mach, well not for main control, it can be used for non time critical things such as a PLC. Even if it was possible you would still need the pinouts so that yopu could configure Mach to the right pins.
First thing you need to do is find out what the pinouts are and also make sure that your machine uses Step and Direction signals. Did your machine come with electrical diagrams? they should show the pinouts.
Hood