I see a couple of issues here:
1) You cannot use a DB9 serial cable as a stepper cable. A typical serial cable is made from 28~30AWG wire which is much too small to carry the current needed for stepper motors. You need at least a 22AWG cable, I like to use 20AWG wire when building G540 cables as it is a bit more robust. Take a look at:
http://www.soigeneris.com/custom_made_g540_stepper_cables-details.aspx for custom made G540 cables. (my web site)
2) I suspect that since you have a DB9 on your motor that you have one of those motors where they put the G540 current programming resistor in the connector on the motor. Gecko DOES NOT recommend this as it is a really, really bad idea. The current programming circuit is a low voltage analog circuit. If the current programming resistor is at the motor that means you are running the two wires for the resistor all the way to the motor and back. This will let electrical interference from the high voltage stepper motor wires into the current programming circuit and the G540 will not work properly.
I would suggest making or buying some proper stepper cables and moving the current programming resistor to the G540 end of the cable where it belongs. If you want to order some custom made cables I can put a DB9 on both ends so they will plug into your motors but will put the current programming resistor at the G540 end (suggest getting EZ-G540 option.)
Also, with USB it is possible to overwhelm the USB connection with electrical noise. Don't run the USE cable along with the stepper cables and use a separate 5V power supply for the SmoothStepper:
http://www.soigeneris.com/meanwell_ps_05_power_supplies-details.aspx . I have seen many a USB device not work properly because the PC can't put out enough current on the USB port. Using a separate power supply solves many problems.