This single instance checking is done in the GUI. To Craig's point, this is what tries to prevent multiple Mach programs running.
When the Mach4GUI.exe program is executed, it loads the Mach4Core.dll file. Mach4Core.dll then loads the plugins. If something is hanging the core (most likely a motion plugin) after Mach4GUI is closed, then it is possible that the GUI shuts down and the cores remains running in the background. In this scenario, it is possible to re-launch Mach4GUI.exe and startup another GUI, core, and a set of plugins. Because the original instance of Mach4GUI is not running, but the core still is!
Now, this is not a good thing at all, obviously. It is why we don't allow two Mach programs running at the same time. My GUESS at this point, because I have no information about what Mach build you are running or what motion controller you are using (things you should really mention when you are having problems), is that is what happened. Two motion plugins fighting over the same motion hardware is probably going to affect performance at the very least!
Steve