Mach3 macros are written in a VB script language called Cypress Enable. If you Google "Cypress Enable", you'll find the Cypress website, and can download a PDF of the language reference manual. The Mach3-specific function calls and variables are (sort of....) defined in the Wiki on the Artsoft website, and various places on this forum. Learning to write Mach3 macros can cause you to tear your hair out, but it is doable. The available information is spotty, and some is just plain wrong. But, there are people here who have learned the hard way, so if you get stuck, just ask.
Mach3 plug-ins are written in C++, using a now very outdated version of VisualStudio. I forget what version is required, but that too is documented in the Plug-In programmers guide on the ArtSoft website. You cannot use newer versions - the plug-ins will compile, but they won't work....
Screensets can be created using a variety of tools. Screen4, is probably the most common, though I'd suggest you instead try MachScreen, which seems MUCH more stable, and easier to learn. Screen4 works, but it is riddled with quirks, and crashes periodically, often trashing your screenset in the process. If you use it, make *frequent* backups. Screen4 can be downloaded from the ArtSoft website. I think you can download MachScreen from somewhere on this site, but I'm not sure. Do some Googling and you'll find it. It was written by a Mach3 user, and is a nice piece of work.
Regards,
Ray L.