I think that would do the trick - a couple of pulleys one at either end, with a belt. Fix your "mouse counter" to one of the pulleys.
If you could somehow "spring" the mouse counter so that was actually turned by friction off the belt, you would get very good resolution.
My first thought was to just attach the mouse wheel to the spindle of one of the pulleys, but you woud then need small pulleys to get anything like the resolution you are looking for.
Carrying one from that - why do we need the pulleys - mount the "mouse" on the table and have it sprung onto a small track on the frame of the lathe/mill - either use a friction contact, or a length of fine toothed rack material which is available at most engineering outlets. In my catalogue this goes down to 1mm teeth spacing, I dont know how many slots in a mouse wheel, but that should give you the resolution you are looking for.