Is it a 2 flute router bit? What exact bit are you using? 2 flute endmills come in hundreds of varieties. Some will work for what you're doing, some won't.
For a router, your best choice is a single flute "O" flute bit, at about 10,000 rpm and 60-80ipm. Unless your machine is really rigid, you may need to take shallower cuts than 1/32".