Can you trigger the limit switch with your finger to verify the limit is in fact working?
One other thing I had to do, was increase the debounce on my system to 250 to allow the switch time to reset during the ref command.
(Not sure is thats the right thing to do, but it allowed the system to work correctly)