I’m sure you’ve looked but I just glanced here:
I can tell you how my config steps with grbl and bCNC went (and I had a few of your same issues with it).
First step – understand positive and negative movement. (and machine vs work coordinates). Put a sheet of paper on your board (and I’m assuming you home to the top right of the machine – you should…).
Positive X moves to the right, Positive Y moves back, and Positive Z moves up. Get this going correct first (flip motor leads or change controller configuration)
Second step – Homing. We make homing moves in the positive direction (right, back, up). Get homing working. I did not have the included capacitors on the homing inputs and it drove me nuts – the limit got triggered w/o the switch and then backed off in the wrong direction (this might be what you are seeing).
Third step – begin to understand MCS and WCS – Machine coordinate system and work coordinate system. After my machine homes, it is at MCS (5mm, -5mm, -5mm) as I backoff 5mm from the limit switches. Therefore the limit switches are (0,0,0). Understand MCS should always be negative after homing (this is because we subtract/cut material with this machine - 3d printers have a positive MCS…).
Now when you jog the machine around you can then set a WCS to (0,0,0) and the MCS might be (540.3,-325.0,-36). The controller sends the command to set the WCS to 0 - you can zero X,Y,Z independently.
Fourth step – dial in distances. Now that you are moving in the correct directions, and can home, you can dial in your distances. Basically, the 8mm leadscrew should be 8mm per revolution, but it isn’t… See previous discussions or let me know if you need help.
Good luck, and check those capacitors on the limit switch inputs…