I got my controller working pretty well, here's what I did.
For stairs, I have a RayCast to detect if there's a stair in front of the player. I check the normal and see if the angle is as a stair should be. If it is, I just basically make the player jump a specified value separate from normal jump value. It makes stairs smoother without having to press the jump key a bunch, but still feels a lot more like stairs.
For slopes, I know it checks the angles to make sure you can't climb up a super steep hill, but that seems similar to what you're doing. One way you could do it is basically lock the over transforms when the player is falling, so it'll only fall on the Y dimension. There are multiple ways this could be done.