I find myself falling from a ledge more often than not than climbing up it. I get to the top of a cliff or wall and my character does the climbing up animation but chances are, I end up falling backwards and injuring/downing myself. This could really do with looking at!
Yea, it gets frustrating after a while climbing pillars for ore when it's like a 50/50 chance every time I just randomly fall all the way to the ground and now I have to go back up again. Or getting up mountains in the desert gets really annoying at times.
Same happens to me. I play offline. It seems to start the "Climb to top" animation early at which point you aren't at the top and just fall back down.