Press X, X quickly to drop down a ladder much more quickly.