Do not waste troops and time waiting for them to destroy walls. The quickest and most effective way of getting rid of walls is to blow them up.