Archers are your friends. As soon as you can, place some around the village. They will fire at anything that gets too close.