You start with four villagers on the island. You can only get one more. To find him, explore until you find the pyramid. Somewhere nearby is a person trapped in a spider web. Free them to get your fifth villager.