Save your gems to purchase extra workers. You can reach six workers relatively quickly. Once at that number your island will be much easier to develop.