There are a total of three silkworms. The first can be found just to the right, near the start. The second can be found just after you cross the canyon by building the glass bridge. The third silkworm is an end reward for completing the genie's mission where you must collect 25 glass. Note: To unlock that genie mission, speak to the pirate who wants you to get a metal object from somewhere on the map and bring it back to him.