When on the ship from Kalay to Tolbi, when asked to go get somebody to row, find a weak person first, strong second, weak third, and strong fourth. After the battle with the Kraken, you should arrive at Crossbone Island.