I had attempted this about six times before finally triggering it. I didn't use any waypoints once I started and I put the imp gem on the pedestal and then used it as my last altar- this is only what worked for me however and I am sure other people managed without putting it on the pedestal. I did however just have Ifan for this run which may also have helped. It does seem very twitchy when it will trigger, so put a definite save when you first get to the island.