When you enter the Inn, turn to your left to find a window. Look in first person view through the window. Shoot a grenade egg at the barrel that is marked gun powder. A door will blast open. Go into the door at the side of Jolly's Inn. You will find a hole in the ground that leads to Smuggler's Cavern.