When Babba is chasing you, run him under one of the shadows. A huge chest of gold will fall on his head and he will be unconscious. Make sure Bugs is on one of the switches and the gong will turn towards you. Pick up Babba and throw him at the gong. Do this to all four gongs and you will defeat Babba. Daffy Duck will then pick up the Time Gem and warp him to Counts Castle.