Gems Are a Distraction
As you leap through the world, you'll be able to collect in-game currency in the form of gems. This precious loot will allow you to change how the game appears, but otherwise won't make any real impact. You can safely ignore any that will put your high score run at risk, and in fact, that's precisely what you should do. Worry about the gems only when you're just playing a session to goof around.