When you do not want to be hit by ghosts and lose all your items (Balloons, Beans, and sometimes Stink Pellets) just stand on the ledge until the specter nears you. Tap the Analog-stick so that you will fall and catch the ledge. When you are hanging, they cannot hit you and cause your items to go flying.
Another way to avoid the ghosts and keep from losing your collected items is to use Lumos. Hold Down to maximize the spell. The ghosts will not hit you when the light is on. Sometimes this requires careful timing because you are trying to search and collect items at the same time.