Capturing Boos are introduced as early as Gloomy Manor's "Visual Tricks" mission. However, the game does not explicitly tell you what is the reward for capturing them all.
Mission
How to unlock
Gradual Infiltration
Capture all 5 Boos in "Gloomy Manor".
Hostile Intrusion
Capture all 5 Boos in "Haunted Towers".
Outlandish Interruption
Capture all 5 Boos in "Old Clockworks".
Severe Infestation
Capture all 3 Boos in "Secret Mine".
Terrifying Invasion
Capture all 5 Boos in "Treacherous Mansion".
Boo Binary
Go to the Warehouse in the first level of the Old Clockworks. Once you are there, pick up and unravel the red thread roll at the far right. A boo enemy pops out. You'll notice that his name is ComBooter, and he says "01100010 01101111 01101111." His speech is binary, which spells out "Boo."
Dancing Luigi
Spin the circle pad in a tight circle several times consecutively, making sure that Luigi completes 360-degree turns on each rotation. If you do so quickly enough, Luigi will finally stop spinning with a quick pirouette move, like a ballet dancer.
E. Gadd Medals
You can obtain three golden medals with E. Gadd's face on them as you meet certain requirements:
Beat the Dark Moon quest (main story)
Obtain a 3-star rating on every level
Complete E. Gadd's vault room (collect all gems, boos, and ghosts)
Endless Floors in the Scarescraper
This option creates a Scarescraper with infinite floors as well as its own difficulty, also named "Endless".
Unlockable
How to unlock
Hunter Endless
Beat a Hunter mode in a 25F Scarescraper, then "Create a Scarescraper".
Polterpup Endless
Beat a Polterpup mode in a 25F Scarescraper, then "Create a Scarescraper".
Rush Endless
Beat a Rush mode in a 25F Scarescraper, then "Create a Scarescraper".
Surprise Endless
Beat a Surprise mode in a 25F Scarescraper, then "Create a Scarescraper".
Gold File Frame
Collect all of the three E. Gadd medals and you'll unlock a gold frame that appears around your save file.
Golden Ghost in the Secret Mine Mission: Across the Chasm
The Gold Ghost in the Across the Chasm mission can be accessed by heading down into the old mine. Progress through the entire mine until you come to the large wheel located in the finale room, used for mining. Use the 5000 the make the wheel spin and spit out rocks onto the the conveyor belt (the third rock will be large and golden).
After the golden rock disappears into the warp point, head back up the elevator and into the room with the fire (and the tea pot). Carry the golden rock over to the fire to melt it, flushing out this pretty well hidden Gold Ghost.
Luigi statue
Collect all the gems in the Gloomy Manor.
Mario Strikers Charged Music
Enter the elevator in the last level of the first mansion. You should hear music playing in the background. That music comes from Mario Strikers Charged.
Poltergust 5000 Upgrades
As you find and obtain gold in this game, you will periodically unlock upgrades to the Poltergust 5000, your primary weapon against the ghosts infesting Evershade Valley. These unlockables can result in the ability to deal stronger attacks to ghosts or to allow your Dark-Light Device to have more use.
Unlockable
How to unlock
#1 - Power Gauge Upgrade
Collect 2,000 Gold.
#2 - Dark-Light Upgrade
Collect 4,000 Gold.
#3 - Power Gauge Upgrade
Collect 7,000 Gold.
#4 - Dark-Light Upgrade
Collect 10,000 Gold.
#5 - Super Poltergust
Collect 20,000 Gold.
Rewards From Paintings
Shine your dark light on an uncolored painting for a few seconds and money or some other item will appear.
Saved game file icons
Successfully complete the indicated task to unlock the corresponding E. Gadd medal.
Medal #1: Successfully complete Dark Moon Quest mode.
Medal #2: Get a 3 star rank in all missions in Dark Moon Quest mode.
Medal #3: Successfully complete E. Gadd's Vault Room (all Boos, gems, and ghosts).
Scarescraper Mode
Beat the mission "Visual Tricks" in Gloomy Manor.
Showering Plumber
After you turn the water on in the Haunted Towers area, look for a pipe that spouts water near the entrance. Place Luigi directly under the running water. He will take off his hat and scrub under his armpits, as if he is taking a shower.
Sticky Situation: Ghosts Easter Egg
Everytime you play the mission Sticky Situation in the Gloomy Manor Ghosts will appear in random rooms of the mansion.
There is a good chance that everytime you replay the mission the ghosts will be in different rooms/doing different things.
Very entertaining if you just like to go around the Gloomy Manor sucking up ghosts.
Surprise Mode
Beat the single-player campaign, then select the Scarescraper option from E. Gadd's bunker to unlock this mode.
Vault statues
Successfully complete one of the following tasks to unlock a Luigi statue in the Vault.
Collect all gems Gloomy Manor.
Collect all gems in Haunted Towers.
Collect all gems in Old Clockworks.
Collect all gems in Secret Mine.
Collect all gems in Treacherous Mansion.