Capturing Boos are introduced as early as Gloomy Manor's "Visual Tricks" mission. However, the game does not explicitly tell you what is the reward for capturing them all.

Mission	How to unlock
Gradual Infiltration	Capture all 5 Boos in "Gloomy Manor".
Hostile Intrusion	Capture all 5 Boos in "Haunted Towers".
Outlandish Interruption	Capture all 5 Boos in "Old Clockworks".
Severe Infestation	Capture all 3 Boos in "Secret Mine".
Terrifying Invasion	Capture all 5 Boos in "Treacherous Mansion".