To decline a challenge without losing popularity, first ensure that autosave is off. Then, turn exit Legacy mode without saving. When you load your legacy again, notice that your popularity has been restored and you have not and will not be challenged again.