Learn proper swiping technique to gain more fans. If you swipe up too far the fan will get swept off of the screen. If you move them too little, they will not leave or count towards your total. Try swiping with two fingers to make it faster and easier.