Your general health is ultimately more important than a big spell collection. Health potions are pretty rare, so concentrate on collecting them when you encounter a vendor. They'll come in handy when you're in big battles.