Use your gems for new equipment or on skill scrolls. Try to keep rare equipment with rare members of the party, since equipment is bound once it's equipped. To unbind equipment, first you have to unequip it, and then you have to spend gems to unbind it. Only then can you move the equipment around to other members.