When you speak with Sugar-Lips Habasi at the South Wall Cornerclub in Balmora, he will remove the bounty from your character for a price. However, if you take the time to leave your money on the floor beforehand, you can get rid of your bounty and keep most (if not all) of your gold.