Use the following trick to keep your stolen items if you get caught. Rather than paying someone to relieve the bounty on your head, drop everything in your inventory, except gold, somewhere you will remember. The objects will not be removed from the game. Then, pay a guard. You will not lose your stolen items and the bounty will be gone. As an additional bonus, the person you stole from will not attack you anymore or be sour towards you.