Certain fences only take certain loot. Here are some examples:
Docks fence, AKA Dalia: Artwork & Gems.
South Quarter fence, AKA Heartless Perry: Gems & Metal.
Stonemarket fence AKA Blackmarket Bertha: Gems & Metal.