The Nephilim Shard is given to you by Duncan the first time you see him; immediately after you find Jaunty and the ten bird skulls.