In order to complete the challenge of killing a cougar with a knife, simply wait until the mission 'At Home With Dutch' (given by MacDougal in Blackwater.

At the point where you are confronted by the cougar, turn around and jump up on the rock after the cougar comes at you. Since the cougar will not be able to jump up to you, all you need to do is fall on top of it and spam your knife attack; takes about 3 hits. Skin and you're all set.

Note that this is a later in game mission, so you cannot do this challenge early on.