Brogar's single strike can take away lot of your health during the early part of the game. Before fighting him, learn the Counterattack skill from Craig (level 2 in sword). Place a bet with Craig and challenge him in the arena. During the fight, press [Counterattack] continuously. He has no chance of hurting you, and he will end up losing his health. When he is down, you can take away his weapon, money, potion and a piece of the golden sword (which is a part of the quest).