As time passes without you attacking enemies, you lose skill points. The number of points you lose per second increases as more and more time passes.

Time Laps Points Lost
1-10 seconds 10
11-20 seconds 20
21-30 seconds 30
31+ 40

You will also lose skill points by being attacked.

Attack Points Lost
Critical Attack (hit) 60
Critical Attack (blocked) 60
Deflected Projectile (hit) 30
Deflected Projectile (blocked) 15
Enhanced Light Attack (hit) 30
Enhanced Critical Attack (hit) 70
Enhanced Strong Attack (hit) 50
Force Attack (hit) 30
Force Attack (blocked) 15
Hazard (hit) 40
Light Attack (hit) 20
Light Attack (blocked) 10
Strong Attack (hit) 40
Strong Attack (blocked) 20
Thrown Object (hit) 50