As time passes without you attacking enemies, you lose skill points. The number of points you lose per second increases as more and more time passes.