It is possible to kill the workers before all hell breaks loose. Equip your flashlight, stand close to a worker, then turn away just enough so the "talk" option is no longer available, swing it, then turn back as quick as you can. If you time it right, the flashlight will make contact and kill them in one hit. It might also help if you turn up your turn sensitivity.