So there I was, at $occupation, researching fun things like IDSes and port scanners, when someone lets me know that the staging server was acting real flaky. I investigate it, as I had pushed code earlier that day, and sure enough, staging is not working properly. I investigate further, and there is only one webpage that seems to not be working right. So I poke and prod at the cryptic logs, and finally realize I'm going to have to call in the developers. I do, and am promptly informed of a backup file to untar. I untar it, and suddenly everything works. Turns out that a developer had pushed files without letting anyone know, which broke the site.
This is why we have code push procedures. More importantly, this is why developers do not push code unless it's a dire emergency. Developers are barbarians, mongols, need walled off from the civilized world so they cannot smash and destroy.
annoyed