Taught by my dying servers I've tried to keep a good backup of my running apps so if it ever happens again I don't have to murder my imaginary self and excuse a growing number of people who use my apps.
And from time to time I dump archives from a remote to a NAS.
Imagine my surprise when I open the backup folder and meet a 6 months worth of empty tar archives.
First, I nervously made a manual backup if something were to happen right now. Then I started to investigate all of my archiving scripts for first filtering the archives, then moving them from remote to remote, and finally the scripts that manage created archives - and none of them seemed like they'd ever go wrong and break.
Only after digging through them I've thought of checking logs they ouput and have seen a lot more activity for a script that's only supposed to fire once per day and group that day's archives.
And of course, I misconfigured cron so it fired every minute between 1:00 and 2:00 instead of once at 1:00 -
* 1 * * * instead of 0 1 * * *.
Great I caught that early.