February 2023 Downtime Incident

In early February 2023, Beeblebrox went down unexpectedly and remained down for roughly two weeks.

Cause

The incident was caused by a failure of the system's 32GB SD card, which had become corrupted after a power outage. The server was down for two weeks as I had to have physical access to the system to get it back up and running.

Failure of Backups

When I got access to the system, I found that the 32GB backup drive was full and that it was only storing partial backups. I hadn't checked whether or not it was full, since I only keep one week of backups and figured that one week's worth of website/Minecraft saves would fit on a 32GB drive. The wiki's MySQL database was also not backed up. This situation complicated restoring service.

Fix

I was able to recover all of the data from the SD card, and transferred all necessary data back to the Pi, which I equipped with a new, 128GB SD card and a 128GB backup drive. I also reorganized the various services so that they weren't all running from my personal account, and made daemons for the Minecraft server and the IRC Bouncer so that they automatically start on power-on and restart if they crash. I also took various measures to improve the security of the system, even though this (thankfully) wasn't a security incident.