February 2023 Downtime Incident - Revision history

Hdw at 22:00, 20 February 2023

2023-02-20T22:00:55Z

← Older revision		Revision as of 22:00, 20 February 2023
Line 7:		Line 7:
	== Failure of Backups ==		== Failure of Backups ==

	When I got access to the system, I found that the 32GB backup drive was full and that it was only storing partial backups. I hadn't checked whether or not it was full, since I only keep one week of backups and figured that one week's worth of [[~~Websites\|~~website]]/[[Minecraft]] saves would fit on a 32GB drive. The wiki's MySQL database was also not backed up. This situation complicated restoring service.		When I got access to the system, I found that the 32GB backup drive was full and that it was only storing partial backups. I hadn't checked whether or not it was full, since I only keep one week of backups and figured that one week's worth of [[website]]/[[Minecraft]] saves would fit on a 32GB drive. The wiki's MySQL database was also not backed up. This situation complicated restoring service.

	== Fix ==		== Fix ==

Hdw at 21:59, 20 February 2023

2023-02-20T21:59:33Z

← Older revision		Revision as of 21:59, 20 February 2023
Line 7:		Line 7:
	== Failure of Backups ==		== Failure of Backups ==

	When I got access to the system, I found that the 32GB backup drive was full and that it was only storing partial backups. I hadn't checked whether or not it was full, since I only keep one week of backups and figured that one week's worth of [[website]]/[[Minecraft]] saves would fit on a 32GB drive. The wiki's MySQL database was also not backed up. This situation complicated restoring service.		When I got access to the system, I found that the 32GB backup drive was full and that it was only storing partial backups. I hadn't checked whether or not it was full, since I only keep one week of backups and figured that one week's worth of [[Websites\|website]]/[[Minecraft]] saves would fit on a 32GB drive. The wiki's MySQL database was also not backed up. This situation complicated restoring service.

	== Fix ==		== Fix ==

Hdw: Created page

2023-02-20T21:58:33Z

Created page

New page

In early February 2023, [[Beeblebrox]] went down unexpectedly and remained down for roughly two weeks.

== Cause ==

The incident was caused by a failure of the system's 32GB SD card, which had become corrupted after a power outage. The server was down for two weeks as [https://www.spacehippie.ca/~hdw I] had to have physical access to the system to get it back up and running.

== Failure of Backups ==

When I got access to the system, I found that the 32GB backup drive was full and that it was only storing partial backups. I hadn't checked whether or not it was full, since I only keep one week of backups and figured that one week's worth of [[website]]/[[Minecraft]] saves would fit on a 32GB drive. The wiki's MySQL database was also not backed up. This situation complicated restoring service.

== Fix ==

I was able to recover all of the data from the SD card, and transferred all necessary data back to the Pi, which I equipped with a new, 128GB SD card and a 128GB backup drive. I also reorganized the various services so that they weren't all running from my personal account, and made daemons for the Minecraft server and the [[IRC Bouncer]] so that they automatically start on power-on and restart if they crash. I also took various measures to improve the security of the system, even though this (thankfully) wasn't a security incident.

[[Category:Technical]]