I'm going to start with my apologies for the 24 hour outage. Thankfully there should be little impact beyond that.

I've been running all of this on a single box, and had plans (last night in fact) to expand it to two with a NAS backend to help protect it from failures and improve performance.

Ironically, the morning in which I planned to make those updates was when I experienced some sort of disk failure on the server. I don't know exactly what happened and the exact level of failure (diagnosis of that will be tonight, mostly to see if I need to get a warranty replacement, or if there was some software cause).

The disk failure presented itself as bad blocks and mild data corruption which prevented multiple services from running.

The good news is that it appears nothing of real importance was corrupted. We lost the majority of one database table, but all the table contained was a list of every single activity-pub contact the server could see and it's contents should be reconstructing automatically now (though obviously may take a while).

It might be worth checking your friends/followers to make sure there are no major absences, I don't know if this impacts who you're following or just the basic contact info.

Beyond that, here's what's changed and changing followed by accountability for my mistakes:

* The server now has a NAS backend with disk redundancy which will protect against drive failures, it also helps to share resources between this server and a second server once I have the drive issues figured out.
* Once the original server is fixed/cleared it will be running a load balanced second copy of the webserver (and eventually a copy of the database) to improve performance and reliability. (The nas will allow two copies of the server to share the same media files)

For accountability:

One of the things that made the failure take longer to recover from was the fact that my backups of the database had failed to run due to a typo.

I could have sworn I had checked it to ensure it was running, but clearly I had not.

I fixed the error and confirmed it ran successfully this morning. Database backups are run twice daily and backed up to a remote **encrypted** backup (borgbase.com). I will be checking periodically over the next week or two to confirm that it continues running.

Additionally, with the NAS now set up and available I am running full system snapshots of the database twice a day as well. This means I should have two avenues for recovery across two different methods going forward, which should significantly increase reliability.

I'm not entirely certain what's causing this, but I do know what kicked it off.

I'm doing some migrations of the media files between boxes and it seems to have somehow messed up the thumbnails and some cached remote images. However, I've seen no problems with any uploaded files (aside from thumbnail views of those files).

This appears to be resolving itself over time as the server updates contacts and recaches many of these files.

Migrations should be finished in the next day or two and there should be no significant downtime.

Regarding the downtime that happened last night (while I was trying to sleep which is why it went on for so long).

The short version is a bunch of stuff clogged up the pipes, hung, and just needed a good ol' fashioned restart to fix. I've made some changes to help reduce the chance of that happening again, as always I'm sorry for the trouble.

The longer version is that the php worker processes hung and bogged down the database which brought the whole thing to a screeching halt.

I've changed the limits on those workers so they should have less impact and hopefully not do that again (the downside is that they'll be a little slower on federating updates, but most of the time that shouldn't be noticeable).

I've also taken advantage of the existing downtime to migrate the database over to a second machine with more memory. I originally intended to upgrade the memory of the machine it was on, but unfortunately made the mistake of buying the wrong chips. That plan is still pending. However by migrating it I was able to expand the memory usage significantly which should help performance, the downside is that it's a busier system so the CPU is occasionally busier and can sometimes have a negative performance impact (it's likely negligible but I'm not super confident of that).

In the next week or two I plan to do further hardware upgrades, but with the database migrated already it should be negligible if any downtime.

Once that's done, I'm hoping to implement some high availability options to further reduce downtimes.