Regarding the downtime that happened last night (while I was trying to sleep which is why it went on for so long).

The short version is a bunch of stuff clogged up the pipes, hung, and just needed a good ol' fashioned restart to fix. I've made some changes to help reduce the chance of that happening again, as always I'm sorry for the trouble.

The longer version is that the php worker processes hung and bogged down the database which brought the whole thing to a screeching halt.

I've changed the limits on those workers so they should have less impact and hopefully not do that again (the downside is that they'll be a little slower on federating updates, but most of the time that shouldn't be noticeable).

I've also taken advantage of the existing downtime to migrate the database over to a second machine with more memory. I originally intended to upgrade the memory of the machine it was on, but unfortunately made the mistake of buying the wrong chips. That plan is still pending. However by migrating it I was able to expand the memory usage significantly which should help performance, the downside is that it's a busier system so the CPU is occasionally busier and can sometimes have a negative performance impact (it's likely negligible but I'm not super confident of that).

In the next week or two I plan to do further hardware upgrades, but with the database migrated already it should be negligible if any downtime.

Once that's done, I'm hoping to implement some high availability options to further reduce downtimes.