- Sensitive content
- Filtered word: nsfw
[moved to @f00fc7c8@woem.space] likes this.
Server Crash
Regarding the downtime that happened last night (while I was trying to sleep which is why it went on for so long).
The short version is a bunch of stuff clogged up the pipes, hung, and just needed a good ol' fashioned restart to fix. I've made some changes to help reduce the chance of that happening again, as always I'm sorry for the trouble.
The longer version is that the php worker processes hung and bogged down the database which brought the whole thing to a screeching halt.
I've changed the limits on those workers so they should have less impact and hopefully not do that again (the downside is that they'll be a little slower on federating updates, but most of the time that shouldn't be noticeable).
I've also taken advantage of the existing downtime to migrate the database over to a second machine with more memory. I originally intended to upgrade the memory of the machine it was on, but unfortunately made the mistake of buying the wrong chips. That plan is still pending. However by migrating it I was able to expand the memory usage significantly which should help performance, the downside is that it's a busier system so the CPU is occasionally busier and can sometimes have a negative performance impact (it's likely negligible but I'm not super confident of that).
In the next week or two I plan to do further hardware upgrades, but with the database migrated already it should be negligible if any downtime.
Once that's done, I'm hoping to implement some high availability options to further reduce downtimes.
If you're experiencing particularly slow load times on the network page (the default homepage with your main feed), one thing on your end you can tune is how many items it tries to load at one time.
Go to Settings -> Display -> Content/Layout and you can change "Number of items displayed per page".
Especially as an item includes a post and all of it's comments as one item, this can make a drastic performance difference (on my personal feed 40 takes >30 seconds to load sometimes, but 20 takes < 3 seconds)
Some updates since it's been a messy few days and I really should have said something sooner.
After a post of mine went viral, server encountered some load issues that kinda cascaded.
I've been gradually reducing the impact through tuning, and have ordered more ram sticks to expand the available ram on the server, as well as give me room on another server to provide redundancy (so it can try to spread the load between boxes).
Ram sticks for the second box will arrive tomorrow, I'll review how I want to approach things from there. No planned downtime for tomorrow, but a planned downtime for next Tuesday with the other ram sticks arrive.
Side note: I've made a custom error page to at least make it less painful, no blaring white, automatic attempts to refresh the page, and my email address to reach out about it if needed.
Planned Outage
Power company is coming to work on the power transformer next Monday (7/17).
I've been notified that the planned outage is for 9am to noon US Central Time.
Shiri Bailem
in reply to Shiri Bailem • •