Outage - Self Resolved - Investigating
The server went down for a few hours today and resolved before I could look at it. It's also been a really bad day for my physical health so I've had very little capacity.
I am looking into why it happened but have no firm answers at this time.
Login Screen Issue
There was an issue for a little while that I didn't notice until I rebooted my desktop in which visiting foggyminds.com/ would give you the login screen regardless of your login status (as opposed to foggyminds.com/network or any other address).
Apparently the load balancer was mistakenly caching the root page which caused it to always show the login page.
Server Performance
Just acknowledging that the server has had some spotty performance recently. I'm unable to figure out the cause but continuously investigating.
We now have two load balanced servers, and I can establish that it's not specific to either server. I've updated the load balancer with caching settings which should help alleviate a little (public images now will get cached and not have to go through PHP and database queries).
The database is showing no performance issues that I can see, when the page lags the database queries are not.
Additionally, the server lag appears to be random and per request (as in another identical request made at the same time doesn't lag).
My focus is going to be on examining the PHP-FPM service on both nodes, they're both reporting slow execution.
This looks onerous right now... but I can imagine something like this being incorporated into AC system or dehumidifiers, etc, especially if it gets expanded to an array of diseases...
It doesn't so much prevent disease, but it can at least help with peace of mind and rapid disease control (ie. system goes off, everyone in the area quarantines to limit spread)
Server News
in reply to Server News • •As far as I can tell, the biggest most obvious culprit is the database backup which was happening at that time.
It looks like the database is big enough that backups are no longer simple and I'll need to change my backup method.
Shiri Bailem
in reply to Server News • •Revised method of backups set up, tomorrow I'll test that it's working and then attempt a restore (on a second server so no impact here).
I've also refreshed the virtual network interface as I learned it may have been dropping packets for some unknown reason and it's working fine now.