Outage - Database Stuck
... I can't get 1 day without issues apparently...
When I was asleep, it looks like the database got stuck in some sort of optimize process with everything stuck waiting on that.
Restarting the database server forced that process to clear and cleaned things up.
I suspect what happened was I got overzealous after things started working great and I set the background worker count too high (these workers automatically run in the background doing things like updating contacts, downloading posts, etc).
I tuned that setting down, and increased the number of connections the database allows.
Dear god I feel like a newb... I used to do this stuff professionally and I feel the fact that it's been years.
Biggest performance issue I've had for a little while with this server turns out to be because the hypervisor copied the MAC address when I copied the server. It shouldn't have taken me nearly this long to identify this problem!
Aaaargh!
Performance Issues - Tentatively Resolved?
I feel really dumb for not noticing the cause sooner, I most attribute it to the rareness of the problem and the inconsistency at which it occurred.
I use a virtual server environment for my servers, and one of my measures to improve reliability and performance was to have two instances of the webserver behind a load balancer. In laymen's terms, whenever you're connecting you're assigned to whichever has the least connections and they're otherwise identical (same files, connect to the same database, etc).
Well... turns out when I duplicated the server initially, the software decided to not change the MAC address. Laymen's: The ip address of the server is different, but the network uses the mac address to map ip addresses to boxes, if two boxes have the same mac address then traffic is going to sporadically and randomly switch between them... but also with a bad IP address which means half of the traffic is always getting rejected.
So your connection to the load balancer was fine, but it was struggling to connect to the webservers and the webservers were struggling to connect to the database.
Once I changed that the server immediately became quite zippy!
My sincere apologies for the impact.
Content warning: Unintentionally lewd product, only NSFW if you're kinky, someone should have asked more people for feedback on the design
midɲa likes this.
Definitely worth the subscription!
https://youtu.be/PlCHRkH9tEk?si=IfiUT9ifUGKD_-H1
Welcome To Dropout.tv
What is Dropout.tv? It's too hard to put into words, so we just made this video. For exclusive series like Dimension 20, Game Changer, Make Some Noise, Um Ac...YouTube
Outage - Self Resolved - Investigating
The server went down for a few hours today and resolved before I could look at it. It's also been a really bad day for my physical health so I've had very little capacity.
I am looking into why it happened but have no firm answers at this time.
As far as I can tell, the biggest most obvious culprit is the database backup which was happening at that time.
It looks like the database is big enough that backups are no longer simple and I'll need to change my backup method.
Revised method of backups set up, tomorrow I'll test that it's working and then attempt a restore (on a second server so no impact here).
I've also refreshed the virtual network interface as I learned it may have been dropping packets for some unknown reason and it's working fine now.
A quick test...
It's working partially.
I can post TO Tumblr, and Tumblr can see my comments, but Friendica isn't can't see the comments made to the post on Tumblr lol.
@skye 🏳️⚧️ he/him @hadeantaiga Friendica connects to Tumblr as a puppet account, so none of it is on the fediverse but just in the same feed.
I suppose they probably didn't connect the comments because that would get weird with a whole lot of comments being only visible to you (a whole of context management to figure out if you're replying via tumblr or AP, and whether or not the person you're talking to can see the relevant tumblr or AP comment)
It does offer a quick button to get to the original post immediately though, which is helpful (for Friendica users, the top right little arrow beside the network icon).
I think the feature mostly works to let you post and not worry about leaving a chunk of your audience out. Though I imagine it could use some work and if you have ideas they're always listening on the github: https://github.com/friendica/friendica/issues
hadeantaiga likes this.
Introduction post!
Hello! My name is Eli, I'm a transmasc butch from Tumblr. I'm slowly poking my way around the Fediverse and I'm looking for an interface that lets me do long-form blogging. Friendica is allegedly good for that, so here I am!
Grad student, certified "Moon Nerd".
Nonbinary butch transmasc. Masculine like a misty mountain forest, queer as in fuck you. Converting to Judaism slowly but surely.
Interests:
Games: #Pathfinder #DnD #DragonAge #DiscoElysium #TheSims #CivilizationIV
TV: #Trigun #StarTrek #OurFlagMeansDeath #YouTubeVideoEssays
Books: #SherlockHolmes #TheExpanse #LordOfTheRings #TheHungerGames #SciFi #Fantasy
@hadeantaiga Welcome to the server/fediverse! And welcome as a fellow trans jewish convert!
I'm more than happy to answer any questions you may have with Friendica and to help you figure out if it's the right place for you... but if you go somewhere else please link because you definitely have me interested in your introduction!
Also what's your Tumblr, Friendica supports following Tumblr feeds, and this spurred me on to finally setting up the Tumblr addon which allows crossposting to Tumblr (it does require access to your Tumblr to do so, but uses Oauth so doesn't have your password or control over your account, just the ability to post).
@hadeantaiga friendica's upside is connecting to everything that allows it to connect, downside is that it's a little less user friendly.
Followed your tumblr!
hadeantaiga likes this.
Server News
in reply to Server News • •