I made an oopsie
My apologies to everyone for the long extended outage yesterday.
For background, 2 of the hard drives in the storage array this site is using have gone bad. I got replacements in and set to work migrating storage.
Unfortunately I was overconfident in the process as I had never actually needed to perform such a migration before, let alone in a live environment used by others.
I made assumptions in how the tools would operate in swapping out the drives (I used 'pvmove'...) and didn't realize that the tool I selected would lock the entire filesystem until it was done. (it took ~10 hours to transfer a single disk...)
This was made worse by the fact that the second replacement drive was DOA (it actively prevented the system from booting, so I spent a couple hours troubleshooting that before I realized I hadn't knocked something loose... the system was basically just rejecting the new drive).
There's sadly more downtime to come before this is resolved *but* it should be drastically shorter. Next time I'll be using a different tool to transfer without locking the system, so the downtime will just be 2 reboots (1 to put in the new drive, 1 to take out the old).
The replacement to the replacement drive will be arriving on Tuesday.
Dear lord I can't win lately... NAS went down and I basically had to kick it to get it back up, likely related to a dead disk...
Here's the tally of my environment:
* 2 Proxmox servers
- 1 server failed, I need to send it in for a refund under the protection plan
- I bought a server to replace it already... that server came with a busted ram slot meaning it's already on degraded performance (need to refund it, but want to get a replacement before I send it off...)
- 1 old desktop running as a NAS
- 1 hard drive is throwing non-critical block errors, threatening to fail... I got the warranty exchange set up already but...
- 1 hard drive failed entirely after setting up that warranty exchange and now I don't have the parity to spare a drive, so I need to spare drives before I can start exchanging drives (those drives should arrive on Saturday)
And in all of this... spending way too much money I really shouldn't be spending at all...
And before someone thinks I'm throwing around a few K here... everything I'm getting is renewed, which is probably part of the problem but I sure as hell can't afford new. (Proxmox servers are around $100 each, drives around $45 each)
Shannon Prickett reshared this.
You ever want to see how razor thin support and inclusivity is?
Tell people you're a Jew and hate "Happy Holidays" and "Season's Greetings" because they're faux inclusivity.
People come out of the woodwork that would normally say they're allies to tell you that the harms you experience aren't real, or that the fact that others like it means the harms you experience don't matter.
Let alone the people who don't remotely read said posts before replying...
reshared this
like this
@Cass M 🇨🇦 :mstdn: oh wow I was out of sorts when I replied the first time, I completely blazed past calling out that "Judeo-Christian" is antisemitic:
The biggest thing for me with some of the eat the rich posts I've seen lately, especially those talking about Taylor Swift, is basing judgement off of a net worth score.
(I'm no fan of Taylor Swift, I just know that she barely notched over and only on net worth... which is a junk number, basically a fancy guess how much them and everything they own would sell for, including things like song rights, likeness, etc)
I'll especially note that I don't think any artist really comes close to billionaire earnings without other businesses that are bigger than their art.
Useful reference:
* Highest Paid Actor (in a year) - Most years this floats just under $80 million, some rare extreme years see around $200 million
* Highest Paid Musician (in a year) - this jumps around alot but in 2022 it was $210 million and peaked in 2014 with Dr Dre at $620 million (next highest was Bruce Springsteen in 2021 for $435 million)
Important note not being a billionaire doesn't mean they can't be exploiting, just that billionaires are guaranteed to be intentionally exploiting.
In the case of artists, they have very little control over most of the things. Actors are contractors with little to no employees and no control over the pay of anyone else. Musicians are tied up in contracts and have a limited list of employees (ie. background dancers), and generally are just hiring venues (the venue provides almost all the workers, and they pretty much all suck of course).
Reduced Performance / Reliability
One of the servers went down from hardware failure, thankfully since I run this across multiple boxes with failover it means the site is (obviously) still up.
It might occasionally get a little spotty on connection and especially on performance until that server gets replaced as it means the remaining server is a tad bit overloaded.
It'll probably be a few weeks unfortunately as I don't have the spare funds to pre-purchase a replacement (the protection plan I purchased will cover it, but I've got to mail off the unit, wait for the money, then wait for financial stresses to pass enough that I can order the replacement... then a good week or two delivery time after that)
Got my anti-cistamines today! One order of tit-tacs and one order of antiboyotics!
#hrt #trans #transfemme
reshared this
welcome! to the world of tomorrow!
congratulations! it just gets better from here 💜
Shiri Bailem likes this.
Oh God, I'm LOVING Scott Pilgrim: Takes Off!
It Is NOT What You Expect It Is!
Shannon Prickett reshared this.
@raf 🟣 My jaw hit the floor and never quite got off it the whole rest of the way.
I was half watching it while playing games, expecting a straight retelling with minor changes and more content... I was so confused at the end of episode 1!
My initial expectation is it's going to be like the Matrix 4 movie with lots of callbacks but done so much better
But no spoilers! I'm checking this out!
Shiri Bailem likes this.
Shiri Bailem likes this.
@:labrys-xf-p: HighPriestessTerra :labrys-xf-b: :xf_lesknife: maybe? I hadn't heard anything and the EU's pressure usually comes from other fronts.
I suspect it might be success on Google's counter-ploy add all the same features, and have it automatically translate iMessage's "such-and-such liked your message" to proper reactions... so iMessage users got stuck purely being the less featured bunch
I'm as much eat the rich as the next girl, but I do get tired people applying it casually to entertainers or artists...
It's not a score thing, it's not hit this number in net worth and suddenly you're on the dinner plat. It's based on the fact specifically on the abuses it takes to get there, entertainers being one of the few exceptions on multiple fronts.
Here's the thing about entertainers:
* they're typically not the ones responsible for paying the workers, outside of a small few, actors don't pay the film crew (they're paid by the same people as the crew), and singers/musicians don't pay the majority of the crew (they're paid for by the venue, those they do pay for are probably going to be better compensated for their expertise). The stories of actors donating parts of their salary to crew... that's legit charity as it's not their responsibilities (that responsibility falls on the producers and studio execs).
* they have wildly inflated net worth, because net worth of a performer is typically about how much money they can make, but not necessarily how much money they get.
* they have inconsistent pay
* often those numbers hide other expenses, if they have their own staff or equipment or resources or such... those costs come after the number you're being quoted typically (ie. a musician makes $500 mil on the tour? Their personal crew, tour bus, etc are all paid out from that). This is because a performer is an instance of them being their own business which means they have astronomical expenses under their name.
Of course this doesn't mean they can't be awful, it's not uncommon for them to start their own side businesses (considered a good plan because their stardom likely won't last forever), and it's in those businesses where they often become "The Rich".
like this
@Charlie Stross it's entirely guesswork I agree, but I do think it's also a combination of people not knowing what net worth even is...
Like for you they'd be talking thinks like your "fair market value" if you were to sell off all rights to the laundry files.
Net worth includes a lot of ridiculous number... for the typical person it includes the estimated amounts they can get if they were to sell their car, their home, every single possession of any remote worth to them...
The "net" part of it is what the number is minus debts.
The guesswork of course is just them guessing what you could theoretically sell your whole life for and a guess of how much you owe in debts.
A lot of people think it means income or how much money someone has on hand.
Short Planned Maintenance Tonight
My apologies if this is inconvenient, I opted to do it on shorter notice without a set hour because (a) there's not a lot of activity on the server and (b) I'm really impatient.
I'm doing a hardware upgrade that requires rebooting the network storage backend which will bring down everything for a short time. It should take well under 30 minutes to do the hardware swap and most of the downtime is just going to be the database starting back up (which often takes in the range of another 30 minutes).
As part of this I'll also be deploying some software updates that require a reboot to take effect.
Dear god can we stop with the false "this saying is actually longer", it's terrifying seeing constant historical revisionism even if it's something so minor...
We don't need to edit history to erase all the inconveniences of taking a quippy saying at face value, and this only encourages people to be awful around so very many very real social issues because the quippy line can be purposefully misread in some shitty way that's obviously incorrect... by pulling this constantly it just tells people they're in the right for their bad faith takes.
Hey, it's that time of year again where I remind you that "Happy Holidays" is *** NOT INCLUSIVE ***.
There is no special feature of this time of year that makes "happy holidays" inclusive. It's popular to cite how many holidays occur in winter/fall... but you find just as many in every season.
Happy holidays is purely a stand-in for Merry Christmas, and the whole effort of it is using a nod to minority religions (most commonly my faith) to throw Christmas in our faces while pretending to include us.
Channukah is our least important holiday, acknowledging it when you don't even know our other holidays, let alone what our most sacred holidays are... well, that tells you the whole story itself.
Let alone people don't know when it actually is and will most years be "celebrating" it weeks after it's over.
Holidays don't have to be lumped together, celebrate your holiday. Nobody is offended at you celebrating your holiday, only at it being forced on us.
And "Happy Holidays" only exists as a way to force it on us while pretending to include us.
Random recommendation post:
Kagi: kagi.com
"Premium" Search Engine, free tier to try it out and a small monthly fee to use it regularly. Absolutely better results than Google, no ads, and more privacy. I really do recommend it. (And for my family there is a family plan as well...)
Beeper: beeper.com
A universal chat platform, if you long for the old days of Trillian, GAIM/Pidgin, and the like. Days when you could use your own chat program and just talk to everyone across everything... well this is it again. It's a Matrix service (think open source equivalent to Messenger/Skype/Discord/etc) and bridges to all the other services. Currently it's free, but it's planned to be free + paid extra features.
This has made a world of difference in my experience talking and chatting with people, especially the short list of people who talk primarily on Discord rather than SMS or Messenger. Plus if you're on Beeper or another Matrix service, end-to-end encryption is the default for complete security (You can even use your own client if you like without losing that)
Cosmic Ray reshared this.
WTF?
I honestly haven't the foggiest idea how this happened, but apparently the DNS settings got changed a few days ago on the servers with absolutely no explanation (and to junk nonsense settings for some reason). I'm going to keep an eye on them to make sure they don't change again.
Additionally I think that created a cascade that caused the other problems.
Any posts you've made over the past 2-3 days haven't been sent to other servers, but will start sending now.
As far as the other problems, I think when that happened it caused so many processes to lag and take way longer and more resources than usual as any time it tried to contact another server it timed out on the dns request.
DOS Overload
There's been some recent outages of the server, the root cause I've tracked down to the server getting overloaded with requests (mostly updates from other servers). Those updates have been coming in faster than the server can process them and preventing other requests from coming through.
I've made some tweaks that I believe have resolved it, fingers crossed.
Technical explanation:
The servers ran out of php-fpm threads to handle requests. It was configured with static count of 30 each (60 total). They were definitely impacted significantly by memory leaks which kept the count low.
I've changed it from static to ondemand and increased the count to 100 each, I'll probably go in and increase it again since it's still pegged at that limit almost constantly. But thankfully running on-demand seems to be keeping the memory usage per thread drastically lower.
Where the static assignment of 30 was eating up 8GB of ram, 100 on-demand threads is only taking up 1.3GB.
I'm going to increase it until it's either hitting memory constraints or it's no longer constantly at full capacity.
There's definitely some sort of time and code problem involved as it hit again this morning even with the previous changes, though this time it only impacted updates (making posts/comments/likes, getting new posts). I think reading was unaffected because those operations are faster and require significantly less memory.
For whatever reason, sometime around midnight the server gets hit with a bunch of requests that all seem to lock up, eating up large quantities of memory and then won't exit. (With on-demand the threads exit after 10s of being idle, there was over 100 threads running continuously from midnight until I killed them around 9am). Likewise there was a very massive flood of updates from other servers corresponding to that, so I think it might just be a bunch of large servers sending bulk updates or some such.
New tuning to handle that: I put firmer time limits into PHP to prevent threads from running forever, there's two options for setting max times and the first was getting ignored (I think friendica overrode it? the second should override that and kill any threads going too long)
In addition to that, I set up a rate limiter to the inbox endpoint (where other servers send updates to), this should help keep that from overloading the server (majority of the time it'll just be slowing them down by a second or two unless the server is overloaded, at which point the rate limit should help get it accessible for users)
This is two years old but could easily be mistaken for current news.
Also a great summary of the whole situation and still entirely relevant.
youtu.be/INCXqWzH5vk?si=8fUbaE…
#israel #israel-gaza #Israel-GazaWar #Palestine #palestine-israelConflict
Uncomplicating The "Complicated" Palestine/Israel Conflict - SOME MORE NEWS
For 20% off your first order, visit http://MackWeldon.com/somenews AND ENTER PROMO CODE: somenewsHi. Is The Palestine/Israel Conflict Actually Complicated? P...YouTube
Ooof, just watched the new episode of Dimension 20.
Dropout is genuinely super fantastic about content warnings on their content... but they fell a little short on this one, mostly because they did something new and extreme... and didn't realize that they needed stronger warnings.
And of course in the FB group there are a whole bunch of people going "It's not their fault you didn't read the content warnings" like douches.
like this
reshared this
Shiri Bailem likes this.
It's really wild coming into Baldur's Gate 3 after playing Solasta.
If you're not aware: Solasta is an unlicensed D&D 5e game that uses SRD content plus some of their own content to flesh it out some.
Interestingly enough, I'd absolutely say that Solasta is waaaaaay more faithful to D&D than BG3 is. And it's battle system is much better done, both between being more faithful and supporting things like proper flight and spider climb (as opposed to BG3 which doesn't have spider climb and implements flight as just a fancy long range jump basically).
It does a much better job of short rests vs long rests, let alone fast travel. Solasta doesn't have you warping back to camp mid dungeon, but instead you have to find a safe spot to camp in the dungeon if you want a long rest. And fast travel is done by accelerated movement rather than warping, so you can't use it to magically escape a dungeon.
That's not to say Solasta is outright better.
BG3 shines on drastically better story and voice acting, as well as making your dialogue choices really feel meaningful. I was never able to bond with my own characters really in Solasta like I can in BG3.
Oh, and BG3 feels like a complete package since so many races and classes in Solasta are behind DLC.
If someone told me they wanted to play BG3 specifically for the 5e combat and mechanics, I'd definitely point them to Solasta over BG.
#DungeonsAndDragons5e #DungeonsAndDragons #BaldursGate3 #BG3
🦁Ams 🦁 reshared this.
Oh dear god, I watched Last Week Tonight and have been introduced to cop slide.
And I can't stop looping this one: youtube.com/shorts/_EqrIX6J92s
Policeman injured after tumbling down children's slide
A Boston police officer in full uniform was recorded tumbling down a children’s slide at a newly renovated playground.Boston Mayor Michelle Wu said: “I will ...YouTube
Who approved it?
Raroun likes this.
Gotta admit... I'm sold on Kagi.
The experience and quality has been fantastic using their free tier (100 free searches a month), enough that I decided to pay for it to keep using it.
Gist: it's a search engine that chose to rely on subscriptions instead of ads and data mining... and it shows in the quality of the search results. There isn't a whole page of them linking to other partner or owned services before you get any results. On top of that there's a crapload of customization and control of your experience (such as choosing what things it tries to automatically answer).
As mentioned, there's a free tier of 100 searches a month, that's actually good for most people to use for a week or two. No sensitive data needed, so I recommend giving it a try.
like this
reshared this
Shiri Bailem likes this.
30 Days of US Healthcare: DIR Fees
youtube.com/watch?v=I61wCVh0Y1…
Day 23 of 30 Days of US Healthcare youtube.com/playlist?list=PLpM…
Saw this and decided to share my commentary on it because it's extremely harmful:
Unfortunately nothing here is valid. They've solved the finger problem and it's pretty much only old models that show up there. People are bad at even as well (let alone the fact that real faces are asymmetrical). Purposeless or lack of intent is also common in human art (there are whole art styles around it). And lack of perspective often is because the real art it's emulating lacks perspective.
And those AI detection pages are very very inaccurate. They can't remotely tell you if it's actual AI work, just if they think there's a chance of it being AI work.
All of this, including the detection sites are causing problems for real people who get accused of their very real artwork being AI. People get shut out of communities and told that they need to entirely change their personal style if they want to be accepted because it's too "AI like".
And finally the comment about it taking an incredible amount of resources: it actually takes remarkably little. You can run these AI systems on a single desktop computer with a low to mid range graphics card. (One of the problems for these AI companies and one of the reasons they're so widespread is because it's so easy and cheap to run it yourself)
Note: I don't have the energy for an image description, if you reply with one I'll edit this post to include it.
Just a reminder:
Mastodon != ActivityPub != Fediverse
It's been driving me up the wall how often people act like Mastodon is a synonym for both AP and Fedi.
like this
[moved to @f00fc7c8@woem.space] reshared this.
Just giving a shout out to Fediseer.com for an interesting project (and the gui project available at fediseer.lemmings.world
If you're an instance admin I think it's worth checking out, setting up, and bookmarking for your moderation toolkit.
I don't endorse any sort of automated blocking off of said lists, but I do endorse having a database you can reference when deciding whether to defederate an instance, especially if people have filled in proof links.
Shiri Bailem
in reply to Shiri Bailem • •Oh, and the NAS going down is critical because those $100 boxes (a) don't have enough disk space for everything and (b) it's how I'm maintaining the ability to high(-ish) availability transfer between the two nodes (this saved me when the one proxmox system died).
Those are tiny form factor systems, but they have room for a second hard drive. I need to get a couple of hard drives to drop into them so they have better local storage, afterwards I can set them up as ZFS and use proxmox's features to sync the drives between the two nodes... that way they're not relying on the nas for anything that's not just large.