Skip to main content

Edward Carpenter: Law represents from age to age the code of the dominant or ruling class, slowly accumulated, no doubt, and slowly modified, but always added to and always administered by the ruling class. Today the code of the dominant class may perhaps best be denoted by the word Respectability-and if we ask why this code has to a great extent overwhelmed the codes of the other classes and got the law on its side (so far that in the main it characterises those classes who do not conform to it as the criminal classes), the answer can only be: Because it is the code of the classes who are in power. Respectability is the code of those who have the wealth and the command, and as these have also the fluent pens and tongues, it is the standard of modern literature and the press. It is not necessarily a better standard than others, but it is the one that happens to be in the ascendant; it is the code of the classes that chiefly represent modern society; it is the code of the Bourgeoisie. It is different from the Feudal code of the past, of the knightly classes, and of Chivalry; it is different from the Democratic code of the future-of brotherhood and of equality; it is the code of the Commercial age and its distinctive watchword is-property. https://wordsmith.social/protestation/quotes#quote9502

Eugene V. Debs: The master class has always declared the wars; the subject class has always fought the battles. The master class has had all to gain and nothing to lose, while the subject class has had nothing to gain and all to lose - especially their lives. They have always taught and trained you to believe it to be your patriotic duty to go to war and to have yourselves slaughtered at their command. But in all the history of the world you, the people, have never had a voice in declaring war, and strange as it certainly appears, no war by any nation in any age has ever been declared by the people. And here let me emphasize the fact - and it cannot be repeated too often - that the working class who fight all the battles, the working class who make the supreme sacrifices, the working class who freely shed their blood and furnish the corpses, have never yet had a voice in either declaring war or making peace. It is the ruling class that invariably does both. They alone declare war and they alone make peace. https://wordsmith.social/protestation/quotes#quote9503

Paulo Freire: All the actions of the dominant class manifest its need to divide in order to facilitate the preservation of the oppressor state. Its interference in the unions, favoring certain "representatives" of the dominated classes (who actually represent the oppressor, not their own comrades); its promotion of individuals who reveal leadership capacity and could signify a threat if they were not "softened up" in this way; its distribution of benefits to some and penalties to others: all these are ways of dividing in order to preserve the system which favors the elite. https://wordsmith.social/protestation/quotes#quote9504

Short Planned Maintenance Tonight

My apologies if this is inconvenient, I opted to do it on shorter notice without a set hour because (a) there's not a lot of activity on the server and (b) I'm really impatient.

I'm doing a hardware upgrade that requires rebooting the network storage backend which will bring down everything for a short time. It should take well under 30 minutes to do the hardware swap and most of the downtime is just going to be the database starting back up (which often takes in the range of another 30 minutes).

As part of this I'll also be deploying some software updates that require a reboot to take effect.

WTF?

I honestly haven't the foggiest idea how this happened, but apparently the DNS settings got changed a few days ago on the servers with absolutely no explanation (and to junk nonsense settings for some reason). I'm going to keep an eye on them to make sure they don't change again.

Additionally I think that created a cascade that caused the other problems.

Any posts you've made over the past 2-3 days haven't been sent to other servers, but will start sending now.

As far as the other problems, I think when that happened it caused so many processes to lag and take way longer and more resources than usual as any time it tried to contact another server it timed out on the dns request.

DOS Overload

There's been some recent outages of the server, the root cause I've tracked down to the server getting overloaded with requests (mostly updates from other servers). Those updates have been coming in faster than the server can process them and preventing other requests from coming through.

I've made some tweaks that I believe have resolved it, fingers crossed.

Technical explanation:

The servers ran out of php-fpm threads to handle requests. It was configured with static count of 30 each (60 total). They were definitely impacted significantly by memory leaks which kept the count low.

I've changed it from static to ondemand and increased the count to 100 each, I'll probably go in and increase it again since it's still pegged at that limit almost constantly. But thankfully running on-demand seems to be keeping the memory usage per thread drastically lower.

Where the static assignment of 30 was eating up 8GB of ram, 100 on-demand threads is only taking up 1.3GB.

I'm going to increase it until it's either hitting memory constraints or it's no longer constantly at full capacity.

in reply to Server News

Server News
Server News
•

There's definitely some sort of time and code problem involved as it hit again this morning even with the previous changes, though this time it only impacted updates (making posts/comments/likes, getting new posts). I think reading was unaffected because those operations are faster and require significantly less memory.

For whatever reason, sometime around midnight the server gets hit with a bunch of requests that all seem to lock up, eating up large quantities of memory and then won't exit. (With on-demand the threads exit after 10s of being idle, there was over 100 threads running continuously from midnight until I killed them around 9am). Likewise there was a very massive flood of updates from other servers corresponding to that, so I think it might just be a bunch of large servers sending bulk updates or some such.

New tuning to handle that: I put firmer time limits into PHP to prevent threads from running forever, there's two options for setting max times and the first was getting ignored (I think friendica overrode it? the second should override that and kill any threads going too long)

In addition to that, I set up a rate limiter to the inbox endpoint (where other servers send updates to), this should help keep that from overloading the server (majority of the time it'll just be slowing them down by a second or two unless the server is overloaded, at which point the rate limit should help get it accessible for users)

⇧