Skip to main content

You can follow us in other languages. Visit our website for more information https://wordsmith.social/protestation/social

Lenin: The fundamental law of revolution, which has been confirmed by all revolutions and especially by all three Russian revolutions in the twentieth century, is as follows: for a revolution to take place it is not enough for the exploited and oppressed masses to realise the impossibility of living in the old way, and demand changes; for a revolution to take place it is essential that the exploiters should not be able to live and rule in the old way. It is only when the "lower classes" do not want to live in the old way and the "upper classes" cannot carry on in the old way that the revolution can triumph. This truth can be expressed in other words: revolution is impossible without a nation-wide crisis (affecting both the exploited and the exploiters). It follows that, for a revolution to take place, it is essential, first, that a majority of the workers (or at least a majority of the class-conscious, thinking, and politically active workers) should fully realise that revolution is necessary, and that they should be prepared to die for it; second, that the ruling classes should be going through a governmental crisis, which draws even the most backward masses into politics (symptomatic of any genuine revolution is a rapid, tenfold and even hundredfold increase in the size of the working and oppressed masses-hitherto apathetic-who are capable of waging the political struggle), weakens the government, and makes it possible for the revolutionaries to rapidly overthrow it. https://wordsmith.social/protestation/quotes#quote3970

Lenin: To seek out, investigate, predict, and grasp that which is nationally specific and nationally distinctive, in the concrete manner in which each country should tackle a single international task: victory over opportunism and Left doctrinairism within the working-class movement; the overthrow of the bourgeoisie; the establishment of a Soviet republic and a proletarian dictatorship-such is the basic task in the historical period that all the advanced countries (and not they alone) are going through. The chief thing-though, of course, far from everything-the chief thing, has already been achieved: the vanguard of the working class has been won over, has ranged itself on the side of Soviet government and against parliamentarianism, on the side of the dictatorship of the proletariat and against bourgeois democracy. All efforts and all attention should now be concentrated on the next step, which may seem-and from a certain viewpoint actually is -less fundamental, but, on the other hand, is actually closer to a practical accomplishment of the task. That step is: the search after forms of the transition or the approach to the proletarian revolution. https://wordsmith.social/protestation/quotes#quote4079

WTF?

I honestly haven't the foggiest idea how this happened, but apparently the DNS settings got changed a few days ago on the servers with absolutely no explanation (and to junk nonsense settings for some reason). I'm going to keep an eye on them to make sure they don't change again.

Additionally I think that created a cascade that caused the other problems.

Any posts you've made over the past 2-3 days haven't been sent to other servers, but will start sending now.

As far as the other problems, I think when that happened it caused so many processes to lag and take way longer and more resources than usual as any time it tried to contact another server it timed out on the dns request.

DOS Overload

There's been some recent outages of the server, the root cause I've tracked down to the server getting overloaded with requests (mostly updates from other servers). Those updates have been coming in faster than the server can process them and preventing other requests from coming through.

I've made some tweaks that I believe have resolved it, fingers crossed.

Technical explanation:

The servers ran out of php-fpm threads to handle requests. It was configured with static count of 30 each (60 total). They were definitely impacted significantly by memory leaks which kept the count low.

I've changed it from static to ondemand and increased the count to 100 each, I'll probably go in and increase it again since it's still pegged at that limit almost constantly. But thankfully running on-demand seems to be keeping the memory usage per thread drastically lower.

Where the static assignment of 30 was eating up 8GB of ram, 100 on-demand threads is only taking up 1.3GB.

I'm going to increase it until it's either hitting memory constraints or it's no longer constantly at full capacity.

in reply to Server News

Server News
Server News
•

There's definitely some sort of time and code problem involved as it hit again this morning even with the previous changes, though this time it only impacted updates (making posts/comments/likes, getting new posts). I think reading was unaffected because those operations are faster and require significantly less memory.

For whatever reason, sometime around midnight the server gets hit with a bunch of requests that all seem to lock up, eating up large quantities of memory and then won't exit. (With on-demand the threads exit after 10s of being idle, there was over 100 threads running continuously from midnight until I killed them around 9am). Likewise there was a very massive flood of updates from other servers corresponding to that, so I think it might just be a bunch of large servers sending bulk updates or some such.

New tuning to handle that: I put firmer time limits into PHP to prevent threads from running forever, there's two options for setting max times and the first was getting ignored (I think friendica overrode it? the second should override that and kill any threads going too long)

In addition to that, I set up a rate limiter to the inbox endpoint (where other servers send updates to), this should help keep that from overloading the server (majority of the time it'll just be slowing them down by a second or two unless the server is overloaded, at which point the rate limit should help get it accessible for users)

⇧