The website for our biggest client went down and the server went haywire. Though for this client we don’t provide any infrastructure, so we called their it partner to start figuring this out.
They started blaming us, asking is if we had upgraded the website or changed any PHP settings, which all were a firm no from us. So they told us they had competent people working on the matter.

TL;DR their people isn’t competent and I ended up fixing the issue.

Hours go by, nothing happens, client calls us and we call the it partner, nothing, they don’t understand anything. Told us they can’t find any logs etc.

So we setup a conference call with our CXO, me, another dev and a few people from the it partner.
At this point I’m just asking them if they’ve looked at this and this, no good answer, I fetch a long ethernet cable from my desk, pull it to the CXO’s office and hook up my laptop to start looking into things myself.

IT partner still can’t find anything wrong. I tail the httpd error log and see thousands upon thousands of warning messages about mysql being loaded twice, but that’s not the issue here.
Check top and see there’s 257 instances of httpd, whereas 256 is spawned by httpd, mysql is using 600% cpu and whenever I try to connect to mysql through cli it throws me a too many connections error.

I heard the IT partner talking about a ddos attack, so I asked them to pull it off the public network and only give us access through our vpn. They do that, reboot server, same problems.

Finally we get the it partner to rollback the vm to earlier last night. Everything works great, 30 min later, it crashes again. At this point I’m getting tired and frustrated, this isn’t my job, I thought they had competent people working on this.

I noticed that the db had a few corrupted tables, and ask the it partner to get a dba to look at it. No prevail.

5’o’clock is here, we decide to give the vm rollback another try, but first we go home, get some dinner and resume at 6pm. I had told them I wanted to be in on this call, and said let me try this time.

They spend ages doing the rollback, and then for some reason they have to reconfigure the network and shit. Once it booted, I told their tech to stop mysqld and httpd immediately and prevent it from start at boot.

I can now look at the logs that is leading to this issue. I noticed our debug flag was on and had generated a 30gb log file. Tail it and see it’s what I’d expect, warmings and warnings, And all other logs for mysql and apache is huge, so the drive is full. Just gotta delete it.

I quietly start apache and mysql, see the website is working fine, shut it down and just take a copy of the var/lib/mysql directory and etc directory just go have backups.

Starting to connect a few dots, but I wasn’t exactly sure if it was right. Had the full drive caused mysql to corrupt itself? Only one way to find out. Start apache and mysql back up, and just wait and see. Meanwhile I fixed that mysql being loaded twice. Some genius had put load mysql.so at the top and bottom of php ini.

While waiting on the server to crash again, I’m talking to the it support guy, who told me they haven’t updated anything on the server except security patches now and then, and they didn’t have anyone familiar with this setup. No shit, it’s running php 5.3 -.-

Website up and running 1.5 later, mission accomplished.

Worst one I’ve seen so far is when I was working for my previous community another developer joined to help me, without the permission of me or the other lead developer he pushed a client-side update. We didn’t think it was a big deal, but once we began reviewing the code it became a big deal... he had placed our SQL credentials into that file that every client downloads. All the person had to do was open the file and could connect to our SQL which contained 50k+ players info, primarily all in-game stuff except IPs which we want to protect at all costs.

Issue becomes, what he was trying to do required the games local database on the client-side, but instead he tried connecting to it as an external database so he decided to copy server-side code and used on the client.

Anyways, the database had a firewall that blocked all connections except the server and the other lead dev and myself. We managed to change the credentials and pull the file away before any harm was done to it, about 300 people had downloaded the file within an hours period, but nothing happened luckily. IP to the DB, username, password, etc, were all changed just to keep it protected.

So far this is the worst, hopefully it doesn’t get worse than this :/

Just had an old coworker from a previous job send me some stuff for a php script he was having issues with.

There was too much glory in what he was trying to do: mixing php inside of jquery code, not using strict types would have prevented like 10 issues he was having on his script on another portion, mixing headers, weirdly named variables, poorly constructed, reused db connections, 0 oop or proper dependency management in his code, horrible use of sessions and cookies, O (n²) logic all over the place.

But the cake.....are y'all ready for it? It was code screenshots, not even of just the section, no, the full page, from a windows machine (to make it better he is hosting the application on an IIS server and his configuration was not properly set) but I digress, back to the cake:

He was writing his code inside of wordpad :P

FUCKING WORDPAD

I just politely told him that I was busy at the moment and happily ignored him. Dude is not a good person to begin with imo, for example, he brought the subject of homosexuality during one of our talks after he saw me talking to my bf, who just so happens to be gay, his statement was "I do not understand how there can be gay people when there are women that are so hot"

My comeback was "I do not understand how we can be heterosexual when there are some really attractive dudes out there, see how stupid your logic sounds? attractiveness is not the basis for homosexuality ye dipstick" he let it go after that, but close minded people like that are not really my cup of tea.

I started at a company to develop an "uber" clone. Hired by the company's cto. I was happy initially as i had been unemployed for a while but that's because i didn't see the shitstorm coming. The task was build this using php, well 2 weeks later and db locking issues because mysql only allows 100 connections and the website takes over 200mb per request, i tried using the meteor framework, a lil better but the orphaned process would require me to reboot every 2 days. So enter erlang, built in 3 weeks works amazing problems none here... Well in comes the cto (which came in once a week). Apparently he had been reviewing my code and didn't understand it. He couldn't understand no for loops etc and demanded that it be made understandable to a normal dev. Did normal devs write uber no. Anyhow i spent the next 6 Weeks refactoring trying to make elixir looks like imperative programming, he finally gave up, so now I'm deep committed writing an API, finish in a week cto comes in and "why aren't you using patch" i don't need it, well another day implanting a patch api that will never be used. Ok done. Now we have a meeting with the investors who i worked in the same building with and they want a frontend built. I explained i was a backend dev and they needed a uiux expert. Next week cto comes back with this jquery fire pit and stolen bootstrap theme and take me with implementing it. This time we scrap the api change some of the backend logic and implement rest from the 90s one static page per request. After 3 months working with jquery I'm let go because of finical issues. I told them i was a backend dev but they didn't listen if the cto would've gotten a frontend expert things would be different but what to expect from a cto who's coding legacy is creating WordPress plugins.

Hopefully things will be better soon I'm tired of living on the streets.

87 idle connections to the database from this app. Doing nothing.. CLOSE YOUR DAMN CONNECTIONS!!

Am I the only one who gets intimidated when shit its roof?

Yesterday, during crucial business hours, one of the major OMS db column type got overflowed. Caused around 30 mins downtime and then later, pool of all connections with high concurrent requests flushed down stream which caused thunder herd.

One by one.. all services went down; Fucking java service couldn't even start because of load..

This is the moment I fell in love with GoLang. We shard request using GoLang service, it just started and picked up the load beautifully..

At the end, it is around 6 millions business loss, but a good lesson learned :)

I spent a whole day trying to debug why the backend would stop sending data after a while (-> it filled up its connection pool)
but after I turned pooling off, there were a number of dangling Client-Idle-Read DB connections
//not using Entity Framework in these N services, it's pure SQL and DB-AccessLayer here

Never happened before, looked up conf, looked up SO
Changed connectionClose to mandatory Dispose
Still didnt work.

long story short. Turns out in 1 of the services, the DataReader to POCO static method, was disposing/closing the READER and not the CONNECTION.

I sent it as tuple instead of 2 separate variables (reader, connection), and the Item1, Item2 got mixed up (:

Totally my fault, and imma avoid name-less tuples from now on like they a plague -.- thats my 2024-resolution

I'm starting to rank languages based on how easy is to create and work with multi dimensional arrays.

Converting a class that makes lot of DB connections from Java to kotlin made me realise that. Huge turnoff 😞

At the office
5 website is down !
Searching for answer ... Noting. Nginx is calm, php is calm, DB to many connections :O but the DB is interne acces only !!!

Internal ddos WTF

Drupal 8 website -> sorry guy i just fucked up and write 8Go of useless log in watchdog table because something went wrong

Actual log : %errormessage %errortime %vardump

Me : damm he fucked up and cannot write some complet log 🤣

Do you know some module to limit this table size and write acces ?

Needed to setup the database for the web app on one of our Raspberry pis...

Gave the group's sysadmin the sql file and told him to tell me when he was done so I could change the connections in the app. - < two days ago

While he was supposed to take care of it I continued developing the app.

Today I went to ask him what was up with the db, said he wasn't able to import.

This guy, didn't read the error, didn't tell me about it, asked no-one for help and just went back to YouTube for two fucking days.

I'm so done with group projects

Sorta rant.

Now that this drama shite is calming down, here's my tuppence on how connections management should be implemented.

We really need something that is in between blocking and doing nothing. This, in my opinion, is temporary silencing. Having the ability to mute a person or rant, or notifications altogether, for a set period of time would be very useful when a situation needs de-escalating. As this social network (let's not beat around the bush, this is a social network) grows in size, this will be a handy tool for calming storms without burning bridges permanently.

To sum up, I think it would be a good idea to have three available options for notification muting:

- mute notifications/this person/rant for 24 hours
- mute notifications/this person/rant for 7 days
- mute this person/rant forever

As for implementation, I'd wrap up the call to the user's notification assembly like so:

If(!notifBlocked(<typeofnotif>)
{
notif.push()
}

And use a date field in the DB to handle timescales.

Also, I suggest the addition of this tag specifically for suggestions, just so we're not all using different tags

@dfox

tried to stress-test an authenticated websocket endpoint (that makes 2-3 DB calls) by opening closing randomly and it crashed after 20-30 times within a few seconds

I was focused on the middleware glitching out, but error was in the DB-Postgres coz of multiple-connections

Even if I increase the upper limit of simultaneous open connections, the problem at-scale will still exist
If I tried to use a static forever-open connection, it errors out coz 1-command-at-a-time per connection

so im constrained on both sides -.-
Either I rate-limit the endpoint in general and force-close open connections or I cache Organisation-level info that rarely changes

this is one of the few times I miss MS-SQL, it can take a beating but still serve without much complains or losing data consistency -_-

When the DBA restores the live db to staging and all your connections fail on passwords. Every fucking day!

Writing a test helper that handles all the fuckery with DB connections and transactions when the legacy under test uses multiple DB connections is so 👌👌👌👌👌👌👌👌👌👌👌👌👌👌

I can write tests for code that was untestable before 💦💦💦
*nerd-gasms*

Had to optimize today a DB File / MySQL 5.7

I'm sometimes sad that software cannot bite people in their crotches, shit in their faces and fuck their eyes out.

The config was non commented and a disaster.

Yay. Let's set thread_cache to 16K while max connections is 50.

And raise limits like max files without adjusting SystemD / ulimit

And OF COURSE MySQL sucks...

So let's migrate to PostGres.

*Brainfart*

This has happened way too often.

Really. Software should have the right to punish dumb people. At least ... 10 kv shock to teach a lesson.