Stupidest client ever:
I once had a client that requested me a new website, all went well and get paid.
After a month our two, he called and requested a change on basically all pages. I sent him a estimate of 2h, he approved, signed a contract and I've made the changes. He never paid me that changes. After 2 months ignoring my calls and emails I've rollback to the pre-changes website. He called me anything you can think off and that I would be sorry for done that... I was like... WTF??
- Pay me and I'll put the changes online.
He replied:
- you will be sorry. See you in court.
We went to court because of FUCKING 200€...
Court decision? The client was obligated to pay me the 200€ and all the court expenses...

C: application not working
Me: k. What changed?
C: we didn't make changes
Me: k... *gets a tech team (W) on the phone*
W: Hey, what's broken?
Me: C's application. How do things look?
W: running healthy. I'll check logs.
Me: thanks. *gets tech team (S) on the line*
S: hey, everything clear on our end, will check logs.
Me: thanks *gets tech team (U)*
U: hey! They asked us to deploy their new version today during normal deployment time. Is it acting up?
Me: C, what did you change?
C: nothing major, just how we connect to W and S...
W&S: are you shitting me???
Me: U, will you please roll it back?
C: no! Must stay on this version, you need to fix your side!!
Me: nope. *calls U boss (UG)*
UG: U, you have my permission to roll back, they need to fix. C, if your boss doesn't like it, have them call me.
*rollback fixes problem*

IF I FUCKING ASK YOU WHAT THE FUCK YOU CHANGED, YOU BETTER TELL ME THE TRUTH, OR I WILL STRIP YOUR CODE OFF OUR FUCKING SYSTEMS AND SHOVE IT DOWN YOUR THROAT. MY JOB IS TO HELP YOU AND YOU NEED TO BACK TO FUCK UP AND NOT GET IN THE WAY OF MY JOB OR YOU WON'T HAVE ONE ANYMORE.

Senior: Why did you refactor those ten files?

Junior: There was a method copy-pasted in every one of them, so I moved it to a utils class.

Senior: Don't you know we will have to test all of those changed classes again? Please, rollback!

Junior: ok.

... two days later ...

Senior: Why did you just copy-paste that method? Don't you know it's bad practice?

> Hey I use Arch
> Did you know that I use Arch Linux?
> Arch Linux with i3 is best
> Have I told you that I use Arch Linux?
> Windows is crap, I use Arch
> I love the rollback function in Arch Linux
> i3 GAPS on ARCH LINUX IS BEST
> My girlfriend is Arch Linux
> I fuck Arch Linux
> I name my kids Arch Linux

My biggest dev blunder. I haven't told a single soul about this, until now.

👻👻👻👻👻👻

So, I was working as a full stack dev at a small consulting company. By this time I had about 3 years of experience and started to get pretty comfortable with my tools and the systems I worked with.

I was the person in charge of a system dealing with interactions between people in different roles. Some of this data could be sensitive in nature and users had a legal right to have data permanently removed from our system. In this case it meant remoting into the production database server and manually issuing DELETE statements against the db. Ugh.

As soon as my brain finishes processing the request to venture into that binary minefield and perform rocket surgery on that cursed database my sympathetic nervous system goes into high alert, palms sweaty. Mom's spaghetti.

Alright. Let's do this the safe way. I write the statements needed and do a test run on my machine. Works like a charm 😎

Time to get this over with. I remote into the server. I paste the code into Microsoft SQL Server Management Studio. I read through the code again and again and again. It's solid. I hit run.

....

Wait. I ran it?

....

With the IDs from my local run?

...

I stare at the confirmation message: "Nice job dude, you just deleted some stuff. Cool. See ya. - Your old pal SQL Server".

What did I just delete? What ramifications will this have? Am I sweating? My life is over. Fuck! Think, think, think.

You're a professional. Handle it like one, goddammit.

I think about doing a rollback but the server dudes are even more incompetent than me and we'd lose all the transactions that occurred after my little slip. No, that won't fly.

I do the only sensible thing: I run the statements again with the correct IDs, disconnect my remote session, and BOTTLE THAT SHIT UP FOREVER.

I tell no one. The next few days I await some kind of bug report or maybe a SWAT team. Days pass. Nothing. My anxiety slowly dissipates. That fateful day fades into oblivion and I feel confident my secret will die with me. Cool ¯\_(ツ)_/¯

So. My grandma (280 km away) gave me a call.
Grandma: "Hiii. I have a problem with my PC."
Me thinking "ok. Just an ordinary tech support call from granny..."
Me: "Ok. What's the issue?"
Grandma: "Windows tells me there is no space left on drive F, non-stop"
Me: "As far as I know there is no drive F on your PC"
Grandma: "Would you be so kind to have a look?"

I agreed to log on via team viewer.

To cut an already long story short, there was this April Update which sent a bug all around the globe.

This bug exposed an internal drive Windows uses to do its Windows stuff - whatever it is.

Regularly these kind of drives are hidden from users eyes...

I finally solved it by applying a rollback on Windows.
The update is gone and with the next update there will be a fix from May to hide this drives again.

Microsoft!
I do you scare grandparents all around the globe?!??

MOTHER FUCKER IDIOTS!!!

SO I HAVE TO ROLLBACK OUR PROD DEPLOYMENT BECAUSE IM THE ONLY FUCKER WHO UNDERSTANDS THE SIMPLE CONCEPT OF "DONT PUSH UNFINISHED CHANGES"?!?
DAMN!!!! FUCK YOU ALL...

ARGH. I wrote a long rant containing a bunch of gems from the codebase at @work, and lost it.

I'll summarize the few I remember.

First, the cliche:
if (x == true) { return true; } else { return false; };
Seriously written (more than once) by the "legendary" devs themselves.

Then, lots of typos in constants (and methods, and comments, and ...) like:
SMD_AGENT_SHCEDULE_XYZ = '5-year-old-typo'

and gems like:

def hot_garbage
magic = [nil, '']
magic = [0, nil] if something_something
success = other_method_that_returns_nothing(magic)
if success == true
return true # signal success
end
end

^ That one is from our glorious self-proclaimed leader / "engineering director" / the junior dev thundercunt on a power trip. Good stuff.

Next up are a few of my personal favorites:

Report.run_every 4.hours # Every 6 hours
Daemon.run_at_hour 6 # Daily at 8am

LANG_ENGLISH = :en
LANG_SPANISH = :sp # because fuck standards, right?

And for design decisions...

The code was supposed to support multiple currencies, but just disregards them and sets a hardcoded 'usd' instead -- and the system stores that string on literally hundreds of millions of records, often multiple times too (e.g. for payment, display fees, etc). and! AND! IT'S ALWAYS A FUCKING VARCHAR(255)! So a single payment record uses 768 bytes to store 'usd' 'usd' 'usd'

I'd mention the design decisions that led to the 35 second minimum pay API response time (often 55 sec), but i don't remember the details well enough.

Also:
The senior devs can get pretty much anything through code review. So can the dev accountants. and ... well, pretty much everyone else. Seriously, i have absolutely no idea how all of this shit managed to get published.

But speaking of code reviews: Some security holes are allowed through because (and i quote) "they already exist elsewhere in the codebase." You can't make this up.

Oh, and another!
In a feature that merges two user objects and all their data, there's a method to generate a unique ID. It concatenates 12 random numbers (one at a time, ofc) then checks the database to see if that id already exists. It tries this 20 times, and uses the first unique one... or falls through and uses its last attempt. This ofc leads to collisions, and those collisions are messy and require a db rollback to fix. gg. This was written by the "legendary" dev himself, replete with his signature single-letter variable names. I brought it up and he laughed it off, saying the collisions have been rare enough it doesn't really matter so he won't fix it.

Yep, it's garbage all the way down.

The website for our biggest client went down and the server went haywire. Though for this client we don’t provide any infrastructure, so we called their it partner to start figuring this out.
They started blaming us, asking is if we had upgraded the website or changed any PHP settings, which all were a firm no from us. So they told us they had competent people working on the matter.

TL;DR their people isn’t competent and I ended up fixing the issue.

Hours go by, nothing happens, client calls us and we call the it partner, nothing, they don’t understand anything. Told us they can’t find any logs etc.

So we setup a conference call with our CXO, me, another dev and a few people from the it partner.
At this point I’m just asking them if they’ve looked at this and this, no good answer, I fetch a long ethernet cable from my desk, pull it to the CXO’s office and hook up my laptop to start looking into things myself.

IT partner still can’t find anything wrong. I tail the httpd error log and see thousands upon thousands of warning messages about mysql being loaded twice, but that’s not the issue here.
Check top and see there’s 257 instances of httpd, whereas 256 is spawned by httpd, mysql is using 600% cpu and whenever I try to connect to mysql through cli it throws me a too many connections error.

I heard the IT partner talking about a ddos attack, so I asked them to pull it off the public network and only give us access through our vpn. They do that, reboot server, same problems.

Finally we get the it partner to rollback the vm to earlier last night. Everything works great, 30 min later, it crashes again. At this point I’m getting tired and frustrated, this isn’t my job, I thought they had competent people working on this.

I noticed that the db had a few corrupted tables, and ask the it partner to get a dba to look at it. No prevail.

5’o’clock is here, we decide to give the vm rollback another try, but first we go home, get some dinner and resume at 6pm. I had told them I wanted to be in on this call, and said let me try this time.

They spend ages doing the rollback, and then for some reason they have to reconfigure the network and shit. Once it booted, I told their tech to stop mysqld and httpd immediately and prevent it from start at boot.

I can now look at the logs that is leading to this issue. I noticed our debug flag was on and had generated a 30gb log file. Tail it and see it’s what I’d expect, warmings and warnings, And all other logs for mysql and apache is huge, so the drive is full. Just gotta delete it.

I quietly start apache and mysql, see the website is working fine, shut it down and just take a copy of the var/lib/mysql directory and etc directory just go have backups.

Starting to connect a few dots, but I wasn’t exactly sure if it was right. Had the full drive caused mysql to corrupt itself? Only one way to find out. Start apache and mysql back up, and just wait and see. Meanwhile I fixed that mysql being loaded twice. Some genius had put load mysql.so at the top and bottom of php ini.

While waiting on the server to crash again, I’m talking to the it support guy, who told me they haven’t updated anything on the server except security patches now and then, and they didn’t have anyone familiar with this setup. No shit, it’s running php 5.3 -.-

Website up and running 1.5 later, mission accomplished.

Day 1: Client requested this change! Please do it!
[Make the change]
Day 2: Hold on, this is a new request and out of scope. we should reject this change.
[Rollback change]
Day 3: Client said they really need it. Please make the change.
[rollmyeyes]

My preciousssss!!

Fucking assholes! Just spent 3h debugging for bugs that weren't there.. Our client insisted we must rollback the whole update, because gui was broken.. after analysing data & testing I figure out that there must be something 'wrong' as there was no data to copy from in the first place...so there should be no bug..
Aaand here goes the best part: they didn't want to point out missing data bug, they just wanted one restriction to be removed, because it 'broke GUI', to allow for empty value on save... WTF?! How can you insist that gui is buggy & that you don't want an update, if you just want something to be optional?! Which was done immediately, one change in one js file?! Dafaaaaaq?!

Kids, English is important!! Otherwise you end up debugging ghosts for 3+ hours withou a cigarette...and waking up a coworker with bad news of rollback at half to midnight... Aaaaaaaargh!!!
сука блять 

Dev: I think I just deleted data I shouldn't have. Can you do a rollback?
Me: No, I'm a DBA, I don't touch data. Did you commit?
Dev: Yes, I committed.
Me: Your data is gone.
Dev: But don't you guys do backups?
Me: We backup the Prod servers. We only back up Dev on request. Did you request a backup in Dev?
Dev: No.
Me: ...
Dev: The Testers are going to be pissed.

Devs: We fixed a bug, so watch out for some changes.
Management: The new values are too bad, you need to change it back.
Devs: But ... it was a bug?
Management: Rollback ... now!

I swear, the moment shareholders are involved, management is just about who has the best lie.

Best code performance incr. I made?

Many, many years ago our scaling strategy was to throw hardware at performance problems. Hardware consisted of dedicated web server and backing SQL server box, so each site instance had two servers (and data replication processes in place)

Two servers turned into 4, 4 to 8, 8 to around 16 (don't remember exactly what we ended up with). With Window's server and SQL Server licenses getting into the hundreds of thousands of dollars, the 'powers-that-be' were becoming very concerned with our IT budget. With our IT-VP and other web mgrs being hardware-centric, they simply shrugged and told the company that's just the way it is.

Taking it upon myself, started looking into utilizing web services, caching data (Microsoft's Velocity at the time), and a service that returned product data, the bottleneck for most of the performance issues. Description, price, simple stuff. Testing the scaling with our dev environment, single web server and single backing sql server, the service was able to handle 10x the traffic with much better performance.

Since the majority of the IT mgmt were hardware centric, they blew off the results saying my tests were contrived and my solution wouldn't work in 'the real world'. Not 100% wrong, I had no idea what would happen when real traffic would hit the site.

With our other hardware guys concerned the web hardware budget was tearing into everything else, they helped convince the 'powers-that-be' to give my idea a shot.

Fast forward a couple of months (lots of web code changes), early one morning we started slowly turning on the new framework (3 load balanced web service servers, 3 web servers, one sql server). 5 minutes...no issues, 10 minutes...no issues,an hour...everything is looking great. Then (A is a network admin)...
A: "Umm...guys...hardly any of the other web servers are being hit. The new servers are handling almost 100% of the traffic."
VP: "That can't be right. Something must be wrong with the load balancers. Rollback!"
A:"No, everything is fine. Load balancer is working and the performance spikes are coming from the old servers, not the new ones. Wow!, this is awesome!"
<Web manager 'Stacey'>
Stacey: "We probably still need to rollback. We'll need to do a full analysis to why the performance improved and apply it the current hardware setup."
A: "Page load times are now under 100 milliseconds from almost 3 seconds. Lets not rollback and see what happens."
Stacey:"I don't know, customers aren't used to such fast load times. They'll think something is wrong and go to a competitor. Rollback."
VP: "Agreed. We don't why this so fast. We'll need to replicate what is going on to the current architecture. Good try guys."
<later that day>
VP: "We've received hundreds of emails complementing us on the web site performance this morning and upset that the site suddenly slowed down again. CEO got wind of these emails and instructed us to move forward with the new framework."

After full implementation, we were able to scale back to only a few web servers and a single sql server, saving an initial $300,000 and a potential future savings of over $500,000. Budget analysis considering other factors, over the next 7 years, this would save the company over a million dollars.

At the semi-annual company wide meeting, our VP made a speech.

VP: "I'd like to thank everyone for this hard fought journey to get our web site up to industry standards for the benefit of our customers and stakeholders. Most of all, I'd like to thank Stacey for all her effort in designing and implementation of the scaling solution. Great job Stacy!"
<hands her a blank white envelope, hmmm...wonder what was in it?>

A few devs who sat in front of me turn around, network guys to the right, all look at me with puzzled looks with one mouth-ing "WTF?"

Not entirely sure if should be said an IT company but yes I would mention SHAREit.

The well known file sharing software, I remember using it most of the times for file sharing between my Android devices, it previously was only meant for "File Sharing" but now they're just more into ads, video recommendations, news etc etc "useless stuffs".
I mean who the hell in this world would like to open a file sharing app for trending WhatsApp video recommendations? Getting notifications in a language unknown to me? Who has even asked you all for this.

I know ads gives you profit but atleast not the trending movies and all for a file sharing app.

It was good overall in the start but now not they're not working in a way they used to.

PS: image below is a SHAREit notification for someone doing black magic on a kid. See, stupid stuff! not even a news to be read just fake stuff. Atleast show some serious ads or else just rollback to your previous version.

That paranoia feeling when you push an upgrade and something stops working, you rollback, it still doesn't work..
"Has it ever worked at all?"

I can't even deal with this. We just deployed a new update to our system, and everything was going smoothly. And then, out of nowhere, we started getting a bunch of error messages and user complaints.

Why do these things always happen? We spent hours trying to figure out the source of the problem, and it turns out it was because we didn't do enough testing before the deployment. Are you kidding me?

I know that testing can be time-consuming, but seriously, this is ridiculous. It's frustrating when something like this happens, especially when we're under a tight deadline. And to make matters worse, we had to roll back the deployment and start all over again. I just want to throw my computer out the window.

Uuuugghhh!

It's 17:55... Did much work that day since I came in earlier than usual, so I could leave in time and do some shopping with the girlfriend.

A colleague comes in to my room, a tad distressed. He had accidentally ran a fixture script on a production environment database (processing a shipload of records per minute), truncating all tables...

Using AWS RDS to rollback the transaction log takes up about 20m. I had to do that about 5 times to estimate the date and time of when the fixture script ran... Since there was no clear point in time...

Finally I get to the best state of the data I could get. I log in remotely run some queries. All is well again... With minor losses in data.

I try to download a dump using pg_dump and apparently my version is mismatched with the server. I add the latest version to aptitudes source list of postgres repo and I am ready to remove and purge the current postgres client and extensions...

sudo apt-get remove post*
Are you sure? (Y/n) *presses enter and enters into a world of pain*

Apparently a lot of system critical applications start with post... T_T

That moment when you magically lose your undo history and can't rollback the shit code you just wrote.

Overtime rant, I suspect they will try to denie me half of the overtime pay for the past week cause I did more than ordered to reach their arbitrary deadline.
I will in return offer to delete half the work I did. Rollback changesets and delete the code.

Who at Microsoft thought that this is a good idea?

I wish I did not update, last update I did was a year ago and I was happy with it, now Windows feels like chaos ... Half the things are Dark themed and the other half is not. Lets see what the future holds

Also that Windows.old thing takes up 20GB that even if installation fails, it never succeeds in rolling back ... At least I was never able to rollback ...

Has anyone else noticed the blackest black theme messing up with the divider lines again?

Maybe it's just me...

Living on the edge!

One or two years ago I managed to deploy a DDL change directly on the production server. As I knew there was a backup job which will run every day at noon and at midnight. So I run my script some minutes after noon. So far so good. But somehow I tested it badly in my test environment and the UI of the application throws error after error now in production.

Well, just revert the db to the latest recovery point with the backup, I thought.

It became clear then after a couple of minutes of searching the backup folder for the db backup that there was no such file. The youngest backup file was 3 years old.

Now what happened: The backup script had a switch "simulate=true" and then simulated a successful backup on each run. Therefore the monitoring system got no alerts for not correctly executing those jobs correctly. Then the monitoring job which should do the backupfolder surveillance stuck with green, because there was a valid backup file inside. But it did not check for a specific creation date.

Now this database is the one we need for doing our daily business and is really crucial. Therefore It was easier to emergencyfix the application than doing a rollback of the db 🙄

Well, not really a data loss story, but close to one.

JetBrain's IDE and its possibility to rollback my "sudo rm *" command in the wrong folder just in 3 clicks is one of the few reasons why I didn't throw myself out of a window yet.

I am learning java at school and my teacher asked me to make a work on JTA (java transaction API). There's not a lot of tutos on it on the web so I say to myself "go on, give it a try, you'll only learn by trying."
I finally find how to make the @TransactionType, where to put the @Stateless, my test works, nice. Finally I want to try a case where it shouldn't work, just to be sure the rollback works well. The test goes and... NullPointerException. Wtf ! Normally, my catch is supposed to, well, catch the error !

And finally, I was just stupid. My catch worked great. But I put a "throw e" inside.

Now I wanna hides under blankets, cry, eat cake and never see my coworkers again.

Join's bridge: "hey man, something is wrong with your DB. our app can't connect in any environment, it started after our code release last night"

"Every other app connecting is working as expected, could you rollback your release?"

"nah, that can't be it. we validated it works"

"Then why am i on an outage bridge? call me if it's still broken after you rollback"

When the UX team decides to rollback a big change you've just finished implementing

So today I accidentally updated more than 3000 records in a table and wasn’t using transactions so I couldn’t Rollback, LOL had the whole team freak out this Monday morning and we had to pull a backup

The split second feeling of EXECUTE an UPDATE and SET value without putting WHERE clause.

I froze for a moment with cold sweat that I don't know what to do. My mind went blank.

Thank God it is just the entire list of customer details that is not relate to money issue.

Anyone can suggest the best practice for this type of accident UPDATE / DELETE?
Does using BEGIN TRANS ROLLBACK is safer way to execute?

During one of our 'pop-up' meetings last week.

Ralph: "The test code the developers are checking in is a mess. They don't know what they are doing."

ex.
var foo = SomeLibrary.GetFoo();
Assert.IsNotNull(foo);

Fred: "Ha ha..someone should talk to HR about our hiring practices. These people are literally driving the company backwards."
Me: "I think unit testing is complete waste of time."
- You could almost see the truck hit the wall and splatter watermelon everwhere..took Ralph and Fred a couple of seconds to respond
Fred: "Uh..unit testing is industry best practice. There is scientific evidence that prove testing reduces bugs and increases code quality"
Ralph: "Over 90% of our deployments are rolled back because of bugs. Unit testing will eliminate that."
Me: "Sorry, I disagree."
- Stepping on kittens wouldn't have gotten a worse look from Fred and Ralph
Fred: 'Pretty sure if you ask any professional developer, they'll tell you unit testing and code coverage reduces bugs.'
Me: "I'm not asking anyone else, I'm asking you. Find one failed deployment, just one, over the past 6 months that unit testing or code coverage would have prevented."
- good 3 seconds of awkward silence.
Ralph: "Well, those rollbacks are all mostly due to server mis-configurations. That's not a fair comparison."
Me: "I'm using your words. Unit tests reduces bugs and lack of good tests is the direct reason why we have so many failed deployments"
Boss: "Yea, Ralph...you and Fred kinda said that."
Fred: "No...we need to write good tests. Not this mess."
Me: "Like I said, show me one test you've written that would have prevented a rollback. Just one."
Ralph: "So, what? We do nothing?"
Me: "No, we have to stop worshiping this made up 80% code coverage idol. If not, developers are going to keep writing useless test code just to meet some percent. If we wrote device drivers or frameworks for other developers maybe, but we write CRUD apps. We execute a stored procedure or call a service. This 80% rule doesn't fit for code we write."
Fred: "If the developers took their head out of their ass.."
Me: "Hey!..uh..no, they are doing exactly what they are being told. Meet the 80% requirement, even if doesn't make sense."
Ralph: "Nobody told them to write *that* code."
Boss: "My gosh, what have you and Fred been complaining about for the past hour?"
- Ralph looks at his monitor and brilliantly changes the subject
Ralph: "Oh my f-king god...Trump said something stupid again ..."

At that point I put my headphones on went back to what I was doing. I'm pretty sure Fred and Ralph spent the rest of the day messaging back-n-forth, making fun of me or some random code I wrote 3 years ago (lots of typing and giggling). How can highly educated grown men (one has a masters in CS) get so petty and insecure?

fuck oracle. fuck my company.

Using Oracle VM Manager/Servers to host Oracle Phone transfer solution without support coverage from Oracle.

Requiring Unix sysadmins to update to latest release and not telling that we do not have coverage from Oracle if anything goes wrong.

Gues what.. We've updated to Oracle VM Manager/Server 3.4.5 which was released this year and it uses fucking XEN hypervisor version 4.4.4 which has been deprecated and dead since who knows when. Latest release of XEN is 4.11. But that is not an issue, whatever, enterprise, legacy software, etc.

This fucking update introduced memory leak on the hypervisor which has been reported as per xen 4.4.4 history. Furthermore, we have no support from Oracle which means that I have to dig through mailing lists and limited information on the net since oracle has freakin support wall on nearly each of the major bugs found on that shitty software.

I have no idea whether any newer version of xen will work with that old Oracle Linux kernel or not.

Furthermore, Oracle provided great documentation on how to rollback the fcking update. Reinstall the hypervisor. Riiiight. XEN does not have export/import feature.

eh

The more problems I encounter with Windows the more I have the impression these infested chimps never implemented a real system repair/restoration feature but just faked it with randomized error messages.

Now Windows 10 can't even rollback updates anymore but lands in an infinite boot loop that even a reset to an earlier restoration point won't fix.
And the Windows repair option lists two Windows installations even though there's just one.

Somehow the roles of Linux and Windows have switched nowadays, by now I had to reinstall Win10 5 times on three PCs in the last 12 months.
Windows 8 was shit too but at least it didn't fucking break within a week.

Well, this week was a week from hell. It was a short 3 day week, and all of my internal Customers, who are normally pretty reasonable, just all unloaded on me at the same time. "We need this now!" "Have you done this?" "Why didn't you do that?" "We need you to do this, because our migration takes place in 30 minutes." (first notice of the migration). And then to top everything off, I'm creating a rollback DDL, and I've spent a couple of hours pulling my hair out, because a set of columns that need rolled back aren't in Prod, so I can't roll them back, because my own DDL drops them, and broadcast my natural meltdown to the entire DevOps team, feeling like an utter jackass after I realize my mistake. And even at quitting time, they are still walking up, and texting, and emailing. Holy f**k, I'm only going to be gone four days, two of them weekend, and will be back. All of this while trying to sell my house and pack boxes and move to an apartment. Can I retire now? Looks at retirement account... Nope, I'll be working until I'm 95. Just shoot me already!

I took a job with a software company to manage their product, which was a SaaS property maintenance system for real estate, social housing, etc.

There was no charge to real estate agents to use it but maintenance contractors had to use credits to take a job, which they pre-purchased. They recharged their credit costs back to the real estate agent on their invoice).

Whether this pricing model is good or not, that's what it was. So, in I came, and one of the first things management wanted me to deal with was a long-standing problem where nobody in the company ever considered a contractor's credits could go into the negative. That is, they bought some credits once, then kept taking jobs (and getting the real estate agent to pay for the credits), and went into negative credits, never paying another cent to this software company.

So, I worked with product and sales and finance and the developers to create a series of stories to help get contractors' back into positive credits with some incentives, and most certainly preventing anyone getting negative again.

The code was all tested, all was good, and this was the whole sprint. We released it ...

... and then suddenly real estate agents were complaining reminders to inspect properties were being missed and all sorts of other date-related events were screwed up.

I couldn't understand how this happened. I spoke with the software manager and he said he added a couple of other pieces of code into the release.

In particular, the year prior someone complained a date on a report was too squished and suggested a two-digit year be used. Some atrocious software developer worked on it who, quite seriously, didn't simply change the formatting of that one report. No, he modified the code everywhere to literally store two-digit years in the database. This code sat unreleased for a year and then .... for no perceivable reason, the moron software manager decided he'd throw it into this sprint without telling me or anybody else, or without it being tested.

I told him to rollback but he said he'd already had developers fixing the problems as they came up. He seemed to be confident they'd sort it out soon.

Yet, as the day went on more and more issues arose. I spoke to him with the rest of the management team and said we need to revert the code but he said they couldn't because they hadn't been making pull requests that were exclusive to specific tickets but instead contained lots of work all in one. He didn't think they could detangle it and said the only way to fix was "play whack-a-mole" when issues came up.

I only stayed in that company for three months; there was simply way too much shit to fix and to this day I still have no idea the reasoning that went on in the head of anyone involved with that piece of code.

Me: Ok I've updated the docs, I'll open a PR with the changes

Maintainer: Looks great! Can you remove the changes to the package-lock.json? (I assume it got updated when you ran npm install to start the webserver)

Me: Ok sure, I'll update it soon

And this is where the troubles begin. The file was commited 2 commits ago, so I have to roll back to then. However, the remote repository has been updated since then, so I git fetch to keep up to date.

This makes the rollback a hell of a lot harder, so I run git log to see the history. I try a reset, but I went back to the wrong commit, and now a shit ton of files are out of sync.

I frantically google 'reset a git reset', and come across the reflog command. Running that fucks things up even worse, and now so much shit is out of sync that even git seems confused.

I try to fix the mess I've created, and so I git pull from my forked repo to get myself back to where I was. Git starts screaming at me about out of sync files, so I try to find a way to overwrite local changes from the origin.

And by this point, the only way to describe what the local repo looks like is a dumpster fire clusterfuck that was involved in a train wreck

I resolved the mess by just deleting the local copy and git cloning again from my fork.

I gotta learn how to use Git better

Not much of a SQL Dev, still an apprentice and had a basic run throughs. Client needed a migration script to run, which I was assigned. Took me a good 6/7 days to make, transfer over a secure (and VERY slow) network took 2 hours. Infrastructure 3rd party took 2 days to clear and run. After all that process. I then realise, I left the fucking rollback in

That moment when something goes wrong in the server for the first time at 3AM, your last manual backup was 5 hours ago and you stay up until 7AM to rollback and also setup incremental backups every 15 minutes... FML xD

C/C++ is so fucking unreadable. Who had the cancer idea to use abbreviations for EVERYTHING?? If a word is longer than 3 chars you can be sure those psychologically unstable devs will use a shortcut.
Pointer ==> ptr
test_and_set ==> fucking t_a_s
rollback ==> rb
I don't want to play a shitty word guessing game for every fucking function in your horribly documented api.

Why can't my team including my boss learn to stop making assumptions... And mixing seperate issues into one...

If there's a fucking production issue, first step is to reproduce it... AKA ask what the user did and what he expects....

Not...
User: hey we call this url and get an error
Dev: ok rollback

Monday morning: "Hey uuuh the client receives 3 mails instead of one and only one of them is good, there's a problem, go fix it"

Yesterday, me: - "Hey I've been looking everywhere, made hundreds of tests, there's a problem with the files attached to the mail, they're unreadable"
-" I told you it's in the code, you didn't look deep enough"

This evening: "Umm it seems that there's indeed a permission issue. So I'm gonna rollback everything you've done since Monday"

One year. One year and I live this hell.

Ok so first off i cant get my damn mic working, despite every setting change i can find and every driver update i can do and every damn mic i've used. Then i try updating windows ten because maybe thats the problem, sure enough after i've done that my whole damn computer is messed up. Even opening folders or applications take between five and ten minutes. Not to mention i can't open my windows settings to rollback. I literally build and fix computers for a living and yet i have not clue what else i can do at this point. Think i might renounce technology and float off into the Atlantic on a plank of wood :/

Allright, so now I have to extend a brand new application, released to LIVE just weeks ago by devs at out client's company. This application is advertised as very well structured, easy to work on, µservices-based masterpiece.

Well either I lack a loooot of xp to understand the "µservices", "easy to work on" and "well structured" parts in this app or I'm really underpaid to deal with all of this...

- part of business logic is implemented in controllers. Good luck reusing it w/o bringing up all the mappings...
- magic numbers every-fucking-where... I tried adding some constants to make it at least a tiny bit more configurable... I was yelled at by the lead dev of the app for this later.
- crud-only subservices (wrapped by facade-like services, but still.. CRUD (sub)services? Then what's a repository for...?). As a result devs didn't have a place where they could write business logic. So business logic is now in: controllers (also responsible for mapping), helpers (also application layer; used by controllers; using services).
- no transactions wrapping several actions, like removing item from CURRENT table first and then recreating it in HISTORY table. No rollback/recovery mechanism in service layers if things go South.
- no clean-code. One can easily find lines (streams) 400+ cols long.
- no encapsulation. Object fields are accessed directly
- Controllers, once get result from Services (i.e. Facade), must have a tree of: if (result instanceof SomeService.SomeSubservice1.Item1) {...} else if (result instanceof SomeService.SomeSubservice2.Item4) {...} etc. to build a proper DTO. IMO this is not a way to make abstraction - application should NOT know services' internals.
- µservices use different tables (hats off for this one!) but their records must have the same IDs. E.g. if I order a burger and coke - there are 2 order items in my order #442. When I make a payment I create an invoice which must have an id #442. And I'm talking about data layer, not service or application (dto)! Shouldn't µservices be loosely coupled and be able to serve independently...? What happens if I reuse InvoiceµService in some other app?

What are your thoughts?

Had been trying to get the latest build rolled out for the past 3 days. Every morning, I wake up around 6am, drink a nice cup of coffee, while listening to Ariana Grande (I don't know why, but for some reason, she randomly started coming on my playlist a lot) and start rolling out, and sure enough new errors start spiking, ultimately rollback.

Conclusion: don't listen to Ariana Grande when rolling out to production! 🤔

Can someone rollback the 2020 patch already ? It's buggy

That feeling, when your changes almost caused a rollback and you hear words like "revenue impact" being discussed 😨

<sanityCheck> //asking for a friend

Some clever b*****ds wrecked a section of our production mysql db. To fix it I need to rollback the affected records 2 weeks - around 50/300 tables are affected, the other data must remain intact.

Currently my plan is to take a 2 week old dump and cherry pick the data I need from it, then combine it with a dump of the db in it's current state, drop the db and recreate it.

I know this approach will work - but it's risky, a pain in the ass and dealing with 300mb text files is tedious so since I only need to start in around 8 hours I figured It wouldn't hurt to post my approach and see if anyone thinks my plan is borderline retarded.

If you have any advice .etc that will make my life easier I would greatly appreciate it.

So in your opinion...

- is there a better/safer way?
- do you know of any db dump merge tools?
- have a recommended (linux) text editor for large text files?
- have you made any personal mistakes/fuck ups in the past you think I should avoid?
- am I just being a moron and overthinking this?
- if I am being a moron - In your humble opinion has the time come for me to give up all hope and pursue my dream of becoming a professional couch surfer?

</sanityCheck>

Note: Alternatively, if your just pissed that my rant is asking for a solution instead of simply trashing the people that created my situation and your secretly wishing it was on SO where it belongs so you can moderate/edit/downvote/mark the shit out it, feel welcome to troll me in the comments (getting dev advice just doesn't feel reliable without a troll - you matter to me). Afterwards If your panties are still in a bunch I'll post it on SO and dm a link to you to personally moderate - my days already fucked and I wouldn't want to ruin yours too.

!dev

Battle of titans:

Me vs Windows Update.

No, this time not about automatic restarts interrupting my work because I've postponed "important" updates for far too long.

Well, technically I lied.

Last evening windows force rebooted to install updates. OK then, I'll go home. This morning I saw windows update frozen at some percentage.

This evening I've got 3 forced restarts in 5 minutes. Only notepad++ with an opened and unsaved document saved my life.

Apparently, windows update follows a simple logic:

if(user.IsUsingComputer && lastUpgradeAttempt.HasFailedWithTotalFreeze && versionUpgrade.CanInstallOnlyWithFreshWindowsInstallation)
{
update();
}

Seriosly. Why am I forced to install an update only to see it lock up the entire computer and rollback after I intervene? Last time I had this issue only reinstalling windows helped, which costs about 2 days. Excluding one day to try suggested solutions, remove drivers, av scanner, etc.

Is it really that hard to implement logic like:

If(lastUpdateAttempt.HasFuckedUp)
{
windowsUpdate.FuckOff();
}

New update available on your phone...

Wait people to teste it before installing. It look stable, ok i will install it

After the update

Fucking shit the clock is on the left side 🤮 what the fuck is this shit android... The settings menu is now a ducking mess. The finger print scan is so slow i can cooke an egg before he unlock the phone. The multi tasking screen is like iphone and look like a garbedge. Some applications rushing running.

I fixed a bug properly... Took down an entire application systems, sometimes you just gotta monkey patch that shit.

So it was a 15 year old cold fusion system and chrome had deprecated some window pop up feature, so I tracked it to the shared function that triggered this, fixed it there, tested it and even got it all past qa.

Turned out some of the other modules on the app had some other logic around this that made it not work there, they had implemented the fallback check without any fallback logic.

Time to rollback a 3 week sprint...

(defn rant []
(go
(let [feature-pool (atom [:ui-fixes])]
(loop []
(<! wait-for-release-week)
(->>
shit-load-of-features
(reset! feature-pool)
develop-features
ask-for-feature-status
deploy-in-last-minute
find-a-bug-and-rollback release
(recur))))

(rant)

so there was this issue regarding our company's system which tends to be a problem for sometime now, its a recurring issue caused by the data that the users needs to encode to the system

today another issue arised, our senior supervisor, not knowing that this issue was already recurring and there is already a documented step procedure on how to address it, suggested or come up with a another solution which would task one of our co-developer to push a temporary code to production during business hours just to accommodate the issue and rollback the code after

take note that its during business hours and more than a hundreds of branches of the company are using the said system

what was he thinking !!

thankfully one of our colleagues voiced out explaining that this issue was already recurring and already has a procedural solution, but still our brainy-know-it-all-stubborn-close-minded heck of a supervisor insisted that the solution has computational impact and still insisted that they push a temporary code to the production, what an idiot!!

fast forward our colleagues ended up standing their ground, even if our supervisor is highly doubtful at them, and executed the already established solution instead of pushing a temporary code to the production which was such a bullshit idea

damn those close minded people they shouldn't have reach that position in the first place!!

I personally don't have a funny dev sin story (not that I didn't commit any).
My internship colleague should update a value of a row in production. So he wrote a SQL command and forgot the where clause. This was the first time the company tested there rollback mechanism and it didn't work. For the next 2 weeks my colleague was busy updating 2000ish rows to make it work again

Run untested SQL on prod db. Yeah I love adrenaline

Why are some people too lazy to write commit messages and just amend onto eg. A merge...

How is it possible?
I installed a treeview module. I followed all the documentation and the module is showing. Perfect!

Me: create 3 roots,
Module: all is ok.
Me: 😊create 2 nodes,
Module: everything is fine.
Me: 😁3rd node
Module: ... Kaput. tree displays like the nodes are in random order.

Me: 🤨Check the database, and fix the set.
Module: Aah much better.
Me: 😃Try to change an int.
Module: Noooooo! Big mistake!!!
Me: 🤔Ok, ok, ok, rollback! 😧
Module: still in random situation
Me: 😶 and now what?

19:30, still in the office fixing server issue. Lync message pops up from "let's call him the ba", help! Need to rollback a change on someone else's project that has just been deployed changing the username and password from Xxxxxx Xxx2016 to xxxxxx xxx2016 to stop people from accessing it.... Genius design! By the way, I've just had a look at that 40 page doc of changes you emailed him. When do you want it? Tuesday. Wtf? (Bank holiday here on Monday so one day to make the changes) Ok, if you're busy, it's only text changes so can some of us help? It's only a find and replace right? Er, do you have visual studio? What's that? FU

Me to my team: demo to the client is postponed, we'll show it the day after tomorrow.

Them: nice, then we can put in production also the new feature xyz.

Me: mmm... Is it tested and everything ok? Then yes, let's deploy it.

Bad decision. Now everything is not working. Rollback needed!

Wtf is wrong with AWS CDK? I add one EC2 instance and it deletes all my queues and api resources, then fails to create my EC2, does some fucking rollback then attempts and fails at some fucking rollback clean up.
So it nuked my entire fucking stack because why?
Because FUCKING WHY JEFF. WHY IS YOUR ROCKET SHAPED LIKE A PENIS AND WHY IS YOUR SHITTY FUCKING DEPLOYMENT SYSTEM FUCKING WITH ME LIKE THIS.
I can’t do this anymore. I’ve been doing this for months now. I really don’t know whether to laugh or have a mental break down.

Complete Disaster Kunt. That’s what I’m calling this shit from now on. I just don’t get how it can fuck itself up so much

Arg. I messed up bad today. Updated 17k records in database by mistake and had to rollback and apologize...

I won't rush on a task for rest of the day

When the deploy didn't work as expected and nobody is able to rollback...

Any advice on how to deal with gatekeeping developers? How to deal with red tape?

I work with people that are resistant to code and process change. Continuous pedantic pushback on nearly anything; one raised a fuss over metrics not being satisfactory at a 5% threshold for alerting stating that 4.99% metrics variance wouldn't trigger an alert.

It's genuinely as though my coworkers are all scared of code based on the way they behave. They don't seem to code very often either.

I'm someone that codes quickly but I have to constantly write proposals for quite literally any change to the codebase. Even IF there were issues we could always rollback (and even then we have metrics, alerts, canary rollouts, feature flags, etc etc). As a quick aside, my pace isn't related to the pushback nor experience/skill level. Just affects my morale and mental heth to be blocked.

I can communicate effectively and I try to be as clear as possible in my proposals but this is absolutely driving me up the wall and killing my motivation.

This is a faang-level company and I would've expected better.

Any advice on how to best navigate this? Is this the norm???

Always feels cringy when this colleague of mine calls the devs of their application as "<insert_application_name> champions". Like champions of what? Champions of production rollback?

Just spend two fucking days debugging a few methods in our program. I used unit tests to call those methods (don't want to navigate through the complete program workflow)...
Yeah, guess what: the test cleanup includes a fucking rollback transaction function. So NONE NADA ZERO FUCKING CHANGES WILL BE PERSISTED. Fuck me

We need to rollback

Best Practices for Implementing CI/CD Pipelines in a Microservices Architecture
Hello everyone,

I'm currently working on implementing CI/CD pipelines for our microservices-based application and I'm looking for some best practices and advice. Our architecture consists of several microservices, each with its own repository and development team. We've been using Jenkins for our build automation, but we're open to exploring other tools if they offer better integration or features.

Here are a few specific areas where I need guidance:

Pipeline Design: How should we structure our CI/CD pipelines to handle multiple microservices efficiently? Should each microservice have its own pipeline, or is there a better approach?

Deployment Strategies: What deployment strategies work best for microservices to ensure zero downtime and easy rollback? We're considering blue-green deployments and canary releases, but would love to hear about your experiences.

Tool Recommendations: Are there any CI/CD tools or platforms that are particularly well-suited for microservices architectures? We're particularly interested in tools that offer good integration with Kubernetes.

Testing and Quality Assurance: How do you handle testing in a microservices environment? What types of tests do you include in your CI/CD pipelines to ensure the quality and stability of each microservice?

Monitoring and Logging: What are the best practices for monitoring and logging in a microservices setup? How do you ensure that you have visibility into the performance and issues of individual microservices?

Any insights, resources, or examples from your own implementations would be greatly appreciated. Thank you in advance for your help!

Linux noob... just accidentally made myself (not root) owner on all files. seemed fine for a while, but I couldn't sudo or even ssh. had to rollback my server nearly a month to fix it 😭

Junior: "I've deleted the unnecessary migration files after a rollback, just like you recommended me, but the DB-table that tracks which migrations where run still shows them after running migrations"
Me: "That shouldn't happen 🤔 How did you delete them?"
J: "I deleted the code in the Up and down scripts"....

Is it so hard to understand that a migration will "run" even if it doesn't do anything?

Branch the branch, deploy the rollback!

nobody has any clue how to rollback or how the options we have work under the hood

how good is our devops out of 10?

Fucking stupid spring-boot-devtools dependency !!!

Started work at 11AM and was working on a Rest API system using Spring Boot. Got to know about Spring Boot dev tools and added it to my project.

Later in the evening my endpoints started throwing exceptions for no god damn reason. Invalidated the caches, restarted my IDE and laptop. Rolled back my code to almost vanilla branch !!! YET THE ENDPOINTS KEPT THROWING RANDOM EXCEPTIONS.

This went on till 1:30 AM (I live in a country where work-life balance is not a thing for software developers :)). Frustrated, the last thing I tried was to rollback the devtools dependency from my POM file. AND MY ENDPOINTS STARTED WORKING AGAIN 🤬

What the actual ffffuhkkk !!!!

To all those who contribute to spring-boot-devtools, you guys are doing a great job and it isn’t a personal attack towards you (I really mean it). It just messed up my project in some way and I was extremely frustrated.