"You gave us bad code! We ran it and now production is DOWN! Join this bridgeline now and help us fix this!"

So, as the author of the code in question, I join the bridge... And what happens next, I will simply never forget.

First, a little backstory... Another team within our company needed some vendor client software installed and maintained across the enterprise. Multiple OSes (Linux, AIX, Solaris, HPUX, etc.), so packaging and consistent update methods were a a challenge. I wrote an entire set of utilities to install, update and generally maintain the software; intending all the time that this other team would eventually own the process and code. With this in mind, I wrote extensive documentation, and conducted a formal turnover / training season with the other team.

So, fast forward to when the other team now owns my code, has been trained on how to use it, including (perhaps most importantly) how to send out updates when the vendor released upgrades to the agent software.

Now, this other team had the responsibility of releasing their first update since I gave them the process. Very simple upgrade process, already fully automated. What could have gone so horribly wrong? Did something the vendor supplied break their client?

I asked for the log files from the upgrade process. They sent them, and they looked... wrong. Very, very wrong.

Did you run the code I gave you to do this update?

"Yes, your code is broken - fix it! Production is down! Rabble, rabble, rabble!"

So, I go into our code management tool and review the _actual_ script they ran. Sure enough, it is my code... But something is very wrong.

More than 2/3rds of my code... has been commented out. The code is "there"... but has been commented out so it is not being executed. WT-actual-F?!

I question this on the bridge line. Silence. I insist someone explain what is going on. Is this a joke? Is this some kind of work version of candid camera?
Finally someone breaks the silence and explains.

And this, my friends, is the part I will never forget.

"We wanted to look through your code before we ran the update. When we looked at it, there was some stuff we didn't understand, so we commented that stuff out."

You... you didn't... understand... my some of the code... so you... you didn't ask me about it... you didn't try to actually figure out what it did... you... commented it OUT?!

"Right, we figured it was better to only run the parts we understood... But now we ran it and everything is broken and you need to fix your code."

I cannot repeat the things I said next, even here on devRant. Let's just say that call did not go well.

So, lesson learned? If you don't know what some code does? Just comment that shit out. Then blame the original author when it doesn't work.

You just cannot make this kind of stuff up.

A wild Darwin Award nominee appears.

Background: Admins report that a legacy nightly update process isn't working. Ticket actually states problem is obviously in "the codes."

Scene: Meeting with about 20 people to triage the issue (blamestorming)

"Senior" Admin: "update process not working, the file is not present"
Moi: "which file?"
SAdmin: "file that is in ticket, EPN-1003"
Moi: "..." *grumbles, plans murder, opens ticket*
...
Moi: "The config dotfile is missing?"
SAdmin: "Yes, file no there. Can you fix?"
Moi: "Engineers don't have access to the production system. Please share your screen"
SAdmin: "ok"
*time passes, screen appears*
Moi: "ls the configuration dir"
SAdmin: *fails in bash* > ls

*computer prints*
> ls
_.legacyjobrc

Moi: *sees issues, blood pressure rises* "Please run list all long"
SAdmin: *fails in bash, again* > ls ?
Moi: *shakes* "ls -la"
SAdmin: *shonorable mention* > ls -la

*computer prints*
> ls -la
total 1300
drwxrwxrwx- 18 SAdmin {Today} -- _.legacyjobrc

Moi: "Why did you rename the config file?"
SAdmin: "Nothing changed"
Moi: "... are you sure?"
SAdmin: "No, changed nothing."
Moi: "Is the job running as your account for some reason?"
SAdmin: "No, job is root"
Moi: *shares screenshot of previous ls* This suggests your account was likely used to rename the dotfile, did you share your account with anyone?
SAdmin: "No, I rename file because could not see"
Moi: *heavy seething* so, just to make sure I understand, you renamed a dotfile because you couldn't see it in the terminal with ls?
SAdmin: "No, I rename file because it was not visible, now is visible"
Moi: "and then you filed a ticket because the application stopped working after you renamed the configuration file? You didn't think there might be a correlation between those two things?"
SAdmin: "yes, it no work"
Interjecting Director: "How did no one catch this? Why were there no checks, and why is there no user interface to configure this application? When I was writing applications I cared about quality"
Moi: *heavy seething*
IDjit: "Well? Anyone? How are we going to fix this"
Moi: "The administrative team will need to rename the file back to its original name"
IDjit: "can't the engineering team do this?!"
Moi: "We could, but it's corporate policy that we have no access to those environments"
IDjit: "Ok, what caused this issue in the first place? How did it get this way?!"

TFW you think you've hit the bottom of idiocy barrel, and the director says, "hold my mango lassi."

toxic workplace; leaving

I haven't wanted to write this rant. I haven't even wanted to talk to anyone (save my gf, ofc). I've just been silently fuming.

I wrote a much longer rant going into far too much detail, but none of that is relevant, so I deleted it and wrote this shorter (believe it or not) version instead. And then added in more details because details.

------

On Tuesday, as every Tuesday, I had a conference call with the rest of the company. For various, mostly stupid reasons, the boss yelled at and insulted me for twenty minutes straight in front of everyone, telling me how i'm disorganized, forgetful, how can't manage my time, can't manage myself let alone others, how I don't have my priorities straight, etc. He told the sales team to get off the call, and then proceeded to yell and chew at me for another twenty minutes in front of the frontend contractor about basically the same things. The call was 53 minutes, and he spent 40 minutes of it telling me how terrible I've been. No exaggeration, no spin. The issues? I didn't respond to an email (it got lost in my ever-filling inbox), and I didn't push a very minor update last week (untested and straight to prod, ofc). (Side note: he's yelled at me for ~15 minutes before for being horribly disorganized and unable to keep up on Trello -- because I had a single card in the wrong column. One card, out of 60+ over two boards. Never mind that most have time estimates, project tags, details, linked to cards on his boards, columns for project/qa/released, labels for deferred, released to / rejected from qa, finished, in production, are ordered by priority, .... Yep. I'm totes disorganized.)

Anyway, I spent most of conference call writing "Go fuck yourself," "Choke on a cat and die asshole," "Shit code, low pay, and broken promises. what a prize position," etc. or flipping him off under the camera on our conference-turn-video-call (switched due to connection issues, because ofc video is more stable than audio-only in his mind).

I'm just.
so, so done.

I did nothing the rest of the day on Tuesday, and basically just played games on Wednesday. I did one small ticket -- a cert replacement since that was to expire the next day -- but the rest was just playing CrossCode. (fun game, fyi; totally recommend.)

Today? It's 3:30pm and I can't be bothered to do anything. I have an "urgent" project to finish by Monday, literally "to give [random third party sales guy] a small win". Total actual wording. I was to drop all other tasks (even the expiring cert lol) and give this guy his small win. fucking whatever. But the project deals with decent code -- it's a minor extension to the first project I did for the company (see my much earlier rants), back when I was actually applying myself and learning something (everything) new, enjoying myself, and architecting+writing my own code. So I might actually do the project, but It's been two days and I haven't even opened single file yet.

But yeah. This place is total and complete shit. Dealing with the asshole reminds me of dealing with my parents while growing up, and that's a subject I don't want to broach -- far too many toxic memories.

So, I'm quitting as soon as I find something new.
and with luck, this will be before assface hires my replacement-to-be, and who will hopefully quit as soon as s/he sees the abysmal codebase. With even more luck, the asshole king himself will get to watch his company die due to horrible mismanagement. (though ofc he'll never attribute it to himself. whatever.)

I just never want to see or think about him again.
(nor this fetid landfill of a codebase. bleh.)

With luck, this will be one of my last rants about this toxic waste dump and its king of the pile.
Fourty fucking minutes, what the fuck.

So my boss sent me a msg at 00:05 to update production.

And here I was at 7:30am reading it with a poker face thinking "no dude, my working hours are not like that, lol. Get a life."

Doot doot.

My day: Eight lines of refactoring around a 10-character fix for a minor production issue. Some tests. Lots of bloody phone calls and conference calls filled with me laughing and getting talked over. Why? Read on.

My boss's day: Trying very very hard to pin random shit on me (and failing because I'm awesome and fuck him). Six hours of drama and freaking out and chewing and yelling that the whole system is broken because of that minor issue. No reading, lots of misunderstanding, lots of panic. Three-way called me specifically to bitch out another coworker in front of me. (Coworker wasn't really in the wrong.) Called a contractor to his house for testing. Finally learned that everything works perfectly in QA (duh, I fixed it hours ago). Desperately waited for me to push to prod. Didn't care enough to do production tests afterwards.

My day afterwards: hey, this Cloudinary transform feature sounds fun! Oh look, I'm done already. Boo. Ask boss for update. Tests still aren't finished. Okay, whatever. Time for bed.

what a joke.

Oh, I talked to the accountant after all of this bullshit happened. Apparently everyone that has quit in the last six years has done so specifically because of the boss. Every. single. person.

I told him it was going to happen again.

I also told him the boss is a druggie with a taste for psychedelics. (It came up in conversation. Absolutely true, too.) It's hilarious because the company lawyer is the accountant's brother.

So stupid.

Seven months ago:
===============
Project Manager: - "Guys, we need to make this brand new ProjectX, here are the specs. What do you think?"

Bored Old Lead: - "I was going to resign this week but you've convinced me, this is a challenge, I never worked with this stack, I'm staying! I'll gladly play with this framework I never used before, it seems to work with this libA I can use here and this libB that I can use here! Such fun!"

Project Manager: - "Awesome! I'm counting on you!"

Six months ago:
====================
Cprn: - "So this part you asked me to implement is tons of work due to the way you're using libA. I really don't think we need it here. We could use a more common approach."

Bored Old Lead: - "No, I already rewrote parts of libB to work with libA, we're keeping it. Just do what's needed."

Cprn: - "Really? Oh, I see. It solves this one issue I'm having at least. Did you push the changes upstream?"

Bored Old Lead: - "No, nobody uses it like that, people don't need it."

Cprn: - "Wait... What? Then why did you even *think* about using those two libs together? It makes no sense."

Bored Old Lead: - "Come on, it's a challenge! Read it! Understand it! It'll make you a better coder!"

Four months ago:
==============
Cprn: - "That version of the framework you used is loosing support next month. We really should update."

Bored Old Lead: - "Yeah, we can't. I changed some core framework mechanics and the patches won't work with the new version. I'd have to rewrite these."

Cprn: - "Please do?"

Bored Old Lead: - "Nah, it's a waste of time! We're not updating!"

Three months ago:
===============
Bored Old Lead: - "The code you committed doesn't pass the tests."

Cprn: - "I just run it on my working copy and everything passes."

Bored Old Lead: - "Doesn't work on mine."

Cprn: - "Let me take a look... Ah! Here you go! You've misused these two options in the framework config for your dev environment."

Bored Old Lead: - "No, I had to hack them like that to work with libB."

Cprn: - "But the new framework version already brings everything we need from libB. We could just update and drop it."

Bored Old Lead: - "No! Can't update, remember?"

Last Friday:
=========
Bored Old Lead: - "You need to rewrite these tests. They work really slow. Two hours to pass all."

Cprn: - "What..? How come? I just run them on revision from this morning and all passed in a minute."

Bored Old Lead: - "Pull the changes and try again. I changed few input dataset objects and then copied results from error messages to assertions to make the tests pass and now it takes two hours. I've narrowed it to those weird tests here."

Cprn: - "Yeah, all of those use ORM. Maybe it's something with the model?"

Bored Old Lead: - "No, all is fine with the model. I was just there rewriting the way framework maps data types to accommodate for my new type that's really just an enum but I made it into a special custom object that needs special custom handling in the ORM. I haven't noticed any issues."

Cprn: - "What!? This makes *zero* sense! You're rewriting vendor code and expect everything to just work!? You're using libs that aren't designed to work together in production code because you wanted a challenge!?? And when everything blows up you're blaming my test code that you're feeding with incorrect dataset!??? See you on Monday, I'm going home! *door slam*"

Today:
=====
Project Manager: - "Cprn, Bored Old Lead left on Friday. He said he can't work with you. You're responsible for Project X now."

When you're a junior sysadmin but still have to maintain ALL the production server:

How it looks:

$ sudo apt-get update

How it feels:

& sudo [ $[ $RANDOM % 6 ] == 0 ] && rm -rf / || echo *Click*

For some idiotic reason, I ran
UPDATE users
SET email="myname@mycompany.com"; in production.

No where clause. Oh drat.

Contrary to most people I really love to receive email related to jobs when I'm in holiday. I keep important alerts on.

It's like:
email: ***urgent, server down***
me (sipping mojito by the pool): fuck them. let's them deal with that
email: ***requirements all wrong, must develop the feature again***
me (enjoying a dinner): oh, I told them 100 times!, fuck all of them, work for me now, stupid moron.
email: I destroyed by mistake the db with an update..."
me (dancing like crazy): ahahaha I told you that support guys should not have access to production db, fuckfuck you, fix it yourself!!!

and so on..... I don't know, it just boost my pleasure during holiday.

One:

Had a stack of harddrives with my important data, two USB drives and a 4.7gb disc, two or three cloud storage accounts.

Needed a restore:
Knocked the stack of hard drives onto the floor (all broken), stood on one of the flash drives, found the other one in a pocket of a pair of trousers which just came out of the washing machine, dvd too scratched to read and couldn't verify my cloud storage account because I lost the password to the connected email account and the backup email account to verify that one didn't exist anymore. Fucking hell.

Two:

Production database with not that much yet but at least some production data which wasn't backupped.
Friend: can I reboot the db machine?
Me: yup!
Friend: what's the luks crypt password?
Me: 😯😐😓😫😲😧😭

End of story 😅

For the record, the first one actually happened (I literally cried afterwards) and that taught me to update my recovery email addresses more often!

"Read-only Friday" rule: On Fridays, you don't deploy new versions, don't merge code into production, don't update databases, and a lot of others "DON'Ts"

FUCKING TELEGRAM FUCK YOU STAY IN YOUR FUCKING API DOCUMENTATION AND STOP FUCKING TESTING YOUR SHIT ON A PRODUCTION SYSTEM WHY WOULD YOU DO THAT FUCK OFF WHY AM I EVEN DEVELOPING SHIT FOR YOUR PLATFORM ANYMORE WHEN FOLLOWING YOUR DOCUMENTATION LEADS TO FUCKING ERRORS AND WE HAVE TO DECOMPILE AND REVERSE ENGINEER YOUR FUCKING "OPEN SOURCE" APPS BECAUSE YOU DONT EVEN BOTHER TO FUCKING UPDATE THE SOURCE CODE ONCE A YEAR WHAT THE FUCK

Thank you for your attention

Old story.

Colleague: "I'm gonna update some tables and stored procedures later"

*phone rings two hours later*

Colleague: "Hi! Just wanted to tell you I accidently dropped all our production databases. Have fun recovering them 😊"

Well shit.

Summary: Burnout, and everything's broken.

I don't feel like doing a damn thing today. I look at the code and cringe. I look at Slack and think "ugh. i can't." Mental capitals are even too much work.

(I've started reading "Zen and the Art of Motorcycle Maintenance" to try and combat burnout. I'll write a rant/story about it here if I find it helpful. but all I want to do today is drink tea and read.)

But onto the story:

Heroku is deprecating support for and will automatically upgrade any old verisons of Postgres running on its platform after August something (like five days from now).

I performed the upgrade to PG10 on Sunday (and late into the night), provisioning a new follower, blah blah blah.

However, the version of Rails we're using (4.2.x) doesn't support PG10 sequences, so I manually added in support via a monkeypatch. I did this on our QA servers first, obviously, and everything worked as expected. After half a day of no issues, I did the same on production, and again: everything worked as expected.

But today? I keep hearing about new things that are broken. One specific type of alert doesn't work for one specific person (wat). Can't send [redacted] at all. Can't update merchants! Yet there are magically no errors logged.

That last one (well, two) are just great; let me explain: when there's an error concerning merchants, the error gets caught, isn't logged or recorded anywhere so it just disappears, and the rescue block triggers a json response instead and happily exits. This is for an internal admin tool, so returning a user-friendly error is kinda stupid anyway, but masking what actually happened? fuck that dev with an obelisk made from spikes and solidified pain. That json response is also lovely: it's a 200 OK returning {status: 1, data: "[generic message containing incorrect IT jargon]"}. Doesn't even say "error" anywhere. Bloody everything about this pattern is absolutely wrong. Even the friggin' text.

Fucking hell. I want to pipe the entire codebase into shred and walk out the door.

But I digress. So many things are broken, my motivation is wanning to a sliver, and I have a conference call today where I'll undoubtedly be asked why everything is on smoking and/or on fire, and my huge and overly productive week last week will ofc mean nothing by contrast.

Ugh.

`shred ~/dev/work -zfu -n 32 &; ./brew tea --hot && wine ~/takeabreak.exe`

Always use SELECT-query with the same conditions before you DELETE/UPDATE in a production database.

Yep. So the dev teams boss says it's fine to run a production environment on a single Windows instance with the db on that same instance, which they already totally lost once from a reboot after an auto update before I came along tasked with fixing the cluster fuck they created.

This from a man who somehow runs a dev team while using gmail via the web because he can't use an email client, uses email to track tasks but can't because they get lost amongst his 3000+ unread emails, has a screen dirtier than a hookers vag on half priced Tuesday, and got a new laptop but had to get his daughter to set it up and transfer his data because he couldn't.

But ok... you have a degree, You must know what you're doing.
It's ok though, I'll keep covering your incompetent ass while you keep raping the company because no one listens.
Peoples ignorance and arrogance astounds me.

On an afternoon the day before delivery, we discovered a crashing bug. At around 2 AM, we had found the cause and fixed it. A short sleep at home, then back to office at 8 AM because the delivery was 200 devices containing that software, and they had to be updated manually because production had put in the old image.

We seized all available computers, even those from marketing who were... surprised. Half-way in the update, we calculated that we wouldn't have enough time until the freight service would show up.

So we asked the secretary that she should be a bit flirty to the parcel guy, invite him to a coffee and chat around to buy us more time. We closed the last parcel just when he figured that he had to continue with his tour.

Type T ( something something... )
...
Type * TP = T;
TP.do()

> Be me
> Production code
> Update Type structure
> SEGV
> Wtf why was this even working
> Ask senior
> senior y u do dis
> bc
> Why were no errors
...
> mfw
> He says

"I added -fpermissive and forgot..."

On production since May

...

...

...

... 2007

DATA COMMUNICATION BETWEEN SERVER AND AN APP..

1. Write all data into files,
2. Make the files as zip,
3. Send the zip to server,
4. Server will unzip the zip file,
5. Read all the data line by line from the files and update the data.

** TRUST ME, THIS IS A PRODUCTION APP I HAVE SEEN FROM A CLIENT **

My worst dev sin was leaving out the 'WHERE' in a SQL update statement on a production database

Set every booking to be owned by the same sales consultant 👀

Every once in a while, execute a SQL update query on a production DB without a where clause.

I messed up carelessly in production. Learnt how SQL queries bite you in the ass when it knows you are under pressure.

Was hosting an online quiz kinda thing during my college techfest. Tens of thousands of people participating.

Using MySQL as database and thousands of queries were being executed. Everyone were pretty excited as the event just opened up.

None of the teams could solve one particular level. Turns out the solution was wrong and was asked by the organisers to change the solution for that particular level. Usual stuff, right?

Was too lazy to open up the web UI for the back office and so, straight ahead logged in to the MySQL server and ran the UPDATE query on the table consisting of the solutions.

It had been a couple of hours and the organisers came to me with a weird problem. There were no changes in the scoreboard for the last two hours. Everyone were stuck wherever they were. Weird, right?

I then realized.

Fk.

In that dreaded query, I had only run

UPDATE 'qa' SET answer = 'something'

leaving out the where clause, specifying the question to update, like

WHERE qno=13

As a result, solutions to all the questions were updated to the same answer. After hastily fixing everything back, I had the dreaded conversation.

Org: What was the problem?

Me: It was the cache.

Org: Damn thing. Always messes up.

Me: *sheepishly* yeah

Probably the most embarrassing moment in my life, wrt coding 😑

Ended in an UPDATE without WHERE query to a core table in a Drupal project (in a dev database stored in the same server that production database was)

My manager is instructing my team to add a feature that can only be enabled for users by running an update script in the database.

When I argued that it's not really "complete" if it can't be turned on without someone going into the production database, I was told that not only is it complete, but they plan to have our non-technical customer service enable it for customers if the customer requests it...

Apparently giving everyone and their brother write access to prod is a good idea, but implementing a checkbox is a "waste of time and would cost too much money".

Probably going to float my resume... :-p

OMFG I don't even know where to start..
Probably should start with last week (as this is the first time I had to deal with this problem directly)..

Also please note that all packages, procedure/function names, tables etc have fictional names, so every similarity between this story and reality is just a coincidence!!

Here it goes..

Lat week we implemented a new feature for the customer on production, everything was working fine.. After a day or two, the customer notices the audit logs are not complete aka missing user_id or have the wrong user_id inserted.
Hm.. ok.. I check logs (disk + database).. WTF, parameters are being sent in as they should, meaning they are there, so no idea what is with the missing ids.

OK, logs look fine, but I notice user_id have some weird values (I already memorized most frequent users and their ids). So I go check what is happening in the code, as the procedures/functions are called ok.

Wow, boy was I surprised.. many many times..
In the code, we actually check for user in this apps db or in case of using SSO (which we were) in the main db schema..
The user gets returned & logged ok, but that is it. Used only for authentication. When sending stuff to the db to log, old user Id is used, meaning that ofc userid was missing or wrong.

Anyhow, I fix that crap, take care of some other audit logs, so that proper user id was sent in. Test locally, cool. Works. Update customer's test servers. Works. Cool..

I still notice something off.. even though I fixed the audit_dbtable_2, audit_dbtable_1 still doesn't show proper user ids.. This was last week. I left it as is, as I had more urgent tasks waiting for me..

Anyhow, now it came the time for this fuckup to be fixed. Ok, I think to myself I can do this with a bit more hacking, but it leaves the original database and all other apps as is, so they won't break.
I crate another pck for api alone copy the calls, add user_id as param and from that on, I call other standard functions like usual, just leave out the user_id I am now explicitly sending with every call.
Ok this might work.

I prepare package, add user_id param to the calls.. great, time to test this code and my knowledge..

I made changes for api to incude the current user id (+ log it in the disk logs + audit_dbtable_1), test it, and check db..
Disk logs fine, debugging fine (user_id has proper value) but audit_dbtable_1 still userid = 0.

WTF?! I go check the code, where I forgot to include user id.. noup, it's all there. OK, I go check the logging, maybe I fucked up some parameters on db level. Nope, user is there in the friggin description ON THE SAME FUCKING TABLE!!
Just not in the column user_id...

WTF..Ok, cig break to let me think..

I come back and check the original auditing procedure on the db.. It is usually used/called with null as the user id. OK, I have replaced those with actual user ids I sent in the procedures/functions. Recheck every call!! TWICE!! Great.. no fuckups. Let's test it again!
OFC nothing changes, value in the db is still 0. WTF?! HOW!?

So I open the auditing pck, to look the insides of that bloody procedure.. WHAT THE ACTUAL FUCK?!
Instead of logging the p_user_sth_sth that is sent to that procedure, it just inserts the variable declared in the main package..
WHAT THE ACTUAL FUCK?! Did the 'new guy' made changes to this because he couldn't figure out what is wrong?! Nope, not him. I asked the CEO if he knows anything.. Noup.. I checked all customers dbs (different customers).. ALL HAD THIS HARDOCED IN!!! FORM THE FREAKING YEAR 2016!!! O.o

Unfuckin believable.. How did this ever work?!

Looks like at the begining, someone tried to implement this, but gave up mid implementation.. Decided it is enough to log current user id into BLABLA variable on some pck..
Which might have been ok 10+ years ago, but not today, not when you use connection pooling.. FFS!!

So yeah, I found easter eggs from years ago.. Almost went crazy when trying to figure out where I fucked this up. It was such a plan, simple, straight-forward solution to auditing..

If only the original procedure was working as it should.. bloddy hell!!

Pushed an update to production but forgot to turn off the debug option to print SQL queries to the log. Now all I see is SQL queries.

It's now hopeless to read the log with tail ¯\_(ツ)_/¯

Our team really needs some workflow arrangement, and this time it was me who screwed up.
So we have to push an update to the Play Store and the App Store the Friday, the app is well tested on test environment then production environment, we got the ok so I uploaded a build, the app management team then continued the process of publishing..

During the weekend the app was approved and live to almost 500k user that can receive the update.

I got a phone call from the Project Manager at almost midnight, the time was really suspicious so I answered.
- Me: Hello.
- PM: Hi, sorry to call you now but the app is live and we have a problem.
- Me: what kind of problem? Let me check.

So I updated the app on my phone and opened it while I am on call.. I almost had heart attack!! WE PUBLISHED A VERSION POINTING TO THE TEST ENVIRONMENT. Holly shit

- Me: shit call the app management team NOW.

Eventually we removed the app from sale (unpublished it) and we submitted a new version immediately, once it was approved the next day we made the app available again (so for those who didn’t update yet, there will be no update to a faulted version, and no new users landing to a version with test data), I received one or two calls from friends telling me why the app is not on the store (our app is used nationally, so it’s really important).

Thank God there was no big show on twitter or other social media.. but it’s really a good lesson to learn.

I understand this is totally my fault, thankfully I didn’t get fired 😅

Today was a manic-depressive kind of day. Spent the morning helping some developers with getting their code to run a stored procedure to drop old partitions, but it wasn't working on their end. It was a fairly simple proc. But working with partitions is a little like working with an array. I figured out that they were passing the wrong timestamp, and needed to add +1 to delete the right partition. Got that sorted out, and things were good. Lunch time.

After lunch I did some busy work, and then the PO comes up at about 2PM and says he's assigned some requests to me. The first was just attaching some scripts. Easy. The second, the user wants a couple of schemas exported ... at 6PM. I've been in the office since 6:45AM.

While I'm setting up some commands to run for the data export, a BA walks up and asks if I'm filling in for another DBA who is out for a few weeks. Yep. There's a change request that hasn't been assigned, and he normally does the work. I ask when it's due. Well, the pre-implementation was supposed to be done in the morning, but it wasn't, and we're in the implementation window ... half way through. I bring up the change task, and look at. Create new schema and users. That's all it says. The BA laughs. I tell I need more to go on. 10 minutes later he sends an email with the information. There's only two hours left in the window, and I can only use half of it, because the production guys have to their stuff, and we're in their window. Now I'm irritated, because I'm new to Oracle, and it's an unforgiving mistress. Fortunately, another DBA says he'll do it, so that we can get it done in time. But can't work it either, because Dev DBAs don't have access to QA, and the process required access for this task. Gets shelved until the access issue is resolved. It's now after 4:15PM. I'm going to in traffic with that 6PM deadline.

I manage to get home and to the computer by 5:45PM. Log in. Start VPN. Box pops on screen. Java needs to update. I chose skip update. Box pops up again. It won't let me log in until Java is current. Passed.

I finally get logged in, and it's 6:10PM. I'm late getting the job started. I pull up Putty and log into the first box, and paste my pre-prepared command in the command line and hit error. Command not found. I'm tired, so it's a moment to sink in. I don't have time for this.

I log into DBArtisan and pull up the first data base, use the wizard to set the job, and off it goes. Yay. Bring up the second database, and have enter the connect info. Host not found. Wut? Examine host name. Yep, it's correct. Try a different method. Host not found. Go back to Putty. Log in. Past string. Launch. Command not found. Now my brain is quitting on me. Why now? It's after 6:30PM. Fiddle with some settings, reset $Oracle home. Try again. Yay. It works. I'm done. It's after 7PM.

There is nothing like technology to snatch the euphoria of a success away from you. It's a love-hate thing, but I wouldn't trade it for anything else. I'm done. Good night.

Was busy today doing some house work. Everything was normal till afternoon.

Me thinking of using laptop noticed that there was no laptop

My office laptop and another bag was stolen from my home .

It had most of the production details and source code. Running behind police for help now.

Informed my boss too about this incident.

Will update on what happens tomorrow in office.

Fuck that person who stole my office laptop and destroyed my Peace.

UPDATE on production. Forgot the WHERE... bugger... Panic time!

Ran a script on production to scrape ~1000 sites continously and update our ~50.000 productions from the data. On the same server as our site was running. Needless to say, with traffic and scraping, our server had almost 100% CPU and ram usage all the time for 2 weeks until I realised my fuckup

I goofed up and forgot the WHERE clause in my UPDATE query. Accidentally all of production possibly updated because we don't do test databases. I think it didn't actually go through because I cancelled it but now I need to restore a backup and compare data. Which means explaining this to the co-owner who can help me with a restore. I'm mortified, more so because it was a stupid thing to do to begin with.

Fucking shit i just had a 3 days chat with google's cloud engineer about an issue i had in a project. eventually the issue occured due to an update they made on some projects involving IAM changes that required some changes from my part in my security toles. Like wtf haven't you heard of data fixes when you roll out such changes?! I just had my production env down for 72hours for their fuckup.
At least send an email regarding it so we could set it up in time

Do I want to continue?
Y -> vacation lost (Production server is down).

N -> Ok, I will gather more packages for you to update next time.
😭😭

When your college approaches you with an update and want to test it in production....

Being mocked by the chef because I am "almost a week on a task"... Yeah dude, if a script which should update hundred thousands of main entities in production with a 45 min duration even though it is already paralleled is a half day job why don't you do it fucking yourself?

Employer: Hey, we are moving an API update live tomorrow morning that could affect our apps. Can you regression test the apps to make sure they all work?

Me: The API team is pushing code overtop of live endpoints that can break them?

Employer: Yes, we need the updates to work with a new product we are developing.

Me: And nobody thought about versioning these endpoints so we guarantee uptime on all existing services using them now?

Employer: We looked at that but it cost extra and required us to use the cloud solution so we don’t use versioning.

Me: Okkk… I also take it that the API’s don’t have integration tests written?

Employer: What are integration tests? Are unit tests the same thing?

Me: No, so when do I need to regression test all 7 production apps?

Employer: The API’s are moving to production at 4am and we need it signed off by 7am.

Me: I only have 3 hours to regression test 7 production apps at 4am? Each app, if I just skim over them, would take me 2 hours each. I will do my best but that’s a very short time to ensure complete functionality.

Employer: Don’t you have unit tests?

Ever had a day that felt like you're shoveling snow from the driveway? In a blizzard? With thunderstorms & falling unicorns? Like you shovel away one m² & turn around and no footprints visible anymore? And snow built up to your neck?

Today my work day was like that.. xcept shit..shit instead of pretty & puffy snow!!

Working on things a & b, trying to not mess either one up, then comes shit x, coworker was updating production.. ofc something went wrong.. again not testing after the update..then me 'to da rescue'.. :/ hardly patch things up, so it works..in a way.. feature c still missing due to needed workarounds.. going back to a and b.. got disrupted by the same coworker who is nver listening, but always asking too much..

And when I think I finally have the b thing figured out a f-ing blocker from one of our biggest clients.. The whole system is unresponsive.. Needles to say, same guy in support for two companies (their end), so they filed the jira blocker with the wrong customer that doesn't have a SLA so no urgent emails..and then the phone calls.. and then the hell broke loose.. checking what is happening.. After frantic calls from our dba to anyone who even knows that our customer exists if they were doing sth on the db.. noup, not a single one was fucking with the prod db.. The hell! Materialised view created 10 mins ago that blocked everything..set to recreate every 10 minutes..with a query that I am guessing couldn't even select all that data in under 15.. dafaaaq?! Then we kill it..and again it is there.. We found out that customers dbas were testing something on live environment, oblivious that they mamaged to block the entire db..
FML, I'm going pokemon hunting.. :/ codename for ingress n beer..

Sometimes your music app knows just the right song to play.

Story:
Production program was working (has been for a long time). But suddenly it starts failing. I spent a long ass time trying to see what went wrong.

Problem:
Security update on the server 🙃

Now I've got the client, his minions, and the users emailing me to fix this. But I didn't start this fire!

Song: We didn't start the fire, by Billy Joel

When will X be finished?!

Um..I don't know cuz thing Y is blocking me from doing X implementation and it makes no sense.. I have no idea what is wrong with it and can't tell how long it'll take me to fix, hopefully by the end of today I'll have thing Y fixed enough so I can finish thing X..

Ok, will you be able to update production today?! Just put it on production when you're done with X.

Uuuumm... Yeah... // still need to know how X will behave in cases a, b, c & b+c...

I'm writing this while waiting for abc feedback.. this is going great.. :/

I know prod is not used as prod yet, but still... fuuuuu!!

So, in my company we where initially about 20 programmers doing two big projects.

The client (who also is the owner of the company) keep asking more and more and more things. Each 3 months we update the site but the client doesn't start the marketing or anything else, so the app don't have any users.

After two years of development, 26 micro services, one big web platform in Python (web2py, bad decision) and a hybrid mobile app the client decide to shut down the project because it was "a little bit illegal".

The second project have the same problems, but this project does have marketing, the shitty part is after two year and a lot of development now the project isn't viable because the market is gone.

The boss calls, says he have some problems and he will fire 18 persons and reduce the payment of the rest, he ask us to "hold" for the good times.

The great idea he had for earn money is rewriting a WordPress app that have 4 years in production to angular (because he, who knows why, thinks angular is the best shit out there)

I want to quit but even with the reduced payment I know he pays way more than the market average, plus I'm still student.

Yesterday I killed a production server with a handful of sites running on it. 😬

Created an observer for one of my Laravel models. It was generating new translated slugs when another post gets translated by an API. While implementing also an updated method besides the created observation I obviously updated the models slug.
Pretty confident this small update will work I just pushed to production. Tested it live.

BOOM

Hard reset on the server redoing the changes. Searching where I fucked up this time...

Finding me observing the updating while updating the updating of the updating by the updating. 🤦

I run update without where on mysql console on production database Today.

CLASSIC

Just because I needed to fix database after bug fix on the backend of the application.
I thought I wrote good sql statement after executing it on my local machine and then everything got bad.

Luckily it was only one column with some cached statistics data and I checked that it was not important data before I actually started fixing stuff but still ...
Almost got hard attack afterwards.
Made a script to fix this column and it took me only 15 minutes but still...

Bug was caused in part I got no unit tests and application grow after 3 years of development from simple one for one customer and volumes of documents around 50k to over 40 customers and volumes over 2mil per month, don’t know how many pages each, just in one year after we completed all needed features.

I have daily backups and logs of every api operation but still.
I think this got to far for one backend developer.

I got scared that I will loose money cause I am contractor and the only backend developer working on it.
I am so tired of this right now I think I need a break from work.

Responsibility is killing me so hard right now.
It will take a week to get back to normal.

So I did a clean Windows 10 restore recently on my laptop from Insider program to just Anniversary Update . Went away from my computer for a day or so by the time I got updates completed and Visual Studio up and running. So earlier today I went to start back up development on a project of mine to come across the emulators not working. The thing is that I lost 6 hrs of production to figure this out. I tried everything possible so I gave up and reinstalled VS to just remember I forgot to turn on my Hyper-V in BIOS setting. So I'm half way in VS reinstalling and I can't do anything about it. GOD FUCKEN DAMMIT W10!

Supporting a production upgrade on a Friday night without extra allowances because I screwed up the one on Tuesday night! Yay!

I was Just college fresher who completed his Engineering. My first week in the office. And a system was provided to me, since it was support project so I was given direct access to production database.

Fresher + Production Database + Access of Admin credentials = Worst Possible Combination

So it was my night shift, I was told to update new tariff plan for our client (which was one of the largest telecom service in India) .

If someone recharges for more than 200 Rupee, that person will get 10% or 20% extra talk time. Which was only applicable for particular circle (Like Bihar and Rajasthan).

Since I was fresher, I was told to update given query from my senior employee which he shared on the shared folder. Production downtime was in the mid night, so at that time I updated that query on the production database.

Query successfully updated. I completed my night shift, went home and slept.

When I woke up, I saw my mobile it had 200+ missed calls from different locations of India. They were Circle heads of that telecom service provider who contacted me. I realized something unexpected is expecting me.

Then at that moment my team lead called me and he asked me to come office right away.

Reminding you I was a fresher, I was shivering. What have I done there?

When I reached office, I came to know that the query I updated on production bombarded.

Every person who recharged that day (duration from midnight to morning 10 AM) got 10 times or 20 times more talktime.

A part of Query was something like this where error was made:

TalkTime = RechargeAmount + RechargeAmount * 10/100; (Bihar)
or

TalkTime = RechargeAmount + RechargeAmount * 20/100; (Rajasthan)
But instead of this query, I updated below one:

TalkTime = RechargeAmount + RechargeAmount * 10;
or

TalkTime = RechargeAmount + RechargeAmount * 20;
In a span of 10 hours, that telecom service lost revenue of 6.5 crore Rupees. Thanks to recovery team they were able to recover 6 crore but still 50 lakh Rupees were in loss.

One small query, and approx 1 million dollar was on stake.

Aftermath of this incident

My Mistake:

I should have taken those queries on mail. Or, there should have been mail communication regarding this.
Never ever do anything over oral communication. Senior employee who did this denied and said he provided correct query, and I had no proof of communication.
I told them, it was me who executed that query on production. Since I was fresher, and took my responsibility of that incident. My team lead rescued me from that situation.
Lesson Learned:

Always test your query and code multiple times before you execute or Go live it on production.
Always have email communication for every action you take on production.
Power comes with responsibility. If you have admin credentials of production never use it for update/delete/drop until you are sure.
Don’t take your job lightly.
I was not fired from that Job, but I have learnt my lesson very well.

Am I the only developer in existence who's ever dealt with Git on Windows? What a colossal train wreck.

1. Authentication. Since there is no ssh key/git url support on Windows, you have to retype your git credentials Every Stinking Time you push. I thought Git Credential Manager was supposed to save your credentials? And this was impossible over SSH (see below). The previous developer had used an http git URL with his username and password baked in for authentication. I thought that was a horrific idea so I eventually figured out how to use a Bitbucket App password.

2. Permissions errors
In order to commit and push updates, I have to run Git for Windows as Administrator.

3. No SSH for easy git access
Here's where I confess that this is a Windows Server machine running as some form of production. Please don't slaughter me! I am not the server admin.
So, I convinced the server guy to find and install some sort of ssh service for Windows just for the off times we have to make a hot fix in production. (Don't ask, but more common than it should be.)
Sadly, this ssh access is totally useless as the git colors are all messed up, the line wrap length and window size are just weird (seems about 60 characters wide by 25 lines tall) and worse of all I can't commit/push in git via ssh because Permissions. Extremely aggravating.

4. Git on Windows hangs open and locks the index file
Finally, we manage to have Git for Windows hang quite frequently and lock the git index file, meaning that we can't do anything in git (commit, push, pull) without manually quitting these processes from task manager, then browsing to the directory and deleting the .git/index.lock file.

Putting this all together, here's the process for a pull on this production server:
Launch a VNC session to the server. Close multiple popups from different services. Ask Windows to please not "restart to install updates". Launch git for Windows. Run a git pull. If the commits to be pulled involve deleting files, the pull will fail with a permissions error. Realize you forgot to launch as Administrator. Depending on how many files were deleted in the last update, you may need to quit the application and force close the process rather than answer "n" for every "would you like to try again?" file. Relaunch Git as Administrator. Run Git pull. Finally everything works.

At this point, I'd be grateful for any tips, appreciate any sympathy, and understand any hatred. Windows Server is bad. Git on Windows is bad.

Boss just told me that he thought him making changing directly on production was "OK", because it always seemed to automatically update the other environments when his changes appeared "magically" a week later.

Yeah, because the frequent deployments I do and then come across your un-approved changes which I'm constantly flagging up has nothing to do with it!!

*facepalm*

mongpoop

this is how our upgrade from mongoose v5 to v6 went

v5: strictQuery is false by default

v6: strictQuery is true be default

^ realized this once our update went live, we are now manually migrating several hundreds of production data.

v7: strictQuery is false by default.

what!!!

A newly joined developer (who was supposed to be very senior) comes and asks me how to write a test cos for some reason the person didn't know how to mock.

In Java,
(same for any other implementation which has an interface)
Writes Arraylist list =.....
Instead of List list = Arraylist...

Deployed code (another engineer from another country helped to deploy since this new senior dev didn't have access yet.

But the new senior dev didn't update relevant files in production code which brought down the site for nearly an hour. Mistake aside, the first reaction from this new senior dev is 'WHY DIDN'T THE DEV THAT WAS HELPING DIDN'T DO THE FILE UPDATE?'

This was followed by some other complaints such as our branching stragies are wrong. When in fact the new senior dev made a mistake by just making assumptions on our git branching strategies and we already advised on correct process.

Out of all these, guess this is the best part. The senior dev never tested code locally! Just wrote code, unit test and send to QA and somehow the test passed through. I learnt this when I realised this dev... has not even set up the local environment yet.

I keep saying new but this Senior dev been around like 3 months! This person is in another team within our larger team but shares same code base. I am puzzled how do you not set up your environment for 3 months. Don't you ask for help if you are stuck? I am pretty sure the env is still not setup.

Am I over reacting or is this one disgusting developer who doesn't even qualify for an intern let alone a senior dev? It's so revolting I can't even bring myself to offer help.

How to manage updates on production as web agency

1: Update the GitHub private repo with your changes
2: Write an email to the customer sysadmin and ask to sync in staging and later production
3: Wait 2 weeks and do another ping about it
4: The sysadmin later will do that
5: In the meantime there is a security update of your CMS and a lot of plugins to update
6: Try to update from the CMS panel and there are no write permission
7: Cry a lot and write another email to go back to point 2

Joined a new startup as a remote dev, feeling a bit micromanaged. So this week I joined an established startup as a senior mobile dev where I work remotely.

Previous two devs got fired and two new guys got hired (me as a senior dev and another senior dev as a teamlead, also third senior dev will join next week).

Situation is that codebase is really crappy (they invested 4 years developing the android app which hasn't even been released yet). It seems that previous devs were piggybacking on old architecture and didn't bother to update anything, looking at their GIT output I could tell that they were working at 20-30% capacity and just accepting each other MR's usually with no comments meaning no actual code reviews. So codebase already is outdated and has lots of technical debt. Anyways, I like the challenges so a crappy codebase is not really a problem.

Problem is that management seems to be shitting bricks now and because they got burned by devs who treated this as a freelance gig (Im talking taking 8-10 weeks pto in a given year, lots of questionable sick leaves and skipping half of the meetings) now after management fired them it seems that they are changing their strategy into micro managament and want to roll this app out into production in the next 3 months or so lol. I started seeing redflags, for example:

1. Saw VP's slack announcement where he is urging devs to push code everyday. I'm a senior dev and I push code only when I'm ready and I have at least a proof of concept that's working. Not a big fan of pushing draft work daily that is in in progress and have to deal with nitpicky comments on stuff that is not ready yet. This was never a problem in 4-5 other jobs I worked in over the years.

2. Senior dev who's assigned as the teamlead on my team has been working for 1 month and I can already see that he hates the codebase, doesn't plan on coding too much himself and seems like he plans on just sitting in meetings and micromanaging me and other dev who will join soon. For example everyday he is asking me on how I am doing and I have to report this to him + in a separate daily meeting with him and product. Feels weird.

3. Same senior dev/teamlead had a child born yesterday. While his wife was in hospital the guy rushed home to join all work meetings and to work on the project. Even today he seems to be working. That screams to me like a major redflag, how will he be able to balance his teamlead position and his family life? Why management didn't tell him to just take a few days off? He told me himself he is a senior dev who helped other devs out, but never was in an actual lead position. I'm starting to doubt if he will be able to handle this properly and set proper boundaries so that management wouldn't impact mental health.

Right now this is only my 1st week. They didn't even have a proper backend documentation. Not a problem. I installed their iOS app which is released and intercepted the traffic so I know how backend works so I can implement it in android app now.

My point is that I'm not a child who needs hand holding. I already took on 2 tickets and gonna push an MR with fixes. This is my first week guys. In more corporate companies people sit 2 months just reading documentation and are not expected to be useful for first few months. All I want is for management to fuckoff and let me do my thing. I already join daily standup, respond to my teamlead daily and I ping people if I need something. I take on responsibility and I deliver.

How to handle this situation? I think maybe I came off as too humble in the interview or something, but basically I feel like I'm being treated like a junior or something. I think I need to deliver a few times and establish some firm boundaries here.

In all workplaces where I worked I was trusted and given freedom. I feel like if they continue treating me like a junior/mid workhorse who needs to be micromanaged I will just start interviewing for other places soon.

Even seniors make mistakes. In case you were ever doubting yourself - just remember that.

I just had a very senior level programmer on my staff add a function to a production system that issues an SQL UPDATE query without a WHERE clause. Fortunately, only the 1st entry succeeded and the rest failed due to "duplicate record" errors. Clearly he had intended to do a SELECT to check if an entry was present. If it was present, do an UPDATE, otherwise do an INSERT (think UPSERT - but done manually). However instead in the insert part they were both UPDATE's. The first update was normal looking but the second UPDATE was just this weird malformed-looking thing where he tried to do an UPDATE but to every field including the key fields. Clearly he was thinking about an insert but actually writing it as an update. Every now and then I need to remind myself that these things happen. The guy's not dumb - just made a mistake.

I'm just happy it "failed unsuccessfully".

There's very little good use cases for mongo, change my mind.

Prototyping maybe? Rails can prototype, create/update/destroy db schemas really quickly anyways.

If you're doing a web app, there's tons of libs that let you have a store in your app, even a fake mongo on the browser.

Are the reads fast? When I need that, use with redis.

Can it be an actual replacement for an app's db? No. Safety mechanisms that relational dbs have are pretty much must haves for a production level app.
Data type checks, null checks, foreign key checks, query checks.
All this robustness, this safety is something critical to maintain the data of an app sane.

Screw ups in the app layer affecting the data are a lot less visible and don't get noticed immediately (things like this can happen with relational dbs but are a lot less likely)

Let's not even get into mutating structures. Once you pick a structure with mongo, you're pretty much set.
Redoing a structure is manual, and you better have checks afterwards.

But at the same time, this is kind of a pro for mongo, since if there's variable data, as in some fields that are not always present, you don't need to create column for them, they just go into the data.

But you can have json columns in postgres too!

Is it easier to migrate than relational dbs? yes, but docker makes everything easy also.

My company insists on working in one production environment to save time and every time I try to convince them to set up a work flow with a dev and test environment, they tell me we don't have the time...

Even after I set one up anyway as I'm scared shitless to touch production. They tell me it's faster doing it all in one environment.

They launch an update. Site buggy as hell and doesn't load 90% of the content...

Sigh....

I finally get the go ahead to push an update to a client site into production. Client then sends out marketing email about changes later that afternoon. The next day the client decides they are not ready to start fulfilling orders from one of the new products I added. Now I have to rush a change to remove the product links from the site.

Windows 10 insider preview had a critical bug like half a year ago where most browsers would freeze the PC. I've reported it multiple times but my feedback didnt get any attention. The bug made it to production months later...

Now, the story is repeating. I've discovered a bug. Everytime there's a software update for apps that with a built-in updater in the main .exe (JetBrains IDEs, VS etc.) the updater fails, program is corrupted and needs to be reinstalled. For instance, there's a minor bump in Android Studio version, if you try to use autoupdater it will corrupt your Android Studio and you'll have to reinstall.

Been trying to reach out to them but the only "real issues" that are highlighted are "no CPU temp in task manager" or "pls improve automatic problem resolving"

Why the fuck do you even have a feedback program if you're ignoring the people who are reaching out to you, you pieces of shit

There's a production site with api call limits. We experienced waaaaay more visitors than usual and I forgot to increase the api call limit, so new users weren't able to do anything. FML again, update emails sent out.

Finally made my node production server stable enough that I could focus on writing tests*. I start by setting up docker, mocking cognito, preparing the database and everything. Reading up on Node test suites and following a short tut to set up my first unit test. Didn't go smoothly, but it's local and there are no deadlines so who cares. 4 days later, first assert.equal(1+1, 2) passes and I'm happy.

I start writing all sorts of tests, installing everything required into "devDependancies," and getting the joy of having some tests pass on first try with all asserts set up, feels good!

I decide to make a small update to production, so I add a test, run and see it fail, implement the feature, re-run and, it passes!

I push the feature to develop, test it, and it works as intended. Merge that to master and subsequently to one of my ec2 production servers**, and lo and behold, production server is on a bootloop claiming it "Cannot find module `graphql`". But how? I didn't change any production dependencies, and my package lock json is committed so wth?

I google the issue, but can't find anything relevant. The only thing that I could guess was that some dependencies (including graphql) were referenced*** in both, prod and dev, and were omitted when installed on a prod NODE_ENV, but googling that specific issue yielded no results, and I would have thought npm would be clever enough to see that and would always install those dependencies (spoiler: it didn't for me).

With reduced production capacity (having one server down) I decided to npm uninstall all dev dependencies anyway and see what happens. Aaaaand it works.....

So now I have a working production server, but broken local tests, and I'm not sure why npm is behaving like this...

* Yes I see the irony.
** No staging because $$$, also this is a personal project.
*** I am not directly referencing the same thing twice, it's probably a subdependency somewhere.

When your staging/test server dies just as the boss wants a big update completed. Straight to production it is!

Developed an update to our database procedures and tested it with local copy. After a few days everything was ready. Opened our server and started the update. After a couple of rows an error occurred. Turns out our production db is older version and does not support some syntax I used. It became a bit longer day at work...

Is it good or bad that I forgot to push an critical buggfix to production just before I went home?

Hopefully I can update from home (think one of our servers has an backup of the ssh keys)

This one was from my Tech Lead, She had to update phone number of a customer in Database on production server. And guess what she forgot 'WHERE'. Next we were facing each other with poker face. :|

p.s: fortunately we had backup of just 4 hours back. Still we lost data of about 100 people.

TL;DR: fear of bricking my laptop due to typo pinning.

The worst nightmare i am living in right now...

I was noticing i did need some software in sid so i decided to use apt pinning for said software...

I configure the system, ok test looks good... I push it to production, run it on the system....and the nightmare starts.

Lits of packages get updated, and i am screaming 'noooooooo' since debian sid softwarz can sometimes break everything! I discovered that i did test my apt pinning config for the presence of the amount of numbers, but not at their value... Sooo, by accident swapping pin numbers for stable and unstable you get... Your worst apt-get update nightmare...

I hope it does not become a brick.

That's pretty awesome -_- angular decides to simply discontinue older (half a year) versions of their cli tool by overwriting the older releases with an update notice. Causing my project (jsRant) to break until I update to the newest version.

This new version requires a different configuration so I have to update my entire project just to be able to deploy to production again....

A comment from the heart: f*** you angular.

My workplace is still using xml based configuration, and non-spring boot projects.

So every spring boot tutorial I find feels like "Look at how easy you can get this running" and then it's just actually a toy you can't get into production.

Also it kind of bugs me that you need to be online to actually be able to initialize/create a spring boot project and every single tutorial says so.
You can make a local network m2 repository, but can one make a spring initializer service?

Either way, migrating every single project to Spring boot is a no-no,

And I'm stuck with like 5 prototypes of SSO integration from which only 2 work, and the other 3 have their own problems.

One does redirect to the login and all, but the SAML endpoint gets 404 on response when you log in.

One is on OpenID Connect, but I would need to update the project from Spring 3 to Spring 5 to get it working, which upon attempting to do seems to break everything else.

One has an external library handling the security context just the way we are accustomed to, but it only does a 401 forbidden when you go without logging in and I'm starting to think it is actually one of those that require you to extract the token or something manual like that, which wouldn't work for us

The other two are spring boot tutorials that worked out of the box, both SAML and OpenID, still can't use those for the main projects.

I'm tired of dealing with this configuration hell, been two months at this, I want to get features done as usual, not be stuck configuring stuff that might or might not work.

Rant aside, I think I figured I need to use a different Security adapter, but I needed to vent.

beware of font choices in chat apps; a coworker joked in the room that "well, sure, of course it's okay to update in production in the middle of the day" and for some reason, the other coworker didn't see the quotes because of the weird font they use, and also didn't stop to think, and went ahead and ran the deployment script. In production. In the middle of the day. With active users.

The good news is that those folks who logged back in got to use the new version a whole lot earlier than anyone was expecting. :\

I'm starting to FUCKing hate those 1-5 lines minified 3rdparty javascript snippets that everyone seems to be bombarding their website with.

Why on gods ungodly earth would you ever dream of injecting this kind of style into my <head> tag.
Without any warning whatsoever.

You couldn't be FUCKing bothered to be thinking about the consequences, before pushing such an update to your production server.
That's how you leave your users website broken, or ugly af.

I know it's an easy fix, simply remove the snippet that injects this crap, if only I was allowed to say "no don't keep this FUCKing crap" to the customer.

UPDATE someColumn = 'someChange'
without the where on production (not me)

Installed on my 2x 980 TI SLI the newest Nvidia Driver 384.76.

Played games on it and some Video Editing. Noticed that I have bad laggs and checked clocks.

Wow clocks were locked at half the max Output. Checked if I can do anything to increase it, nope it's locked at half the Performance.

Tried to install previous Driver, install failed. Went 2 Drivers back and installed the 382.33.

Everything worked again.

WTF are the guys at NVIDIA coding while being drunk?? How the fuck can you lock the powertarget of the GPU at half the Performance and after 4 days not releasing a hotfix? Are they testing this shit even or forcing it into production?

Currently NVIDIA has really crappy drivers and with every update it gets even worse.

And the only option you got is GPU's from NVIDIA and AMD. And Vega is not even released yet.

Hope that AMD will beat the shit out of NVIDIA...

I personally don't have a funny dev sin story (not that I didn't commit any).
My internship colleague should update a value of a row in production. So he wrote a SQL command and forgot the where clause. This was the first time the company tested there rollback mechanism and it didn't work. For the next 2 weeks my colleague was busy updating 2000ish rows to make it work again

Ever given access to production data to correct a data entry that doesn't have a PATCH api ready?

Ever ran UPDATE table set COLUMN="value"; without WHERE condition?

Ever done both of these sequentially?

Yeah. DB snapshots cost a bomb to restore :(

My first dev job was for a .net shop. Until then, I had only worked in Java and PHP. This place didn't have the normal team structure, and I soon found that I was going to be working solo on the projects I was responsible for. I'm my first week there, I was tasked with making make revisions to an application in a new language, with a new toolkit, solo. A few weeks later was the most intense day I've had as a dev, as I put in the change control to release my update to production.

CTO: I want you to take up a side project and push it to production. Atleast 1 per quarter. Also, you have to keep making tiny enhancements to the website on weekends. And it would be great if you can mentor and help the other team with architecture. And, don't you feel itchy that the app you made in that hackathon is not used by any user? Productionize it. Don't forget to update me on your primary task at the end of biweekly sprint, that goes without said.

Shame on you Facebook Developers!

I’m neither messy or anal coder, but whenever I push something to production I make sure to handle all debugger warnings... I don’t think it’s perfectionism, rather good practice, makes important bugs easier to find. ;)

A single Facebook SDK Framework has 70+ unhandled warnings, trivial things, could be fixed in a half day...

I had to use ‘inhibit_all_warnings!’ flag literally for the first time, not to get frustrated every time it gets updated :/

P.S Not to mention I had to change the dependency manager because (with the newest update) they have written themselves out of compatibility with the one I was using... C’mon guys. Not cool.

First year on the job. Was already good at writing software, but bad at practices and administration. One such software was being tested live, while still in development. I was developing on the production database... .
Yeah.
I was working on an edit feature of sales records, in a table that already contained hundreds of subsidized sales of very expensive products. Based on that, the supplier had to compensate the shops with half the price of every item.
I forgot to add a where clause to the update. Lost all sales data. On production.
Asked the admin if there are backups and he says yes, checks to discover that the backup script failed for the last week (since it became live)

Whole thing was incredibly stupid. I made a ton of stupid mistakes, and so did the other people involved. The loss was around 1 year of my income. Luckily the client decided to brush it off as losses and claim some tax benefits and it all ended well.

Working with a data scientist on an update to a machine learning api that has dinner logic change with a new model.

He's wondering why his PRs are falling.

He's trying to merge into development from a branch created off of main.

He's renamed all the functions and classes and never updated the call points.

He's using new packages but never includes them in the requirements file, so the docket builds are failing.

His class method definitions don't contain self and are throwing syntax errors.

I've been working with him for 4 days to get him to understand branching, linting, unit testing, and not blindly copy and pasting snippets from jupyter notebook into production api code!

Before get get source code for freelance job, the person who cantact me say the job is to continue the project for some update and tweak.

The UI from design is beautiful and he gave good explaination for the project and the update, continue to conversarion, negosiation and deal.

but he is not the IT guy and also the project is not his work or something that he do previosly. All the person who work on that project is already leave and not contactable.

And here that I get:
- source code
- domain cred.

And here what's missing:
- documentation
- .env file
- db backup / old db cred.
- server and hosting cred.

And after some hour of learning the code I find out that:
- latest commit was 2 year ago and different from production version.
- most of the branch is RnD.
- the code have many wtf/minute lol

And for now I still re-negotiate with the person who give me the project with 2 suggestion from me.
- continue with this code with condition, he need to search for the missing part at least backup db or documentation.
- recreate the project with more time

And here's one funny part of the code.
randomNumber(){
return 5 // this number was choose by dev team at random
}

I'm given a simple assignment to update email templates. I tot it would be a breeze.

It turn out SURPRISE! After the updating of template is done. I deploy the code in the development environment.

I tried to access the email template like how the user will see to verify all is good. It turn out i am facing error.

So uhh ok, i went to check the logs to see what the hiccups. It turn out that a table is missing. But this is production code. So my question how the hell did the production environment has the table but dev don't.....

One of our corporate clients insists that we push changes and updates to production on either Saturday or Sunday. I've worked weekends for longer than I care to admit now.

A major update that completely changes both the data model and UI is scheduled for deployment this Sunday.

Monday is going to be hell with support requests.

What's better in general:
#1 master merge -> production
#2 master merge -> button -> production

And why?

At work we currently use #2 because sometimes we need to make manual changes or update images on prod before we can fully update the code

Edit: This should have been a 'question' not a joke/meme, sorry about that

Azure, great development slots! Must have, now I can have developer, staging and production. The greatest no downtime when swapping a new server in....

Everything crashes? WTF?

OKAY, so swapping to a service that authenticates users makes the authentication part crash :/

Phew development slots ROLL BACK...

No the entire service was broken. Rolling back, all non authenticating controllers work, but the authentication never happens, so server is working, but the users cant use it. Fuck!

Delete everything. Recreate. The setting persists. WTF. Delete again, recreate, reinitialize, republish, it works as it should when tested phew.

Creating new service experiencing cant replicate. Hmm, okay must have been a glitch. Next, update, YEAH swap, no downtime!!!

*EXPLOSION* ..... RINSE AND REPEAT:/

Best: finding Laravel and how well works out with production deployment with GitLab

Worst: didnt implement this right from the beginning. Had to copy paste from the dev folder to production folder FOR 3 MONTHS when i wanted to update the website.

I was hoping that I never have to build stupid websites again, but here I am...

The thing I hate the most, is the hassle to have an easy to update dev, staging and prod environment. Fuck wordpress, fuck drupal, fuck joomla.

git pull && composer install && npm install should be all that's needed to get the latest code in an environment.

composer require *** to install plugins. No stupid web interface where users install plugins in production env.

I don't want to create database dumps just because these fuckers think that you should store configurations in the database.

Is there any clean CMS primarily for professional programmers? Or are they all just made for retarded subhumans?

Quick update on our partner's API that doesn't work (see previous rant).

They gave the wrong URL! Wow!! Well we have the new URL but
the production credentials don't work!!!

Recently many of us may have seen that viral image of a BSOD in a Ford car, saying the vehicle cannot be driven due to an update failure.
I haven't been able to verify the story in established news sources, so I won't be further commenting on it, specifically.
But the prospects of the very concept are quite... concerning.

Deploying updates and patches to software can be reasonably called *the software industry*. We almost have no V0 software in production nowadays, anywhere (except for some types of firmware).
Thus, as car and other devices become more and more reliant on larger software rather than much shorter onboard firmware, infrastructure for online updates becomes mandatory.
And large scale, major updates for deployed software on many different runtime environments can be messy even on the most stable situations and connections (even k8s makes available rolling updates with tests on cloud infrastructure, so the whole thing won't come crashing down).
Thereby, an update mess on automotive-OS software is a given, we just have to wait for it.

When it comes... it will be a mess. Auto manufacturers will adopt a "move fast and break things" approach, because those who don't will appear to be outcompeted by those who deploy lots of shiny things, very often.
It will lead to mass outages on otherwise dependable transportation - private transportation.
Car owners, the demographic that most strongly overlaps with every other powerful demographic, will put significant pressure on governments to do something about it.
Governments (and I might be wrong here) will likely adapt existing recall implementation laws to apply to automotive OS software updates.
That means having to go to the auto shop every time there is a software update.
If Windows may be used as a reference for update frequency, that means several times per day.
A more reasonable expectation would be once per month.
Still completely impossible for large groups of rural car owners.
That means industry instability due to regulation and shifting demographics, and that could as well affect the rest of the software industry (because laws are pesky like that, rules that apply to cars could easily be used to reign in cloud computing software).

Thus... Please, someone tells me I overlooked something or that I am underestimating the adaptability of the powers at play, because it seems like a storm is on the horizon, straight ahead.

Was feeling over-confident one day, wrote a delete query and pushed to production without double checking it.

Turns out the where clause always returned true and deleted all rows in the production table.

Mind you, from that day onwards I'm using update queries instead of delete..

The biggest mistake my colleague done is -

update query for admin_reports table without where clause in mysql in production db. Right after that no admin reports. More than 1000 rows affected.

Glad we luckily we have some data in staging machine.. I don't know Why TF our devops team not taking backup. Hope they will from now.

Nom I'm using python to dump the data from staging and save it local file and then export to production.

#HisLifeSucks
#HeartBeatsFast

Work on something locally for a week. Everything works. Take a copy of the production db and run update scripts to the copy to make sure it goes smoothly. Nothing works. Damn.

worst mistake was probably introducing an infinite loop in the category tree for e-commerce site...

in the vein of true agile and considering MVPs and what not we had not yet automated everything. the client would send category updates as a spreadsheet and i had a script to generate the sql and jam it into the site. having run the script several times in the past I thought I'd just throw the update into production and call it a weekend...

it wasn't long before I started fielding calls that the site was unstable. no page would load and the server kept crashing under trivial load. well an entire frantic weekend later I discovered the category load hit an edge case I hadn't considered and I had introduced an infinite loop in the navigation of the site.

i'd like to say I learned my lesson and never just threw changes into production again, but what can I say - I like living on the edge. I did however learn that loop detection can be a valuable thibg

Someone mentioned antivirus and custom code... it reminded me of when Symantec Endpoint Protection received an update and some of our production servers had our custom apps quarantined. We had about four servers that had most of our custom apps removed. When it happened I thought someone was monkeying with the servers (they were our task servers used for task and service jobs). It took me about an hour to realize what happened then another hour to get the SEP admins to disable it until we could get another patch. Fun day.

Deploy new script on production and then server time are outdated suddenly, plus old timestamp data inside db changed to outdated time. Who update the data inside db? Mindblow~

Nothin to get a rant simmering like reFuckingRanting! All good, I'll have another shooter while I wait to reinstall, reconfig, rebuild, rewipe, and reRefuse any and all Windows clammy hand of aid it forces down my soul. One of these days, when the whole realm understands we don't need this vast array of exactly the same shit but this one's dick is a little bigger so lets fucking make it, this and not that, and rebuild, push the update, need 4 more updates by noon next day. Nothing stays stable team green, NOTHING. Fuck anyone trying to actually ...make something..We got vulns and updates, backdates, and breaking changes on all 58 of our same shit production line shit shows. I can't count one time in this few year pain olympics that in a single 8 hour span of time the same shit that was working like a fucking wet glove in a horses ass at whiskey 1 was NOT fucking anywhere near coherent come whiskey not 1. Just sayin, is this a cock show boys? Or is is just a wild and rambunctious thought to maybe start compiling/combining some key role playing pieces of softwar? If not, I'll just prep for another round of fucks, and carry on. Sadly, this shit is addicting against many odds. Enjoy your lunches

I tried to run a migration I wrote with a an older dump from production. Turns out it had a cost of 150129966.7 and didn't finish within half an hour. Turns out I didn't understand joins in update queries as well as I thought