Friend: "What is devRant?"
Me: "A place where programmers tell jokes and complain."
Friend: "Why dont you just do that irl?"
Me: "Because we never test in production"

I worked with a good dev at one of my previous jobs, but one of his faults was that he was a bit scattered and would sometimes forget things.

The story goes that one day we had this massive bug on our web app and we had a large portion of our dev team trying to figure it out. We thought we narrowed down the issue to a very specific part of the code, but something weird happened. No matter how often we looked at the piece of code where we all knew the problem had to be, no one could see any problem with it. And there want anything close to explaining how we could be seeing the issue we were in production.

We spent hours going through this. It was driving everyone crazy. All of a sudden, my co-worker (one referenced above) gasps “oh shit.” And we’re all like, what’s up? He proceeds to tell us that he thinks he might have been testing a line of code on one of our prod servers and left it in there by accident and never committed it into the actual codebase. Just to explain this - we had a great deploy process at this company but every so often a dev would need to test something quickly on a prod machine so we’d allow it as long as they did it and removed it quickly. It was meant for being for a select few tasks that required a prod server and was just going to be a single line to test something. Bad practice, but was fine because everyone had been extremely careful with it.

Until this guy came along. After he said he thought he might have left a line change in the code on a prod server, we had to manually go in to 12 web servers and check. Eventually, we found the one that had the change and finally, the issue at hand made sense. We never thought for a second that the committed code in the git repo that we were looking at would be inaccurate.

Needless to say, he was never allowed to touch code on a prod server ever again.

Everyone has a test environment. If you're lucky, you'll have a separate production environment too.

Testing in production doesn't hurt.

IT
FUCKING
KILLS

Just ask Boeing.

So I'm working on a live map of my school's bus system, and I needed some filler images to test out how the stops were being drawn on each route...and honestly wish I could push this version to production

Customer :Can you build a system that rates our product by XYZ standard?
Us: Sure!

*time passes*

Us: Ta-da!
Customer: Okay, here are some good and some bad products!

*products get rated shit to supershit*

Customer: No, that's wrong. Some of these are as good as we can, they should be rated best!
Us: okay, we offset the results.

*products get rated good to barely okay*

Customer: Great! Can you sign that the system rates by XYZ Standard?
Us: No.
Customer : But we paid you to rate by XYZ standard!
Us: By XYZ standard your products are bad, you can either have your products rated by standard or pass the test.
Customer: Unacceptable!
Us: Improve production?
Customer : Not possible, the job is done when you rate the products good by XYZ standard.

A more experienced friend told me
"don't be a pussy, test in production"

I'm the one fixing the bugs, not him

Twitter doing test on production... Interesting. At least it worked 😂

Hey, Root? How do you test your slow query ticket, again? I didn't bother reading the giant green "Testing notes:" box on the ticket. Yeah, could you explain it while I don't bother to listen and talk over you? Thanks.

And later:
Hey Root. I'm the DBA. Could you explain exactly what you're doing in this ticket, because i can't understand it. What are these new columns? Where is the new query? What are you doing? And why? Oh, the ticket? Yeah, I didn't bother to read it. There was too much text filled with things like implementation details, query optimization findings, overall benchmarking results, the purpose of the new columns, and i just couldn't care enough to read any of that. Yeah, I also don't know how to find the query it's running now. Yep, have complete access to the console and DB and query log. Still can't figure it out.

And later:

Hey Root. We pulled your urgent fix ticket from the release. You know, the one that SysOps and Data and even execs have been demanding? The one you finished three months ago? Yep, the problem is still taking down production every week or so, but we just can't verify that your fix is good enough. Even though the changes are pretty minimal, you've said it's 8x faster, and provided benchmark findings, we just ... don't know how to get the query it's running out of the code. or how check the query logs to find it. So. we just don't know if it's good enough.

Also, we goofed up when deploying and the testing database is gone, so now we can't test it since there are no records. Nevermind that you provided snippets to remedy exactly scenario in the ticket description you wrote three months ago.

And later:

Hey Root: Why did you take so long on this ticket? It has sat for so long now that someone else filed a ticket for it, with investigation findings. You know it's bringing down production, and it's kind of urgent. Maybe you should have prioritized it more, or written up better notes. You really need to communicate better. This is why we can't trust you to get things out.

*twitchy smile*

FML. An overreaching supergenius "architect" and a database team:

A: "We have decided that apps should use mysql. Install a MySQL so we match cloud"
DBA: "we don't have an image or experience with MySQL. We have mssql and Oracle "
A: "ok, use mssql in data center and mysql in production cloud"
DBA: "that's... not going to work well"
A: "just do it!"

...

Me, reading this shit, sends email: "ignoring the fact that we have more than 500 queries in this application which will need to be checked and most likely rewritten, how are we supposed to test the mysql queries without production access?"
A: "just use mssql local and MySQL in cloud"
M: "... Just to make sure I understand, you want us to write queries for mssql, test them locally, and then write separate queries, with a separate SQL connection abstraction that deploys to production? Again, how are we going to test this?"
A: "no, use same queries, should be fine"
M: "they really won't, they're different dialects"
A: "do the needful, make work!"

If karma were a thing, this person would have long since exploded into a cloud of atomized blood.

Interview (first job):

Interviewer: So what languages do you know?

Me: Well, i learnd C, C++ and Matlab scripting, but i'm learning C# as a personal project.

Interviewer: Perfect!

First day:

Interviewer(now boss): So, a guy is leaving next week and you will be replacing him. He has 70 projects and you will be responsible for this production test platform in JAVA

Confession of the day:
1. I work in release mode
2. I work on the main branch only
3. I test on production

I quit and my last day is next week.

Apparently management has decided that I should spend my last day implementing a new feature for a customer where I have been the only developer, and release it to production (without first implementing it in test) the same day. A feature that potentially could cripple a whole workflow if done wrong.

Of course I advised not to release untested code to production on a friday, just before the only person that knows how it works leaves the company. But no, “the customer reaaaaaally wants it before summer, so just be careful not to write any bugs”.

I’m not saying that I’m intentionally gonna write bad code - but if I do, I’m not gonna pick up the phone when it calls.

Every company has a test environment. Some are lucky enough, to also have a production environment

So Facebook declared millions of people dead ... I guess that's what happens when you test in production

Guess who just pushed a whole week's work straight into production without a single damn test and everything works fine?😎😎😎

IF LIVES DEPEND ON A SYSTEM

1. Code review, collaboration, and knowledge sharing (each hour of code review saves 33 hours of maintenance)

2. TDD (40% — 80% reduction in production bug density)

3. Daily continuous integration (large code merges are a major source of bugs)

4. Minimize developer interruptions (an interrupted task takes twice as long and contains twice as many defects)

5. Linting (catches many typo and undefined variable bugs that static types could catch, as well as a host of stylistic issues that correlate with bug creation, such as accidentally assigning when you meant to compare)

6. Reduce complexity & improve modularity  -- complex code is harder to understand, test, and maintain

-Eric Elliott

My boss last week backed me up with a client. I was working in a production enviorment and the client refused to test the changes made when I told them. So we things started to go wrong and they called to my boss complaning.

He said:
Well you are right. The things are broken but he (me) stayed last night waiting for your response and you didn't gave one. So he is going to work only in the afternoon and you will have to wait.

I must buy a beer for my boss.

I extracted a tangled action to its own api, and wrote a test for it.

The test failed.

I added debugging, more debugging, all the debugging. It still fails. But I can at least see why it fails!

It turns out the api finds and updates the wrong user. It finds and updates the wrong user EVEN WHEN THERE ARE NO OTHER USERS.

WHAT THE SALAMI.

Also, the user lookup it uses is extremely roundabout and takes several seconds with ~2 million users. Normally I'd fix the lookup, but it has been in production for several years, and I'm terrified it will break something if I fix it.

Blargherhagrid.

!dev I'd just helped a client cut over to a new fiber connection and then left for Vegas, about 2 days into the trip my wife and I decided to hit a breakfast spot that had bottomless mimosa's, which was of course a claim we had to test.

As we are walking(stumbling) out of the restaurant I get a call that the connection has crashed and the entire car dealership is unable to sell cars, which they tell me is important functionality.

So I make it up to my room and break out the laptop, luckily the mgmt interfaces are still available externally so I'm able to log in and then have the fun challenge of 1) not falling off of my chair 2) not accidentally making a change that kills what connection I have in and 3) fixing their actual issue.

Took me almost an hour to find a simple OSPF issue but at least got them working and happy. However by that time I was beginning to sober up, which is the absolute worst thing that can happen while day-drinking and ended up basically causing me to be be hung-over for the rest of the night, including my wifes friends wedding, which she wasn't thrilled about...

The moral of this story is to make sure to NOT stop drinking while dealing with unexpected production impacting events.

When you test in Production. Been there 😁

Some years back I was working in a project that essentially dealt with all things related to foreigners and foreign affairs in Switzerland. You could manage entry visas, work permits, citizenship, international warrants, Interpol requests, etc.

One of the test managers (from client side - i.e. the government) was once manually "testing" and mixed up the production and test instance, to both of which he was logged in at the time.
The test case then ended up setting up an entry ban against himself, as he used his own name for testing...

Next time he returned from vacation the border control at the airport were like "Uhm, Sir, we can't let you into the country. Please come with us." :D :D

(He managed to clear that up in end, I dare say, though, that he learned his lesson.)

I was only seventeen back then and I was a Java Developer Intern, not knowing much about enterprise oriented coding.

The project leader in our dev team saw a lot of potential and passion in my work, but was convinced I wasn't taught enough to do the right thing.

I was mainly doing shitty mappers and services back then, which were somewhat used but never lasted long and were ditched a few months later, which always bummed me out. I wanted to make an impact on REAL projects that would deploy into production.

So Mister Mentor (GDPR forbid to use the actual name), who was always first to come and last to leave the office, taught me what it means to code for real.

We stayed after 5pm until 7-8pm multiple times a week and he taught me in a deeply understanding and calm way how to:
- Git (SVN)
- Refactor
- SOA
- Annotate
- Deploy
- Unit Test

And most importantly:
- How to debug like an absolute BOSS
(We even debugged native Java Libraries just for fun to see if we could break them)

Fast-forward a month later and little intern me made his first commit on production.

Without Mister Mentor, I wouldn't be half as good of a developer as I am today.

A few days after deploying a big important Website into production, I wanted to copy the whole thing including DB back onto our test server for future testing/bug fixing if something comes up. (Last changes were done on production server before going live)
So I opened SSH, removed everything on the test sever aaaaand then I realized I was connected to production...
Took about an hour to get everything up and running again. We didn't tell the client and hoped it would not be noticed.

So we hired an intern and his first task was to change a few things in email layout for our client, which is an investment bank.

I told to one of my developers to make his local database dump and setup the project for an intern. When intern completed the task, my developer thought that title "Dow Jones index crashed" was pretty funny title for a test.

What he didn't thought through enough, is that he forgot to configure fake SMTP server and he had production database dump with real email addresses.

I had really awkward 20 minutes conversation with our client. Fuck my life.

I worked for over 13 hours yesterday on super-urgent projects. I got so much done it's insane.

Projects:
1) the printer auto-configuration script.
2) changing Stripe from test mode to live mode in production
3) website responsiveness

I finished two within five minutes and pushed to both QA and Production. actually urgent, actually necessary. Easy change.

The printer auto-configure script was honestly fun to write, if very involved. However, the APIs I needed to call to fetch data, create a printer client, etc... none of them were tested, and they were _all_ broken in at least two ways. The CTO (api guy in my previous rant) was slow at fixing them, so getting the APIs working took literally four hours. One of them (test print) still doesn't work.

Responsiveness... this was my first time making a website responsive. Ever. Also, one of the pages I needed to style was very complicated (nested fixed-aspect-ratio + flexbox); I ended up duplicating the markup and hacking the styling together just to make it work. The code is horrible. But! "Friday's the day! it's going live and we're pushing traffic to it!" So, I invested a lot of time and energy into making it ready and as pretty as I could, and finally got it working. That page alone took me two hours.

The site and the printer script (and obv the Stripe change as well) absolutely needed to be done by this morning. Super important.

well.

1) Auto-configure script. Ostensibly we would have an intern come in and configure the printers. However, we have no printers that need configuring, so she did marketing instead. :/ Also, the docs Epson sent us only work for the T88V printer (we have exactly one, which we happened to set up and connect to). They do not work for the T88VI printers, which is what we ordered. and all we'll ever be ordering. So. :/ I'll need to rewrite a large chunk of my code to make this work. Joy :/

2) Stripe Live mode. Nobody even seemed to notice that we were collecting info in Test mode, or that I fixed it. so. um. :/

3) Responsiveness.
Well. That deadline is actually next Wednesday. The marketing won't even start until then, and I haven't even been given the final changes yet (like come on). Also! I asked for a QA review last night before I'd push it to production. One person glanced at it. Nobody else cared. Nobody else cared enough to look in the morning, either, so it's still on QA. Super-important deadline indeed. :/

Honestly?

I feel like Alice (from Dilbert) after she worked frantically on urgent projects that ended up just being cancelled. (That one where Wally smells that lovely buttery-popcorn scent of unnecessary work.)

I worked 13 hours yesterday.
for nothing.

fucking. hell.

A quote to live by

I started this new freelance project where I am building some android libraries for the client. Anyways, during meeting I was about to present my results and suddenly backend seemed to be down. I looked into the round "are your servers down?"
Team Lead: "Yeah our cto, also our only backend dev, is developing a new feature."
Me: "Okay but why is production down?",
Team Lead: "Ah dont worry we always test on production. We have a pretty solid hardware, we will even upgrade it soon!"
Me:"... How about you just separate your stage environments and have a develop environment?"
Client: "see, this is where our strength is. We dont need a develop stage we have very strong hardware and our backend is fully in PHP"

Thanks God I'm a freelancer

I have to start my best moment last year with a confession: I moved from Dev to Test half a decade ago. Naturally I do a lot of automation. My Best moment was when Dev said my automation code is so well structured that he wants to work on that and not an the production code anymore. Gave me that warm "still got it" feeling 😊

Data Analyst: “the task failed in test, can we try running it in production?”

My life as a Data Engineer.

#9

Of course they don't use git. And also they don't use SSH all changes get committed by FTP.

#9.1

When I started he gave me root access and I had to clone the whole fucking thing, wich was about 2gb, via FTP.

#9.2

He stumbled when I told him, that I will test all changes first on my local machine. They were used to work in production.

😓🔨

Soooo, i'm assigned to finish a half made prototype of a production line functional test platform.

1 - Some wires were faulty so someone decided to change them.

2 - wild shortcut appears *music fades, screen goes black*

3 - now i have to fix the prototype

4 - Devil's voice whispers on my ear *welcome to heeeeellllllll*

Old boss story. This guy was nice but a terrible boss. Also relevant, he has a background in IT so should know better.

Him: So when you wanna check a password is correct you just unhash it in the database?
Me: *facepalm*

Me: Hey we should be doing unit and integration testing at a minimum to lower bugs.
Him: We don't need those, we're not a bank. If a problem comes up we just fix it and push to production.
(A while later)
Him(in email): Why do we keep getting bugs reported. Don't you devs test your code.

I was mildly annoyed at that one.

Him: We're always over budget on projects, how can we fix this.
Me: What if we increase our quotes.(technically there are other ways as well but not really possible at that time)
Him: We can't do that, clients won't want to pay.

Me: *finishing off my handover as I'm leaving for a new job*
Him: Wow you do a lot of work

If any of you guys is (or was until recently) a developer for Smappee, the payload message test was a success

Fixing someone else's code who left the job.
Production suddenly not working, cannot debug locally, cannot deploy to a test environment because it does not exists anymore.
There should be a contract clause that developer need to support his project for 2 years after he leaves his job.

We have 2 layers of testing environments and production.

I tested the changes on the 1st layer, bud since it was 5min to lunch i did not test on 2nd layer which is connected to the production DB. I pushed to production and caused 5+ websites to go full retard and went to lunch.

Came back to 19emails and 3+ skype msgs about "why the fck would you do that..."

Estimated damages nearly 20k EUR and i lost some permissions for two weeks, but my great boss helped me out and cheered me up by telling stories how he took down multiple servers too

plot twist: im the team leader of our office now :)

Never ever have a tab open on production server database. Changed all users passwords by mistake. Thought was my test database.

I have to refactor code from an intern. He's VERY lucky that he already left the company.

If I'd say he programms like the first human that would be very insulting to that first human.

It looks like code at first sight, but when you try to understand what he was doing to achieve his goal you get a brainfuck. Duplicate code, unused code, dumb variable names like blRszN.

He wrote unittests like "expects Exception to be thrown or Server returns Statuscode 500".
Yes, Exception, the generic one.

THESE FUCKING TESTS ARE GREEN BECAUSE YOU DID NOT ACTUALLY TEST SOMETHING.
GREEN IN THIS CONTEXT MEANS: YOUR PRODUCTION CODE IS A BIG PILE OF SHIT.

I already removed 2 bugs in a test which caused another exception than the "expected" one and the test does still not reach the actual method under test.

Dumb fucktard.

The sad thing: The fuckers who did the code reviews and let this shit pass are still here writing code.

So far all designers I worked with do the following:
1. Use "creativity" to come up with stuff that the system does not allow implementing, for example: Changing clock color in mobile statusbar to Blue!
2. Use "creativity" to come up with a heavily customized calendar for a windows software which requires building the control from scratch, but they explain their creativity by saying: Can't you use CSS?
3. Provide iOS only design which follows android guidelines and refuse to provide android styles for at least pages that to be handled differently on each platform, for example, we had a checkout page in an app, and they wanted the same style for both WITHOUT building custom control for it, they said: Can't you use the android custom control inside iOS
4. They design for a website and send same mockups for me to implement on mobile apps, the problem is a web page runs on a big screen when the mobile app doesn't have room for half the stuff they designed but they must look exactly the same as website !!
5. They send entire PSD with no color codes and say: You can extract icons, and colors from psd ... When they should provide them as per our request which is: SVG for Android and PDF for iOS with the color codes, but no, they are lazy!
6. They ask the team to create a page in the app which is almost production ready just so that they test different font sizes and see how it will look on the phone
7. Same as #6 for images that contain text

The list goes on, but those are by the far the ones that made me one step away from resigning, some of them made me resign...

So, a few years ago I was working at a small state government department. After we has suffered a major development infrastructure outage (another story), I was so outspoken about what a shitty job the infrastructure vendor was doing, the IT Director put me in charge of managing the environment and the vendor, even though I was actually a software architect.
Anyway, a year later, we get a new project manager, and she decides that she needs to bring in a new team of contract developers because she doesn't trust us incumbents.
They develop a new application, but won't use our test team, insisting that their "BA" can do the testing themselves.
Finally it goes into production.
And crashes on Day 1. And keeps crashing.
Its the infrastructure goes out the cry from her office, do something about it!
I check the logs, can find nothing wrong, just this application keeps crashing.
I and another dev ask for the source code so that we can see if we can help find their bug, but we are told in no uncertain terms that there is no bug, they don't need any help, and we must focus on fixing the hardware issue.
After a couple of days of this, she called a meeting, all the PMs, the whole of the other project team, and me and my mate. And she starts laying into us about how we are letting them all down.
We insist that they have a bug, they insist that they can't have a bug because "it's been tested".
This ends up in a shouting match when my mate lost his cool with her.
So, we went back to our desks, got the exe and the pdb files (yes, they had published debug info to production), and reverse engineered it back to C# source, and then started looking through it.
Around midnight, we spotted the bug.
We took it to them the next morning, and it was like "Oh". When we asked how they could have tested it, they said, ah, well, we didn't actually test that function as we didn't think it would be used much....
What happened after that?
Not a happy ending. Six months later the IT Director retires and she gets shoed in as the new IT Director and then starts a bullying campaign against the two of us until we quit.

Aaaah, I fucking love it to death, when customers spontaneously decide to hire a separate, unrelated company to add new content pages to the website developed by our company.

That furuncle of a company must have had real pro devs to just create a new /html folder, dump their shit content in there and just manually add links in the existing CMS pages.
HOLY FUCK!
As you might already have expected, the /html folder contains:
- static *.html files for every page
- inline CSS in the *.html
- the crappiest PHP mailing script I have ever witnessed
- images with random resolutions, mostly too small

The layout of these puke-ridden pages obviously doesn't fit neither the existing color palette, nor has anything common with the current layout or typography at all.
These bastards don't even use Git!

Come on, dear customer, could you PLEASE fucking NOT hire a completely separate company to do OUR job?
PLEASE? PLEASE?!

I had to compare the whole deployment folder with our repo to find out what else these brain-damaged cunts changed in our code!

Just got this notification. I see virgin active UK likes to test in production. I too like to live dangerously.

Production server credentials "test/test" 😱😨😱

It's about a guy that knows better.

I was working as a subcontractor on a bigger system. We (subs) were not allowed to deploy code, we had to wait for contractor to deploy.

One day I got an email that my code is bugged and that my feature is not working on production. I checked it on test env, everything was fine. Then I checked if the code I wrote was deployed. It was not.

I send an email explaining that if they deployed my code it would be working. Then I got a response. There was a bug in my code.

Another email. I asked how would they know? Do they have a test on their environment that failed?

No. There is one guy that READ my code and he said it should not work, so he will not deploy it. He was not a programmer, he was a business consultant responsible for the documentation.

His issue was that I used a function that was not in a class. So if the function is not declared it's obvious it will not work. I had to explain to him in another email, that you can use object of another class inside your class and then call a function, that is not in your class. It was the last time this guy blocked my deploy.

TL;DR, I had to explain a non-dev how object composition works in order to have my code deployed. Took four emails.

When you let an Intern test in production

Hiring process is fucking broken ok?
We all do have something else to do, nobody wants to do "homework" for 4 fucking hours. Which let's be real, isn't 4 hours. It's always more. I try to squeeze it in a least amount of time which means mistakes will be made. I always try to show my knowledge of the language and it's features. But, you didn't do X. That's it, that is a no from us.
Dude, I just wrote a high production grade small project with 90%+ test coverage and you are telling me that those 2 small shit I made is a big deal? Fuck off

Most companies I worked with have a code full of shit and here I present to you, with a poetry and it's a no because of X?

My bet is that if I ever started to work there I would find a code that isn't tested and is in shit state

\rant

Friend of mine: so I wonder how do you test your applications in the startup?

Me: testing? *grabs his coffee laughing*

Actually we have a complete build pipeline from commit/pull-request to dev and production environments. No tests. Really. We are in rapid product development / research state.

We change technologies and approaches like our underwear (and yeah, this is frequently). If we settled some day and understood the basic problems of the whole feature palette, we'll talk about tests again.

I hate it when I ask the customer where did they test it or where the error is.. On test or on production? And the answer is yes. O.o

Me: We really need to improve our unit test coverage.
Team/Boss: <sarcasm> Haha yeah.
Production Bug: I'm doing something nasty to a client, because a dev broke something but no test coverage.
Boss: How could we have prevented this?!

When you test your backend code thoroughly before pushing to production, but a fatal exception with the much larger production userbase causes one of your vital threads to die with a NullPointerException.

Let me tell you the story of how a feature request no one asked for got put in an early grave:

PM walks into weekly meeting with a single use case that one user called in about, despite never having this issue during the past year and a half that our app has been in production. PM's boss (genuinely one of the best people i have ever worked with) happens to sit in this particular meeting for no reason other than he felt like he should once in a while.

PM brings up use case and wants to devote 3 weeks' development time and another 3 weeks to test RIGHT NOW while other projects are already in motion. PM's boss speaks up with this: "Listen if this guy is really this upset, we can just tell him to build his own service. All the other end users have no problems with this, so it's not worth spending the resources on, i don't think."

And that is how i went from "this is bullshit" to "i love you" in the span of 20 minutes.

So, to anyone defending IBM at this point, a member of a client's offshore team used their paystub as test data. Aaaaand I was horrified by what I saw.

Their pay is less than $2/hr ($3973/yr, 300k INR).

I can't even. Not only that someone would pay so little to a supposedly degreed professional (I question the validity of that claim based on performance, that's a story for another time), but that companies feel comfortable giving full production system access to people I would not blame for taking bribes.

Fuck.

Client: can you filter boats by location?
Me: Let me see... As you know, there are three remote systems that feed data into your database. I'd have to make a connection between the location records. But I can't rely on coordinates, name, ID or anything else. You'd have to manually create those links for me by remote systems records IDs. Telling me that record XY from system A is identical to record YX from system B, etc...
Client: How many records are we talking about?
Me: 504.

Three days later...

Client: Got it, is that enough for you in excel?
Me: Let me see... Very nice work, I can work with that.
Client: I almost died on it!

An hour later...

Me: Got it, test it and let's run it on the production version.
Client: It works beautifully.

A minute later...

Can we filter the ships by ports?
Me: Let me see... Yes, it's theoretically possible, but it's the same situation as with places...
Client: How many records are we talking about?
Me: 12,647.

Skype relayed to me the sound of something heavy falling, something grunting. Something dying.

OMFG I don't even know where to start..
Probably should start with last week (as this is the first time I had to deal with this problem directly)..

Also please note that all packages, procedure/function names, tables etc have fictional names, so every similarity between this story and reality is just a coincidence!!

Here it goes..

Lat week we implemented a new feature for the customer on production, everything was working fine.. After a day or two, the customer notices the audit logs are not complete aka missing user_id or have the wrong user_id inserted.
Hm.. ok.. I check logs (disk + database).. WTF, parameters are being sent in as they should, meaning they are there, so no idea what is with the missing ids.

OK, logs look fine, but I notice user_id have some weird values (I already memorized most frequent users and their ids). So I go check what is happening in the code, as the procedures/functions are called ok.

Wow, boy was I surprised.. many many times..
In the code, we actually check for user in this apps db or in case of using SSO (which we were) in the main db schema..
The user gets returned & logged ok, but that is it. Used only for authentication. When sending stuff to the db to log, old user Id is used, meaning that ofc userid was missing or wrong.

Anyhow, I fix that crap, take care of some other audit logs, so that proper user id was sent in. Test locally, cool. Works. Update customer's test servers. Works. Cool..

I still notice something off.. even though I fixed the audit_dbtable_2, audit_dbtable_1 still doesn't show proper user ids.. This was last week. I left it as is, as I had more urgent tasks waiting for me..

Anyhow, now it came the time for this fuckup to be fixed. Ok, I think to myself I can do this with a bit more hacking, but it leaves the original database and all other apps as is, so they won't break.
I crate another pck for api alone copy the calls, add user_id as param and from that on, I call other standard functions like usual, just leave out the user_id I am now explicitly sending with every call.
Ok this might work.

I prepare package, add user_id param to the calls.. great, time to test this code and my knowledge..

I made changes for api to incude the current user id (+ log it in the disk logs + audit_dbtable_1), test it, and check db..
Disk logs fine, debugging fine (user_id has proper value) but audit_dbtable_1 still userid = 0.

WTF?! I go check the code, where I forgot to include user id.. noup, it's all there. OK, I go check the logging, maybe I fucked up some parameters on db level. Nope, user is there in the friggin description ON THE SAME FUCKING TABLE!!
Just not in the column user_id...

WTF..Ok, cig break to let me think..

I come back and check the original auditing procedure on the db.. It is usually used/called with null as the user id. OK, I have replaced those with actual user ids I sent in the procedures/functions. Recheck every call!! TWICE!! Great.. no fuckups. Let's test it again!
OFC nothing changes, value in the db is still 0. WTF?! HOW!?

So I open the auditing pck, to look the insides of that bloody procedure.. WHAT THE ACTUAL FUCK?!
Instead of logging the p_user_sth_sth that is sent to that procedure, it just inserts the variable declared in the main package..
WHAT THE ACTUAL FUCK?! Did the 'new guy' made changes to this because he couldn't figure out what is wrong?! Nope, not him. I asked the CEO if he knows anything.. Noup.. I checked all customers dbs (different customers).. ALL HAD THIS HARDOCED IN!!! FORM THE FREAKING YEAR 2016!!! O.o

Unfuckin believable.. How did this ever work?!

Looks like at the begining, someone tried to implement this, but gave up mid implementation.. Decided it is enough to log current user id into BLABLA variable on some pck..
Which might have been ok 10+ years ago, but not today, not when you use connection pooling.. FFS!!

So yeah, I found easter eggs from years ago.. Almost went crazy when trying to figure out where I fucked this up. It was such a plan, simple, straight-forward solution to auditing..

If only the original procedure was working as it should.. bloddy hell!!

Half way through a 2 hour deployment, and a fucking test fails.
Yes, it takes 2+hours to run the tests.

Rerun in test environment: pass
Rerun in prod: pass
Rerun changed test in prod: fail..

Why, why you got to hate me for?
I love it when production is the place where config gets changed.

Just test it on production.

Typos kill, kids! And deploying to production.

Instead of "for item in items" in my script, I accidentally did "for items in items". Thus, an exponential loop has been entering things into the database for the past few hours before I found the place to fix it.

By the way, this runs on cron every minute. So there are processes still running exponentially right now, possibly 180+.

Yeah, I'm setting up a a test server instead now.

We started a project in January for which I was the sole developer, to automate tedious interaction with a vendor's ticketing system. We have a storage environment with about 400,000 commodity disks attached(for this vendor-- there are other vendors too), in sites around the US and Canada. With a weekly failure rate of about 0.0005%, that means about 200 disks a week need to be replaced.

This work-- hardware investigation through storage appliance frontends, internal ticket creation, external ticket creation, watching the external ticket for updates to include in our internal ticket --was all manual, and for around 200 issues a week, it was done by one guy for two years. He was hopelessly behind. This is all automated now, and this morning, I pushed this automation from dev/test to production.

It feels great to see your work helping people around you.

Wow i must have been brain dead when i wrote this code. Needed to exclude certain elements from response for the the list of objects.

for (obj : objects) {
If (obj.skipFromResponse()) {
break
}
add obj to response
}

I used break instead of continue at the if condition which meant it would break out of the loop at the first instance of condition being met.

This went through qa and has been in production for 4 weeks so how did this not break before. Well little did i know the list of objects was sorted and all the test data, qa data and everything so far in production coincidentally only had the last element with matching condition. This meant it returned everything correctly so far.

Today was the first time there was a situation where this caused incorrect output. Luckily as soon as I heard the description of the issue I remembered to check the merged PR and hung my head in shame for making such trivial error. I must have written way more complicated code without any problem but this made me embarrassed to even admit. 🤦‍♂️

Tomcat

manager: "hey, we have this old java software running and need it to be compatible with our brand new crm system"
me: "okay, i've never workes with jsp before, but i'll have a look at it"
code: undocumented, who would have guessed
manager: "oh and btw. you must test your code on our live system, with our production database, but make sure not to brake anything, our last backup was 20 months ago."
me: "..."

Can't say I am a religious type, not really into discovering/discussing religions and comparing them either... BUT I do hope there is someplace (like hell on steroids!!) for developers who don't test their code before checking it in and/or puting it on production..
Also another question, can I plead not guilty due to insanity when killing such devs?!? O.o FUuuuuuuuu!!!!

They test in production

Our team really needs some workflow arrangement, and this time it was me who screwed up.
So we have to push an update to the Play Store and the App Store the Friday, the app is well tested on test environment then production environment, we got the ok so I uploaded a build, the app management team then continued the process of publishing..

During the weekend the app was approved and live to almost 500k user that can receive the update.

I got a phone call from the Project Manager at almost midnight, the time was really suspicious so I answered.
- Me: Hello.
- PM: Hi, sorry to call you now but the app is live and we have a problem.
- Me: what kind of problem? Let me check.

So I updated the app on my phone and opened it while I am on call.. I almost had heart attack!! WE PUBLISHED A VERSION POINTING TO THE TEST ENVIRONMENT. Holly shit

- Me: shit call the app management team NOW.

Eventually we removed the app from sale (unpublished it) and we submitted a new version immediately, once it was approved the next day we made the app available again (so for those who didn’t update yet, there will be no update to a faulted version, and no new users landing to a version with test data), I received one or two calls from friends telling me why the app is not on the store (our app is used nationally, so it’s really important).

Thank God there was no big show on twitter or other social media.. but it’s really a good lesson to learn.

I understand this is totally my fault, thankfully I didn’t get fired 😅

I love Test-Driven Development!

And because of that fact, my heart shatters into thousands of pieces, when I recognize error events on our production nodes which are pointing onto a golden hammer function in a legacy project.

This particular function has about 300 lines with a bunch of subfunction calls and instantiations of helper-classes returning information for workflow.

Refactoring this code to apply proper unit-tests requires a way bigger investment than simply deal with 30 eventlogs a day, because this kind of payment is barely used by customers of our webshop.

This fact is a little itch each day of my work.

Guess it will make me go insane one day
¯\_(ツ)_/¯

xD

So this just happened...don't test on production kids (....sometimes you have no choice BUT DON'T F**KING DO IT OK!?)

It's 5 AM and I don't want to shit on anybody's party but trust me when I say most of you here complaining about legacy code don't know the meaning of the word.

As someone who maintained a PHP4 codebase with an average file length of 3000+ lines for almost 4 years, I feel you, I feel your pain and your helplessness. But I've seen it all and I've done it all and unless you've witnessed your IDE struggle to highlight the syntax, unless you had to make regular changes in a test-less SVN's working copy that **is** the production and unless you are the reason that working copy exists because you've had enough of `new_2_old_final_newest.php` naming scheme, you do not know legacy. If you still don't believe me bare in mind I said "is" as in: "this system is still in production".

But also bare hope. Because as much grief as it cost me and countless before me, today of all days, without a warning, it got green lit for userbase migration to a newer platform. And if this 20 years of generous custom features and per client implemented services can be shut down even though it brings more profit than all the other products combined, so can happen to any of your projects. 🙏

Unfortunately, I do mean *any*.

I used to work on a production management team, whose job was, among other things, safeguarding access to production. Dev teams would send us requests all the time to, "run a quick SQL script."

Invariably, the SQL would include, "SELECT * FROM db_config."

We would push the tickets back, and the devs would call us, enraged. I learned pretty quickly that they didn't have any real interest in dev, test, or staging environments, and just wanted to do everything in prod, and see if it works.

But they would give up their protests pretty fast when I offered to let them speak to a manager when they were upset I wouldn't run their SQL.

This was some time ago. A Legendary bug appeared. It worked in the dev environment, but not in the test and production environment.

It had been a week since I was working on the issue. I couldn't pinpoint the problem. We CANNOT change the code that was already there, so we needed to override the code that was written. As I was going at it, something happened.

---
Manager: "Hey, it's working now. What did you do?"
Me: *Very confused because I know I was nowhere close to finding the real source of the problem* Oh, it is? Let me check.
Also me: *Goes and check on the test and prod environment and indeed, it's already working*
Also me to the power of three: *Contemplates on life, the meaning of it, of why I am here, who's going to throw out the trash later, asking myself whether my buddies and I will be drinking tonight, only to realize that I am still on the phone with my manager*
Me again: "Oh wow, it's working."
Manager: "Great job. What were the changes in the code?"
Me: "All I did was put console logs and pushed the changes to test and prod if they were producing the same log results."
Manager: "So there were no changes whatsoever, is that what you mean?"
Me: "Yep. I've no idea why it just suddenly worked."
Manager: "Well, as long as it's working! Just remove those logs and deploy them again to the test and prod environment and add 'Test and prod fix' to the commit comment."
Me: "But what if the problem comes up again? I mean technically we haven't resolved the issue. The only change I made were like 20 lines of console logs! "
Manager: "It's working, isn't it? If it becomes a problem, we'll work it out later."
---

I did as I was told, and Lo and Behold, the problem never occurred again.

Was the system playing a joke on me? The system probably felt sorry for me and thought, "Look at this poor fucker, having such a hard time on a problem he can't even comprehend. That idiotic programmer had so many sleepless nights and yet still couldn't find the solution. Guess I gotta do my job and fix it for him. I'm the only one doing the work around here. Pathetic Homo sapiens!"

Don't get me wrong, I'm glad that it's over but..

What the fuck happened?

Dear client, if you can't be bothered to check more than two data points during several test imports, why are you surprised your production import has errors in the other 10k+ data points? We told you to check thoroughly, and you swore it was fine. But great now I get to unfuck production while you're mad you can't go live yet.

Why the hell do you keep commit+push your shit to the master branch! We have develop, feature and hotfix, you *************!

Just because you want to test something in prd environment, you don't mess with the master to build the production image. And you do not even rebase fucking develop branch and keep it out of sync you POS!

Excuse my language, thank you.

I was testing database migration and there was some issues which I couldn't fix them. So I drop the table just to test if the issues are still there.
"Why nothing have been changed?" After some minutes I realize it was production database which I had dropped 😓

So... Heard back from a recruiter today. Lovely lass.

I’d passed over a submission for her tech demo.

The brief was basically just to create a small simple module that calculates shit, nae effort.

But, when the recruiter had me on the phone she said “I know it’s a silly small module but try and run it up like you would a production ready app”.

The job spec and recruiter were keen on me demonstrating TDD, not specific on js version, final runtime, etc. The job was a senior spec at a higher salary range. So it warranted some effort, and demonstrating more than a simple module.

“Okay, cool, nae bother, let’s crack on.”

The feedback in the response from the dev today:
“He’s over-engineered tests, build...”

SUCK MY LEFT TESTICLE YOU FUCKWIT.

Talk to your recruiters, not me.

The feedback included a phrase I never hope to hear from a developer I work with:
“Tests are good but...” 😞

It was a standard 98% test suite from an RGR cycle, no more or less than I’d expect in prod.

The rest of the feedback was misguided or plain wrong. It was useful to see because I know now when they say they have “high standards” they mean: we listen to the dude who put the factory pattern in a JS brief.

Oh shit also: “someone’s done chmod 777” was in there as a sarcastic comment in the feedback. It was his fucking unarchive tool 😞

My response was brief and polite: “cheers for the consideration, all the best, James”

It’s honestly not worth warning them. Or, asking why they’d criticise something they’d asked me to do.

If you want a shitty js module, ask for a shitty js module and no more.

Discovered yesterday that my boss does tests with production database... and I'm responsible for the fucking backups and he doesn't even care to let me now when so I can at least schedule one at slave. Come on... it's not that hard to let others know or test on your own machine...

New mobile challenge I created:

... treat this as production code and show us your best practises and thought processes ... please list and explain any third party libraries used.

First submission:

I've used library X, although it has a major UI defect when rotating the device, it should be good for a simple test.

... wrong

I goofed up and forgot the WHERE clause in my UPDATE query. Accidentally all of production possibly updated because we don't do test databases. I think it didn't actually go through because I cancelled it but now I need to restore a backup and compare data. Which means explaining this to the co-owner who can help me with a restore. I'm mortified, more so because it was a stupid thing to do to begin with.

So after 6 months of asking for production API token we've finally received it. It got physically delivered by a courier, passed as a text file on a CD. We didn't have a CD drive. Now we do. Because security. Only it turned out to be encrypted with our old public key so they had to redo the whole process. With our current public key. That they couldn't just download, because security, and demanded it to be passed in the fucking same way first. Luckily our hardware guy anticipated this and the CD drives he got can burn as well. So another two weeks passed and finally we got a visit from the courier again. But wait! The file was signed by two people and the signatures weren't trusted, both fingerprints I had to verify by phone, because security, and one of them was on vacation... until today when they finally called back and I could overwrite that fucking token and push to staging environment before the final push to prod.

Only for some reason I couldn't commit. Because the production token was exactly the same as the fucking test token so there was *nothing to commit!*

BECAUSE FUCKING SECURITY!

My co-worker, still studying but working as a "senior dev", just decided that we don't need a test/staging environment anymore. We just "validate" (we also don't use the word "test" anymore) newly created features in production.

Makes absolutely sense...

Thank god I have a new job from february on!

When the test database has better specifications & performance than the production one.. FML! 😭😭😭😭😭

Ever dealt with people who don’t understand a damn thing they’re coding and just copy and paste stuff around and say that it works? It’s amazing how long people can skate by with only knowing 50% of how what they’re working on works.

We had some dumb know it all on our team who would regularly ship half-done stuff to production and then scramble to fix it after the users bitched about how it didn’t meet the requirements.

This stupid person changed an else to an if having no idea what any of the logic did in this section of code, didn’t adequately test it, and it somehow made it through code review because the better devs were out of office.

This bullshit goes to production, fucks up 200,000 records, and users start complaining about it. Shitty developer refuses to revert the offending code until multiple people on the team overrule them. They spend the next week unfucking the data and decides to just take a day off on Wednesday because it’s been “too much mental energy to fix.” The shit wasn’t even fully resolved yet.

Some people seriously do not belong in this industry. This person’s thought process was:
“Changing an else to an if can’t possibly have significant consequences. Let’s just change this so my code change executes to see what happens.”

Still not sure how they weren’t fired when this happened. They unfortunately got to quit on their own two months later.

In my three years experience so far I can honestly say that 100% of the developers I've worked with are narrow sighted with regards to how they develop.

As in, they lack the capacity to anticipate multiple scenarios.

They code with one unique scenario in mind and their work ends up not passing tests or generates bugs in production.

Not to say I'm the best at foreseeing every possible scenario, but I at least TRY to anticipate and test my code as much as possible to identify problems and edge cases.

I usually take much more time to complete tasks than my colleagues, but my work usually passes tests and comes back bug free. Whereas my colleagues get applauded for completing tasks quickly but end up spending lots of time fixing up after themselves when tests fail or bugs appear.
Probably more time wasted than if they had done the job correctly from the start. Yet they're considered to be effecient devs because they work "fast".

Frustrating...

Deploying to production 8pm Saturday evening.

Because I have confidence in our test suite, QA procedures, infrastructure and my team members are not morons.

Please don't create a test server that doesn't have the same data with the production server, and vice versa.

YOU WANT ME TO TEST AND DEVELOP IN MOTHERFUCKING PRODUCTION?!!!

Whaaaaaat, I'm changing shit, what if somebody goes to buy this product and I've made it super-cheap? ATM, there's two fuckin options for shipping, both different costs. the best bit? RIGHT NOW, THE USER CAN CHOOSE TO PAY LESS FOR THEIR SHIPPING.

HOLY FUCK.

Discovered pro tip of my life :
Never trust your code

Achievements unlocked :
Successfully running C++ GPU accelerated offscreen rendering engine with texture loading code having faulty validation bug over a year on production for more than 1.5M daily Android active users without any issues.

History : Recently I was writing a new rendering engineering that uses our GPU pipeline engine.. and our prototype android app benchmark test always fails with black rendering frame detection assertion.

Practice:
Spend more than a month to debug a GPU pipeline system based on directed acyclic graph based rendering algorithm.

New abilities added :
Able to debug OpenGL ES code on Android using print statement placed in source code using binary search.

But why?
I was aware of the issue over a month and just ignored it thinking it's a driver bug in my android device.. but when the api was used by one of Android dev, he reported the same issue. In the same day at night 2:59AM ....

Satan came to me and told me that " ok listen man, here is what I am gonna do with you today, your new code will be going production in a week, and the renderer will give you just one black frame after random time, and after today 3AM, your code will not show GL Errors if you debug or trace. Buhahahaha ahhaha haahha..... Puffff"

And he was gone..
Thanks satan for not killing me.. I will not trust stable production code anymore enevn though every line is documented and peer reviewed.

about 6 years ago I was working for a large consulting company on a government project. I put in a change for a stored procedure that hard coded the partition to 0, except 0 didn't exist on production, just on test. several thousand government employees couldn't access it for a day. 😞

Hey, we need a service to resize some images. Oh, it’ll also need a globally diverse cache, with cache purging capabilities, only cache certain images in the United States, support auto scaling, handle half a petabyte of data , but we don’t know when it’ll be needed, so just plan on all of it being needed at once. It has to support a robust security profile using only basic HTTP auth, be written in Java, hosted on-prem, and be fully protected from ddos attacks. It must be backwards compatible with the previous API we use, but that’s poorly documented, you’ll figure it out. Also, it must support being rolled out 20% of the way so we can test it, and forget about it, and leave two copies of our app in production.

You can re-use the code we already have for image thumbnails even though it’s written in Python, caches nothing and is hosted in the cloud. It should be easy. This guy can show you how it all works.

I didn't leave, I just got busy working 60 hour weeks in between studying.

I found a new method called matrix decomposition (not the known method of the same name).

Premise is that you break a semiprime down into its component numbers and magnitudes, lets say 697 for example. It becomes 600, 90, and 7.

Then you break each of those down into their prime factorizations (with exponents).
So you get something like

>>> decon(697)
offset: 3, exp: [[Decimal('2'), Decimal('3')], [Decimal('3'), Decimal('1')], [Decimal('5'), Decimal('2')]]
offset: 2, exp: [[Decimal('2'), Decimal('1')], [Decimal('3'), Decimal('2')], [Decimal('5'), Decimal('1')]]
offset: 1, exp: [[Decimal('7'), Decimal('1')]]

And it turns out that in larger numbers there are distinct patterns that act as maps at each offset (or magnitude) of the product, mapping to the respective magnitudes and digits of the factors.

For example I can pretty reliably predict from a product, where the '8's are in its factors.

Apparently theres a whole host of rules like this.
So what I've done is gone an started writing an interpreter with some pseudo-assembly I defined. This has been ongoing for maybe a month, and I've had very little time to work on it in between at my job (which I'm about to be late for here if I don't start getting ready, lol).

Anyway, long and the short of it, the plan is to generate a large data set of primes and their products, and then write a rules engine to generate sets of my custom assembly language, and then fitness test and validate them, winnowing what doesn't work.

The end product should be a function that lets me map from the digits of a product to all the digits of its factors.

It technically already works, like I've printed out a ton of products and eyeballed patterns to derive custom rules, its just not the complete set yet. And instead of spending months or years doing that I'm just gonna finish the system to automatically derive them for me. The rules I found so far have tested out successfully every time, and whether or not the engine finds those will be the test case for if the broader system is viable, but everything looks legit.

I wouldn't have persued this except when I realized the production of semiprimes *must* be non-eularian (long story), it occured to me that there must be rich internal representations mapping products to factors, that we were simply missing.

I'll go into more details in a later post, maybe not today, because I'm working till close tonight (won't be back till 3 am), but after 4 1/2 years the work is bearing fruit.

Also, its good to see you all again. I fucking missed you guys.

I run a service with 157311 (mostly inactive) users and I test in production.

When your college approaches you with an update and want to test it in production....

Borrowed from Reddit and Twitter:
Everybody has a testing environment. Some people are lucky enough enough to have a totally separate environment to run production in.

small victories... leaving little fuck you notes all over the code..
along with all the test cunts that still get incremented on the production..

Things that seem "simple" but end up taking a long ass time to actually deploy into production:

1. Using a new payment processor:

"It's just a simple API, I'll be done in 2 hours"

LOL sure it is, but testing orders and setting up a sandbox or making sure you have credentials right, and then switching from test to life and retesting, and then... fuck

2. Making changes to admin stats.

"'I just have to add this column and remove that one... maybe like a couple of hours"

YOU WISH

3. Anything Javascript

"Hah, what, that's like a button, np"

125 minutes later...

console.log('before foo');
console.log(this.foo)
etc..

Employer: Hey, we are moving an API update live tomorrow morning that could affect our apps. Can you regression test the apps to make sure they all work?

Me: The API team is pushing code overtop of live endpoints that can break them?

Employer: Yes, we need the updates to work with a new product we are developing.

Me: And nobody thought about versioning these endpoints so we guarantee uptime on all existing services using them now?

Employer: We looked at that but it cost extra and required us to use the cloud solution so we don’t use versioning.

Me: Okkk… I also take it that the API’s don’t have integration tests written?

Employer: What are integration tests? Are unit tests the same thing?

Me: No, so when do I need to regression test all 7 production apps?

Employer: The API’s are moving to production at 4am and we need it signed off by 7am.

Me: I only have 3 hours to regression test 7 production apps at 4am? Each app, if I just skim over them, would take me 2 hours each. I will do my best but that’s a very short time to ensure complete functionality.

Employer: Don’t you have unit tests?

If you don't know, there are 2 types of bug fixes:

Hot Fix - Patch files directly on the production

Quick Fix - Deploy fix on production and then test it

Yet another day at work:

My job is to write test libraries for web services and test others code. Yes I know to code, and have a niche in software testing.
Sometimes developers (whose code I find bugs in) get so defensive and scream in emails and meetings if I point out an issue in their code.
Today, when I pointed a bug in his repo, a developer questioned me in an email asking if I even understood his code, and as a tester I shouldn’t look at his code and only blackbox test it.

I wish I can educate the defensive developer that sometimes, it’s okay to make mistakes and be corrected. That’s how we deliver services that doesn’t suck in production.

(I'll give some context before the rant: I'm part if the IT department of a manufacturing company (actually I'm 1/2 of the department), and all the applications (old an new - except the ones used on production line) used in the company are my responsibility, that including most of databases too... Also, English isn't my native language so there will be some words or phrases that I'll probably write wrong... Sorry for that, if there are any corrections, I'll be glad to hear them)

So...

There will be an implementation of new "control point" on the "shipping department" which consists on a electromechanical equipment controlled by a PLC. And despite the original concept was a collaboration between 2 departments (we, IT, and Production Control), I was never taken in consideration about anything of the project... To be fair, I forget about its existence until two weeks ago.

So, a few days I learned that there are a huge delay regarding the original deadline (mainly because the supplier was delayed with the delivery of their system), and since two weeks (less, actually, because some holydays in between) I'm learning how to integrate that "P.o.S" into an existing application on a PC using a serial communication (not the main problem, as I've done that before... With another brand of PLC's) while avoiding buying any additional software (to get the communication done and in a easy way) and that sort of things... But discovering in the process that it will be necessary to acquire such additional SW in order to finish the job ASAP.

When suddenly I get the "news" that it's almost all my duty (and responsibility) to meet the original deadline, because it doesn't matter how the other departments screw all the schedule, it's the job of IT to get the shit done in time... And what is worst: they didn't said that in such straight manner, no, the implied it while making a quick test with the general manager.

I mean, WTF? Besides doing a "respectable" number of "user support" activities in a dialy basis, I also need to manage the activities of other departments? And also fix their screw ups on a schedule that I just learned days before?

And also there is a coworker (one of whom screwed up) that, almost every time she see me, is asking "how much until you'll finish?"

As I read on a meme years ago: "please, give patience, because if you give strength, I'll need bail money too..."

Damn... I don't know of the benefits of this work are worth all this nonsense

I'm getting really tired of those dumbass programmers that do not understand shit and then come to me when production breaks. (I am also a programmer, not really a DevOps engineer, but I'm the least worst at DevOps stuff, so it's my job...).

We're programming some kind of document management tool. Today we had a release, and one of the new features is to download all of your documents as a zip file, which is asynchronuously generated. When it's done, the user gets a mail with the download link to the zip file.

The feature works basically, but today it broke our production service, as somebody was running a test of it.

Turns out all the documents are loaded into memory to be zipped. So if you have 2 gigs of documents, a container with memory restrictions in that area will crash.

I asked the programmer who reported this «ops problem» to me, why he didn't just shit the files into a temp foler in order to zip them in there.

He told me that he wanted to do so, but did not know how to mock this for a unit test, and therefore went to the in-memory «solution», which was easier for him to mock.

For fuck's sake, unit tests and mocks are fucking tools, not ends in itself! I don't give a fuck about your pointless mocking code when the application crashes!

When I got to deal with such dumbasses, I'd prefer to mock those motherfuckers with a leaky bucket of liquid shit, which basically accomplishes the same task from my perspective: dripping shit all over the place and make everything suck as fuck.

"I always test in production"

This sentence by a colleague made me a murderer (almost).
Jesus...

You get to work, things have broken down in the night, you have no access to production or even test environment and you have to guess why. You do the same job as somebody in other countries for less pay while everyone else has this laid back approach where the time they actually spend working is negligible. Until the sheer amount of entropy in your organization wears you down and you just become part of the problem.

So last week I really fucked up

I had this new implementation that was supposedly to be integrating smoothly into the rest of the service. It depended on a serialized model made by a data scientist. I test it in local, in QA environment: no problem.

So, Friday, 4pm, I decide to deploy to production. I check once from the app: the service throw an error. Panic attack, my chief is at my desk, we triy to understand what went wrong. I make calls with cUrls: no problem. Everything seems fine. I recheck from the app again: no problem.
We dedice to let it in prod, as the feature work. I go get some beers with the guys, to celebrate the deploy.

Fast-forward the next morning, 11am, my phone ring: it's a colleague of my chief. "Please check Slack, a client is trying to use the feature, it's broken"

FUUUUUUUUUUUUCK!!!

Panic attack again. I go to the computer, check the errors: two types of errors. One I can fix, the other from a missing package on the machine that the data guy used.

Needless to say, I had a fairly good weekend.

Lessons learned:
- make sure Dev, QA and Prod are exactly the same (use Ansible or Container)
- never deploy on a Friday afternoon if you don't have a quick way to revert

Bug I had to fix today: some elements in our React app were being swapped with other elements.

We had `<foo>bar</foo>` on the component but on the html `foo` was being swapped with some other element in our app. It's contents ("bar" in the example) were being left in place, though, so we were getting `<baz>bar</baz>`.

This would only happen when running on production mode. On development everything was fine.

Also, everything seemed fine on the React dev tools. `foo` was where it was supposed to be, but on the html it was somewhere else.

Weirdest shit I ever saw when using React. I found a way to go around it and applied that fix, but I'm still trying to track this down to the source.

The worst part was waiting for fucking webpack to finish the production bundle on every fucking change I wanted to test. I didn't miss the change-save-compile-test flow at all.

What a shit day.

Next year I will strive to achieve the best test coverage on all our components and design all our new features using best-practice agile methodology with a realtime user involvement.

Reverend on 7 January: Fuck that, we need to ship this shit to production now.

Me: *asks boss for the id of his test store so I can apply experimental schema changes to test out a new dashboard app*

Boss: *gives me production store id and doesn't say anything*

Fate: … "You got lucky this time."

This is the CEO of the company btw. Startups. *sigh*

I dunno about coolest, but I did sort of cement my reputation as the "database guy" in my first job because of this.

My first job was with a group maintaining a series of websites. Because of the nature of the websites, every morning we had to pull the records from one database on one network, sneaker net the data to a database on another network, and import the data via custom data import function.

However, the live site would crash after 100 or so records were imported. The dba at the live site had to script out a custom data partitioning script to do his daily duties, but it definitely messed up his productivity.

Turns out, the custom mass import function had recycled the standard import function, which was only used to import 1 record at a time, and it never closed its database connections, because it never needed to. A one line fix to production code was delivered 6 months later (because that was our release cycle) and I came up with the temporary work around, which was basically removing the connection limit. It would still crash with the work around, but only with multiple days worth of data. So basically only on Monday. Also developed the test set for the import (15k+ records).

TL;DR Dear boss, firstly, you always get someone to review anything important done by a fucking intern.
Secondly, you do not give access to your fucking client's production server to an intern.
Thirdly, you don't ask your fucking intern to test the intern's work that has not been reviewed by anyone directly on your client's fucking production server.

Last week, the boss and one of the lead devs (the only guy with some serious knowledge about systems and networking) decided to give me (an intern who barely has any work experience) the task of fixing or finding an alternate solution to allowing their support team access to their client machines. Currently they used a reverse SSH tunnel and an intermediary VH but for some reason, that was very unreliable in terms of availability. I suggested using OpenVPN and explained how it would work. Seemed to be a far better idea and they accepted. After several days of working through documentations and guides and everything, I figured out how OpenVPN works and managed to deploy a TEST server and successfully test remote access using two VMs. On seeing my tests, the boss told me that he wanted to test it on the client network. I agreed. Today he comes to me and he tells me to prepare testing for tomorrow and that the client technician is going to give me access to one of their boxes. And then he adds, "It's a working prod server. We'll see if we can make it work on that" and left. I gaped at him for a while and asked another dev guy in the room if what I heard was right. He confirmed. Turns out, the lead dev and the boss's son (who also works here) had had a huge argument since morning on the same issue and finally the dev guy had washed it off his hands and declared that if anything goes wrong from testing it on production, it's entirely the boss's own fault. That's when the boss stepped in and approached me. I ran back to his office and began to explain why prod servers don't top the list of things you can fuck around with. But he simply silenced me saying, "What can go wrong?" and added, "You shouldn't stay still. You should keep moving". Okay, like firstly what the fuck and secondly, what the fuck?.
Even though OpenVPN client is not the scariest thing to install, tomorrow's going to be fun.

I didn’t turn down a dev freelance project when the client decided against going with best practices because the solution I offered was a well-established design pattern but created a need for a financial management change she didn’t like. I stupidly built what she asked for. It worked fine in the 3rd party vendor test environment but failed on production. After hours of analysis of code to ensure no changes happened to my source during test->prod deployment, and the vendor denying they had config differences between them, and the client refusing to pay, all I could do was abandon the project.

Remember to always setup and test your production setup!

Seem like a local college forgot to do some QA 😂

When you're YouTube and you test in production on the Apple TV app...

Just now... Got a job to create patch files for a couple of jars, which may or may not have varying class files. In total, I have to decompile, check, add and synchronize about 30 class files in 6 jars with a new functionality (that I didn't write). 🙂

FUCK PRODUCTION! WHY CANT YOU MAINTAIN ONE MOTHERFUCKING JAR?

OH? YOU'RE SUPERSTITIOUS THAT ONE TINY, ANT-SHIT SIZED CHANGE IN ONE SIMPLE FUNCTIONALITY WILL FUCK UP *OUR* PRODUCT?

FUCK MANAGEMENT! YOU DON'T HAVE CONFIDENCE IN YOUR *OWN* PRODUCT!

OH? CUSTOMER COMES FIRST? HAVE THE BALLS TO DEFEND YOUR OWN FUCKING SELF AND PRODUCT TO THE CLIENT OR THEY'RE GONNA MAKE YOU YOUR BITCH AND TIE A GAGBALL DIPPED IN HOT SAUCE AROUND YOUR MOUTH! HOW.. THE FUCK.. DID YOU MISS THAT LOGIC??????

Best part, they want it by tomorrow, and they don't wanna test it. Guess who's gonna get slaughtered after a week? ME! 🙂

FUCKING MICROSOFT IIS SHIT.
I'm a .NET dev since 13 years and EVERY FUCKING TIME STUPID IIS MOTHERFUCKER AND STUPID WINDOWS SERVER have a different problem setting up because of some permission.
You can't never get a site up in IIS without loosing time and patience having weird 400/500.x errors because every fucking machine have to set up some tweaky and hidden permissions.
I have 2 identical fucking win servers and deploying a .NET core applications and on one works (test server) and obviously, on the production server it gives troubles.

FUCK YOU MICROSOFT FUCK YOU I would take the IIS devs personally here and whip them to death until they don't resolve the fucking thing

Don't test complex functionality before production.. neither unit, nor system tests involved for my android app. Releaseing solely based on hope!

Simultaneously opening ssh sessions to test and production system, finally stopping the application in the factory.

It was me.

Now my client does not want to rely on Amazon S3 because of the One Outage that it ever had a couple what weeks ago I forgot already. So my dumbass blurts out well we could always just back up to some other image or file storing website. But now I'm expected to implement this right away when I really haven't thought about it at all I mean I would have to write some sort of failover and some sort of daily or syncing mechanism. I guess I should forget about any direct upload to S3 code that I have written. Really I guess I have to wrap all of the image and file handling stuff with my own solution. Which actually that will be very nice when it is done and I could use this on other projects but it's quite a lot of work for something that I don't feel we really need at this stage in development. Just because you're using stuff on production that has am enormous red TEST label in the way of the ui doesn't mean i can code bullet proof software any faster

BANE OF MY FUCKING EXISTENCE. STOP POLLUTING MY PRODUCTION CODE WITH TEST CODE, YOU FUCKING CRETINS.

Living on the edge!

One or two years ago I managed to deploy a DDL change directly on the production server. As I knew there was a backup job which will run every day at noon and at midnight. So I run my script some minutes after noon. So far so good. But somehow I tested it badly in my test environment and the UI of the application throws error after error now in production.

Well, just revert the db to the latest recovery point with the backup, I thought.

It became clear then after a couple of minutes of searching the backup folder for the db backup that there was no such file. The youngest backup file was 3 years old.

Now what happened: The backup script had a switch "simulate=true" and then simulated a successful backup on each run. Therefore the monitoring system got no alerts for not correctly executing those jobs correctly. Then the monitoring job which should do the backupfolder surveillance stuck with green, because there was a valid backup file inside. But it did not check for a specific creation date.

Now this database is the one we need for doing our daily business and is really crucial. Therefore It was easier to emergencyfix the application than doing a rollback of the db 🙄

Well, not really a data loss story, but close to one.

"Our system has detected that this message is[nl]421-4.7.0 suspicious due to the very low reputation of the sending IP address"

Oh shit..

Note to self: Don't test on production servers. Gmail has now blocked my ass.

Migrating to another sub domain for the 4th time.
First they say 'oh this sub domain is just for demo and you can freely test your code on it .
Few days later they say the same thing to another team and then :

- 'dude ! Is the website down because of your code ?'

- 'The code is not complete yet ! I've been adding some features! '

-'Is it possible that you migrate to another demo sub domain? Content production team wants to upload more content on that sub domain '

-'Sure ! Why not :('

Was just reading some of the OpenVPN scripts to renew a certificate where I forgot to source the vars file first (apparently OpenVPN stores those in a separate file that you always have to source first, and I tend to forget it sometimes).

Reading the revoke-full script that OpenVPN provides, it's just bash so I can read it no problem. But traversing through it and trying to understand it... Horrible! There's a test file in $RT named keys/revoke-test.pem. It's not used anywhere in OpenVPN for anything useful as far as I'm aware. The script however - the script that's running on a production server! - attempts to remove this file. It doesn't exist. Test files do (or at least should) not exist in production. They're not supposed to be there.

It exports empty variables. Some of them are set by the sourced vars file, some aren't. Not entirely sure why it's exporting variables as empty when they're uninitialized, or why it doesn't just unset the ones that are initialized.

And finally it goes ahead and revokes the key file that I'm actually concerned about through regular OpenSSL and verifies it.

Not to mention that the lack of the sourced vars file, which admittedly I should think about in the current status quo, if it *always* needs to be sourced anyway... Why doesn't the script do that itself then? One less thing to go wrong. But hey, proper design?

Gore. I don't have any other words for it.

And before anyone tells me that I should go and fix it if I'm so worried about it. Remember, I am not a developer. That's the job of the developers that made this in the first place.

Day 0: thank you for being an Amazon Customer, your database is about to be upgrade in the near future with or without your consent! Tough titties motherfucker!

Day 16: ok, every upgraded by hand in the test environment, everything seems stable, let's go make preparations for production!

Day 16.5: ssh user@<prod_bastion_ip> --yada --yada
Unable to connect

Oooook, let's try again,
Unable to connect

Day 16.5.1: WHY THE FUCK NOT, the IP is fucking right, the cert is right, the user is right, the..... fucking.... EC2 instance has been......... terminated.....

FML!
---

Why! why can't people leave things alone.

Excuse me while I hit the bourbon 🥃

Most intense day for me was at the very start of my career. Internship... went with product manager to client's office while PM installed new test version of our product for on-site integration testing. Shortly after deploy, client manager came over to ask why production had gone down...

Turns out that manually typing DB name as part of deployment script is not, erm, risk free. PM entered production DB name and took out a very busy call centre for a few hours. Agents in tears, customers raging on phones, etc! After we restored and got everything back up and running, he reached me the keyboard and said "You're doing it this time."

My attempt was problem free, thankfully. Earned many brownie points that day.

Managers don't understand that there will ALWAYS be bugs shipped to production, no matter how hard you try to prevent or test against them.

Devs: lol

inb4 any comments really, i've seen facebook, instagram, and all the 'big players' crash and have bugs multiple times before, so don't go around swinging your dick like your company's software has no known bugs (don't even get me started on the devrant mobile app) I'm just saying bugs are a fact of software

Ever have a bug that *only* occurs in your production environment? How do you test potential fixes? 😜

Dev sent out a code review request.
I take about an hour, ask questions, make suggestions, general feedback, etc.

Today I noticed none of my questions were answered, developer closed the review, and the code merged into the production branch.

So I email him, asking him why the review was closed and why none of my concerns were addressed before merging to production.

Dev: "No one responded or left feedback, so I thought it was OK to merge up."
Me: "I reviewed and left feedback within the hour you sent the request."
Dev: "Oh yea...you did. Sorry. The code is already in production, but if you still want to leave feedback, create a work item, and I'll take a look."

No you won't.

An example of the code...The dev added an async method to a test harness *console app*. Why? .. check in comment was "Improves performance and enhances the developer experience.."
NO IT DOESN'T!

OK..that's off my chest. No one is getting punched in the face today.

First day back from holiday: after 30 minutes of work (excludes start-up, catch-up etc.) The P.O. (product owner) comes to me

Telling me I needed to switch project, ok I thought at least they switched the project from what ever it was to a propper OTAP street while I was away

Few tickets later the P.O. asked me if tickets x could be deployed to our test servers as well as production.. (note the ticket was already merged with our develop branch and he wanted only that single ticket x to be deployed)

WTF is the point of a OTAP street if you're going to deploy it to every server type at once?

So day one after my holiday I already needed to fight the P.O. again

At least I wasn't disturbed during my vacation... Witch is a first.

Just needing somewhere to let some steam off

Tl;dr: perfectly fine commandline system is replaced by bad ui system because it has a ui.

For a while now we have had a development k8s cluster for the dev team. Using helm as composing framework everything worked perfectly via the console. Being able to quickly test new code to existing apps, and even deploy new (and even third party apps) on a simar-to-production system was a breeze.

Introducing Rancher

We are now required to commit every helm configuration change to a git repository and merge to master (master is used on dev and prod) before even being able to test the the configuration change, as the package is not created until after the merge is completed.

Rolling out new tags now also requires a VCS change as you have to point to the docker image version within a file.

As we now have this awesome new system, the ops didn't see a reason to give us access to kubectl. So the dev team is stuck with a ui, but this should give the dev team more flexibility and independence, and more people from the team can roll releases.

Back to reality: since the new system we have hogged more time from ops than we have done in a while, everyone needs to learn a new unintuitive tool, and the funny thing, only a few people can actually accept VCS changes as it impacts dev and prod. So the entire reason this was done, so it is reachable to more people, is out the window.

"Real devs test in production", in practice.

This was actually the second such notification I received. Not sure if this is standard for mobile app testing...

I was experimenting with a load test suite called 'Siege' to build and scale increasingly complex searches against our new site search engine. I assumed that an old iMac couldn't have generated a crushing load against the beta servers and I learned two things the day I wrote and started that script before heading to lunch:

1) Beta and Production shared MSSQL instances
2) That single iMac was more than enough to take the whole production site down...

the web developer equivalent of waiting for code to compile is waiting for your local test DB to be populated with recent data from the production DB

When you test in production

Why is web development such a headache?

I'm writing a responsive wesbite from scratch. All goes perfect, even cross browser.
It all works, adapts to screen size etc. Nice! About to get this code into production.

Me: I'll test the iPhone 5 viewport size before I push the code...

Responsive Developer Tools:

FireFox: nu uh, there's a magic random 1px margin to every element on your page now, which you cannot find in your css or in the computed tab. It's magical.

Me: weird, what if I change the viewport size to the iPhone 6's dimensions?

Issue persists.

Me: hmm, what if I add or substract one fucking pixel from the viewport width or height?

FireFox: What 1px margin? Don't know what you're talking about ... There never was one...

Me: ok, weird (sets viewport size back to the iPhone 5 format for testing)

FireFox: I present to you: the magic random 1px margin.

I'm at a loss. I really am. Been clicking and unclicking almost every responsive part of my css I could find for this page and it just doesn't want to work persistently. And I swear to god that it worked a week ago in that exact viewport size. It's so frustrating.

Overengineering. Finding the right point between overdesign and no design at all. That's where fancy languages and unusual patterns being hit by real world problems, and you need to deal with all that utter mess you created being architecture astronaut. Isn't that funny how you realize that another fancy tool is fundamentally incompatible with the task you need to solve, and you realize it after a month of writing workarounds and hacks.

But on the other hand, duct tape slacking becomes a mess even quicker.

Not being able to promote projects. You may code the shit out of side project and still get zero response, absolutely no impact. That's why your side projects often becomes abandoned.

Oversleeping. You thought tomorrow was productive day, but you wake up oversleeped, your head aches, your mind is not clear and you be like "fuck that, I'm staying in bed watching memes all day". But there's job that has to be done, and that bothers you.

Writing tests. Oh, words can't describe how much I hate writing tests, any kind of. I tried testing so many times in high school, at university, even at production, but it seems like my mind is just doesn't accept it. I know that testing is fundamentally important, but my mind collapses every time I try to write a single fucking test, resulting in terrible headache. I don't know why it's like that, but it is, and I better repl the shit out of pure function than write fucking tests.

Since I have seen a lot of people uploading this kind of stuff lately, here is Xiaomi's test in production, back in 2017 November...

I'm genuinely shocked at the number of people I see on here bashing automated testing as a waste of time, simply because my entire career has taught me the opposite (and it's usually only non-technical managers I see who don't want to see "time wasted" writing tests.)

I'm also just as genuinely curious - what do you guys do instead? Just don't test and deal with production issues as they occur? Pass it off to a separate UAT / human-based testing department and let them sign it off? Assume that because you're using Haskell / some other discipline it'll work if it compiles?

Ok I'm officially losing my fucking mind!

I've been trying to solve a connection bug that only occurs in production which is cool if THIS FUCKING APP DIDN'T TAKE 30 MINUTES TO DEPLOY!!!

Been busy for 4 hours and I've only been able to test 4 minor fixes.

I was Just college fresher who completed his Engineering. My first week in the office. And a system was provided to me, since it was support project so I was given direct access to production database.

Fresher + Production Database + Access of Admin credentials = Worst Possible Combination

So it was my night shift, I was told to update new tariff plan for our client (which was one of the largest telecom service in India) .

If someone recharges for more than 200 Rupee, that person will get 10% or 20% extra talk time. Which was only applicable for particular circle (Like Bihar and Rajasthan).

Since I was fresher, I was told to update given query from my senior employee which he shared on the shared folder. Production downtime was in the mid night, so at that time I updated that query on the production database.

Query successfully updated. I completed my night shift, went home and slept.

When I woke up, I saw my mobile it had 200+ missed calls from different locations of India. They were Circle heads of that telecom service provider who contacted me. I realized something unexpected is expecting me.

Then at that moment my team lead called me and he asked me to come office right away.

Reminding you I was a fresher, I was shivering. What have I done there?

When I reached office, I came to know that the query I updated on production bombarded.

Every person who recharged that day (duration from midnight to morning 10 AM) got 10 times or 20 times more talktime.

A part of Query was something like this where error was made:

TalkTime = RechargeAmount + RechargeAmount * 10/100; (Bihar)
or

TalkTime = RechargeAmount + RechargeAmount * 20/100; (Rajasthan)
But instead of this query, I updated below one:

TalkTime = RechargeAmount + RechargeAmount * 10;
or

TalkTime = RechargeAmount + RechargeAmount * 20;
In a span of 10 hours, that telecom service lost revenue of 6.5 crore Rupees. Thanks to recovery team they were able to recover 6 crore but still 50 lakh Rupees were in loss.

One small query, and approx 1 million dollar was on stake.

Aftermath of this incident

My Mistake:

I should have taken those queries on mail. Or, there should have been mail communication regarding this.
Never ever do anything over oral communication. Senior employee who did this denied and said he provided correct query, and I had no proof of communication.
I told them, it was me who executed that query on production. Since I was fresher, and took my responsibility of that incident. My team lead rescued me from that situation.
Lesson Learned:

Always test your query and code multiple times before you execute or Go live it on production.
Always have email communication for every action you take on production.
Power comes with responsibility. If you have admin credentials of production never use it for update/delete/drop until you are sure.
Don’t take your job lightly.
I was not fired from that Job, but I have learnt my lesson very well.

Normally you would have:
- Management
- SysOps
- DevOps
- Devs

In the company (30-50 workers) we only have:
- Management
- SysOps (they don't know how to deploy apps beyond FTP to a webhost either)
- Devs

Jepp, management does not want a specific DevOps department, because he thinks every single "I just finished the Javascript course on codecademy" person knows how to deploy an app beyond dragging&dropping it to a webhost with FTP...

I tried to propose to them that I handle DevOps and teach it to others, so we can deploy code that we deem "production ready" in a more proper manner...
They refused...
They rather stick to "just use FTP to push any changes we made directly to the production server and test changes there"

New twist on an old favorite.

Background:
- TeamA provides a service internal to the company.
- That service is made accessible to a cloud environment, also has a requirement to be made available to machines on the local network so you can develop against it.
- Company is too cheap/stupid to get a s2s vpn to their cloud provider.
- Company also only hosts production in the cloud, so all other dev is done locally, or on production non-similar infra, local dev is podman.
- They accomplish service connectivity by use of an inordinately complicated edge gateway/router/firewall/message translator/ouija board/julienne fry maker, also controlled by said service team.

Scenario:
Me: "Hey, we're cool with signing requests using an x509 cert. That said, doing so requires different code than connecting to an unsecured endpoint. Please make this service accessible to developer machines and lower environments on the internal network so we can, you know, develop."
TeamA: "The service should be accessible to [cloud ip range]"
Me: "Yes, that's a production range. We need to be able to test the signing code without testing in production"
TeamA: "Can you mock the data?"
Me: "The code we are testing is relating to auth, not business logic"
TeamA: "What are you trying to do?"
Me: "We are trying to test the code that uses the x509 you provide to connect to the service"
TeamA: "Can you deploy to the cloud"
Me: "Again, no, the cloud is only production per policy, all lower environments are in the local data center"
TeamA: "can you try connecting to the gateway?"
Me: "Yes, we have, it's not accessible, it only has public DNS, and only allows [cloud ip range]"
TeamA: "it work when we try it"
Me: "Can you please supply repro steps so we can adjust our process"
TeamA: "Yes, log into the gateway and try issuing the call from there"
Me: (╯°□°)╯︵ ┻━┻

tl;dr: Works on my server

probably every time I see my tests failing.

Each time I am writing tests I'm convincing myself "it's an investment", "spend 2 hours now to save 2 days later", "unit-tests are good".

And each time I'm chasing away ideas like "perhaps they are right, perhaps writing unit tests is a waste of time..", "this code is simple, it should ever break - why test it??", "In the 2 hours I'll spend writing those UT I could build another feature"

Yes, it is terribly annoying to write tests, especially after writing the production code (code-first approach). Why test code that you know works, right?

But after a few weeks, months or years, when the time comes to change your feature: enhance it, refactor it, build an integration with/from it, etc, I feel like a child who found a forgotten favourite candy in his pocket when I see my tests failing.

It means I did a very good job writing them
It means it was not a waste of time
it means these tests will now save me hours or days of trial-and-error change→compile→deploy→test cycles.

So yeah, whenever I see my tests fail, I feel warm and fussy inside :)

I used a test user account to make live payment transaction with test wallet money on production, accounting department just called... I think I'm in hot waters!

Adding a proof of concept directly to production and handing it over without a single test is God's blessing.

I hate dev politics...

PM: Hey there is a weird error happening when I upload this file on production, but it works on our test environments.

Me: After looking at this error, I don't find any issues with the code, but this variable is set when the application is first loaded, I bet it wasn't loaded correctly our last deployment and we just need to reload the application.

Senior Dev: We need to output all of the errors and figure out where this error is coming from. Dump out all the errors on everything in production!!

Me: That's dumb... the code works on test... it's not the code.. it's the application.

Senior dev: %$*^$>&÷^> $

Me: Hey I have an idea! If test works... I can go ahead and deploy last week's changes to prod and dump those errors you were talking about!!

Senior Dev: OK

Me: *runs Jenkins job the deploys the new code and restarts the application*

PM: YAY you fixed it!!

Senior Dev: Did you sump put those errors like I said.

Me: Nope didn't touch a thing... I just deployed my irrelevant changes to that error and reloaded the application.

A couple of years ago I was working on a fairly large system with a complex (by necessity) access control architecture.

As is usually the case with those projects, it's awkward for developers to repro bugs that have to do with a user's accesses in production when we are not allowed to replicate production data in test, let alone locally.

We had a bug where I ended up making myself a new row in the production database for a thing I could have access to without affecting real data to repro it safely. I identified the bug so I could repro it in dev/test and removed the row and ensured everything worked normally, whew scary.

Have you ever walked into the office one day, and everyone is hunched over in a semicircle around one person's workstation, before one turns around to look at you and says - after a pause - "... ltlian?.."

Turns out I had basically "poisoned the well" with my dummy entity in a way where production now threw 500 for everyone BUT me who had transitive access to this post-non-entity. Due to the scope of the system, it had taken about a day for this to gradually propagate in terms of caching and eventual consistencies; new entities coming in was expected, but not that they disappear.

Luckily I had a decent track record for this to be a one-off. I sometimes think about how I would explain testing in prod and making it faceplant before going home for the day, other than "I assumed it would be fine". I would fire me.

Is this what I think it is?

Take over responsibility you fucking morons!

We are the engineering team and we cannot know how you operate our product in every detail. And for god's sake don't blame us when shit happens in production when you don't test upcoming deployments by yourself!

* Developing a new "My pages" NBV offer/order solution for customer
_Thursday
Customer: Are we ready for testing?
Me: Almost, we need to receive the SSL cert and then do a full test run to see if your sales services get the orders correctly. At this point, all orders made via this flow are tagged so they will not be sent to the Sales services. We also still need to implement the tracking to see who has been exposed to what in My Pages.
Customer: Ok, great!

_Friday
Customer: My web team needs these customers to have fake offers on them, to validate the layout and content
Me: Ok, my colleague can fix this by Tuesday - he has all the other things with higher prio from you to complete first
Customer: Ok! Good!

_Sunday
Me: Good news, got the SSL cert installed and have verified the flow from my side. Now you need to verify the full flow from your side.
Customer: Ok! Great! Will do.

_Monday
*quiet*

_Tuesday
Customer: Can you see how things are going? Any good news?
Me: ???
*looks into the system*
WTF!?!
- Have you set this into production on your side? We are not finished with the implementation on our side!
Customer: Oh, sorry - well, it looked fine when we tested with the test links you sent (3 weeks ago)
Me: But did you make a complete test run, and make sure that Sales services got the order?
Customer: Oh, no they didn't receive anything - but we thought that was just because of it being a test link
Me: Seriously - you didn't read what i wrote last Thursday?
Customer: ...
Me: Ok, so what happens if something goes wrong - who get's blamed?
Customer: ...
Me: FML!!!

Why we always found the worst problems 2days after going to production? No matter how we tested?

Wow or wtf to these banks API. was integrating an API for a service which accept JSON input.
Okay fair enough, that would be fine
Spent an hour writing code(purescript) most of time spent was on writing Types based on the API doc. after that okay let me test the API it failed.
I was what happened? So tested the API from postman with the payload from the doc, it worked. What how?
used a JSON diff to compare the payload from postman and the log. Looked same to me after spending few hours checking what is wrong with it .trying changing value to pasting the body of the log request in postman and trying everything failed.
Later went to the original working payload provided by them and changing the order. It started throwing error. I was like wait what?
It must be only on there UAT. created a payload with production creds and hoping to our production server (they have IP whitelist) ran the curl with proper payload as expected it worked. Later for same payload changed the order or one key and tried it failed.
Just why????
I don't want to create a JSON with keys on specific order. Also it's not even sorted order.

Tl;Dr Im the one of the few in my area that sees sftping as the prod service account shouldn't be a deployment process. And the ONLY ONE THAT CARES THAT THIS IS GONNA BREAK A BUNCH OF SHIT AT SOME POINT.

The non tl;dr:

For a whole year I've been trying to convince my area that sshing as the production service account is not the proper way to deploy and/or develop batch code. My area (my team and 3 sister teams) have no concept of using version control for our various Unix components (shell scripts and configuration files) that our CRITICAL for our teams ongoing success. Most develop in a "prodqa like" system and the remainder straight in production. Those that develop straight in prodqa have no "test" deployment so when they ssh files straight to actual production. Our area has no concept of continuous integration and automated build checking. There is no "test cases", no "systems testing" or "regression testing". No gate checks for changing production are enforced. There is a standing "approved" deployment process by the enterprise (my company is Whyyyyyyyyyy bigger than my area ) but no one uses it. In fact idk anyone in my area who knows HOW to deploy using the official deployment method. Yes, there is privileged access management on the service account. Yes the managers gets notified everytime someone accesses the privileged production account. The managers don't see fixing this as a priority. In fact I think I've only talk to ONE other person in my area who truly understands how terrible it is that we have full production change access on a daily basis. Ive brought this up so many times and so many times nothing has been done and I've tried to get it changed yet nothing has happened and I'm just SO FUCKING SICK that no one sees how big of a deal this. I mean, overall I live the area I work in, I love the people, yet this one glaring deficiency causes me so much fucking stress cause it's so fucking simple to fix.
We even have an newer enterprise deployment. Method leveraging a product called "urban code deploy" (ucd) to deploy a git repository. JUST FUCKING GIT WITH THE PROGRAM!!!!..... IT WAS RELEASED FUCKING 12 YEARS AGO......

Please..... Please..... I just want my otherwise normally awesome team to understand the importance and benefits of version control and approved/revertable deployments

Optimization issue pops out with one of our queries.
> Team leader: You need to do this and that, it's a thing you know NOTHING about but don't worry, the DBA already performed all the preliminary analysis, it's tested and it should work. Just change these 2 lines of code and we're good to go
> ffwd 2 days, ticket gets sent back, it's not working
> Team leader: YOU WERE SUPPOSED TO TEST IT YOUR CHANGE IS NOT WORKING
> IHateForALiving: try it on our production machine and you'll see the exact same error, it's been there for years
> Team leader: BUT YOU WERE SUPPOSED TO TEST IT

Just so we're clear, when I perform a change in the code, I test the changes I made. I don't know in which universe I should be held accountable for tards breaking features 10 years ago, but you can't seriously expect me to test the whole fucking software from scratch every time I add an index to the db.

So glad to be staying a new job next week. Today a junior colleague asked me what the best way to test something would be as it won't work locally. Knowing this has a good chance of taking down the server, I suggest he sounds up a server on his AWS account. My manager comes in, oh no I don't want him doing it on AWS use the production server instead. By the time stuff States hitting the fan I will be gone.

Best parte do being a dev?

Fix bugs in production! Because life is to short to test in UAT

A newly joined developer (who was supposed to be very senior) comes and asks me how to write a test cos for some reason the person didn't know how to mock.

In Java,
(same for any other implementation which has an interface)
Writes Arraylist list =.....
Instead of List list = Arraylist...

Deployed code (another engineer from another country helped to deploy since this new senior dev didn't have access yet.

But the new senior dev didn't update relevant files in production code which brought down the site for nearly an hour. Mistake aside, the first reaction from this new senior dev is 'WHY DIDN'T THE DEV THAT WAS HELPING DIDN'T DO THE FILE UPDATE?'

This was followed by some other complaints such as our branching stragies are wrong. When in fact the new senior dev made a mistake by just making assumptions on our git branching strategies and we already advised on correct process.

Out of all these, guess this is the best part. The senior dev never tested code locally! Just wrote code, unit test and send to QA and somehow the test passed through. I learnt this when I realised this dev... has not even set up the local environment yet.

I keep saying new but this Senior dev been around like 3 months! This person is in another team within our larger team but shares same code base. I am puzzled how do you not set up your environment for 3 months. Don't you ask for help if you are stuck? I am pretty sure the env is still not setup.

Am I over reacting or is this one disgusting developer who doesn't even qualify for an intern let alone a senior dev? It's so revolting I can't even bring myself to offer help.

most memorable bug I fixed

Line 1:
- throw new Error(‘test’)
+ // throw new Error(‘test’)

yes, this was committed code in production

I really really hope that no one post this,a friend texted it to me and I wanted to share it because made my day.

Idk where it comes, so feel free if know where this came from to post it:

//FUN PART HERE
# Do not refactor, it is a bad practice. YOLO

# Not understanding why or how something works is always good. YOLO

# Do not ever test your code yourself, just ask. YOLO

# No one is going to read your code, at any point don’t comment. YOLO

# Why do it the easy way when you can reinvent the wheel? Future-proofing is for pussies. YOLO

# Do not read the documentation. YOLO

# Do not waste time with gists. YOLO

# Do not write specs. YOLO also matches to YDD (YOLO DRIVEN DEVELOPMENT)

# Do not use naming conventions. YOLO

# Paying for online tutorials is always better than just searching and reading. YOLO

# You always use production as an environment. YOLO

# Don’t describe what you’re trying to do, just ask random questions on how to do it. YOLO

# Don’t indent. YOLO

# Version control systems are for wussies. YOLO

# Developing on a system similar to the deployment system is for wussies! YOLO

# I don’t always test my code, but when I do, I do it in production. YOLO

# Real men deploy with ftp. YOLO

So YOLO Driven Development isn’t your style? Okay, here are a few more hilarious IT methodologies to get on board with.

*The Pigeon Methodology*
Boss flies in, shits all over everything, then flies away.

*ADD (Asshole Driven Development)*
An old favourite, which outlines any team where the biggest jerk makes all the big decisions. Wisdom, process and logic are not the factory default.

*NDAD (No Developers Allowed in Decisions)*
Methodology Developers of all kinds are strictly forbidden when it comes to decisions regarding entire projects, from back end design to deadlines, because middle and top management know exactly what they want, how it should be done, and how long it will take.

*FDD (Fear Driven Development)*
The analysis paralysis that can slow an entire project down, with developments afraid to make mistakes, break the build, or cause bugs. The source of a developer’s anxiety could be attributed to a failure in sharing information, or by implicating that team members are replaceable.

*CYAE (Cover Your Ass Engineering)*
As Scott Berkun so eloquently put it, the driving force behind most individual efforts is making sure that when the shit hits the fan, you are not to blame.

So, just pulled another all-nighter..

On our platform I switched a quite big customer to another stock keeping system to pull them into automatic FEFO handling etc. Just a better stock keeping system overall.. I made it.. *self hi-5*

Evidently the crons caught that change, and CLEARED ALL THE STOCK LEVELS as they're now managed by said system...

Had to pull the counts, locations, expiry dates and lot numbers from the history table and old database fields, add them to an Excel sheet and then add all gathered locations by hand back into the new system, whilst also setting the new settings for them.

39 unique products that were gathered over 190+ sku objects... (Somebody didn't get object oriented, or was trying to KISS themselves, clearly...)

That's 6 hours of extra work for a stupid fuck up.. Oops? (:

We basically don't unit test at work. I write some tests for my code and honest to God people complain I'm wasting time saying a test bed and manual tests are good enough. We don't write test beds for about half of our production code and rely on integration tests for the rest. We only test release builds which have been symbol stripped, I get handed a crash report with no stack trace that I'm unable to reproduce and expected to stay late to fix it for some arbitrary internal deadline.

I've since moved to R&D where basically I'm left to do my own thing so it's better.

We don't project manage. Project leads take time estimates and double them so management might cut them some slack. This doesn't matter because management made up time estimates before the project started. Last project I was on had a timeline of 3 months and took a year.

We have released broken products. Not that any of the above really matters, our software products have made about 50k revenue in 2 years. There are 6 people on software. Fortunately hardware has made about 3 mill. That said our hardware customers are getting frustrated with us as we keep fucking up, shipping broken products and missing deadlines.

I've been working there about a year and a half and will be looking for a job at the end of the current project.

I joined devRant about when I was most pissed off with my job, my rant frequency has definitely gone down since I moved over to R&D.

So, we (I'm the backend guy and work with a UI dev) are building this product portfolio management tool for our client and they have a set of 250 users. The team has two point of contacts for the 250 users who maintain the master data, help users with data quality, tool guidance, reporting and other stuff. So one day one of these two support users come to me and say : Hey I'm not able to add new transactions coz a customer is missing.

We have the provision to create / maintain customers.

I check the production DB, application code, try creating the customer and then the transaction, everything works perfectly fine.

I ask the user for a screen sharing session, the user starts reproducing the error like this :

We have a 3 system landscape - Dev / Test and Prod

U : Logs into the test system url, creates the customer.

U : Points out the toast saying customer creation is successful.

U : opens a new tab, opens the production system, tries creating the transaction, searches for the customer and says " see !! cant find the customer here ! the master data management apps never work !! "

FML?.

When you open a previous employees production JUnit tests and see the below:

@Test
public void test1(){
...
}

Works in production?

Yes: Copy changes to dev and test.
No: Start humming and walk away from the computer.

If I had a nickel every time the unit tests failed not because something was wrong in the code, but because someone had messed up the unit test I'd be able to retire early.

I just spent the better part of 10 hours hunting down a bug in some production code only for the test to be wrong because the person who wrote it had mocked the http response incorrectly.

Nothing I did to "fix" the code worked, because nothing was wrong with it...

I'm still studying computer science/programming, I still have one year to do in order to graduate (Master). I am in a work study program so I'm working for a company half of the time, and I'm studying the other half. It is important to mention that I am the only web developer of the company

When I arrived in the company 9 months ago, I was given a Vue project which had been developed by a trainee a few weeks before my arrival and I was asked to correct a few things, it was mostly about css. Then, I was ask to add a few functionalities, nothing really hard to code, and we were supposed to test the solution in a staging environment, and if everything was ok, deploy it to prod.

However, the more I did what I was asked, the more functionalities I had to implement, until I reached a point where I had to modify the API, create new routes, etc. I'm not complaining about that, that's my job and I like it. But the solution was supposed to be ready when I arrived, it was also supposed to be tested and deployed.

The problem is, the person emitting these demands (let's call him guy X) is not from the IT service, it's a future user of the website in the admin side. The demands kept going and going and going because, according to him, the solution was not in a good enough state to be deployed, it missed too many (un)necessary features. It kept going for a few months.

The best is yet to come though : guy X was obviously a superior, and HIS superior started putting pressure on me through mails, saying the app was already supposed to be in production and he was implying that I wasn't working fast enough. Luckily, my IT supervisor was aware of what was going on and knew I obviously wasn't to blame.

In the end, the solution was eagerly deployed in production, didn't go through the staging environment and was opened to the users. Now, guy X receives complaints because none of what I did was tested (it was by me, but I wasn't going to test every single little thing because I didn't have time). Some users couldn't connect or use this or that feature and I am literally drowning in mails, all from guy X, asking me to correct things because users are blocked and it's time consuming for him to do some of the things the website was doing manually.

We are here now just because things have been done in a rush, I'm still working on it and trying to fix prod problems and it's pissing me off because we HAVE a staging environment that was supposed to prevent me from working against the clock.

On a final note, what's funny is that the code I'm modifying, the pre-existing one needs to be refactored because bits and pieces are repeated sometimes 5 times where it should have been externalized and imported from another file. But I don't know when and if I will ever be able to do that.

I could have given more context but it's 4am and I'm kinda tired, sorry if I'm not clear or anything. That's my first rant

Writing a feature critical for production in 2 hours of solid focus during the morning.

6 hours later it's still not in the build because:
* tech lead wants the code to move to a partial class instead of an extension method, delaying the UX review. No guidelines for this ever existed.

* after seeing the result, the UX team wants some element to be dynamic. A line. A friggin horizontal line.

* after adding the dynamic shiny frigggggin line, I try to test the feature with the server. It is still not deployed because the server guy went home. "The PR was not merged so I assumed we'll add it tomorrow".

Another day at the meat grinder.