Motivational quote!!!!

I worked with a good dev at one of my previous jobs, but one of his faults was that he was a bit scattered and would sometimes forget things.

The story goes that one day we had this massive bug on our web app and we had a large portion of our dev team trying to figure it out. We thought we narrowed down the issue to a very specific part of the code, but something weird happened. No matter how often we looked at the piece of code where we all knew the problem had to be, no one could see any problem with it. And there want anything close to explaining how we could be seeing the issue we were in production.

We spent hours going through this. It was driving everyone crazy. All of a sudden, my co-worker (one referenced above) gasps “oh shit.” And we’re all like, what’s up? He proceeds to tell us that he thinks he might have been testing a line of code on one of our prod servers and left it in there by accident and never committed it into the actual codebase. Just to explain this - we had a great deploy process at this company but every so often a dev would need to test something quickly on a prod machine so we’d allow it as long as they did it and removed it quickly. It was meant for being for a select few tasks that required a prod server and was just going to be a single line to test something. Bad practice, but was fine because everyone had been extremely careful with it.

Until this guy came along. After he said he thought he might have left a line change in the code on a prod server, we had to manually go in to 12 web servers and check. Eventually, we found the one that had the change and finally, the issue at hand made sense. We never thought for a second that the committed code in the git repo that we were looking at would be inaccurate.

Needless to say, he was never allowed to touch code on a prod server ever again.

Someone is testing on production...

"Don't worry about testing too much, push it to production asap"

Got this notification. Someone is testing on production environment.

Oh, man, I just realized I haven't ranted one of my best stories on here!

So, here goes!

A few years back the company I work for was contacted by an older client regarding a new project.

The guy was now pitching to build the website for the Parliament of another country (not gonna name it, NDAs and stuff), and was planning on outsourcing the development, as he had no team and he was only aiming on taking care of the client service/project management side of the project.

Out of principle (and also to preserve our mental integrity), we have purposely avoided working with government bodies of any kind, in any country, but he was a friend of our CEO and pleaded until we singed on board.

Now, the project itself was way bigger than we expected, as the wanted more of an internal CRM, centralized document archive, event management, internal planning, multiple interfaced, role based access restricted monster of an administration interface, complete with regular user website, also packed with all kind of features, dashboards and so on.

Long story short, a lot bigger than what we were expecting based on the initial brief.

The development period was hell. New features were coming in on a weekly basis. Already implemented functionality was constantly being changed or redefined. No requests we ever made about clarifications and/or materials or information were ever answered on time.

They also somehow bullied the guy that brought us the project into also including the data migration from the old website into the new one we were building and we somehow ended up having to extract meaningful, formatted, sanitized content parsing static HTML files and connecting them to download-able files (almost every page in the old website had files available to download) we needed to also include in a sane way.

Now, don't think the files were simple URL paths we can trace to a folder/file path, oh no!!! The links were some form of hash combination that had to be exploded and tested against some king of database relationship tables that only had hashed indexes relating to other tables, that also only had hashed indexes relating to some other tables that kept a database of the website pages HTML file naming. So what we had to do is identify the files based on a combination of hashed indexes and re-hashed HTML file names that in the end would give us a filename for a real file that we had to then search for inside a list of over 20 folders not related to one another.

So we did this. Created a script that processed the hell out of over 10000 HTML files, database entries and files and re-indexed and re-named all this shit into a meaningful database of sane data and well organized files.

So, with this we were nearing the finish line for the project, which by now exceeded the estimated time by over to times.

We test everything, retest it all again for good measure, pack everything up for deployment, simulate on a staging environment, give the final client access to the staging version, get them to accept that all requirements are met, finish writing the documentation for the codebase, write detailed deployment procedure, include some automation and testing tools also for good measure, recommend production setup, hardware specs, software versions, server side optimization like caching, load balancing and all that we could think would ever be useful, all with more documentation and instructions.

As the project was built on PHP/MySQL (as requested), we recommended a Linux environment for production. Oh, I forgot to tell you that over the development period they kept asking us to also include steps for Windows procedures along with our regular documentation. Was a bit strange, but we added it in there just so we can finish and close the damn project.

So, we send them all the above and go get drunk as fuck in celebration of getting rid of them once and for all...

Next day: hung over, I get to the office, open my laptop and see on new email. I only had the one new mail, so I open it to see what it's about.

Lo and behold! The fuckers over in the other country that called themselves "IT guys", and were the ones making all the changes and additions to our requirements, were not capable enough to follow step by step instructions in order to deploy the project on their servers!!!

[Continues in the comments]

Testing in production doesn't hurt.

IT
FUCKING
KILLS

Just ask Boeing.

Friday afternoon going straight to production without testing.

Gotta love testing in production. Nice work Niantic 🤣

F🤬k!! why doesn’t this work on the testing server!l! I swear it worked local.

Meanwhile live:

Another day at work...

A more experienced friend told me
"don't be a pussy, test in production"

I'm the one fixing the bugs, not him

Fixing bugs in production like...

How many can relate?

Alpha testing be like...

User: *Clicks on staging environment*

Giant Warning Dialog: YOU ARE CURRENTLY ENTERING THE STAGING ENVIRONMENT

Users: Ok

App: *Completely different colour, I’m talking bright unsightly yellow*

User: Ok

Giant Yellow and Red Flashing Banner at the Top of the Screen: WARNING YOU ARE CURRENTLY USING STAGING, THIS AREA IS FOR TESTING ONLY

User: The production environment sure is acting strange today. It’s a weird colour and I don’t recognize any of the data, it’s all just dummy filler data. I better create a ticket for the dev team to check o—….. no wait I’ll send an email CC everyone including the CEO and sound the alarm production is currently down and filled with giant warning messages.

Manager: OH MY GOD PRODUCTION IS DOWN DID YOU HEAR ABOUT THIS??? WHAT THE FUCK COULD THESE WARNING MESSAGES BE THAT’S ONLY SUPPOSED TO HAPPEN ON STAGING! THE CEO IS BREATHING DOWN MY NECK YOU NEED TO GET THIS FIXED IMMEDIATELY!!!!!!!

Dev: …

When you catch Facebook Devs testing in prod (beta app though)

Just in bank testing app in production

Doot doot.

My day: Eight lines of refactoring around a 10-character fix for a minor production issue. Some tests. Lots of bloody phone calls and conference calls filled with me laughing and getting talked over. Why? Read on.

My boss's day: Trying very very hard to pin random shit on me (and failing because I'm awesome and fuck him). Six hours of drama and freaking out and chewing and yelling that the whole system is broken because of that minor issue. No reading, lots of misunderstanding, lots of panic. Three-way called me specifically to bitch out another coworker in front of me. (Coworker wasn't really in the wrong.) Called a contractor to his house for testing. Finally learned that everything works perfectly in QA (duh, I fixed it hours ago). Desperately waited for me to push to prod. Didn't care enough to do production tests afterwards.

My day afterwards: hey, this Cloudinary transform feature sounds fun! Oh look, I'm done already. Boo. Ask boss for update. Tests still aren't finished. Okay, whatever. Time for bed.

what a joke.

Oh, I talked to the accountant after all of this bullshit happened. Apparently everyone that has quit in the last six years has done so specifically because of the boss. Every. single. person.

I told him it was going to happen again.

I also told him the boss is a druggie with a taste for psychedelics. (It came up in conversation. Absolutely true, too.) It's hilarious because the company lawyer is the accountant's brother.

So stupid.

I just sent an automated email titled "Gary is a Dinosaur!" to a lot of humourless clients because the ancient application I was testing assumed I was in the production environment. 🙃

Lesson to self: stop using bogus names in testing.

Still, it could've been a lot worse... 😂

Hey, Root? How do you test your slow query ticket, again? I didn't bother reading the giant green "Testing notes:" box on the ticket. Yeah, could you explain it while I don't bother to listen and talk over you? Thanks.

And later:
Hey Root. I'm the DBA. Could you explain exactly what you're doing in this ticket, because i can't understand it. What are these new columns? Where is the new query? What are you doing? And why? Oh, the ticket? Yeah, I didn't bother to read it. There was too much text filled with things like implementation details, query optimization findings, overall benchmarking results, the purpose of the new columns, and i just couldn't care enough to read any of that. Yeah, I also don't know how to find the query it's running now. Yep, have complete access to the console and DB and query log. Still can't figure it out.

And later:

Hey Root. We pulled your urgent fix ticket from the release. You know, the one that SysOps and Data and even execs have been demanding? The one you finished three months ago? Yep, the problem is still taking down production every week or so, but we just can't verify that your fix is good enough. Even though the changes are pretty minimal, you've said it's 8x faster, and provided benchmark findings, we just ... don't know how to get the query it's running out of the code. or how check the query logs to find it. So. we just don't know if it's good enough.

Also, we goofed up when deploying and the testing database is gone, so now we can't test it since there are no records. Nevermind that you provided snippets to remedy exactly scenario in the ticket description you wrote three months ago.

And later:

Hey Root: Why did you take so long on this ticket? It has sat for so long now that someone else filed a ticket for it, with investigation findings. You know it's bringing down production, and it's kind of urgent. Maybe you should have prioritized it more, or written up better notes. You really need to communicate better. This is why we can't trust you to get things out.

*twitchy smile*

Testing in production??

Don't know whether to laugh or cry!!😂😥

!rant
!!git

Who here uses `master` for development?

My boss (api guy) tried to convince me that was normal practice. I gently told him that it sounded crazy and very very bad.

Here's the dev path I'm enforcing on my repos:

(feature branches) -> dev -> qa* -> master -> production*

*: the build server auto-pulls from these branches, and pushes any passing builds to staging/production.

Everyone works on their own feature branches, and when they're happy with their work, they merge it into `dev`. `dev`, therefore, is for feature integration testing. After everything is working well on `dev`, it gets merged into `qa` for the testers to fawn over and beat with sticks. Anything that passes QA gets merged into `master`, where it sits until we're ready to release it. When that time comes (it's usually right away, but not always), `master` gets merged into `production`.

This way, `master` is always stable and contains the newest code, so it's perfect for forking/etc. Is this standard practice, or should I be doing something different?

Also, api guy encourages something he calls "running a racetrack" -- each dev has their own branch (their initials) and they push to that throughout the day. everyone else pulls from it regularly and pushes to their own branch. When anyone's happy with their code, they push from their (updated) branch to `qa` (I insisted on `dev` instead.)

Supposedly this drastically reduces the number of merge conflicts when pushing to an upstream branch due to having a more recent ancestor node?

I don't quite follow that, but it seems to me that merging/pushing throughout the day would just make them happen sooner? idk.

What are your thoughts?

Why are job postings so bad?
Like, really. Why?

Here's four I found today, plus an interview with a trainwreck from last week.
(And these aren't even the worst I've found lately!)

------

Ridiculous job posting #1:
* 5 years React and React Native experience -- the initial release of React Native was in May 2013, apparently. ~5.7 years ago.
* Masters degree in computer science.
* Write clean, maintainable code with tests.
* Be social and outgoing.

So: you must have either worked at Facebook or adopted and committed to both React and React Native basically immediately after release. You must also be in academia (with a masters!), and write clean and maintainable code, which... basically doesn't happen in academia. And on top of (and really: despite) all of this, you must also be a social butterfly! Good luck ~

------

Ridiculous job posting #2:
* "We use Ruby on Rails"
* A few sentences later... "we love functional programming and write only functional code!"

Cue Inigo Montoya.

------

Ridiculous job posting #3:
* 100% remote! Work from anywhere, any time zone!
* and following that: You must have at least 4 work hours overlap with your coworkers per day.
* two company-wide meetups per quarter! In fancy places like Peru and Tibet! ... TWO PER QUARTER!?

Let me paraphrase: "We like the entire team being remote, together."

------

Ridiculous job posting #4:
* Actual title: "Developer (noun): Superhero poised to change the world (apply within)"
* Actual excerpt: "We know that headhunters are already beating down your door. All we want is the opportunity to earn our right to keep you every single day."
* Actual excerpt: "But alas. A dark and evil power is upon us. And this… ...is where you enter the story. You will be the Superman who is called upon to hammer the villains back into the abyss from whence they came."

I already applied to this company some time before (...surprisingly...) and found that the founder/boss is both an ex cowboy dev and... more than a bit of a loon. If that last part isn't obvious already? Sheesh. He should go write bad fantasy metal lyrics instead.

------

Ridiculous interview:
* Service offered for free to customers
* PHP fanboy angrily asking only PHP questions despite the stack (Node+Vue) not even freaking including PHP! To be fair, he didn't know anything but PHP... so why (and how) is he working there?
* Actual admission: No testing suite, CI, or QA in place
* Actual admission: Testing sometimes happens in production due to tight deadlines
* Actual admission: Company serves ads and sells personally-identifiable customer information (with affiliate royalties!) to cover expenses
* Actual admission: Not looking for other monetization strategies; simply trying to scale their current break-even approach.

------

I find more of these every time I look. It's insane.
Why can't people be sane and at least semi-intelligent?

Worst dev team failure I've experienced?

One of several.

Around 2012, a team of devs were tasked to convert a ASPX service to WCF that had one responsibility, returning product data (description, price, availability, etc...simple stuff)
No complex searching, just pass the ID, you get the response.

I was the original developer of the ASPX service, which API was an XML request and returned an XML response. The 'powers-that-be' decided anything XML was evil and had to be purged from the planet. If this thought bubble popped up over your head "Wait a sec...doesn't WCF transmit everything via SOAP, which is XML?", yes, but in their minds SOAP wasn't XML. That's not the worst WTF of this story.

The team, 3 developers, 2 DBAs, network administrators, several web developers, worked on the conversion for about 9 months using the Waterfall method (3~5 months was mostly in meetings and very basic prototyping) and using a test-first approach (their own flavor of TDD). The 'go live' day was to occur at 3:00AM and mandatory that nearly the entire department be on-sight (including the department VP) and available to help troubleshoot any system issues.

3:00AM - Teams start their deployments
3:05AM - Thousands and thousands of errors from all kinds of sources (web exceptions, database exceptions, server exceptions, etc), site goes down, teams roll everything back.

3:30AM - The primary developer remembered he made a last minute change to a stored procedure parameter that hadn't been pushed to production, which caused a side-affect across several layers of their stack.

4:00AM - The developer found his bug, but the manager decided it would be better if everyone went home and get a fresh look at the problem at 8:00AM (yes, he expected everyone to be back in the office at 8:00AM).

About a month later, the team scheduled another 3:00AM deployment (VP was present again), confident that introducing mocking into their testing pipeline would fix any database related errors.

3:00AM - Team starts their deployments.
3:30AM - No major errors, things seem to be going well. High fives, cheers..manager tells everyone to head home.
3:35AM - Site crashes, like white page, no response from the servers kind of crash. Resetting IIS on the servers works, but only for around 10 minutes or so.
4:00AM - Team rolls back, manager is clearly pissed at this point, "Nobody is going fucking home until we figure this out!!"
6:00AM - Diagnostics found the WCF client was causing the server to run out of resources, with a mix of clogging up server bandwidth, and a sprinkle of N+1 scaling problem. Manager lets everyone go home, but be back in the office at 8:00AM to develop a plan so this *never* happens again.

About 2 months later, a 'real' development+integration environment (previously, any+all integration tests were on the developer's machine) and the team scheduled a 6:00AM deployment, but at a much, much smaller scale with just the 3 development team members.

Why? Because the manager 'froze' changes to the ASPX service, the web team still needed various enhancements, so they bypassed the service (not using the ASPX service at all) and wrote their own SQL scripts that hit the database directly and utilized AppFabric/Velocity caching to allow the site to scale. There were only a couple client application using the ASPX service that needed to be converted, so deploying at 6:00AM gave everyone a couple of hours before users got into the office. Service deployed, worked like a champ.

A week later the VP schedules a celebration for the successful migration to WCF. Pizza, cake, the works. The 3 team members received awards (and a envelope, which probably equaled some $$$) and the entire team received a custom Benchmade pocket knife to remember this project's success. Myself and several others just stared at each other, not knowing what to say.

Later, my manager pulls several of us into a conference room
Me: "What the hell? This is one of the biggest failures I've been apart of. We got rewarded for thousands and thousands of dollars of wasted time."
<others expressed the same and expletive sediments>
Mgr: "I know..I know...but that's the story we have to stick with. If the company realizes what a fucking mess this is, we could all be fired."
Me: "What?!! All of us?!"
Mgr: "Well, shit rolls downhill. Dept-Mgr-John is ready to fire anyone he felt could make him look bad, which is why I pulled you guys in here. The other sheep out there will go along with anything he says and more than happy to throw you under the bus. Keep your head down until this blows over. Say nothing."

Everyone has these projects sooner or later;

PM: 'You have 2 weeks to finish. It HAS to be done by then because they will start using it ASAP!!'
Me: Working day and night to meet the deadline.

After 2 years PM comes by;

PM: remember that program? They're testing it now. They plan to use it in production.

Rghuu*~ (that's me dying a little on the inside)

Worst bad practice..

Manager: I need code today
Developer (thinking) : let me give it without unit test. Anyways tester will test it.

Manager to tester: complete testing fast.

Tester(thinking): developer must have unit tested it. Let me skip it.
Enjoy testing completed.

God help clients.. 😊

Production is Testing
Software Freedom is Proprietary
Email is Version Control

Stop fucking deploying to production and testing there !

Team lead my ass 😠

Production environment. Where else would I do all my testing?

My first testing job in the industry. Quite the rollercoaster.

I had found this neat little online service with a community. I signed up an account and participated. I sent in a lot of bug reports. One of the community supervisors sent me a message that most things in FogBugz had my username all over it.

After a year, I got cocky and decided to try SQL injection. In a production environment. What can I say. I was young, not bright, and overly curious. Never malicious, never damaged data or exposed sensitive data or bork services.

I reported it.

Not long after, I got phone calls. I was pretty sure I was getting charged with something.

I was offered a job.

Three months into the job, they asked if I wanted to do Python and work with the automators. I said I don't know what that is but sure.

They hired me a private instructor for a week to learn the basics, then flew me to the other side of the world for two weeks to work directly with the automation team to learn how they do it.

It was a pretty exciting era in my life and my dream job.

Testing in production

"We use production DB in testing, so be careful with it"

Yes. The way to go. Testing in production for a banking site .

Testing in prod.
On Friday evening.
Before long holiday.

Because why not.

Sex talk between programmers.

She: I'm a virgin.
He: Don't worry. They call me the virginslayer007.
She: Oh! So how many virgins have you slayed till now?
He: That would be ONE in a few minutes.
She: So u r also a virgin then..
He: Don't worry. I watched so many video tutorials. We just have to do exactly as they did. Best thing is that it can be done both for testing and production purposes.
She: Let's stick to testing purposes for now.

Every so often I remember that the code I wrote is running in production and real customers are using it and I feel a little bit sick

Client: "I did not receive the email that should be send after that event. Please fix."
Me:
* Checks code - ok
* Tests feature in locally - ok
* Tests feature in production - ok
* checks values in database - ok
* 2 hours wasted - ok
"Please help me dear CTO, idk what else I could check or how I should even respond to this."
CTO: "hmm, the clients account uses a adminstrative email address for testing. Let me just check if it is in the mailbox."
*checks* "Yeah, that's the email you're looking for, right?"
Me: *experiences relief, anger, blood lust and disappointment at the same time* "Could you please respond to the client for me, I need a break. Thanks"

In my current work, I have two systems to work on (let's name em Systems A and B). Both basically do the same thing; both allow users to book facilities available to them.

System A is already in production. My job is to fix any bugs that come up on said system. System B is an improved version that they wanted me to develop. This would follow a different framework etc. I am already halfway through this system.

Now, here's the fucked up part. The code for system A is a massive clusterfuck. It has unused commented code dated back to ancient times where men had the brain of an ape.

And don't get me started on the fucking logic. One part of the code was to retrieve and display the timeslots available for a chosen facility. The code to do that alone takes up 500++ fucking lines, filled with ajax commands, html manipulation and commented, unused codes..AND THAT'S JUST THE FRONTEND!

The fucking backend was not a problem of smelly code anymore. Nope. It was like a programmer had code diarrhea and shat his backend code all over the project. If I had a pin board, I would have made a crazy wall just to understand what some fucknut was trying to achieve.

Anyway, my supervisor told me to fix some bugs on System A. Knowing how the code was, I told her that I could refactor the code. Since I've already achieved that function on System B, with a shorter and cleaner code, I could just copy that and use on System A. But nope. She SPECIFICALLY told me to just "do whatever to fix the bugs. I don't want to waste time on System A." Okay. Makes sense to me. Whatever. I didn't wanna fuck my head up looking through that mess of a cesspool. So, I came up with a few hacks, not thinking of clean code and fixed whatever bugs there was. I then just pushed to the repo (after testing of course).

This bloody morning, supervisor came in and gave me more bugs to fix. When I thought she was done, she said "Hey. I saw the fix you made to the system. The bugs are fixed but the retrieval of the timeslots is now pretty slow. Could you see what is the problem?"

Slow.. She said that it was slow. And asked if I could fix it. I already told her what the problem was and she did not want me to waste time on it. But she wants me to fix it. WHAT THE FUCK IS WRONG IN HER BLOODY HEAD! I SWEAR TO GOD... UGHHHHH I swear I was already waterboarding her in my head. YOU WANT FAST?? How bout fucking allowing me to refactor the code?? Fucking shit head. I think I should take up yoga.

Some years back I was working in a project that essentially dealt with all things related to foreigners and foreign affairs in Switzerland. You could manage entry visas, work permits, citizenship, international warrants, Interpol requests, etc.

One of the test managers (from client side - i.e. the government) was once manually "testing" and mixed up the production and test instance, to both of which he was logged in at the time.
The test case then ended up setting up an entry ban against himself, as he used his own name for testing...

Next time he returned from vacation the border control at the airport were like "Uhm, Sir, we can't let you into the country. Please come with us." :D :D

(He managed to clear that up in end, I dare say, though, that he learned his lesson.)

When even your spammers are testing in production...

FUCKING TELEGRAM FUCK YOU STAY IN YOUR FUCKING API DOCUMENTATION AND STOP FUCKING TESTING YOUR SHIT ON A PRODUCTION SYSTEM WHY WOULD YOU DO THAT FUCK OFF WHY AM I EVEN DEVELOPING SHIT FOR YOUR PLATFORM ANYMORE WHEN FOLLOWING YOUR DOCUMENTATION LEADS TO FUCKING ERRORS AND WE HAVE TO DECOMPILE AND REVERSE ENGINEER YOUR FUCKING "OPEN SOURCE" APPS BECAUSE YOU DONT EVEN BOTHER TO FUCKING UPDATE THE SOURCE CODE ONCE A YEAR WHAT THE FUCK

Thank you for your attention

A few days after deploying a big important Website into production, I wanted to copy the whole thing including DB back onto our test server for future testing/bug fixing if something comes up. (Last changes were done on production server before going live)
So I opened SSH, removed everything on the test sever aaaaand then I realized I was connected to production...
Took about an hour to get everything up and running again. We didn't tell the client and hoped it would not be noticed.

Be me, new dev on a team. Taking a look through source code to get up to speed.

Dev: **thinking to self** why is there no package lock.. let me bring this up to boss man

Dev: hey boss man, you’ve got no package lock, did we forget to commit it?

Manager: no I don’t like package locks.

Dev: ...why?

Manager: they fuck up computer. The project never ran with a package lock.

Dev: ..how will you make sure that every dev has the same packages while developing?

Manager: don’t worry, I’ve done this before, we haven’t had any issues.

**couple weeks goes by**

Dev: pushes code

Manager: hey your feature is not working on my machine

Dev: it’s working on mine, and the dev servers. Let’s take a look and see

**finds out he deletes his package lock every time he does npm install, so therefore he literally has the latest of like a 50 packages with no testing**

Dev: well you see you have some packages here that updates, and have broken some of the features.

Manager: >=|, fix it.

Dev: commit a working package lock so we’re all on the same.

Manager: just set the package version to whatever works.

Dev: okay

**more weeks go by**

Manager: why are we having so many issues between devs, why are things working on some computers and not others??? We can’t be having this it’s wasting time.

Dev: **takes a look at everyone’s packages** we all have different packages.

Manager: that’s it, no one can use Mac computers. You must use these windows computers, and you must install npm v6.0 and node v15.11. Everyone must have the same system and software install to guarantee we’re all on the same page

Dev: so can we also commit package lock so we’re all having the same packages as well?

Manager: No, package locks don’t work.

**few days go by**

Manager: GUYS WHY IS THE CODE DEPLOYING TO PRODUCTION NOT WORKING. IT WAS WORKING IN DEV

DEV: **looks at packages**, when the project was built on dev on 9/1 package x was on version 1.1, when it was approved and moved to prod on 9/3 package x was now on version 1.2 which was a change that broke our code.

Manager: CHANGE THE DEPLOYMENT SCRIPTS THEN. MAKE PROD RSYNC NODE_MODULES WITH DEV

Dev: okay

Manager: just trust me, I’ve been doing this for years

Who the fuck put this man in charge.

So I took on a fairly big project and poured my heart and soul into it, was the biggest thing I did yet. I kept on sending beta's to the customer after each change for review! Kept on insisting that they review it, the answer was always "this looks amazing keep doing what you're doing"! After I finished and pushed everything to production.

They didn't use it for nearly 6 months! And then out of the blue they call me saying that half of the app is wrong.. WTF? Where was this information during testing! I informed them that the changes would take some time since I need to do migrations and change the whole database schema.

In which they replied "but you already finished it once won't changing things make it easier? We shouldn't pay for your mistakes"
I don't know how I handled that but they should be thankful they were half way across the country 😠😠😠😠

Pushing code to production without testing

When you give production access to the intern

Testing in production?
Nice.
And that is an official Samsung App.

Rather than singling out one person, I wanna present what I see as incompetent/stupid/ignorant:
- no will to learn
- failure to follow the very specific instructions & later asking for help when they FUBR sth & not even knowing what they did to fuck up in the first place
- asking how to solve stuff, then ignoring the suggestions & doing sth totally against recommendations
- failure to remember most basic stuff, especially if not writing it down to look at later when needed
- failure to check logs & 'google' stuff before asking why something isn't working the way they want it
- after two weeks, asking me how feature xy works, mind you they coded it, not me
- asking me why they did something in a specific way - WTF, am I a mind reader?! Who designed that crap?! Me or you?!!
- being passive/aggressive & snarky when told to do something or being asked why isn't it done already
- not testing their shit properly
- not making backups when upgrading (production) servers
- not checking the input value, no validation.. even after many many debacles on production with null ref exceptions
- failure to admit they fucked up
- not learning from (their) mistakes

My bank testing in production environment 😂🤣😂🤣

I see you’re testing in a production environment. I too like to live dangerously.

So today was the worst day of my whole (just started) career.

We have a huge client like 700k users. Two weeks ago we migrated all their services to our aws infrastructure. I basically did most of the work because I'm the most skilled in it (not sure anymore).

Today I discovered:
- Mail cron was configured the wrong way so 3000 emails where waiting to be sent.
- The elastic search service wasn't yet whitelisted so didn't work for two weeks.
- The cron which syncs data between production db en testing db only partly worked.

Just fucking end me. Makes me wonder what other things are broken. I still have a lot to learn... And I might have fucked their trust in me for a bit.

PM says you are spending too much time testing. Fast forward to a production bug and they are standing behind you while you are frantically coding saying "this should have been tested more".

looks like somebody's testing in production 😅

Why bother testing if you can just push to production

Once I purposely made a very obvious mistake to see if QA was actually testing due to how many bugs were getting through to production.

Someone is caught while testing on prod.

If you see someone ranting about a colleague who made a semi-major fuckup which was not recognised during dev and stage testing and made it to production where it was discovered three days after deployment just when he went on holiday, well, that colleague was me.

1. Slack. Pretty good chat app for dev companies, I use it to prevent people standing next to my desk 40 times a day.

2. Unit testing tools, especially when fully automated using a git master branch hook, something like codeship/jenkins, and a deployment service.

3. Jetbrains IDEs. I love Vim, but Jetbrains makes theming, autocompleting & code style checks with mixed templating languages a breeze.

4. Urxvt terminal. It's a bit of work at the start, but so extremely fast and customizable.

5. Cinnamon or i3. Not really dev tools, but both make it easy to organize many windows.

6. A smart production bug logger. I tend to use Bugsnag, Rollbar or Sentry.

7. A good coffee machine. Preferably some high pressure espresso maker which costs more than the CEO's car, using organic fairtrade hipster beans with a picture of a laughing south american farmer. And don't you dare fuck it up with sugar.

8. Some high quality bars of chocolate. Not to consume yourself, but to offer to coworkers while they wait for you to fix a broken deploy. The importance of office politics is not to be underestimated.

Old boss story. This guy was nice but a terrible boss. Also relevant, he has a background in IT so should know better.

Him: So when you wanna check a password is correct you just unhash it in the database?
Me: *facepalm*

Me: Hey we should be doing unit and integration testing at a minimum to lower bugs.
Him: We don't need those, we're not a bank. If a problem comes up we just fix it and push to production.
(A while later)
Him(in email): Why do we keep getting bugs reported. Don't you devs test your code.

I was mildly annoyed at that one.

Him: We're always over budget on projects, how can we fix this.
Me: What if we increase our quotes.(technically there are other ways as well but not really possible at that time)
Him: We can't do that, clients won't want to pay.

Me: *finishing off my handover as I'm leaving for a new job*
Him: Wow you do a lot of work

Reply with #code to #customer

"Everybody has a testing environment. Some people are lucky enough to have a totally seperate environment to run production in."
-Unknown

Testing in production like...

STOP. TESTING. IN. PRODUCTION.
STOP. TESTING. IN. PRODUCTION.
STOP. TESTING. IN. PRODUCTION.
STOP. TESTING. IN. PRODUCTION.

Deploying something to production without testing it first because it "should" work.

Someone is testing in production :D

Wordpress does not suck. If you know how to work it.

Past period I saw so many rants on WP. My rant is that it is not 100% WP fault. Yes there are seriously structural problems in WP but that does not mean you cannot create top-notch websites.

At my work we create those top-notch WP sites. Blazing fast and manageable. Seriously we got a customer request to make the site slower because it loaded pages to fast (ea; you hardly could see you switched pages).

- We ONLY use a strict set of plugins that we think are stable, useful.
- We have everything in composer (and our own Satis) for plugins.
- We use custom themes & classes. Our code is MVC with Twig.
- In our track history we have 0 hacked websites for the past 2 years.
- Everything runs stable 24/7
- We have OTAP (testing, acceptance & production environments)
- We patch really fast

These are sites going from $15k++ and we know our shit.

Don't hate on WP if you have no clue what you are doing yourself.

That is my rant.

Speaking of pushing test notifications in prod...

We have 2 layers of testing environments and production.

I tested the changes on the 1st layer, bud since it was 5min to lunch i did not test on 2nd layer which is connected to the production DB. I pushed to production and caused 5+ websites to go full retard and went to lunch.

Came back to 19emails and 3+ skype msgs about "why the fck would you do that..."

Estimated damages nearly 20k EUR and i lost some permissions for two weeks, but my great boss helped me out and cheered me up by telling stories how he took down multiple servers too

plot twist: im the team leader of our office now :)

Manager: what is the estimations for this task A nd B?

Me: Task A: 3 months for 1 guys, and task B: 2 Months

Manager: ok, u can have a fresher, and finish task A, and u urself can pick task B, u can train him and bring him up to pace...

Me: (trying to recalibrate my estimations)...

Manager: oh and u have 3 weeks to deliver production ready scalable quality code with junits, documentation and testing done...

Me: then why the fuck did u bother for the estimates?

Manager: oh that is just for the process complaince...I don't want any trouble in audit...

My worst devSin was testing in production once because I was too lazy to set up the dev environment locally.
Never will I do that mistake again!

So, a few years ago I was working at a small state government department. After we has suffered a major development infrastructure outage (another story), I was so outspoken about what a shitty job the infrastructure vendor was doing, the IT Director put me in charge of managing the environment and the vendor, even though I was actually a software architect.
Anyway, a year later, we get a new project manager, and she decides that she needs to bring in a new team of contract developers because she doesn't trust us incumbents.
They develop a new application, but won't use our test team, insisting that their "BA" can do the testing themselves.
Finally it goes into production.
And crashes on Day 1. And keeps crashing.
Its the infrastructure goes out the cry from her office, do something about it!
I check the logs, can find nothing wrong, just this application keeps crashing.
I and another dev ask for the source code so that we can see if we can help find their bug, but we are told in no uncertain terms that there is no bug, they don't need any help, and we must focus on fixing the hardware issue.
After a couple of days of this, she called a meeting, all the PMs, the whole of the other project team, and me and my mate. And she starts laying into us about how we are letting them all down.
We insist that they have a bug, they insist that they can't have a bug because "it's been tested".
This ends up in a shouting match when my mate lost his cool with her.
So, we went back to our desks, got the exe and the pdb files (yes, they had published debug info to production), and reverse engineered it back to C# source, and then started looking through it.
Around midnight, we spotted the bug.
We took it to them the next morning, and it was like "Oh". When we asked how they could have tested it, they said, ah, well, we didn't actually test that function as we didn't think it would be used much....
What happened after that?
Not a happy ending. Six months later the IT Director retires and she gets shoed in as the new IT Director and then starts a bullying campaign against the two of us until we quit.

Me: Hey boss, if you ever need someone to get into doing DevOps related tasks for the team, I'd be more than happy to take that on.

Boss: We don't really need any dedicated person to work on that, but if we do in the future, I'll let you know.

Fast forward a few days: I am now unable to deploy bug fixes to our testing environment, now in the cloud, because all access has been blocked for everyone except the two numbskulls who thought it'd be a great idea to move EVERYTHING over (apps, configuration manager, proxies, etc) first.

Oh, and this bug is affecting production.

I just blocked some of the top management from connecting to our WLAN because I was testing a verifing feature for said WiFi that kicks all devices not listed in the DB.
It happened while my boss/senior/guidance was trying to show them the advantages of a centrally managed infrastructure.
He covered my ass well and tried to sell it to them as proof of a secure solution, that unknown devices couldn't log in.
I feel like human trash right now, but that's what you get for testing in production.

- Be me (Dev)
- Develop website for client that has an online payment integration.
- Notice there is no one for QA and testing.
- Try to raise voice about it but get shot down by management.
- Get order to push the project to production (because the deadline has arrived, therefore the project must be ready)
- Production performance has an error margin of 15% (customers getting their card over/under charged)
- Get blamed and flamed for everything cuz well, its all the dev's fault.

FML

IKEA testing in production?

I'm coming off a lengthy staff augmentation assignment awful enough that I feel like I need to be rehabilitated to convince myself that I even want to be a software developer.

They needed someone who does .NET. It turns out what they meant was someone to copy and paste massive amounts of code that their EA calls a "framework." Just copy and paste this entire repo, make a whole ton of tweaks that for whatever reason never make their way back into the "template," and then make a few edits for some specific functionality. And then repeat. And repeat. Over a dozen times.

The code is unbelievable. Everything is stacked into giant classes that inherit from each other. There's no dependency inversion. The classes have default constructors with a comment "for unit testing" and then the "real" code uses a different one.

It's full of projects, classes, and methods with weird names that don't do anything. The class and method names sound like they mean something but don't. So after a dozen times I tried to refactor, and the EA threw a hissy fit. Deleting dead code, reducing three levels of inheritance to a simple class, and renaming stuff to indicate what it does are all violations of "standards." I had to go back to the template and start over.

This guy actually recorded a video of himself giving developers instructions on how to copy and paste his awful code.

Then he randomly invents new "standards." A class that reads messages from a queue and processes them shouldn't process them anymore. It should read them and put them in another queue, and then we add more complication by reading from that queue. The reason? We might want to use the original queue for something else one day. I'm pretty sure rewriting working code to meet requirements no one has is as close as you can get to the opposite of Agile.

I fixed some major bugs during my refactor, and missed one the second time after I started over. So stuff actually broke in production because I took points off the board and "fixed" what worked to add back in dead code, variables that aren't used, etc.

In the process, I asked the EA how he wanted me to do this stuff, because I know that he makes up "standards" on the fly and whatever I do may or may not be what he was imagining. We had a tight deadline and I didn't really have time to guess, read his mind, get it wrong, and start over. So we scheduled an hour for him to show me what he wanted.

He said it would take fifteen minutes. He used the first fifteen insisting that he would not explain what he wanted, and besides he didn't remember how all of the code he wrote worked anyway so I would just have to spend more time studying his masterpiece and stepping through it in the debugger.

Being accountable to my team, I insisted that we needed to spend the scheduled hour on him actually explaining what he wanted. He started yelling and hung up. I had to explain to management that I could figure out how to make his "framework" work, but it would take longer and there was no guarantee that when it was done it would magically converge on whatever he was imagining. We totally blew that deadline.

When the .NET work was done, I got sucked into another part of the same project where they were writing massive 500 line SQL stored procedures that no one could understand. They would write a dozen before sending any to QA, then find out that there was a scenario or two not accounted for, and rewrite them all. And repeat. And repeat. Eventually it consisted of, one again, copying and pasting existing procedures into new ones.

At one point one dev asked me to help him test his procedure. I said sure, tell me the scenarios for which I needed to test. He didn't know. My question was the equivalent of asking, "Tell me what you think your code does," and he couldn't answer it. If the guy who wrote it doesn't know what it does right after he wrote it and you certainly can't tell by reading it, and there's dozens of these procedures, all the same but slightly different, how is anyone ever going to read them in a month or a year? What happens when someone needs to change them? What happens when someone finds another defect, and there are going to be a ton of them?

It's a nightmare. Why interview me with all sorts of questions about my dev skills if the plan is to have me copy and paste stuff and carefully avoid applying anything that I know?

The people are all nice except for their evil XEB (Xenophobe Expert Beginner) EA who has no business writing a line of code, ever, and certainly shouldn't be reviewing it.

I've tried to keep my sanity by answering stackoverflow questions once in a while and sometimes turning evil things I was forced to do into constructive blog posts to which I cannot link to preserve my anonymity. I feel like I've taken a six-month detour from software development to shovel crap. Never again. Lesson learned. Next time they're not interviewing me. I'm interviewing them. I'm a professional.

Somebody at Samsung is testing something in production 🤭

It's "Find my mobile" which is preinstalled by default on Samsung devices

Just got this notification. I see virgin active UK likes to test in production. I too like to live dangerously.

Well the clown strikes again,

How do u break production and a testing environment in one night?

One full month preping for same thing that revolves around one config file and assured us he was confident,

He wasn't
he managed to fuck it up so bad for the team d brass lost d plot,

I'm not one for condemning people but my God Dante's inferno woulda had an extra ring if he worked with this buck,

The stupidity has shattered my belief in sunshine and rainbows

"Who needs a staging server, test suites and continuous integration anyways haha"
-company i just joined

So my department is "integrating CI/CD"

Right now, there's a very anti-automation culture in the deployment process, and out of our many applications, almost none have automated testing. And my groups is the only one that uses feature branching - one of the few groups that uses branching at all beyond "master, dev"

So yeah... You could see how this is already ENTIRELY fucked from the very beginning.

First thing they want to do is add better support for a process... Which goes directly against CI/CD.

The process is that to deploy to production (even after it is manually approved by manager), someone in another department needs to press a button to manually deploy. This, as far as I can tell, is for business rule reasons rather than technical ones.

They want us to improve that (the system will stay exactly the same with some streamlined options for said button pressers)

I'm absolutely astounded at the way our management wants to do something but goes in exactly the opposite direction. It's like the found an article of what CI/CD was and then took notes on exactly what not to do.

testing in production

Last night my boss played with our access points in the warehouse for a client, he messed something up and they stopped working.

I asked a person from our service to fix them

Service: he fucked something up again?
Me: yup
S: can you fix them?
M: yup
S: then why ask me?
M: it's not my job 😂

He swapped them, and got mad.

Friend of mine: so I wonder how do you test your applications in the startup?

Me: testing? *grabs his coffee laughing*

Actually we have a complete build pipeline from commit/pull-request to dev and production environments. No tests. Really. We are in rapid product development / research state.

We change technologies and approaches like our underwear (and yeah, this is frequently). If we settled some day and understood the basic problems of the whole feature palette, we'll talk about tests again.

Another case of testing in production, who knew EA actually did testing?

Freelance project I was working on was deployed. Without my knowledge. At 11pm. Their in-house "tech guy" thought that the preview build i gave them was good enough for deployment. Massive bug, broke their api endpoints.
Got a call at 2 in the morning,asking for a fix. I told them how it was their fault and the App they deployed had TESTING written right on the main screen.
They promised additional payment to get me to fix it asap.
Went through the commit history (thank goodness their tech guy knew git, fuck him for committing on production though) and the crash reports.
Removed three lines. All became right with the world again. 😎

If you're going to request CRITICAL changes to thousands of records in the database, and approve it through testing which is done on an exact replica of production, then tell me it was done incorrectly after the fact it has been implemented and you didn't actually review the changes made to the data or business logic that you requested then you are an idiot. Our staging environment is there to ensure all the changes are accurate you useless human. Its the data you provided, I didn't just magically pull it from thin air to make yours and my job a pain the ass.

After months of development, testing, testing and even more testing the app was ready for deployment to production. Happy days, the end was in sight!

I had a week's leave so I handed over the preparation for deployment to my Senior Developer and left it in his capable hands while I enjoyed the sun and many beers.

I came back on the day of deployment and proudly pressed the deploy button. Hurrah!

Not long after I got loads of phone calls from around the country as the app wasn't working. What madness is this?! We tested this for months!

Turns out my Senior didn't like the way I'd written the SQL queries so he changed them. Which is obviously both annoying and unprofessional, but even worse he got a join wrong so the memory usage was a billion times more and it drained the network bandwidth for the whole site when I tried to debug it.

I got all the grief for the app not working and for causing many other incidents by running queries that killed the network.

So...much... rage!!!

When you test your backend code thoroughly before pushing to production, but a fatal exception with the much larger production userbase causes one of your vital threads to die with a NullPointerException.

Russia’s biggest pharmacy pushes hardcoded localhost testing helloworld shit to production

Worst collaboration experience story?

I was not directly involved, it was a Delphi -> C# conversion of our customer returns application.

The dev manager was out to prove waterfall was the only development methodology that could make convert the monolith app to a lean, multi-tier, enterprise-worthy application.

Starting out with a team of 7 (3 devs, 2 dbas, team mgr, and the dev department mgr), they spent around 3 months designing, meetings, and more meetings. Armed with 50+ page specification Word document (not counting the countless Visio workflow diagrams and Microsoft Project timeline/ghantt charts), the team was ready to start coding.
The database design, workflow, and UI design (using Visio), was well done/thought out, but problems started on day one.
- Team mgr and Dev mgr split up the 3 devs, 1 dev wrote the database access library tier, 1 wrote the service tier, the other dev wrote the UI (I'll add this was the dev's first experience with WPF).
- Per the specification, all the layers wouldn't be integrated until all of them met the standards (unit tested, free from errors from VS's code analyzer, etc)
- By the time the devs where ready to code, the DBAs were already tasked with other projects, so the Returns app was prioritized to "when we get around to it"

Fast forward 6 months later, all the devs were 'done' coding, having very little/no communication with one another, then the integration. The service and database layers assumed different design patterns and different database relationships and the UI layer required functionality neither layers anticipated (ex. multi-users and the service maintaining some sort of state between them).

Those issues took about a month to work out, then the app began beta testing with real end users. App didn't make it 10 minutes before users gave up. Numerous UI logic errors, runtime errors, and overall app stability. Because the UI was so bad, the dev mgr brought in one of the web developers (she was pretty good at UI design). You might guess how useful someone is being dropped in on complex project , months after-the-fact and being told "Fix it!".

Couple of months of UI re-design and many other changes, the app was ready for beta testing.

In the mean time, the company hired a new customer service manager. When he saw the application, he rejected the app because he re-designed the entire returns process to be more efficient. The application UI was written to the exact step-by-step old returns process with little/no deviation.

With a tremendous amount of push-back (TL;DR), the dev mgr promised to change the app, but only after it was deployed into production (using "we can fix it later" excuse).

Still plagued with numerous bugs, the app was finally deployed. In attempts to save face, there was a company-wide party to celebrate the 'death' of the "old Delphi returns app" and the birth of the new. Cake, drinks, certificates of achievements for the devs, etc.

By the end of the project, the devs hated each other. Finger pointing, petty squabbles, out-right "FU!"s across the cube walls, etc. All the team members were re-assigned to other teams to separate them, leaving a single new hire to fix all the issues.

Caught someone testing in production on my way home the other day....

Thanks Amazon, that was a very helpful and informative notification.

> Worst work culture you've experienced?

It's a tie between my first to employers.

First: A career's dead end.
Bosses hardly ever said the truth, suger-coated everything and told you just about anything to get what they wanted. E.g. a coworker of mine was sent on a business trip to another company. They had told him this is his big chance! He'd attend a project kick-off meeting, maybe become its lead permanently. When he got there, the other company was like "So you're the temporary first-level supporter? Great! Here's your headset".

And well, devs were worth nothing anyway. For every dev there were 2-3 "consultants" that wrote detailed specifications, including SQL statements and pseudocode. The dev's job was just to translate that to working code. Except for the two highest senior devs, who had perfect job security. They had cooked up a custom Ant-based build system, had forked several high-profile Java projects (e.g. Hibernate) and their code was purposely cryptic and convoluted.
You had no chance to make changes to their projects without involuntarily breaking half of it. And then you'd have to beg for a bit of their time. And doing something they didn't like? Forget it. After I suggested to introduce automated testing I was treated like a heretic. Well of course, that would have threatened their job security. Even managers had no power against them. If these two would quit half a dozen projects would simply be dead.

And finally, the pecking order. Juniors, like me back then, didn't get taught shit. We were just there for the work the seniors didn't want to do. When one of the senior devs had implemented a patch on the master branch, it was the junior's job to apply it to the other branches.

Second: A massive sweatshop, almost like a real-life caricature.
It was a big corporation. Managers acted like kings, always taking the best for themselves while leaving crumbs for the plebs (=devs, operators, etc). They had the spacious single offices, we had the open plan (so awesome for communication and teamwork! synergy effects!). When they got bored, they left meetings just like that. We... well don't even think about being late.
And of course most managers followed the "kiss up, kick down" principle. Boy, was I getting kicked because I dared to question a decision of my boss. He made my life so hard I got sick for a month, being close to burnout. The best part? I gave notice a month later, and _he_still_was_surprised_!
Plebs weren't allowed anything below perfection, bosses on the other hand... so, I got yelled at by some manager. Twice. For essentially nothing, things just bruised his fragile ego. My bosses response? "Oh he's just human". No, the plebs was expected to obey the powers that be. Something you didn't like? That just means your attitude needs adjustment. Like with the open plan offices: I criticized the noise and distraction. Well that's just my _opinion_, right? Anyone else is happily enjoying it! Why can't I just be like the others? And most people really had given up, working like on a production line.

The company itself, while big, was a big ball of small, isolated groups, sticking together by office politics. In your software you'd need to call a service made by a different team, sooner or later. Not documented, noone was ever willing to help. To actually get help, you needed to get your boss to talk to their boss. Then you'd have a chance at all.

Oh, and the red tape. Say you needed a simple cable. You know, like those for $2 on Amazon. You'd open a support ticket and a week later everyone involved had signed it off. Probably. Like your boss, the support's boss, the internal IT services' boss, and maybe some other poor sap who felt important. Or maybe not, because the justification for needing that cable wasn't specific enough. I mean, just imagine the potential damage if our employees owned a cable they shouldn't!

You know, after these two employers I actually needed therapy. Looking back now, hooooly shit... that's why I can't repeat often enough that we devs put up with way too much bullshit.

When you accidentally publish the testing version of your app to the production channel instead of the beta channel... 😣

I was asked to look into a site I haven't actively developed since about 3-4 years. It should be a simple side-gig.
I was told this site has been actively developed by the person who came after me, and this person had a few other people help out as well.
The most daunting task in my head was to go through their changes and see why stuff is broken (I was told functionality had been removed, things were changed for the worse, etc etc).
I ssh into the machine and it works. For SOME reason I still have access, which is a good thing since there's literally nobody to ask for access at the moment.
I cd into the project, do a git remote get-url origin to see if they've changed the repo location. Doesn't work. There is no origin. It's "upstream" now. Ok, no biggie. git remote get-url upstream. Repo is still there. Good.
Just to check, see if there's anything untracked with git status. Nothing. Good.
What was the last thing that was worked on? git log --all --decorate --oneline --graph. Wait... Something about the commit message seems familiar. git log. .... This is *my* last commit message. The hell?
I open the repo in the browser, login with some credentials my browser had saved (again, good because I have no clue about the password). Repo hasn't gotten a commit since mine. That can't be right.
Check branches. Oh....Like a dozen new branches. Lots of commits with text that is really not helpful at all. Looks like they were trying to set up a pipeline and testing it out over and over again.
A lot of other changes including the deletion of a database config and schema changes. 0 tests. Doesn't seem like these changes were ever in production.
...
At least I don't have to rack my head trying to understand someone else's code but.... I might just have to throw everything that was done into the garbage. I'm not gonna be the one to push all these changes I don't know about to prod and see what breaks and what doesn't break
.
I feel bad for whoever worked on the codebase after me, because all their changes are now just a waste of time and space that will never be used.

In one of my first jobs i developed an (ugly and heavly under-payed) e-commerce/media platform for a customer.
That customer was constantly making fun of his bald partner telling how he was gay, liked dicks, etc., drawing dicks and bananas as sample website logos or uploading dildo/penis images as images, he was always like this.
Once the website was ready for production i removed all the "testing" posts and images and told the client to insert some real content and alert me when it was ready for release.
Well some time after the release i got a call from that client, for the first time he was serious:

C: Hi, why there are dildo images on the server? (the website in production was full of dildo/penis images instead of actual product images, he even photoshopped the head of his partner on a penis and uploaded it!!!)
R: ehm... i told you it was on production and to stop uploading bad content....
C: Ummm ok, please fix it immediatly, thanks!

!rant

One of our clients discovered a bug on our site about an hour ago. I wrote up a fix, and after very little testing, pushed it into production. I then immediately left to go home.

You can say I like to live my life dangerously.

I work for a bank and every production release date it's a chaos... Like, for real, devs running to get their stories approved by the testing team and last minute scope changes that, if not made, would make the whole app fail (real shitty management as you can see).

Longstoryshort, a dev didn't finished one of his stories and create 7 major bugs with another... Today that was my breakfast, took me 4 hours and get it all done and approved... We didn't make the release tho, but I scored some major points with this.

Funny thing, tomorrow I'm telling my PM I'll leave the company for a better job, so that will be their breakfast.

Got pulled out of bed at 6 am again this morning, our VMs were acting up again. Not booting, running extremely slow, high disk usage, etc.

This was the 6 time in as many weeks this happened. And always the marching orders were the same. Find the bug, smash the bug, get it working with the least effort. I've dumped hundreds of hours maintaining this broken shitheap of a system, putting off other duties to keep mission critical stations running.

The culprits? Scummy consultants, Windows 10 1709, and Citrix Studio.

Xen Server performed well enough, likely due to its open source origins and Centos architecture.

Whelp. DasSeahawks was good and pissed. Nothing like getting rousted out of bed after a few scant hours rest for patching the same broken system.

DasSeahawks lost his temper. Things went flying. Exorcists were dispatched and promptly eaten.

Enough. No consultants, no analysts, and no experts touched it. No phone calls, no manuals, not even a google search. Just a very pissed admin and his minion declaring blitzkrieg.

We made our game plan, moved the users out, smoked our cigs, chugged monster, and queued a gnu-metal playlist on spotify.

Then we took a wrecking ball to the whole setup. User docs were saved, all else was rm -r * && shred && summon -u Poseidon -beast Land_Cracken.

Started at 3pm and finished just after midnight. Rebuilt all the vms with RDP, murdered citrix studio (and their bullshit licenses), completely blocked Windows 10 updates after 1607, and load balanced the network.

So what do we get when all the experts are fired? Stabbed lightning. VMs boot in less than 10 seconds, apps open instantly, and server resources are half their previous usage state. My VMs are now the fastest stations in our complex, as they should be.

Next to do: install our mxgpu, script up snapshots and heartbeat, destroy Windows ads/telemetry, and setup PDQ. damn its good to be good!

What i learned --> never allow testing to go to production, consultants will fuck up your shit for a buck, and vendors are half as reliable over consultants. Windows works great without Microsoft, thin clients are overpriced, and getting pissed gets things done.

This my friends, is why admins are assholes.

How to NOT write unit tests:

A colleague of mine has developed a new package of software, many of our new projects are going to use. So in his presentation of the new functionalities he also showed us that he used unit tests to cover some of his code. So i asked him to show me that all tests passes.

He: I can show you, but one test suit will fail currently.

Me: Why?? You told us, everything is finished and works fine.

He: That's right, but they will fail because I'm currently not in the customer VPN.

Me: Excuse me, WHAT??

He: Yes, I'm not in the VPN that connects me to this one customers facility in Hungary, where the counterpart of the software is runnung live.

Me: YOU WROTE UNIT TESTS THAT TEST AGAINST A RUNNING LIVE FACILITY??

He: Yes, so I can check, that the telegramms I send are right. If I get back the right acknowledgement, the telegramm structure is right and my code is working.

Me: You know, that is not the porpose of unit tests? You know, that these test should run in any environment?

He: But they are proving, that my code is working. Everytime I change something I connect to the customer and let the tests run.

Me: ...

Despite the help of some other developers we could not convince him that this was not good and he should remove them. So now this package is used in 2 new projects and this test suit is still failing, everytime you execute all unit tests.

For some reason, when testing if a block of code is firing, I always do console.log('fart'); More often than not I forget to remove it before going to production. I feel good knowing that's out there on dozens of sites right now.

So this just happened...don't test on production kids (....sometimes you have no choice BUT DON'T F**KING DO IT OK!?)

Testing in a production environment is like closing a door in a to kill a snake and electricity goes off 📴

We developed an application and deployed on production (but not launched)
And business team already created lot of garbage or dummy data. Reporting systems are huge pile of bars, stats and shit.

Now, has to destroy and clean production.

Already advise them to do experiments or testing dev or staging.

Damn. First time in my career experienced this. Has to delete production.

Spent a lot of time designing a proper HTTP (dare I even say RESTful) API for our - what is until now a closed system, using a little-known/badly-supported message-over-websocket protocol to do RPC-style communications - supposedly enterprise-grade product.

I make the API spec go through several rounds of review with the rest of the dev team and customers/partners alike. After a few iterations, everybody agrees that the spec will meet the necessary requirements.

I start implementing according to spec. Because this is the first time we're actually building proper HTTP handling into the product, but we of course have to make it work at least somewhat with the RPC-style codebase, it's mostly foundational work. But still, I manage to get some initial endpoints fully implemented and working as per the spec we agreed. The first PR is created, reviews are positive, the direction is clear and what's there already works.

At this point in time, I leave on my honeymoon for two weeks. Naturally, I assume that the remaining endpoints will be completed following the outlines/example of the endpoints which I built. When I come back, the team mentions that the implementation is completed and I believe all is well.

The feature is deployed selectively to some alpha customers to start validation testing before the big rollout. It's been like that for a good month, until a few days ago when I get a question related to a PoC integration which they can't seem to get to work.

I start investigating and notice that the API hasn't been implemented according to the previously agreed upon spec at all. Not only did the team manage to implement the missing functionality in strange and some even broken ways, they also managed to refactor my previously working endpoints into being non-compliant.

Now, I'm a flexible guy. It's not because something isn't done exactly as I've imagined it that it's automatically bad. However, I know from experience that designing a good/clear/future-proof API is a tricky exercise. I've put a lot of time and effort into deliberate design decisions that made up the spec that we all reviewed repeatedly and agreed upon. The current implementation might also be fine, but I now have to go over each endpoint again and reason about whether the implementation still fulfills the requirements (both soft and hard) that we set out to meet.

I'm met with resistance, pushback and disbelief from product management and dev co-workers alike when I raise the concern that the API might actually not be production-ready (while I'm frantically rewriting my integration tests and figuring out how the actual implementation works in comparison to what was spec'ed).

Oh, and did I mention that product management wants to release this by end-of-week?!

This morning I was looking in our database in order to solve a problem with a user registration and I accidentally noticed some users registered with unusual email addresses (temporary mail services, Russian providers and so on...).
I immediately thought about malicious users so I dug into the logs and I found that the registration requests started from an IP address belonging to our company (we have static IP addresses). My first reaction was: «OMG! Russian hackers infiltrated into our systems and started registering new users!»
So, I found the coworker owning the laptop from which the requests were sent and I went to him in order to warn him that someone violated his computer.
And he said: «Ah! Those 7 users? Yeah, I was doing some tests, I registered them. My email address was already registered so I created some new ones».
Really, man? Really? WTF

I used to work on a production management team, whose job was, among other things, safeguarding access to production. Dev teams would send us requests all the time to, "run a quick SQL script."

Invariably, the SQL would include, "SELECT * FROM db_config."

We would push the tickets back, and the devs would call us, enraged. I learned pretty quickly that they didn't have any real interest in dev, test, or staging environments, and just wanted to do everything in prod, and see if it works.

But they would give up their protests pretty fast when I offered to let them speak to a manager when they were upset I wouldn't run their SQL.

Dammit Carl!
Testing in production again?

Me: Im testing a new feature that is not on production yet for 30m and can not make it work. (I asked the developer for any idea why is not working)
Dev: i just tested, works fine.
Me: i just tested again with no luck. I’m i missing anything?
Dev: (Developer comes to my desk). Lets see what you doing wrong here. (After 5seconds). You're not on UAT. You have been redirected to the production and you've been testing there all this time.
Me: 😩

I was testing database migration and there was some issues which I couldn't fix them. So I drop the table just to test if the issues are still there.
"Why nothing have been changed?" After some minutes I realize it was production database which I had dropped 😓

I absolutely HATE that stage fright feeling I get when I'm about to launch new software into production mode for a client! Anyone else feel that? Makes me want to vomit thinking of all the promises I've made that it will work fine and then all the things I don't even realize could go wrong. I never have enough testing resources because client budgets tend to favor shiny features at the expense of testing.

I just auto charged myself because I forgot I tested my 7 day trial in production last week

🤦‍♂️🤡🤦‍♂️🤡🤦‍♂️🤡🤦‍♂️🤡🤦‍♂️🤡🤦‍♂️🤡🤦‍♂️🤡🤦‍♂️🤡🤦‍♂️🤡🤦‍♂️🤡🤦‍♂️🤡🤦‍♂️🤡🤦‍♂️🤡🤦‍♂️🤡🤦‍♂️🤡🤦‍♂️🤡🤦‍♂️🤡🤦‍♂️🤡🤦‍♂️🤡🤦‍♂️🤡🤦‍♂️🤡🤦‍♂️🤡

-not commenting
-leaving console logs behind in production
-not testing if it works in IE
-using root too much
-using if instead of switch
-never staying consistent with naming conventions
-starting projects and never finishing

Thank you Tata sky

Come on... Whatever we say, these guys won't just stop testing in production

Yet another dev team testing in a production environment

Somebody is testing in production.

My co-worker, still studying but working as a "senior dev", just decided that we don't need a test/staging environment anymore. We just "validate" (we also don't use the word "test" anymore) newly created features in production.

Makes absolutely sense...

Thank god I have a new job from february on!

Year 2020 is deployed into production without testing.
Isn't it?

When your college approaches you with an update and want to test it in production....

Testing a brand new build on production FTW 👍

I run a service with 157311 (mostly inactive) users and I test in production.

Testing in production... (German public TV)

Borrowed from Reddit and Twitter:
Everybody has a testing environment. Some people are lucky enough enough to have a totally separate environment to run production in.

We have a production release tomorrow and we didn't have any kind of testing other than the unit tests we wrote. 💣💥🤯🎆

"I always test in production"

This sentence by a colleague made me a murderer (almost).
Jesus...

Hello DevRant community! It’s been a while, almost 5 years to be exact. The last time I posted here, I was a newbie, grappling with the challenges of a new job in a completely new country. Oh, how time flies!

Fast forward to today, and it’s been quite the journey. The codebase that once seemed like an indecipherable maze is now my playground. The bugs that used to keep me up at night are now my morning coffee puzzles. And the team, oh the team! We’ve moved from awkward nods to inside jokes and shared victories.

But let’s talk about the real hero here - the coffee machine. The unsung hero that has fueled late-night coding sessions and early morning stand-ups. It’s seen more heated debates than the PR comments section. If only it could talk, it would probably write its own rant about the indecisiveness of developers choosing between cappuccino and latte.

And then there are the unforgettable ‘learning opportunities’ - moments like accidentally shutting down the production server or dropping the customer database. Yes, they were panic-inducing crises of apocalyptic proportions at that time, but in hindsight, they were valuable lessons. Lessons about the importance of thorough testing, proper version control, reliable backup systems, and most importantly, owning up to our mistakes.

So here’s to the victories and failures, the bugs and fixes, the refactorings and 'wontfix’s. Here’s to the incredible journey of growth and learning. And most importantly, here’s to this amazing community that’s always been there with advice, sympathy, humor, and support.

Can’t wait to see what the next 5 years bring! 🥂

Startup-ing 101, from Fitbit:
- spy on users
- sell data
- cut production costs
- mutilate people's bodies, leaving burn scars that will never heal
- announce the recall, get PR, and make the refund process impossibly convoluted
- never give actual refunds
- claim that yes, fitbit catches fire, but only the old discontinued device, just to mess with search results and make the actual info (that all devices catch fire) hard to find
- try hard to obtain the devices in question, so people who suffered have no evidence
- give bogus word salad replies to the press

This is what one of the people burned has to say:
"I do not have feeling in parts of my wrist due to nerve damage and I will have a large scar that will be with me the rest of my life. This was a traumatic experience and I hope no one else has to go through it. So, if you own a Fitbit, please reconsider using it."

Ladies and gentlemen, cringefest starts. One of fitbit replies:
"Fitbit products are designed and produced in accordance with strict standards and undergo extensive internal and external testing to ensure the safety of our users. Based on our internal and independent third party testing and analysis, we do not believe this type of injury could occur from normal use. We are committed to conducting a full investigation. With Google's resources and global platform, Fitbit will be able to accelerate innovation in the wearables category, scale faster, and make health even more accessible to everyone. I could not be more excited for what lies ahead".

In the future, corporate speech will be autogenerated.

(if you wear fitbit, just be aware of this.)

Things that seem "simple" but end up taking a long ass time to actually deploy into production:

1. Using a new payment processor:

"It's just a simple API, I'll be done in 2 hours"

LOL sure it is, but testing orders and setting up a sandbox or making sure you have credentials right, and then switching from test to life and retesting, and then... fuck

2. Making changes to admin stats.

"'I just have to add this column and remove that one... maybe like a couple of hours"

YOU WISH

3. Anything Javascript

"Hah, what, that's like a button, np"

125 minutes later...

console.log('before foo');
console.log(this.foo)
etc..

I guess someone has been testing in production

> Client complains acceptance testing is too expensive and that he'll do it himself.
> Client complains acceptance testing wasn't thorough enough when a bug hits production.

Ever had a day that felt like you're shoveling snow from the driveway? In a blizzard? With thunderstorms & falling unicorns? Like you shovel away one m² & turn around and no footprints visible anymore? And snow built up to your neck?

Today my work day was like that.. xcept shit..shit instead of pretty & puffy snow!!

Working on things a & b, trying to not mess either one up, then comes shit x, coworker was updating production.. ofc something went wrong.. again not testing after the update..then me 'to da rescue'.. :/ hardly patch things up, so it works..in a way.. feature c still missing due to needed workarounds.. going back to a and b.. got disrupted by the same coworker who is nver listening, but always asking too much..

And when I think I finally have the b thing figured out a f-ing blocker from one of our biggest clients.. The whole system is unresponsive.. Needles to say, same guy in support for two companies (their end), so they filed the jira blocker with the wrong customer that doesn't have a SLA so no urgent emails..and then the phone calls.. and then the hell broke loose.. checking what is happening.. After frantic calls from our dba to anyone who even knows that our customer exists if they were doing sth on the db.. noup, not a single one was fucking with the prod db.. The hell! Materialised view created 10 mins ago that blocked everything..set to recreate every 10 minutes..with a query that I am guessing couldn't even select all that data in under 15.. dafaaaq?! Then we kill it..and again it is there.. We found out that customers dbas were testing something on live environment, oblivious that they mamaged to block the entire db..
FML, I'm going pokemon hunting.. :/ codename for ingress n beer..

I know politics is not allowed here, but I have to share this gem with you.

One day before the election for the European parliament the website of the German city Bochum showed a wrong bar diagram with false results of the election for a few seconds.

Everyone was loling. But I was like WTF? They were testing in production. And also they included data were the party AFD had about 50% of the votes. Are they retarded or so?

Me: Im testing a new feature that is not on production yet for 30m and can not make it work. (I asked the developer for any idea why is not working)
Dev: i just tested, works fine.
Me: i just tested again with no luck. I’m i missing anything?
Dev: (Developer comes to my desk). Lets see what you doing wrong here. (After 5seconds). You have been redirected to the production site and you've been testing there all this time.
Me: 😩

Yet another day at work:

My job is to write test libraries for web services and test others code. Yes I know to code, and have a niche in software testing.
Sometimes developers (whose code I find bugs in) get so defensive and scream in emails and meetings if I point out an issue in their code.
Today, when I pointed a bug in his repo, a developer questioned me in an email asking if I even understood his code, and as a tester I shouldn’t look at his code and only blackbox test it.

I wish I can educate the defensive developer that sometimes, it’s okay to make mistakes and be corrected. That’s how we deliver services that doesn’t suck in production.

Testing an attendace machine API one by one so i know what does what.

And there’s an API for wiping all the attendance data stored in the machine.

I didn’t realize it until i push the damn button.

The attendance machine just become fresh like new 😱😨😰

Shit.

A testing session just become an extreme sport.

Thankfully the IT guy has a backup but just up to last month.

Well, it’s better than nothing isn’t it?

He just tell his boss that the machine was run out of memory and the attendance data for the current month were not saved.

And he ask him to buy me some machine for testing.

Yes, i was living on the edge by testing in the production machine.

Remember to always setup and test your production setup!

Seem like a local college forgot to do some QA 😂

Me: *asks boss for the id of his test store so I can apply experimental schema changes to test out a new dashboard app*

Boss: *gives me production store id and doesn't say anything*

Fate: … "You got lucky this time."

This is the CEO of the company btw. Startups. *sigh*

Does anyone know of any apps/companies that are using .Net core in production?

I’ve started a project at work which consists of a webapi written in dotnet core, a react spa for the front end and xunit for testing.

I’m just curious as there are loss of sites about things written in Rails, or django, but almost nothing about dotnet core.

Woops! I was debugging a particularly snarky issue in Production the other day. This morning I realized about 60 minutes into my new coding I hadn't changed my profile back. I was testing form submissions on a live customer's site.

Ok, here goes...

I was once asked to evaluate upgrade options for an online shop platform.

The thing was built on Zend 1, but that's not the problem.

The geniuses that worked on it before didn't have any clue about best practices, framework convention, modular thinking, testing, security issues...nothing!

There were some instances when querying was done using a rudimentary excuse for a model layer. Other times, they would just use raw queries and just ignore the previous method. Sometimes the database calls were made in strange function calls inside randomly loaded PHP files from different folders from all over the place. Sometimes they used JOINs to get the data from multiple tables, sometimes they would do a bunch of single table queries and just loop every data set to format it using multiple for loops.

And, best of all, there were some parts of the app that would just ignore any ideea of frameworks, conventions and all that and would be just a huge PHP file full of spagetti code just spalshed around, sometimes with no apparent logic to it. Queries, processing, HTML...everything crammed in one file...

The most amazing thing was that this code base somehow managed to function in production for more than 5 years and people actualy used it...

Imagine the reaction I got from the client the moment I said we should burn it to the ground and rebuild the whole thing from scratch...

Good thing my boss trusted me and backed me up (he is a great guy by the way) and we never had to go along with that Frankenstein monster...

TL;DR Dear boss, firstly, you always get someone to review anything important done by a fucking intern.
Secondly, you do not give access to your fucking client's production server to an intern.
Thirdly, you don't ask your fucking intern to test the intern's work that has not been reviewed by anyone directly on your client's fucking production server.

Last week, the boss and one of the lead devs (the only guy with some serious knowledge about systems and networking) decided to give me (an intern who barely has any work experience) the task of fixing or finding an alternate solution to allowing their support team access to their client machines. Currently they used a reverse SSH tunnel and an intermediary VH but for some reason, that was very unreliable in terms of availability. I suggested using OpenVPN and explained how it would work. Seemed to be a far better idea and they accepted. After several days of working through documentations and guides and everything, I figured out how OpenVPN works and managed to deploy a TEST server and successfully test remote access using two VMs. On seeing my tests, the boss told me that he wanted to test it on the client network. I agreed. Today he comes to me and he tells me to prepare testing for tomorrow and that the client technician is going to give me access to one of their boxes. And then he adds, "It's a working prod server. We'll see if we can make it work on that" and left. I gaped at him for a while and asked another dev guy in the room if what I heard was right. He confirmed. Turns out, the lead dev and the boss's son (who also works here) had had a huge argument since morning on the same issue and finally the dev guy had washed it off his hands and declared that if anything goes wrong from testing it on production, it's entirely the boss's own fault. That's when the boss stepped in and approached me. I ran back to his office and began to explain why prod servers don't top the list of things you can fuck around with. But he simply silenced me saying, "What can go wrong?" and added, "You shouldn't stay still. You should keep moving". Okay, like firstly what the fuck and secondly, what the fuck?.
Even though OpenVPN client is not the scariest thing to install, tomorrow's going to be fun.

fuck this!
spent an hour trying get my website working (on a raspberry) ... no errors, dev tools gave nothing, php gives nothing mySQL related... weird.. debugged my code for an hour when it me... db on my pc for testing is not the same one as the "production" server. i am so fucking stupid... i need some sleep

So, awesome clip to use for testing... Problem is, I found this in the codebase for a production app.

*Facepalm*

testing in production

Another member of the team updated the production JSON configuration of the project with a missing comma, this broke a system that's not yet live and where there aren't any real users (only used for demos/testing).

Instead of having a good laugh about how silly this was, the CTO/CEO removed their write access to production..

I'm using my devrant avatar for some UI testing :P , thanks @trogus and @dfox
(no production and plagiarism intended)

Why is web development such a headache?

I'm writing a responsive wesbite from scratch. All goes perfect, even cross browser.
It all works, adapts to screen size etc. Nice! About to get this code into production.

Me: I'll test the iPhone 5 viewport size before I push the code...

Responsive Developer Tools:

FireFox: nu uh, there's a magic random 1px margin to every element on your page now, which you cannot find in your css or in the computed tab. It's magical.

Me: weird, what if I change the viewport size to the iPhone 6's dimensions?

Issue persists.

Me: hmm, what if I add or substract one fucking pixel from the viewport width or height?

FireFox: What 1px margin? Don't know what you're talking about ... There never was one...

Me: ok, weird (sets viewport size back to the iPhone 5 format for testing)

FireFox: I present to you: the magic random 1px margin.

I'm at a loss. I really am. Been clicking and unclicking almost every responsive part of my css I could find for this page and it just doesn't want to work persistently. And I swear to god that it worked a week ago in that exact viewport size. It's so frustrating.

Jesus fuckin' Christ. I own a webshop together with someone else. This guy is so fuckin' stupid. Yesterday I've deployed a release to our acceptance environment. I talked with him extensively about it. This morning I texted him to check out all the new stuff.

5 minutes later he texts.me back: I would suggest changing option x. Uuh... what option x? We don't have that any more.

Dude! What the fuck! We talked extensively about acceptance testing so you know it is in our acceptance environment, not production, asshole.

And then again, he asks for the link to the acceptance, which I gave him twice already.

Are you really that stupid??

A story about a helpless intern : On a fine day an intern was assigned a feature to develope. He worked his ass off, completed it, submitted it for testing, the build was approved by the tester and got released the next day. Fast forward a few days, the feature is failing in few cases at the production.Everbody, starts pointing fingers at the intern.
The intern wonders...How the fuck am I gonna know this fucking use case, do they really expect me to to build a full proof feature without telling me about all the possible senarios...And how the fuck did the tester approved this...? I mean, now that I know this senario, it seems pretty obvious that it should have been tested...!
Note : This also happened to another developer who recently joined...The PM failed to properly communicate all the requirements and the fucking lazy ass tester did not consider all the possible senarios. And the script failed in the production...!
Note : It's 4 fucking AM and the intern still can't sleep...

Client calls me requesting a new simple feature.
Connect to FTP server.
Edit some PHP pages and upload them back, check if the changes actually worked.

Basically implementing and testing a new feature on a live production website...

PS. It didn't work the first or second time

"Real devs test in production", in practice.

This was actually the second such notification I received. Not sure if this is standard for mobile app testing...

A coworker reviews if my applications work from time to time (as it is his job).
*sends request*
"Hey nothing works I can't get a single response"
*I check his request*
S-T-O-P using production users when testing, d#head.

Overengineering. Finding the right point between overdesign and no design at all. That's where fancy languages and unusual patterns being hit by real world problems, and you need to deal with all that utter mess you created being architecture astronaut. Isn't that funny how you realize that another fancy tool is fundamentally incompatible with the task you need to solve, and you realize it after a month of writing workarounds and hacks.

But on the other hand, duct tape slacking becomes a mess even quicker.

Not being able to promote projects. You may code the shit out of side project and still get zero response, absolutely no impact. That's why your side projects often becomes abandoned.

Oversleeping. You thought tomorrow was productive day, but you wake up oversleeped, your head aches, your mind is not clear and you be like "fuck that, I'm staying in bed watching memes all day". But there's job that has to be done, and that bothers you.

Writing tests. Oh, words can't describe how much I hate writing tests, any kind of. I tried testing so many times in high school, at university, even at production, but it seems like my mind is just doesn't accept it. I know that testing is fundamentally important, but my mind collapses every time I try to write a single fucking test, resulting in terrible headache. I don't know why it's like that, but it is, and I better repl the shit out of pure function than write fucking tests.

Most intense day for me was at the very start of my career. Internship... went with product manager to client's office while PM installed new test version of our product for on-site integration testing. Shortly after deploy, client manager came over to ask why production had gone down...

Turns out that manually typing DB name as part of deployment script is not, erm, risk free. PM entered production DB name and took out a very busy call centre for a few hours. Agents in tears, customers raging on phones, etc! After we restored and got everything back up and running, he reached me the keyboard and said "You're doing it this time."

My attempt was problem free, thankfully. Earned many brownie points that day.

A couple of years ago I was working on a fairly large system with a complex (by necessity) access control architecture.

As is usually the case with those projects, it's awkward for developers to repro bugs that have to do with a user's accesses in production when we are not allowed to replicate production data in test, let alone locally.

We had a bug where I ended up making myself a new row in the production database for a thing I could have access to without affecting real data to repro it safely. I identified the bug so I could repro it in dev/test and removed the row and ensured everything worked normally, whew scary.

Have you ever walked into the office one day, and everyone is hunched over in a semicircle around one person's workstation, before one turns around to look at you and says - after a pause - "... ltlian?.."

Turns out I had basically "poisoned the well" with my dummy entity in a way where production now threw 500 for everyone BUT me who had transitive access to this post-non-entity. Due to the scope of the system, it had taken about a day for this to gradually propagate in terms of caching and eventual consistencies; new entities coming in was expected, but not that they disappear.

Luckily I had a decent track record for this to be a one-off. I sometimes think about how I would explain testing in prod and making it faceplant before going home for the day, other than "I assumed it would be fine". I would fire me.

I'm so happy that I have the first working version of the program. Just couple of bug fixing and unit testing and I'll push the code to production.

Day 1: Just couple of bug fixing and unit testing and I'll push the code to production.

Day 2: Just couple of bug fixing and unit testing and I'll push the code to production.
.........

Day 200: Just couple of bug fixing and unit testing and I'll push the code to production.

😞

New twist on an old favorite.

Background:
- TeamA provides a service internal to the company.
- That service is made accessible to a cloud environment, also has a requirement to be made available to machines on the local network so you can develop against it.
- Company is too cheap/stupid to get a s2s vpn to their cloud provider.
- Company also only hosts production in the cloud, so all other dev is done locally, or on production non-similar infra, local dev is podman.
- They accomplish service connectivity by use of an inordinately complicated edge gateway/router/firewall/message translator/ouija board/julienne fry maker, also controlled by said service team.

Scenario:
Me: "Hey, we're cool with signing requests using an x509 cert. That said, doing so requires different code than connecting to an unsecured endpoint. Please make this service accessible to developer machines and lower environments on the internal network so we can, you know, develop."
TeamA: "The service should be accessible to [cloud ip range]"
Me: "Yes, that's a production range. We need to be able to test the signing code without testing in production"
TeamA: "Can you mock the data?"
Me: "The code we are testing is relating to auth, not business logic"
TeamA: "What are you trying to do?"
Me: "We are trying to test the code that uses the x509 you provide to connect to the service"
TeamA: "Can you deploy to the cloud"
Me: "Again, no, the cloud is only production per policy, all lower environments are in the local data center"
TeamA: "can you try connecting to the gateway?"
Me: "Yes, we have, it's not accessible, it only has public DNS, and only allows [cloud ip range]"
TeamA: "it work when we try it"
Me: "Can you please supply repro steps so we can adjust our process"
TeamA: "Yes, log into the gateway and try issuing the call from there"
Me: (╯°□°)╯︵ ┻━┻

tl;dr: Works on my server

How the hell some devs are releasing to production before passing to QA, Alpha testing, staging at least handle all type of errors and hide your API Keys.

TL;DR: When picking vendors to outsource work to, vet them really well.

Backstory:
Got a large redesign project that involves rebuilding a website's main navigation (accessibility reasons).
Project is too big just for our dev team to handle with our workload so we got to bring a 3rd party vendor to help us. We do this often so no big deal.
But, this time the twist was Senior Management already had retained hours with a dev shop so they want us to use them for project. Okay...

It begins:
Have our scope / discovery meeting about the changes and our expected DevOps workflow.

Devs work Local and push changes to our Github, that kicks off the build and we test on Dev, then it goes to Staging for more testing & PM review. Once ready we can push to prod, or whenever needed. All is agreed, everyone was happy.

Emailed the vendors' project manager to ask for their devs Github accounts so we can add them to the project. Got no reply for 3 days.

4th day, I get back "Who sets up the Github accounts?"

fuck me. they've never used Github before but in our scope meeting 4 days ago you said Github was fine...??

Whatever, fuck it. I'll make the accounts and add them.

Added 4 devs to the repo and setup new branch. 40min later get an email that they can't setup dev environment now, the dev doesn't know how to setup our CMS locally, "not working for some reason."

So, they ask for permission to develop on our STAGING server.. "because it's already setup"... they want to actively dev on our staging where we get PM/Senior Management approvals?

We have dev, staging, production instances and you want to dev in staging, not dev?... nay nay good sir.

This is whom senior management wants us to use, already paid for via retainer no less. They are a major dev shop and they're useless...

😢😭

Cant wait for today's progress checkup meeting. 😐😐

/rant

Does anyone work on a team with multiple stacks?

For example we have batch jobs in Java but also have a JS front-end and APIs.

How do you divide the developers and the work across these projects?

Currently everyone does everything but I feel like this is inefficient and hard to develop expertise. And different people or even the same person will make the same mistakes over and over again because they don't know how to do X or they forget or overlook some quirk. When I switched Beck to JS took me like a week to get a Promises nailed down again. And this morning someone else had a production bug and couldn't figure it out. But when I looked at the code I could pretty much see where an issue could be (uncaught exception in a promise)

Also the testing frameworks are very different and there's a lot of infrastructure technical debt, things that really should've been done a long time or fixed but no one had the time or expertise to do it or notice it (until it causes a production issue and then everyone is like WTF is happening??!!!!).

I'm not the manager but I always feel that the team needs to be split along the language lines and specific people need to own these projects to review and code changes for all these common newbie errors. And also developer enough expertise to foresee problems before it becomes a production issue.

I'm testing a shitforbrains code in production because I don't have any other options. He went on vacation and said that the code should work, and if it doesn't it is my problem.

I am contemplating taking a shit under his desk and working from home. Then he would have to deal with my shit aswell. Literally.
The tables have turned my friend...

I'm genuinely shocked at the number of people I see on here bashing automated testing as a waste of time, simply because my entire career has taught me the opposite (and it's usually only non-technical managers I see who don't want to see "time wasted" writing tests.)

I'm also just as genuinely curious - what do you guys do instead? Just don't test and deal with production issues as they occur? Pass it off to a separate UAT / human-based testing department and let them sign it off? Assume that because you're using Haskell / some other discipline it'll work if it compiles?

Since I have seen a lot of people uploading this kind of stuff lately, here is Xiaomi's test in production, back in 2017 November...

The most crazy issue I've fixed was caused by a TCP behavior which I didn't know, called the "half-closed connection".
There was a third-party application installed on a production server which called a LDAP server for retrieving users information. During the day we had several users using the application and all worked fine. During the night, when the application was not accessed, something happened and the first call to the application in the morning was stuck for about 5 minutes before returning a response. I tried to reproduce the issue in a testing environment without success. Then I discovered that the application and the LDAP server were located on two different networks, with a firewall between them. And firewalls sometimes drop old connections. For this reason network applications usually implement a keep-alive mechanism. Well, the default LDAP Java libraries don't set the keep-alive on their connections. So, I found a library called "libdontdie", which force the keep-alive on the connections. I installed the library on the server, loaded it at the startup and the weird stuck behavior in the morning disappeared.

Testing in production eh?

So glad to be staying a new job next week. Today a junior colleague asked me what the best way to test something would be as it won't work locally. Knowing this has a good chance of taking down the server, I suggest he sounds up a server on his AWS account. My manager comes in, oh no I don't want him doing it on AWS use the production server instead. By the time stuff States hitting the fan I will be gone.