Motivational quote!!!!

I worked with a good dev at one of my previous jobs, but one of his faults was that he was a bit scattered and would sometimes forget things.

The story goes that one day we had this massive bug on our web app and we had a large portion of our dev team trying to figure it out. We thought we narrowed down the issue to a very specific part of the code, but something weird happened. No matter how often we looked at the piece of code where we all knew the problem had to be, no one could see any problem with it. And there want anything close to explaining how we could be seeing the issue we were in production.

We spent hours going through this. It was driving everyone crazy. All of a sudden, my co-worker (one referenced above) gasps “oh shit.” And we’re all like, what’s up? He proceeds to tell us that he thinks he might have been testing a line of code on one of our prod servers and left it in there by accident and never committed it into the actual codebase. Just to explain this - we had a great deploy process at this company but every so often a dev would need to test something quickly on a prod machine so we’d allow it as long as they did it and removed it quickly. It was meant for being for a select few tasks that required a prod server and was just going to be a single line to test something. Bad practice, but was fine because everyone had been extremely careful with it.

Until this guy came along. After he said he thought he might have left a line change in the code on a prod server, we had to manually go in to 12 web servers and check. Eventually, we found the one that had the change and finally, the issue at hand made sense. We never thought for a second that the committed code in the git repo that we were looking at would be inaccurate.

Needless to say, he was never allowed to touch code on a prod server ever again.

Oh, man, I just realized I haven't ranted one of my best stories on here!

So, here goes!

A few years back the company I work for was contacted by an older client regarding a new project.

The guy was now pitching to build the website for the Parliament of another country (not gonna name it, NDAs and stuff), and was planning on outsourcing the development, as he had no team and he was only aiming on taking care of the client service/project management side of the project.

Out of principle (and also to preserve our mental integrity), we have purposely avoided working with government bodies of any kind, in any country, but he was a friend of our CEO and pleaded until we singed on board.

Now, the project itself was way bigger than we expected, as the wanted more of an internal CRM, centralized document archive, event management, internal planning, multiple interfaced, role based access restricted monster of an administration interface, complete with regular user website, also packed with all kind of features, dashboards and so on.

Long story short, a lot bigger than what we were expecting based on the initial brief.

The development period was hell. New features were coming in on a weekly basis. Already implemented functionality was constantly being changed or redefined. No requests we ever made about clarifications and/or materials or information were ever answered on time.

They also somehow bullied the guy that brought us the project into also including the data migration from the old website into the new one we were building and we somehow ended up having to extract meaningful, formatted, sanitized content parsing static HTML files and connecting them to download-able files (almost every page in the old website had files available to download) we needed to also include in a sane way.

Now, don't think the files were simple URL paths we can trace to a folder/file path, oh no!!! The links were some form of hash combination that had to be exploded and tested against some king of database relationship tables that only had hashed indexes relating to other tables, that also only had hashed indexes relating to some other tables that kept a database of the website pages HTML file naming. So what we had to do is identify the files based on a combination of hashed indexes and re-hashed HTML file names that in the end would give us a filename for a real file that we had to then search for inside a list of over 20 folders not related to one another.

So we did this. Created a script that processed the hell out of over 10000 HTML files, database entries and files and re-indexed and re-named all this shit into a meaningful database of sane data and well organized files.

So, with this we were nearing the finish line for the project, which by now exceeded the estimated time by over to times.

We test everything, retest it all again for good measure, pack everything up for deployment, simulate on a staging environment, give the final client access to the staging version, get them to accept that all requirements are met, finish writing the documentation for the codebase, write detailed deployment procedure, include some automation and testing tools also for good measure, recommend production setup, hardware specs, software versions, server side optimization like caching, load balancing and all that we could think would ever be useful, all with more documentation and instructions.

As the project was built on PHP/MySQL (as requested), we recommended a Linux environment for production. Oh, I forgot to tell you that over the development period they kept asking us to also include steps for Windows procedures along with our regular documentation. Was a bit strange, but we added it in there just so we can finish and close the damn project.

So, we send them all the above and go get drunk as fuck in celebration of getting rid of them once and for all...

Next day: hung over, I get to the office, open my laptop and see on new email. I only had the one new mail, so I open it to see what it's about.

Lo and behold! The fuckers over in the other country that called themselves "IT guys", and were the ones making all the changes and additions to our requirements, were not capable enough to follow step by step instructions in order to deploy the project on their servers!!!

[Continues in the comments]

Testing in production doesn't hurt.

IT
FUCKING
KILLS

Just ask Boeing.

Gotta love testing in production. Nice work Niantic 🤣

Another day at work...

Alpha testing be like...

Looks like someone is testing in production

Just in bank testing app in production

Doot doot.

My day: Eight lines of refactoring around a 10-character fix for a minor production issue. Some tests. Lots of bloody phone calls and conference calls filled with me laughing and getting talked over. Why? Read on.

My boss's day: Trying very very hard to pin random shit on me (and failing because I'm awesome and fuck him). Six hours of drama and freaking out and chewing and yelling that the whole system is broken because of that minor issue. No reading, lots of misunderstanding, lots of panic. Three-way called me specifically to bitch out another coworker in front of me. (Coworker wasn't really in the wrong.) Called a contractor to his house for testing. Finally learned that everything works perfectly in QA (duh, I fixed it hours ago). Desperately waited for me to push to prod. Didn't care enough to do production tests afterwards.

My day afterwards: hey, this Cloudinary transform feature sounds fun! Oh look, I'm done already. Boo. Ask boss for update. Tests still aren't finished. Okay, whatever. Time for bed.

what a joke.

Oh, I talked to the accountant after all of this bullshit happened. Apparently everyone that has quit in the last six years has done so specifically because of the boss. Every. single. person.

I told him it was going to happen again.

I also told him the boss is a druggie with a taste for psychedelics. (It came up in conversation. Absolutely true, too.) It's hilarious because the company lawyer is the accountant's brother.

So stupid.

I just sent an automated email titled "Gary is a Dinosaur!" to a lot of humourless clients because the ancient application I was testing assumed I was in the production environment. 🙃

Lesson to self: stop using bogus names in testing.

Still, it could've been a lot worse... 😂

Testing in production??

Don't know whether to laugh or cry!!😂😥

Why are job postings so bad?
Like, really. Why?

Here's four I found today, plus an interview with a trainwreck from last week.
(And these aren't even the worst I've found lately!)

------

Ridiculous job posting #1:
* 5 years React and React Native experience -- the initial release of React Native was in May 2013, apparently. ~5.7 years ago.
* Masters degree in computer science.
* Write clean, maintainable code with tests.
* Be social and outgoing.

So: you must have either worked at Facebook or adopted and committed to both React and React Native basically immediately after release. You must also be in academia (with a masters!), and write clean and maintainable code, which... basically doesn't happen in academia. And on top of (and really: despite) all of this, you must also be a social butterfly! Good luck ~

------

Ridiculous job posting #2:
* "We use Ruby on Rails"
* A few sentences later... "we love functional programming and write only functional code!"

Cue Inigo Montoya.

------

Ridiculous job posting #3:
* 100% remote! Work from anywhere, any time zone!
* and following that: You must have at least 4 work hours overlap with your coworkers per day.
* two company-wide meetups per quarter! In fancy places like Peru and Tibet! ... TWO PER QUARTER!?

Let me paraphrase: "We like the entire team being remote, together."

------

Ridiculous job posting #4:
* Actual title: "Developer (noun): Superhero poised to change the world (apply within)"
* Actual excerpt: "We know that headhunters are already beating down your door. All we want is the opportunity to earn our right to keep you every single day."
* Actual excerpt: "But alas. A dark and evil power is upon us. And this… ...is where you enter the story. You will be the Superman who is called upon to hammer the villains back into the abyss from whence they came."

I already applied to this company some time before (...surprisingly...) and found that the founder/boss is both an ex cowboy dev and... more than a bit of a loon. If that last part isn't obvious already? Sheesh. He should go write bad fantasy metal lyrics instead.

------

Ridiculous interview:
* Service offered for free to customers
* PHP fanboy angrily asking only PHP questions despite the stack (Node+Vue) not even freaking including PHP! To be fair, he didn't know anything but PHP... so why (and how) is he working there?
* Actual admission: No testing suite, CI, or QA in place
* Actual admission: Testing sometimes happens in production due to tight deadlines
* Actual admission: Company serves ads and sells personally-identifiable customer information (with affiliate royalties!) to cover expenses
* Actual admission: Not looking for other monetization strategies; simply trying to scale their current break-even approach.

------

I find more of these every time I look. It's insane.
Why can't people be sane and at least semi-intelligent?

Everyone has these projects sooner or later;

PM: 'You have 2 weeks to finish. It HAS to be done by then because they will start using it ASAP!!'
Me: Working day and night to meet the deadline.

After 2 years PM comes by;

PM: remember that program? They're testing it now. They plan to use it in production.

Rghuu*~ (that's me dying a little on the inside)

My first testing job in the industry. Quite the rollercoaster.

I had found this neat little online service with a community. I signed up an account and participated. I sent in a lot of bug reports. One of the community supervisors sent me a message that most things in FogBugz had my username all over it.

After a year, I got cocky and decided to try SQL injection. In a production environment. What can I say. I was young, not bright, and overly curious. Never malicious, never damaged data or exposed sensitive data or bork services.

I reported it.

Not long after, I got phone calls. I was pretty sure I was getting charged with something.

I was offered a job.

Three months into the job, they asked if I wanted to do Python and work with the automators. I said I don't know what that is but sure.

They hired me a private instructor for a week to learn the basics, then flew me to the other side of the world for two weeks to work directly with the automation team to learn how they do it.

It was a pretty exciting era in my life and my dream job.

Testing in production

"We use production DB in testing, so be careful with it"

Yes. The way to go. Testing in production for a banking site .

Every so often I remember that the code I wrote is running in production and real customers are using it and I feel a little bit sick

Sex talk between programmers.

She: I'm a virgin.
He: Don't worry. They call me the virginslayer007.
She: Oh! So how many virgins have you slayed till now?
He: That would be ONE in a few minutes.
She: So u r also a virgin then..
He: Don't worry. I watched so many video tutorials. We just have to do exactly as they did. Best thing is that it can be done both for testing and production purposes.
She: Let's stick to testing purposes for now.

In my current work, I have two systems to work on (let's name em Systems A and B). Both basically do the same thing; both allow users to book facilities available to them.

System A is already in production. My job is to fix any bugs that come up on said system. System B is an improved version that they wanted me to develop. This would follow a different framework etc. I am already halfway through this system.

Now, here's the fucked up part. The code for system A is a massive clusterfuck. It has unused commented code dated back to ancient times where men had the brain of an ape.

And don't get me started on the fucking logic. One part of the code was to retrieve and display the timeslots available for a chosen facility. The code to do that alone takes up 500++ fucking lines, filled with ajax commands, html manipulation and commented, unused codes..AND THAT'S JUST THE FRONTEND!

The fucking backend was not a problem of smelly code anymore. Nope. It was like a programmer had code diarrhea and shat his backend code all over the project. If I had a pin board, I would have made a crazy wall just to understand what some fucknut was trying to achieve.

Anyway, my supervisor told me to fix some bugs on System A. Knowing how the code was, I told her that I could refactor the code. Since I've already achieved that function on System B, with a shorter and cleaner code, I could just copy that and use on System A. But nope. She SPECIFICALLY told me to just "do whatever to fix the bugs. I don't want to waste time on System A." Okay. Makes sense to me. Whatever. I didn't wanna fuck my head up looking through that mess of a cesspool. So, I came up with a few hacks, not thinking of clean code and fixed whatever bugs there was. I then just pushed to the repo (after testing of course).

This bloody morning, supervisor came in and gave me more bugs to fix. When I thought she was done, she said "Hey. I saw the fix you made to the system. The bugs are fixed but the retrieval of the timeslots is now pretty slow. Could you see what is the problem?"

Slow.. She said that it was slow. And asked if I could fix it. I already told her what the problem was and she did not want me to waste time on it. But she wants me to fix it. WHAT THE FUCK IS WRONG IN HER BLOODY HEAD! I SWEAR TO GOD... UGHHHHH I swear I was already waterboarding her in my head. YOU WANT FAST?? How bout fucking allowing me to refactor the code?? Fucking shit head. I think I should take up yoga.

Some years back I was working in a project that essentially dealt with all things related to foreigners and foreign affairs in Switzerland. You could manage entry visas, work permits, citizenship, international warrants, Interpol requests, etc.

One of the test managers (from client side - i.e. the government) was once manually "testing" and mixed up the production and test instance, to both of which he was logged in at the time.
The test case then ended up setting up an entry ban against himself, as he used his own name for testing...

Next time he returned from vacation the border control at the airport were like "Uhm, Sir, we can't let you into the country. Please come with us." :D :D

(He managed to clear that up in end, I dare say, though, that he learned his lesson.)

So I took on a fairly big project and poured my heart and soul into it, was the biggest thing I did yet. I kept on sending beta's to the customer after each change for review! Kept on insisting that they review it, the answer was always "this looks amazing keep doing what you're doing"! After I finished and pushed everything to production.

They didn't use it for nearly 6 months! And then out of the blue they call me saying that half of the app is wrong.. WTF? Where was this information during testing! I informed them that the changes would take some time since I need to do migrations and change the whole database schema.

In which they replied "but you already finished it once won't changing things make it easier? We shouldn't pay for your mistakes"
I don't know how I handled that but they should be thankful they were half way across the country 😠😠😠😠

When even your spammers are testing in production...

When you give production access to the intern

FUCKING TELEGRAM FUCK YOU STAY IN YOUR FUCKING API DOCUMENTATION AND STOP FUCKING TESTING YOUR SHIT ON A PRODUCTION SYSTEM WHY WOULD YOU DO THAT FUCK OFF WHY AM I EVEN DEVELOPING SHIT FOR YOUR PLATFORM ANYMORE WHEN FOLLOWING YOUR DOCUMENTATION LEADS TO FUCKING ERRORS AND WE HAVE TO DECOMPILE AND REVERSE ENGINEER YOUR FUCKING "OPEN SOURCE" APPS BECAUSE YOU DONT EVEN BOTHER TO FUCKING UPDATE THE SOURCE CODE ONCE A YEAR WHAT THE FUCK

Thank you for your attention

Testing in production?
Nice.
And that is an official Samsung App.

My bank testing in production environment 😂🤣😂🤣

So today was the worst day of my whole (just started) career.

We have a huge client like 700k users. Two weeks ago we migrated all their services to our aws infrastructure. I basically did most of the work because I'm the most skilled in it (not sure anymore).

Today I discovered:
- Mail cron was configured the wrong way so 3000 emails where waiting to be sent.
- The elastic search service wasn't yet whitelisted so didn't work for two weeks.
- The cron which syncs data between production db en testing db only partly worked.

Just fucking end me. Makes me wonder what other things are broken. I still have a lot to learn... And I might have fucked their trust in me for a bit.

I see you’re testing in a production environment. I too like to live dangerously.

Rather than singling out one person, I wanna present what I see as incompetent/stupid/ignorant:
- no will to learn
- failure to follow the very specific instructions & later asking for help when they FUBR sth & not even knowing what they did to fuck up in the first place
- asking how to solve stuff, then ignoring the suggestions & doing sth totally against recommendations
- failure to remember most basic stuff, especially if not writing it down to look at later when needed
- failure to check logs & 'google' stuff before asking why something isn't working the way they want it
- after two weeks, asking me how feature xy works, mind you they coded it, not me
- asking me why they did something in a specific way - WTF, am I a mind reader?! Who designed that crap?! Me or you?!!
- being passive/aggressive & snarky when told to do something or being asked why isn't it done already
- not testing their shit properly
- not making backups when upgrading (production) servers
- not checking the input value, no validation.. even after many many debacles on production with null ref exceptions
- failure to admit they fucked up
- not learning from (their) mistakes

looks like somebody's testing in production 😅

"Everybody has a testing environment. Some people are lucky enough to have a totally seperate environment to run production in."
-Unknown

So, a few years ago I was working at a small state government department. After we has suffered a major development infrastructure outage (another story), I was so outspoken about what a shitty job the infrastructure vendor was doing, the IT Director put me in charge of managing the environment and the vendor, even though I was actually a software architect.
Anyway, a year later, we get a new project manager, and she decides that she needs to bring in a new team of contract developers because she doesn't trust us incumbents.
They develop a new application, but won't use our test team, insisting that their "BA" can do the testing themselves.
Finally it goes into production.
And crashes on Day 1. And keeps crashing.
Its the infrastructure goes out the cry from her office, do something about it!
I check the logs, can find nothing wrong, just this application keeps crashing.
I and another dev ask for the source code so that we can see if we can help find their bug, but we are told in no uncertain terms that there is no bug, they don't need any help, and we must focus on fixing the hardware issue.
After a couple of days of this, she called a meeting, all the PMs, the whole of the other project team, and me and my mate. And she starts laying into us about how we are letting them all down.
We insist that they have a bug, they insist that they can't have a bug because "it's been tested".
This ends up in a shouting match when my mate lost his cool with her.
So, we went back to our desks, got the exe and the pdb files (yes, they had published debug info to production), and reverse engineered it back to C# source, and then started looking through it.
Around midnight, we spotted the bug.
We took it to them the next morning, and it was like "Oh". When we asked how they could have tested it, they said, ah, well, we didn't actually test that function as we didn't think it would be used much....
What happened after that?
Not a happy ending. Six months later the IT Director retires and she gets shoed in as the new IT Director and then starts a bullying campaign against the two of us until we quit.

Old boss story. This guy was nice but a terrible boss. Also relevant, he has a background in IT so should know better.

Him: So when you wanna check a password is correct you just unhash it in the database?
Me: *facepalm*

Me: Hey we should be doing unit and integration testing at a minimum to lower bugs.
Him: We don't need those, we're not a bank. If a problem comes up we just fix it and push to production.
(A while later)
Him(in email): Why do we keep getting bugs reported. Don't you devs test your code.

I was mildly annoyed at that one.

Him: We're always over budget on projects, how can we fix this.
Me: What if we increase our quotes.(technically there are other ways as well but not really possible at that time)
Him: We can't do that, clients won't want to pay.

Me: *finishing off my handover as I'm leaving for a new job*
Him: Wow you do a lot of work

Someone is testing in production :D

Testing in production like...

STOP. TESTING. IN. PRODUCTION.
STOP. TESTING. IN. PRODUCTION.
STOP. TESTING. IN. PRODUCTION.
STOP. TESTING. IN. PRODUCTION.

Me: Hey boss, if you ever need someone to get into doing DevOps related tasks for the team, I'd be more than happy to take that on.

Boss: We don't really need any dedicated person to work on that, but if we do in the future, I'll let you know.

Fast forward a few days: I am now unable to deploy bug fixes to our testing environment, now in the cloud, because all access has been blocked for everyone except the two numbskulls who thought it'd be a great idea to move EVERYTHING over (apps, configuration manager, proxies, etc) first.

Oh, and this bug is affecting production.

Wordpress does not suck. If you know how to work it.

Past period I saw so many rants on WP. My rant is that it is not 100% WP fault. Yes there are seriously structural problems in WP but that does not mean you cannot create top-notch websites.

At my work we create those top-notch WP sites. Blazing fast and manageable. Seriously we got a customer request to make the site slower because it loaded pages to fast (ea; you hardly could see you switched pages).

- We ONLY use a strict set of plugins that we think are stable, useful.
- We have everything in composer (and our own Satis) for plugins.
- We use custom themes & classes. Our code is MVC with Twig.
- In our track history we have 0 hacked websites for the past 2 years.
- Everything runs stable 24/7
- We have OTAP (testing, acceptance & production environments)
- We patch really fast

These are sites going from $15k++ and we know our shit.

Don't hate on WP if you have no clue what you are doing yourself.

That is my rant.

Speaking of pushing test notifications in prod...

My worst devSin was testing in production once because I was too lazy to set up the dev environment locally.
Never will I do that mistake again!

Somebody at Samsung is testing something in production 🤭

It's "Find my mobile" which is preinstalled by default on Samsung devices

IKEA testing in production?

I just blocked some of the top management from connecting to our WLAN because I was testing a verifing feature for said WiFi that kicks all devices not listed in the DB.
It happened while my boss/senior/guidance was trying to show them the advantages of a centrally managed infrastructure.
He covered my ass well and tried to sell it to them as proof of a secure solution, that unknown devices couldn't log in.
I feel like human trash right now, but that's what you get for testing in production.

Manager: what is the estimations for this task A nd B?

Me: Task A: 3 months for 1 guys, and task B: 2 Months

Manager: ok, u can have a fresher, and finish task A, and u urself can pick task B, u can train him and bring him up to pace...

Me: (trying to recalibrate my estimations)...

Manager: oh and u have 3 weeks to deliver production ready scalable quality code with junits, documentation and testing done...

Me: then why the fuck did u bother for the estimates?

Manager: oh that is just for the process complaince...I don't want any trouble in audit...

testing in production

Hotstar, India's largest online streaming app, casually testing in production

Last night my boss played with our access points in the warehouse for a client, he messed something up and they stopped working.

I asked a person from our service to fix them

Service: he fucked something up again?
Me: yup
S: can you fix them?
M: yup
S: then why ask me?
M: it's not my job 😂

He swapped them, and got mad.

I'm coming off a lengthy staff augmentation assignment awful enough that I feel like I need to be rehabilitated to convince myself that I even want to be a software developer.

They needed someone who does .NET. It turns out what they meant was someone to copy and paste massive amounts of code that their EA calls a "framework." Just copy and paste this entire repo, make a whole ton of tweaks that for whatever reason never make their way back into the "template," and then make a few edits for some specific functionality. And then repeat. And repeat. Over a dozen times.

The code is unbelievable. Everything is stacked into giant classes that inherit from each other. There's no dependency inversion. The classes have default constructors with a comment "for unit testing" and then the "real" code uses a different one.

It's full of projects, classes, and methods with weird names that don't do anything. The class and method names sound like they mean something but don't. So after a dozen times I tried to refactor, and the EA threw a hissy fit. Deleting dead code, reducing three levels of inheritance to a simple class, and renaming stuff to indicate what it does are all violations of "standards." I had to go back to the template and start over.

This guy actually recorded a video of himself giving developers instructions on how to copy and paste his awful code.

Then he randomly invents new "standards." A class that reads messages from a queue and processes them shouldn't process them anymore. It should read them and put them in another queue, and then we add more complication by reading from that queue. The reason? We might want to use the original queue for something else one day. I'm pretty sure rewriting working code to meet requirements no one has is as close as you can get to the opposite of Agile.

I fixed some major bugs during my refactor, and missed one the second time after I started over. So stuff actually broke in production because I took points off the board and "fixed" what worked to add back in dead code, variables that aren't used, etc.

In the process, I asked the EA how he wanted me to do this stuff, because I know that he makes up "standards" on the fly and whatever I do may or may not be what he was imagining. We had a tight deadline and I didn't really have time to guess, read his mind, get it wrong, and start over. So we scheduled an hour for him to show me what he wanted.

He said it would take fifteen minutes. He used the first fifteen insisting that he would not explain what he wanted, and besides he didn't remember how all of the code he wrote worked anyway so I would just have to spend more time studying his masterpiece and stepping through it in the debugger.

Being accountable to my team, I insisted that we needed to spend the scheduled hour on him actually explaining what he wanted. He started yelling and hung up. I had to explain to management that I could figure out how to make his "framework" work, but it would take longer and there was no guarantee that when it was done it would magically converge on whatever he was imagining. We totally blew that deadline.

When the .NET work was done, I got sucked into another part of the same project where they were writing massive 500 line SQL stored procedures that no one could understand. They would write a dozen before sending any to QA, then find out that there was a scenario or two not accounted for, and rewrite them all. And repeat. And repeat. Eventually it consisted of, one again, copying and pasting existing procedures into new ones.

At one point one dev asked me to help him test his procedure. I said sure, tell me the scenarios for which I needed to test. He didn't know. My question was the equivalent of asking, "Tell me what you think your code does," and he couldn't answer it. If the guy who wrote it doesn't know what it does right after he wrote it and you certainly can't tell by reading it, and there's dozens of these procedures, all the same but slightly different, how is anyone ever going to read them in a month or a year? What happens when someone needs to change them? What happens when someone finds another defect, and there are going to be a ton of them?

It's a nightmare. Why interview me with all sorts of questions about my dev skills if the plan is to have me copy and paste stuff and carefully avoid applying anything that I know?

The people are all nice except for their evil XEB (Xenophobe Expert Beginner) EA who has no business writing a line of code, ever, and certainly shouldn't be reviewing it.

I've tried to keep my sanity by answering stackoverflow questions once in a while and sometimes turning evil things I was forced to do into constructive blog posts to which I cannot link to preserve my anonymity. I feel like I've taken a six-month detour from software development to shovel crap. Never again. Lesson learned. Next time they're not interviewing me. I'm interviewing them. I'm a professional.

Freelance project I was working on was deployed. Without my knowledge. At 11pm. Their in-house "tech guy" thought that the preview build i gave them was good enough for deployment. Massive bug, broke their api endpoints.
Got a call at 2 in the morning,asking for a fix. I told them how it was their fault and the App they deployed had TESTING written right on the main screen.
They promised additional payment to get me to fix it asap.
Went through the commit history (thank goodness their tech guy knew git, fuck him for committing on production though) and the crash reports.
Removed three lines. All became right with the world again. 😎

Well the clown strikes again,

How do u break production and a testing environment in one night?

One full month preping for same thing that revolves around one config file and assured us he was confident,

He wasn't
he managed to fuck it up so bad for the team d brass lost d plot,

I'm not one for condemning people but my God Dante's inferno woulda had an extra ring if he worked with this buck,

The stupidity has shattered my belief in sunshine and rainbows

"Who needs a staging server, test suites and continuous integration anyways haha"
-company i just joined

Caught someone testing in production on my way home the other day....

After months of development, testing, testing and even more testing the app was ready for deployment to production. Happy days, the end was in sight!

I had a week's leave so I handed over the preparation for deployment to my Senior Developer and left it in his capable hands while I enjoyed the sun and many beers.

I came back on the day of deployment and proudly pressed the deploy button. Hurrah!

Not long after I got loads of phone calls from around the country as the app wasn't working. What madness is this?! We tested this for months!

Turns out my Senior didn't like the way I'd written the SQL queries so he changed them. Which is obviously both annoying and unprofessional, but even worse he got a join wrong so the memory usage was a billion times more and it drained the network bandwidth for the whole site when I tried to debug it.

I got all the grief for the app not working and for causing many other incidents by running queries that killed the network.

So...much... rage!!!

Friend of mine: so I wonder how do you test your applications in the startup?

Me: testing? *grabs his coffee laughing*

Actually we have a complete build pipeline from commit/pull-request to dev and production environments. No tests. Really. We are in rapid product development / research state.

We change technologies and approaches like our underwear (and yeah, this is frequently). If we settled some day and understood the basic problems of the whole feature palette, we'll talk about tests again.

Another case of testing in production, who knew EA actually did testing?

If you're going to request CRITICAL changes to thousands of records in the database, and approve it through testing which is done on an exact replica of production, then tell me it was done incorrectly after the fact it has been implemented and you didn't actually review the changes made to the data or business logic that you requested then you are an idiot. Our staging environment is there to ensure all the changes are accurate you useless human. Its the data you provided, I didn't just magically pull it from thin air to make yours and my job a pain the ass.

> Worst work culture you've experienced?

It's a tie between my first to employers.

First: A career's dead end.
Bosses hardly ever said the truth, suger-coated everything and told you just about anything to get what they wanted. E.g. a coworker of mine was sent on a business trip to another company. They had told him this is his big chance! He'd attend a project kick-off meeting, maybe become its lead permanently. When he got there, the other company was like "So you're the temporary first-level supporter? Great! Here's your headset".

And well, devs were worth nothing anyway. For every dev there were 2-3 "consultants" that wrote detailed specifications, including SQL statements and pseudocode. The dev's job was just to translate that to working code. Except for the two highest senior devs, who had perfect job security. They had cooked up a custom Ant-based build system, had forked several high-profile Java projects (e.g. Hibernate) and their code was purposely cryptic and convoluted.
You had no chance to make changes to their projects without involuntarily breaking half of it. And then you'd have to beg for a bit of their time. And doing something they didn't like? Forget it. After I suggested to introduce automated testing I was treated like a heretic. Well of course, that would have threatened their job security. Even managers had no power against them. If these two would quit half a dozen projects would simply be dead.

And finally, the pecking order. Juniors, like me back then, didn't get taught shit. We were just there for the work the seniors didn't want to do. When one of the senior devs had implemented a patch on the master branch, it was the junior's job to apply it to the other branches.

Second: A massive sweatshop, almost like a real-life caricature.
It was a big corporation. Managers acted like kings, always taking the best for themselves while leaving crumbs for the plebs (=devs, operators, etc). They had the spacious single offices, we had the open plan (so awesome for communication and teamwork! synergy effects!). When they got bored, they left meetings just like that. We... well don't even think about being late.
And of course most managers followed the "kiss up, kick down" principle. Boy, was I getting kicked because I dared to question a decision of my boss. He made my life so hard I got sick for a month, being close to burnout. The best part? I gave notice a month later, and _he_still_was_surprised_!
Plebs weren't allowed anything below perfection, bosses on the other hand... so, I got yelled at by some manager. Twice. For essentially nothing, things just bruised his fragile ego. My bosses response? "Oh he's just human". No, the plebs was expected to obey the powers that be. Something you didn't like? That just means your attitude needs adjustment. Like with the open plan offices: I criticized the noise and distraction. Well that's just my _opinion_, right? Anyone else is happily enjoying it! Why can't I just be like the others? And most people really had given up, working like on a production line.

The company itself, while big, was a big ball of small, isolated groups, sticking together by office politics. In your software you'd need to call a service made by a different team, sooner or later. Not documented, noone was ever willing to help. To actually get help, you needed to get your boss to talk to their boss. Then you'd have a chance at all.

Oh, and the red tape. Say you needed a simple cable. You know, like those for $2 on Amazon. You'd open a support ticket and a week later everyone involved had signed it off. Probably. Like your boss, the support's boss, the internal IT services' boss, and maybe some other poor sap who felt important. Or maybe not, because the justification for needing that cable wasn't specific enough. I mean, just imagine the potential damage if our employees owned a cable they shouldn't!

You know, after these two employers I actually needed therapy. Looking back now, hooooly shit... that's why I can't repeat often enough that we devs put up with way too much bullshit.

Thanks Amazon, that was a very helpful and informative notification.

In one of my first jobs i developed an (ugly and heavly under-payed) e-commerce/media platform for a customer.
That customer was constantly making fun of his bald partner telling how he was gay, liked dicks, etc., drawing dicks and bananas as sample website logos or uploading dildo/penis images as images, he was always like this.
Once the website was ready for production i removed all the "testing" posts and images and told the client to insert some real content and alert me when it was ready for release.
Well some time after the release i got a call from that client, for the first time he was serious:

C: Hi, why there are dildo images on the server? (the website in production was full of dildo/penis images instead of actual product images, he even photoshopped the head of his partner on a penis and uploaded it!!!)
R: ehm... i told you it was on production and to stop uploading bad content....
C: Ummm ok, please fix it immediatly, thanks!

Yet another dev team testing in a production environment

Got pulled out of bed at 6 am again this morning, our VMs were acting up again. Not booting, running extremely slow, high disk usage, etc.

This was the 6 time in as many weeks this happened. And always the marching orders were the same. Find the bug, smash the bug, get it working with the least effort. I've dumped hundreds of hours maintaining this broken shitheap of a system, putting off other duties to keep mission critical stations running.

The culprits? Scummy consultants, Windows 10 1709, and Citrix Studio.

Xen Server performed well enough, likely due to its open source origins and Centos architecture.

Whelp. DasSeahawks was good and pissed. Nothing like getting rousted out of bed after a few scant hours rest for patching the same broken system.

DasSeahawks lost his temper. Things went flying. Exorcists were dispatched and promptly eaten.

Enough. No consultants, no analysts, and no experts touched it. No phone calls, no manuals, not even a google search. Just a very pissed admin and his minion declaring blitzkrieg.

We made our game plan, moved the users out, smoked our cigs, chugged monster, and queued a gnu-metal playlist on spotify.

Then we took a wrecking ball to the whole setup. User docs were saved, all else was rm -r * && shred && summon -u Poseidon -beast Land_Cracken.

Started at 3pm and finished just after midnight. Rebuilt all the vms with RDP, murdered citrix studio (and their bullshit licenses), completely blocked Windows 10 updates after 1607, and load balanced the network.

So what do we get when all the experts are fired? Stabbed lightning. VMs boot in less than 10 seconds, apps open instantly, and server resources are half their previous usage state. My VMs are now the fastest stations in our complex, as they should be.

Next to do: install our mxgpu, script up snapshots and heartbeat, destroy Windows ads/telemetry, and setup PDQ. damn its good to be good!

What i learned --> never allow testing to go to production, consultants will fuck up your shit for a buck, and vendors are half as reliable over consultants. Windows works great without Microsoft, thin clients are overpriced, and getting pissed gets things done.

This my friends, is why admins are assholes.

Dammit Carl!
Testing in production again?

Oracle What the fuck is wrong with you ?

How greedy can a company become ? In Germany we have a word for people like you. Eine dreckige Ratte. ( a dirty rat) I thought already that you learned from the Open Office drama but no you just do the same shit again. Why do i have to pay for JDK ? Of course i dont have to pay but what do i see in the terms and conditions for the new jdk ?

You may not: use the Programs for any data processing or any commercial, production, or internal business purposes other than developing, testing, prototyping, and demonstrating your Application;

Which basically means you could sue me for just using your JDK.

Wtf ? Are you completley nuts ? This basically means that people have to use from now on OpenJDK to avoid your snitchy trap. And i hope that this will not withstand in lawsuit if you decide to do such.

Wow how deep can you sink ?

Spent a lot of time designing a proper HTTP (dare I even say RESTful) API for our - what is until now a closed system, using a little-known/badly-supported message-over-websocket protocol to do RPC-style communications - supposedly enterprise-grade product.

I make the API spec go through several rounds of review with the rest of the dev team and customers/partners alike. After a few iterations, everybody agrees that the spec will meet the necessary requirements.

I start implementing according to spec. Because this is the first time we're actually building proper HTTP handling into the product, but we of course have to make it work at least somewhat with the RPC-style codebase, it's mostly foundational work. But still, I manage to get some initial endpoints fully implemented and working as per the spec we agreed. The first PR is created, reviews are positive, the direction is clear and what's there already works.

At this point in time, I leave on my honeymoon for two weeks. Naturally, I assume that the remaining endpoints will be completed following the outlines/example of the endpoints which I built. When I come back, the team mentions that the implementation is completed and I believe all is well.

The feature is deployed selectively to some alpha customers to start validation testing before the big rollout. It's been like that for a good month, until a few days ago when I get a question related to a PoC integration which they can't seem to get to work.

I start investigating and notice that the API hasn't been implemented according to the previously agreed upon spec at all. Not only did the team manage to implement the missing functionality in strange and some even broken ways, they also managed to refactor my previously working endpoints into being non-compliant.

Now, I'm a flexible guy. It's not because something isn't done exactly as I've imagined it that it's automatically bad. However, I know from experience that designing a good/clear/future-proof API is a tricky exercise. I've put a lot of time and effort into deliberate design decisions that made up the spec that we all reviewed repeatedly and agreed upon. The current implementation might also be fine, but I now have to go over each endpoint again and reason about whether the implementation still fulfills the requirements (both soft and hard) that we set out to meet.

I'm met with resistance, pushback and disbelief from product management and dev co-workers alike when I raise the concern that the API might actually not be production-ready (while I'm frantically rewriting my integration tests and figuring out how the actual implementation works in comparison to what was spec'ed).

Oh, and did I mention that product management wants to release this by end-of-week?!

How to NOT write unit tests:

A colleague of mine has developed a new package of software, many of our new projects are going to use. So in his presentation of the new functionalities he also showed us that he used unit tests to cover some of his code. So i asked him to show me that all tests passes.

He: I can show you, but one test suit will fail currently.

Me: Why?? You told us, everything is finished and works fine.

He: That's right, but they will fail because I'm currently not in the customer VPN.

Me: Excuse me, WHAT??

He: Yes, I'm not in the VPN that connects me to this one customers facility in Hungary, where the counterpart of the software is runnung live.

Me: YOU WROTE UNIT TESTS THAT TEST AGAINST A RUNNING LIVE FACILITY??

He: Yes, so I can check, that the telegramms I send are right. If I get back the right acknowledgement, the telegramm structure is right and my code is working.

Me: You know, that is not the porpose of unit tests? You know, that these test should run in any environment?

He: But they are proving, that my code is working. Everytime I change something I connect to the customer and let the tests run.

Me: ...

Despite the help of some other developers we could not convince him that this was not good and he should remove them. So now this package is used in 2 new projects and this test suit is still failing, everytime you execute all unit tests.

We developed an application and deployed on production (but not launched)
And business team already created lot of garbage or dummy data. Reporting systems are huge pile of bars, stats and shit.

Now, has to destroy and clean production.

Already advise them to do experiments or testing dev or staging.

Damn. First time in my career experienced this. Has to delete production.

Somebody is testing in production.

Testing in a production environment is like closing a door in a to kill a snake and electricity goes off 📴

I used to work on a production management team, whose job was, among other things, safeguarding access to production. Dev teams would send us requests all the time to, "run a quick SQL script."

Invariably, the SQL would include, "SELECT * FROM db_config."

We would push the tickets back, and the devs would call us, enraged. I learned pretty quickly that they didn't have any real interest in dev, test, or staging environments, and just wanted to do everything in prod, and see if it works.

But they would give up their protests pretty fast when I offered to let them speak to a manager when they were upset I wouldn't run their SQL.

"I always test in production"

This sentence by a colleague made me a murderer (almost).
Jesus...

Thank you Tata sky

Come on... Whatever we say, these guys won't just stop testing in production

-not commenting
-leaving console logs behind in production
-not testing if it works in IE
-using root too much
-using if instead of switch
-never staying consistent with naming conventions
-starting projects and never finishing

My co-worker, still studying but working as a "senior dev", just decided that we don't need a test/staging environment anymore. We just "validate" (we also don't use the word "test" anymore) newly created features in production.

Makes absolutely sense...

Thank god I have a new job from february on!

Borrowed from Reddit and Twitter:
Everybody has a testing environment. Some people are lucky enough enough to have a totally separate environment to run production in.

Woops! I was debugging a particularly snarky issue in Production the other day. This morning I realized about 60 minutes into my new coding I hadn't changed my profile back. I was testing form submissions on a live customer's site.

Testing in production... (German public TV)

When your college approaches you with an update and want to test it in production....

Yet another day at work:

My job is to write test libraries for web services and test others code. Yes I know to code, and have a niche in software testing.
Sometimes developers (whose code I find bugs in) get so defensive and scream in emails and meetings if I point out an issue in their code.
Today, when I pointed a bug in his repo, a developer questioned me in an email asking if I even understood his code, and as a tester I shouldn’t look at his code and only blackbox test it.

I wish I can educate the defensive developer that sometimes, it’s okay to make mistakes and be corrected. That’s how we deliver services that doesn’t suck in production.

I know politics is not allowed here, but I have to share this gem with you.

One day before the election for the European parliament the website of the German city Bochum showed a wrong bar diagram with false results of the election for a few seconds.

Everyone was loling. But I was like WTF? They were testing in production. And also they included data were the party AFD had about 50% of the votes. Are they retarded or so?

Ever had a day that felt like you're shoveling snow from the driveway? In a blizzard? With thunderstorms & falling unicorns? Like you shovel away one m² & turn around and no footprints visible anymore? And snow built up to your neck?

Today my work day was like that.. xcept shit..shit instead of pretty & puffy snow!!

Working on things a & b, trying to not mess either one up, then comes shit x, coworker was updating production.. ofc something went wrong.. again not testing after the update..then me 'to da rescue'.. :/ hardly patch things up, so it works..in a way.. feature c still missing due to needed workarounds.. going back to a and b.. got disrupted by the same coworker who is nver listening, but always asking too much..

And when I think I finally have the b thing figured out a f-ing blocker from one of our biggest clients.. The whole system is unresponsive.. Needles to say, same guy in support for two companies (their end), so they filed the jira blocker with the wrong customer that doesn't have a SLA so no urgent emails..and then the phone calls.. and then the hell broke loose.. checking what is happening.. After frantic calls from our dba to anyone who even knows that our customer exists if they were doing sth on the db.. noup, not a single one was fucking with the prod db.. The hell! Materialised view created 10 mins ago that blocked everything..set to recreate every 10 minutes..with a query that I am guessing couldn't even select all that data in under 15.. dafaaaq?! Then we kill it..and again it is there.. We found out that customers dbas were testing something on live environment, oblivious that they mamaged to block the entire db..
FML, I'm going pokemon hunting.. :/ codename for ingress n beer..

Testing an attendace machine API one by one so i know what does what.

And there’s an API for wiping all the attendance data stored in the machine.

I didn’t realize it until i push the damn button.

The attendance machine just become fresh like new 😱😨😰

Shit.

A testing session just become an extreme sport.

Thankfully the IT guy has a backup but just up to last month.

Well, it’s better than nothing isn’t it?

He just tell his boss that the machine was run out of memory and the attendance data for the current month were not saved.

And he ask him to buy me some machine for testing.

Yes, i was living on the edge by testing in the production machine.

Things that seem "simple" but end up taking a long ass time to actually deploy into production:

1. Using a new payment processor:

"It's just a simple API, I'll be done in 2 hours"

LOL sure it is, but testing orders and setting up a sandbox or making sure you have credentials right, and then switching from test to life and retesting, and then... fuck

2. Making changes to admin stats.

"'I just have to add this column and remove that one... maybe like a couple of hours"

YOU WISH

3. Anything Javascript

"Hah, what, that's like a button, np"

125 minutes later...

console.log('before foo');
console.log(this.foo)
etc..

I guess someone has been testing in production

Ok, here goes...

I was once asked to evaluate upgrade options for an online shop platform.

The thing was built on Zend 1, but that's not the problem.

The geniuses that worked on it before didn't have any clue about best practices, framework convention, modular thinking, testing, security issues...nothing!

There were some instances when querying was done using a rudimentary excuse for a model layer. Other times, they would just use raw queries and just ignore the previous method. Sometimes the database calls were made in strange function calls inside randomly loaded PHP files from different folders from all over the place. Sometimes they used JOINs to get the data from multiple tables, sometimes they would do a bunch of single table queries and just loop every data set to format it using multiple for loops.

And, best of all, there were some parts of the app that would just ignore any ideea of frameworks, conventions and all that and would be just a huge PHP file full of spagetti code just spalshed around, sometimes with no apparent logic to it. Queries, processing, HTML...everything crammed in one file...

The most amazing thing was that this code base somehow managed to function in production for more than 5 years and people actualy used it...

Imagine the reaction I got from the client the moment I said we should burn it to the ground and rebuild the whole thing from scratch...

Good thing my boss trusted me and backed me up (he is a great guy by the way) and we never had to go along with that Frankenstein monster...

Me: *asks boss for the id of his test store so I can apply experimental schema changes to test out a new dashboard app*

Boss: *gives me production store id and doesn't say anything*

Fate: … "You got lucky this time."

This is the CEO of the company btw. Startups. *sigh*

TL;DR Dear boss, firstly, you always get someone to review anything important done by a fucking intern.
Secondly, you do not give access to your fucking client's production server to an intern.
Thirdly, you don't ask your fucking intern to test the intern's work that has not been reviewed by anyone directly on your client's fucking production server.

Last week, the boss and one of the lead devs (the only guy with some serious knowledge about systems and networking) decided to give me (an intern who barely has any work experience) the task of fixing or finding an alternate solution to allowing their support team access to their client machines. Currently they used a reverse SSH tunnel and an intermediary VH but for some reason, that was very unreliable in terms of availability. I suggested using OpenVPN and explained how it would work. Seemed to be a far better idea and they accepted. After several days of working through documentations and guides and everything, I figured out how OpenVPN works and managed to deploy a TEST server and successfully test remote access using two VMs. On seeing my tests, the boss told me that he wanted to test it on the client network. I agreed. Today he comes to me and he tells me to prepare testing for tomorrow and that the client technician is going to give me access to one of their boxes. And then he adds, "It's a working prod server. We'll see if we can make it work on that" and left. I gaped at him for a while and asked another dev guy in the room if what I heard was right. He confirmed. Turns out, the lead dev and the boss's son (who also works here) had had a huge argument since morning on the same issue and finally the dev guy had washed it off his hands and declared that if anything goes wrong from testing it on production, it's entirely the boss's own fault. That's when the boss stepped in and approached me. I ran back to his office and began to explain why prod servers don't top the list of things you can fuck around with. But he simply silenced me saying, "What can go wrong?" and added, "You shouldn't stay still. You should keep moving". Okay, like firstly what the fuck and secondly, what the fuck?.
Even though OpenVPN client is not the scariest thing to install, tomorrow's going to be fun.

Does anyone know of any apps/companies that are using .Net core in production?

I’ve started a project at work which consists of a webapi written in dotnet core, a react spa for the front end and xunit for testing.

I’m just curious as there are loss of sites about things written in Rails, or django, but almost nothing about dotnet core.

So, awesome clip to use for testing... Problem is, I found this in the codebase for a production app.

*Facepalm*

Overengineering. Finding the right point between overdesign and no design at all. That's where fancy languages and unusual patterns being hit by real world problems, and you need to deal with all that utter mess you created being architecture astronaut. Isn't that funny how you realize that another fancy tool is fundamentally incompatible with the task you need to solve, and you realize it after a month of writing workarounds and hacks.

But on the other hand, duct tape slacking becomes a mess even quicker.

Not being able to promote projects. You may code the shit out of side project and still get zero response, absolutely no impact. That's why your side projects often becomes abandoned.

Oversleeping. You thought tomorrow was productive day, but you wake up oversleeped, your head aches, your mind is not clear and you be like "fuck that, I'm staying in bed watching memes all day". But there's job that has to be done, and that bothers you.

Writing tests. Oh, words can't describe how much I hate writing tests, any kind of. I tried testing so many times in high school, at university, even at production, but it seems like my mind is just doesn't accept it. I know that testing is fundamentally important, but my mind collapses every time I try to write a single fucking test, resulting in terrible headache. I don't know why it's like that, but it is, and I better repl the shit out of pure function than write fucking tests.

testing in production

Does anyone work on a team with multiple stacks?

For example we have batch jobs in Java but also have a JS front-end and APIs.

How do you divide the developers and the work across these projects?

Currently everyone does everything but I feel like this is inefficient and hard to develop expertise. And different people or even the same person will make the same mistakes over and over again because they don't know how to do X or they forget or overlook some quirk. When I switched Beck to JS took me like a week to get a Promises nailed down again. And this morning someone else had a production bug and couldn't figure it out. But when I looked at the code I could pretty much see where an issue could be (uncaught exception in a promise)

Also the testing frameworks are very different and there's a lot of infrastructure technical debt, things that really should've been done a long time or fixed but no one had the time or expertise to do it or notice it (until it causes a production issue and then everyone is like WTF is happening??!!!!).

I'm not the manager but I always feel that the team needs to be split along the language lines and specific people need to own these projects to review and code changes for all these common newbie errors. And also developer enough expertise to foresee problems before it becomes a production issue.

New twist on an old favorite.

Background:
- TeamA provides a service internal to the company.
- That service is made accessible to a cloud environment, also has a requirement to be made available to machines on the local network so you can develop against it.
- Company is too cheap/stupid to get a s2s vpn to their cloud provider.
- Company also only hosts production in the cloud, so all other dev is done locally, or on production non-similar infra, local dev is podman.
- They accomplish service connectivity by use of an inordinately complicated edge gateway/router/firewall/message translator/ouija board/julienne fry maker, also controlled by said service team.

Scenario:
Me: "Hey, we're cool with signing requests using an x509 cert. That said, doing so requires different code than connecting to an unsecured endpoint. Please make this service accessible to developer machines and lower environments on the internal network so we can, you know, develop."
TeamA: "The service should be accessible to [cloud ip range]"
Me: "Yes, that's a production range. We need to be able to test the signing code without testing in production"
TeamA: "Can you mock the data?"
Me: "The code we are testing is relating to auth, not business logic"
TeamA: "What are you trying to do?"
Me: "We are trying to test the code that uses the x509 you provide to connect to the service"
TeamA: "Can you deploy to the cloud"
Me: "Again, no, the cloud is only production per policy, all lower environments are in the local data center"
TeamA: "can you try connecting to the gateway?"
Me: "Yes, we have, it's not accessible, it only has public DNS, and only allows [cloud ip range]"
TeamA: "it work when we try it"
Me: "Can you please supply repro steps so we can adjust our process"
TeamA: "Yes, log into the gateway and try issuing the call from there"
Me: (╯°□°)╯︵ ┻━┻

tl;dr: Works on my server

"Real devs test in production", in practice.

This was actually the second such notification I received. Not sure if this is standard for mobile app testing...

A story about a helpless intern : On a fine day an intern was assigned a feature to develope. He worked his ass off, completed it, submitted it for testing, the build was approved by the tester and got released the next day. Fast forward a few days, the feature is failing in few cases at the production.Everbody, starts pointing fingers at the intern.
The intern wonders...How the fuck am I gonna know this fucking use case, do they really expect me to to build a full proof feature without telling me about all the possible senarios...And how the fuck did the tester approved this...? I mean, now that I know this senario, it seems pretty obvious that it should have been tested...!
Note : This also happened to another developer who recently joined...The PM failed to properly communicate all the requirements and the fucking lazy ass tester did not consider all the possible senarios. And the script failed in the production...!
Note : It's 4 fucking AM and the intern still can't sleep...

So glad to be staying a new job next week. Today a junior colleague asked me what the best way to test something would be as it won't work locally. Knowing this has a good chance of taking down the server, I suggest he sounds up a server on his AWS account. My manager comes in, oh no I don't want him doing it on AWS use the production server instead. By the time stuff States hitting the fan I will be gone.

Most intense day for me was at the very start of my career. Internship... went with product manager to client's office while PM installed new test version of our product for on-site integration testing. Shortly after deploy, client manager came over to ask why production had gone down...

Turns out that manually typing DB name as part of deployment script is not, erm, risk free. PM entered production DB name and took out a very busy call centre for a few hours. Agents in tears, customers raging on phones, etc! After we restored and got everything back up and running, he reached me the keyboard and said "You're doing it this time."

My attempt was problem free, thankfully. Earned many brownie points that day.

Jesus fuckin' Christ. I own a webshop together with someone else. This guy is so fuckin' stupid. Yesterday I've deployed a release to our acceptance environment. I talked with him extensively about it. This morning I texted him to check out all the new stuff.

5 minutes later he texts.me back: I would suggest changing option x. Uuh... what option x? We don't have that any more.

Dude! What the fuck! We talked extensively about acceptance testing so you know it is in our acceptance environment, not production, asshole.

And then again, he asks for the link to the acceptance, which I gave him twice already.

Are you really that stupid??

Since I have seen a lot of people uploading this kind of stuff lately, here is Xiaomi's test in production, back in 2017 November...

Testing in production eh?

TL;DR: When picking vendors to outsource work to, vet them really well.

Backstory:
Got a large redesign project that involves rebuilding a website's main navigation (accessibility reasons).
Project is too big just for our dev team to handle with our workload so we got to bring a 3rd party vendor to help us. We do this often so no big deal.
But, this time the twist was Senior Management already had retained hours with a dev shop so they want us to use them for project. Okay...

It begins:
Have our scope / discovery meeting about the changes and our expected DevOps workflow.

Devs work Local and push changes to our Github, that kicks off the build and we test on Dev, then it goes to Staging for more testing & PM review. Once ready we can push to prod, or whenever needed. All is agreed, everyone was happy.

Emailed the vendors' project manager to ask for their devs Github accounts so we can add them to the project. Got no reply for 3 days.

4th day, I get back "Who sets up the Github accounts?"

fuck me. they've never used Github before but in our scope meeting 4 days ago you said Github was fine...??

Whatever, fuck it. I'll make the accounts and add them.

Added 4 devs to the repo and setup new branch. 40min later get an email that they can't setup dev environment now, the dev doesn't know how to setup our CMS locally, "not working for some reason."

So, they ask for permission to develop on our STAGING server.. "because it's already setup"... they want to actively dev on our staging where we get PM/Senior Management approvals?

We have dev, staging, production instances and you want to dev in staging, not dev?... nay nay good sir.

This is whom senior management wants us to use, already paid for via retainer no less. They are a major dev shop and they're useless...

😢😭

Cant wait for today's progress checkup meeting. 😐😐

/rant

The most crazy issue I've fixed was caused by a TCP behavior which I didn't know, called the "half-closed connection".
There was a third-party application installed on a production server which called a LDAP server for retrieving users information. During the day we had several users using the application and all worked fine. During the night, when the application was not accessed, something happened and the first call to the application in the morning was stuck for about 5 minutes before returning a response. I tried to reproduce the issue in a testing environment without success. Then I discovered that the application and the LDAP server were located on two different networks, with a firewall between them. And firewalls sometimes drop old connections. For this reason network applications usually implement a keep-alive mechanism. Well, the default LDAP Java libraries don't set the keep-alive on their connections. So, I found a library called "libdontdie", which force the keep-alive on the connections. I installed the library on the server, loaded it at the startup and the weird stuck behavior in the morning disappeared.

My first rant. Woohoo!

Honestly I do the whole shebang ussualy depending on what the needs are from network to servers to coding because for some reason nobody has any technical experience where I work.
I just started app development for a gamedev startup and I am in sheer awe of the amount of transpiling/compiling etc that needs to be done for an multiplatform app for iOS and android with js(x)/typescript, html, css.

I remember when I could just write some spaghetti code to make it working by following a couple of tutorials. Then refractoring and testing it for a couple of hours and be done with it. push it into production.

Now I am lost having to learn OOP, functional programming, reactjs, react native, express, webpack, mongodb, babel, and the list goes on and on...
Why not just make a new backend that does all of that in another language which supports all of that.

I have no formal education in programming/coding and the last time I learned JS it was just some if else, switches and simple dom manipulation.

I just want to get to coding a freakin' game but I have to learn JSX for the front and typescript on the backend.

I am this close to going back to ye ol' lamp stack and quitting this job. 😥

I'm testing a shitforbrains code in production because I don't have any other options. He went on vacation and said that the code should work, and if it doesn't it is my problem.

I am contemplating taking a shit under his desk and working from home. Then he would have to deal with my shit aswell. Literally.
The tables have turned my friend...

Someone is testing in production on a Friday again 😑

Someone is testing in production 😂

* Developing a new "My pages" NBV offer/order solution for customer
_Thursday
Customer: Are we ready for testing?
Me: Almost, we need to receive the SSL cert and then do a full test run to see if your sales services get the orders correctly. At this point, all orders made via this flow are tagged so they will not be sent to the Sales services. We also still need to implement the tracking to see who has been exposed to what in My Pages.
Customer: Ok, great!

_Friday
Customer: My web team needs these customers to have fake offers on them, to validate the layout and content
Me: Ok, my colleague can fix this by Tuesday - he has all the other things with higher prio from you to complete first
Customer: Ok! Good!

_Sunday
Me: Good news, got the SSL cert installed and have verified the flow from my side. Now you need to verify the full flow from your side.
Customer: Ok! Great! Will do.

_Monday
*quiet*

_Tuesday
Customer: Can you see how things are going? Any good news?
Me: ???
*looks into the system*
WTF!?!
- Have you set this into production on your side? We are not finished with the implementation on our side!
Customer: Oh, sorry - well, it looked fine when we tested with the test links you sent (3 weeks ago)
Me: But did you make a complete test run, and make sure that Sales services got the order?
Customer: Oh, no they didn't receive anything - but we thought that was just because of it being a test link
Me: Seriously - you didn't read what i wrote last Thursday?
Customer: ...
Me: Ok, so what happens if something goes wrong - who get's blamed?
Customer: ...
Me: FML!!!

Ya know...just testing in production 😶

A loooong time ago...

I've started my first serious job as a developer. I was young yet enthusiastic as well as a kind of a greenhorn. First time working in a business, working with a team full of experienced full-lowered ultra-seniors which were waiting to teach me the everything about software engineering.

Kind of.

Beside one senior which was the team lead as well there were two other devs. One of them was very experienced and a pretty nice guy, I could ask him anytime and he would sit down with me a give me advice. I've learned a lot of him.

Fast forward three months (yes, three months).

I was not that full kind of greenhorn anymore and people started to give me serious tasks. I had some experience in doing deployments and stuff from my other job as a sysadmin before so I was soon known as the "deployment guy", setting up deployments for our projects the right way and monitoring as well as executing them. But as it should be in every good team we had to share our knowledge so one can be on vacation or something and another colleague was able to do the task as well.

So now we come to the other teammate. The one I was not talking about till now. And that for a reason.

He was very nice too and had a couple of years as a dev on his CV, but...yeah...like...

When I switched some production systems to Linux he had to learn something about Linux. Everytime he encountered an error message he turned around and asked me how to fix it. Even. For. The. Simplest. Error. He. Could. Google. Up.

I mean okay, when one's new to a system it's not that easy, but when you have an error message which prints out THE SOLUTION FOR THE ERROR and he asks me how to fix it...excuse me?

This happened over 30 times.
A. Week.

Later on I had to introduce him to the deployment workflow for a project, so he could eventually deploy the staging environment and the production environment by hisself.

I introduced him. Not for 10 minutes. I explained him the whole workflow and the very main techniques and tools used for like two hours. Every then and when I stopped and asked him if he had any questions. He had'nt! Wonderful!

Haha. Oh no.

So he had to do his first production deployment. I sat by his side to monitor everything. He did well. One or two questions but he did well.

The same when he did his second prod deploy. Everythings fine.

And then. It. Frikkin. Begins.

I was working on the project, did some changes to the code. Okay, deploy it to dev, time for testing.

Hm.

Error checking out git. Okay, awkward. Got to investigate...

On the dev server were some files changed. Strange. The repo was all up to date. But these changes seemed newer because they were fixing at least one bug I was working on.

This doubles the strangeness.

I want over to my colleague's desk.

I asked him about any recent changes to the codebase.

"Yeah, there was a bug you were working on right? But the ticket was open like two days so I thought I'll fix it"

What the Heck dude, this bug was not critical at all and I had other tasks which were more important. Okay, but what about the changed files?

"Oh yeah, I could not remember the exact deployment steps (hint from the author: I wrote them down into our internal Wiki, he wrote them done by hisself when introducing him and after all it's two frikkin commands), so I uploaded them via FTP"

"Uhm... that's not how we do it buddy. We have to follow the procedure to avoid..."

"The boss said it was fine so I uploaded the changes directly to the production servers. It's so much easier via FTP and not this deployment crap, sorry to say that"

You. Did. What?

I could not resist and asked the boss about this. But this had not Effect at all, was the long-time best-buddy-schmuddy-friend of the boss colleague's father.

So in the end I sat there reverting, committing and deploying.

Yep

It's soooo much harder this deployment crap.

Years later, a long time after I quit the job and moved to another company, I get to know that the colleague now is responsible for technical project management.

Hm.

Project Management.

Karma's a bitch, right?

How many people do unit testing?. Am always caught my deadline. I just fix errors in production. The client timeline are not feasible

Following my first rant, my boss had the brilliant idea of running the old and the new architecture in parallel. I had advised that it won’t be ideal since the same Scala code was ingesting into 2 different Kinesis streams and one was running an old KCL written in Java where as other was consumed by a Firehose delivery stream(eventually we will be ingesting it into Firehose directly). I had told few manual + automated tests on Code as well as from a functionality of the new architecture and a set of tests for checking the integration of the new Producer code with Consumer.

The statement I got from my boss was “This is the test, we test it on production in parallel”. My boss had a brilliant idea to fucking test the new code on the production directly but running them in parallel without accounting for undefined behaviour it might cause in the current production system. I mean my boss should get a Nobel peace prize for shattering our mental peace.

Anywho, we started the deployment today at 5AM in the morning. I had all the aws services deployed. Was just waiting to deploy the new Collector code which we did at 5AM. Immediately after 5 minutes the system went bonkers, there was fire, blood, demons and I was smoking a cigarette with the biggest “I told you so smile” on my face. I’ve just written an email to my boss and have told him calmly that “Listen motherfucker, 90 percent of the software companies aren’t idiots to focus on testing and quality. We need to start spending time on testing and quality else we’ll again be in the same soup after few weeks again”.waiting for his reply

*Repost of my own accidentally deleted post*
A Short story that i made on an Android component
===============================

Once upon a time there used to be a ViewPager who was not able to load a Fragment UI.
All the ViewPagers in town can properly load the Fragrant UI but this one was little different.

He wanted to be more then just a ViewPager. He used to see an Activity that can load anything. He was inspired from the Activity and wanted to be like the Activity but his destiny made him just a ViewPager.

So he refused to cooperate. He started to protest silently, No log, nothing.

Everyone assumed this ViewPager have a bug in it. but he was planning something really big that will left everyone in shock and awe moment.

He was planning to rise against the evil 😈 developers who continuously making him to load Fragrant UI

He assembled the biggest army of the bugs that humanity ever seen to counter the developers.

He distributed these bugs in all over the developer's code to make them fire from their work.

Even he taught bugs to not caught in QA testing but appear in production randomly.
So they silently started going into production

And then chaos is erupted all around the world, bugs started to surface and interrupted the daily life of humanity.

In this chaos the ViewPager RAISED!
And took over all the base classes.

ViewPager was unaware of few facts. this unnecessary rise in his power made whole system unstable

Without the base classes the system finally collapsed and then ViewPager as well with the system.

This was the end of everything for the ViewPager but he was satisfied as he lived the life he always wanted

THE END

Testing in production? Stupidity?
Yes.

Some one is testing in production... Shall I report this ? (I'm student btw)

Tl;Dr Im the one of the few in my area that sees sftping as the prod service account shouldn't be a deployment process. And the ONLY ONE THAT CARES THAT THIS IS GONNA BREAK A BUNCH OF SHIT AT SOME POINT.

The non tl;dr:

For a whole year I've been trying to convince my area that sshing as the production service account is not the proper way to deploy and/or develop batch code. My area (my team and 3 sister teams) have no concept of using version control for our various Unix components (shell scripts and configuration files) that our CRITICAL for our teams ongoing success. Most develop in a "prodqa like" system and the remainder straight in production. Those that develop straight in prodqa have no "test" deployment so when they ssh files straight to actual production. Our area has no concept of continuous integration and automated build checking. There is no "test cases", no "systems testing" or "regression testing". No gate checks for changing production are enforced. There is a standing "approved" deployment process by the enterprise (my company is Whyyyyyyyyyy bigger than my area ) but no one uses it. In fact idk anyone in my area who knows HOW to deploy using the official deployment method. Yes, there is privileged access management on the service account. Yes the managers gets notified everytime someone accesses the privileged production account. The managers don't see fixing this as a priority. In fact I think I've only talk to ONE other person in my area who truly understands how terrible it is that we have full production change access on a daily basis. Ive brought this up so many times and so many times nothing has been done and I've tried to get it changed yet nothing has happened and I'm just SO FUCKING SICK that no one sees how big of a deal this. I mean, overall I live the area I work in, I love the people, yet this one glaring deficiency causes me so much fucking stress cause it's so fucking simple to fix.
We even have an newer enterprise deployment. Method leveraging a product called "urban code deploy" (ucd) to deploy a git repository. JUST FUCKING GIT WITH THE PROGRAM!!!!..... IT WAS RELEASED FUCKING 12 YEARS AGO......

Please..... Please..... I just want my otherwise normally awesome team to understand the importance and benefits of version control and approved/revertable deployments

While smoke testing in production, I had to delete the sample entity I created to test the released feature, which is not a big deal

Until 20 minutes later, when I realized that I attached a couple of sub entities under it that contained actual live data

I just had the most embarrassing moment in programming... I am writing an administration / client / invoice webapp and I was testing an export function that worked locally, because everything that was being exported was inside the folder.

So I exported the files in test production, but some invoices didn't exist. So when they don't exist, the system creates a new invoice.

Because I was running on the test production (with client data) the system emailed the created invoices to the clients.. now I have to contact some clients and tell them the invoices were sent accidentally.

Testing in production

Testing in production?

Oooh I have quite a few,
My favourite: accidently left a log. Debug("bollocks") in a try catch this made it through testing and does (still) occasionally go into production log files.

Worst: wrote an interceptor for jboss with the intent of checking cache for some lookup data. I picked the wrong one of two similarly named methods and instead queried the database, I effectively wrote a denial of service utility into our app

Testing crap on production in the middle of the day.. flm

When i had finished my work and was about to leave i saw commit in master labeled "Testing on production"... tomorrow will be an interesting day

I am still confused why people treat testing as secondary position? Tester are paid less and they hire lower quality engineers. I think testing is as important as any other phase of development, like design or implementation. . . and yeah we do testing of our own code. The only thing I can do in my case is to see that people who change my code may not break basic functionalities. And again about edge cases, try to handle some other left to be seen in production( those which I could not think of due to lack of time) I take care not to leave edge case but sometime cannot do it. I just hope people realise the worth of testing.

Updating a WordPress plugin in production without testing.

One of the founders of my startup does not write tests....

Me: we have to all write tests. I explain unit tests, integration, etc.

Him: I do... I have a test that checks if the app crashes or not.

Me: that is not what I meant (writes test because he won’t) 🙄

Trying to make a deadline, waiting for more info from an analist so I can implement the science stuff correctly

Today I receive the email with the info I need. Email is cc'ed to the client. And it starts with complaining about me not having implemented the science stuff. The info I need is attached in this email. Arghgh, how am I supposed to do that if I don't have the information I need.

Apart from that she was testing in the production environment. How do you work with people like this.

But hey, I just got my devRant stickers ;)

Debugging a task, that's sending emails to too many customers.
Supervisor: "Never mind, just test in production, there is a dry run flag for the tasks."
Just in case I test locally...
Flags tried:
--dryrun="TRUE" => Error, failed to send mail.
--dryrun=TRUE => Error, failed to send mail.
--dryrun="true" => Not trying to send mail.

If it's THIS PICKY a little more documentation would be nice.
And by a little more I mean: more than the task base class in a giant php monstrosity without phpdocs expecting its code to be self-documenting.

Testing in production 😂

I don't know if my boss just wants me to learn how to use a new internal deployment process or just likes giving me unnecessary low-value work to take up time...

I could and have just copied the program via SFTP and unzip it to set it up....

(This is a testing and does not need to be in production...)

I have better things I could be doing and just want to get this done and closed but ...

I actually learnt this last year but here I go in case someone else steps into this shit.

Being a remote work team, every other colleague of mine had some kind of OS X device but I was working this Ubuntu machine.

Turns out we were testing some Ruby time objects up to a nanosecond precision (I think that's the language defaults since no further specification was given) and all tests were green in everyone's machine except mine. I always had some kind of inconsistency between times.

After not few hours of debugging and beating any hard enough surface with our heads, we discovered this: Ruby's time precision is up to nanoseconds on Linux (but just us on OS X) indeed but when we stored that into PostgreSQL (its time precision is up to microseconds) and retrieved it back it had already got its precision cut down; hence, when compared with a non processed value there was a difference. THIS JUST DOES NOT HAPPEN IN OS X.

We ended up relying on microseconds. You know, the production application runs on Ubuntu too. Fuck this shit.

Hope it helps :)

P.s.: I'm talking about default configs, if anyone knows another workaround to this or why is this the case please share.

The company I worked for had to do deletion runs of customer data (files and database records) every year, mainly for legal reasons. Two months before the next run they found out that the next year would bring multiple times the amount of objects, because a decade ago they had introduced a new solution whose data would be eligible for deletion for the first time.

The existing process was not be able to cope with those amounts of objects and froze to death gobbling up every bit of ram on the testing system. So my task was to rewrite the exising code, optimize api calls and somehow I ended up in multithreading the whole process. It worked and is most probably still in production today. 💨

Out of the frying pan, into the fire:
So in my first job, I thought it's just us operating so crazy: meddling with arcane C/C++ code from the 80's, shooting our code to production without testing, fixing hundred of customers data base entries by hand, letting an intern alter some core component (to have more logging) and directly push it to prod...
Silly me.
I mean I suspected, that maybe it's not only this tiny little company acting wild, that also the bigger companies with all their ISO certified processes, agile blabla, professional tooling whatsoever - will also have their skeleton in the closet,.. like some obscure assembler part buried in the heart of your code base nobody dares to touch...
How Pieter Hintjens asked about the state of the industry and all the fads so bluntly put it:
"It's all bullshit."
But we are humans, so we better jump on the bandwagon if we want to keep our jobs... and somehow try to keep that trashy house of cards from crashing down.

While addressing a Senior Dev's (SD) query from another team.

SD: why is this field mandatory? Can't it be just optional? Any other work around?
Me: Is your code changes already pushed in Devo? In that case, we provide a value which will work since you are not concerned about it.
SD: Yes. It's pushed till production. And, I want to test changes in Prod.
Me: (shared some codes) and explained that this feature for testing is only available in Devo.
SD: I know that. (Shared me a ticket) I want this field to be optional. That's it.
Me: (read the entire ticket. Didn't find anything related their) Told him, I will discuss with team. And meanwhile, for Devo, you can use this value.

Next morning, I accidentally came over some other ticket raised by him only which had the correct doubts regarding request to support this field in production
Now, I don't know why did he share a wrong ticket with me.
And, how will it even help him if that field was even optional.
THAT JUST WONT WORK IN PRODUCTION.

I will discuss with my team and see what can happen.

So I am debugging a connector library for an api that users curl.

I am fighting my ass off with errors and a lack of debug, testing or thought for CI

Take a poke around this set of classes only to find.

Postman token in the opts. and a removal of ssl check. What you straight copied from postman.

Like seriously clean up your fucking code if you are gonna put something out as production ready to your team.

Console.log('fuck');

Crying right now. Tldr, feature demanded, never used. 8 months ago, bosses demand two features. Disable sync of products they choose and set a reserve quantity on products they choose. Code is done and in production, but the only time that choice has ever run is in testing. With over 10,000 products, not one has ever had sync disabled, or a reserve quantity !=0. I need a new job.

I'm so sick of this task.

It's been three weeks now and I have the same problem as last week. It's very frustrating to have my name on this. I could have finished this in a few days if there weren't so many dependencies on other people or if those people would at least care enough to validate their work. I worked on more complex tasks and the most annoying part is knowing how easy it is but still being unable to finish it. I'm tired of attending stand-up meetings and giving the same "update" over and over again.

The team lead who borrowed me (team 2) messaged me yesterday asking what's holding it up. I told him I'm stuck with this thing that I need and that I already reached out to someone to get it so he already knows it's not my fault it's taking so long. Keep in mind that I cannot "just" get it myself, I would if I could, but imagine logging in and then paying right away. What items are you paying for? How much do they cost? It's a chain of events involving different teams.

Last week was spent with a team (team 3) that wouldn't respond to follow-ups. It came to a point where my original team (team 1) and the team who borrowed me (team 2) were all in the same group chat as team 3's lead. All for this one small fucking thing that I need. They kept saying "It's fixed" when it's not and I spent hours taking and sending screenshots to say "It's not there, seeeee?"

Finally, they made the changes last Friday after my work shift. Someone else from the team I'm working with (team 2) validated it, "Yep, it's fixed. The thing is there now." Now someone from team 2 has to make the changes to their API so the thing would appear in the response and then I will call that API to get the thing. While I'd love to volunteer, it's a completely different task assigned to a different developer and since I'm just borrowed by this team, I don't have access to all of their code except this module I'm working on and cannot assess the impact of making the changes. Not to mention that I would have to learn their entire process which I've already proven they are not really sure of themselves.

Why do I need to learn the entire process? Isn't that overkill? Glad you asked. I had an amazing on-boarding during the first week while setting up their shit and got a lot of "You need these first" errors, none of those things are committed to the repository, some are but are in different branches that I couldn't have guessed. Just refer to the previous rant.

So the smartest move is to wait for them to make the changes to the API. I told team 2 that I'm getting an internal server error with the API. They re-deployed and it works but after testing, the thing I need isn't in the response. They said they already made the changes to get the thing. It's not there. Now this guy I'm talking to follows up again with the developer who's working on the API and while all of this is happening, they decided to change the requirements. Team 2 is having production issues and my concerns are of course, not prioritized.

My team lead (team 1, the original) is obviously pissed off by team 3 from last week and now my team mates (team 1) are annoyed at both team 2 and 3 because of their broken applications and processes. Team 2 also made changes in their application without proper communication so the team 1 developers are affected.

I just want all of this to be over so I can return to my original team.

Sometime last year I had an internship at a small company.

Test servers weren't a thing, and after local testing, it would go to production with a backup of the files that we would put back as soon as we notice something was broken or off.

We used symfony and sonata admin was part of the bundle.

One day, boss asks me to show all the items in a table on the admin page instead of 30 rows.

Me being good guy intern say "sure no problem" so after finding the magic number, I set it to 0 instead of 30.

I gave my work reviewed by my supervisor (senior dev there) and he approved it.

I try to upload the file over FTP. No permissions.

Ask the other dev what it's about, his response: "no idea"

So he tries, fails and decides to try SSH.

Somehow, after fiddling for 20 minutes with ssh, we managed to upload the file.
As soon as we did we hear a scream from the boss's office, we refresh the site, and no matter what page we went to, all we saw was white and the logo of the company in the top left corner.

So this time, we fiddled around with ssh to restore the file for 20 minutes.

Finally succeed all goed back to normal.

A little while later, we call a meeting with the bosses and ask to rewrite the website, BAM, we get approval.

We said "two weeks tops", well that lasted 3 months.

In the end bosses are Uber happy with the work and everything ended well.
Also, development speed has multiplied.

After three months of development, my first contribution to the client is going live on their servers in less than 12 hours. And let me say, I shall never again be doing that much programming in one go, because the last week and a half has been a nightmare... Where to begin...

So last Monday, my code passed to our testing servers, for QA to review and give its seal of approval. But the server was acting up and wouldn't let us do much, giving us tons of timeouts and other errors, so we reported it to the sysadmin and had to put off the testing.

Now that's all fine and dandy, but last Wednesday we had to prepare the release for 4 days of regression testing on our staging servers, which meant that by Wednesday night the code had to be greenlight by QA. Tuesday the sysadmin was unable to check the problem on our testing servers, so we had to wait to Wednesday.

Wednesday comes along, I'm patching a couple things I saw, and around lunch time we deploy to the testing servers. I launch our fancy new Postman tests which pass in local, and I get a bunch of errors. Partially my codes fault, partially the testing env manipulating server responses and systems failing.

Fifteen minutes before I leave work on the day we have to leave everything ready to pass to staging, I find another bug, which is not really something I can ignore. My typing skills go to work as I'm hammering line after line of code out, trying to get it finished so we can deploy and test when I get home. Done just in time to catch the bus home...

So I get home. Run the tests. Still a couple failures due to the bug I tried to resolve. We ask for an extension till the following morning, thus delaying our deployment to staging. Eight hours later, at 1AM, after working a full 8 hours before, I push my code and leave it ready for deployment the following morning. Finally, everything works and we can get our code up to staging. Tests had to be modified to accommodate the shitty testing environment, but I'm happy that we're finally done there.

Staging server shits itself for half a day, so we end up doing regression tests a full day late, without a change in date for our upload to production (yay...).

We get to staging, I run my tests, all green, all working, so happy. I keep on working on other stuff, and the day that we were slated to upload to production, my coworkers find that throughout the development (which included a huge migration), code was removed which should not have. Team panics. Everyone is reviewing my commits (over a hundred commits) trying to see what we're missing that is required (especially legal requirements). Upload to production is delayed one day because of this. Ended up being one class missing, and a couple lines of code, which is my bad (but seriously, not bad considering I'm a Junior who was handed this project as his first task at his first job).

I swear to God, from here on out, one feature per branch and merge request. Never again shall I let this happen. I don't even know why it was allowed to happen, it breaks our branch policies. But ohel... I will now personally oppose crap like this too...

Now if you'll excuse me... I'm going to be highly unproductive and rest, because I might start balding otherwise after these weeks...

Before I started working, I used to feel like I depended on documentation and the internet a little too much owing to ultra crappy long term memory. After spending some time at my internship going through code written by "professional developers" several years senior to me and trying to write unit tests for it (surprise: the code was in production without having underwent any sort of testing), I feel like the amount of time I spend online reading usage recommendations, alternates for optimisation, best practices for writing clean and descriptive code and all that is a lot more rewarding. Some bad things help you feel good about yourself.

Just joined a new company and can only describe the merge process as madness.....is it or am I the one that is mad?!

They have the following branches:

UAT#_Development branch
UAT#_Branch (this kicks of a build to a machine named UAT#)

Each developer has a branch with the # being a number 1 to 6 except 5 which has been reserved for UAT_Testing branch.

They are working on a massive monolith (73 projects), it has direct references to projects with no nuget packages. To build the solution requires building other solutions in a particular order, in short a total fucking mess.

Developer workflow:

Branch from master with a feature or hotfix branch

Make commits to said branch and test manually as there are no automated tests

Push the commits to their UAT#_Development branch, this branch isn't recreated each time and may have differences to all the other UAT#_Development branches.

Once happy create a pull request to merge from UAT#_Development to UAT#_Branch you can approve your own pull request, this kicks off a build and pushes it to a server that is named UAT#.

Developer reviews changes on the UAT# server.

QA team create a UAT/year/month/day branch. Then tell developers to merge their UAT#_branch branches in to the previously created branch, this has to be done in order and that is done through a flurry of emails.

Once all merges are in it then gets pushed to a UAT_Testing branch which kicks off a build, again not a single automated test, and is manually tested by the QA team. If happy they create a release branch named Release/year/month/day and push the changes into it.

A pull request from the release branch is then made to pre-live environment where upon merge a build is kicked off. If that passes testing then a pull request to live is created and the code goes out into production.

Ahhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh it's a total mess. I knew when I took on this job it would be a challenge but nothing has prepped me for the scale of the challenge!! My last place it was trunk based development, commit straight to master, build kicks off with automated testing and that just gets pushed through each of the environments, so easy, so simple!

They tell me this all came about because they previously used EntityFramework EDMX models for the database and it caused merge hell.

Sometimes, certain features don't work in my app if it's uploaded to the production track in Google Play.

It works in debug mode.
It works in release mode.
It works in the internal testing track.
...but it won't work in the production track? Y'know, the one that actually matters? Theoretically, there shouldn't really be a difference if the same exact APK was uploaded to the internal testing and production track, but... idk dude.

As a result, I implemented a secret way to test a feature in the production build (it's an app to remotely control OBS Studio): if the first connection you added is named "yayeet_" and you open the disclaimer, it tests the feature. Luckily, I got some of the stuff figured out, but I just thought the way I had to test it in production was dumb.

In my first few months of my first dev job, I written this fragile piece of code in, trigger warning, PHP that sent out email reports to my clients. It was a two men team, and we have no clue about TDD or how to do unit testing for such code. We would just run that piece of code manually do send out dummy emails to ensure things were working.

One day the code broke. I was told by my boss to fix it. Spent the entire day trying to fix but couldn't get anything done. Finally at around 7pm my boss came by and asked why is it I couldn't get it fixed. He helped me troubleshoot and fixed it. And subsequently told me "c'mon man you're better than this."

It turns out that he changed a part of a code that was supposed return an array of strings to an array of objects, adding a second attribute that wasn't even in use.

So what that meant is that he changed a piece of working code, to include a property he didn't need, committed and push to production without even manually testing it. AND TALKED SHIT TO ME.

That was the day I learned git blame and began my journey on TDD.

That shitty moment when you are finally about to release your code, after about one month of developing and testing, and making sure everything is OK, imagining: "Oh we're finally releasing this feature, I have worked so hard on it, it's going to kick some ass!" but surprisingly things get fucked up on production server... I mean seriously? Stupid middleware I killed myself to get to work messed up. Where the hell have you been in staging, you stupid little bug? You happy now? My CTO giving me awkward looks and shit like: "I'm sorry but you have to come fix it, during weekend." The best way to fuck up my mood, today is the last day of week for god's sake!
I hate releasing like this. seriously SAG in this release!

A very long rant.. but I'm looking to share some experiences, maybe a different perspective.. huge changes at the company.

So my company is starting our microservices journey (we have a 359 retail websites at this moment)

First question was: What to build first?

The first thing we had to do was to decide what we wanted to build as our first microservice. We went looking for a microservice that can be used read only, consumers could easily implement without overhauling production software and is isolated from other processes.

We’ve ended up with building a catalog service as our first microservice. That catalog service provides consumers of the microservice information of our catalog and its most essential information about items in the catalog.

By starting with building the catalog service the team could focus on building the microservice without any time pressure. The initial functionalities of the catalog service were being created to replace existing functionality which were working fine.

Because we choose such an isolated functionality we were able to introduce the new catalog service into production step by step. Instead of replacing the search functionality of the webshops using a big-bang approach, we choose A/B split testing to measure our changes and gradually increase the load of the microservice.

Next step: Choosing a datastore

The search engine that was in production when we started this project was making user of Solr. Due to the use of Lucene it was performing very well as a search engine, but from engineering perspective it lacked some functionalities. It came short if you wanted to run it in a cluster environment, configuring it was hard and not user friendly and last but not least, development of Solr seemed to be grinded to a halt.

Elasticsearch started entering the scene as a competitor for Solr and brought interesting features. Still using Lucene, which we were happy with, it was build with clustering in mind and being provided out of the box. Managing Elasticsearch was easy since there are REST APIs for configuration and as a fallback there are YAML configurations available.

We decided to use Elasticsearch since it provides us the strengths and capabilities of Lucene with the added joy of easy configuration, clustering and a lively community driving the project.

Even bigger challenge? Which programming language will we use

The team responsible for developing this first microservice consists out of a group web developers. So when looking for a programming language for the microservice, we went searching for a language close to their hearts and expertise. At that time a typical web developer at least had knowledge of PHP and Javascript.

What we’ve noticed during researching various languages is that almost all actions done by the catalog service will boil down to the following paradigm:

- Execute a HTTP call to fetch some JSON

- Transform JSON to a desired output

- Respond with the transformed JSON

Actions that easily can be done in a parallel and asynchronous manner and mainly consists out of transforming JSON from the source to a desired output. The programming language used for the catalog service should hold strong qualifications for those kind of actions.

Another thing to notice is that some functionalities that will be built using the catalog service will result into a high level of concurrent requests. For example the type-ahead functionality will trigger several requests to the catalog service per usage of a user.

To us, PHP and .NET at that time weren’t sufficient enough to us for building the catalog service based on the requirements we’ve set. Eventually we’ve decided to use Node.js which is better suited for the things we are looking for as described earlier. Node.js provides a non-blocking I/O model and being event driven helps us developing a high performance microservice.

The leap to start programming Node.js is relatively small since it basically is Javascript. A language that is familiar for the developers around that time. While Node.js is displaying some new concepts it is relatively easy for a developer to start using it.

The beauty of microservices and the isolation it provides, is that you can choose the best tool for that particular microservice. Not all microservices will be developed using Node.js and Elasticsearch. All kinds of combinations might arise and this is what makes the microservices architecture so flexible.

Even when Node.js or Elasticsearch turns out to be a bad choice for the catalog service it is relatively easy to switch that choice for magic ‘X’ or component ‘Z’. By focussing on creating a solid API the components that are driving that API don’t matter that much. It should do what you ask of it and when it is lacking you just replace it.

Many more headaches to come later this year ;)

Level 1 support moron dishing out bad instructions from his flowchart.

Wanted me to edit config files for a production setup, which would've killed shipping for all stations, in the middle of our shipping rush.

Fixed the problem while in the escalation queue for level 2. L2 confirms the fix, and bemoans the shit documentation L1 provided.

If its a business class (mission critical) system, hire decent support staff! You might try testing people for reading/listening comprehension, and then paying them a decent wage! This isn't good for my blood pressure...

Critical bug in production? Sorry, can't fix it right now: We've got a build running with 1 hour of build time left, 8 hours of automatic tests, 3 days of manual testing, and a partridge in a pear tree. Your fix can be in the next release.

evaluating whether i should make it more agile and flexible or just get it slapped into production. always waging war in between proper developer testing vs my own uat testing (of which we as devlopers also need to do)

So our software clubs discord bot went rogue and deleted a metric ass ton of messages

This is what I get for testing in production. Fml

An integration test is testing the real db (production twin or similar) or an in-memory db?

Dealing with government bureaucracy today. Prepare for pure anger.

First of all, what fucking dipshit site does testing and maintenance in production without letting users know? Bitch I'm getting an invalid date error when I use your own stupid date selector and I had to waste the office lady's time asking about it because you couldn't be arsed to either test your shit properly or actually take it down if it's broken. Who made that stupid ass decision and why the fuck did nobody question it. Fuck you.

As part of a technical test, I've been asked to test and report bugs in the production application of the company. Is that normal? Or are they making me do free work for them?.
So far I've only seen challenges like this to be done on a custom application for test.

So a third-party service that I implemented is going to production and me and the PO were testing that yesterday, didn't see any orders coming in the service backend.. so we send a mail to them.

This morning they respond with saying that they can't have both the test server running and the production server...

What the hell is this... :/

A dev decided to overwrite the master branch with his code saying its better. That it fixes the major bugs that all of us couldn't solve.

Against my better judgement of firing him, I decided to test it.

Firing up the testing site, we made test databases to use and we went to house.

In the middle of testing, I noticed the test DBs weren't being changed. While everyone was still testing, I looked at the code. It wasn't made to test on any databases, it was specifically designed for the actual production server.

However the damage was done. In a secret dashboard in the code, someone sent instructions to drop the tables, effectively ruining the production server.

We had the dev go to an offline backup site that only went online every 10 minutes a day to make new backups. So we shut down the production server, setup a maintenance page. I get my ass chewed out again, and we were sitting ducks.

I don't think the dev had enough punishment, so I grabbed his laptop and made a full backup of his data, and locked the SSD in a safe.

I downloaded a Windows 98 and put it on a flash drive. And installed it all on his SSD. The dev is now a proud (pirate) owner of Windows 98.

He came back and started balling on his desk. We all looked at him with a pity, but he deserved it.

I'll give him the drive on Monday.
Do you think he learned his lesson?

Google play console is so fucking confusing, I'm trying to get my app to open beta testing stage, I've completed all the steps and done the full roll out, but the status of the app is still pending publication (3 days), wtf does "Pending publication" means do I need to publish the app for production in order for beta testing release to work, or I have to wait for google to approve the beta release for publish?
I'm lost

Testing new server deployment in test env all works, then production it all breaks down. Network didn't allowed the right traffic. Took me whole week to find that out. Until some networking engineer said, you know there is a firewall between those networks?

Working on a security testing tool that's purpose and use has been overstated by the staff engineer and product owner but no team wants to use it and everyone else in security second guesses if it should exist. Oh, also no documentation on how to use it, and you have to figure out how to use it. The tool has been developed and passed down from multiple people who each developed it differently and have all left the company now. No code reviews exactly exist so every functionality has been assumed to work my PO, SM and Staff Engineer, thus questioned when you bring up something that you're not sure works. Constantly redeploying to production at a timezone that's too early for your country but done to proviide minimal damage to the application for customers in case something goes wrong.

Upside is, you're leaving the team in a week and feel sorry for whoever is going to handle this next.

@dfox when deleting a post I am able to report it. I didn't want to try because I learned my lesson about bug testing in production, but thought I'd let you know and if it's fine I'll play with it :D

Handling null like null is to be handled in production. And in current times null is good news (The app is of one of the most important german newspapers)

A young new dev was working on his first ticket, about a bug during parsing of an uploaded excel file. Our issue was that if the file contained an empty line, all remaining rows were ignored. So the task included extending our tests to cover this case. After 2 weeks (!), his merge request comes in. His idea (without ever asking for help) was to parse the whole file (in some cases huge) in the production code a second time, just to count the rows (!!) and save the count in a public static int field, which was verified in his new test.

Working in a new repo. Halp.

One nightmarish project that was doomed from the beginning, had me as the sole developer. I could hardly sleep when we began testing on a separate test system, but with (nearly) all the config stored in shared memory and copied from the production system, I dreaded, half awake, that the production server data base connection was still configured in the test system and that it was shooting all it's test data repeatedly to prod.

Finally drove to company in middle of the night at 4 o'clock. Checked everything was OK, tried to sleep 3 hours before the start of the work day.

This system also had the most hideous memory corruption in some shared memory that was used across several processes and should have been thoroughly protected by a mutex, but somehow, sometimes this crucial map, that was used to speed up the access to all the customer data just contained garbage.

Still haunts me to that day. (Like xkcd's unresolved tension of a non-matching parenthesis - an unresolved bug.

In my initial days as a web developer, i was assigned a task, to implement a cart share functionality in an e commerce company.
I made the functionality and tested on my system.
Result: working good.

Pushed it to beta testing environment.
Resilt: working good.

Pushed to pre production environment.
Result: working good.

Pushed to live site.
Result: 😀 Error in live site..
So a call comes to me from my team lead..
Asks what was the issue...

Me: i dont know either.
....

After 3-4 hrs:
I found the reason.
My system, beta test env, pre prod env are all having latest php version (5.6 i guess)
But the live server had old version of php.

Me: laughed like anything.
I didn't know that these things would matter in such a great level.

Moral of the story:
Be one with the force (server in this case)

IOS 13 for my Ipad broke its battery indicator it's been stuck at 59% for the entire day. Just really hope it doesn't give up the ghost during the day...