Everyone has a test environment. If you're lucky, you'll have a separate production environment too.

Got this notification. Someone is testing on production environment.

Oh, man, I just realized I haven't ranted one of my best stories on here!

So, here goes!

A few years back the company I work for was contacted by an older client regarding a new project.

The guy was now pitching to build the website for the Parliament of another country (not gonna name it, NDAs and stuff), and was planning on outsourcing the development, as he had no team and he was only aiming on taking care of the client service/project management side of the project.

Out of principle (and also to preserve our mental integrity), we have purposely avoided working with government bodies of any kind, in any country, but he was a friend of our CEO and pleaded until we singed on board.

Now, the project itself was way bigger than we expected, as the wanted more of an internal CRM, centralized document archive, event management, internal planning, multiple interfaced, role based access restricted monster of an administration interface, complete with regular user website, also packed with all kind of features, dashboards and so on.

Long story short, a lot bigger than what we were expecting based on the initial brief.

The development period was hell. New features were coming in on a weekly basis. Already implemented functionality was constantly being changed or redefined. No requests we ever made about clarifications and/or materials or information were ever answered on time.

They also somehow bullied the guy that brought us the project into also including the data migration from the old website into the new one we were building and we somehow ended up having to extract meaningful, formatted, sanitized content parsing static HTML files and connecting them to download-able files (almost every page in the old website had files available to download) we needed to also include in a sane way.

Now, don't think the files were simple URL paths we can trace to a folder/file path, oh no!!! The links were some form of hash combination that had to be exploded and tested against some king of database relationship tables that only had hashed indexes relating to other tables, that also only had hashed indexes relating to some other tables that kept a database of the website pages HTML file naming. So what we had to do is identify the files based on a combination of hashed indexes and re-hashed HTML file names that in the end would give us a filename for a real file that we had to then search for inside a list of over 20 folders not related to one another.

So we did this. Created a script that processed the hell out of over 10000 HTML files, database entries and files and re-indexed and re-named all this shit into a meaningful database of sane data and well organized files.

So, with this we were nearing the finish line for the project, which by now exceeded the estimated time by over to times.

We test everything, retest it all again for good measure, pack everything up for deployment, simulate on a staging environment, give the final client access to the staging version, get them to accept that all requirements are met, finish writing the documentation for the codebase, write detailed deployment procedure, include some automation and testing tools also for good measure, recommend production setup, hardware specs, software versions, server side optimization like caching, load balancing and all that we could think would ever be useful, all with more documentation and instructions.

As the project was built on PHP/MySQL (as requested), we recommended a Linux environment for production. Oh, I forgot to tell you that over the development period they kept asking us to also include steps for Windows procedures along with our regular documentation. Was a bit strange, but we added it in there just so we can finish and close the damn project.

So, we send them all the above and go get drunk as fuck in celebration of getting rid of them once and for all...

Next day: hung over, I get to the office, open my laptop and see on new email. I only had the one new mail, so I open it to see what it's about.

Lo and behold! The fuckers over in the other country that called themselves "IT guys", and were the ones making all the changes and additions to our requirements, were not capable enough to follow step by step instructions in order to deploy the project on their servers!!!

[Continues in the comments]

I love starting my day by finding things like this in a production environment...

User: *Clicks on staging environment*

Giant Warning Dialog: YOU ARE CURRENTLY ENTERING THE STAGING ENVIRONMENT

Users: Ok

App: *Completely different colour, I’m talking bright unsightly yellow*

User: Ok

Giant Yellow and Red Flashing Banner at the Top of the Screen: WARNING YOU ARE CURRENTLY USING STAGING, THIS AREA IS FOR TESTING ONLY

User: The production environment sure is acting strange today. It’s a weird colour and I don’t recognize any of the data, it’s all just dummy filler data. I better create a ticket for the dev team to check o—….. no wait I’ll send an email CC everyone including the CEO and sound the alarm production is currently down and filled with giant warning messages.

Manager: OH MY GOD PRODUCTION IS DOWN DID YOU HEAR ABOUT THIS??? WHAT THE FUCK COULD THESE WARNING MESSAGES BE THAT’S ONLY SUPPOSED TO HAPPEN ON STAGING! THE CEO IS BREATHING DOWN MY NECK YOU NEED TO GET THIS FIXED IMMEDIATELY!!!!!!!

Dev: …

Twitter doing test on production... Interesting. At least it worked 😂

I just sent an automated email titled "Gary is a Dinosaur!" to a lot of humourless clients because the ancient application I was testing assumed I was in the production environment. 🙃

Lesson to self: stop using bogus names in testing.

Still, it could've been a lot worse... 😂

GF: Who is this girl?
Me: I need a backup plan if this production environment fails

[This makes me sound really bad at first, please read the whole thing]

Back when I first started freelancing I worked for a client who ran a game server hosting company. My job was to improve their system for updating game servers. This was one of my first clients and I didn't dare to question the fact that he was getting me to work on the production environment as they didn't have a development one setup. I came to regret that decision when out of no where during the first test, files just start deleting. I panicked as one would and tried to stop the webserver it was running on but oh no, he hasn't given me access to any of that. I thought well shit, I might as well see where I fucked up since it was midnight for him and I wasn't able to get a hold of him. I looked at every single line hundreds of times trying to see why it would have started deleting files. I found no cause. Exhausted, (This was 6am by this point) I pretty much passed out. I woke up around 5 hours later with my face on my keyboard (I know you've all done that) only to see a good 30 messages from the client screaming at me. It turns out that during that time every single client's game server had been deleted. Before responding and begging for forgiveness, I decided to take another crack at finding the root of the problem. It wasn't my fault. I had found the cause! It turns out a previous programmer had a script that would run "rm -rf" + (insert file name here) on the old server files, only he had fucked up the line and it would run "rm -rf /". I have never felt more relieved in my life. This script had been disabled by the original programmer but the client had set it to run again so that I could remake the system. Now, I was never told about this specific script as it was for a game they didn't host anymore.
I realise this is getting very long so I'll speed it up a bit.
He didn't want to take the blame and said I added the code and it was all my fault. He told me I could be on live chat support for 3 months at his company or pay $10,000. Out of all of this I had at least made sure to document what I was doing and backup every single file before I touched them which managed to save my ass when it came to him threatening legal action. I showed him my proof which resulted in him trying to guilt trip me to work for him for free as he had lost about 80% of his clients. By this point I had been abused constantly for 4 weeks by this son of a bitch. As I was underage he had said that if we went to court he'd take my parents house and make them live on the street. So how does one respond? A simple "Fuck off you cunt" and a block.

That was over 8 years ago and I haven't heard from him since.

If you've made it this far, congrats, you deserve a cookie!

Seven months ago:
===============
Project Manager: - "Guys, we need to make this brand new ProjectX, here are the specs. What do you think?"

Bored Old Lead: - "I was going to resign this week but you've convinced me, this is a challenge, I never worked with this stack, I'm staying! I'll gladly play with this framework I never used before, it seems to work with this libA I can use here and this libB that I can use here! Such fun!"

Project Manager: - "Awesome! I'm counting on you!"

Six months ago:
====================
Cprn: - "So this part you asked me to implement is tons of work due to the way you're using libA. I really don't think we need it here. We could use a more common approach."

Bored Old Lead: - "No, I already rewrote parts of libB to work with libA, we're keeping it. Just do what's needed."

Cprn: - "Really? Oh, I see. It solves this one issue I'm having at least. Did you push the changes upstream?"

Bored Old Lead: - "No, nobody uses it like that, people don't need it."

Cprn: - "Wait... What? Then why did you even *think* about using those two libs together? It makes no sense."

Bored Old Lead: - "Come on, it's a challenge! Read it! Understand it! It'll make you a better coder!"

Four months ago:
==============
Cprn: - "That version of the framework you used is loosing support next month. We really should update."

Bored Old Lead: - "Yeah, we can't. I changed some core framework mechanics and the patches won't work with the new version. I'd have to rewrite these."

Cprn: - "Please do?"

Bored Old Lead: - "Nah, it's a waste of time! We're not updating!"

Three months ago:
===============
Bored Old Lead: - "The code you committed doesn't pass the tests."

Cprn: - "I just run it on my working copy and everything passes."

Bored Old Lead: - "Doesn't work on mine."

Cprn: - "Let me take a look... Ah! Here you go! You've misused these two options in the framework config for your dev environment."

Bored Old Lead: - "No, I had to hack them like that to work with libB."

Cprn: - "But the new framework version already brings everything we need from libB. We could just update and drop it."

Bored Old Lead: - "No! Can't update, remember?"

Last Friday:
=========
Bored Old Lead: - "You need to rewrite these tests. They work really slow. Two hours to pass all."

Cprn: - "What..? How come? I just run them on revision from this morning and all passed in a minute."

Bored Old Lead: - "Pull the changes and try again. I changed few input dataset objects and then copied results from error messages to assertions to make the tests pass and now it takes two hours. I've narrowed it to those weird tests here."

Cprn: - "Yeah, all of those use ORM. Maybe it's something with the model?"

Bored Old Lead: - "No, all is fine with the model. I was just there rewriting the way framework maps data types to accommodate for my new type that's really just an enum but I made it into a special custom object that needs special custom handling in the ORM. I haven't noticed any issues."

Cprn: - "What!? This makes *zero* sense! You're rewriting vendor code and expect everything to just work!? You're using libs that aren't designed to work together in production code because you wanted a challenge!?? And when everything blows up you're blaming my test code that you're feeding with incorrect dataset!??? See you on Monday, I'm going home! *door slam*"

Today:
=====
Project Manager: - "Cprn, Bored Old Lead left on Friday. He said he can't work with you. You're responsible for Project X now."

Worst dev team failure I've experienced?

One of several.

Around 2012, a team of devs were tasked to convert a ASPX service to WCF that had one responsibility, returning product data (description, price, availability, etc...simple stuff)
No complex searching, just pass the ID, you get the response.

I was the original developer of the ASPX service, which API was an XML request and returned an XML response. The 'powers-that-be' decided anything XML was evil and had to be purged from the planet. If this thought bubble popped up over your head "Wait a sec...doesn't WCF transmit everything via SOAP, which is XML?", yes, but in their minds SOAP wasn't XML. That's not the worst WTF of this story.

The team, 3 developers, 2 DBAs, network administrators, several web developers, worked on the conversion for about 9 months using the Waterfall method (3~5 months was mostly in meetings and very basic prototyping) and using a test-first approach (their own flavor of TDD). The 'go live' day was to occur at 3:00AM and mandatory that nearly the entire department be on-sight (including the department VP) and available to help troubleshoot any system issues.

3:00AM - Teams start their deployments
3:05AM - Thousands and thousands of errors from all kinds of sources (web exceptions, database exceptions, server exceptions, etc), site goes down, teams roll everything back.

3:30AM - The primary developer remembered he made a last minute change to a stored procedure parameter that hadn't been pushed to production, which caused a side-affect across several layers of their stack.

4:00AM - The developer found his bug, but the manager decided it would be better if everyone went home and get a fresh look at the problem at 8:00AM (yes, he expected everyone to be back in the office at 8:00AM).

About a month later, the team scheduled another 3:00AM deployment (VP was present again), confident that introducing mocking into their testing pipeline would fix any database related errors.

3:00AM - Team starts their deployments.
3:30AM - No major errors, things seem to be going well. High fives, cheers..manager tells everyone to head home.
3:35AM - Site crashes, like white page, no response from the servers kind of crash. Resetting IIS on the servers works, but only for around 10 minutes or so.
4:00AM - Team rolls back, manager is clearly pissed at this point, "Nobody is going fucking home until we figure this out!!"
6:00AM - Diagnostics found the WCF client was causing the server to run out of resources, with a mix of clogging up server bandwidth, and a sprinkle of N+1 scaling problem. Manager lets everyone go home, but be back in the office at 8:00AM to develop a plan so this *never* happens again.

About 2 months later, a 'real' development+integration environment (previously, any+all integration tests were on the developer's machine) and the team scheduled a 6:00AM deployment, but at a much, much smaller scale with just the 3 development team members.

Why? Because the manager 'froze' changes to the ASPX service, the web team still needed various enhancements, so they bypassed the service (not using the ASPX service at all) and wrote their own SQL scripts that hit the database directly and utilized AppFabric/Velocity caching to allow the site to scale. There were only a couple client application using the ASPX service that needed to be converted, so deploying at 6:00AM gave everyone a couple of hours before users got into the office. Service deployed, worked like a champ.

A week later the VP schedules a celebration for the successful migration to WCF. Pizza, cake, the works. The 3 team members received awards (and a envelope, which probably equaled some $$$) and the entire team received a custom Benchmade pocket knife to remember this project's success. Myself and several others just stared at each other, not knowing what to say.

Later, my manager pulls several of us into a conference room
Me: "What the hell? This is one of the biggest failures I've been apart of. We got rewarded for thousands and thousands of dollars of wasted time."
<others expressed the same and expletive sediments>
Mgr: "I know..I know...but that's the story we have to stick with. If the company realizes what a fucking mess this is, we could all be fired."
Me: "What?!! All of us?!"
Mgr: "Well, shit rolls downhill. Dept-Mgr-John is ready to fire anyone he felt could make him look bad, which is why I pulled you guys in here. The other sheep out there will go along with anything he says and more than happy to throw you under the bus. Keep your head down until this blows over. Say nothing."

Advice that I give to interns/grads:

In uni/college, you're taught *how* to code something to achieve a goal, and 99% of the time the code will work and do the job in a lab.

But when building things for a real production environment, you learn the 100 ways how *not* to code, from seeing things break left right and centre - basically everything and anything can break your code, whether it is users, the OS, other people's code, legacy code, lag, concurrency, the alignment of the moon to your server...

Every company has a test environment. Some are lucky enough, to also have a production environment

It's 2018 and I am forced to add new features on project that was deployed in 2003 and lastly updated at 2009.

Best part is that PHP version on the production server is [drum roll] 4.3.8. I found out that this version was the latest at July 2003. On top of that, server runs on Windows Server 2003 and the database is oracle (meaning php driver is needed).

I have spent a WHOLE WEEK just trying to recreate that environment in order to start working on the new features.

Sorry for grumping but I had to take it out somewhere.

Just last month we finished a big project for a client. The company had their lawyers make a contract to transfer ownership for the website to them.

One of the requirements was to deliver all code in a word file. And just that. In the whole contract was not a word about transfering the production environment to their servers. Or supplying a working version. The files and database. No just the word file.

As of today still wondering what they where planning to do with a word file with hunderds of pages of code.

Offcourse we deliverd a working version to their servers. But why are there people making decisions about things they understand nothing at all.

My first testing job in the industry. Quite the rollercoaster.

I had found this neat little online service with a community. I signed up an account and participated. I sent in a lot of bug reports. One of the community supervisors sent me a message that most things in FogBugz had my username all over it.

After a year, I got cocky and decided to try SQL injection. In a production environment. What can I say. I was young, not bright, and overly curious. Never malicious, never damaged data or exposed sensitive data or bork services.

I reported it.

Not long after, I got phone calls. I was pretty sure I was getting charged with something.

I was offered a job.

Three months into the job, they asked if I wanted to do Python and work with the automators. I said I don't know what that is but sure.

They hired me a private instructor for a week to learn the basics, then flew me to the other side of the world for two weeks to work directly with the automation team to learn how they do it.

It was a pretty exciting era in my life and my dream job.

Production environment. Where else would I do all my testing?

I hate people... I hate stupid people even more...

A person asked on slack about where download a Programming Language server called Railo. The official site is no longer up because the software was forked and acquired by a new company.

I suggested just to download that fork since it's more stable. They said no, they needed to mimic their production environment. Makes sense, so I left it alone since I couldn't help further.

Another person on slack asked which version of Railo they need. The OPs response was, "Oh whatever version you have."

My response was... "WTF... the latest version of Railo is 4.3 and the fork is 4.5... the only difference is the new name and a couple of security fixes. If you want to mimic production then you need the exact copy.. otherwise, the fork will be your best bet."

Nope.. I need Railo... any version. They say again.

PM (on slack): "we’re about to deploy to production".
Me: "ok"
… I keep on working on a task / remain available for any post deployment issues …
PM (5 minutes later on slack): "deployment broke production! We need to handle this NOW!"

My dev colleague has already called it a day, but I’m still online

Me: "ok I don’t have access to prod, can you describe what’s going on? I can’t reproduce on any other environment"

PM: …

10 minutes go by

Me: "anybody there?"

PM: …

45 minutes later, I realize PM is offline

The following day:
PM: "ok we got prod running again" (turns out it was client’s fault for not updating a config we as devs can’t access)
PM: "but we’re REALLY UPSET! You guys need to be available to intervene for any issues following deployment to production! At least one of you should be available!"

Me: "but, but…" 🫠

Most satisfying bug I've fixed?

Fixed a n+1 issue with a web service retrieving price information. I initially wrote the service, but it was taken over by a couple of 'world class' monday-morning-quarterbacks.
The "Worst code I've ever seen" ... "I can't believe this crap compiles" types that never met anyone else's code that was any good.

After a few months (yes months) and heavy refactoring, the service still returned price information for a product. Pass the service a list of product numbers, service returns the price, availability, etc, that was it.

After a very proud and boisterous deployment, over the next couple of days the service seemed to get slower and slower. DBAs started to complain that the service was causing unusually high wait times, locks, and CPU spikes causing problems for other applications. The usual finger pointing began which ended up with "If PaperTrail had written the service 'correctly' the first time, we wouldn't be in this mess."

Only mattered that I initially wrote the service and no one seemed to care about the two geniuses that took months changing the code.

The dev manager was able to justify a complete re-write of the service using 'proper development methodologies' including budgeting devs, DBAs, server resources, etc..etc. with a projected year+ completion date.

My 'BS Meter' goes off, so I open up the code, maybe 5 minutes...tada...found it. The corresponding stored procedure accepts a list of product numbers and a price type (1=Retail, 2=Dealer, and so on). If you pass 0, the stored procedure returns all the prices.
Code basically looked like this..

public List<Prices> GetPrices(List<Product> products, int priceTypeId)
{
foreach (var item in products)
{
List<int> productIdsParameter = new List<int>();
productIdsParameter.Add(item.ProductID);
List<Price> prices = dataProvider.GetPrices(productIdsParameter, 0);
foreach (var price in prices)
{
if (price.PriceTypeID == priceTypeId)
{
prices = dataProvider.GetPrices(productIdsParameter, price.PriceTypeID);
return prices;
}
* Omitting the other 'WTF?' code to handle the zero price type
}
}
}

I removed the double stored procedure call, updated the method signature to only accept the list of product numbers (which it was before the 'major refactor'), deployed the service to dev (the issue was reproducible in our dev environment) and had the DBA monitor.

The two devs and the manager are grumbling and mocking the changes (they never looked, they assumed I wrote some threading monstrosity) then the DBA walks up..

DBA: "We're good. You hit the database pretty hard and the CPU never moved. Execution plans, locks, all good to go."
<dba starts to walk away>
DevMgr: "No fucking way! Putting that code in a thread wouldn't have fix it"
Me: "Um, I didn't use threads"
Dev1: "You had to. There was no way you made that code run faster without threads"
Dev2: "It runs fine in dev, but there is no way that level of threading will work in production with thousands of requests. I've got unit tests that prove our design is perfect."
Me: "I looked at what the code was doing and removed what it shouldn't be doing. That's it."
DBA: "If the database is happy with the changes, I'm happy. Good job. Get that service deployed tomorrow and lets move on"
Me: "You'll remove the recommendation for a complete re-write of the service?"
DevMgr: "Hell no! The re-write moves forward. This, whatever you did, changes nothing."
DBA: "Hell yes it does!! I've got too much on my plate already to play babysitter with you assholes. I'm done and no one on my team will waste any more time on this. Am I clear?"

Seeing the dev manager face turn red and the other two devs look completely dumbfounded was the most satisfying bug I've fixed.

You can keep telling me how to code it, and I can read about how to code it, but until I've successfully coded it in a production environment 5-10 times it's all just theoretical to me.

I started this new freelance project where I am building some android libraries for the client. Anyways, during meeting I was about to present my results and suddenly backend seemed to be down. I looked into the round "are your servers down?"
Team Lead: "Yeah our cto, also our only backend dev, is developing a new feature."
Me: "Okay but why is production down?",
Team Lead: "Ah dont worry we always test on production. We have a pretty solid hardware, we will even upgrade it soon!"
Me:"... How about you just separate your stage environments and have a develop environment?"
Client: "see, this is where our strength is. We dont need a develop stage we have very strong hardware and our backend is fully in PHP"

Thanks God I'm a freelancer

I recently joined this big MNC after shutting down my own startup. I was trying to automate their build process properly. They were currently using grunt and I favor gulp, so I offered to replace the build process with gulp and manage it properly.
I was almost done with it in development environment and QA was being done for production.
In the meantime I was trying to fix some random bug in a chrome extension backend. I pushed some minor changes to production which was not going to affect the main site. That was in the afternoon.
This Friday my senior rushed to me. It was like he ran six floors to reach me. He asked, did you push the new build system to production, I refused. He then went to the computer nearby and opened the code.
It was Friday and I was about to leave. But being a good developer, I asked what's the problem. He told me that one complete module is down and the developers responsible for them left for the day already and are unreachable.
I worked on that module multiple times last month, so I offered my help. He agreed and we get to work.
The problem was in the Angular front end. So we immediately knew that the build process is screwed. I accidentally kept the gulp process open for anyone, so I immediately rebuilt using grunt and deployed again, but to no success.
Then I carefully analyzed all the commits to the module to find out that I was the one who pushed the change last. That was the chrome extention. I quickly reverted the changes and deployed and the module was live again. The senior asked, how did you do that? I told the truth.
He was surprised that how come that change affect the complete site too. We identified it after an hour. It was the grunt task which includes all the files from that particular module, including chrome extension in the build process.
He mailed the QA team to put Gulp in increased priority and approved the more structural changes, including more scrutiny before deployment and backup builds.
The module was down for more than 5 hours and we got to know only after the client used it for their own process. I was supposed to be fired for this. But instead everyone appreciated my efforts to fix things.
I guess I am in a good company 😉

This image comes to my mind way to often.

If your application cannot fully function in a development environment, with the excuse that certain functionalities can only be executed in production, the reason is that your system is a huge piece of shit.

I see you’re testing in a production environment. I too like to live dangerously.

One of our newly-joined junior sysadmin left a pre-production server SSH session open. Being the responsible senior (pun intended) to teach them the value of security of production (or near production, for that matter) systems, I typed in sudo rm --recursive --no-preserve-root --force / on the terminal session (I didn't hit the Enter / Return key) and left it there. The person took longer to return and the screen went to sleep. I went back to my desk and took a backup image of the machine just in case the unexpected happened.

On returning from wherever they had gone, the person hits enter / return to wake the system (they didn't even have a password-on-wake policy set up on the machine). The SSH session was stil there, the machine accepted the command and started working. This person didn't even look at the session and just navigated away elsewhere (probably to get back to work on the script they were working on).

Five minutes passes by, I get the first monitoring alert saying the server is not responding. I hoped that this person would be responsible enough to check the monitoring alerts since they had a SSH session on the machine.
Seven minutes : other dependent services on the machine start complaining that the instance is unreachable.

I assign the monitoring alert to the person of the day. They come running to me saying that they can't reach the instance but the instance is listed on the inventory list. I ask them to show me the specific terminal that ran the rm -rf command. They get the beautiful realization of the day. They freak the hell out to the point that they ask me, "Am I fired?". I reply, "You should probably ask your manager".

Lesson learnt the hard-way. I gave them a good understanding on what happened and explained the implications on what would have happened had this exact same scenario happened outside the office giving access to an outsider. I explained about why people in _our_ domain should care about security above all else.

There was a good 30+ minute downtime of the instance before I admitted that I had a backup and restored it (after the whole lecture). It wasn't critical since the environment was not user-facing and didn't have any critical data.

Since then we've been at this together - warning engineers when they leave their machines open and taking security lecture / sessions / workshops for new recruits (anyone who joins engineering).

My bank testing in production environment 😂🤣😂🤣

Hot fix in production environment. This is why earthquakes are so deadly in Italy.

Yep. So the dev teams boss says it's fine to run a production environment on a single Windows instance with the db on that same instance, which they already totally lost once from a reboot after an auto update before I came along tasked with fixing the cluster fuck they created.

This from a man who somehow runs a dev team while using gmail via the web because he can't use an email client, uses email to track tasks but can't because they get lost amongst his 3000+ unread emails, has a screen dirtier than a hookers vag on half priced Tuesday, and got a new laptop but had to get his daughter to set it up and transfer his data because he couldn't.

But ok... you have a degree, You must know what you're doing.
It's ok though, I'll keep covering your incompetent ass while you keep raping the company because no one listens.
Peoples ignorance and arrogance astounds me.

Let me tell you a story.

Our company has a homegrown monitoring solution. Keeps track of our deployments and alerts us when something is broken. Really nice for the most part, except a little issue where we get up to 25 alerts PER DAY that our PRODUCTION ENVIRONMENT IS DOWN. Including weekends.

With this many false positives, we quickly learn to ignore the alerts and miss real incidents.

So we approached this team, remember its our own tool, and told them about the problem. Turns out it is a known issue. And here's the kicker: they aren't planning on fixing it!

It gets better. Rather than fix this glaring issue, their solution is to make ANOTHER ALERT that lets us know the monitoring is misbehaving.

To recap, we can now expect to get up to 25 false positive alerts per day that our production is down, followed immediately by more alerts that the monitor is broken, which means we can ignore the previous alert.

As our PM said when he heard this: fuck that noise. We are escalating the shit out of this!

"Everybody has a testing environment. Some people are lucky enough to have a totally seperate environment to run production in."
-Unknown

Fixing someone else's code who left the job.
Production suddenly not working, cannot debug locally, cannot deploy to a test environment because it does not exists anymore.
There should be a contract clause that developer need to support his project for 2 years after he leaves his job.

The moment when you as an apprentice have to maintain the code of a java web application which was mostly written in 2004 with java 1.4.....
The most terrible part is that the production environment runs still java 1.7 and you think about the fancy java 8 features all the time😐😑😐

me: "Why is the QA guy manually typing JSON into a production environment?"
asshole: "That's not your responsibility."
me: "Why didn't you just migrate data? This is dangerous."
asshole: "You need to go sit down."

So, a few years ago I was working at a small state government department. After we has suffered a major development infrastructure outage (another story), I was so outspoken about what a shitty job the infrastructure vendor was doing, the IT Director put me in charge of managing the environment and the vendor, even though I was actually a software architect.
Anyway, a year later, we get a new project manager, and she decides that she needs to bring in a new team of contract developers because she doesn't trust us incumbents.
They develop a new application, but won't use our test team, insisting that their "BA" can do the testing themselves.
Finally it goes into production.
And crashes on Day 1. And keeps crashing.
Its the infrastructure goes out the cry from her office, do something about it!
I check the logs, can find nothing wrong, just this application keeps crashing.
I and another dev ask for the source code so that we can see if we can help find their bug, but we are told in no uncertain terms that there is no bug, they don't need any help, and we must focus on fixing the hardware issue.
After a couple of days of this, she called a meeting, all the PMs, the whole of the other project team, and me and my mate. And she starts laying into us about how we are letting them all down.
We insist that they have a bug, they insist that they can't have a bug because "it's been tested".
This ends up in a shouting match when my mate lost his cool with her.
So, we went back to our desks, got the exe and the pdb files (yes, they had published debug info to production), and reverse engineered it back to C# source, and then started looking through it.
Around midnight, we spotted the bug.
We took it to them the next morning, and it was like "Oh". When we asked how they could have tested it, they said, ah, well, we didn't actually test that function as we didn't think it would be used much....
What happened after that?
Not a happy ending. Six months later the IT Director retires and she gets shoed in as the new IT Director and then starts a bullying campaign against the two of us until we quit.

My worst devSin was testing in production once because I was too lazy to set up the dev environment locally.
Never will I do that mistake again!

Me: Hey boss, if you ever need someone to get into doing DevOps related tasks for the team, I'd be more than happy to take that on.

Boss: We don't really need any dedicated person to work on that, but if we do in the future, I'll let you know.

Fast forward a few days: I am now unable to deploy bug fixes to our testing environment, now in the cloud, because all access has been blocked for everyone except the two numbskulls who thought it'd be a great idea to move EVERYTHING over (apps, configuration manager, proxies, etc) first.

Oh, and this bug is affecting production.

It's about a guy that knows better.

I was working as a subcontractor on a bigger system. We (subs) were not allowed to deploy code, we had to wait for contractor to deploy.

One day I got an email that my code is bugged and that my feature is not working on production. I checked it on test env, everything was fine. Then I checked if the code I wrote was deployed. It was not.

I send an email explaining that if they deployed my code it would be working. Then I got a response. There was a bug in my code.

Another email. I asked how would they know? Do they have a test on their environment that failed?

No. There is one guy that READ my code and he said it should not work, so he will not deploy it. He was not a programmer, he was a business consultant responsible for the documentation.

His issue was that I used a function that was not in a class. So if the function is not declared it's obvious it will not work. I had to explain to him in another email, that you can use object of another class inside your class and then call a function, that is not in your class. It was the last time this guy blocked my deploy.

TL;DR, I had to explain a non-dev how object composition works in order to have my code deployed. Took four emails.

Well the clown strikes again,

How do u break production and a testing environment in one night?

One full month preping for same thing that revolves around one config file and assured us he was confident,

He wasn't
he managed to fuck it up so bad for the team d brass lost d plot,

I'm not one for condemning people but my God Dante's inferno woulda had an extra ring if he worked with this buck,

The stupidity has shattered my belief in sunshine and rainbows

If you're going to request CRITICAL changes to thousands of records in the database, and approve it through testing which is done on an exact replica of production, then tell me it was done incorrectly after the fact it has been implemented and you didn't actually review the changes made to the data or business logic that you requested then you are an idiot. Our staging environment is there to ensure all the changes are accurate you useless human. Its the data you provided, I didn't just magically pull it from thin air to make yours and my job a pain the ass.

It's almost weekend!

But wait! My colleague just changed some code in the production environment. Whoop! Guess what! It's broken now

Fuck you, stop bothering me. I have to celebrate weekend with non-existing friends.

Half way through a 2 hour deployment, and a fucking test fails.
Yes, it takes 2+hours to run the tests.

Rerun in test environment: pass
Rerun in prod: pass
Rerun changed test in prod: fail..

Why, why you got to hate me for?
I love it when production is the place where config gets changed.

It's 17:55... Did much work that day since I came in earlier than usual, so I could leave in time and do some shopping with the girlfriend.

A colleague comes in to my room, a tad distressed. He had accidentally ran a fixture script on a production environment database (processing a shipload of records per minute), truncating all tables...

Using AWS RDS to rollback the transaction log takes up about 20m. I had to do that about 5 times to estimate the date and time of when the fixture script ran... Since there was no clear point in time...

Finally I get to the best state of the data I could get. I log in remotely run some queries. All is well again... With minor losses in data.

I try to download a dump using pg_dump and apparently my version is mismatched with the server. I add the latest version to aptitudes source list of postgres repo and I am ready to remove and purge the current postgres client and extensions...

sudo apt-get remove post*
Are you sure? (Y/n) *presses enter and enters into a world of pain*

Apparently a lot of system critical applications start with post... T_T

We started a project in January for which I was the sole developer, to automate tedious interaction with a vendor's ticketing system. We have a storage environment with about 400,000 commodity disks attached(for this vendor-- there are other vendors too), in sites around the US and Canada. With a weekly failure rate of about 0.0005%, that means about 200 disks a week need to be replaced.

This work-- hardware investigation through storage appliance frontends, internal ticket creation, external ticket creation, watching the external ticket for updates to include in our internal ticket --was all manual, and for around 200 issues a week, it was done by one guy for two years. He was hopelessly behind. This is all automated now, and this morning, I pushed this automation from dev/test to production.

It feels great to see your work helping people around you.

This is the craziest shit... MY FUCKING SERVER JUST SET ON FIRE!!!

Like seriously its hot news (can't resist the puns), it's actually really bad news and I'm just in shock (it's not everyday you find out your running the hottest stack in the country :-P)... I thought it slow as fuck this morning but the office internet was also on the fritz so I carried on with my life until EVERYTHING went down (completely down - poof gone) and within 2 minutes I had a technician from the data centre telling me that something to do with fans had failed and they caught fire, melted and have become one with the hardware. WTF? The last time I went to the data centre it was so cold I pissed sitting down for 2 days because my dick vanished.

I'm just so fucking torn right now because initially I was absolutely fucking ecstatic - 1 week ago after a year of doomsday bitching about having a single point of failure and me not being a sysadmin only to have them look at me like I'm some kind of techie flat earther I finally got approval to spend around 5x more per month and migrate all our software to containerized micro services.

I'll admit this is a bit worse than I expected but thanks to last week at least I have recent off site images of the drives - because big surprise I have to set this monolithic beast back up (No small feat - its gonna be a long night) on a fresh VPS, I also have to do it on premises or the data will only finish uploading sometime next week.

Pro Tip: If your also pleading for more resources/better production environment only to be stone walled the second you mention there's a cost attached be like me - I gave them an ultimatum, either I deploy the software on a stack that's manageable or they man the fuck up and pay a sys admin (This idea got them really amped up until they checked how much decent sys admins cost).

Now I have very flexible pockets because even if I go rambo the max server costs would only be 15-20% of a sys admins paycheck even though that is 13 x more than our current costs.

Our team really needs some workflow arrangement, and this time it was me who screwed up.
So we have to push an update to the Play Store and the App Store the Friday, the app is well tested on test environment then production environment, we got the ok so I uploaded a build, the app management team then continued the process of publishing..

During the weekend the app was approved and live to almost 500k user that can receive the update.

I got a phone call from the Project Manager at almost midnight, the time was really suspicious so I answered.
- Me: Hello.
- PM: Hi, sorry to call you now but the app is live and we have a problem.
- Me: what kind of problem? Let me check.

So I updated the app on my phone and opened it while I am on call.. I almost had heart attack!! WE PUBLISHED A VERSION POINTING TO THE TEST ENVIRONMENT. Holly shit

- Me: shit call the app management team NOW.

Eventually we removed the app from sale (unpublished it) and we submitted a new version immediately, once it was approved the next day we made the app available again (so for those who didn’t update yet, there will be no update to a faulted version, and no new users landing to a version with test data), I received one or two calls from friends telling me why the app is not on the store (our app is used nationally, so it’s really important).

Thank God there was no big show on twitter or other social media.. but it’s really a good lesson to learn.

I understand this is totally my fault, thankfully I didn’t get fired 😅

"It works on our end", the sentence that made me lose my shit.

I've been working on a project were we're supposed to integrate an API into our system.

When trying to get some user id's (UUID) from said API, we got a type-error in the response (???), so I called their integration support and asked what the fuck they were doing (not really, i was kinda calm at this point).

The answer I got was following:

Integration guy: "Uh, bro, like, I don't even know, it's probably on your end"

Me: "We literally used this endpoint with the same parameters yesterday, and got a result we expected. I noticed you updated your API this morning, did you make any major changes?"

Integration guy: "Yeah we changed the type of user id from string to number"

Me: "So, you changed the type of a UUID (uuid4) from string to number? How did you not think that would be an issue? I can see in your forums that everyone else is having the same issue."

Integration guy: "Nah, it's probably a bug in your code, it works on our end"

Me in my mind: *IT WORKS ON YOUR END?!? IT DOESN'T FUCKING MATTER IF IT WORKS ON YOUR END, FUCKTARD.*

What I actually said: "Uhm, I'm not sure if works on your end either, I'm not even sure how this change made it to production. But hey, thanks I guess, bye."

WHY AM I NOT ABLE TO YELL AT PEOPLE WHEN THEY ARE BEING RETARDED???

But really though, when you're maintaining an API, you shouldn't fucking care if things work on your end in your dev environment. What matters is how it works in production, for the end user/users.

And I know that 99% of cases it's the users fault by entering the wrong parameters or trying to request with wrongly setup auth and what not, but still.

Don't ASSUME nothing's wrong on your end. It's your fucking job to fix the issues.

And guess what? The problem was on their side.
I'm going fucking bald.

Don't you just love it when something works in your development environment, your staging environment and then randomly breaks on production? :^)

Why the hell do you keep commit+push your shit to the master branch! We have develop, feature and hotfix, you *************!

Just because you want to test something in prd environment, you don't mess with the master to build the production image. And you do not even rebase fucking develop branch and keep it out of sync you POS!

Excuse my language, thank you.

Today I build a queue to spread the load of the 300.000 daily caculations. To prevent slow server response time from to many analist calculating at the same time.

First run on the server I managed to get the server load to 120% and get us offline for 30 minutes.

Accepation environment and production are on the same hardware.

Today was not a good day.

I have discovered a fresh hell

Some guy I’ve never met or heard of in the office lobbed a comment at one of my *approved and merged* pull requests. He doesn’t say anything specific, only that my REST urls are not in line with naming convention. That’s all he says, and I’ve already walked the URL consumers through the code and given them the URLS.

I’m really annoyed that this guy won’t just say what he has in mind, but fine whatever this is a professional environment and developers are not known for being a diplomatic people. Let it go and get your work done!

I do some googling and find an obvious change that needs to happen- I implement it, open a new pull request and inform my URL consumers of the change.

This rando still isn’t satisfied and still won’t say what needs to change. I am on round 3 of this wonderful cycle and this guy is acting all fuckin HAUGHTY about it. “Here is a list of conventions I found googling, you should read them even if it takes 4 hours because it will benefit your career”

Sure dog you’re probably right on that one but we are in a professional environment and at this point you are holding up production so you can wave your dick around! Just SAY WHAT YOU MEAN SO WE CAN MAKE THE CHANGES AND GET OUR WORK DONE

This was some time ago. A Legendary bug appeared. It worked in the dev environment, but not in the test and production environment.

It had been a week since I was working on the issue. I couldn't pinpoint the problem. We CANNOT change the code that was already there, so we needed to override the code that was written. As I was going at it, something happened.

---
Manager: "Hey, it's working now. What did you do?"
Me: *Very confused because I know I was nowhere close to finding the real source of the problem* Oh, it is? Let me check.
Also me: *Goes and check on the test and prod environment and indeed, it's already working*
Also me to the power of three: *Contemplates on life, the meaning of it, of why I am here, who's going to throw out the trash later, asking myself whether my buddies and I will be drinking tonight, only to realize that I am still on the phone with my manager*
Me again: "Oh wow, it's working."
Manager: "Great job. What were the changes in the code?"
Me: "All I did was put console logs and pushed the changes to test and prod if they were producing the same log results."
Manager: "So there were no changes whatsoever, is that what you mean?"
Me: "Yep. I've no idea why it just suddenly worked."
Manager: "Well, as long as it's working! Just remove those logs and deploy them again to the test and prod environment and add 'Test and prod fix' to the commit comment."
Me: "But what if the problem comes up again? I mean technically we haven't resolved the issue. The only change I made were like 20 lines of console logs! "
Manager: "It's working, isn't it? If it becomes a problem, we'll work it out later."
---

I did as I was told, and Lo and Behold, the problem never occurred again.

Was the system playing a joke on me? The system probably felt sorry for me and thought, "Look at this poor fucker, having such a hard time on a problem he can't even comprehend. That idiotic programmer had so many sleepless nights and yet still couldn't find the solution. Guess I gotta do my job and fix it for him. I'm the only one doing the work around here. Pathetic Homo sapiens!"

Don't get me wrong, I'm glad that it's over but..

What the fuck happened?

Testing in a production environment is like closing a door in a to kill a snake and electricity goes off 📴

when PM demos app's dev environment instead of production

Since we are using the same password on all our servers (both QA and Production environment) my team somehow decided that it would be easier to copy the private SSH key for to ALL servers and add the public key to the authorized.keys file.

This way we SSH without password and easily add it to new servers, it also means that anyone who gets into one server can get to all of them.

I wasn't a fan of the same password on all servers, but this private key copying is just going against basic security principles.

Do they want rogue connections? Because that's how you get them.

My co-worker, still studying but working as a "senior dev", just decided that we don't need a test/staging environment anymore. We just "validate" (we also don't use the word "test" anymore) newly created features in production.

Makes absolutely sense...

Thank god I have a new job from february on!

Okay. So my dumbass boss took this project that had a steep timeline. I told him straight up, it won't work because we won't make the timeline. If we do this, I will be the one bending over backwards to deliver. I don't like to promise and fail. I got the oh don't worry let's just try. If we don't make it that's fine. Unfortunately that's not how I work. I refuse to deliberately fail. So I say okay and we begin. I suggested open source is the fastest way to deliver bit the fucked up part is, I am the only senior dev in the team. I will be expected to reverse engineer the open source app to connect our own deployment parameters. Use tech I have never used before. Connect frontend and backend. Handle dns bullshit. I have literally been working on Vibes and coffee for the past two weeks because ofcourse I ran into so many issues. Now I have an extension for Monday and I hate to fail. So I am not sleeping or resting just working on a fucking java app I didnt build and I am expected to make it work seemlessly on our production environment. I made some progress. Deployed frontend, deployed backend. Forgot to connect production dB so I decided to go with azure database for mysql driver since we have credits on azure. Now my java app is pissing itself over ssl handshake. I generate my keystore and add it and now java socket just times out. I want to pummel somebody or a punching bag that looks like my boss.

Yet another dev team testing in a production environment

So after 6 months of asking for production API token we've finally received it. It got physically delivered by a courier, passed as a text file on a CD. We didn't have a CD drive. Now we do. Because security. Only it turned out to be encrypted with our old public key so they had to redo the whole process. With our current public key. That they couldn't just download, because security, and demanded it to be passed in the fucking same way first. Luckily our hardware guy anticipated this and the CD drives he got can burn as well. So another two weeks passed and finally we got a visit from the courier again. But wait! The file was signed by two people and the signatures weren't trusted, both fingerprints I had to verify by phone, because security, and one of them was on vacation... until today when they finally called back and I could overwrite that fucking token and push to staging environment before the final push to prod.

Only for some reason I couldn't commit. Because the production token was exactly the same as the fucking test token so there was *nothing to commit!*

BECAUSE FUCKING SECURITY!

Borrowed from Reddit and Twitter:
Everybody has a testing environment. Some people are lucky enough enough to have a totally separate environment to run production in.

The WTF moment when I realized that the main production DB server was configured with **dynamic** private IP. After maintenance upgrade and reboot the rest of environment stopped. When I explained to sys admin what caused the production breakdown hi still did not get that :/

Facebook breaking production environment a lot recently. I've got a bad feeling about whatever they're doing.

It feels like a they're covering up some shit.

Well look at me, high as fuck and drunk past reasoning fixing the production environment. Wish me luck.

You get to work, things have broken down in the night, you have no access to production or even test environment and you have to guess why. You do the same job as somebody in other countries for less pay while everyone else has this laid back approach where the time they actually spend working is negligible. Until the sheer amount of entropy in your organization wears you down and you just become part of the problem.

I don't know how managers are planning deadlines and counting December as a full working month!

Most companies that I worked with, count either half a month or push the deadline until the end of January when the workforce is back but not here.

Our division manager has promised the customer that the production environment will be ready on the first week of January, without even consulting the team or checking the schedule like WTF!
The person responsible for setting the infrastructure was on vacation for 2 weeks and he didn't hand over the access to production or share the progress done.
Fast forward, the manager went to slack and pinged the whole company with full caps message that the production should be done today.

Fun times :/

developer makes a "missed-a-semicolon"-kind of mistake that brings your non-production infrastructure down.

manager goes crazy. rallies the whole team into a meeting to find "whom to hold accountable for this stupid mistake" ( read : whom should I blame? ).

spend 1-hour to investigate the problem. send out another developer to fix the problem.

... continue digging ...
( with every step in the software development lifecycle handbook; the only step missing was to pull the handbook itself out )

finds that the developer followed the development process well ( no hoops jumped ).
the error was missed during the code review because the reviewer didn't actually "review" the code, but reported that they had "reviewed and merged" the code

get asked why we're all spending time trying to fix a problem that occurred in a non-production environment. apparently, now it is about figuring out the root cause so that it doesn't happen in production.

we're ALL now staring at the SAME pull request. now the manager is suddenly more mad because the developer used brackets to indicate the pseudo-path where the change occurred.
"WHY WOULD YOU WASTE 30-SECONDS PUTTING ALL THOSE BRACES? YOU'RE ALREADY ON A BRANCH!"

PS : the reason I didn't quote any of the manager's words until the end was because they were screaming all along, so, I'd have to type in ALL CAPS-case. I'm a CAPS-case-hater by-default ( except for the singular use of "I" ( eye; indicating myself ) )

WTF? I mean, walk your temper off first ( I don't mean literally, right now; for now, consider it a figure of speech. I wish I could ask you to do it literally; but no, I'm not that much of a sadist just yet ). Then come back and decide what you actually want to be pissed about. Then think more; about whether you want to kill everyone else's productivity by rallying the entire team ( OK, I'm exaggerating, it's a small team of 4 people; excluding the manager ) to look at an issue that happened in a non-production environment.
At the end of the week, you're still going to come back and say we're behind schedule because we didn't get any work done.
Well, here's 4 hours of our time consumed away by you.

This manager also has a habit of saying, "getting on X's case". Even if it is a discussion ( and not a debate ). What is that supposed to mean? Did X commit such a grave crime that they need to be condemned to hell?

I miss my old organization where there was a strict no-blame policy. Their strategy was, "OK, we have an issue, let's fix it and move on."
I've gotten involved ( not caused it ) in even bigger issues ( like an almost-data-breach ) and nobody ever pointed a finger at another person.
Even though we all knew who caused the issue. Some even went beyond and defended the person. Like, "Them. No, that's not possible. They won't do such dumb mistakes. They're very thorough with their work."
No one even talked about the person behind their back either ( at least I wasn't involved in any such conversation ). Even later, after the whole issue had settled down. I don't think people brought it up later either ( though it was kind of a hush-hush need-to-know event )

Now I realize the other unsaid-advantage of the no-blame policy. You don't lose 4 hours of your so-called "quarantine productivity". We're already short on productivity. Please don't add anymore. 🙏

I didn’t turn down a dev freelance project when the client decided against going with best practices because the solution I offered was a well-established design pattern but created a need for a financial management change she didn’t like. I stupidly built what she asked for. It worked fine in the 3rd party vendor test environment but failed on production. After hours of analysis of code to ensure no changes happened to my source during test->prod deployment, and the vendor denying they had config differences between them, and the client refusing to pay, all I could do was abandon the project.

So last week I really fucked up

I had this new implementation that was supposedly to be integrating smoothly into the rest of the service. It depended on a serialized model made by a data scientist. I test it in local, in QA environment: no problem.

So, Friday, 4pm, I decide to deploy to production. I check once from the app: the service throw an error. Panic attack, my chief is at my desk, we triy to understand what went wrong. I make calls with cUrls: no problem. Everything seems fine. I recheck from the app again: no problem.
We dedice to let it in prod, as the feature work. I go get some beers with the guys, to celebrate the deploy.

Fast-forward the next morning, 11am, my phone ring: it's a colleague of my chief. "Please check Slack, a client is trying to use the feature, it's broken"

FUUUUUUUUUUUUCK!!!

Panic attack again. I go to the computer, check the errors: two types of errors. One I can fix, the other from a missing package on the machine that the data guy used.

Needless to say, I had a fairly good weekend.

Lessons learned:
- make sure Dev, QA and Prod are exactly the same (use Ansible or Container)
- never deploy on a Friday afternoon if you don't have a quick way to revert

Ever had a day that felt like you're shoveling snow from the driveway? In a blizzard? With thunderstorms & falling unicorns? Like you shovel away one m² & turn around and no footprints visible anymore? And snow built up to your neck?

Today my work day was like that.. xcept shit..shit instead of pretty & puffy snow!!

Working on things a & b, trying to not mess either one up, then comes shit x, coworker was updating production.. ofc something went wrong.. again not testing after the update..then me 'to da rescue'.. :/ hardly patch things up, so it works..in a way.. feature c still missing due to needed workarounds.. going back to a and b.. got disrupted by the same coworker who is nver listening, but always asking too much..

And when I think I finally have the b thing figured out a f-ing blocker from one of our biggest clients.. The whole system is unresponsive.. Needles to say, same guy in support for two companies (their end), so they filed the jira blocker with the wrong customer that doesn't have a SLA so no urgent emails..and then the phone calls.. and then the hell broke loose.. checking what is happening.. After frantic calls from our dba to anyone who even knows that our customer exists if they were doing sth on the db.. noup, not a single one was fucking with the prod db.. The hell! Materialised view created 10 mins ago that blocked everything..set to recreate every 10 minutes..with a query that I am guessing couldn't even select all that data in under 15.. dafaaaq?! Then we kill it..and again it is there.. We found out that customers dbas were testing something on live environment, oblivious that they mamaged to block the entire db..
FML, I'm going pokemon hunting.. :/ codename for ingress n beer..

A service had/has been logging hundreds of errors in the development environment and I reached out to the owning process mgr that the error was occurring and perhaps a good opportunity to log additional data to help troubleshoot the issue if the problem ever made its way to production. He responded saying the error was related to a new feature they weren't going to implement in the backing dev database (TL;DR), and they know it works in production (my spidey sense goes off).
They deployed the changes to production this morning and immediately starting throwing errors (same error I sent)
Mgr messaged me a little while ago "Did you make any changes to the documentation service? We're getting this error .."

50% sure someone misspelled something in a config, but only thing they are logging is 'Unable to parse document'. Nothing that indicates an issue with the service they're using.

Have you ever been interrupted because a marketing workmate had a friend on the phone who needed advice on a WordPress hosting, and wanted your advice right now?

Because I have.

When we had a massive server failure and our production environment was down.

Seriously, what the fuck is wrong with people nowadays.

Day 0: thank you for being an Amazon Customer, your database is about to be upgrade in the near future with or without your consent! Tough titties motherfucker!

Day 16: ok, every upgraded by hand in the test environment, everything seems stable, let's go make preparations for production!

Day 16.5: ssh user@<prod_bastion_ip> --yada --yada
Unable to connect

Oooook, let's try again,
Unable to connect

Day 16.5.1: WHY THE FUCK NOT, the IP is fucking right, the cert is right, the user is right, the..... fucking.... EC2 instance has been......... terminated.....

FML!
---

Why! why can't people leave things alone.

Excuse me while I hit the bourbon 🥃

When you're developing it's very well advised to run your software locally in an environment as much as possible matching the real environment.

So for example, if you're running linux on production then you also run it locally to run your code.

Here's where people need to shut the fuck up:

No, mac is not good for linux development. Not unless portability is already a concern that you have and even then it might be counter productive. So many times when people say this, portability isn't not a concern. What runs on servers is up to them.

If your servers are going to be centos, then you develop with centos. Not with debian, gentoo, ubuntu, maxosx, etc.

Even different linux distros are a headache for portability when it's just to support a few desktops for development so don't think that macosx is going to cut it. It might not be as radical a difference as between windows and linux traditionally is but it's still not good for "linux" development. I don't think people making that statement really know what linux is now how different distributions work.

What you use for your graphical operating system doesn't matter to much but when you run your code then there's a simple solution.

Another thing people need to shut up about. It's not docker, unless you're already in Linux where docker is one of many options such as chroot or lxc.

This question always comes up, how do you developer for linux in windows? No it's not docker it's virtual machine.

It's that simple. You download the ISO for the distro you want and then install it on a VM. What does docker for windows do? It runs a linux VM that runs docker.

This may come as a great shock to developers around the world but it is possible to run linux in a VM and then any linux application your want including docker.

Another option is to shove a box in the corner, install what you need on it, share the file system and have people use that to run their code. It really is that easy.

My brain: you have 30 min until you gotta catch the bus, that’s enough for that css fix you wanted to implement on the website!

My development environment: Hey that looks a lot better and works perfectly fine!

My website in production: Fuck you. Fuck your changes. Fuck your bus.

What do we learn? Precompile assets before pushing to production and don’t push to production if there’s a bus to catch 🙃

Living on the edge!

One or two years ago I managed to deploy a DDL change directly on the production server. As I knew there was a backup job which will run every day at noon and at midnight. So I run my script some minutes after noon. So far so good. But somehow I tested it badly in my test environment and the UI of the application throws error after error now in production.

Well, just revert the db to the latest recovery point with the backup, I thought.

It became clear then after a couple of minutes of searching the backup folder for the db backup that there was no such file. The youngest backup file was 3 years old.

Now what happened: The backup script had a switch "simulate=true" and then simulated a successful backup on each run. Therefore the monitoring system got no alerts for not correctly executing those jobs correctly. Then the monitoring job which should do the backupfolder surveillance stuck with green, because there was a valid backup file inside. But it did not check for a specific creation date.

Now this database is the one we need for doing our daily business and is really crucial. Therefore It was easier to emergencyfix the application than doing a rollback of the db 🙄

Well, not really a data loss story, but close to one.

Developing in your local environment? A few bugs here and there. Deploying it to production in a completely different environment? The whole app is a bug...

Ever have a bug that *only* occurs in your production environment? How do you test potential fixes? 😜

Jesus fuckin' Christ. I own a webshop together with someone else. This guy is so fuckin' stupid. Yesterday I've deployed a release to our acceptance environment. I talked with him extensively about it. This morning I texted him to check out all the new stuff.

5 minutes later he texts.me back: I would suggest changing option x. Uuh... what option x? We don't have that any more.

Dude! What the fuck! We talked extensively about acceptance testing so you know it is in our acceptance environment, not production, asshole.

And then again, he asks for the link to the acceptance, which I gave him twice already.

Are you really that stupid??

Tonight, I gave birth to my first fully functional and deployment-ready dockerized application, "lestadium_web_1"!!
The big baby contains a Laravel showcase website with some React already in production! Big bonus with that, it's connected to a database, and I managed to setup some environment variables so nothing too dangerous is built within the image!

Fuck, that was exhausting but I'm so happy to finally understand how to make my stuff work, how they work and how to find some examples to get inspired from 😍

Today during a follow-up meeting of the grand project I'm workng on...

TL: ... and I want to start working on the production environment and have it ready by next month.
Me: (interrupts) hold up! We are not ready, we have a huge backlog of technical tasks that need to be addressed and we are still not in possession of the very crucial business and functional requirements that you are supposed to provide. The acceptation environment is just set up on infra perspective but does not have anything running yet! The API we depend on is still not ready because you keep adding change tasks to it. We have a mountain of work to do to even get to a first release to integration yet and there is still the estimations on data loads and systems... your dream will not be possible until at least Q2 of 2024.
TL: stop being so negative @neatnerdprime and try to be more customer friendly. I want it by the end of the next month.
Me: remember what I said to you about moving prematurely. Remember I don't take any responsibility if things break because you rush the project. Please, reconsider!
TL: I just want it, please do it

FUCK YOU YOU SORRY EXCUSE OF A PEOPLE PERSON KNOWING JACK SHIT AND JUST LICKING THE MIDDLE MANAGEMENT ASSHOLE TO RECEIVE ATTABOY PETS ON YOUR UGLY ASS BALD HEAD AND CROOKED TEETH. YOU SHOULD FUCKING DIE IN A FURNACE AND LEAVE NO TRACE BEHIND.

Boss: "You hardcoded the redirect uri in the code (Early on during development and forgot about it, because apple OAuth is a piece of shit), but don't worry I fixed it by hardcoding the uri with the production host into the config file where clearly all settings are fetched from the OS Environment variables at runtime. This will surely fix the problem in staging we have, no need to thank me"

So, I'll explain how my day went with just 2 sentences.

1: Difference between crontab -e and crontab -r.
2: The keys 'E' and 'R' are right next to each other!

If you've understood what just happened, your next question should be "Jason, which environment?"

"PRODUCTION!!!"

Well... I can think of several bugs that I found on a previous project, but one of the worst (if not the worst, because the damage scope) it's one bug that only appears for a couple of days at the end of every month.

What happens is the following: this bug occurs in a submodule designed (heh) to control the monthly production according the client requirements (client says "I want 1000 thoot picks", that submodule calculates the daily production requirements in order to full fill the order).

Ideally, that programming need to be done once a week (for the current month), because the quantities are updated by client on the same schedule, and one of the edge cases is that when the current date is >= 16th of the month, the user can start programming the production of the following month.

So, according to this specific case, there's an unidentified, elusive, and nasty bug that only shows up on the two last days of every month, when it doesn't allow to modify/create anything for the following month. I mean, normally, whenever you try to edit/create new data, the application shows either an estimated of the quantities to produce, or the previous saved data. But on those specific days it doesn't show any information at all, disregarding of there's something saved or not.

The worst thing is that such process involves both a very overcomplicated stored procedure, and an overcomplicated functionality on the client side (did I mentioned that it dynamically generates a pseudo-spreadsheet with the procedure dataset? Cell by cell), that absolutely no one really fully understands, and the dude that made those artifacts is no longer available (and by now, I'm not so sure that he even remember what he done there).

One of the worst thing is that at this point, it's easier to handle with that error rather to redesign all of that (not because technical limitations, but for bureaucratic and management issues).

The another worst thing (the most important none) is that this specific bug can create a HUGE mess as it prevents the programming of the production to be done the next day (you know, people tends to procrastinate and start doing things at the very end of the day/week/month)... And considering that the company could lose a huge amount of money by every minute without production, you can guess the damage scope of this single bug.

Anyway, this bug has existed since, I don't know, 2015 (Q4?) and we have tried so many things trying to solve it, but that spaghettis refuse to be understood (specially the stored procedure, as it has dynamically generated queries). During my tenure (that ended last year) I spent a good amount of time (considering what I mentioned on the last rant, about the toxic environment) trying to solve that, just giving up after the first couple of weeks.

Anyway... I'm guessing that this particular bug will survive another 4-ish years, or even outlive the current full development team... But, who knows ¯\_(ツ)_/¯ ?

I'm still studying computer science/programming, I still have one year to do in order to graduate (Master). I am in a work study program so I'm working for a company half of the time, and I'm studying the other half. It is important to mention that I am the only web developer of the company

When I arrived in the company 9 months ago, I was given a Vue project which had been developed by a trainee a few weeks before my arrival and I was asked to correct a few things, it was mostly about css. Then, I was ask to add a few functionalities, nothing really hard to code, and we were supposed to test the solution in a staging environment, and if everything was ok, deploy it to prod.

However, the more I did what I was asked, the more functionalities I had to implement, until I reached a point where I had to modify the API, create new routes, etc. I'm not complaining about that, that's my job and I like it. But the solution was supposed to be ready when I arrived, it was also supposed to be tested and deployed.

The problem is, the person emitting these demands (let's call him guy X) is not from the IT service, it's a future user of the website in the admin side. The demands kept going and going and going because, according to him, the solution was not in a good enough state to be deployed, it missed too many (un)necessary features. It kept going for a few months.

The best is yet to come though : guy X was obviously a superior, and HIS superior started putting pressure on me through mails, saying the app was already supposed to be in production and he was implying that I wasn't working fast enough. Luckily, my IT supervisor was aware of what was going on and knew I obviously wasn't to blame.

In the end, the solution was eagerly deployed in production, didn't go through the staging environment and was opened to the users. Now, guy X receives complaints because none of what I did was tested (it was by me, but I wasn't going to test every single little thing because I didn't have time). Some users couldn't connect or use this or that feature and I am literally drowning in mails, all from guy X, asking me to correct things because users are blocked and it's time consuming for him to do some of the things the website was doing manually.

We are here now just because things have been done in a rush, I'm still working on it and trying to fix prod problems and it's pissing me off because we HAVE a staging environment that was supposed to prevent me from working against the clock.

On a final note, what's funny is that the code I'm modifying, the pre-existing one needs to be refactored because bits and pieces are repeated sometimes 5 times where it should have been externalized and imported from another file. But I don't know when and if I will ever be able to do that.

I could have given more context but it's 4am and I'm kinda tired, sorry if I'm not clear or anything. That's my first rant

The most crazy issue I've fixed was caused by a TCP behavior which I didn't know, called the "half-closed connection".
There was a third-party application installed on a production server which called a LDAP server for retrieving users information. During the day we had several users using the application and all worked fine. During the night, when the application was not accessed, something happened and the first call to the application in the morning was stuck for about 5 minutes before returning a response. I tried to reproduce the issue in a testing environment without success. Then I discovered that the application and the LDAP server were located on two different networks, with a firewall between them. And firewalls sometimes drop old connections. For this reason network applications usually implement a keep-alive mechanism. Well, the default LDAP Java libraries don't set the keep-alive on their connections. So, I found a library called "libdontdie", which force the keep-alive on the connections. I installed the library on the server, loaded it at the startup and the weird stuck behavior in the morning disappeared.

Works in production?

Yes: Copy changes to dev and test.
No: Start humming and walk away from the computer.

A loooong time ago...

I've started my first serious job as a developer. I was young yet enthusiastic as well as a kind of a greenhorn. First time working in a business, working with a team full of experienced full-lowered ultra-seniors which were waiting to teach me the everything about software engineering.

Kind of.

Beside one senior which was the team lead as well there were two other devs. One of them was very experienced and a pretty nice guy, I could ask him anytime and he would sit down with me a give me advice. I've learned a lot of him.

Fast forward three months (yes, three months).

I was not that full kind of greenhorn anymore and people started to give me serious tasks. I had some experience in doing deployments and stuff from my other job as a sysadmin before so I was soon known as the "deployment guy", setting up deployments for our projects the right way and monitoring as well as executing them. But as it should be in every good team we had to share our knowledge so one can be on vacation or something and another colleague was able to do the task as well.

So now we come to the other teammate. The one I was not talking about till now. And that for a reason.

He was very nice too and had a couple of years as a dev on his CV, but...yeah...like...

When I switched some production systems to Linux he had to learn something about Linux. Everytime he encountered an error message he turned around and asked me how to fix it. Even. For. The. Simplest. Error. He. Could. Google. Up.

I mean okay, when one's new to a system it's not that easy, but when you have an error message which prints out THE SOLUTION FOR THE ERROR and he asks me how to fix it...excuse me?

This happened over 30 times.
A. Week.

Later on I had to introduce him to the deployment workflow for a project, so he could eventually deploy the staging environment and the production environment by hisself.

I introduced him. Not for 10 minutes. I explained him the whole workflow and the very main techniques and tools used for like two hours. Every then and when I stopped and asked him if he had any questions. He had'nt! Wonderful!

Haha. Oh no.

So he had to do his first production deployment. I sat by his side to monitor everything. He did well. One or two questions but he did well.

The same when he did his second prod deploy. Everythings fine.

And then. It. Frikkin. Begins.

I was working on the project, did some changes to the code. Okay, deploy it to dev, time for testing.

Hm.

Error checking out git. Okay, awkward. Got to investigate...

On the dev server were some files changed. Strange. The repo was all up to date. But these changes seemed newer because they were fixing at least one bug I was working on.

This doubles the strangeness.

I want over to my colleague's desk.

I asked him about any recent changes to the codebase.

"Yeah, there was a bug you were working on right? But the ticket was open like two days so I thought I'll fix it"

What the Heck dude, this bug was not critical at all and I had other tasks which were more important. Okay, but what about the changed files?

"Oh yeah, I could not remember the exact deployment steps (hint from the author: I wrote them down into our internal Wiki, he wrote them done by hisself when introducing him and after all it's two frikkin commands), so I uploaded them via FTP"

"Uhm... that's not how we do it buddy. We have to follow the procedure to avoid..."

"The boss said it was fine so I uploaded the changes directly to the production servers. It's so much easier via FTP and not this deployment crap, sorry to say that"

You. Did. What?

I could not resist and asked the boss about this. But this had not Effect at all, was the long-time best-buddy-schmuddy-friend of the boss colleague's father.

So in the end I sat there reverting, committing and deploying.

Yep

It's soooo much harder this deployment crap.

Years later, a long time after I quit the job and moved to another company, I get to know that the colleague now is responsible for technical project management.

Hm.

Project Management.

Karma's a bitch, right?

Rant to discuss Domain Driven Design (DDD) architecture.

Did anyone implemented on production?
How scalable and easy to use?
Do you recommend it? if yes in which environment? in which business types?

We called a customer because that on their server a directory is missing which was important for production.

Turned out that they didn't miss a directory because they worked in the development environment of the same customer but in a different location. For the last 3 months.

A newly joined developer (who was supposed to be very senior) comes and asks me how to write a test cos for some reason the person didn't know how to mock.

In Java,
(same for any other implementation which has an interface)
Writes Arraylist list =.....
Instead of List list = Arraylist...

Deployed code (another engineer from another country helped to deploy since this new senior dev didn't have access yet.

But the new senior dev didn't update relevant files in production code which brought down the site for nearly an hour. Mistake aside, the first reaction from this new senior dev is 'WHY DIDN'T THE DEV THAT WAS HELPING DIDN'T DO THE FILE UPDATE?'

This was followed by some other complaints such as our branching stragies are wrong. When in fact the new senior dev made a mistake by just making assumptions on our git branching strategies and we already advised on correct process.

Out of all these, guess this is the best part. The senior dev never tested code locally! Just wrote code, unit test and send to QA and somehow the test passed through. I learnt this when I realised this dev... has not even set up the local environment yet.

I keep saying new but this Senior dev been around like 3 months! This person is in another team within our larger team but shares same code base. I am puzzled how do you not set up your environment for 3 months. Don't you ask for help if you are stuck? I am pretty sure the env is still not setup.

Am I over reacting or is this one disgusting developer who doesn't even qualify for an intern let alone a senior dev? It's so revolting I can't even bring myself to offer help.

I really really hope that no one post this,a friend texted it to me and I wanted to share it because made my day.

Idk where it comes, so feel free if know where this came from to post it:

//FUN PART HERE
# Do not refactor, it is a bad practice. YOLO

# Not understanding why or how something works is always good. YOLO

# Do not ever test your code yourself, just ask. YOLO

# No one is going to read your code, at any point don’t comment. YOLO

# Why do it the easy way when you can reinvent the wheel? Future-proofing is for pussies. YOLO

# Do not read the documentation. YOLO

# Do not waste time with gists. YOLO

# Do not write specs. YOLO also matches to YDD (YOLO DRIVEN DEVELOPMENT)

# Do not use naming conventions. YOLO

# Paying for online tutorials is always better than just searching and reading. YOLO

# You always use production as an environment. YOLO

# Don’t describe what you’re trying to do, just ask random questions on how to do it. YOLO

# Don’t indent. YOLO

# Version control systems are for wussies. YOLO

# Developing on a system similar to the deployment system is for wussies! YOLO

# I don’t always test my code, but when I do, I do it in production. YOLO

# Real men deploy with ftp. YOLO

So YOLO Driven Development isn’t your style? Okay, here are a few more hilarious IT methodologies to get on board with.

*The Pigeon Methodology*
Boss flies in, shits all over everything, then flies away.

*ADD (Asshole Driven Development)*
An old favourite, which outlines any team where the biggest jerk makes all the big decisions. Wisdom, process and logic are not the factory default.

*NDAD (No Developers Allowed in Decisions)*
Methodology Developers of all kinds are strictly forbidden when it comes to decisions regarding entire projects, from back end design to deadlines, because middle and top management know exactly what they want, how it should be done, and how long it will take.

*FDD (Fear Driven Development)*
The analysis paralysis that can slow an entire project down, with developments afraid to make mistakes, break the build, or cause bugs. The source of a developer’s anxiety could be attributed to a failure in sharing information, or by implicating that team members are replaceable.

*CYAE (Cover Your Ass Engineering)*
As Scott Berkun so eloquently put it, the driving force behind most individual efforts is making sure that when the shit hits the fan, you are not to blame.

New twist on an old favorite.

Background:
- TeamA provides a service internal to the company.
- That service is made accessible to a cloud environment, also has a requirement to be made available to machines on the local network so you can develop against it.
- Company is too cheap/stupid to get a s2s vpn to their cloud provider.
- Company also only hosts production in the cloud, so all other dev is done locally, or on production non-similar infra, local dev is podman.
- They accomplish service connectivity by use of an inordinately complicated edge gateway/router/firewall/message translator/ouija board/julienne fry maker, also controlled by said service team.

Scenario:
Me: "Hey, we're cool with signing requests using an x509 cert. That said, doing so requires different code than connecting to an unsecured endpoint. Please make this service accessible to developer machines and lower environments on the internal network so we can, you know, develop."
TeamA: "The service should be accessible to [cloud ip range]"
Me: "Yes, that's a production range. We need to be able to test the signing code without testing in production"
TeamA: "Can you mock the data?"
Me: "The code we are testing is relating to auth, not business logic"
TeamA: "What are you trying to do?"
Me: "We are trying to test the code that uses the x509 you provide to connect to the service"
TeamA: "Can you deploy to the cloud"
Me: "Again, no, the cloud is only production per policy, all lower environments are in the local data center"
TeamA: "can you try connecting to the gateway?"
Me: "Yes, we have, it's not accessible, it only has public DNS, and only allows [cloud ip range]"
TeamA: "it work when we try it"
Me: "Can you please supply repro steps so we can adjust our process"
TeamA: "Yes, log into the gateway and try issuing the call from there"
Me: (╯°□°)╯︵ ┻━┻

tl;dr: Works on my server

TL;DR: When picking vendors to outsource work to, vet them really well.

Backstory:
Got a large redesign project that involves rebuilding a website's main navigation (accessibility reasons).
Project is too big just for our dev team to handle with our workload so we got to bring a 3rd party vendor to help us. We do this often so no big deal.
But, this time the twist was Senior Management already had retained hours with a dev shop so they want us to use them for project. Okay...

It begins:
Have our scope / discovery meeting about the changes and our expected DevOps workflow.

Devs work Local and push changes to our Github, that kicks off the build and we test on Dev, then it goes to Staging for more testing & PM review. Once ready we can push to prod, or whenever needed. All is agreed, everyone was happy.

Emailed the vendors' project manager to ask for their devs Github accounts so we can add them to the project. Got no reply for 3 days.

4th day, I get back "Who sets up the Github accounts?"

fuck me. they've never used Github before but in our scope meeting 4 days ago you said Github was fine...??

Whatever, fuck it. I'll make the accounts and add them.

Added 4 devs to the repo and setup new branch. 40min later get an email that they can't setup dev environment now, the dev doesn't know how to setup our CMS locally, "not working for some reason."

So, they ask for permission to develop on our STAGING server.. "because it's already setup"... they want to actively dev on our staging where we get PM/Senior Management approvals?

We have dev, staging, production instances and you want to dev in staging, not dev?... nay nay good sir.

This is whom senior management wants us to use, already paid for via retainer no less. They are a major dev shop and they're useless...

😢😭

Cant wait for today's progress checkup meeting. 😐😐

/rant

I started fully exploring different aspects of tech in a middle school technology class where the teacher gave me a good grade as long as I did something that could be useful or interesting. I learned how to design webpages by playing with inspect element, and then decided to make my own with Notepad. One of my friends showed me how to use Sublime Text, and I found that I loved programming. Other things I did in there included using two desktops with NIC's wired directly to each other with an old version of Synergy and a VNC server, and at one point, I built a server node out of old dell Optiplex desktops the school had piled in a storage room.

Last year in high school, I took a class on VB.net and made some money afterwards by freelance refreshing legacy spaghetti, and got burned pretty badly by a person offering $25,000 for a major POS to backend CMS integration rewrite. The person told me that I had finished second, and that another dev had gotten the reward, but that he liked my code. A few days later, I was notified through a *cough*very convoluted*cough* system of mine by a trigger that ran once during startup in a production environment and reported the version number as well as a few other bits, and I was able to see that *cough*someone*cough* had been using my code. I stopped programming for at least six months straight because I didn't want to go back.

This year in high school, I'm taking the engineering class I didn't get into last year, and I realized that Autodesk Inventor supports VBA. I got back into programming with a lot of copy-paste and click-once "installers" to get my modelling assignments done faster than my classmates. Last week, one of my friends asked me to help him fix his VB program, which I did, and now I'm hooked again.

I've always been an engineer at heart, but now I'm conflicted with going into I.T., mechanical or robotical engineering, or being a software developer.

A little long, but that's how I got to where I am now. (I still detest those who take advantage of defenseless programmers. There's a special place for them.)

Integrating Google recaptcha into my web service. For some reason it always errors, both on a production and development environment, correct domains configured, and with he simplest setup. I'm fucking lost, documentation assumes it actually works. Similar errors on stack overflow and Google groups either got no answers or have obvious issues.

Fuck this man

My company insists on working in one production environment to save time and every time I try to convince them to set up a work flow with a dev and test environment, they tell me we don't have the time...

Even after I set one up anyway as I'm scared shitless to touch production. They tell me it's faster doing it all in one environment.

They launch an update. Site buggy as hell and doesn't load 90% of the content...

Sigh....

When the CTO/CEO of your "startup" is always AFK and it takes weeks to get anything approved by them (or even secure a meeting with them) and they have almost-exclusive access to production and the admin account for all third party services.

Want to create a new messaging channel? Too bad! What about a new repository for that cool idea you had, or that new microservice you're expected to build. Expect to be blocked for at least a week.

When they also hold themselves solely responsible for security and operations, they've built their own proprietary framework that handles all the authentication, database models and microservice communications.

Speaking of which, there's more than six microservices per developer!

Oh there's a bug or limitation in the framework? Too bad. It's a black box that nobody else in the company can touch. Good luck with the two week lead time on getting anything changed there. Oh and there's no dedicated issue tracker. Have you heard of email?

When the systems and processes in place were designed for "consistency" and "scalability" in mind you can be certain that everything is consistently broken at scale. Each microservice offers:

1. Anemic & non-idempotent CRUD APIs (Can't believe it's not a Database Table™) because the consumer should do all the work.
2. Race Conditions, because transactions are "not portable" (but not to worry, all the code is written as if it were running single threaded on a single machine).
3. Fault Intolerance, just a single failure in a chain of layered microservice calls will leave the requested operation in a partially applied and corrupted state. Ger ready for manual intervention.
4. Completely Redundant Documentation, our web documentation is automatically generated and is always of the form //[FieldName] of the [ObjectName].
5. Happy Path Support, only the intended use cases and fields work, we added a bunch of others because YouAreGoingToNeedIt™ but it won't work when you do need it. The only record of this happy path is the code itself.

Consider this, you're been building a new microservice, you've carefully followed all the unwritten highly specific technical implementation standards enforced by the CTO/CEO (that your aware of). You've decided to write some unit tests, well um.. didn't you know? There's nothing scalable and consistent about running the system locally! That's not built-in to the framework. So just use curl to test your service whilst it is deployed or connected to the development environment. Then you can open a PR and once it has been approved it will be included in the next full deployment (at least a week later).

Most new 'services' feel like the are about one to five days of writing straightforward code followed by weeks to months of integration hell, testing and blocked dependencies.

When confronted/advised about these issues the response from the CTO/CEO
varies:

(A) "yes but it's an edge case, the cloud is highly available and reliable, our software doesn't crash frequently".
(B) "yes, that's why I'm thinking about adding [idempotency] to the framework to address that when I'm not so busy" two weeks go by...
(C) "yes, but we are still doing better than all of our competitors".
(D) "oh, but you can just [highly specific sequence of undocumented steps, that probably won't work when you try it].
(E) "yes, let's setup a meeting to go through this in more detail" *doesn't show up to the meeting*.
(F) "oh, but our customers are really happy with our level of [Documentation]".

Sometimes it can feel like a bit of a cult, as all of the project managers (and some of the developers) see the CTO/CEO as a sort of 'programming god' because they are never blocked on anything they work on, they're able to bypass all the limitations and obstacles they've placed in front of the 'ordinary' developers.

There's been several instances where the CTO/CEO will suddenly make widespread changes to the codebase (to enforce some 'standard') without having to go through the same review process as everybody else, these changes will usually break something like the automatic build process or something in the dev environment and its up to the developers to pick up the pieces. I think developers find it intimidating to identify issues in the CTO/CEO's code because it's implicitly defined due to their status as the "gold standard".

It's certainly frustrating but I hope this story serves as a bit of a foil to those who wish they had a more technical CTO/CEO in their organisation. Does anybody else have a similar experience or is this situation an absolute one of a kind?

Once I deleted two tables in a production environment 🤬😩😱

Once I maintained one of the most used and fucked up codebases on the market with almost 1M+ daily users. (cannot say more, sorry).

It's written in PHP and is absolutely terrifying,
the first time I saw some lines of code I was about to scream and cry.

- spaghetti code
- no indentation
- random SQL query unoptimized
- unused vars
- Code is split among several files with no logical reasoning
- Mixed procedural and oop programming
- Unsanitised user input (yes, you got it right)

No test environment, no backup database, every commit goes straight to production.

It's a real disaster but the company prefers to keep it as it is without refactoring or anything else.

Just to make it clear:
It's not hatred against PHP, it's against the code's current status and the older programmers which used to work on it.

"It is pointless to use just a fraction of the data in a homologation environment"
Those words reveal the truth in our creed.
We work in the deepest of back-ends to serve the front.
No data is true. Everything can be edited.
We are Data Engineers.

And for those words to take hold, a junior must execute a leap of faith, and push a hotfix into production.

TIL don't rely too much on in memory databases if your client runs development and production environment on the same machine.

Just don't

Debugception, when you are debuggin someones debug code in a production environment but nobody knows what he was debugging because he left the company.

Ok so some of you have probably seen my previous rants about my computer science teacher and our project but I'm just going to summarize all of them and share with you more of my pain.

1. He edits in the production environment. Its a laravel project and he is creating test database migrations IN THE PRODUCTION ENVIRONMENT AND SWITCHING BACK AND FORTH FROM MASTER AND DEV.

2. He edits in vim and doesn't follow codestyle even though I printed him off a piece of paper and emailed it to him.

3. He doesn't have any ethic when it comes to more complex things like laravel homestead.

4. He doesnt want me to release features even though he takes really long to do them.

While I love vim and it is my editor of choice, some things should be done in an ide. This is really annoying me and I'm really just considering handing him the project if he can't follow basic outline.

Tried to install WAMP manually to learn PHP. It almost put me off development altogether, but great when it worked and I learnt a lot doing it. The next time I used Xampp before moving on to vagrant & VMs.

Sometimes setting up the Dev environment is the hardest part of learning. But better to learn in dev than production!

Trying to make a deadline, waiting for more info from an analist so I can implement the science stuff correctly

Today I receive the email with the info I need. Email is cc'ed to the client. And it starts with complaining about me not having implemented the science stuff. The info I need is attached in this email. Arghgh, how am I supposed to do that if I don't have the information I need.

Apart from that she was testing in the production environment. How do you work with people like this.

But hey, I just got my devRant stickers ;)

When your IT VP starts speaking blasphemy:

"Team,
We all know what’s going on with the API. Next week we may see 6x order volumes.

We need to do everything possible to minimize the load on our prod database server.

Here are some guidelines we’re implementing immediately:

· I’m revoking most direct production SQL access. (even read only). You should be running analysis queries and data pulls out of the replication server anyway.
· No User Management activities are allowed between 9AM and 9PM EST. If you’re going to run a large amount of updates, please coordinate with a DBA to have someone monitoring.
· No checklist setup/maintenance activities are allowed at all. If this causes business impact please let me know.
· If you see are doing anything in [App Name] that’s running long, kill it and get a DBA involved.

Please keep the communication level high and stay vigilant in protecting our prod environment!"

RIP most of what I do at work.

I think UPS' Api documentation and service must be the worst documented and build API I have ever seen from a corporate.

1. The developer website is a mess. A total mess. You can barely find the API type you are looking for.

2. When you get the API and download the documentation, the files, .pdf etc is still a mess. Pages long that most are craps.

3. Each request returns Status Code 200. Even if it is an error. This blew my mind.

4. Each request, based on error type or based on tracking activity returns different JSON schema.

For example, the JSON Schema for a shipment in transit is different from JSON schema for a shipment that has been delivered. A shipment that has been returned, a shipment that required signature etc. They are different from each other.

5. And the worst. They do not provide with test tracking codes. I have found some on internet, but they do not work in development and production environment.

I'm writing all the dev things I know in a docs site as a means to be hireable should I need to switch jobs.

I'm not gonna go too deep on how I'm doing it. One style I'm enjoying is making every article take only one page long, and if they take longer, maybe consider breaking it into another article.
Fuck long articles. Yes, that's a bit autistic.

But I will describe the challenges I'm finding (which are quite many) in further detail.

One of them is that words can be ambiguous. Production can mean the production environment but it can also mean production in plain english.
And there are tons of cases like this.

Because of this, I felt a lot of confusion in my beginner days. So it my objective to write this as to prevent as much confusion as possible.

Granted, I don't want to write "development for dummies". Software is complex. But because it's complex on its own, I don't want to add complexity to the learning process through obscure language usage.

"Fine", I say, "I'll disambiguate". But this means I find myself branching out very often into fundamental or commonly used software terms like "framework", "model", "scaffold", "algorithm", "viewport", "breakpoint", etc.

Another challenge is reaching good levels of completitude.
This means I have to explain that obscure CLI flag I never used in my life.

If I don't do this, then what makes my docs different than these superficial dev.to or medium posts? Nothing.
But trying to explain EVERYTHING about a software can generate a lot of frustration: I never finish.

It also makes me wonder "do I even know shit?". I think some amount of insecurity is healthy and pushes myself forward.
But at some point it's kind of making me feel like shit. Maybe I just need to keep learning.

A developer couldn't get a application performance monitoring (APM) tool to trace his application. They claimed that their libraries and their configurations were alright and that the APM tool was non-performant.

The developer then argues with sysadmin that the APM tool can't trace the application and that there's nothing wrong with the application or the configurations. When sysadmin questions whether the developer got the tool to work anywhere, they say, "No" and head off to make it work at least in one place. They come back saying that it works on their development environment (which is their local machine). Sysadmin claims that the system configurations on the server instances cannot be matched by the development environment and there could be a lot more factors to be considered for the problem. The sysadmin asks to prove it on a server instance on one of the test environments and then they'd agree that it is a problem with the tool. They also argue that this is not the only application that uses the APM tool and the tool happily traces other applications with no issues.

The developer tries the same configuration on a staging instance and fails. In order to make it work, they silently uninstall the existing version of the APM tool and then compiles an unstable branch of the tool. It finally works with this version.

They go back to the sysadmin and show that it works on the staging environment, but does not on production. After banging their head on the wall for a while, the sysadmin figure that the tool had been swapped out for the unstable branch that was manually compiled. When questioned, the developer responds, "It works with this version on staging, so deploy the same version on production"

WTF? You don't deploy an unstable branch to production. Just because you can't make it work on the stable branch doesn't mean that it is the problem with the tool itself. There's a big difference between a stable branch and a non-stable branch. How would you feel if the sysadmin retorted by asking you to deploy the staging branch of your application to production?

Lead dev runs the program I gave him to set up a bunch of processes that run for one database.

It has a GUI that seems native to his windows environment......but it sort of is not.

The program runs, asks for the .csv file that is to be parsed into the database.

Lead dev: Ok, what is this though?
Me (his boss) "Don't worry about it"

Him: "Holy shit what the fuck is this??? TELL ME!!!"

Me: DON'T WORRY ABOUT IT

Him: "WTF DID YOU MAKE THIS IN???!

ME: DON'T WORRY ABOUT IT

CMS Admin (another one of my employees) "Would you TWO SHUT THE FUCK UP!!!?"

New Guy (mainly a frontend dev): ........

Meanwhile, in production, no one knows if your gui app is built in Lazarus and Free Pascal, as long as it works.

I really need to stop doing this to the lead dev, dude already keeps trying to choke me for writing things in perl.

On another note, Object Pascal is pretty cool. Might write a book on it for those that want to do CLI based applications on it, I have no clue why every book on the subject costs in euros, but there should be more shit written for beginners, language is awesome and one can get lots of mileage from Lazarus and FPC

As my first dev job, I took over role of solo programmer maintaining all kinds of custom-made software used by local ISP. It was about 10 years ago.

My first question was where can I find test environment and repo. Apparently there was none and I should learn and develop on production.
My sin was to quickly give up on setting up both test and repo.

My second sin was to continue using the same copy&paste PHTML with register_globals enabled, building over it without attempting to refactor it with templates. I did not use globals in any new code at least.

And I suppose my third sin was that I was playing games when I was done with my tasks. I could have used that time to refactor a bit.

But I think in the end I was absolved from them since I was the only one suffering from this. I stayed with company until it got sold and helped migrate data over (along with myself).

After three months of development, my first contribution to the client is going live on their servers in less than 12 hours. And let me say, I shall never again be doing that much programming in one go, because the last week and a half has been a nightmare... Where to begin...

So last Monday, my code passed to our testing servers, for QA to review and give its seal of approval. But the server was acting up and wouldn't let us do much, giving us tons of timeouts and other errors, so we reported it to the sysadmin and had to put off the testing.

Now that's all fine and dandy, but last Wednesday we had to prepare the release for 4 days of regression testing on our staging servers, which meant that by Wednesday night the code had to be greenlight by QA. Tuesday the sysadmin was unable to check the problem on our testing servers, so we had to wait to Wednesday.

Wednesday comes along, I'm patching a couple things I saw, and around lunch time we deploy to the testing servers. I launch our fancy new Postman tests which pass in local, and I get a bunch of errors. Partially my codes fault, partially the testing env manipulating server responses and systems failing.

Fifteen minutes before I leave work on the day we have to leave everything ready to pass to staging, I find another bug, which is not really something I can ignore. My typing skills go to work as I'm hammering line after line of code out, trying to get it finished so we can deploy and test when I get home. Done just in time to catch the bus home...

So I get home. Run the tests. Still a couple failures due to the bug I tried to resolve. We ask for an extension till the following morning, thus delaying our deployment to staging. Eight hours later, at 1AM, after working a full 8 hours before, I push my code and leave it ready for deployment the following morning. Finally, everything works and we can get our code up to staging. Tests had to be modified to accommodate the shitty testing environment, but I'm happy that we're finally done there.

Staging server shits itself for half a day, so we end up doing regression tests a full day late, without a change in date for our upload to production (yay...).

We get to staging, I run my tests, all green, all working, so happy. I keep on working on other stuff, and the day that we were slated to upload to production, my coworkers find that throughout the development (which included a huge migration), code was removed which should not have. Team panics. Everyone is reviewing my commits (over a hundred commits) trying to see what we're missing that is required (especially legal requirements). Upload to production is delayed one day because of this. Ended up being one class missing, and a couple lines of code, which is my bad (but seriously, not bad considering I'm a Junior who was handed this project as his first task at his first job).

I swear to God, from here on out, one feature per branch and merge request. Never again shall I let this happen. I don't even know why it was allowed to happen, it breaks our branch policies. But ohel... I will now personally oppose crap like this too...

Now if you'll excuse me... I'm going to be highly unproductive and rest, because I might start balding otherwise after these weeks...

Hell of a Docker

One application in c++. 4 in c# targeting Linux. Several logging places, Several configuration files , dozens of different folders to access (read/write). Many applications being called from just one that orchestrates everything.

OS is Linux. Installation is to be made inside a docker image and later placed in a container by means of several bash files and python scripts. All these are part of a legacy set of applications.

They’ve asked me to just comment out one line which took 3 days to find out because they didn’t remember where it was and in which application it was and what was in that line.

After changing it, I was asked to create a test environment which must have resemblance to the current server in production. 12 days later And many errors, headaches, problems with docker, I got it done.

Test starts and then, problems with docker volumes, network, images, docker-composer, config files and applications, started to appear.

1 month later, I still have problems and can’t run all applications at least once completely using the whole set.

Just one simple task of deploying locally some applications, which would take one or two days, is becoming a nightmare.

Conclusion: While still trying to figure out why an infinite loop was caused by some DB connection attempt in an application, I am collecting a great amount of hate for docker. It might be good for something, that’s for sure, but in my experience so far, it is far worse than any expectations I had before using it.

Lesson learned: Must run away from tasks involving that shit!

Need some advice here.

So hello everyone! I recently moved abroad for work, for the sake of the experience and the excitement of learning how developers in Latin America tackle specific problems. To my surprise, the dev team is actually composed solely of Europeans and Americans.

I work for a relatively new startup with an ambitious goal. I love the drive everyone has, but my major gripe is with my team lead. He's adverse to any change, and any and all proposals made to improve quality of throughput are shot down in flames. Our stack is a horrendous mess patched together with band-aids, nothing is documented, there are NO unit tests for our backend and the same goes for our frontend. The team has been working on a database/application migration for about a month now, which I find ridiculous because the entire situation could have been avoided by following very rudimentary DevOps practices (which I'm shunned for mentioning). I should also add that for whatever reason containerization and microservices are also taboo, which I find hillarious because of our currently convoluted setup with elastic beanstalk and the the constant complaints between our development environment and production environments differing too much.

I've been tasked with managing a Wordpress site for the past 3 weeks, hardly what I would consider exciting. I've written 6 pages in the past two weeks so our marketing team can move off of squarespace to save some money and allow us more control. Due to the shit show that is our "custom theme" I had to write these pages in a manner that completely disregard existing style rules by disabling them entirely on these pages. Now, ironically they would like to change the blog's base theme but this would invertedly cause other pages created before I arrived to simply not work, which means I would have to rewrite them.

Before I took the role of writing an entire theme from scratch and updating these existing pages to work adequately, I proposed moving to a headless wordpress setup. In which case we could share assets in a much more streamline manner between our application and wordpress site and unify our styles. I was shot down almost immediately. Due to a grave misunderstanding of how wordpress works, no one else on the team seems to understand just how easy it is to fetch data from wordpress's api.

In any event, I also had a tech meeting today with developers from partner companies and realized no one knew what the fuck they were talking about. The greater majority of these self proclaimed senior developers are actually considered junior developers in the United States. I actually recoiled at the thought that I may have made a great mistake leaving the United States to look a great tech gig.

I mean no disrespect to Latin America, or any European countries, I've met some really incredible developers from Russia, the Ukraine, Italy, etc. in the past and I'm certainly not trying to make any blanket statements. I just want to know what everyone thinks, if I should maybe move back to the states and header over to the bay/NY. I'm from the greater Boston area, where some really great stuff is going on but I guess I also wanted a change of scenery.

Dumb mistake from when I was still working:

My work laptop’s SSD went haywire, and I/O would spike every 10 minutes or so for ~50 ms. The hardware guy said he could replace the SSD right away, or I could endure it for a few weeks and get a new laptop instead. Obviously, I agreed to wait. The stutter noticeably affected screen rendering, but I didn’t notice any other issues. Little did I know that every time it happened, all input was ignored (as in: not queued). Normally it wouldn’t matter, because hitting a random ~50 ms window is hard. How-the-f×ck-ever…

A few days later — without getting into “why” — I was forced to apply a patch in production. So I opened an SSH session to prod in one terminal, spun up a dev environment in another, copied the database schema from prod to dev, and made sure to test everything. No issues, so I jumped to prod, applied the patch, restarted services, jumped back to dev, and cleaned up the now-unnecessary database. Only to discover that my “jumped back to dev” keystroke didn’t register.

TL;DR: idiot 'team leader' does mindless merge to master. Precious time wasted in a high pressure deadline environment.

So, i work currently at one of Belgiums largest consulting company's at brussels airport, we are moving their analytics platform to the cloud.

We use puppet to manage the systems.

When i started i noticed immediately that their 'development workflow' is hardly to be named as such, because they simply change stuff directly on server , manual 'temporary' fixes everywhere, hardcoded stuff, non validated code... Basically the way one would develop in their garage, not in a consulting company as this one. But that is just the beginning.

A month ago i did a major effort to equalize all the discrepancies between the codebase and the server. Ensured entire codebase to be validated, syntax checked, parsed, tested... It works. A 'great codebase overhaul' commit was PR'ed to master and got merged.

Yesterday the team lead, i'll call him 'B-tard' from here on, has also 'equalized the discrepancies between codebase, server and the restnof the stale branches on the repo' . i was doing my other work on my branch so no fucks given. This is where i should have given some fucks.

Anyways, today. The day starts every day with merging the master branch into your working branh because you need the latest working codebase, right?

Wrong!
This fucking dipshit smug b-tard has done a mindless merge of the entire codebase, effectively removing ALL validated working code for provisioning servers. Control blocks, lookup functions, lambda's... Basically everything he did not understand.

At the same time the project is already way beyond the allotted budget in pkney and time, so there is a huge pressure to have a working 'production' environment TODAY!

THIS MOTHERFUCKING B-TARD JUST MADE THAT IMPOSSIBLE.

i'm loving this assignment, i'm loving the PM, the collegues, the environment, the location... everything. All but this fuckibg b-tard that somehow got his position by sucking dick or licking ass or both...

I wanna get out asap.

Oh... While typing this and arriving at the room of the office... It is locked, i have no key.

Fucking asshole!

Working on production issue,
Kind of nervous checking logs and so on...
Ops manager and PO who were looking over my shoulder this whole time start shooting the breeze.

I know what they were trying to do. They are trying to create a relaxed environment.

But the issue is that the talk is very distracting. If you want to shoot the breeze please go somewhere else.

Anyway just did that, asked them to leave. They weren't happy about it. But I really needed the silence.

I don't always wreck the production environment, but when I do it's because I'm trying to refactor a piece of code with a "// this is here for reasons, don't change this" comment above it.

Working from development environment. New API module works like a charm.

Migrated to production. Whole CRM breakes down.

Migraine intensifies.

Any devs out there worked building Golang microservices for a production environment?

I don’t have a specific question really. Just wondering who is out there on a similar path!

I’m building using Golang, Google Cloud, Docker, Kunernetes, and Terraform currently on a personal product bound for production!

Why is cd so anoying. I tried serval stuff with all kind of setups. But everything just doesn't work good or really expensive. I just want a easy way to have a develop and production environment without to much problems or an high price card.

Does anyone havr any tips. Already wasted so much time on it

Continuing my last random post. (Please don't bother to take a look at it.)
But, hey you. Yes you, get yourself one beer/whisky and cheers!!
Why? Because my first django project ran successfully in staging environment!! Ok, There were few little bugs. But I fixed most of them.
I don't drink. So please go and enjoy on behalf of me.
And don't drink too much. Keep one bottle for production deployment.
P.S. This is just a beginning of the new journey! Still, lot to learn and experience.

When a major bug fixad recently by you reappears on production environment 'cause someone else (not you of course...) merged old branch content to live branch...

Storytime.

once upon a time, there was a dream: we need to test the vagrant setups for our Devs, so that they can run these against the production environment of puppet without problems.

in the year of 2016, the once lone ranger - our team lead - created the ticket. don't. even. ask.

the idea was to build these vagrant setups via bamboo, log the results and fix the setups afterwards.

after weeks of brain fuckery (aka daily business), home office madness, beer, java specs, more beer and many failed builds, I made it.

bamboo now builds the fuckers via a dedicated agent now and I closed the ticket today \o/.

Just joined a new company and can only describe the merge process as madness.....is it or am I the one that is mad?!

They have the following branches:

UAT#_Development branch
UAT#_Branch (this kicks of a build to a machine named UAT#)

Each developer has a branch with the # being a number 1 to 6 except 5 which has been reserved for UAT_Testing branch.

They are working on a massive monolith (73 projects), it has direct references to projects with no nuget packages. To build the solution requires building other solutions in a particular order, in short a total fucking mess.

Developer workflow:

Branch from master with a feature or hotfix branch

Make commits to said branch and test manually as there are no automated tests

Push the commits to their UAT#_Development branch, this branch isn't recreated each time and may have differences to all the other UAT#_Development branches.

Once happy create a pull request to merge from UAT#_Development to UAT#_Branch you can approve your own pull request, this kicks off a build and pushes it to a server that is named UAT#.

Developer reviews changes on the UAT# server.

QA team create a UAT/year/month/day branch. Then tell developers to merge their UAT#_branch branches in to the previously created branch, this has to be done in order and that is done through a flurry of emails.

Once all merges are in it then gets pushed to a UAT_Testing branch which kicks off a build, again not a single automated test, and is manually tested by the QA team. If happy they create a release branch named Release/year/month/day and push the changes into it.

A pull request from the release branch is then made to pre-live environment where upon merge a build is kicked off. If that passes testing then a pull request to live is created and the code goes out into production.

Ahhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh it's a total mess. I knew when I took on this job it would be a challenge but nothing has prepped me for the scale of the challenge!! My last place it was trunk based development, commit straight to master, build kicks off with automated testing and that just gets pushed through each of the environments, so easy, so simple!

They tell me this all came about because they previously used EntityFramework EDMX models for the database and it caused merge hell.

A very long rant.. but I'm looking to share some experiences, maybe a different perspective.. huge changes at the company.

So my company is starting our microservices journey (we have a 359 retail websites at this moment)

First question was: What to build first?

The first thing we had to do was to decide what we wanted to build as our first microservice. We went looking for a microservice that can be used read only, consumers could easily implement without overhauling production software and is isolated from other processes.

We’ve ended up with building a catalog service as our first microservice. That catalog service provides consumers of the microservice information of our catalog and its most essential information about items in the catalog.

By starting with building the catalog service the team could focus on building the microservice without any time pressure. The initial functionalities of the catalog service were being created to replace existing functionality which were working fine.

Because we choose such an isolated functionality we were able to introduce the new catalog service into production step by step. Instead of replacing the search functionality of the webshops using a big-bang approach, we choose A/B split testing to measure our changes and gradually increase the load of the microservice.

Next step: Choosing a datastore

The search engine that was in production when we started this project was making user of Solr. Due to the use of Lucene it was performing very well as a search engine, but from engineering perspective it lacked some functionalities. It came short if you wanted to run it in a cluster environment, configuring it was hard and not user friendly and last but not least, development of Solr seemed to be grinded to a halt.

Elasticsearch started entering the scene as a competitor for Solr and brought interesting features. Still using Lucene, which we were happy with, it was build with clustering in mind and being provided out of the box. Managing Elasticsearch was easy since there are REST APIs for configuration and as a fallback there are YAML configurations available.

We decided to use Elasticsearch since it provides us the strengths and capabilities of Lucene with the added joy of easy configuration, clustering and a lively community driving the project.

Even bigger challenge? Which programming language will we use

The team responsible for developing this first microservice consists out of a group web developers. So when looking for a programming language for the microservice, we went searching for a language close to their hearts and expertise. At that time a typical web developer at least had knowledge of PHP and Javascript.

What we’ve noticed during researching various languages is that almost all actions done by the catalog service will boil down to the following paradigm:

- Execute a HTTP call to fetch some JSON

- Transform JSON to a desired output

- Respond with the transformed JSON

Actions that easily can be done in a parallel and asynchronous manner and mainly consists out of transforming JSON from the source to a desired output. The programming language used for the catalog service should hold strong qualifications for those kind of actions.

Another thing to notice is that some functionalities that will be built using the catalog service will result into a high level of concurrent requests. For example the type-ahead functionality will trigger several requests to the catalog service per usage of a user.

To us, PHP and .NET at that time weren’t sufficient enough to us for building the catalog service based on the requirements we’ve set. Eventually we’ve decided to use Node.js which is better suited for the things we are looking for as described earlier. Node.js provides a non-blocking I/O model and being event driven helps us developing a high performance microservice.

The leap to start programming Node.js is relatively small since it basically is Javascript. A language that is familiar for the developers around that time. While Node.js is displaying some new concepts it is relatively easy for a developer to start using it.

The beauty of microservices and the isolation it provides, is that you can choose the best tool for that particular microservice. Not all microservices will be developed using Node.js and Elasticsearch. All kinds of combinations might arise and this is what makes the microservices architecture so flexible.

Even when Node.js or Elasticsearch turns out to be a bad choice for the catalog service it is relatively easy to switch that choice for magic ‘X’ or component ‘Z’. By focussing on creating a solid API the components that are driving that API don’t matter that much. It should do what you ask of it and when it is lacking you just replace it.

Many more headaches to come later this year ;)

When the project officially closes as successful, and you see it in action in the production environment, and leadership and customers are satisfied.

I'm given a simple assignment to update email templates. I tot it would be a breeze.

It turn out SURPRISE! After the updating of template is done. I deploy the code in the development environment.

I tried to access the email template like how the user will see to verify all is good. It turn out i am facing error.

So uhh ok, i went to check the logs to see what the hiccups. It turn out that a table is missing. But this is production code. So my question how the hell did the production environment has the table but dev don't.....

So nice, a good, well structured DTAP environment with Acceptance and Test containing a recent dump from Production so that bugs can be reproduced properly........ *wakes up*

How do you manage production, development and test environment in iOS/Xcode? Any good sources out there? 💻

Just found out our pre-production environment is the qa for another team.

Infra team has recommended to ignore the nomenclature. It's just a name they said.

Hey devs or sysadmin here in devRant I wanna know what hypervisor are you using in production or dev environment??

I will annex the hypervisor that I know and I work on, but are free to add more.

Vote with a "++" in the hypervisors that you use.

When it comes to config files for any kind of application, I tend to make sure that I understand what each config does, and that for each environment, they have the correct settings.

However some of my co-workers don't bother understanding what the configs mean and so I have people copying and pasting config details from development environments into staging and production configs. They think that just changing hostnames is more than enough.

So they ended up wasting a good 3 hours trying to figure out why sessions were always invalidating and cached objects were not caching. I had a look at the config and realised their mistake in like 3 mins.

Friday: Run your security test on production after hours.
Saturday: Wait, do development instead.
Today: Ya know, development is a critical environment too. Just don't test anything at all.

🙄

Today at work, I had to do a dry run of our new production environment config for our VM server.
So I setup ESXi in a vm in my workstation. Installed it it, no problem. Setup some VMs, still going good. Then my supervisor came and said I have to make some tweaks and use vCenter. I thought: meh, that came late but ok. Download the vCenter installer from our internal CDN and ran it. Sorry general config, blah blah blah. "Setup with embedded or external platform service controller?" -"Embedded" - "minimal requirements: 11GB ram, 250GB disk storage, 2 CPU cores". Well... Fuck me. My workstation specs: 8 GB RAM, 128 GB SSD, 4 cores.

What the bloody fucking hell does this stuff need 250 GB disk space for?!

Well at least I've got a permanent upgrade for my workstation.

The main developer at the company just got a new gig and now I'm the sole dev. Everything in the production environment is legacy and in C#. Guess it's time to take a crash course...

Fuck when your client find a bug in production, but you can't replicate in your developmment environment. So sad 😣

for the 3rd time ive tried introducing some version control on a project that really needs it because it has multiple people working on it.

And because the last time my efforts got shut down because in practice people thought it was too much of a hassle to develop locally rather than on the shared development server directly, I made a feature that would let people checkout branches on said server...

Apparently the action of; saving > committing > pushing to your feature branch > merge after aproval, is still too much for people to comprehend; "I think this is too convoluted can't we just keep pushing to the production server to check our work and then commit and push to the master branch"

So I just got pissed and said fuck it, no more git then, I'm not even going to put any effort into changing tooling here anymore, and this is a massive project where we have to manually remove code that isnt ready yet from the staging environment.

Are the people I'm working with just this stupid or am I really overengineering this solution because I think 4 people should not be working on the same file at the same time without any form of version control and just direct upload to FTP.

(and yes, I know I should leave this job already, but social anxiety of starting at a new company is a big obstacle for me)

Setting up a production environment for a product is so shitty!!!! Why God, why????

You are just a great programmer when you build features in production environment.

I’ve been at this issue at work for four days now and no progress and I feel really bad because we have important stories to pick up and I feel I’m wasting my capacity like this because I haven’t fixed it. Basically, only in our QA environment (one before production) our services is not acknowledging duplicate events posted by Kafka, thus keeps reprocessing them. I’ve spent so long trying to diagnose the code, which is the same in all envs currently, seeing how this suddenly occurred, restarted things, went through complications of using different tools, asked for help from others a lot but IVE gotten NOWHERE. Idr wanna say to my team that I should prioritise other things because we have deadlines but I feel this issue is important to fix but I just can’t figure out how. Now I’m worried this whole sprint will go without me doing anything and then fingers pointed at me later

Has anyone ever tried to export a table in SAP HANA to csv format? Unfortunately, we are not allowed to access our SAP HANA environment directly from our development laptop (the SAP environment is a production environment, of course we don't have a develop or test environment nor does anyone feel the need to create one), so we need to email ourselves a csv file with the data we need to develop our models in python (I'm not kidding).

Anyhow, among other things, I need to manually add column names in the file, some columns are quoted, some are not, I cannot choose my delimiter, ...

Is this vendor lock-in the year 2023?

Who else is frustrated/burnout at building products that never gets into production?

When I work for a company I always tend to do everything with good practices, spend a lot of time thinking on the best ways to build x feature, and then the company falls into the infinite loop of adding stupid features, and then I've been working for 2 years and 0 paid customers. Funny that we've Sentry, GA, Hotjar sitting there doing nothing.

I'm honestly hating the startup environment rn. Good thing is that I've learnt a lot and salary is good. But also I lost all motivation.

Any recommendations for a tired dev?

In our company, "UAT".
We using staging environment and most of the data is either missing or corrupt. They don't refresh the data, saying it can impose some security issue.
How the hell are we supposed to complete UAT when there's no data that's in production!!!!

I was hoping that I never have to build stupid websites again, but here I am...

The thing I hate the most, is the hassle to have an easy to update dev, staging and prod environment. Fuck wordpress, fuck drupal, fuck joomla.

git pull && composer install && npm install should be all that's needed to get the latest code in an environment.

composer require *** to install plugins. No stupid web interface where users install plugins in production env.

I don't want to create database dumps just because these fuckers think that you should store configurations in the database.

Is there any clean CMS primarily for professional programmers? Or are they all just made for retarded subhumans?

- Teammate discovers a standard PaaS feature isn’t working and breaks core functionality in dev environment
- Teammate creates a support ticket to the PaaS company
- PaaS company says that they’re aware of the issue but don’t have a solution yet and advises to disable the feature for now
- Teammate ships the feature and leaves it enabled on production.
- Teammate thinks that “oh we know it’s broken, nobody is going to use it anyway”
- Customer uses the feature
- Shit hits the fan
- Teammate: *shocked pikachu face*

Currently debugging a project that was written over 4 years ago...

At first all was well in the world, besides the ever present issue off our goddamn legacy framework. This framework was written 7 years ago on top of an existing open source one, because the existing one was 'lacking some features' & 'did not feel right'.

Now those might be perfectly fine reasons to write a layer on top of a framework, but please, for all future devs sanities, write fucking documentation and maintain it if you're going to use said framework in all major projects!!

Anyhow back to the situation at hand, I'm getting familiar with the project, sighing at the use of our stupid legacy framework, attempting to recreate the reported bugs...

Turns out I can't, well I get other bugs & errors, but not the reported ones. I go to the production server, where I suddenly do can reproduce them...

Already thinking, fuck my life, and scared for the results... I try a 'git status' on the production server....

And yep, there it is, lo and behold, fucking changes on production, that are not in git, fuck you previous dev who worked on this and your stupid lazy ass modifcations on production!

Bleh, already feeling royally pissed, there's only 1 thing I can do, push changes back to git in a seperate branch, and pray I can merge them back in master on my dev environment without to much issues...

Only I first have to get our sysadmi. to allow pushing from a production server back to our git server...

Sigh, going to put on my headphones, retreat to my me space and try to sort out this shitpile now...

Gah, I just received this Ubuntu 18.04 VM with 8 cores and 8 gigs of ram, and since it'll be a production server both serving public and "private" networks (yes, shout at me, but projects won't be about hosting sensitive information, I wouldn't put all that on one server), and I'm struggling between my options.

Docker, or not docker?

The server's main use is to host our growing blog and install Varnish, which will hog some ram after a while. I use Laradock for my dev projets, it's really easy to develop with it, but I am unsure if it fits a production environment with performance, security and traffic load in mind :(

I read Docker has stability issues (in 2016-2017), and can bring the machine down with it, I don't know if I should just install the software (nginx, apache, percona/mysql/maria) without "containerizing" it and go for it

I'm lost xD

Who inside BitTitan is doing live testing on production? Would you kindly revert the changes and do testing on a test environment? I've seen the 4 changes so far and they clearly not working.
Please we need to finish this migration

We have pre-prod and prod test environments. Prod is supposed to be exactly the same as a client's production environment. Find out our prod test environment is nothing like the client's production environment... for over 3 years.

The joy of ops - when the customer says "my reporting environment isn't fast enough - can I get my reports from the production environment?" and we've specifically created the reporting environment for insulating the system from performance-hogging queries...

Using preview / beta builds on production environment.

The highlighted lines are a part of a flask app I'm writing in Python2(not python3 because I'm a bit too lazy to fix few dependency errors). All functions work as expected and all templates are rendered individually, and routes are all defined. check_date checks for invalid dates like 32Jan, 2018. It returns 0 if date is valid. add_data basically returns 0 if it decides to add user data into the database(db).
The problem is that line 60 renders but lines 54,57 don't. Any ideas as to what might be going wrong.?
PS: I'm building this app for learning and not for a production environment...

If your code is giving HTTP 500 error on a production server, go kill yourself, until you are having a development environment on the production server. In that case, kill your manager! 💢

It annoys me immensely when I struggle with myself, criticizing my own lack of knowledge in certain areas and my colleagues say: "You'll learn by doing". No, I won't, that's a foolish dogma.

I won't and I have never learned by 'doing'. The best results I've obtained have been through understanding every last bit of what's under the hood of a particular functionality. I'm not going to understand the white box by constantly probing the black box, it's just unsatisfactory and insufficient information. It's even dangerous to base yourself on the black box results because you often might get false positives.

I got through university by massive multilateral sensory focus: kinesthetic (writing things down), auditory (listening to the professor), visual (observing graphs and models of the material taught), conscious (mentalizing it all and interlinking information so that later it's accessible from long-term memory). I can confirm this is necessary for the brain because a Neurologist once told me just that.

At least for me, I had the most horrible grades (D's and F's) in freshman year with the 'learn by doing' method and the best grades (A, A+) with the multi-sensory method in later years as I matured my studying methods. In fact, with that method I've continuously outsmarted other people who had 10 years more experience than me ('experts', 'consultants',..) but they preferred to stay in the ignorant 'bro zone' rather than learning things properly. Even worse, the day they arrived on the scene, they completely broke the production environment and messed it up for the whole team. I felt like banging my head on my desk. It just makes me disappointed in the system.

If you follow popular method, you'll soon find yourself in the same problems that arise from doing what everyone else does. What happens at that point? That's right, they have to call in someone who actually bothered learning things.

As back end developer, I rarely have hands on production environment. When it happens, I need to ask my way around and since the office is empty that day, I ask the client directly. They give me a URL. Right away, I ask the credentials.
"Just connect to the URL"
"You mean, you have an open access of this software, having critical information of more than 50 000 persons, to the web?"
*Silence* "hahaha it appears that way"
Thankfully, a tactful manager handled the situation astutely and we never heard about it anymore.
Don't we love all happy ending?

Sure boss, we don't need staging. Let's just copy some tables from our customer's server to our testing machine, overwrite our data with theirs and start testing "simulating" their environment. It's not that we need to test for our production, right?

What are you using for local environment? I'm stuck on xampp for 10 years. I realized that I'm having terrible pipeline from developing to production.

Today our PM planned to deploy in production an e-commerce based on PrestaShop.
A colleague of mine mamaged to implement everything that was necessary, and I made a small script to add random sales on random products every sunday.
We tested it several times in our environment, on multiple machines, and everything was working fine.

BUT
Today we launched the script on production server, and we was a little mistake.
"A bug? Say no more pal, I'll fix it!".

Fixed, tested on local environment, deployed and.... The first steps weren't working.

"Fatal error".
That's what I got. No exceptions, no error messages, no references.. Just "fatal error".
We spent two hours looking for the problem, thinking it was a server error that was just outputting that shitty message.

And you know what? Some fucking fat cocksucker son of a bitch thought it was an excellent idea to stop the code execution with a simple and very helpful "fatal error".

"oh, wait, there is an error here, let me print die(" fatal error"), ao the other developer will be able to find what's going on", he thought.

FUCK YOU MORON.

TL;DR: Avoid French software, they are a bounch of asshole (except some goos guy..)

How to set the up a stagging environment for a github branch in heroku.
Lets say i have a master branch and a dev branch. All the changes and updates first pushed to dev branch and after successful review and test in goes to master branch.
But the issue is as i'm following the gitflow of keeping the master branch always deployable.since my heroku app is linked to the master branch, when i try to test the dev branch in production environment of heroku,sometimes it breaks for some error.and at that time the sites goes down untill i redeploy the master branch as it's the stable version .So how do i test a branch in heroku production environment while also keeping the the site active with master branch. it that makes any sense 🤡 plz help

Is there a unit test for dance moves? I really gotta see some green lights before I move this one into production. Side note: anyone have a UAT environment for this? Thx

First run of an import procedure in the production environment.
Spent all morning with an "Unsupported media type" error.
Finds out that the provided password was wrong and that the Webservice always return that message when there's an error.
Any type of error...

I wanna know who use xenserver and why??

Currently am using xenserver 6.5 in a production environment and today i start to test xenserver 7.

which type are you ??

**Manager:** Hey, we've got a little hiccup in the production environment. I know it's Friday evening and you're probably daydreaming about pizza, but could you give it a peek?

**Type 1:** Man, this is like finding a needle in a haystack while wearing sunglasses at night. Might take me a few hours... or days. But hey, wish me luck and have an epic weekend!

**Type 2:** Eureka! Found the gremlin. It looks like XYZ person tried to be a bit too creative on commit number 2234324. Maybe they had too much caffeine? Anyway, could you have a chat with them? And oh, may your weekend be as smooth as a fresh jar of peanut butter.

**Type 3:** Detective mode activated! Found the sneaky bug. It was XYZ person's "masterpiece" in commit number 2234324. But fear not! I've put on my superhero cape and fixed it in commit number 345453345.

**Type 4:** This issue again? It's like a recurring bad dream about forgetting your pants! I've revamped the whole thing so we don't have to relive this nightmare. If someone tries to pull this off again, our CI/CD will roast them like a marshmallow over a campfire.

**Type 5:** Ta-da! Fixed the glitch, jazzed up the design, and sprinkled in some extra logging magic. Now, troubleshooting will be as easy as pie. Speaking of which, I've got time for a coffee and maybe a slice of pie before heading out. Cheers!
Type 6 **Gloomy**: Oh, the digital clouds have gathered again. This issue is like a never-ending rain on a Monday morning. I've peered into the abyss of our code, and it's... well, it's deep and dark. I'll need some time, a flashlight, and maybe a comforting blanket. If you don't hear from me in a few hours, send in a search party with some hot cocoa.

Sometimes I have to connect to production database and alter my dev environment so I can “log in” as a user and see what’s wrong with their account. Once in a while there is a legitimate website issue that is unique to that user’s profile. Other times it’s user error, like the user not understanding that they have to connect their membership to their online account (they think signing up for an account will connect it automatically).

I don’t like circumventing the user’s log in like this, but sometimes it’s necessary since the website is so confusing. I inherited this website, so many of the problems were formed way before I took over.

My stakeholders want a log in as user feature for website admins to use. My manager and PM don’t think that’s a good idea right now since there are over two dozen people with admin access and admin access means access to everything in the admin (there aren’t options to give permissions as needed).

Testing on the hot-standby server in the production environment

In my initial days as a web developer, i was assigned a task, to implement a cart share functionality in an e commerce company.
I made the functionality and tested on my system.
Result: working good.

Pushed it to beta testing environment.
Resilt: working good.

Pushed to pre production environment.
Result: working good.

Pushed to live site.
Result: 😀 Error in live site..
So a call comes to me from my team lead..
Asks what was the issue...

Me: i dont know either.
....

After 3-4 hrs:
I found the reason.
My system, beta test env, pre prod env are all having latest php version (5.6 i guess)
But the live server had old version of php.

Me: laughed like anything.
I didn't know that these things would matter in such a great level.

Moral of the story:
Be one with the force (server in this case)