I worked with a good dev at one of my previous jobs, but one of his faults was that he was a bit scattered and would sometimes forget things.

The story goes that one day we had this massive bug on our web app and we had a large portion of our dev team trying to figure it out. We thought we narrowed down the issue to a very specific part of the code, but something weird happened. No matter how often we looked at the piece of code where we all knew the problem had to be, no one could see any problem with it. And there want anything close to explaining how we could be seeing the issue we were in production.

We spent hours going through this. It was driving everyone crazy. All of a sudden, my co-worker (one referenced above) gasps “oh shit.” And we’re all like, what’s up? He proceeds to tell us that he thinks he might have been testing a line of code on one of our prod servers and left it in there by accident and never committed it into the actual codebase. Just to explain this - we had a great deploy process at this company but every so often a dev would need to test something quickly on a prod machine so we’d allow it as long as they did it and removed it quickly. It was meant for being for a select few tasks that required a prod server and was just going to be a single line to test something. Bad practice, but was fine because everyone had been extremely careful with it.

Until this guy came along. After he said he thought he might have left a line change in the code on a prod server, we had to manually go in to 12 web servers and check. Eventually, we found the one that had the change and finally, the issue at hand made sense. We never thought for a second that the committed code in the git repo that we were looking at would be inaccurate.

Needless to say, he was never allowed to touch code on a prod server ever again.

Oh, man, I just realized I haven't ranted one of my best stories on here!

So, here goes!

A few years back the company I work for was contacted by an older client regarding a new project.

The guy was now pitching to build the website for the Parliament of another country (not gonna name it, NDAs and stuff), and was planning on outsourcing the development, as he had no team and he was only aiming on taking care of the client service/project management side of the project.

Out of principle (and also to preserve our mental integrity), we have purposely avoided working with government bodies of any kind, in any country, but he was a friend of our CEO and pleaded until we singed on board.

Now, the project itself was way bigger than we expected, as the wanted more of an internal CRM, centralized document archive, event management, internal planning, multiple interfaced, role based access restricted monster of an administration interface, complete with regular user website, also packed with all kind of features, dashboards and so on.

Long story short, a lot bigger than what we were expecting based on the initial brief.

The development period was hell. New features were coming in on a weekly basis. Already implemented functionality was constantly being changed or redefined. No requests we ever made about clarifications and/or materials or information were ever answered on time.

They also somehow bullied the guy that brought us the project into also including the data migration from the old website into the new one we were building and we somehow ended up having to extract meaningful, formatted, sanitized content parsing static HTML files and connecting them to download-able files (almost every page in the old website had files available to download) we needed to also include in a sane way.

Now, don't think the files were simple URL paths we can trace to a folder/file path, oh no!!! The links were some form of hash combination that had to be exploded and tested against some king of database relationship tables that only had hashed indexes relating to other tables, that also only had hashed indexes relating to some other tables that kept a database of the website pages HTML file naming. So what we had to do is identify the files based on a combination of hashed indexes and re-hashed HTML file names that in the end would give us a filename for a real file that we had to then search for inside a list of over 20 folders not related to one another.

So we did this. Created a script that processed the hell out of over 10000 HTML files, database entries and files and re-indexed and re-named all this shit into a meaningful database of sane data and well organized files.

So, with this we were nearing the finish line for the project, which by now exceeded the estimated time by over to times.

We test everything, retest it all again for good measure, pack everything up for deployment, simulate on a staging environment, give the final client access to the staging version, get them to accept that all requirements are met, finish writing the documentation for the codebase, write detailed deployment procedure, include some automation and testing tools also for good measure, recommend production setup, hardware specs, software versions, server side optimization like caching, load balancing and all that we could think would ever be useful, all with more documentation and instructions.

As the project was built on PHP/MySQL (as requested), we recommended a Linux environment for production. Oh, I forgot to tell you that over the development period they kept asking us to also include steps for Windows procedures along with our regular documentation. Was a bit strange, but we added it in there just so we can finish and close the damn project.

So, we send them all the above and go get drunk as fuck in celebration of getting rid of them once and for all...

Next day: hung over, I get to the office, open my laptop and see on new email. I only had the one new mail, so I open it to see what it's about.

Lo and behold! The fuckers over in the other country that called themselves "IT guys", and were the ones making all the changes and additions to our requirements, were not capable enough to follow step by step instructions in order to deploy the project on their servers!!!

[Continues in the comments]

Friday afternoon going straight to production without testing.

F🤬k!! why doesn’t this work on the testing server!l! I swear it worked local.

Meanwhile live:

'Never deploy to production before weekend or holidays.'

Deploy to Production

Worst thing you've seen another dev do? Long one, but has a happy ending.

Classic 'Dev deploys to production at 5:00PM on a Friday, and goes home.' story.

The web department was managed under the the Marketing department, so they were not required to adhere to any type of coding standards and for months we fought with them on logging. Pre-Splunk, we rolled our own logging/alerting solution and they hated being the #1 reason for phone calls/texts/emails every night.

Wanting to "get it done", 'Tony' decided to bypass the default logging and send himself an email if an exception occurred in his code.

At 5:00PM on a Friday, deploys, goes home.

Around 11:00AM on Sunday (a lot folks are still in church at this time), the VP of IS gets a call from the CEO (who does not go to church) about unable to log into his email. VP has to leave church..drive home and find out he cannot remote access the exchange server. He starts making other phone calls..forcing the entire networking department to drive in and get email back up (you can imagine not a group of happy people)

After some network-admin voodoo, by 12:00, they discover/fix the issue (know it was Tony's email that was the problem)

We find out Monday that not only did Tony deploy at 5:00 on a Friday, the deployment wasn't approved, had features no one asked for, wasn't checked into version control, and the exception during checkout cost the company over $50,000 in lost sales.

Was Tony fired? Noooo. The web is our cash cow and Tony was considered a top web developer (and he knew that), Tony decided to blame logging. While in the discovery meeting, Tony told the bosses that it wasn't his fault logging was so buggy and caused so many phone calls/texts/emails every night, if he had been trained properly, this problem could have been avoided.

Well, since I was responsible for logging, I was next in the hot seat.

For almost 30 minutes I listened to every terrible thing I had done to Tony ever since he started. I was a terrible mentor, I was mean, I was degrading, etc..etc.
Me: "Where is this coming from? I barely know Tony. We're not even in the same building. I met him once when he started, maybe saw him a couple of times in meetings."
Andrew: "Aren't you responsible for this logging fiasco?"
Me: "Good Lord no, why am I here?"
Andrew: "I'll rephrase so you'll understand, aren't you are responsible for the proper training of how developers log errors in their code? This disaster is clearly a consequence of your failure. What do you have to say for yourself?"
Me: "Nothing. Developers are responsible for their own choices. Tony made the choice to bypass our logging and send errors to himself, causing Exchange to lockup and losing sales."
Andrew: "A choice he made because he was not properly informed of the consequences? Again, that is a failure in the proper use of logging, and why you are here."
Me: "I'm done with this. Does John know I'm in here? How about you get John and you talk to him like that."
'John' was the department head at the time.
Andrew:"John, have you spoken to Tony?"
John: "Yes, and I'm very sorry and very disappointed. This won't happen again."
Me: "Um...What?"
John: "You know what. Did you even fucking talk to Tony? You just sit in your ivory tower and think your actions don't matter?"
Me: "Whoa!! What are you talking about!? My responsibility for logging stops with the work instructions. After that if Tony decides to do something else, that is on him."
John: "That is not how Tony tells it. He said he's been struggling with your logging system everyday since he's started and you've done nothing to help. This behavior ends today. We're a fucking team. Get off your damn high horse and help the little guy every once in a while."
Me: "I don't know what Tony has been telling you, but I barely know the guy. If he has been having trouble with the one line of code to log, this is the first I've heard of it."
John: "Like I said, this ends today. You are going to come up with a proper training class and learn to get out and talk to other people."

Over the next couple of weeks I become a powerpoint wizard and 'train' anyone/everyone on the proper use of logging. The one line of code to log. One line of code.

A friend 'Scott' sits close to Tony (I mean I do get out and know people) told me that Tony poured out the crocodile tears. Like cried and cried, apologizing, calling me everything but a kitchen sink,...etc. It was so bad, his manager 'Sally' was crying, her boss 'Andrew', was red in the face, when 'John' heard 'Sally' was crying, you can imagine the high levels of alpha-male 'gotta look like I'm protecting the females' hormones flowing.

Took almost another year, Tony released a change on a Friday, went home, web site crashed (losses were in the thousands of $ per minute this time), and Tony was not let back into the building on Monday (one of the best days of my life).

Had to wake four people up at 2 am to fix a crashing service.

10/10 would deploy to production on Friday night again.

One of our clients deploy their own server app. So this happened after a prod deployment. (4am)

*Cellphone rings while sleeping*

Client : we need you on the conference call now. URGENT!

*Gets on conference call*
*Client explain the problem*
*Explaining to the client that the problem is in their side (https connection not working, either network or certificate problem)*
*Client doesn't believe it and pushes me for a fix that I have no control on*
*4 hours later in a heated conversation*

Client : ok problem is on our side. We used our SSL certificate from staging with production and thought it would work.

Me :

My team handles infrastructure deployment and automation in the cloud for our company, so we don't exactly develop applications ourselves, but we're responsible for building deployment pipelines, provisioning cloud resources, automating their deployments, etc.

I've ranted about this before, but it fits the weekly rant so I'll do it again.

Someone deployed an autoscaling application into our production AWS account, but they set the maximum instance count to 300. The account limit was less than that. So, of course, their application gets stuck and starts scaling out infinitely. Two hundred new servers spun up in an hour before hitting the limit and then throwing errors all over the place. They send me a ticket and I login to AWS to investigate. Not only have they broken their own application, but they've also made it impossible to deploy anything else into prod. Every other autoscaling group is now unable to scale out at all. We had to submit an emergency limit increase request to AWS, spent thousands of dollars on those stupidly-large instances, and yelled at the dev team responsible. Two weeks later, THEY INCREASED THE MAX COUNT TO 500 AND IT HAPPENED AGAIN!

And the whole thing happened because a database filled up the hard drive, so it would spin up a new server, whose hard drive would be full already and thus spin up a new server, and so on into infinity.

Thats probably the only WTF moment that resulted in me actually saying "WTF?!" out loud to the person responsible, but I've had others. One dev team had their code logging to a location they couldn't access, so we got daily requests for two weeks to download and email log files to them. Another dev team refused to believe their server was crashing due to their bad code even after we showed them the logs that demonstrated their application had a massive memory leak. Another team arbitrarily decided that they were going to deploy their code at 4 AM on a Saturday and they wanted a member of my team to be available in case something went wrong. We aren't 24/7 support. We aren't even weekend support. Or any support, technically. Another team told us we had one day to do three weeks' worth of work to deploy their application because they had set a hard deadline and then didn't tell us about it until the day before. We gave them a flat "No" for that request.

I could probably keep going, but you get the gist of it.

When you receive a new task to disable a feature that hasn’t been used for months and deploy the changes to production, the last thing you expect is:
> deployment successful
> 5 seconds pass
>
>
> you got mail
> why does this no work anymore

Are you fucking kidding me!

TLDR; I just screwed a production server and rendered it useless!!!

Long story:

I went to install a product that we built at the customer's site, and was given a Linux running server, to deploy our app.

I work in windows, and barely know the basic Linux commands.

So I look at the files in the home directory, and see that the are a lot of files, so I ask the customer if it is ok that I move all the files to a separate directory.

He agrees, and me thinking that I am smart, proceed to enter the following commands in the terminal:

mkdir old
mv /* old

Of course I got an error that I don't have permission so my next command was:

sudo mv /* old

And that was the end of that computer.

The amazing part of the story is that as soon as it happened, I understood so much about Linux.

The file structure, sudo, the power of the terminal, aliases and so much more...

Product Owner: "Our definition of done is putting it on production. So you are only done if it's on production. Otherwise our sprint goal is failed."

So we put it on production.

After deploy, some content manager appears: "Why is the system doing things? I was told this should not happen today."

"Erm, we have put the feature on production as we are only done if it's on production."

"Well, yes. But it should not be live yet!"

Oh well. Communication, or the lack thereof, does never fail to amaze me ¯\_(ツ)_/¯

*production is down*
Ops: At 5pm? On a Friday? *checks deploy history* God! Who did the deploy
Dev: It was a small patch, a tiny patch. It shouldn't have....
Ops: Deploy on a Friday evening?
Colleague: I didn't think it would...

Ops (on the outside) : *takes a deep breath* Its okay Dev, we can fix this. Don't worry

Me(in my mind) : for fuck sakes! Are you fucking kidding me?*** **** *** god damn it! *****

"Read-only Friday" rule: On Fridays, you don't deploy new versions, don't merge code into production, don't update databases, and a lot of others "DON'Ts"

PM (on slack): "we’re about to deploy to production".
Me: "ok"
… I keep on working on a task / remain available for any post deployment issues …
PM (5 minutes later on slack): "deployment broke production! We need to handle this NOW!"

My dev colleague has already called it a day, but I’m still online

Me: "ok I don’t have access to prod, can you describe what’s going on? I can’t reproduce on any other environment"

PM: …

10 minutes go by

Me: "anybody there?"

PM: …

45 minutes later, I realize PM is offline

The following day:
PM: "ok we got prod running again" (turns out it was client’s fault for not updating a config we as devs can’t access)
PM: "but we’re REALLY UPSET! You guys need to be available to intervene for any issues following deployment to production! At least one of you should be available!"

Me: "but, but…" 🫠

local vs production

I was only seventeen back then and I was a Java Developer Intern, not knowing much about enterprise oriented coding.

The project leader in our dev team saw a lot of potential and passion in my work, but was convinced I wasn't taught enough to do the right thing.

I was mainly doing shitty mappers and services back then, which were somewhat used but never lasted long and were ditched a few months later, which always bummed me out. I wanted to make an impact on REAL projects that would deploy into production.

So Mister Mentor (GDPR forbid to use the actual name), who was always first to come and last to leave the office, taught me what it means to code for real.

We stayed after 5pm until 7-8pm multiple times a week and he taught me in a deeply understanding and calm way how to:
- Git (SVN)
- Refactor
- SOA
- Annotate
- Deploy
- Unit Test

And most importantly:
- How to debug like an absolute BOSS
(We even debugged native Java Libraries just for fun to see if we could break them)

Fast-forward a month later and little intern me made his first commit on production.

Without Mister Mentor, I wouldn't be half as good of a developer as I am today.

The solution for this one isn't nearly as amusing as the journey.

I was working for one of the largest retailers in NA as an architect. Said retailer had over a thousand big box stores, IT maintenance budget of $200M/year. The kind of place that just reeks of waste and mismanagement at every level.

They had installed a system to distribute training and instructional videos to every store, as well as recorded daily broadcasts to all store employees as a way of reducing management time spend with employees in the morning. This system had cost a cool 400M USD, not including labor and upgrades for round 1. Round 2 was another 100M to add a storage buffer to each store because they'd failed to account for the fact that their internet connections at the store and the outbound pipe from the DC wasn't capable of running the public facing e-commerce and streaming all the video data to every store in realtime. Typical massive enterprise clusterfuck.

Then security gets involved. Each device at stores had a different address on a private megawan. The stores didn't generally phone home, home phoned them as an access control measure; stores calling the DC was verboten. This presented an obvious problem for the video system because it needed to pull updates.

The brilliant Infosys resources had a bright idea to solve this problem:

- Treat each device IP as an access key for that device (avg 15 per store per store).
- Verify the request ip, then issue a redirect with ANOTHER ip unique to that device that the firewall would ingress only to the video subnet
- Do it all with the F5

A few months later, the networking team comes back and announces that after months of work and 10s of people years they can't implement the solution because iRules have a size limit and they would need more than 60,000 lines or 15,000 rules to implement it. Sad trombones all around.

Then, a wild DBA appears, steps up to the plate and says he can solve the problem with the power of ORACLE! Few months later he comes back with some absolutely batshit solution that stored the individual octets of an IPV4, multiple nested queries to the same table to emulate subnet masking through some temp table spanning voodoo. Time to complete: 2-4 minutes per request. He too eventually gives up the fight, sort of, in that backhanded way DBAs tend to do everything. I wish I would have paid more attention to that abortion because the rationale and its mechanics were just staggeringly rube goldberg and should have been documented for posterity.

So I catch wind of this sitting in a CAB meeting. I hear them talking about how there's "no way to solve this problem, it's too complex, we're going to need a lot more databases to handle this." I tune in and gather all it really needs to do, since the ingress firewall is handling the origin IP checks, is convert the request IP to video ingress IP, 302 and call it a day.

While they're all grandstanding and pontificating, I fire up visual studio and:

- write a method that encodes the incoming request IP into a single uint32
- write an http module that keeps an in-memory dictionary of uint32,string for the request, response, converts the request ip and 302s the call with blackhole support
- convert all the mappings in the spreadsheet attached to the meetings into a csv, dump to disk
- write a wpf application to allow for easily managing the IP database in the short term
- deploy the solution one of our stage boxes
- add a TODO to eventually move this to a database

All this took about 5 minutes. I interrupt their conversation to ask them to retarget their test to the port I exposed on the stage box. Then watch them stare in stunned silence as the crow grows cold.

According to a friend who still works there, that code is still running in production on a single node to this day. And still running on the same static file database.

#TheValueOfEngineers

Accidently deleted user table on production database and some backups where broken. Had to deploy a 4 days old backup...

1. Slack. Pretty good chat app for dev companies, I use it to prevent people standing next to my desk 40 times a day.

2. Unit testing tools, especially when fully automated using a git master branch hook, something like codeship/jenkins, and a deployment service.

3. Jetbrains IDEs. I love Vim, but Jetbrains makes theming, autocompleting & code style checks with mixed templating languages a breeze.

4. Urxvt terminal. It's a bit of work at the start, but so extremely fast and customizable.

5. Cinnamon or i3. Not really dev tools, but both make it easy to organize many windows.

6. A smart production bug logger. I tend to use Bugsnag, Rollbar or Sentry.

7. A good coffee machine. Preferably some high pressure espresso maker which costs more than the CEO's car, using organic fairtrade hipster beans with a picture of a laughing south american farmer. And don't you dare fuck it up with sugar.

8. Some high quality bars of chocolate. Not to consume yourself, but to offer to coworkers while they wait for you to fix a broken deploy. The importance of office politics is not to be underestimated.

That paranoia when you deploy for production and you keep checking if it isn't down every 10 minutes.

We have a pretty simple rule in our team:

Do not deploy to production on Friday.

Well thanks to the client being very slow to reply to me, they only signed off on launching the app at 15:30 on Friday, for a big campaign the app was built to facilitate starting Saturday.

Guess who had to bite the bullet and launch a new app into production at 15:30 on Friday.

Guess who got a text from his boss at 19:30 that there was a critical change required tonight.

Guess who was making code changes and deploying to production at 21:15 on Friday night while drinking Gin and tonic...

Nb This was a project only i was assigned to and came in as a rush job at the last minute.

Fixing someone else's code who left the job.
Production suddenly not working, cannot debug locally, cannot deploy to a test environment because it does not exists anymore.
There should be a contract clause that developer need to support his project for 2 years after he leaves his job.

just deploy to production friday night

TGIF guys

Being forced to deploy to production on friday?

It's about a guy that knows better.

I was working as a subcontractor on a bigger system. We (subs) were not allowed to deploy code, we had to wait for contractor to deploy.

One day I got an email that my code is bugged and that my feature is not working on production. I checked it on test env, everything was fine. Then I checked if the code I wrote was deployed. It was not.

I send an email explaining that if they deployed my code it would be working. Then I got a response. There was a bug in my code.

Another email. I asked how would they know? Do they have a test on their environment that failed?

No. There is one guy that READ my code and he said it should not work, so he will not deploy it. He was not a programmer, he was a business consultant responsible for the documentation.

His issue was that I used a function that was not in a class. So if the function is not declared it's obvious it will not work. I had to explain to him in another email, that you can use object of another class inside your class and then call a function, that is not in your class. It was the last time this guy blocked my deploy.

TL;DR, I had to explain a non-dev how object composition works in order to have my code deployed. Took four emails.

Me: Hey boss, if you ever need someone to get into doing DevOps related tasks for the team, I'd be more than happy to take that on.

Boss: We don't really need any dedicated person to work on that, but if we do in the future, I'll let you know.

Fast forward a few days: I am now unable to deploy bug fixes to our testing environment, now in the cloud, because all access has been blocked for everyone except the two numbskulls who thought it'd be a great idea to move EVERYTHING over (apps, configuration manager, proxies, etc) first.

Oh, and this bug is affecting production.

So my department is "integrating CI/CD"

Right now, there's a very anti-automation culture in the deployment process, and out of our many applications, almost none have automated testing. And my groups is the only one that uses feature branching - one of the few groups that uses branching at all beyond "master, dev"

So yeah... You could see how this is already ENTIRELY fucked from the very beginning.

First thing they want to do is add better support for a process... Which goes directly against CI/CD.

The process is that to deploy to production (even after it is manually approved by manager), someone in another department needs to press a button to manually deploy. This, as far as I can tell, is for business rule reasons rather than technical ones.

They want us to improve that (the system will stay exactly the same with some streamlined options for said button pressers)

I'm absolutely astounded at the way our management wants to do something but goes in exactly the opposite direction. It's like the found an article of what CI/CD was and then took notes on exactly what not to do.

Pressing that “deploy to production” button when it usually consists of multiple projects and databases updating at once.

Maybe I should look at separating them one year.

Because I’m a fucking cowboy and a charlatan, and because I hate sleep and despise feeling refreshed and happy, I’m working pretty much full time as a contractor (I’m the full stack dev. I do everything) on a (well funded) startup alongside my day job.

Tonight I had to make some quick (lol “quick”) changes to a core piece of the platform.

Now before continuing please refer back to the first line of this rant.

So instead of writing new functionality, I copied and pasted another section.

I renamed all references of “new_order” to, cleverly “new_order2”.

I know.

I deploy to production...

My phone starts blowing up. In short, everything is fucked.

I’m going over the query, checking the production database. Why is this manifesting like this? It all looks correct.

2 HOURS of broken sales, pissed off customers, pissed off service agents and I see that there was still one reference of “new_order” that should have been “new_order2”.

I am a piece of shit.

After months of development, testing, testing and even more testing the app was ready for deployment to production. Happy days, the end was in sight!

I had a week's leave so I handed over the preparation for deployment to my Senior Developer and left it in his capable hands while I enjoyed the sun and many beers.

I came back on the day of deployment and proudly pressed the deploy button. Hurrah!

Not long after I got loads of phone calls from around the country as the app wasn't working. What madness is this?! We tested this for months!

Turns out my Senior didn't like the way I'd written the SQL queries so he changed them. Which is obviously both annoying and unprofessional, but even worse he got a join wrong so the memory usage was a billion times more and it drained the network bandwidth for the whole site when I tried to debug it.

I got all the grief for the app not working and for causing many other incidents by running queries that killed the network.

So...much... rage!!!

So you want full stack engineers to: design, do UX, create front end, build backend and deploy it in your mono repo stupid manual deployment "kubernetes cluster", add monitoring alerting manually, review others PR, QA our own apps and features, manually sync to Production, use VPN otherwise we cannot connect to anything, 2factor auth, do SRE, architecture diagrams, demo, run agile ceremonies, and learn a legacy coding language which was never mentioned in the job description. Did I miss anything?

it was 12am when we are ready to launch our new web design which requires a lot of hardwork and routing processes. my team lead was the one who pushes the button to production using "cap production deploy" command. everyone in the room (including PM) was like counting down like launching a rocket to space. the feeling is great knowing that everyone was sleepy at that time. im glad it went smooth and everyone congratulates each other.

Been there for two weeks

[Team lead] why didn't you deploy to production like i told you, while i was sick?
[Me] nobody told me i should do that
[TL] i wrote you on slack
[Me] I'm pretty sure you didn't
*TL scrolls through history, can't find proof*
[TL] okay I can't find anything, I probably told you in a hangout call

He did not, I would remember that...

What kind of rusty asshole develops an FTP client which seemingly treats uppercase and lowercase filenames as exactly the same and is not able to fucking understant UTF-8 filenames!?
OK or maybe it was the shitty ass server to which I had to deploy the website to.
I've never been so pissed in my life.
It's already an asshole torture to upload 2.3 giggle bytes of pixel jizz, but 5 hours later, when the site has been made public, you find out that 25% of these images' filenames were automatically renamed during the extraction because some asshole dev thought it was a great idea to not even inform the user about this behaviour.
Fixing filenames in production while your boss is really pissed next to you the hole time is not a great feeling. Especially when you accidentally purge the whole image cache and the PHP image transform task then blocks thus making the whole site not loading any more images for 40 minutes.

WHAT AN ASSRAPE!

Please don't comment. I'm still too pissed to read comments. Thanks.

“PHP is evil” is not just a joke.

PHP is usually percieved as a language which is not so consistent and has some opinionated historical aspects but allows rapid development because it’s easy. They say PHP doesn’t focus on that “purist shit” such as concepts and “just gets things done”.

Hovewer, this is not true. PHP lures you in and lies to you promising saving time on development, but everything, and I mean EVERYTHING written in PHP is doomed to turn into a bloody mess sooner or later.

You have to be an AI to manage the growing PHP codebase and add features without breaking anything. With every feature it gets harder and harder. If you’re still a human managing a human team, you have to enforce guidelines. Automatic error preventon measures are made of code themselves so the cost of deploying them ona late stage can be ridiculous. And you never deploy them on early stage because you want to “save time”. Your people have to spend more and more time everyday checking on that guidelines. Your development process only becomes slower and slower. If you try to push things, your project will crumble to dust.

To make PHP at least decent, you have to figure out all this by yourself on an early stage. When you’re done, you spent a lot of time creating the buggy, ad-hoc, unspecified and unsupported alternative of what works out of the box in other languages. And you still code in PHP and still have all its disadvantages in your project’s DNA.

PHP is evil because it promises and never delivers. PHP is evil because it lies to you and it already fucked over so many of us.

If you want to code in PHP, do it under your pillow. Code your own silly projects.

If your project has the word “production” somewhere in its plans, PHP is not the way to go.

Amen.

This is the craziest shit... MY FUCKING SERVER JUST SET ON FIRE!!!

Like seriously its hot news (can't resist the puns), it's actually really bad news and I'm just in shock (it's not everyday you find out your running the hottest stack in the country :-P)... I thought it slow as fuck this morning but the office internet was also on the fritz so I carried on with my life until EVERYTHING went down (completely down - poof gone) and within 2 minutes I had a technician from the data centre telling me that something to do with fans had failed and they caught fire, melted and have become one with the hardware. WTF? The last time I went to the data centre it was so cold I pissed sitting down for 2 days because my dick vanished.

I'm just so fucking torn right now because initially I was absolutely fucking ecstatic - 1 week ago after a year of doomsday bitching about having a single point of failure and me not being a sysadmin only to have them look at me like I'm some kind of techie flat earther I finally got approval to spend around 5x more per month and migrate all our software to containerized micro services.

I'll admit this is a bit worse than I expected but thanks to last week at least I have recent off site images of the drives - because big surprise I have to set this monolithic beast back up (No small feat - its gonna be a long night) on a fresh VPS, I also have to do it on premises or the data will only finish uploading sometime next week.

Pro Tip: If your also pleading for more resources/better production environment only to be stone walled the second you mention there's a cost attached be like me - I gave them an ultimatum, either I deploy the software on a stack that's manageable or they man the fuck up and pay a sys admin (This idea got them really amped up until they checked how much decent sys admins cost).

Now I have very flexible pockets because even if I go rambo the max server costs would only be 15-20% of a sys admins paycheck even though that is 13 x more than our current costs.

about to deploy half baked project on production. wish me luck

I don't know if I'm being pranked or not, but I work with my boss and he has the strangest way of doing things.
- Only use PHP
- Keep error_reporting off (for development), Site cannot function if they are on.
- 20,000 lines of functions in a single file, 50% of which was unused, mostly repeated code that could have been reduced massively.
- Zero Code Comments
- Inconsistent variable names, function names, file names -- I was literally project searching for months to find things.
- There is nothing close to a normalized SQL Database, column ID names can't even stay consistent.
- Every query is done with a mysqli wrapper to use legacy mysql functions.
- Most used function is to escape stirngs
- Type-hinting is too strict for the code.
- Most files packed with Inline CSS, JavaScript and PHP - we don't want to use an external file otherwise we'd have to open two of them.
- Do not use a package manger composer because he doesn't have it installed.. Though I told him it's easy on any platform and I'll explain it.
- He downloads a few composer packages he likes and drag/drop them into random folder.
- Uses $_GET to set values and pass them around like a message contianer.
- One file is 6000 lines which is a giant if statement with somewhere close to 7 levels deep of recursion.
- Never removes his old code that bloats things.
- Has functions from a decade ago he would like to save to use some day. Just regular, plain old, PHP functions.
- Always wants to build things from scratch, and re-using a lot of his code that is honestly a weird way of doing almost everything.
- Using CodeIntel, Mess Detectors, Error Detectors is not good or useful.
- Would not deploy to production through any tool I setup, though I was told to. Instead he wrote bash scripts that still make me nervous.
- Often tells me to make something modern/great (reinventing a wheel) and then ends up saying, "I think I'd do it this way... Referes to his code 5 years ago".
- Using isset() breaks things.
- Tens of thousands of undefined variables exist because arrays are creates like $this[][][] = 5;
- Understanding the naming of functions required me to write several documents.
- I had to use #region tags to find places in the code quicker since a router was about 2000 lines of if else statements.
- I used Todo Bookmark extensions in VSCode to mark and flag everything that's a bug.
- Gets upset if I add anything to .gitignore; I tried to tell him it ignores files we don't want, he is though it deleted them for a while.
- He would rather explain every line of code in a mammoth project that follows no human known patterns, includes files that overwrite global scope variables and wants has me do the documentation.
- Open to ideas but when I bring them up such as - This is what most standards suggest, here's a literal example of exactly what you want but easier - He will passively decide against it and end up working on tedious things not very necessary for project release dates.
- On another project I try to write code but he wants to go over every single nook and cranny and stay on the phone the entire day as I watch his screen and Im trying to code.

I would like us all to do well but I do not consider him a programmer but a script-whippersnapper. I find myself trying to to debate the most basic of things (you shouldnt 777 every file), and I need all kinds of evidence before he will do something about it. We need "security" and all kinds of buzz words but I'm scared to death of this code. After several months its a nice place to work but I am convinced I'm being pranked or my boss has very little idea what he's doing. I've worked in a lot of disasters but nothing like this.

We are building an API, I could use something open source to help with anything from validations, routing, ACL but he ends up reinventing the wheel. I have never worked so slow, hindered and baffled at how I am supposed to build anything - nothing is stable, tested, and rarely logical. I suggested many things but he would rather have small talk and reason his way into using things he made.

I could fhave this project 50% done i a Node API i two weeks, pretty fast in a PHP or Python one, but we for reasons I have no idea would rather go slow and literally "build a framework". Two knuckleheads are going to build a PHP REST framework and compete with tested, tried and true open source tools by tens of millions?

I just wanted to rant because this drives me crazy. I have so much stress my neck and shoulder seems like a nerve is pinched. I don't understand what any of this means. I've never met someone who was wrong about so many things but believed they were right. I just don't know what to say so often on call I just say, 'uhh..'. It's like nothing anyone or any authority says matters, I don't know why he asks anything he's going to do things one way, a hard way, only that he can decipher. He's an owner, he's not worried about job security.

This was some time ago. A Legendary bug appeared. It worked in the dev environment, but not in the test and production environment.

It had been a week since I was working on the issue. I couldn't pinpoint the problem. We CANNOT change the code that was already there, so we needed to override the code that was written. As I was going at it, something happened.

---
Manager: "Hey, it's working now. What did you do?"
Me: *Very confused because I know I was nowhere close to finding the real source of the problem* Oh, it is? Let me check.
Also me: *Goes and check on the test and prod environment and indeed, it's already working*
Also me to the power of three: *Contemplates on life, the meaning of it, of why I am here, who's going to throw out the trash later, asking myself whether my buddies and I will be drinking tonight, only to realize that I am still on the phone with my manager*
Me again: "Oh wow, it's working."
Manager: "Great job. What were the changes in the code?"
Me: "All I did was put console logs and pushed the changes to test and prod if they were producing the same log results."
Manager: "So there were no changes whatsoever, is that what you mean?"
Me: "Yep. I've no idea why it just suddenly worked."
Manager: "Well, as long as it's working! Just remove those logs and deploy them again to the test and prod environment and add 'Test and prod fix' to the commit comment."
Me: "But what if the problem comes up again? I mean technically we haven't resolved the issue. The only change I made were like 20 lines of console logs! "
Manager: "It's working, isn't it? If it becomes a problem, we'll work it out later."
---

I did as I was told, and Lo and Behold, the problem never occurred again.

Was the system playing a joke on me? The system probably felt sorry for me and thought, "Look at this poor fucker, having such a hard time on a problem he can't even comprehend. That idiotic programmer had so many sleepless nights and yet still couldn't find the solution. Guess I gotta do my job and fix it for him. I'm the only one doing the work around here. Pathetic Homo sapiens!"

Don't get me wrong, I'm glad that it's over but..

What the fuck happened?

Quick reminder: don't deploy to production on you last working day before holidays :)
Wish you all happy new year)
GG!

When a salty Ruby developer deploy codes to production

Earlier this year I had to deploy an "emergency" fix to production for (luckily) an internal facing, but customer impacting, web application.

It was only the login page they were changing. I backed up the original, copied the new file into place, and marked my task complete.

Then I went and read the details on the incident. Someone discovered that if you supply ANY valid username and leave the password blank, you're in! Put the wrong password and you're blocked, of course. But blank? You must be legit!

Curious, I looked at the timestamp on the original file I had backed up to see how long it had been like this.

4 years.

I work on a warehouse dev team. One day this past year, I was trying to deploy a new build to a QA server. Earlier that day I had been looking at the logs on the production server and had left the ssh session open. I had been working for less than a year out of college at this point and shouldn't have had access to deploy to the production server.

Long story short I deployed my QA build to the production server and saw there were problems connection to our production database. Then my heart dropped in my chest as I realized I had just brought down our production server.

I managed to get the server back up by rolling back in about 5 minutes and no one ever knew except some people on my team.

I felt horrible for the longest time. Later in the year another guy that joined my team that has about 20 years of experience under his belt did the exact same thing, but needed help rolling it back. Needless to say, that made me feel a lot better. 😂

Definitely the worst moment of my year.

Once we got an urgent requirement to add double hashing the password in a web application. It had to go to the production ASAP. The developer which was working on it, added 2 alerts in Javascript to display entered password and encrypted password. Finally change was ready to deploy but in hurry she forgot to remove the alerts. In rush and excitement, that change was shipped to the production. The alert says 'your password is 123', 'your password is xyz'.

After some time got phone calls from users and manager. Manager said, 'how the hell our application got HACKED? If anything happens to..........'. To cut it short, he was furious. We knew exact reason and solution. Didn't take couple of minutes to resolve this issue.

But it was funny mistake and that released that days pressure off.

"Hey, we got this new untested artifact a few minutes ago. Can you deploy it to the 2500 POS in production and stay at the office in case something goes wrong?"

My old boss used to deploy sometimes at the last second. I was about to walk out of the office (friday at around 5:30pm) when he said "Oh wait i wanted to deploy this website you've created" Before i could argue he deployed the website and ofcourse it broke in production..
This caused me to stay for another hour and a half in order to fix it...

It's Friday afternoon sooo... let's deploy to production and go on vacation for a month. 😜

I started learning Golang today and really like it.

The error handling is *excellent*. It always works the same way and is standardized, unlike the hell that NodeJS error handling is (.catch(), try).

Modules confused the fuck out of me. I eventually figured out how they worked, but Go really doesn't try to make it easy to have multiple source folders...

I'll probably be re-writing my Discord Bot in Golang soon. Being able to have just one binary output will make things infinitely easier. Compile-time variables are another feature that's nice and easy to implement.

The goal is only having to upload a single binary to deploy on production from my CI script that has all keys and stuff inside. Feels good to finally throw all that old bad JS code out and starting completely fresh.

So i just saw a post about not pushing to production on Friday's and that reminded me of something.

A team we are working with (we needed help on something so we got another company to send us a team and do it for us) recently push to production on a Friday at around 5pm.

We didn't know that they had done it until a shit ton of errors started happening and we had no idea why (it was working so far, none of us did anything, so what the fuck is going on).

To make things better, one of my colleagues tried calling them and they wouldn't answer the phone. They knew what they did, so they went home before we could notice it and we had to correct their mistake.

We had everyone calling us saying they had gotten an error, and they needed to get home but couldn't because of it (it messed up a process that only happens at the end of the day before people go home. I can explain more in the comments if anyone is confused, or just wants to know).

Eventually everything was solved, but that was a very stressful ending of the week.

So yeah, don't deploy on Friday's please and thank you

If you don't know, there are 2 types of bug fixes:

Hot Fix - Patch files directly on the production

Quick Fix - Deploy fix on production and then test it

So last week I really fucked up

I had this new implementation that was supposedly to be integrating smoothly into the rest of the service. It depended on a serialized model made by a data scientist. I test it in local, in QA environment: no problem.

So, Friday, 4pm, I decide to deploy to production. I check once from the app: the service throw an error. Panic attack, my chief is at my desk, we triy to understand what went wrong. I make calls with cUrls: no problem. Everything seems fine. I recheck from the app again: no problem.
We dedice to let it in prod, as the feature work. I go get some beers with the guys, to celebrate the deploy.

Fast-forward the next morning, 11am, my phone ring: it's a colleague of my chief. "Please check Slack, a client is trying to use the feature, it's broken"

FUUUUUUUUUUUUCK!!!

Panic attack again. I go to the computer, check the errors: two types of errors. One I can fix, the other from a missing package on the machine that the data guy used.

Needless to say, I had a fairly good weekend.

Lessons learned:
- make sure Dev, QA and Prod are exactly the same (use Ansible or Container)
- never deploy on a Friday afternoon if you don't have a quick way to revert

!rant

Super awesome day today.

1. Got up early to do a risky production deploy and it worked!
2. Three PRs approved before lunch.
3. Got some time to continue learning scala.
4. Coffee and cupcakes with some refugees and discussed work as a software engineer.
5. Tried virtual reality for the first time. Really fun.
6. Helped prepare our goals for this quarter and present them to the department.
7. Department meeting had free local craft beer and pretzels.
8. Went bouldering after work and flashed a 6c.
9. Curled up with my wife watching Netflix.

I really love my life sometimes.

Things that seem "simple" but end up taking a long ass time to actually deploy into production:

1. Using a new payment processor:

"It's just a simple API, I'll be done in 2 hours"

LOL sure it is, but testing orders and setting up a sandbox or making sure you have credentials right, and then switching from test to life and retesting, and then... fuck

2. Making changes to admin stats.

"'I just have to add this column and remove that one... maybe like a couple of hours"

YOU WISH

3. Anything Javascript

"Hah, what, that's like a button, np"

125 minutes later...

console.log('before foo');
console.log(this.foo)
etc..

TL;DR Dear boss, firstly, you always get someone to review anything important done by a fucking intern.
Secondly, you do not give access to your fucking client's production server to an intern.
Thirdly, you don't ask your fucking intern to test the intern's work that has not been reviewed by anyone directly on your client's fucking production server.

Last week, the boss and one of the lead devs (the only guy with some serious knowledge about systems and networking) decided to give me (an intern who barely has any work experience) the task of fixing or finding an alternate solution to allowing their support team access to their client machines. Currently they used a reverse SSH tunnel and an intermediary VH but for some reason, that was very unreliable in terms of availability. I suggested using OpenVPN and explained how it would work. Seemed to be a far better idea and they accepted. After several days of working through documentations and guides and everything, I figured out how OpenVPN works and managed to deploy a TEST server and successfully test remote access using two VMs. On seeing my tests, the boss told me that he wanted to test it on the client network. I agreed. Today he comes to me and he tells me to prepare testing for tomorrow and that the client technician is going to give me access to one of their boxes. And then he adds, "It's a working prod server. We'll see if we can make it work on that" and left. I gaped at him for a while and asked another dev guy in the room if what I heard was right. He confirmed. Turns out, the lead dev and the boss's son (who also works here) had had a huge argument since morning on the same issue and finally the dev guy had washed it off his hands and declared that if anything goes wrong from testing it on production, it's entirely the boss's own fault. That's when the boss stepped in and approached me. I ran back to his office and began to explain why prod servers don't top the list of things you can fuck around with. But he simply silenced me saying, "What can go wrong?" and added, "You shouldn't stay still. You should keep moving". Okay, like firstly what the fuck and secondly, what the fuck?.
Even though OpenVPN client is not the scariest thing to install, tomorrow's going to be fun.

Don't drink an deploy (production)

Just a tip I read today :
Don't deploy anything to production this week. So u don't endup debugging during the holiday and abandoning ur friends and family. ..
Happy holidays 🎉

When u r about to deploy to production. God damn luck every single time. 😒😒😒

Most intense day for me was at the very start of my career. Internship... went with product manager to client's office while PM installed new test version of our product for on-site integration testing. Shortly after deploy, client manager came over to ask why production had gone down...

Turns out that manually typing DB name as part of deployment script is not, erm, risk free. PM entered production DB name and took out a very busy call centre for a few hours. Agents in tears, customers raging on phones, etc! After we restored and got everything back up and running, he reached me the keyboard and said "You're doing it this time."

My attempt was problem free, thankfully. Earned many brownie points that day.

First day back from holiday: after 30 minutes of work (excludes start-up, catch-up etc.) The P.O. (product owner) comes to me

Telling me I needed to switch project, ok I thought at least they switched the project from what ever it was to a propper OTAP street while I was away

Few tickets later the P.O. asked me if tickets x could be deployed to our test servers as well as production.. (note the ticket was already merged with our develop branch and he wanted only that single ticket x to be deployed)

WTF is the point of a OTAP street if you're going to deploy it to every server type at once?

So day one after my holiday I already needed to fight the P.O. again

At least I wasn't disturbed during my vacation... Witch is a first.

Living on the edge!

One or two years ago I managed to deploy a DDL change directly on the production server. As I knew there was a backup job which will run every day at noon and at midnight. So I run my script some minutes after noon. So far so good. But somehow I tested it badly in my test environment and the UI of the application throws error after error now in production.

Well, just revert the db to the latest recovery point with the backup, I thought.

It became clear then after a couple of minutes of searching the backup folder for the db backup that there was no such file. The youngest backup file was 3 years old.

Now what happened: The backup script had a switch "simulate=true" and then simulated a successful backup on each run. Therefore the monitoring system got no alerts for not correctly executing those jobs correctly. Then the monitoring job which should do the backupfolder surveillance stuck with green, because there was a valid backup file inside. But it did not check for a specific creation date.

Now this database is the one we need for doing our daily business and is really crucial. Therefore It was easier to emergencyfix the application than doing a rollback of the db 🙄

Well, not really a data loss story, but close to one.

Lets fix this bug in production on a Friday afternoon. (did that three times on the same project). Never went wrong :)

I recently tried to apply the same data analytics rationale that I use at work to my personal life. This is not a rant, it is more like an data storytelling of an actual use case I would like some input on.

I set a goal - gotta thin up a bit and calm down my ticker - and got a (almost unreasonably expensive) field expert consultant to yell at me about it for a couple hours.

I unravel the metrics - there is like a million weight-related KPIs and most say nothing at all. I have never seen an non-infrastructure measurable subject that could not be resumed to 2-5 performance metrics. I got overall weight, how well my nine-years-old business suit fits me, heart rate, and day-after relative muscle pain (it will make sense soon).

Then its data-pipeline time. I bought a cheap weight scale and smartwatch, and every morning I input the data in an app. Yes, I try to put on the suit every morning. It still does not fit.

After establishing a baseline, I tried to fit different approaches. Doing equipment-free exercises, going to the gym, dieting. None was actually feasible in the long run, but trying different approaches does highlight the impacts and the handling profile of each method.

Looking at the now-gathered data, one thing was obvious - can't do dieting because it is not doable to have a shopping list and meals for me and another for the family.
Gym is also off the table - too much overhead. I spend more time on the trip there and back than actually there.
And home exercise equipment is either super crappy or very expensive. But it is also the most reasonable approach.

So it is solutions time. I got a nice exercise bycicle (not a peloton), an yoga mat (the wife already had that one) and an exercise program that uses only those two resources. Not as efficient without dieting, not as measurable and broad as the gym, but it fits my workflow. Deploy to production!

A few months pass and the dataset grows. The signal is subtle but has support - it works! The handling, however, needs improvement, since I cannot often enough get with the exercise program. Some mornings are just after some hard days.

I start thinking about what else I can improve in the program, but it is already pretty lean and full of compromises.
So I pull an engineer and start thinking about the support systems and draft profile. What else could be draining my willpower and morning time?

Chores. Getting the kids ready for school, firing up the moka pot, setting the off-brand roomba, folding the overnight-dried clothes, cooking breakfast, doing the dishes, cleaning the toilets. All part of my morning routine. It might benefit from some automation.

Last month I got that machine our elders call "wasteful" and "useless crap lazy entitled Americans invented because they feel oh-so-insulted for simply doing something by hand like everyone always did" - a "dish-washer".
Heh, I remember how hard was to convince my mother-in-law that an remote-controled electric garage door would not make she look like an spoiled brat.
Still to early to call, but I think that the dishwasher just saved me about 25 mins every morning. It might be enough to save willpower for me to do more exercise.

This is all so reflective of all data analytics cases really are out in the wild - the analytics phase seems so small compared to the gathering and practical problem-solving all around. And yet d.a. is what tells you that you are doing the wrong thing all along. Or on what you should work next.

Just needing somewhere to let some steam off

Tl;dr: perfectly fine commandline system is replaced by bad ui system because it has a ui.

For a while now we have had a development k8s cluster for the dev team. Using helm as composing framework everything worked perfectly via the console. Being able to quickly test new code to existing apps, and even deploy new (and even third party apps) on a simar-to-production system was a breeze.

Introducing Rancher

We are now required to commit every helm configuration change to a git repository and merge to master (master is used on dev and prod) before even being able to test the the configuration change, as the package is not created until after the merge is completed.

Rolling out new tags now also requires a VCS change as you have to point to the docker image version within a file.

As we now have this awesome new system, the ops didn't see a reason to give us access to kubectl. So the dev team is stuck with a ui, but this should give the dev team more flexibility and independence, and more people from the team can roll releases.

Back to reality: since the new system we have hogged more time from ops than we have done in a while, everyone needs to learn a new unintuitive tool, and the funny thing, only a few people can actually accept VCS changes as it impacts dev and prod. So the entire reason this was done, so it is reachable to more people, is out the window.

Other team lead: Hi DevOps Team, We need you to deploy this app to production. It's maintainers gave up on it in 2019, but we looked at it and it feels right.

Me: Uhm. That's not going to work. It'll fail the security scan before you can even finish the build in CI.

Other team lead: Yeah, this app is the right thing to do, and we needed it last week, but since that won't work, we'll just use this other very very infant technology that was just born yesterday. It's not stable in production, or on MySQL, or in AWS at all, but it's the other direction we can to go.

Me: What problem are you trying to solve in the first place?

Other team lead: Oh, we need access to the read from the production database.

*Finished the deploy*
*Dusts collar*
"Easy pesy"
Few hours later
*slack tone*
Production inaccessible! Blackbox crawler failed with message 5xx.
And that was the day little Charlie learnt dev-ops is not fun and thrilling.

I hate dev politics...

PM: Hey there is a weird error happening when I upload this file on production, but it works on our test environments.

Me: After looking at this error, I don't find any issues with the code, but this variable is set when the application is first loaded, I bet it wasn't loaded correctly our last deployment and we just need to reload the application.

Senior Dev: We need to output all of the errors and figure out where this error is coming from. Dump out all the errors on everything in production!!

Me: That's dumb... the code works on test... it's not the code.. it's the application.

Senior dev: %$*^$>&÷^> $

Me: Hey I have an idea! If test works... I can go ahead and deploy last week's changes to prod and dump those errors you were talking about!!

Senior Dev: OK

Me: *runs Jenkins job the deploys the new code and restarts the application*

PM: YAY you fixed it!!

Senior Dev: Did you sump put those errors like I said.

Me: Nope didn't touch a thing... I just deployed my irrelevant changes to that error and reloaded the application.

Before i go on to celebrate 2018 and get buggy at midnight - i must deploy to production.

probably every time I see my tests failing.

Each time I am writing tests I'm convincing myself "it's an investment", "spend 2 hours now to save 2 days later", "unit-tests are good".

And each time I'm chasing away ideas like "perhaps they are right, perhaps writing unit tests is a waste of time..", "this code is simple, it should ever break - why test it??", "In the 2 hours I'll spend writing those UT I could build another feature"

Yes, it is terribly annoying to write tests, especially after writing the production code (code-first approach). Why test code that you know works, right?

But after a few weeks, months or years, when the time comes to change your feature: enhance it, refactor it, build an integration with/from it, etc, I feel like a child who found a forgotten favourite candy in his pocket when I see my tests failing.

It means I did a very good job writing them
It means it was not a waste of time
it means these tests will now save me hours or days of trial-and-error change→compile→deploy→test cycles.

So yeah, whenever I see my tests fail, I feel warm and fussy inside :)

I always hate that part of the project where you finish all the coding and start trying to deploy into production for the first time.

So we are preparing to deploy the changes onto production and some fucker decides to play with the fire alarm.

Two days ago I wrote the deployment instructions. 5 lines. I sent them to the devops four days before the release (two days before usual).
A colleague of mine leashed out and had me send another message to say to ignore my instructions because they "generated too much entropy" he is releasing too his application and we should create a single instruction file. Okay, I see no reason to do that nor how that helps the devops. A longer file is not easier to understand than a smaller one.
Today the devops deploy our application. They make a backup of the new files and promptly overwrite the original copies with the files from production.
I lost 3 hours today. My colleague is refusing to communicate the error properly to the devops and I have a meeting in 20 minutes. I love my job.

Normally you would have:
- Management
- SysOps
- DevOps
- Devs

In the company (30-50 workers) we only have:
- Management
- SysOps (they don't know how to deploy apps beyond FTP to a webhost either)
- Devs

Jepp, management does not want a specific DevOps department, because he thinks every single "I just finished the Javascript course on codecademy" person knows how to deploy an app beyond dragging&dropping it to a webhost with FTP...

I tried to propose to them that I handle DevOps and teach it to others, so we can deploy code that we deem "production ready" in a more proper manner...
They refused...
They rather stick to "just use FTP to push any changes we made directly to the production server and test changes there"

Ok I'm officially losing my fucking mind!

I've been trying to solve a connection bug that only occurs in production which is cool if THIS FUCKING APP DIDN'T TAKE 30 MINUTES TO DEPLOY!!!

Been busy for 4 hours and I've only been able to test 4 minor fixes.

Thursday afternoon. Client gives us the go to deploy the latest release to production.
Friday late afternoon; my colleague - "wait, did we ever deploy"? Me - "uh, nope".
"Alright have a good weekend"

New twist on an old favorite.

Background:
- TeamA provides a service internal to the company.
- That service is made accessible to a cloud environment, also has a requirement to be made available to machines on the local network so you can develop against it.
- Company is too cheap/stupid to get a s2s vpn to their cloud provider.
- Company also only hosts production in the cloud, so all other dev is done locally, or on production non-similar infra, local dev is podman.
- They accomplish service connectivity by use of an inordinately complicated edge gateway/router/firewall/message translator/ouija board/julienne fry maker, also controlled by said service team.

Scenario:
Me: "Hey, we're cool with signing requests using an x509 cert. That said, doing so requires different code than connecting to an unsecured endpoint. Please make this service accessible to developer machines and lower environments on the internal network so we can, you know, develop."
TeamA: "The service should be accessible to [cloud ip range]"
Me: "Yes, that's a production range. We need to be able to test the signing code without testing in production"
TeamA: "Can you mock the data?"
Me: "The code we are testing is relating to auth, not business logic"
TeamA: "What are you trying to do?"
Me: "We are trying to test the code that uses the x509 you provide to connect to the service"
TeamA: "Can you deploy to the cloud"
Me: "Again, no, the cloud is only production per policy, all lower environments are in the local data center"
TeamA: "can you try connecting to the gateway?"
Me: "Yes, we have, it's not accessible, it only has public DNS, and only allows [cloud ip range]"
TeamA: "it work when we try it"
Me: "Can you please supply repro steps so we can adjust our process"
TeamA: "Yes, log into the gateway and try issuing the call from there"
Me: (╯°□°)╯︵ ┻━┻

tl;dr: Works on my server

I deployed to production in the middle of a zoom meeting with a client.

It was just a tiny change in the settings of a field that I 99% sure wouldn't produce any other major problem but we really wanted this client and we distracted them while the code was building.

Nerves racin all the way

Here: this is BDD, a Linter, and Git. CI your shit to staging before you CI to production. Never deploy on a Friday.

Don't need Netflix when you have a production deployment right before a long weekend. It has failed since last two weeks due to vulnerabilities present in one of libraries(P.S. FUCK JAVASCRIPT and Post release vulnerability scans!). You have rewritten the whole functionality from scratch twice! Security gates finally open for you, welcoming with arms wide open. So you click Deploy! DAFUQ!! FUCK MY LIFE! Deployment failed! It's only a 3 hour window to deploy! You frantically re-review your code, is it me?? Not again!! It isn't! Well, why is the deployment failing, you work against the clock. Going through configs, code, documentation! WTF is it?? Should I give up and raise a support ticket? Nope! You login to the server, sifting through logs and configs, there's a couple of other tickets with today's deadline. What are you going to do? And you get a hint! You take the hunch, change the config 5 minutes before deadline!
Get merge request approved, wait for the build, hit DEPLOY!! Nail biting 3 minutes! Your eyes fixed on the logs! Building..... Pushing instances..... Starting App..... SUCCESS!!! Finish the remaining tickets! Your long weekend still exists!

A loooong time ago...

I've started my first serious job as a developer. I was young yet enthusiastic as well as a kind of a greenhorn. First time working in a business, working with a team full of experienced full-lowered ultra-seniors which were waiting to teach me the everything about software engineering.

Kind of.

Beside one senior which was the team lead as well there were two other devs. One of them was very experienced and a pretty nice guy, I could ask him anytime and he would sit down with me a give me advice. I've learned a lot of him.

Fast forward three months (yes, three months).

I was not that full kind of greenhorn anymore and people started to give me serious tasks. I had some experience in doing deployments and stuff from my other job as a sysadmin before so I was soon known as the "deployment guy", setting up deployments for our projects the right way and monitoring as well as executing them. But as it should be in every good team we had to share our knowledge so one can be on vacation or something and another colleague was able to do the task as well.

So now we come to the other teammate. The one I was not talking about till now. And that for a reason.

He was very nice too and had a couple of years as a dev on his CV, but...yeah...like...

When I switched some production systems to Linux he had to learn something about Linux. Everytime he encountered an error message he turned around and asked me how to fix it. Even. For. The. Simplest. Error. He. Could. Google. Up.

I mean okay, when one's new to a system it's not that easy, but when you have an error message which prints out THE SOLUTION FOR THE ERROR and he asks me how to fix it...excuse me?

This happened over 30 times.
A. Week.

Later on I had to introduce him to the deployment workflow for a project, so he could eventually deploy the staging environment and the production environment by hisself.

I introduced him. Not for 10 minutes. I explained him the whole workflow and the very main techniques and tools used for like two hours. Every then and when I stopped and asked him if he had any questions. He had'nt! Wonderful!

Haha. Oh no.

So he had to do his first production deployment. I sat by his side to monitor everything. He did well. One or two questions but he did well.

The same when he did his second prod deploy. Everythings fine.

And then. It. Frikkin. Begins.

I was working on the project, did some changes to the code. Okay, deploy it to dev, time for testing.

Hm.

Error checking out git. Okay, awkward. Got to investigate...

On the dev server were some files changed. Strange. The repo was all up to date. But these changes seemed newer because they were fixing at least one bug I was working on.

This doubles the strangeness.

I want over to my colleague's desk.

I asked him about any recent changes to the codebase.

"Yeah, there was a bug you were working on right? But the ticket was open like two days so I thought I'll fix it"

What the Heck dude, this bug was not critical at all and I had other tasks which were more important. Okay, but what about the changed files?

"Oh yeah, I could not remember the exact deployment steps (hint from the author: I wrote them down into our internal Wiki, he wrote them done by hisself when introducing him and after all it's two frikkin commands), so I uploaded them via FTP"

"Uhm... that's not how we do it buddy. We have to follow the procedure to avoid..."

"The boss said it was fine so I uploaded the changes directly to the production servers. It's so much easier via FTP and not this deployment crap, sorry to say that"

You. Did. What?

I could not resist and asked the boss about this. But this had not Effect at all, was the long-time best-buddy-schmuddy-friend of the boss colleague's father.

So in the end I sat there reverting, committing and deploying.

Yep

It's soooo much harder this deployment crap.

Years later, a long time after I quit the job and moved to another company, I get to know that the colleague now is responsible for technical project management.

Hm.

Project Management.

Karma's a bitch, right?

I'm still studying computer science/programming, I still have one year to do in order to graduate (Master). I am in a work study program so I'm working for a company half of the time, and I'm studying the other half. It is important to mention that I am the only web developer of the company

When I arrived in the company 9 months ago, I was given a Vue project which had been developed by a trainee a few weeks before my arrival and I was asked to correct a few things, it was mostly about css. Then, I was ask to add a few functionalities, nothing really hard to code, and we were supposed to test the solution in a staging environment, and if everything was ok, deploy it to prod.

However, the more I did what I was asked, the more functionalities I had to implement, until I reached a point where I had to modify the API, create new routes, etc. I'm not complaining about that, that's my job and I like it. But the solution was supposed to be ready when I arrived, it was also supposed to be tested and deployed.

The problem is, the person emitting these demands (let's call him guy X) is not from the IT service, it's a future user of the website in the admin side. The demands kept going and going and going because, according to him, the solution was not in a good enough state to be deployed, it missed too many (un)necessary features. It kept going for a few months.

The best is yet to come though : guy X was obviously a superior, and HIS superior started putting pressure on me through mails, saying the app was already supposed to be in production and he was implying that I wasn't working fast enough. Luckily, my IT supervisor was aware of what was going on and knew I obviously wasn't to blame.

In the end, the solution was eagerly deployed in production, didn't go through the staging environment and was opened to the users. Now, guy X receives complaints because none of what I did was tested (it was by me, but I wasn't going to test every single little thing because I didn't have time). Some users couldn't connect or use this or that feature and I am literally drowning in mails, all from guy X, asking me to correct things because users are blocked and it's time consuming for him to do some of the things the website was doing manually.

We are here now just because things have been done in a rush, I'm still working on it and trying to fix prod problems and it's pissing me off because we HAVE a staging environment that was supposed to prevent me from working against the clock.

On a final note, what's funny is that the code I'm modifying, the pre-existing one needs to be refactored because bits and pieces are repeated sometimes 5 times where it should have been externalized and imported from another file. But I don't know when and if I will ever be able to do that.

I could have given more context but it's 4am and I'm kinda tired, sorry if I'm not clear or anything. That's my first rant

A newly joined developer (who was supposed to be very senior) comes and asks me how to write a test cos for some reason the person didn't know how to mock.

In Java,
(same for any other implementation which has an interface)
Writes Arraylist list =.....
Instead of List list = Arraylist...

Deployed code (another engineer from another country helped to deploy since this new senior dev didn't have access yet.

But the new senior dev didn't update relevant files in production code which brought down the site for nearly an hour. Mistake aside, the first reaction from this new senior dev is 'WHY DIDN'T THE DEV THAT WAS HELPING DIDN'T DO THE FILE UPDATE?'

This was followed by some other complaints such as our branching stragies are wrong. When in fact the new senior dev made a mistake by just making assumptions on our git branching strategies and we already advised on correct process.

Out of all these, guess this is the best part. The senior dev never tested code locally! Just wrote code, unit test and send to QA and somehow the test passed through. I learnt this when I realised this dev... has not even set up the local environment yet.

I keep saying new but this Senior dev been around like 3 months! This person is in another team within our larger team but shares same code base. I am puzzled how do you not set up your environment for 3 months. Don't you ask for help if you are stuck? I am pretty sure the env is still not setup.

Am I over reacting or is this one disgusting developer who doesn't even qualify for an intern let alone a senior dev? It's so revolting I can't even bring myself to offer help.

I really really hope that no one post this,a friend texted it to me and I wanted to share it because made my day.

Idk where it comes, so feel free if know where this came from to post it:

//FUN PART HERE
# Do not refactor, it is a bad practice. YOLO

# Not understanding why or how something works is always good. YOLO

# Do not ever test your code yourself, just ask. YOLO

# No one is going to read your code, at any point don’t comment. YOLO

# Why do it the easy way when you can reinvent the wheel? Future-proofing is for pussies. YOLO

# Do not read the documentation. YOLO

# Do not waste time with gists. YOLO

# Do not write specs. YOLO also matches to YDD (YOLO DRIVEN DEVELOPMENT)

# Do not use naming conventions. YOLO

# Paying for online tutorials is always better than just searching and reading. YOLO

# You always use production as an environment. YOLO

# Don’t describe what you’re trying to do, just ask random questions on how to do it. YOLO

# Don’t indent. YOLO

# Version control systems are for wussies. YOLO

# Developing on a system similar to the deployment system is for wussies! YOLO

# I don’t always test my code, but when I do, I do it in production. YOLO

# Real men deploy with ftp. YOLO

So YOLO Driven Development isn’t your style? Okay, here are a few more hilarious IT methodologies to get on board with.

*The Pigeon Methodology*
Boss flies in, shits all over everything, then flies away.

*ADD (Asshole Driven Development)*
An old favourite, which outlines any team where the biggest jerk makes all the big decisions. Wisdom, process and logic are not the factory default.

*NDAD (No Developers Allowed in Decisions)*
Methodology Developers of all kinds are strictly forbidden when it comes to decisions regarding entire projects, from back end design to deadlines, because middle and top management know exactly what they want, how it should be done, and how long it will take.

*FDD (Fear Driven Development)*
The analysis paralysis that can slow an entire project down, with developments afraid to make mistakes, break the build, or cause bugs. The source of a developer’s anxiety could be attributed to a failure in sharing information, or by implicating that team members are replaceable.

*CYAE (Cover Your Ass Engineering)*
As Scott Berkun so eloquently put it, the driving force behind most individual efforts is making sure that when the shit hits the fan, you are not to blame.

Tl;Dr Im the one of the few in my area that sees sftping as the prod service account shouldn't be a deployment process. And the ONLY ONE THAT CARES THAT THIS IS GONNA BREAK A BUNCH OF SHIT AT SOME POINT.

The non tl;dr:

For a whole year I've been trying to convince my area that sshing as the production service account is not the proper way to deploy and/or develop batch code. My area (my team and 3 sister teams) have no concept of using version control for our various Unix components (shell scripts and configuration files) that our CRITICAL for our teams ongoing success. Most develop in a "prodqa like" system and the remainder straight in production. Those that develop straight in prodqa have no "test" deployment so when they ssh files straight to actual production. Our area has no concept of continuous integration and automated build checking. There is no "test cases", no "systems testing" or "regression testing". No gate checks for changing production are enforced. There is a standing "approved" deployment process by the enterprise (my company is Whyyyyyyyyyy bigger than my area ) but no one uses it. In fact idk anyone in my area who knows HOW to deploy using the official deployment method. Yes, there is privileged access management on the service account. Yes the managers gets notified everytime someone accesses the privileged production account. The managers don't see fixing this as a priority. In fact I think I've only talk to ONE other person in my area who truly understands how terrible it is that we have full production change access on a daily basis. Ive brought this up so many times and so many times nothing has been done and I've tried to get it changed yet nothing has happened and I'm just SO FUCKING SICK that no one sees how big of a deal this. I mean, overall I live the area I work in, I love the people, yet this one glaring deficiency causes me so much fucking stress cause it's so fucking simple to fix.
We even have an newer enterprise deployment. Method leveraging a product called "urban code deploy" (ucd) to deploy a git repository. JUST FUCKING GIT WITH THE PROGRAM!!!!..... IT WAS RELEASED FUCKING 12 YEARS AGO......

Please..... Please..... I just want my otherwise normally awesome team to understand the importance and benefits of version control and approved/revertable deployments

Client: "Can you deploy this in production at 4pm?"

Me: FML!

Until today, I had assumed deploying stuff to prod would NOT be one of my responsabilities in this company. Apparently that's not the case.

Had to deploy my code and pray it didn't break anything. Why is this a big deal at all?

Well you see, there is no repository. At all. No git, no svn, not even duplicate folders. No tests, no pipeline. Just a bunch of CPanels.

Had to manually copy files and folders from the development site to the production site and partially copy a database. "Just drag and drop" were the instructions I was given.

As if using CakePHP2, PHP5 and having to parse fucking Excel files wasn't bad enough, now I have to deal with one of the worst ways to deploy code.

Fuck it, I'm switching on the looking-for-job flag on linkedin.

In August 2021 I asked my bosses for a raise for my extra work that I done for the last 6 months to create the first 4 microservices in the company and deploy them in production on a new infrastructure that is cloud based.
Today 27/01/2022 they reported how happy they are of me and that they will take my proposal *in consideration*.
Now i am searching for a new job.

Funny part: I am simple guy if they would have given me a NAS from Synology with lot of space as a reward I would actually have been happy.

P.S. jokes on them, i left 4 easter eggs hidden in the app, pipeline, db and user manual.

Live deploy gets canned because of a bug that was already on the production server.

Following my first rant, my boss had the brilliant idea of running the old and the new architecture in parallel. I had advised that it won’t be ideal since the same Scala code was ingesting into 2 different Kinesis streams and one was running an old KCL written in Java where as other was consumed by a Firehose delivery stream(eventually we will be ingesting it into Firehose directly). I had told few manual + automated tests on Code as well as from a functionality of the new architecture and a set of tests for checking the integration of the new Producer code with Consumer.

The statement I got from my boss was “This is the test, we test it on production in parallel”. My boss had a brilliant idea to fucking test the new code on the production directly but running them in parallel without accounting for undefined behaviour it might cause in the current production system. I mean my boss should get a Nobel peace prize for shattering our mental peace.

Anywho, we started the deployment today at 5AM in the morning. I had all the aws services deployed. Was just waiting to deploy the new Collector code which we did at 5AM. Immediately after 5 minutes the system went bonkers, there was fire, blood, demons and I was smoking a cigarette with the biggest “I told you so smile” on my face. I’ve just written an email to my boss and have told him calmly that “Listen motherfucker, 90 percent of the software companies aren’t idiots to focus on testing and quality. We need to start spending time on testing and quality else we’ll again be in the same soup after few weeks again”.waiting for his reply

Never! Deploy! To production! On Thursday! *banging my head against the wall*
Now I need to revert some things manually on production ON MY DAY OFF 🤦‍♂️🤦‍♂️

We are devs, we do not test, we directly deploy to production...

I have a web app that is currently running on a production server with no issues, but at the same time, it isn't working on my machine which I used to write, test, and deploy the app. The thing is, I haven't touched the code for a full month.

Now, I know this has to be logical and that there must be an reasonable explanation for all of this that I do not know yet.

However, and out of frustration, my mind wants to believe that there's some sorcery involved here or that a cosmic ray has actually penetrated the machine and messed its registers.

Damn the cosmic ray!

Sooo, turns out, management and senior PMs, technical PMs, service managers and you name it forgot an entire system.

A complete eco-system of applications, queues, services, load-balancers, deploy pipelines, databases, monitoring solutions, etc, etc, that if not handled correctly could effectively put the entire production line to a standstill.

So, waaay too late they make this discovery. In their ignorance. Just utter incompetence. Huge project. Millions of $. And they forget it. Months of meetings probably. Workshops and gettogethers at cozy hotel complex discussing ”the project”? And they do not understand some of the fundamental building blocks…

Basic engineering for these guys must mean something completely different.

I can’t even.

I am so fed up with this organization. It does not stop either.

How is this possible…

Do they even have half a brain?

BI guys ask us to avoid deploy on Fridays cause they don't have time to fix their stuff.

"We cannot test until it is live..." They said.

I hate guys who prefer production driven development.

do you do any rituals to make it psychologically easier to deploy your code into production / replace an existing working system?

(Part 1/2?)

Ohhh my god am I furious and this one's a gem.

Also I'm gonna namespoil all of the entities in my post. If this is against rant rules I'll reframe it.

So the story starts over an year ago. Me, being in a bad place, where I couldn't do a job due to external issues, wanted to try out an internship. Thought I could pull off a 5 hour shift and then attend to my problems.

THE INTERNSHALA ARC:

I apply to a bunch of applications on Angel, Internshala and Indeed.

I was contacted by a few handful of these places. One of them was called "ARCHITECTA SOFTWARE SOLUTIONS". These guys had arranged an online aptitude test for me which I promptly took.

I looked up this company and they seemed like a pretty okay big firm from the outset but didn't have many reviews on Glassdoor and likes of such. (first red flag). Post aptitude test, I was quite sure I fucked up and wouldn't get further contact. Surprisingly, a person from the company sends me his Whatsapp number over chat and asks me to save it. The message is worded like a bulk email (Starting with Hello everyone!!) which I thought was quite odd since the interaction from these platforms has always been a person-to-person contact for me. Since Internshala showed that only around 40 people applied for the position I was quite intrigued but attributed this to my lack of exp in internship operations.

THE WHATSAPP ARC:

I was contacted by the number on WhatsApp saying that they'd be interested in moving forward and I gave them my work experience details.

The person sends me over a development assignment to complete within a few days. The assignment consists of massive scope of details. I'm talking production level concept and implementation. Asks to me implement a custom emotion detection CV model (worded as "emotion camera" lmao), generate a 3d model (specified nowhere and expects to implement a mono-ocular system for the curious) and deploy it over AWS with a website to go along with it and also host that. The website should contain a VR ("360 rotatable") view that can explore the depth-map ("not worded as depth-map") of the face. My first assumption was that they had picked this work up for outsourcing and didn't bother to chip off parts so as to create an assignment out of it (I know very optimistic).

So I shoot it at him on WhatsApp asking which parts of the assignment should I do?

Him: So, which parts CAN you do?

I thought of it as an HR thing.

Me: I could do most of it but given the time-frame of the assignment and my applied position as a web developer it is perhaps out of scope for my application.

Him: Don't worry about the assignment. You can submit when you complete the whole assignment.

I was visibly angry over the stupidity of this man.

Me: This task is a Full-Stack + CV + VR task. It will take over two months to get working. Am I supposed to work on it for that long for an assignment?

Him: Okay just do the basic functionalities like add to cart. But also try to do the camera thing before next week.

At this point I'm sure that they are having trouble handling an eager client and they're offloading work to interns. So I do only the backend and minimal frontend and submit the assignment (a 2 day job done over a weekend).

Nothing. Empty. No messages since then. I tried sending in a Whatsapp message on the application and how to proceed. Then, if I could get to know if I have been rejected. Nothing.

And all this time I can clearly see the account is active as it pushes pretentious motivational quotes over it's Whatsapp status.

My boss is being a stupid cunt. To give you a background we were facing issues with our Collections system. First week December 2019, I and a colleague of mine came up with a new efficient collections architecture. My colleague and I started to Code and create automation scripts mid December and completed it in First week of Jan 2020. This PoC version was supposed to be just between the Dev team(App Dev and Back end, also one from the Ops side to verify the data). I did not receive any feedback on the actual collections system and the data integrity but during this time all they’ve done is take meetings with no real outcome. I raised this and the only email I got is data is looking fine when I know it is not.Now in First week of Feb, he is stressing us to go ahead and deploy the architecture in Production and we have not done any Code Review, Static Code analysis, any real tests on Code and deployment scripts. Have not discussed any metrics for our dashboard and alerting. I have no idea how to handle this cunt. I have even asked for resources to atleast productionalize the code and move ahead the deployment and still no out come. I’ll go in a meeting with him in an hour, I will be very blunt and tell him that whatever he is doing is a foolish way and maybe resign in couple of weeks

During my first job as software engineer I was assign to setup meta data upon implementing the meta data I accidentally forgot to remove the static og:title="Im ocabafox and Im handsome AF" and it was deploy to production. After few days someone send an email that when they share to facebook there is a text Im ocabafox and Im handsome AF. :D

Some things never change

> Have to deploy only one feature to production
> Create branch from latest tag, cherry pick the correct changes
> Mess up merge
> Almost deploy resulting broken code to production
> Realize at last moment
> Let out a loud sigh, start over

Every damn time.

Deploy Updates in the production Server, when:

- it's Friday
- after lunch
- 2 hours before closing time
- the next 5 days are Holidays

Who the hell hardcodes their localhost ports in a web.config without updating the release config to the correct production URLs? And why doesn't our ops team pick up on this shit before clicking their fancy deploy button? And why in holy heaven do we even have a pre-production server if it isn't an exact mirror of production?

God help me, I need a drink.

That's pretty awesome -_- angular decides to simply discontinue older (half a year) versions of their cli tool by overwriting the older releases with an update notice. Causing my project (jsRant) to break until I update to the newest version.

This new version requires a different configuration so I have to update my entire project just to be able to deploy to production again....

A comment from the heart: f*** you angular.

Waiting for the day when i deploy my app to the production server and everything runs as expected.

Friday, forced deploy day for last & current months work. Been stockpiled due to holiday.

Yesterday boss demoed the product to clients so they expect to test today.

Early o'fuck this morning, a coworker managed to drop all secrets and env vars from CI pipelines and trigger a deploy leaving production broken...

It's gonna be a long and busy Friday...

A developer couldn't get a application performance monitoring (APM) tool to trace his application. They claimed that their libraries and their configurations were alright and that the APM tool was non-performant.

The developer then argues with sysadmin that the APM tool can't trace the application and that there's nothing wrong with the application or the configurations. When sysadmin questions whether the developer got the tool to work anywhere, they say, "No" and head off to make it work at least in one place. They come back saying that it works on their development environment (which is their local machine). Sysadmin claims that the system configurations on the server instances cannot be matched by the development environment and there could be a lot more factors to be considered for the problem. The sysadmin asks to prove it on a server instance on one of the test environments and then they'd agree that it is a problem with the tool. They also argue that this is not the only application that uses the APM tool and the tool happily traces other applications with no issues.

The developer tries the same configuration on a staging instance and fails. In order to make it work, they silently uninstall the existing version of the APM tool and then compiles an unstable branch of the tool. It finally works with this version.

They go back to the sysadmin and show that it works on the staging environment, but does not on production. After banging their head on the wall for a while, the sysadmin figure that the tool had been swapped out for the unstable branch that was manually compiled. When questioned, the developer responds, "It works with this version on staging, so deploy the same version on production"

WTF? You don't deploy an unstable branch to production. Just because you can't make it work on the stable branch doesn't mean that it is the problem with the tool itself. There's a big difference between a stable branch and a non-stable branch. How would you feel if the sysadmin retorted by asking you to deploy the staging branch of your application to production?

Our client wants us to deploy all changes to the test server & to the production server at the same time (-___-)
So all bugs which have been founded after that should be hotfixed ASAP :/

Today, I'm making a revolutionary change to our code base. I'm finally deprecating a script that lived in this goddamn repo for way too long. There's about 10 copies of the same script in 10 different directories. The script copies all the code into a "dev" folder and then runs "sed s/prod/dev/g" on all the files, and then overwites a bunch of it with some files suffixed with ".dev". Finally, after fighting the so-called architects, the devs and everyone else that seems to have gotten used to the pain this cursed dumpster fire script has caused us, my merge request is now open and ready to go to get rid of this insanity. Now we won't have to deal with as many "surprises" that happen every goddamn time we deploy to production, overwriting all our hard work by accident, and relieve some of my OCD of having the same script in 10 different places in the repository.

After three months of development, my first contribution to the client is going live on their servers in less than 12 hours. And let me say, I shall never again be doing that much programming in one go, because the last week and a half has been a nightmare... Where to begin...

So last Monday, my code passed to our testing servers, for QA to review and give its seal of approval. But the server was acting up and wouldn't let us do much, giving us tons of timeouts and other errors, so we reported it to the sysadmin and had to put off the testing.

Now that's all fine and dandy, but last Wednesday we had to prepare the release for 4 days of regression testing on our staging servers, which meant that by Wednesday night the code had to be greenlight by QA. Tuesday the sysadmin was unable to check the problem on our testing servers, so we had to wait to Wednesday.

Wednesday comes along, I'm patching a couple things I saw, and around lunch time we deploy to the testing servers. I launch our fancy new Postman tests which pass in local, and I get a bunch of errors. Partially my codes fault, partially the testing env manipulating server responses and systems failing.

Fifteen minutes before I leave work on the day we have to leave everything ready to pass to staging, I find another bug, which is not really something I can ignore. My typing skills go to work as I'm hammering line after line of code out, trying to get it finished so we can deploy and test when I get home. Done just in time to catch the bus home...

So I get home. Run the tests. Still a couple failures due to the bug I tried to resolve. We ask for an extension till the following morning, thus delaying our deployment to staging. Eight hours later, at 1AM, after working a full 8 hours before, I push my code and leave it ready for deployment the following morning. Finally, everything works and we can get our code up to staging. Tests had to be modified to accommodate the shitty testing environment, but I'm happy that we're finally done there.

Staging server shits itself for half a day, so we end up doing regression tests a full day late, without a change in date for our upload to production (yay...).

We get to staging, I run my tests, all green, all working, so happy. I keep on working on other stuff, and the day that we were slated to upload to production, my coworkers find that throughout the development (which included a huge migration), code was removed which should not have. Team panics. Everyone is reviewing my commits (over a hundred commits) trying to see what we're missing that is required (especially legal requirements). Upload to production is delayed one day because of this. Ended up being one class missing, and a couple lines of code, which is my bad (but seriously, not bad considering I'm a Junior who was handed this project as his first task at his first job).

I swear to God, from here on out, one feature per branch and merge request. Never again shall I let this happen. I don't even know why it was allowed to happen, it breaks our branch policies. But ohel... I will now personally oppose crap like this too...

Now if you'll excuse me... I'm going to be highly unproductive and rest, because I might start balding otherwise after these weeks...

Helping out a team, I was documenting some code/processes when I came across several classes that was logging a lot of, IMO, 'junk' that was unnecessary (and I knew wasn't being used in any Splunk alerts/reports)

I offer a refactoring suggestion, simplifying the data being logged, moving the duplicate code to a central location, maybe saving 10~20 lines of code. Didn't think it was a big deal because they were already actively working on the code and it was all new code (nothing deployed to production yet). Sent the suggestion to the lead developer and he responds:

Dev: "Yes, the changes looks fine, but not in scope of the project. Any out of scope work will need to be suggested at the end of the project, reviewed by the team, the project manager and approved by the vice president."

"Out of scope"? Logging data to Splunk needs a vice president's approval? WTF?
YOU PROBABLY HAVE THE PROJECT OPEN IN VISUAL STUDIO RIGHT NOW!!!

Along with the documentation the lead dev said they didn't have time to do, I send his boss and the dev team my suggested changes (before-after screen shots of the code) and offered to do the 2 minutes worth of work (again, this was new code, nothing in production and zero side affects to anything).
I even offered to create the splunk reporting/alerting against the data being logged (another item they said they would not have time to do)

About a minute later the lead dev responds..

Dev: "Those changes look good. I'll have Jake make those changes and we can test the logging when we deploy to dev on Monday. Thanks!"

Of course you will...fracking ass hat.

I'll bet my Battlestar Galactica DVD box set he was going to make the changes himself, brag to his boss how he refactored the code, saving X lines of code..blah blah blah to help *me* with documenting the logging portion.

Meteor is a pain to deploy to production

did on my last project:
1 .Using QA env as dev env
2. Deploy in production not completely tested stuff (90% tested)
3. Run with errors in prod
4. Manual fix in prod
5. Git versioning

I promise to donate once half of my monthly gross income to charity for research on ASD, the year that I'll be able to enjoy all of my holidays without some smartass thinking that it's a great idea to deploy a huge untested upgrade to Production.

36 or sth..
Trying to fix production for xy client without them noticing someone fucked up everything with the previous deploy.. (wasn't me)..
Anyhow, managed to deploy my changes plus fix for the previous fuckup.. in the morning it all worked as it should.
Why it took me so long? Because why bother writing down what changeset was used for deploy.. It's much more fun to guess.. Multiple times.. Anyhow, I managed to figure approximate code for that deploy & merge my changes & fix everything.. + later found out looooads of uncommited changes on the guys computer.. :/ So yeah, never trust a bunneh!!

Wait. Why does this work? It doesn't copy any of the frontend code into the deploy location.

I'm not sure how this works, but it does. Crap, there goes my morning tracking down this wretched spaghetti deploy code.

At least I understand how it works in production. Shit, why is it different between production and our integ servers ,that isn't good. Maybe I can just refactor it.

That was all on Monday. It's now Wednesday and I'm still fucking refactoring something that wasn't actually broken. It just didn't make sense.

Maybe I should just revert my last three days of work on this branch and move on. No! It's too late, I've invested way too much time into this project...

... and I'm almost done, just a few more commits right?

Disclaimer: Technically it's not "our" stack, but we have to use it so....

A webapp we built runs inside the company's network we built it for. Their IT are windows lovers, so everything has to run on Windows servers, even the tablets which are used to access said web app need to have windows.
Their company network isn't accessable from the outside world, so we have access via VPN to get into their network. But this isn't enough to access that shitty windows server our software runs on. After that VPN, you have to connect to a different VPN to which you can only connect to while you're inside the company's network. Then you have access to two servers, one the application is running on and one, well to see if you're changes were deployed correctly because the production server doesn't have a browser on it other than shitty internet explorer 8.

The only way to connect to the server is using RDP. Not even samba or so. To deploy the changes we made to our app, you need to copy paste the files from your local machine to the server. And don't get me started on running mssql migration with the shitty mssql console 😤😤

Why would anyone who isn't a complete idiot use Windows for servers or mssql in the first place????

I'm given a simple assignment to update email templates. I tot it would be a breeze.

It turn out SURPRISE! After the updating of template is done. I deploy the code in the development environment.

I tried to access the email template like how the user will see to verify all is good. It turn out i am facing error.

So uhh ok, i went to check the logs to see what the hiccups. It turn out that a table is missing. But this is production code. So my question how the hell did the production environment has the table but dev don't.....

Sat here trying to decide and finalise my Dev process for Wordpress!

Roots.io clean, good code, deployment to staging and production through git

Vagrant then just push to live (which one?)

Docker then try and figure S*** out

Flywheel local!

Then decide where to deploy:

Digital Ocean or AWS Elastic with load balancers and S***!

Decisions!!

deploying the apps in production...

Devs: i'm confident enough that i can do this. Docker? wtf, i know how to do it.

after successfully deploy in production, 30 minutes later...

Devs: Hey, team lead. I can't access the DB, why?
Team Lead: what? why? what did you do?
Devs: I just successfully deploy in production using the tutum interface deploy button.
Team Lead: Did you uncheck to deploy the DB again?
Devs: Thinking.... hmmmmm No?
Team Lead: Opppsss, that's good. We can't eat our lunch until we fix it. We need to deploy the db back-up again.
Devs: Did I delete the db?
Team Lead: No? probably not you? LOL's
Devs: But who?
Team Lead: It's tutum but it's your mistake to unchecked to redeploy the db before you deploy the apps :D

DevOps / Software Engineer => IT

Ops teams that are "assuring" stability of production by being the only ones who can click to deploy to Prod add no value.

Last weekend I was working on a small project for a friend of mine: a dockerized webapp, plus API backend and DB. I had some problems with the installation on the vps and had to try out different images and never really did a complete setup of my usual dotfiles. Got it running on an Ubuntu distro. Everything great.
It was the first release so I still had to check that every configuration worked ok, like letsencrypt companion container, the reverse proxy and all that stuff, so I decided to clone the whole project on the server tho make the changes there and then commit them from there.
Docker compose, 10 lines of code, change the hosts and password. Boom everything working. Great... Except for the images in the webapp.

WTF? Check the repo, here they are, all ok. I try different build tactics. Nothing. Even building the app on another docker always the same. Checked browser cache, all the correct ports are open. I even though that maybe react was still using some weird websocket I didn't know, but no.
Damn, I spent 5 hours checking why the f*** the server wouldn't make it out.

Then, finally, the realization...

I didn't install the f******* git-lfs plugin and all I was working with were stupid symbolics links! Webpack never even throw an error for any of the stupid images and the browser would only show a corrupted image, when decoding the base64 string.

Literally the solution took 5 minutes.
F*** changes on production, now I do everything on a fully automated CI.

Me to my team: demo to the client is postponed, we'll show it the day after tomorrow.

Them: nice, then we can put in production also the new feature xyz.

Me: mmm... Is it tested and everything ok? Then yes, let's deploy it.

Bad decision. Now everything is not working. Rollback needed!

my stupid ass workplace use the same build server for production and dev. we daily deploy broken shit to out clients. what the fuck

Soo, lets talk about deadlines and how often I come in trouble when there is one near. This week I have (had) three. So I had a lot of problems this week.

But, after all today was a really good day.

I deleted an production database, fucked up my laravel migrations and tomorrow we have to deploy another application. The app is still in development and the site isn´t finished yet.

Friday I need to launch another webapplication. Because I don't want to deploy webapps and apps on friday and I go snowboarding next week, I somehow convinced my client to launch it next week. He agreed with me, so that's a burden off my shoulder. And I'll have an extra day for fixing the last bugs.

Today I worked really hard and almost finished the last application. So I think after all I'll somehow manage it.

Where to begin?
- I purposely overestimate because I can get away with it in current job
- I test in production like 50% of time in current job
- I deploy during working hours
- My javascript is a sphagetti monster in current intranet implementation and I am sorry for the next future maintainer after I'm gone...

Recently many of us may have seen that viral image of a BSOD in a Ford car, saying the vehicle cannot be driven due to an update failure.
I haven't been able to verify the story in established news sources, so I won't be further commenting on it, specifically.
But the prospects of the very concept are quite... concerning.

Deploying updates and patches to software can be reasonably called *the software industry*. We almost have no V0 software in production nowadays, anywhere (except for some types of firmware).
Thus, as car and other devices become more and more reliant on larger software rather than much shorter onboard firmware, infrastructure for online updates becomes mandatory.
And large scale, major updates for deployed software on many different runtime environments can be messy even on the most stable situations and connections (even k8s makes available rolling updates with tests on cloud infrastructure, so the whole thing won't come crashing down).
Thereby, an update mess on automotive-OS software is a given, we just have to wait for it.

When it comes... it will be a mess. Auto manufacturers will adopt a "move fast and break things" approach, because those who don't will appear to be outcompeted by those who deploy lots of shiny things, very often.
It will lead to mass outages on otherwise dependable transportation - private transportation.
Car owners, the demographic that most strongly overlaps with every other powerful demographic, will put significant pressure on governments to do something about it.
Governments (and I might be wrong here) will likely adapt existing recall implementation laws to apply to automotive OS software updates.
That means having to go to the auto shop every time there is a software update.
If Windows may be used as a reference for update frequency, that means several times per day.
A more reasonable expectation would be once per month.
Still completely impossible for large groups of rural car owners.
That means industry instability due to regulation and shifting demographics, and that could as well affect the rest of the software industry (because laws are pesky like that, rules that apply to cars could easily be used to reign in cloud computing software).

Thus... Please, someone tells me I overlooked something or that I am underestimating the adaptability of the powers at play, because it seems like a storm is on the horizon, straight ahead.

When I first deployed to production and handled all the issues gracefully

WHY DOES EVERYTHING BREAK WHEN I WANT TO DEPLOY?

Finally fixed the last few bugs on my project and the thing is pretty much set to go. Then both the production and my local copy (identical in every way except for connecting to different MySQL servers) have the EXACT same query issues. The cursor fetch methods ALL get nothing EVEN WHEN I CAN SEE THE CURSOR HAS THE DATA I REQUESTED in the debugger.

I'm already in hot water for taking longer than expected with this project and this is going to push it back even further.

Used to get really mad at the testing team every time before a production deploy. They only tested the developments that same day, and still I wasn't sure they did a good job. Sometimes they asked for an extra "feature", so we entered into speed-coding mode. Got used to it.

Rules and policies are just for discussions and arguments. When you really have a problem on production, all you need a solution without any law.
You are allowed to execute Alter, Restart Server, Deploy some hacks and many more :D

Today is release day!
Got a whole set of new features to deploy on production!
Also, internet at the office has been dead since 6am.
I'll take a coffee break.

We need to deploy production on friday because the deployment process is managed via SAP tickets and requires almost every time manual intervention. Additional the indexing can only run during weekend.

Accidentally reverted the repository wiping all the non-versioned custom patches in a production client! I thought it was my computer’s terminal. Luckily we’d got a backup from last deploy. Later we customized the bash prompt in all prod servers to avoid confusions like this.

Today our PM planned to deploy in production an e-commerce based on PrestaShop.
A colleague of mine mamaged to implement everything that was necessary, and I made a small script to add random sales on random products every sunday.
We tested it several times in our environment, on multiple machines, and everything was working fine.

BUT
Today we launched the script on production server, and we was a little mistake.
"A bug? Say no more pal, I'll fix it!".

Fixed, tested on local environment, deployed and.... The first steps weren't working.

"Fatal error".
That's what I got. No exceptions, no error messages, no references.. Just "fatal error".
We spent two hours looking for the problem, thinking it was a server error that was just outputting that shitty message.

And you know what? Some fucking fat cocksucker son of a bitch thought it was an excellent idea to stop the code execution with a simple and very helpful "fatal error".

"oh, wait, there is an error here, let me print die(" fatal error"), ao the other developer will be able to find what's going on", he thought.

FUCK YOU MORON.

TL;DR: Avoid French software, they are a bounch of asshole (except some goos guy..)

My brother works in fintech sector. He had created an Options strategy after months of hard work and deployed the strategy in production via CI/CD.
However, the strategy didn't deploy automatically on Monday and few trades didn't happen leading to loss. His boss came down firing at him as to why strategy was not deployed.

Turns out the IT team had changed the password on Friday evening as per their routine password updates.

Deploy new script on production and then server time are outdated suddenly, plus old timestamp data inside db changed to outdated time. Who update the data inside db? Mindblow~

Another 3am production deploy. What's it like to have a real dev team... I legitimately forget.

Sometimes I deploy to production without actually testing the changes. At least I have my unit tests!

So recently I've been feeling like I fooled myself into thinking I'm any good at anything regarding development.

Today I tried to deploy a Console Application that would run nightly. The production systems are much more guarded, as it should be, but I should still be able to schedule a windows task (yeah yeah, windows servers, not the time Linux fanboys and not my choice :P) no problem.

Except I didn't expect that network users can't run jobs, because of a Group Policy about saving passwords on network accounts.

I expected a local administrator account to be available, and it wasn't.

Also a web API isn't available, even though I could telnet to the address on port 443 (HTTPS). A proxy apparently accepts all HTTP/HTTPS traffic and so on.

All this I feel like I should have known....

So am I in my own head, or am I right in thinking maybe I'm not "pro" development yet? Maybe I don't deserve to be "pro".

Thoughts?

Does anyone use file comparison software (Ex. Beyond Compare) on production server to deploy code?

Separation of duties.
I work in a fairly large IT department for a Healthcare company and for security reasons always having to involve application support or other teams even during development phase can be very aggravating when I have to ask for simple things like server log files. And the process to get to deploy in production is paved with bureaucracy and paperwork and emails that have little to do with anything other than just say, I approve, yet we are supposed to be trying to implement agile.

Deploys to Production.
Runtime error.
Open Development server and run in Production setting.
Still runtime error.
Fixes Error.
Error fixed on development.

while (hoursWasted < 3) {
Deploy.
Not working on Prod.
Try other fix.
Still not working, but works perfectly in dev machine.
What the fuck
}

Rage
Go take a walk
Realized I might have deployed to the wrong server
Glanced at deployment path
Realized it's at the wrong server
Reconfigure and Deploy
It works.
Fuck.

Whats been the longest build time you've had??

Finally finished my 21h shift...Deploy on production is tomorrow...Back to the chair in a few hours.

MRW I deploy to production server and forget to add a server domain in "OAuth redirect domains" in Firebase.
Before that I was debugging for 6 hours without success.