Marriage is basically deploying your relationship to production.

Oh, man, I just realized I haven't ranted one of my best stories on here!

So, here goes!

A few years back the company I work for was contacted by an older client regarding a new project.

The guy was now pitching to build the website for the Parliament of another country (not gonna name it, NDAs and stuff), and was planning on outsourcing the development, as he had no team and he was only aiming on taking care of the client service/project management side of the project.

Out of principle (and also to preserve our mental integrity), we have purposely avoided working with government bodies of any kind, in any country, but he was a friend of our CEO and pleaded until we singed on board.

Now, the project itself was way bigger than we expected, as the wanted more of an internal CRM, centralized document archive, event management, internal planning, multiple interfaced, role based access restricted monster of an administration interface, complete with regular user website, also packed with all kind of features, dashboards and so on.

Long story short, a lot bigger than what we were expecting based on the initial brief.

The development period was hell. New features were coming in on a weekly basis. Already implemented functionality was constantly being changed or redefined. No requests we ever made about clarifications and/or materials or information were ever answered on time.

They also somehow bullied the guy that brought us the project into also including the data migration from the old website into the new one we were building and we somehow ended up having to extract meaningful, formatted, sanitized content parsing static HTML files and connecting them to download-able files (almost every page in the old website had files available to download) we needed to also include in a sane way.

Now, don't think the files were simple URL paths we can trace to a folder/file path, oh no!!! The links were some form of hash combination that had to be exploded and tested against some king of database relationship tables that only had hashed indexes relating to other tables, that also only had hashed indexes relating to some other tables that kept a database of the website pages HTML file naming. So what we had to do is identify the files based on a combination of hashed indexes and re-hashed HTML file names that in the end would give us a filename for a real file that we had to then search for inside a list of over 20 folders not related to one another.

So we did this. Created a script that processed the hell out of over 10000 HTML files, database entries and files and re-indexed and re-named all this shit into a meaningful database of sane data and well organized files.

So, with this we were nearing the finish line for the project, which by now exceeded the estimated time by over to times.

We test everything, retest it all again for good measure, pack everything up for deployment, simulate on a staging environment, give the final client access to the staging version, get them to accept that all requirements are met, finish writing the documentation for the codebase, write detailed deployment procedure, include some automation and testing tools also for good measure, recommend production setup, hardware specs, software versions, server side optimization like caching, load balancing and all that we could think would ever be useful, all with more documentation and instructions.

As the project was built on PHP/MySQL (as requested), we recommended a Linux environment for production. Oh, I forgot to tell you that over the development period they kept asking us to also include steps for Windows procedures along with our regular documentation. Was a bit strange, but we added it in there just so we can finish and close the damn project.

So, we send them all the above and go get drunk as fuck in celebration of getting rid of them once and for all...

Next day: hung over, I get to the office, open my laptop and see on new email. I only had the one new mail, so I open it to see what it's about.

Lo and behold! The fuckers over in the other country that called themselves "IT guys", and were the ones making all the changes and additions to our requirements, were not capable enough to follow step by step instructions in order to deploy the project on their servers!!!

[Continues in the comments]

USER: I can't see any data in the page...!
ME: ok, I'll do a check
ME: API calls get no data back. Boss, did you change anything and put it in production?
BOSS: Absolutely not, I just modified the name of what was the "Family" parameter in "Type".
ME: Seems legit. Totally agree. I'm going to lunch. Can you check in the meanwhile why calling the API with "Family" does return nothing? Thanks.

Years ago, when I was a young developer, I was asked to design and implement a complex but usefull feature. It took me some time, but I was really proud of my creation. It passed all the tests and was approved for production deployment. However, while the code was deployed, the customer asked for it to remain disabled for a while.

Months passed, other features were added often breaking the one I created. But I was relentless. I've fixed it every time. I've kept it ready for great launch...

After a few years, when I was already working on a different project, I received an email about my feature. It turned out that everyone simply forgot about it. You might expect that it was finally turned on... Nope, that email was a kind request to remove that feature from code.

When we push directly to production

When everything works fine in localhost but in production

Worst thing you've seen another dev do? Long one, but has a happy ending.

Classic 'Dev deploys to production at 5:00PM on a Friday, and goes home.' story.

The web department was managed under the the Marketing department, so they were not required to adhere to any type of coding standards and for months we fought with them on logging. Pre-Splunk, we rolled our own logging/alerting solution and they hated being the #1 reason for phone calls/texts/emails every night.

Wanting to "get it done", 'Tony' decided to bypass the default logging and send himself an email if an exception occurred in his code.

At 5:00PM on a Friday, deploys, goes home.

Around 11:00AM on Sunday (a lot folks are still in church at this time), the VP of IS gets a call from the CEO (who does not go to church) about unable to log into his email. VP has to leave church..drive home and find out he cannot remote access the exchange server. He starts making other phone calls..forcing the entire networking department to drive in and get email back up (you can imagine not a group of happy people)

After some network-admin voodoo, by 12:00, they discover/fix the issue (know it was Tony's email that was the problem)

We find out Monday that not only did Tony deploy at 5:00 on a Friday, the deployment wasn't approved, had features no one asked for, wasn't checked into version control, and the exception during checkout cost the company over $50,000 in lost sales.

Was Tony fired? Noooo. The web is our cash cow and Tony was considered a top web developer (and he knew that), Tony decided to blame logging. While in the discovery meeting, Tony told the bosses that it wasn't his fault logging was so buggy and caused so many phone calls/texts/emails every night, if he had been trained properly, this problem could have been avoided.

Well, since I was responsible for logging, I was next in the hot seat.

For almost 30 minutes I listened to every terrible thing I had done to Tony ever since he started. I was a terrible mentor, I was mean, I was degrading, etc..etc.
Me: "Where is this coming from? I barely know Tony. We're not even in the same building. I met him once when he started, maybe saw him a couple of times in meetings."
Andrew: "Aren't you responsible for this logging fiasco?"
Me: "Good Lord no, why am I here?"
Andrew: "I'll rephrase so you'll understand, aren't you are responsible for the proper training of how developers log errors in their code? This disaster is clearly a consequence of your failure. What do you have to say for yourself?"
Me: "Nothing. Developers are responsible for their own choices. Tony made the choice to bypass our logging and send errors to himself, causing Exchange to lockup and losing sales."
Andrew: "A choice he made because he was not properly informed of the consequences? Again, that is a failure in the proper use of logging, and why you are here."
Me: "I'm done with this. Does John know I'm in here? How about you get John and you talk to him like that."
'John' was the department head at the time.
Andrew:"John, have you spoken to Tony?"
John: "Yes, and I'm very sorry and very disappointed. This won't happen again."
Me: "Um...What?"
John: "You know what. Did you even fucking talk to Tony? You just sit in your ivory tower and think your actions don't matter?"
Me: "Whoa!! What are you talking about!? My responsibility for logging stops with the work instructions. After that if Tony decides to do something else, that is on him."
John: "That is not how Tony tells it. He said he's been struggling with your logging system everyday since he's started and you've done nothing to help. This behavior ends today. We're a fucking team. Get off your damn high horse and help the little guy every once in a while."
Me: "I don't know what Tony has been telling you, but I barely know the guy. If he has been having trouble with the one line of code to log, this is the first I've heard of it."
John: "Like I said, this ends today. You are going to come up with a proper training class and learn to get out and talk to other people."

Over the next couple of weeks I become a powerpoint wizard and 'train' anyone/everyone on the proper use of logging. The one line of code to log. One line of code.

A friend 'Scott' sits close to Tony (I mean I do get out and know people) told me that Tony poured out the crocodile tears. Like cried and cried, apologizing, calling me everything but a kitchen sink,...etc. It was so bad, his manager 'Sally' was crying, her boss 'Andrew', was red in the face, when 'John' heard 'Sally' was crying, you can imagine the high levels of alpha-male 'gotta look like I'm protecting the females' hormones flowing.

Took almost another year, Tony released a change on a Friday, went home, web site crashed (losses were in the thousands of $ per minute this time), and Tony was not let back into the building on Monday (one of the best days of my life).

One of our clients deploy their own server app. So this happened after a prod deployment. (4am)

*Cellphone rings while sleeping*

Client : we need you on the conference call now. URGENT!

*Gets on conference call*
*Client explain the problem*
*Explaining to the client that the problem is in their side (https connection not working, either network or certificate problem)*
*Client doesn't believe it and pushes me for a fix that I have no control on*
*4 hours later in a heated conversation*

Client : ok problem is on our side. We used our SSL certificate from staging with production and thought it would work.

Me :

You useless piece of shit. I'm here drowning in tickets and you keep using me as your personal debugger. I don't mind helping but the way you frame everything as something I left off and therefore, I should fix it, makes me not want to. I completed the code, the unit tests, the review points, fucking EVERYTHING before I left. All you had to do was create a deployment branch and merge it there when it's time to deploy. That's all.

But no, the first day I came back from vacation, you pester me with "this isn't working, blah blah blah". Did you see the pipeline? Successful, right? What about the production deployment? It passed all the reviews, the tests in all environments and no issues were reported but somehow, it's not working? It took me just a few seconds looking at your shit to figure out that you didn't even merge it, meaning you didn't follow the deployment process and waited for me to come back to submit the fucking MR for a task that's already complete but was just assigned for you to deploy. Bitch, what did you even do?

You copy/paste shit from the internet and don't even bother figuring out how they work. When they don't magically work and you get an error, you send it to me. Even when I give you the commands, if it needs the tiniest amount of tweaking, you just give up and send it back to me. What I fucking hate the most is how you phrase it in a way that it's not working because of me. It's like the new year came and you decided to be an asshole.

I'm not helping you anymore, you cunt. I have bigger issues and you already wasted the first day of my week. All your issues are one search away. I'm not reading your messages and you can tell everyone that you're not having any progress because I'm not helping you and I will gladly take those tickets from you.

I usually cringe when someone rants about a senior not knowing something because I believe that this field is pretty vast and you can't know every single thing but man, your attitude isn't even something I would like in a junior developer. You have the ego of a senior but none of the skills. How did you even get here when you couldn't figure things out yourself?

I find it so funny because when the good senior engineer who finds you disappointing started giving you development tasks, it became more clear that you really don't know what you're talking about when you rush me. Look how the tables turned, you even have the easiest tickets and yet, here you are flailing like a motherfucka.

Figure it out yourself. Fuck you.

My team handles infrastructure deployment and automation in the cloud for our company, so we don't exactly develop applications ourselves, but we're responsible for building deployment pipelines, provisioning cloud resources, automating their deployments, etc.

I've ranted about this before, but it fits the weekly rant so I'll do it again.

Someone deployed an autoscaling application into our production AWS account, but they set the maximum instance count to 300. The account limit was less than that. So, of course, their application gets stuck and starts scaling out infinitely. Two hundred new servers spun up in an hour before hitting the limit and then throwing errors all over the place. They send me a ticket and I login to AWS to investigate. Not only have they broken their own application, but they've also made it impossible to deploy anything else into prod. Every other autoscaling group is now unable to scale out at all. We had to submit an emergency limit increase request to AWS, spent thousands of dollars on those stupidly-large instances, and yelled at the dev team responsible. Two weeks later, THEY INCREASED THE MAX COUNT TO 500 AND IT HAPPENED AGAIN!

And the whole thing happened because a database filled up the hard drive, so it would spin up a new server, whose hard drive would be full already and thus spin up a new server, and so on into infinity.

Thats probably the only WTF moment that resulted in me actually saying "WTF?!" out loud to the person responsible, but I've had others. One dev team had their code logging to a location they couldn't access, so we got daily requests for two weeks to download and email log files to them. Another dev team refused to believe their server was crashing due to their bad code even after we showed them the logs that demonstrated their application had a massive memory leak. Another team arbitrarily decided that they were going to deploy their code at 4 AM on a Saturday and they wanted a member of my team to be available in case something went wrong. We aren't 24/7 support. We aren't even weekend support. Or any support, technically. Another team told us we had one day to do three weeks' worth of work to deploy their application because they had set a hard deadline and then didn't tell us about it until the day before. We gave them a flat "No" for that request.

I could probably keep going, but you get the gist of it.

When you receive a new task to disable a feature that hasn’t been used for months and deploy the changes to production, the last thing you expect is:
> deployment successful
> 5 seconds pass
>
>
> you got mail
> why does this no work anymore

Are you fucking kidding me!

users are the ultimate testers

Whoever said "deploys in production shouldn't be made on friday evening" was unbelievably right.

Devops though

Most satisfying bug I've fixed?

Fixed a n+1 issue with a web service retrieving price information. I initially wrote the service, but it was taken over by a couple of 'world class' monday-morning-quarterbacks.
The "Worst code I've ever seen" ... "I can't believe this crap compiles" types that never met anyone else's code that was any good.

After a few months (yes months) and heavy refactoring, the service still returned price information for a product. Pass the service a list of product numbers, service returns the price, availability, etc, that was it.

After a very proud and boisterous deployment, over the next couple of days the service seemed to get slower and slower. DBAs started to complain that the service was causing unusually high wait times, locks, and CPU spikes causing problems for other applications. The usual finger pointing began which ended up with "If PaperTrail had written the service 'correctly' the first time, we wouldn't be in this mess."

Only mattered that I initially wrote the service and no one seemed to care about the two geniuses that took months changing the code.

The dev manager was able to justify a complete re-write of the service using 'proper development methodologies' including budgeting devs, DBAs, server resources, etc..etc. with a projected year+ completion date.

My 'BS Meter' goes off, so I open up the code, maybe 5 minutes...tada...found it. The corresponding stored procedure accepts a list of product numbers and a price type (1=Retail, 2=Dealer, and so on). If you pass 0, the stored procedure returns all the prices.
Code basically looked like this..

public List<Prices> GetPrices(List<Product> products, int priceTypeId)
{
foreach (var item in products)
{
List<int> productIdsParameter = new List<int>();
productIdsParameter.Add(item.ProductID);
List<Price> prices = dataProvider.GetPrices(productIdsParameter, 0);
foreach (var price in prices)
{
if (price.PriceTypeID == priceTypeId)
{
prices = dataProvider.GetPrices(productIdsParameter, price.PriceTypeID);
return prices;
}
* Omitting the other 'WTF?' code to handle the zero price type
}
}
}

I removed the double stored procedure call, updated the method signature to only accept the list of product numbers (which it was before the 'major refactor'), deployed the service to dev (the issue was reproducible in our dev environment) and had the DBA monitor.

The two devs and the manager are grumbling and mocking the changes (they never looked, they assumed I wrote some threading monstrosity) then the DBA walks up..

DBA: "We're good. You hit the database pretty hard and the CPU never moved. Execution plans, locks, all good to go."
<dba starts to walk away>
DevMgr: "No fucking way! Putting that code in a thread wouldn't have fix it"
Me: "Um, I didn't use threads"
Dev1: "You had to. There was no way you made that code run faster without threads"
Dev2: "It runs fine in dev, but there is no way that level of threading will work in production with thousands of requests. I've got unit tests that prove our design is perfect."
Me: "I looked at what the code was doing and removed what it shouldn't be doing. That's it."
DBA: "If the database is happy with the changes, I'm happy. Good job. Get that service deployed tomorrow and lets move on"
Me: "You'll remove the recommendation for a complete re-write of the service?"
DevMgr: "Hell no! The re-write moves forward. This, whatever you did, changes nothing."
DBA: "Hell yes it does!! I've got too much on my plate already to play babysitter with you assholes. I'm done and no one on my team will waste any more time on this. Am I clear?"

Seeing the dev manager face turn red and the other two devs look completely dumbfounded was the most satisfying bug I've fixed.

Friend: I just love the adrenaline rush caused by bungee jumping
Me: I just love the adrenaline rush caused by deploying untested code to production server on a Friday night

Right-Click > Push to Production...

I recently joined this big MNC after shutting down my own startup. I was trying to automate their build process properly. They were currently using grunt and I favor gulp, so I offered to replace the build process with gulp and manage it properly.
I was almost done with it in development environment and QA was being done for production.
In the meantime I was trying to fix some random bug in a chrome extension backend. I pushed some minor changes to production which was not going to affect the main site. That was in the afternoon.
This Friday my senior rushed to me. It was like he ran six floors to reach me. He asked, did you push the new build system to production, I refused. He then went to the computer nearby and opened the code.
It was Friday and I was about to leave. But being a good developer, I asked what's the problem. He told me that one complete module is down and the developers responsible for them left for the day already and are unreachable.
I worked on that module multiple times last month, so I offered my help. He agreed and we get to work.
The problem was in the Angular front end. So we immediately knew that the build process is screwed. I accidentally kept the gulp process open for anyone, so I immediately rebuilt using grunt and deployed again, but to no success.
Then I carefully analyzed all the commits to the module to find out that I was the one who pushed the change last. That was the chrome extention. I quickly reverted the changes and deployed and the module was live again. The senior asked, how did you do that? I told the truth.
He was surprised that how come that change affect the complete site too. We identified it after an hour. It was the grunt task which includes all the files from that particular module, including chrome extension in the build process.
He mailed the QA team to put Gulp in increased priority and approved the more structural changes, including more scrutiny before deployment and backup builds.
The module was down for more than 5 hours and we got to know only after the client used it for their own process. I was supposed to be fired for this. But instead everyone appreciated my efforts to fix things.
I guess I am in a good company 😉

How a Production Deployment feels like

If you see someone ranting about a colleague who made a semi-major fuckup which was not recognised during dev and stage testing and made it to production where it was discovered three days after deployment just when he went on holiday, well, that colleague was me.

"Everybody has a testing environment. Some people are lucky enough to have a totally seperate environment to run production in."
-Unknown

1. Slack. Pretty good chat app for dev companies, I use it to prevent people standing next to my desk 40 times a day.

2. Unit testing tools, especially when fully automated using a git master branch hook, something like codeship/jenkins, and a deployment service.

3. Jetbrains IDEs. I love Vim, but Jetbrains makes theming, autocompleting & code style checks with mixed templating languages a breeze.

4. Urxvt terminal. It's a bit of work at the start, but so extremely fast and customizable.

5. Cinnamon or i3. Not really dev tools, but both make it easy to organize many windows.

6. A smart production bug logger. I tend to use Bugsnag, Rollbar or Sentry.

7. A good coffee machine. Preferably some high pressure espresso maker which costs more than the CEO's car, using organic fairtrade hipster beans with a picture of a laughing south american farmer. And don't you dare fuck it up with sugar.

8. Some high quality bars of chocolate. Not to consume yourself, but to offer to coworkers while they wait for you to fix a broken deploy. The importance of office politics is not to be underestimated.

Junior dev requests for sudo access on a server instance for some package installation, gets it, figures out how to open the root shell - never goes back. They do everything on root.

Fast forward to production deployment time, their application won't run without elevated privileges. Sysadmin asks why does the application require elevated privileges. Dev answers, "Because I set it up with root" :facepalm:

Got an email earlier this week. It went something like this:

"It looks like your team still hasn't delivered the logging and monitoring solution that we asked for. Can you get it done in time for our production deployment next Friday?"

Um, wait, excuse me, WHAT?

1. You never actually asked for the thing you claim we didn't deliver. In fact, when we brought up the fact that you should probably have some monitoring set up for your servers, you said it would be handled entirely by your own team.
2. I HAVE BEEN WORKING ON THIS PROJECT FOR SIX MONTHS WHY DIDN'T YOU TELL ME YOUR DEADLINE UNTIL NOW
3. I won't even have time to start working on this until the Monday after your prod deployment date. Sorrynotsorry.

I really shouldn't be surprised though. This project has been a clusterfuck from the very beginning so this is just par for the course.

So you want full stack engineers to: design, do UX, create front end, build backend and deploy it in your mono repo stupid manual deployment "kubernetes cluster", add monitoring alerting manually, review others PR, QA our own apps and features, manually sync to Production, use VPN otherwise we cannot connect to anything, 2factor auth, do SRE, architecture diagrams, demo, run agile ceremonies, and learn a legacy coding language which was never mentioned in the job description. Did I miss anything?

I think I want to quit my first applicantion developer job 6 months in because of just how bad the code and deployment and.. Just everything, is.

I'm a C#/.net developer. Currently I'm working on some asp.net and sql stuff for this company.

We have no code standards. Our project manager is somewhere between useless and determinental. Our clients are unreasonable (its the government, so im a bit stifled on what I can say.) and expect absurd things from us. We have 0 automated tests and before I arrived all our infrastructure wasn't correct to our documentation... And we barely had any documentation to begin with.

The code is another horror story. It's out sourced C# asp.net, js and SQL code.. And to very bad programmers in India, no offense to the good ones, I know you exist. Its all spagheti. And half of it isn't spelled correctly.

We have a single, massive constant class that probably has over 2000 constants, I don't care to count. Our SQL projects are a mess with tons of quick fix scripts to run pre and post publishing. Our folder structure makes no sense (We have root/js and root/js1 to make you cringe.) our javascript is majoritly on the asp.net pages themselves inline, so we don't even have minification most of the time.

It's... God awful. The result of a billion and one quick fixes that nobody documented. The configuration alone has to have the same value put multiple times. And now our senior developer is getting the outsourced department to work on moving every SINGLE NORMAL STRING INTO THE DATABASE. That's right. Rather then putting them into some local resource file or anything sane, our website will now be drawing every single standard string from the database. Our SENIOR DEVELOPER thinks this is a good idea. I don't need to go into detail about how slow this is. Want to do it on boot? Fine. But they do it every time the page loads. It's absurd.

Our sql database design is an absolute atrocity. You have to join several tables together just to get anything done. Half of our SP's are failing all the time because nobody really understands the design. Its gloriously awful its like.. The epitome of failed database designs.

But rather then taking a step back and dealing with all the issues, we keep adding new features and other ones get left in the dust. Hell, we don't even have complete browser support yet. There were things on the website that were still running SILVERLIGHT. In 2019. I don't even know how to feel about it.

I brought up our insane technical debt to our PM who told me that we don't have time to worry about things like technical debt. They also wouldn't spend the time to teach me anything, saying they would rather outsource everything then take the time to teach me. So i did. I learned a huge chunk of it myself.

But calling this a developer job was a sick, twisted joke. All our lives revolve around bugnet. Our work is our BN's. So every issue the client emails about becomes BN's. I haven't developed anything. All I've done is clean up others mess.

Except for the one time they did have me develop something. And I did it right and took my time. And then they told me it took too long, forced me to release before it was ready, even though I had never worked on what I was doing before. And it worked. I did it.

They then told me it likely wouldn't even be used anyway. I wasn't very happy at all.

I then discovered quickly the horrors of wanting to make changes on production. In order to make changes to it, we have to... Get this

Write a huge document explaining why. Not to our management. To the customer. The customer wants us to 'request' to fix our application.

I feel like I am literally against a wall. A huge massive wall. I can't get constent from my PM to fix the shitty code they have as a result of outsourcing. I can't make changes without the customer asking why I would work on something that doesn't add something new for them. And I can't ask for any sort of help, and half of the people I have to ask help from don't even speak english very well so it makes it double hard to understand anything.

But what can I do? If I leave my job it leaves a lasting stain on my record that I am unsure if I can shake off.

... Well, thats my tl;dr rant. Im a junior, so maybe idk what the hell im talking about.

Freelance project I was working on was deployed. Without my knowledge. At 11pm. Their in-house "tech guy" thought that the preview build i gave them was good enough for deployment. Massive bug, broke their api endpoints.
Got a call at 2 in the morning,asking for a fix. I told them how it was their fault and the App they deployed had TESTING written right on the main screen.
They promised additional payment to get me to fix it asap.
Went through the commit history (thank goodness their tech guy knew git, fuck him for committing on production though) and the crash reports.
Removed three lines. All became right with the world again. 😎

Production deployment 😜😜😜

After months of development, testing, testing and even more testing the app was ready for deployment to production. Happy days, the end was in sight!

I had a week's leave so I handed over the preparation for deployment to my Senior Developer and left it in his capable hands while I enjoyed the sun and many beers.

I came back on the day of deployment and proudly pressed the deploy button. Hurrah!

Not long after I got loads of phone calls from around the country as the app wasn't working. What madness is this?! We tested this for months!

Turns out my Senior didn't like the way I'd written the SQL queries so he changed them. Which is obviously both annoying and unprofessional, but even worse he got a join wrong so the memory usage was a billion times more and it drained the network bandwidth for the whole site when I tried to debug it.

I got all the grief for the app not working and for causing many other incidents by running queries that killed the network.

So...much... rage!!!

Most ignorant ask from a PM or client?

Migrated to SharePoint 2016 which included Reporting Services, and trying to fix a bug in the reporting services scheduler, I created a report (aka, copied an existing one) 'A Klingon Walks Into a Bar', so it would first in the list and distinct enough so the QA testers would (hopefully) leave it alone.

The PM for the project calls me.
PM: "What is this Klingon report? It looks like a copy of the daily inventory report"
Me: "It is. The reporting service job keeps crashing on certain reports that have daily execution schedules."
PM: "I need you to delete it"
Me: "What? Why? The report is on the dev sharepoint site. I named the report so it was unique and be at the top of the list so I can find it easily."
PM: "The name doesn't conform to our standards and it's confusing the testers."
Me: "The testers? You mean Dan, you, and Heather?"
PM: "Yes, smartass. Can you name the report something like daily inventory report 2, or something else?"
Me: "I could, but since this is in development, no. You've already proofed out the upgrade. You're waiting on me to fix this sharepoint bug. Why do you care what I do on this server? It's going away after the upgrade."
PM: "Yea, about that. We like having the server. It gives us a place to test reports. Would really appreciate it if you would rename or delete that report."
Me: "A test sharepoint reporting services server out of scope, so no, we're not keeping it."
PM: "Having a server just for us would be nice."
Me: "$10,000 nice? We're kinda fudging on the licensing now. If we're keeping it, we will be required to be in compliance. That's a server license, sharepoint license, sql server license, and the dedicated hardware. We talked about that, remember?"
PM: "Why is keeping that report so important to you? I don't want to explain to a VP what a Klingon is."
Me: "I'm not keeping the report or moving it to production. When I figure out the problem, I'll delete the report. OK?"
PM: "I would prefer you delete the report before a VP sees it."
Me: "Why would a VP be looking? They probably have better things to do."
PM: "Jeff wants to see our progress, I'll have to him the site, and he'll see the report."
Me: "OK? You tell Jeff it's a report I'm working on, I'll explain what a Klingon is, Jeff will call me a nerd, and we all move on."
PM: "I'm not comfortable with this upgrade."
Me: "What does that mean?"
PM: "I asked for something simple and I can't be responsible for the consequences. I'll be documenting this situation as a 'no-go' for deployment"
Me: "Oookaayyy?"

I figured out the bug, deleted the 'Klingon' report, and the PM couldn't do anything to delay the deployment.

Before production deployment: Everything is running well, all bugs are fixed, serenity sets in.

Production deployment day: Fire everywhere

goddamit we don't get a fucking break

1. high severity production incident was asked to look into at the end of the day.
2. needed fix in ui.
3. fixed and deployed in 1 hour.
4. issue remained. debugging began.
5. gave up at 1 AM and went to sleep.
6. woke up at 6 and after debugging for 2 hours, identified to be a back end issue.
7. worked with back end team for the fix, and 6 hours and 3 deployments later, it worked.
8. third party vendor reported they are still not receiving one parameter from us.
9. back end team realised they forgot to ask ui to send another parameter.
10. added the parameter in ui, redeployed ui.
11. build and deployment tool broke down. got it fixed. delay of 1.5 hours.
12. finally things are in place. total time 26 hours.

13. found half bottle of vodka, leftover from last weekend. *Priceless*

'If it ain't broken, don't fix it' - Production Deployment

Friday night push to production before the bank holiday.

No good can come of this.

Okay. So my dumbass boss took this project that had a steep timeline. I told him straight up, it won't work because we won't make the timeline. If we do this, I will be the one bending over backwards to deliver. I don't like to promise and fail. I got the oh don't worry let's just try. If we don't make it that's fine. Unfortunately that's not how I work. I refuse to deliberately fail. So I say okay and we begin. I suggested open source is the fastest way to deliver bit the fucked up part is, I am the only senior dev in the team. I will be expected to reverse engineer the open source app to connect our own deployment parameters. Use tech I have never used before. Connect frontend and backend. Handle dns bullshit. I have literally been working on Vibes and coffee for the past two weeks because ofcourse I ran into so many issues. Now I have an extension for Monday and I hate to fail. So I am not sleeping or resting just working on a fucking java app I didnt build and I am expected to make it work seemlessly on our production environment. I made some progress. Deployed frontend, deployed backend. Forgot to connect production dB so I decided to go with azure database for mysql driver since we have credits on azure. Now my java app is pissing itself over ssl handshake. I generate my keystore and add it and now java socket just times out. I want to pummel somebody or a punching bag that looks like my boss.

brace yourselves!!! we are deploying this weekend... (expect more rants to follow as the deployment continues (or fucks up) )

Doing a 2am deployment to production in my batman PJs like its normal. More coffee please!

Going for production deployment..wish me luck !

A coworker changed the application deployment process. He told all three of the other developers who need deployments, but not me. We sit six feet away from each other and I've run/managed deployments for a year longer than him.

His new process doesn't work and he's blaming the dev ops team for not following it. The new process clearly doesn't fit their workflow and never could have.

The lack of deployments have caused production issues and he still won't ping dev ops to remind them about the deployment because "it's not in the new workflow".

He's been painting dev ops as incompetent at the last three retrospectives without having ever personally reminded the deployment guy.

Ugggh.

>site is production ready
>client requests new feature by end of week
>add feature and then site breaks on last day of week
>client freaks out even though we warned this is what happens when you add new features DURING LAUNCH WEEK

After many teeth clenching failed deployment to production attempts and finally realizing I forgot didn't add ssh keys

Developing in your local environment? A few bugs here and there. Deploying it to production in a completely different environment? The whole app is a bug...

It's 1 AM and I've just finished deployment on production. I should go to sleep but what I do, yes scroll through devRant.

If they followed my suggestion and went straight to debugging the server issues they would have been solved it from week 1 and everyone would have thought the migration had a minor performance hiccup. In fact, we have already done such at least twice before and nobody batted an eye.

Instead they self-labelled the migration a failure on first error, setting the stage for apologizing to the client, and put themselves on the spot for a whole staging / production signoff, replication / backup worfklow, almost a blue-green "seamless" deployment reminiscent of DigitalOcean.

Well they're not DigitalOcean, and anyone who has spent any time understanding users knows they will not participate in "new system" tests long enough to find or report issues.

So of course the migration stretched out to almost three months up until the whole reason for the migration - the rapidly escalating risk of the old provider disappearing - hit like a freight train and now they have to go through the problem of debugging the server like I told them to on week 1. Only this time they've set the client mindset against it, lost any chance of reverting, have had grave risk for data loss, and are under pressure to debug other people's code in real-time.

This is why I don't trust devs to do ops. A dev's first solution to any problem is to throw tech at it.

I'm so sick of microservice architecture... in theory it was going to make scaling elastic and deployment easier. In reality it seems to slow productivity to a 🐌 pace.

Anyone have any brilliant suggestions on how to herd these cats in production?

Client: The new page template you sent us looks different on production compared to the other pages that use the same component.

Me: Oh, that's strange since the styling is at the component level. Hmm, let me dig in to it.

Start poking around trying to figure out what I managed to screw up only to find that it looks exactly the same on local and staging. Eventually find another style sheet the client is importing on the production site to change some of the styles.

You know, a change that isn't anywhere to be found in the repo, and no one ever asked for anything to be changed. Their "Dev" decided he would hack in a fix instead of shooting me an email.

Apparently he tried changing the SCSS file but the changes weren't showing up. He changed the minified stylesheet but his changes were overwritten on the next deployment..... #howdoesSASSwork?!

Same client as my last rant so I'm not sure why I'm surprised by this. Oh well, I'll take that hourly rate.

Don't you just love it when you're in the middle of an agreed content freeze and a marketing drone demands an immediate content deployment to production because they made a blog post and it's "super urgent" that it goes live right now.

I always hate that part of the project where you finish all the coding and start trying to deploy into production for the first time.

PM schedules deployment on a Friday... I address my concern about deploying on Fridays, I am assured it won't happen again since it is a special occasion. Next week's Friday... "Guys we need the remaining tasks on jira by today, client wants the app on production by today". smh

Yesterday I deployed a change to production. One of the impacts was that the first service I created for my current company now receives 99.9% less calls and will make it deprecated during the next 2-3 months.

Was a strange feeling to watch the avg number of calls post-deployment. Somehow I even feel bad for the service now sitting there as shadow of its former self. Is that normal 😐?

Long time ago i ranted here, but i have to write this off my chest.

I'm , as some of you know, a "DevOps" guy, but mainly system infrastructure. I'm responsible for deploying a shitload of applications in regular intervals (2 weeks) manually through the pipeline. No CI/CD yet for the vast majority of applications (only 2 applications actually have CI/CD directly into production)

Today, was such a deployment day. We must ensure things like dns and load balancer configurations and tomcat setups and many many things that have to be "standard". And that last word (standard) is where it goes horribly wrong

Every webapp "should" have a decent health , info and status page according to an agreed format.. NOPE, some dev's just do their thing. When bringing the issue up to said dev the (surprisingly standard) answer is "it's always been like that, i'm not going to change". This is a problem for YEARS and nobody, especially "managers" don't take action whatsoever. This makes verification really troublesome.

But that is not the worst part, no no no.

the worst is THIS:

"git push -a origin master"

Oh yes, this is EVERYWHERE, up to the point that, when i said "enough" and protected the master branch of hieradata (puppet CfgMgmt, is a ENC) people lots their shits... Proper gitflow however is apparently something otherworldly.

After reading this back myself there is in fact a LOT more to tell but i already had enough. I'm gonna close down this rant and see what next week comes in.

There is a positive thing though. After next week, the new quarter starts, and i have the authority to change certain aspects... And then, heads WILL roll on the floor.

Most intense day for me was at the very start of my career. Internship... went with product manager to client's office while PM installed new test version of our product for on-site integration testing. Shortly after deploy, client manager came over to ask why production had gone down...

Turns out that manually typing DB name as part of deployment script is not, erm, risk free. PM entered production DB name and took out a very busy call centre for a few hours. Agents in tears, customers raging on phones, etc! After we restored and got everything back up and running, he reached me the keyboard and said "You're doing it this time."

My attempt was problem free, thankfully. Earned many brownie points that day.

Tonight, I gave birth to my first fully functional and deployment-ready dockerized application, "lestadium_web_1"!!
The big baby contains a Laravel showcase website with some React already in production! Big bonus with that, it's connected to a database, and I managed to setup some environment variables so nothing too dangerous is built within the image!

Fuck, that was exhausting but I'm so happy to finally understand how to make my stuff work, how they work and how to find some examples to get inspired from 😍

*Finished the deploy*
*Dusts collar*
"Easy pesy"
Few hours later
*slack tone*
Production inaccessible! Blackbox crawler failed with message 5xx.
And that was the day little Charlie learnt dev-ops is not fun and thrilling.

Don't need Netflix when you have a production deployment right before a long weekend. It has failed since last two weeks due to vulnerabilities present in one of libraries(P.S. FUCK JAVASCRIPT and Post release vulnerability scans!). You have rewritten the whole functionality from scratch twice! Security gates finally open for you, welcoming with arms wide open. So you click Deploy! DAFUQ!! FUCK MY LIFE! Deployment failed! It's only a 3 hour window to deploy! You frantically re-review your code, is it me?? Not again!! It isn't! Well, why is the deployment failing, you work against the clock. Going through configs, code, documentation! WTF is it?? Should I give up and raise a support ticket? Nope! You login to the server, sifting through logs and configs, there's a couple of other tickets with today's deadline. What are you going to do? And you get a hint! You take the hunch, change the config 5 minutes before deadline!
Get merge request approved, wait for the build, hit DEPLOY!! Nail biting 3 minutes! Your eyes fixed on the logs! Building..... Pushing instances..... Starting App..... SUCCESS!!! Finish the remaining tickets! Your long weekend still exists!

When your colleague doesn't add a dependency to package.json and the distribution stops working.
Am I the only one who has got such idiot colleagues? It blows my head every fucking time , I wish I could kill people.

Two days ago I wrote the deployment instructions. 5 lines. I sent them to the devops four days before the release (two days before usual).
A colleague of mine leashed out and had me send another message to say to ignore my instructions because they "generated too much entropy" he is releasing too his application and we should create a single instruction file. Okay, I see no reason to do that nor how that helps the devops. A longer file is not easier to understand than a smaller one.
Today the devops deploy our application. They make a backup of the new files and promptly overwrite the original copies with the files from production.
I lost 3 hours today. My colleague is refusing to communicate the error properly to the devops and I have a meeting in 20 minutes. I love my job.

I hate dev politics...

PM: Hey there is a weird error happening when I upload this file on production, but it works on our test environments.

Me: After looking at this error, I don't find any issues with the code, but this variable is set when the application is first loaded, I bet it wasn't loaded correctly our last deployment and we just need to reload the application.

Senior Dev: We need to output all of the errors and figure out where this error is coming from. Dump out all the errors on everything in production!!

Me: That's dumb... the code works on test... it's not the code.. it's the application.

Senior dev: %$*^$>&÷^> $

Me: Hey I have an idea! If test works... I can go ahead and deploy last week's changes to prod and dump those errors you were talking about!!

Senior Dev: OK

Me: *runs Jenkins job the deploys the new code and restarts the application*

PM: YAY you fixed it!!

Senior Dev: Did you sump put those errors like I said.

Me: Nope didn't touch a thing... I just deployed my irrelevant changes to that error and reloaded the application.

Deployment to production last night, and the whole thing is halted because the DBA didn't find it necessary to show up. 2 hours of calling everyone possible and nothing. Thanks DBA, for your prompt and timely handling of the deployment... 😑

Tl;Dr Im the one of the few in my area that sees sftping as the prod service account shouldn't be a deployment process. And the ONLY ONE THAT CARES THAT THIS IS GONNA BREAK A BUNCH OF SHIT AT SOME POINT.

The non tl;dr:

For a whole year I've been trying to convince my area that sshing as the production service account is not the proper way to deploy and/or develop batch code. My area (my team and 3 sister teams) have no concept of using version control for our various Unix components (shell scripts and configuration files) that our CRITICAL for our teams ongoing success. Most develop in a "prodqa like" system and the remainder straight in production. Those that develop straight in prodqa have no "test" deployment so when they ssh files straight to actual production. Our area has no concept of continuous integration and automated build checking. There is no "test cases", no "systems testing" or "regression testing". No gate checks for changing production are enforced. There is a standing "approved" deployment process by the enterprise (my company is Whyyyyyyyyyy bigger than my area ) but no one uses it. In fact idk anyone in my area who knows HOW to deploy using the official deployment method. Yes, there is privileged access management on the service account. Yes the managers gets notified everytime someone accesses the privileged production account. The managers don't see fixing this as a priority. In fact I think I've only talk to ONE other person in my area who truly understands how terrible it is that we have full production change access on a daily basis. Ive brought this up so many times and so many times nothing has been done and I've tried to get it changed yet nothing has happened and I'm just SO FUCKING SICK that no one sees how big of a deal this. I mean, overall I live the area I work in, I love the people, yet this one glaring deficiency causes me so much fucking stress cause it's so fucking simple to fix.
We even have an newer enterprise deployment. Method leveraging a product called "urban code deploy" (ucd) to deploy a git repository. JUST FUCKING GIT WITH THE PROGRAM!!!!..... IT WAS RELEASED FUCKING 12 YEARS AGO......

Please..... Please..... I just want my otherwise normally awesome team to understand the importance and benefits of version control and approved/revertable deployments

What is more disaster
Than manager thinking to assign a production deployment to non technical person.

Doing the deployment to production, and towards the end one of the support guys looks over at me;

"So, the website in prod is throwing some errors"

Followed by another guy:

"Yeah I'm getting the same, SQL exceptions on the page"

I stare at them panicked for a moment, when one of them goes "just kidding!". Like dude, my heart just skipped several beats!

Any one here ever had something cruel happen to them during a deployment?

My boss is being a stupid cunt. To give you a background we were facing issues with our Collections system. First week December 2019, I and a colleague of mine came up with a new efficient collections architecture. My colleague and I started to Code and create automation scripts mid December and completed it in First week of Jan 2020. This PoC version was supposed to be just between the Dev team(App Dev and Back end, also one from the Ops side to verify the data). I did not receive any feedback on the actual collections system and the data integrity but during this time all they’ve done is take meetings with no real outcome. I raised this and the only email I got is data is looking fine when I know it is not.Now in First week of Feb, he is stressing us to go ahead and deploy the architecture in Production and we have not done any Code Review, Static Code analysis, any real tests on Code and deployment scripts. Have not discussed any metrics for our dashboard and alerting. I have no idea how to handle this cunt. I have even asked for resources to atleast productionalize the code and move ahead the deployment and still no out come. I’ll go in a meeting with him in an hour, I will be very blunt and tell him that whatever he is doing is a foolish way and maybe resign in couple of weeks

A loooong time ago...

I've started my first serious job as a developer. I was young yet enthusiastic as well as a kind of a greenhorn. First time working in a business, working with a team full of experienced full-lowered ultra-seniors which were waiting to teach me the everything about software engineering.

Kind of.

Beside one senior which was the team lead as well there were two other devs. One of them was very experienced and a pretty nice guy, I could ask him anytime and he would sit down with me a give me advice. I've learned a lot of him.

Fast forward three months (yes, three months).

I was not that full kind of greenhorn anymore and people started to give me serious tasks. I had some experience in doing deployments and stuff from my other job as a sysadmin before so I was soon known as the "deployment guy", setting up deployments for our projects the right way and monitoring as well as executing them. But as it should be in every good team we had to share our knowledge so one can be on vacation or something and another colleague was able to do the task as well.

So now we come to the other teammate. The one I was not talking about till now. And that for a reason.

He was very nice too and had a couple of years as a dev on his CV, but...yeah...like...

When I switched some production systems to Linux he had to learn something about Linux. Everytime he encountered an error message he turned around and asked me how to fix it. Even. For. The. Simplest. Error. He. Could. Google. Up.

I mean okay, when one's new to a system it's not that easy, but when you have an error message which prints out THE SOLUTION FOR THE ERROR and he asks me how to fix it...excuse me?

This happened over 30 times.
A. Week.

Later on I had to introduce him to the deployment workflow for a project, so he could eventually deploy the staging environment and the production environment by hisself.

I introduced him. Not for 10 minutes. I explained him the whole workflow and the very main techniques and tools used for like two hours. Every then and when I stopped and asked him if he had any questions. He had'nt! Wonderful!

Haha. Oh no.

So he had to do his first production deployment. I sat by his side to monitor everything. He did well. One or two questions but he did well.

The same when he did his second prod deploy. Everythings fine.

And then. It. Frikkin. Begins.

I was working on the project, did some changes to the code. Okay, deploy it to dev, time for testing.

Hm.

Error checking out git. Okay, awkward. Got to investigate...

On the dev server were some files changed. Strange. The repo was all up to date. But these changes seemed newer because they were fixing at least one bug I was working on.

This doubles the strangeness.

I want over to my colleague's desk.

I asked him about any recent changes to the codebase.

"Yeah, there was a bug you were working on right? But the ticket was open like two days so I thought I'll fix it"

What the Heck dude, this bug was not critical at all and I had other tasks which were more important. Okay, but what about the changed files?

"Oh yeah, I could not remember the exact deployment steps (hint from the author: I wrote them down into our internal Wiki, he wrote them done by hisself when introducing him and after all it's two frikkin commands), so I uploaded them via FTP"

"Uhm... that's not how we do it buddy. We have to follow the procedure to avoid..."

"The boss said it was fine so I uploaded the changes directly to the production servers. It's so much easier via FTP and not this deployment crap, sorry to say that"

You. Did. What?

I could not resist and asked the boss about this. But this had not Effect at all, was the long-time best-buddy-schmuddy-friend of the boss colleague's father.

So in the end I sat there reverting, committing and deploying.

Yep

It's soooo much harder this deployment crap.

Years later, a long time after I quit the job and moved to another company, I get to know that the colleague now is responsible for technical project management.

Hm.

Project Management.

Karma's a bitch, right?

Prepared some stuff for production deployment and submitted my first merge request in this new project. It's so dumb but these things still make me nervous all the time. I checked things over and over again to see if I missed anything or fucked something up.

It's been completed a long time ago so technically, I should be able to just push the code and be done with it but it's like having a mischievous twin who fucks up your life while you're asleep. What if that motherfucker changed something and I forgot to add it, or worse remove it?

All checks done, nothing suspicious, let me push it now..

Me: *starts typing* "git push origin.."

Demon: "..master."

Me: "Sorry, what?"

Demon: "Go on, push it to master."

Me: "No, fuck you."

Demon: "Ma-ma-ma-master! Push it! Come on, you know you want to."

Me: "Even if I did, it's protected."

Demon: "Is it though? Hey, what happens if you push to a protected branch?"

Me: "I don't wanna know. Search it yourself. By the word *protected*, it's pretty damn obvious."

Demon: "You never try, you never know."

Me: "I don't have to know. Therefore, I don't have to try."

Demon: "We learn from our mistakes."

Me: "Suit yourself."

Push completed, merge request created, spends another 10 minutes verifying the changes.

10 minutes later..

Search bar: "What happens if you push to a protected branch?"

I really really hope that no one post this,a friend texted it to me and I wanted to share it because made my day.

Idk where it comes, so feel free if know where this came from to post it:

//FUN PART HERE
# Do not refactor, it is a bad practice. YOLO

# Not understanding why or how something works is always good. YOLO

# Do not ever test your code yourself, just ask. YOLO

# No one is going to read your code, at any point don’t comment. YOLO

# Why do it the easy way when you can reinvent the wheel? Future-proofing is for pussies. YOLO

# Do not read the documentation. YOLO

# Do not waste time with gists. YOLO

# Do not write specs. YOLO also matches to YDD (YOLO DRIVEN DEVELOPMENT)

# Do not use naming conventions. YOLO

# Paying for online tutorials is always better than just searching and reading. YOLO

# You always use production as an environment. YOLO

# Don’t describe what you’re trying to do, just ask random questions on how to do it. YOLO

# Don’t indent. YOLO

# Version control systems are for wussies. YOLO

# Developing on a system similar to the deployment system is for wussies! YOLO

# I don’t always test my code, but when I do, I do it in production. YOLO

# Real men deploy with ftp. YOLO

So YOLO Driven Development isn’t your style? Okay, here are a few more hilarious IT methodologies to get on board with.

*The Pigeon Methodology*
Boss flies in, shits all over everything, then flies away.

*ADD (Asshole Driven Development)*
An old favourite, which outlines any team where the biggest jerk makes all the big decisions. Wisdom, process and logic are not the factory default.

*NDAD (No Developers Allowed in Decisions)*
Methodology Developers of all kinds are strictly forbidden when it comes to decisions regarding entire projects, from back end design to deadlines, because middle and top management know exactly what they want, how it should be done, and how long it will take.

*FDD (Fear Driven Development)*
The analysis paralysis that can slow an entire project down, with developments afraid to make mistakes, break the build, or cause bugs. The source of a developer’s anxiety could be attributed to a failure in sharing information, or by implicating that team members are replaceable.

*CYAE (Cover Your Ass Engineering)*
As Scott Berkun so eloquently put it, the driving force behind most individual efforts is making sure that when the shit hits the fan, you are not to blame.

So, you have to build an app for your company. Once it is done you submit it for the test team who ignores it completely. Then they assume that the best time to test it is after the production deployment and then everything becomes a urgency.
The best part is the PM trying to make you feel guilty about leaving after 12+ hours of work because you didn't finish everything.

Following my first rant, my boss had the brilliant idea of running the old and the new architecture in parallel. I had advised that it won’t be ideal since the same Scala code was ingesting into 2 different Kinesis streams and one was running an old KCL written in Java where as other was consumed by a Firehose delivery stream(eventually we will be ingesting it into Firehose directly). I had told few manual + automated tests on Code as well as from a functionality of the new architecture and a set of tests for checking the integration of the new Producer code with Consumer.

The statement I got from my boss was “This is the test, we test it on production in parallel”. My boss had a brilliant idea to fucking test the new code on the production directly but running them in parallel without accounting for undefined behaviour it might cause in the current production system. I mean my boss should get a Nobel peace prize for shattering our mental peace.

Anywho, we started the deployment today at 5AM in the morning. I had all the aws services deployed. Was just waiting to deploy the new Collector code which we did at 5AM. Immediately after 5 minutes the system went bonkers, there was fire, blood, demons and I was smoking a cigarette with the biggest “I told you so smile” on my face. I’ve just written an email to my boss and have told him calmly that “Listen motherfucker, 90 percent of the software companies aren’t idiots to focus on testing and quality. We need to start spending time on testing and quality else we’ll again be in the same soup after few weeks again”.waiting for his reply

I left for a week and someone deployed my code to production after being completely tested by qa on a Friday night. I get into the office after being gone for a week and am told that production has been down for many customers for several days. In a panic I start troubleshooting my code with the "What the fuck did I do wrong" face.

Development and qa were in a frenzy to figure out what happened, several developers were trying to figure out what went wrong by tracing through the source code for days, fucking days!

In that while time Noone thought to roll back the code. So, I was in a bind and thought "might as well get a box". Before that I looked at the deployment instructions: only the dll's were pushed, no db or resource file changes were pushed. In 20 minutes after I got back: no more problems for any customers and everything is working fine.

SMDH.

At least I found this picture of turtles wearing raspberries.

beware of font choices in chat apps; a coworker joked in the room that "well, sure, of course it's okay to update in production in the middle of the day" and for some reason, the other coworker didn't see the quotes because of the weird font they use, and also didn't stop to think, and went ahead and ran the deployment script. In production. In the middle of the day. With active users.

The good news is that those folks who logged back in got to use the new version a whole lot earlier than anyone was expecting. :\

So, it's been a while since I've been working on my current project and I've never had the "luck" to touch the legacy project wrote in PHP, until this week when I got my first issue.

And damn, this goddamn issue. It was a bug, a very strange bug, that only happens in production and that nobody has any idea what was happening, so yeah, I didn't have anyone to ask and I got less time than usual ( because Thanksgiving ).

And thus, I have no starting point, no previous knowledge on PHP and less time! I expected a very fun week 😀 and it was beyond my expectations.

First I tried to understand what might be causing the issue, but there wasn't any real clue to star with, so no choice, time to read the flow on the code and see what are they're doing and using ( 1k line files, yay, legacy ). Luckily I got some clues, we're using a cookie and a php session variable for the session, ok, let's star with the session variable. Where it's that been initialize ? Well, spoiler alert, I shouldn't start with that, because my search end up in the login method of the API that set a that variable and for some reason in the front end app it was always false and that lead me to think that some of the new backend functions were failing, but after checking the logs I got no luck.

Ok, maybe the cookie it's the issue, I should try open the previous website on the brow...redirect to new project login, What? Why ? I ask around and it's a new feature push on Monday, ok I got Chrome Dev tools I can see which value of the cookie it's been set and THERE IT WAS it has a wrong domain! After 2 days ( I resume a lot of my pain ) I got what I've been looking for, so now I should be able to fix the bug. Then where is the cookie initialized ? In the first file the server hits whenever you tried to enter any page of the app, ok, I found the method, but it's using a function that process the domain and sets it correctly? wtf ? Then how in heaven do I get the incorrect domain ? Hello? Ok, relax, you still have one more day to fix this, let's take it easy.

Then, at the end of the Wednesday, nope I still have no clue how this is happening. I talked with the Devops guy and he explain me how this redirection happens and with what it depends on, I followed the PHP code through and nothing, everything should works fine, sigh. Ok I still have 2 days, because I'm not from US and I'm not in US, so I still have time, but the Sprint is messed up already, so whatever I'm gonna had done this bug anyhow.

Thursday ! I got sick, yay, what else could happen this week. Somehow I managed to work a little and star thinking in what external issue could affect the processing, maybe the redirection was bringing a wrong direction, let's talk with the Devops guy again, and he answer me that the redirection it was being made by PHP code, IN A FILE THAT DOESN'T EXIST IN THE REPOSITORY, amazing, it's just amazing. Then he explained me why this file might be missing and how it's the deployment of this app ( btw the Devops guy it's really cool and I will invite him a beer ) . After that I checked the file and I see a random session_star in the first line of the code, without any configuration, eureka ! There was the cause and I only need to ask someone If that line it's necessary anymore, but oh they're on holiday, damn, well I'll wait till Monday to ask them. But once and for all that bug was done for ! 🎉

What do I learn ? PHP and that I don't want any more tickets of PHP 😆.

I am so sick of this place. Production deployment shouldn't be such a massive cluster fuck of different departments and sometimes redundant configurations. Knowing who to talk to about a problem shouldn't change daily, and it shouldn't be hard to figure out.

My jvm doesn't appear to be running in production. I hunt down the fucker who is supposed to be handling this problem today...but he's out of office and no one knows who his stand in is...GODDAMNIT!

Ok so some of you have probably seen my previous rants about my computer science teacher and our project but I'm just going to summarize all of them and share with you more of my pain.

1. He edits in the production environment. Its a laravel project and he is creating test database migrations IN THE PRODUCTION ENVIRONMENT AND SWITCHING BACK AND FORTH FROM MASTER AND DEV.

2. He edits in vim and doesn't follow codestyle even though I printed him off a piece of paper and emailed it to him.

3. He doesn't have any ethic when it comes to more complex things like laravel homestead.

4. He doesnt want me to release features even though he takes really long to do them.

While I love vim and it is my editor of choice, some things should be done in an ide. This is really annoying me and I'm really just considering handing him the project if he can't follow basic outline.

Had been trying to get the latest build rolled out for the past 3 days. Every morning, I wake up around 6am, drink a nice cup of coffee, while listening to Ariana Grande (I don't know why, but for some reason, she randomly started coming on my playlist a lot) and start rolling out, and sure enough new errors start spiking, ultimately rollback.

Conclusion: don't listen to Ariana Grande when rolling out to production! 🤔

5am now and trying to do a production deployment. It just failed because boss loves making un-approved changes to prod that change entire database schemas. Argh!!

When you are new to a technology and language, but have to make full use of its feature.
You know, make tests, implement production code, planning deployment, build a CI. All by myself.
No regrets, though. Challenge accepted!

How do you guys push changes to you server. ?

I am currently pushing changes to my git repo then pulling those changes on server where I am running the application in production.

I am planning to set up a simple server, to which I will push the changes and it push the changes to the server's running in production.
Or better would be to write a script and run on production servers that will check github for changes

Me: finally we have automated deployment to production
Team Lead: No production deployment still requires manual approval
Me: ok how do you want to handle it slack, webhook, what do you suggest
Team Lead: let's do a proof of concept (POC) for this
Me: Ahh... Poc for this ?
Team Lead: you don't know sh*t ?
Me: well I know you're are creating that here

Next day team change...

Hotswapping/replacing classes on the production server
We call it "russian deployment" .. No offense

I was studying a lot the last year, i learned a lot about Machine Learning/Deep Learning, Data Gathering, Data Analysis, ETL, Model Architecture Design, Training, Fine Tuning, Backend Development, DataBases, API Development, ORMs, Rest, GraphQL, OAuth, CI/CD, Docker, Deployment to Production environments like Heroku, Git and more stuff i dont remember while writing this. I built and keep adding stuff to my Github Portafolio.

Im not able to get a job. I started looking for jobs as Data Scientists, no response never. I take a look at freelancer sites, nothing seems to fit my skills. And when there is a minimal fit, they always want a Full Stack Web Developer, i dont know Frontend Development, i dont like do it.

Dont know what to do or how to land any job.

My options aeems to be:

1.Learn Frontend Dev and work as Full Stack in underpaying freelance jobs

2.Keep applying to Remote-Only startups, but they still wants people with 3+ years of experience.

i cant work in my city, here are not any company startup hiring no one, we are 30 years in the past here.

What you do in my place?

I don't know if my boss just wants me to learn how to use a new internal deployment process or just likes giving me unnecessary low-value work to take up time...

I could and have just copied the program via SFTP and unzip it to set it up....

(This is a testing and does not need to be in production...)

I have better things I could be doing and just want to get this done and closed but ...

Note to self:
Close off ALL ways things could go wrong..

Long story short; I released a new feature, to be able to better follow up on any stock moves, their amounts, locations and even expiry dates. An older tool just bypassed that very verification and nothing was logged or taken out of stock.

~

Taking out an amount for a certain orderline has a shortcut in place to mitigate some of the mandatory steps that pickers need to take in order to verify what's being taken. This little tool only available, visible and possible for a very few select users.

I assigned some orders to one of these people, which made him think it was an urgent batch. It's only one product, for multiple orders, so he went to the location, took out the amount needed and then used the tool to quickly be able to prepare them for shipping.

This bypassed the new methods to check if the location actually had stock to take, which I had just enabled for 1 account.

Luckily I caught the miss-hap as I was monitoring that product first-hand and noticed the batch of orders was collected but the stock amount didn't update.

It was 5min before I was leaving work, so I investigated and then ran to the person in question to ask what he did; which was "I used that tool"

I facepalmed myself internally while blaming myself, as he couldn't know that it wasn't ready to use for that purpose.

The tools to fix this up are there already.. so I used that to fix some missing stock-takes manually.. Though I'll need to close that little tool for these kind of orders for sure, asap, probably when I get home, at least until I bring over its new logic to it.

Happy Tuesday? (:

Today was my first day in OPS room.
Fucking felt weird!
Literally only 1% understood I guess.
Luckily seniors were there with me.
I hope, soon will understand more ;)

Continuing my last random post. (Please don't bother to take a look at it.)
But, hey you. Yes you, get yourself one beer/whisky and cheers!!
Why? Because my first django project ran successfully in staging environment!! Ok, There were few little bugs. But I fixed most of them.
I don't drink. So please go and enjoy on behalf of me.
And don't drink too much. Keep one bottle for production deployment.
P.S. This is just a beginning of the new journey! Still, lot to learn and experience.

So, right now we upload production code by means of FTP.
I said it would be better to use continuous deployment using Docker, but they said it was overkill (I work at a small company).

Because manually uploading by means of FTP is so much better right...

DEBUG=False

It's 3am. Do you know how your production deployment is going?

This has been bothering me for a while. I have an old freelance client of mine I’ve created an web site for (his company) it was small one so I took the complete payment before deployment and I needed no contract. I deployed the complete version of the site on my server, bought the domain for his company under my name and it has been running for a year now.
Lately he had asked me to give admin privileges to his son (cs student 1y) to upload some photos of their new building. I noticed he ruined several functions on the site in doing so, but I was never paid to support that just the hosting for a year.
When I was making the design I made a simple but pretty logo as a placeholder for the site which went in production since they never gave me company logo. All good, no contract small cash all delivered, everyone happy.
Up until few days when I saw my f**king logo cut out from the site as 250px jpeg and made as a huge banner on the company building..
From my pov I would’ve never given permission to use that since its not something i’m proud of and would suggest to make a better one for a fee. I see this as stolen/unauthorized use of intellectual property. But the laws are super shitty in our country so at this point I am stuck at taking their site, domain a hostage until they pay for the logo they used or take it down or taking legal actions.. we never signed anything about that logo.

After three months of development, my first contribution to the client is going live on their servers in less than 12 hours. And let me say, I shall never again be doing that much programming in one go, because the last week and a half has been a nightmare... Where to begin...

So last Monday, my code passed to our testing servers, for QA to review and give its seal of approval. But the server was acting up and wouldn't let us do much, giving us tons of timeouts and other errors, so we reported it to the sysadmin and had to put off the testing.

Now that's all fine and dandy, but last Wednesday we had to prepare the release for 4 days of regression testing on our staging servers, which meant that by Wednesday night the code had to be greenlight by QA. Tuesday the sysadmin was unable to check the problem on our testing servers, so we had to wait to Wednesday.

Wednesday comes along, I'm patching a couple things I saw, and around lunch time we deploy to the testing servers. I launch our fancy new Postman tests which pass in local, and I get a bunch of errors. Partially my codes fault, partially the testing env manipulating server responses and systems failing.

Fifteen minutes before I leave work on the day we have to leave everything ready to pass to staging, I find another bug, which is not really something I can ignore. My typing skills go to work as I'm hammering line after line of code out, trying to get it finished so we can deploy and test when I get home. Done just in time to catch the bus home...

So I get home. Run the tests. Still a couple failures due to the bug I tried to resolve. We ask for an extension till the following morning, thus delaying our deployment to staging. Eight hours later, at 1AM, after working a full 8 hours before, I push my code and leave it ready for deployment the following morning. Finally, everything works and we can get our code up to staging. Tests had to be modified to accommodate the shitty testing environment, but I'm happy that we're finally done there.

Staging server shits itself for half a day, so we end up doing regression tests a full day late, without a change in date for our upload to production (yay...).

We get to staging, I run my tests, all green, all working, so happy. I keep on working on other stuff, and the day that we were slated to upload to production, my coworkers find that throughout the development (which included a huge migration), code was removed which should not have. Team panics. Everyone is reviewing my commits (over a hundred commits) trying to see what we're missing that is required (especially legal requirements). Upload to production is delayed one day because of this. Ended up being one class missing, and a couple lines of code, which is my bad (but seriously, not bad considering I'm a Junior who was handed this project as his first task at his first job).

I swear to God, from here on out, one feature per branch and merge request. Never again shall I let this happen. I don't even know why it was allowed to happen, it breaks our branch policies. But ohel... I will now personally oppose crap like this too...

Now if you'll excuse me... I'm going to be highly unproductive and rest, because I might start balding otherwise after these weeks...

Sat here trying to decide and finalise my Dev process for Wordpress!

Roots.io clean, good code, deployment to staging and production through git

Vagrant then just push to live (which one?)

Docker then try and figure S*** out

Flywheel local!

Then decide where to deploy:

Digital Ocean or AWS Elastic with load balancers and S***!

Decisions!!

Do any of you wonderful devRanters know of good links that cover the topic of best practises when deploying a React/Redux (or any modern web app) to production?

I’m pretty sure I have found all the boxes to check through personal experience but maybe there are tips out there I am unaware of.

🍻

Production deployment going on! Phew!

I think my biggest pet peeve right now is production being different from our test environments.

Middle of Black Friday, in Change Freeze, and I'm doing a Production deployment. Yay...!

One of our corporate clients insists that we push changes and updates to production on either Saturday or Sunday. I've worked weekends for longer than I care to admit now.

A major update that completely changes both the data model and UI is scheduled for deployment this Sunday.

Monday is going to be hell with support requests.

* coughing *
* singing mode on *

For the first time in deployment
There'll be magic, there'll be fun!
For the first time in production
I could notice there's an exception
And I know it is totally crazy

OK Bye

Best: finding Laravel and how well works out with production deployment with GitLab

Worst: didnt implement this right from the beginning. Had to copy paste from the dev folder to production folder FOR 3 MONTHS when i wanted to update the website.

Me: Code checked in, CI and tests passed, deployment kicked off. Huh maybe I won't have to stay late after all!

Production Web Server:

Auto Login feature enables itself when a certain config flag is present - the app will login as an admin without requiring username or password.

I pray to the Continuous Deployment monsters that the config setting never gets copied to production by mistake.

Just discovered that my webpack bundle beeing modified manually (via notepad) by Devops when Deployed to production :/

When Codeship shits itself right after bollocksing up the production deployment of a minor preventive fix.

Hi, so I am fairly new to GitHub and I just wanted to know how do you publish your code on GitHub Public Repo without disclosing your sensetive credentials like DB hostname, username and password, and other keys like Recaptcha secret?

My GitHub repo is connected to Heroku hosting so I cannot replace the keys with something dummy. It will fuck the production deployment up.

Testing new server deployment in test env all works, then production it all breaks down. Network didn't allowed the right traffic. Took me whole week to find that out. Until some networking engineer said, you know there is a firewall between those networks?

What's the most insane deployment scheme you've had to work with? One client has a release schedule that deploys all major projects once a month(!). Bugfixes get deployed once a day (systemwide), so any issue that can't be verified until it's in production has at least a days delay when iterating.

so the deployment from my side went through ok... but a "SYSTEM" defect almost prohibited my changes from being signed off... explained it to the person responsible... and then they sign it off knowing well that it was a "SYSTEM" defect that was outside of scope...

You are the bravest .. when u test changes after deployment into production like a boss 😁😁

Received 'Thank you' e-mail after Test Deployment and UAT !!

Usually i receive that e-mail after production and some issue findings!

After getting contacted late about a change to a service for another team's deployment, giving me 2 weeks to complete, completed and the department meeting after the team thanked a lot of ppl but didn't even mention me or my team. I literally went to the hospital the night it was deployed to production.

Ooohhhh so that's why it's called a sandbox.

Interruptions from production team because of a tiny issue that’s already been fixed waiting for deployment

Deploys to Production.
Runtime error.
Open Development server and run in Production setting.
Still runtime error.
Fixes Error.
Error fixed on development.

while (hoursWasted < 3) {
Deploy.
Not working on Prod.
Try other fix.
Still not working, but works perfectly in dev machine.
What the fuck
}

Rage
Go take a walk
Realized I might have deployed to the wrong server
Glanced at deployment path
Realized it's at the wrong server
Reconfigure and Deploy
It works.
Fuck.

Just found a production project that has good portion of it's Composer dependency tree listed in .gitmodules.

Deployment builds are prepared with --recursive clone.

Why.