I’m a senior dev at a small company that does some consulting. This past October, some really heavy personal situation came up and my job suffered for it. I raised the flag and was very open with my boss about it and both him and my team of 3 understood and were pretty cool with me taking on a smaller load of work while I moved on with some stuff in my life. For a week.

    Right after that, I got sent to a client. “One month only, we just want some presence there since it’s such a big client” alright, I guess I can do that. “You’ll be in charge of a team of a few people and help them technically.” Sounds good, I like leading!

    So I get here. Let’s talk technical first: from being in a small but interesting project using Xamarin, I’m now looking at Visual Basic code, using Visual Studio 2010. Windows fucking Forms.

    The project was made by a single dev for this huge company. She did what she could but as the requirements grew this thing became a behemoth of spaghetti code and User Controls. The other two guys working on the project have been here for a few months and they have very basic experience at the job anyways. The woman that worked on the project for 5 years is now leaving because she can’t take it anymore.

    And that’s not the worse of it. It took from October to December for me to get a machine. I literally spent two months reading on my cellphone and just going over my shitty personal situation for 8 hours a day. I complained to everyone I could and nothing really worked.

    Then I got a PC! But wait… no domain user. Queue an extra month in which I could see the Windows 7 (yep) log in screen and nothing else. Then, finally! A domain user! I can log in! Just wait 2 extra weeks for us to give your user access to the subversion rep and you’re good to go!

    While all of this went on, I didn’t get an access card until a week ago. Every day I had to walk to the reception desk, show my ID and request they call my boss so he could grant me access. 5 months of this, both at the start of the day and after lunch. There was one day in particular, between two holidays, in which no one that could grant me access was at the office. I literally stood there until 11am in which I called my company and told them I was going home.

    Now I’ve been actually working for a while, mostly fixing stuff that works like crap and trying to implement functions that should have been finished but aren’t even started. Did I mention this App is in production and being used by the people here? Because it is. Imagine if you will the amount of problems that an application that’s connecting to the production DB can create when it doesn’t even validate if the field should receive numeric values only. Did I mention the DB itself is also a complete mess? Because it is. There’s an “INDEXES” tables in which, I shit you not, the IDs of every other table is stored. There are no Identity fields anywhere, and instead every insert has to go to this INDEXES table, check the last ID of the table we’re working on, then create a new registry in order to give you your new ID. It’s insane.

   

    And, to boot, the new order from above is: We want to split this app in two. You guys will stick with the maintenance of half of it, some other dudes with the other. Still both targeting the same DB and using the same starting point, but each only working on the module that we want them to work in. PostmodernJerk, it’s your job now to prepare the app so that this can work. How? We dunno. Why? Fuck if we care. Kill you? You don’t deserve the swift release of death.

    Also I’m starting to get a bit tired of comments that go  ‘THIS DOESN’T WORK and ‘I DON’T KNOW WHY WE DO THIS BUT IT HELPS and my personal favorite  ‘??????????????????????

The Perfect Storm:

My worst coding mistake? Yeah, let me tell you about that. I pushed a simple JavaScript/HTML change without knowing that the stupid header was shared with another "not so important" section of the site called "My Account" where people go to pay for their services. I call it the perfect storm because I left early that Friday for a weekend cruise and right before leaving I pushed the change, sent the request to push for production and left. When they noticed that clients were complaining about not being able to pay they started reversing most changes of all teams trying to fix it but they never touched mine because they knew I wasn't working on the backend. My whole team worked over the weekend trying to find the issue while I was having fun in the cruise. They ended up reversing all changes by Sunday night and it took us about 4 more days to figure out that my simple JavaScript/HTML change broke the site and prevented 30 million customers from making payments that weekend plus it broke the whole 2nd release of the month.... yeah, nothing major.

Hey, Root? How do you test your slow query ticket, again? I didn't bother reading the giant green "Testing notes:" box on the ticket. Yeah, could you explain it while I don't bother to listen and talk over you? Thanks.

And later:
Hey Root. I'm the DBA. Could you explain exactly what you're doing in this ticket, because i can't understand it. What are these new columns? Where is the new query? What are you doing? And why? Oh, the ticket? Yeah, I didn't bother to read it. There was too much text filled with things like implementation details, query optimization findings, overall benchmarking results, the purpose of the new columns, and i just couldn't care enough to read any of that. Yeah, I also don't know how to find the query it's running now. Yep, have complete access to the console and DB and query log. Still can't figure it out.

And later:

Hey Root. We pulled your urgent fix ticket from the release. You know, the one that SysOps and Data and even execs have been demanding? The one you finished three months ago? Yep, the problem is still taking down production every week or so, but we just can't verify that your fix is good enough. Even though the changes are pretty minimal, you've said it's 8x faster, and provided benchmark findings, we just ... don't know how to get the query it's running out of the code. or how check the query logs to find it. So. we just don't know if it's good enough.

Also, we goofed up when deploying and the testing database is gone, so now we can't test it since there are no records. Nevermind that you provided snippets to remedy exactly scenario in the ticket description you wrote three months ago.

And later:

Hey Root: Why did you take so long on this ticket? It has sat for so long now that someone else filed a ticket for it, with investigation findings. You know it's bringing down production, and it's kind of urgent. Maybe you should have prioritized it more, or written up better notes. You really need to communicate better. This is why we can't trust you to get things out.

*twitchy smile*

!rant
!!git

Who here uses `master` for development?

My boss (api guy) tried to convince me that was normal practice. I gently told him that it sounded crazy and very very bad.

Here's the dev path I'm enforcing on my repos:

(feature branches) -> dev -> qa* -> master -> production*

*: the build server auto-pulls from these branches, and pushes any passing builds to staging/production.

Everyone works on their own feature branches, and when they're happy with their work, they merge it into `dev`. `dev`, therefore, is for feature integration testing. After everything is working well on `dev`, it gets merged into `qa` for the testers to fawn over and beat with sticks. Anything that passes QA gets merged into `master`, where it sits until we're ready to release it. When that time comes (it's usually right away, but not always), `master` gets merged into `production`.

This way, `master` is always stable and contains the newest code, so it's perfect for forking/etc. Is this standard practice, or should I be doing something different?

Also, api guy encourages something he calls "running a racetrack" -- each dev has their own branch (their initials) and they push to that throughout the day. everyone else pulls from it regularly and pushes to their own branch. When anyone's happy with their code, they push from their (updated) branch to `qa` (I insisted on `dev` instead.)

Supposedly this drastically reduces the number of merge conflicts when pushing to an upstream branch due to having a more recent ancestor node?

I don't quite follow that, but it seems to me that merging/pushing throughout the day would just make them happen sooner? idk.

What are your thoughts?

Why are job postings so bad?
Like, really. Why?

Here's four I found today, plus an interview with a trainwreck from last week.
(And these aren't even the worst I've found lately!)

------

Ridiculous job posting #1:
* 5 years React and React Native experience -- the initial release of React Native was in May 2013, apparently. ~5.7 years ago.
* Masters degree in computer science.
* Write clean, maintainable code with tests.
* Be social and outgoing.

So: you must have either worked at Facebook or adopted and committed to both React and React Native basically immediately after release. You must also be in academia (with a masters!), and write clean and maintainable code, which... basically doesn't happen in academia. And on top of (and really: despite) all of this, you must also be a social butterfly! Good luck ~

------

Ridiculous job posting #2:
* "We use Ruby on Rails"
* A few sentences later... "we love functional programming and write only functional code!"

Cue Inigo Montoya.

------

Ridiculous job posting #3:
* 100% remote! Work from anywhere, any time zone!
* and following that: You must have at least 4 work hours overlap with your coworkers per day.
* two company-wide meetups per quarter! In fancy places like Peru and Tibet! ... TWO PER QUARTER!?

Let me paraphrase: "We like the entire team being remote, together."

------

Ridiculous job posting #4:
* Actual title: "Developer (noun): Superhero poised to change the world (apply within)"
* Actual excerpt: "We know that headhunters are already beating down your door. All we want is the opportunity to earn our right to keep you every single day."
* Actual excerpt: "But alas. A dark and evil power is upon us. And this… ...is where you enter the story. You will be the Superman who is called upon to hammer the villains back into the abyss from whence they came."

I already applied to this company some time before (...surprisingly...) and found that the founder/boss is both an ex cowboy dev and... more than a bit of a loon. If that last part isn't obvious already? Sheesh. He should go write bad fantasy metal lyrics instead.

------

Ridiculous interview:
* Service offered for free to customers
* PHP fanboy angrily asking only PHP questions despite the stack (Node+Vue) not even freaking including PHP! To be fair, he didn't know anything but PHP... so why (and how) is he working there?
* Actual admission: No testing suite, CI, or QA in place
* Actual admission: Testing sometimes happens in production due to tight deadlines
* Actual admission: Company serves ads and sells personally-identifiable customer information (with affiliate royalties!) to cover expenses
* Actual admission: Not looking for other monetization strategies; simply trying to scale their current break-even approach.

------

I find more of these every time I look. It's insane.
Why can't people be sane and at least semi-intelligent?

Confession of the day:
1. I work in release mode
2. I work on the main branch only
3. I test on production

I quit and my last day is next week.

Apparently management has decided that I should spend my last day implementing a new feature for a customer where I have been the only developer, and release it to production (without first implementing it in test) the same day. A feature that potentially could cripple a whole workflow if done wrong.

Of course I advised not to release untested code to production on a friday, just before the only person that knows how it works leaves the company. But no, “the customer reaaaaaally wants it before summer, so just be careful not to write any bugs”.

I’m not saying that I’m intentionally gonna write bad code - but if I do, I’m not gonna pick up the phone when it calls.

My biggest pet peeve at the moment is people without any development experience using version numbers.

Me: "Communicating a release date for the feature towards clients is dangerous, we have a developer shortage, and currently don't really have enough capacity to..."

Manager: "What we release next month doesn't have to be perfect, it is just a v1"

Me: "You mean it's a beta? If that's the case, could you maybe differentiate the requirements of the beta, let's call it a 0.1.0, versus the 1.0.0-rc, the release candidate?"

*Feature is eventually merged into production, barely in a beta state*

Manager: "So I have some ideas for the v2"

Me: "You mean 1.0.0"

Manager: "Let's compromise and call it v1.5"

Me: "Let's compromise, you stop communicating release dates, AND you stop using version numbers..."

Manager: "That's not a compromise..."

Me: "...I wasn't finished... And I won't respond to the recruiter who just offered me a better paying job"

Spent most of the day debugging issues with a new release. Logging tool was saying we were getting HTTP 400’s and 500’s from the backend. Couldn’t figure it out.

Eventually found the backend sometimes sends down successful responses but with statusCode 500 for no reason what so ever. Got so annoyed ... but said the 400’s must be us so can’t blame them for everything.

Turns out backend also sometimes does the opposite. Sends down errors with HTTP 200’s. A junior app Dev was apparently so annoyed that backend wouldn’t fix it, that he wrote code to parse the response, if it contained an error, re-wrote the statusCode to 400 and then passed the response up to the next layer. He never documented it before he left.

Saving the best part for last. Backend says their code is fine, it must be one of the other layers (load balancers, proxies etc) managed by one of the other teams in the company ... we didn’t contact any of these teams, no no no, that would require effort. No we’ve just blamed them privately and that’s that.

#successfulRelease

You know side projects? Well I took on one. An old customer asked to come and take over his latest startups companys tech. Why not, I tought. Idea is sound. Customer base is ripe and ready to pay.

I start digging and the Hardware part is awesome. The guys doing the soldering and imbedded are geniuses. I was impressed AF.

I commit and meet up with CEO. A guy with a vision and sales orientation/contacts. Nice! This shit is gonna sell. Production lines are also set.

Website? WTF is this shit. Owner made it. Gotta give him the credit. Dude doesn't do computers and still managed to online something. He is still better at sales so we agree that he's gonna stick with those and I'll handle the tech.

I bootstrap a new one in my own simplistic style and online it. I like it. The owner likes it. He made me to stick to a tacky logo. I love CSS and bootstrap. You can make shit look good quick.

But I still don't have access to the soul of the product. DBs millions rows of data and source for the app I still behind the guy that has been doing this for over a year.

He has been working on a new version for quite some time. He granted access to the new versions source, but back end and DB is still out of reach. Now for over month has passed and it's still no new version or access to data.

Source has no documentation and made in a flavor of JS frame I'm not familiar with. Weekend later of crazy cramming I get up to speed and it's clear I can't get further without the friggin data.

The V2 is a scramble of bleeding edge of Alpha tech that isn't ready for production and is clearly just a paid training period for the dev. And clearly it isn't going so well because release is a month late. I try to contact, but no reaction. The owner is clueless.

Disheartening. A good idea is going to waste because of some "dev" dropping a ball and stonewalling the backup.

I fucking give him till the end of the next week until I make the hardware team a new api to push the data and refactor the whole thing in proper technologies and cut him off.

Please. If you are a dev and don't have the time to concentrate on the solution don't take it on and kill off the idea. You guys are the key to making things happening and working. Demand your cut but also deserve it by delivering or at least have the balls to tell you are not up for it.

You know what I realized we should always say no for demo driven application development.

We should always ask for enough time do a proper development and if its not enough, shouldn’t write a single line.

Because once we deliver a working demo. Its release ready for them because its FUCKING WORKING..

And trying to explain why this is just a demo and cant be put to production is even bigger pain in the ass than saying no in the beginning.

LESSON LEARNED .

PM: Are you sure we are OK for production release?

ME: Sure!

Never, ever, ever, release on a Friday. But.....
if you do.
Add this step as part of your production push process.

“In order to release on production you merge on production”

Makes sense.

“In order to release on staging you merge on release”

Wait, what?

“The CI for the staging release is called release to production”

What? No, stop!

“You also need to create a tag called pre-production-deployment, so that staging can work”

STOP FFS

“It’s very easy, you just need to read the documentation!”

How do people do not realise they are spitting out bs?

Backend: Sorry the fix we had isn't going to work. Turns out app is sending an "undelivered" status after you call the API where you claimed we had an issue. This is in fact the cause, please address it.

Me: We do not have an "undelivered" status anywhere in our codebase. We do not do this.

Backend: *CC product* turns out this issue is only affecting 0.1% of users, its very minimal. Lets push ahead with the release.

Product: Ok, lets go live.

Me: ... ... ... we all just gonna ignore that "undelivered" bullshit? ... ok ... very stable release, here we go.

Fuck Apple and its review system

So, this started in december. We wanted to publsih an app, after years of development.

Submit to review, and passes on the first try. Well, what do you know. We are on manual release option, so we can release together with the android counterpart. Well yes, but someone notices that the app name is not what was aggreed (App Name instead of AppName). Okay, should be easy, submit the same app, just the name changed. If it passed once, it will pass again, right? HAH

Rejected, because the description, why we use the device’s camera is too general. Well... its the purpose of the app... but whatever, i read the guidelines, okay, its actually documented with exapmles. BUT THEN WHY THE FUCK COULDNT YOU SAY THAT ON THE FIRST UPLOAD?

Whatever, fix it, new version, accepted, ready to release just in time.

It doesindeed roll out,but of course, we notice that the app has a giant issue, but only on specific phones. None of our test phones had this problem, but those who have, essentially cannot use our program. Nasty as it is, the fix is really easy, done in 5 minutes. Upload it asap, literally nothing changed from user point of view, except now it doesnt crash on said devices. Meanwhile 1 star reviews are arriving from these users - of course with all the right. Apple should allow this patch quickly, right? HAH

THE REAL BULLSHIT COMES NOW

With only config files changed, the same binary uploaded we get rejected? What now? Lets read it. “Metadata rejected, no need to upload new binary”.... oh fine only the store page is wrong? Easy. Read the message, what went wrong. “Referencing third party content is nit permitted on the app store” meaning that no android test device should be shown. Fine, your rules. They even send a picutre of the offending element. BUT ITS NOT EVEN ON THE STORE. THATS A SCREENSHOT OF THE APP. HOW IS THAT METADATA? I ask about this, and i get a reply, from either a bot, or a person who cant speak or read english, and only pasted a sample answer, repeating the previous message. WTF. Fine, i guess you are dumb, but since they stop replying to our queries, do the only sensible thing, re-record the offending tutorial video that actually contained an android device. This is about 2 weeks, after the first try to apply a simple patch to a broken app. And still, how did it pass the review 2 times?

Whatever, reupload again, play the waiting game for a week, when the promised average wait time is 2 days, they hit us with a message, that they want to know what patent we use in our apps core functionality. WTF WHY NOW? It didnt bother you for a month, let it release ti production and now you delay a simple patch for this? We send them what they know. Aaaaand they reply: sorry we need more time to review your app. FUUUUUUCKKK YOUUU. You are reviewing a PATCH with close to zero functional change!!! Then, this shit goes on, every week we ask about an ETA, always asking for patience... at the end it took another 3 weeks... so december 15 to jan 21 in total...

FOR. A. SINGLE. FUCKING. PATCH

Bottom line is what is infurating, apple cares that there is an android device in the tutorial video, but they dont care that a significant percentage of our users simply cannot use the app.

Im done

Hey Root, remember that super high-priority ticket that we ignored for five months before demanding you rewrite it a specific way in one day?

Yeah, the new approach we made you use broke the expected usecases, and now the page is completely useless to the support team and they're freaking out. Drop everything you're doing and go fix it! Code-complete for this release is tonight! -- This right after "impacting our business flow" while being collapsed on the fucking floor.

Jesus FUCKING christ, what the fuck is wrong with these people?

If I dropped the ball on a high-priority ticket for two weeks, I'd get fired, let alone for five fucking months.

If I was a manager and demanded a one-day rewrite I can only imagine the amount of chewing out I'd receive, especially on something high-priority.

And let's not forget product ownership: imagine if I screwed up feature planning for someone so badly I made them break a support tool in production. I'd never hear the end of it.

Fucking double standards.

And while I'm at it. Some of the code I've seen in this codebase is awful. Uncommented spaghetti, or an unreadable mess with single-letter variables, super-tightly coupled modules so updates are nearly impossible, typos in freaking constants added across sixty+ files, obviously-incorrect comments, ... . I'll have to start posting snippets to show them off. But could I get away with any of it? ha. Hell no. My code must be absolutely perfect. I hear about any and every flaw, doesn't matter how minor, and nothing can go out until everything is just so.

Hell, I even hear about flaws in other peoples' code during my code reviews. Why? Because I should have fixed it, that's why. But if I do, I get yelled at for "muddying the waters."

Just. JESUS FUCKING CHRIST.

It's like playing a shell game where no matter which shell I pick (or point to their goddamn sleeve where they're clearly hiding it), I get insulted for being so consistently useless, and god damn, how can I never find the fucking pea or follow the damned rules? I'm so terrible and this is why "nobody trusts me." Fuck you.

I'll tell you why I can't find your damned pea: IT'S RATTLING INSIDE YOUR FUCKING HEADS, you ASSHOLE FUCKING IMBECILES.

That's right: one pea among the lot of them.

goddamn I am fucking pissed off.

v2 production release, day 4:

0 Issues reported

*drops mic

I have the best team.

Assigned to a new project team..

Using git, in a creative way. So.. "master" is "dev" branch, usually. Everyone can push their branch to dev server .. so it's "dynamic for us". Production branch is whatever, as long as the branch has the release version. Sometimes, the release comes from "master".. that mean "dev" in normal geek..

That's just Git. The source code is a saturated spagetti of Entity framework and Caliburn. It is littered with antipatterns, especially basebean. Holy Christmas and Easter that baseclass do a lot of stuff that has no place as a base class ..

Fucking frameworks, I'm gonna start to evangelize frameworks as the no1 antipattern.

MS SQL as the main DB, but is dumped to json FILES through a scheduled task to increase read performance on web.

There is a soap endpoint to expose the json files, fml..

I am assuming I was placed here to improve stuff, I have never in my life seen anything like this before.

There is a special place in hell for this repository

I think I want to quit my first applicantion developer job 6 months in because of just how bad the code and deployment and.. Just everything, is.

I'm a C#/.net developer. Currently I'm working on some asp.net and sql stuff for this company.

We have no code standards. Our project manager is somewhere between useless and determinental. Our clients are unreasonable (its the government, so im a bit stifled on what I can say.) and expect absurd things from us. We have 0 automated tests and before I arrived all our infrastructure wasn't correct to our documentation... And we barely had any documentation to begin with.

The code is another horror story. It's out sourced C# asp.net, js and SQL code.. And to very bad programmers in India, no offense to the good ones, I know you exist. Its all spagheti. And half of it isn't spelled correctly.

We have a single, massive constant class that probably has over 2000 constants, I don't care to count. Our SQL projects are a mess with tons of quick fix scripts to run pre and post publishing. Our folder structure makes no sense (We have root/js and root/js1 to make you cringe.) our javascript is majoritly on the asp.net pages themselves inline, so we don't even have minification most of the time.

It's... God awful. The result of a billion and one quick fixes that nobody documented. The configuration alone has to have the same value put multiple times. And now our senior developer is getting the outsourced department to work on moving every SINGLE NORMAL STRING INTO THE DATABASE. That's right. Rather then putting them into some local resource file or anything sane, our website will now be drawing every single standard string from the database. Our SENIOR DEVELOPER thinks this is a good idea. I don't need to go into detail about how slow this is. Want to do it on boot? Fine. But they do it every time the page loads. It's absurd.

Our sql database design is an absolute atrocity. You have to join several tables together just to get anything done. Half of our SP's are failing all the time because nobody really understands the design. Its gloriously awful its like.. The epitome of failed database designs.

But rather then taking a step back and dealing with all the issues, we keep adding new features and other ones get left in the dust. Hell, we don't even have complete browser support yet. There were things on the website that were still running SILVERLIGHT. In 2019. I don't even know how to feel about it.

I brought up our insane technical debt to our PM who told me that we don't have time to worry about things like technical debt. They also wouldn't spend the time to teach me anything, saying they would rather outsource everything then take the time to teach me. So i did. I learned a huge chunk of it myself.

But calling this a developer job was a sick, twisted joke. All our lives revolve around bugnet. Our work is our BN's. So every issue the client emails about becomes BN's. I haven't developed anything. All I've done is clean up others mess.

Except for the one time they did have me develop something. And I did it right and took my time. And then they told me it took too long, forced me to release before it was ready, even though I had never worked on what I was doing before. And it worked. I did it.

They then told me it likely wouldn't even be used anyway. I wasn't very happy at all.

I then discovered quickly the horrors of wanting to make changes on production. In order to make changes to it, we have to... Get this

Write a huge document explaining why. Not to our management. To the customer. The customer wants us to 'request' to fix our application.

I feel like I am literally against a wall. A huge massive wall. I can't get constent from my PM to fix the shitty code they have as a result of outsourcing. I can't make changes without the customer asking why I would work on something that doesn't add something new for them. And I can't ask for any sort of help, and half of the people I have to ask help from don't even speak english very well so it makes it double hard to understand anything.

But what can I do? If I leave my job it leaves a lasting stain on my record that I am unsure if I can shake off.

... Well, thats my tl;dr rant. Im a junior, so maybe idk what the hell im talking about.

Me : I found this code issue, I think we need to fix it
PO: does it affect the user?
Me: not really but we can make it better
PO: do you have a defect for it in *insert issue tracker here*
Me: no, I just noticed it
PO: is there an IM ticket for it?
Me: I don't think so
PO: is this issue already in production?
Me: possibly. Yes. That's why I was wondering if we should fix it.
PO: okay then we will fix it in the 3rd release from now if you still remember it by then.

when your boss wants to change the business logic when it's just 1 week away from production release. ¯\_(ツ)_/¯

Debate (with rant-ish overtones):

FYI, while it is a debate, its a practiseSafeHex debate, which means there is a correct answer, i'm just interested in your responses/thoughts.

Ok lets kick off. So the remote team I work with had an opening for a new iOS developer (unrelated to anything to do with me). They interviewed and hired a guy based off his "amazing" take home challenge.

The challenge consists of 4 screens and was for a senior level position. For the challenge the interviewee created a framework (a iOS library) for each screen, included all the business logic for each screen inside, each one needs to be built separately, exposed some API/functions from each one and then created a main project to stitch it all together.

Now, my opinion is, this is highly unscalable and a ridiculous approach to take as it would add so much unnecessary overhead, for no benefit (I am correct btw).

The interviewee said he did it like this to "show off his skills and to stand out". The remote team loved it and hired him. The challenge said "show us the code standard you would be happy to release to production". I would argue that he has only demonstrated 1 extra skill, and in exchange delivered something that is unscalable, going to be a nightmare to automate and require huge on-boarding and a paradigm shift, for no reason. To me thats a fail for a senior to not realise what he's doing. This person will be required to work alone (in part), make architecture decisions, set the foundation for others etc. Having someone who is willing to just do mad shit to show off, is really not the type of person suited to this role.

Debate!

One day after the release of the website of a medium sized travel company, I made a big mistake by accidentally taking it offline for 1 hour during peak usage (~150 simultaneous visitors).
Turns out deleting the wrong image transformation cache folder in production can hang up the PHP process for taking too much load on regenerating image transformations.

The designer of PHP probably took a big load too while creating the first draft.

In one of my first jobs i developed an (ugly and heavly under-payed) e-commerce/media platform for a customer.
That customer was constantly making fun of his bald partner telling how he was gay, liked dicks, etc., drawing dicks and bananas as sample website logos or uploading dildo/penis images as images, he was always like this.
Once the website was ready for production i removed all the "testing" posts and images and told the client to insert some real content and alert me when it was ready for release.
Well some time after the release i got a call from that client, for the first time he was serious:

C: Hi, why there are dildo images on the server? (the website in production was full of dildo/penis images instead of actual product images, he even photoshopped the head of his partner on a penis and uploaded it!!!)
R: ehm... i told you it was on production and to stop uploading bad content....
C: Ummm ok, please fix it immediatly, thanks!

I work for a bank and every production release date it's a chaos... Like, for real, devs running to get their stories approved by the testing team and last minute scope changes that, if not made, would make the whole app fail (real shitty management as you can see).

Longstoryshort, a dev didn't finished one of his stories and create 7 major bugs with another... Today that was my breakfast, took me 4 hours and get it all done and approved... We didn't make the release tho, but I scored some major points with this.

Funny thing, tomorrow I'm telling my PM I'll leave the company for a better job, so that will be their breakfast.

"Graphics don't matter."

I ranted a while back about gamedev being hard to get into for me, and, today, user @DOSnotCompute posted a similar experience.

I had a couple more thoughts, so thought should post them here (FUCK! It ended up being too fucking long! sorry!)

So I was watching the making of mortal kombat 3 on yt, which was pretty amazing btw because I got to see the actors of the sprites in game which were engraved in my and thousands of others kids minds.

Anyhow, the creators of the series, John Tobias and Ed Boon, were interviewed and what not. And it hit me that while both were the designers, John was the main artist and Ed was the programmer (at least for MK1). Another game that comes to mind Super Meat Boy, and I bet hundreds of others did the same.

And it got me thinking, maybe that's my problem, I just need an artist.

And I think the reason why I never thought of that is because of this idea that graphics don't matter.

"you don't need an artist. You don't need graphics. The most important thing is the gameplay."

What a load of shit.
A lot of people believe that because they got tired of polished AAA games with automatic and predictible gameplay.

People started parrotting this knee jerk of a conclusion since then.

It's dumb. Imagine if Infiminer, one of the games Minecraft was based on, which btw looks terrible, had all the same features Minecraft had.
I would still not touch that shit with a pole.

Graphics ARE important. Games are on the VISUAL medium.
That doesn't mean you're sucking Sony's dick on every AAA release or that every game should be made with UnreUnityCocksReloadedEngine.

Some level of visual craft is required for a game ro be considered such.

(btw, I think most of you guys here get this, not trying to pander, just that I want to make it clear that I'm not accusing this community of being guilty of this)

If a game looks bad (given, bad can be subjective), if it gives the impression that it wasn't seriously made, then you kinda lower your expectations.

People get hyped on games that look good, because it means that the game could be good. Games that look unoriginal or terrible won't get played, wether they're good or not. And I think it's a reasonable reaction.

How many times did I hear things like "Look at x video game from the 90s, the graphics are terrible but it's fun as hell".

That is an absurd statement. The level of production some NES games went through is insane. We're talking millions of dollars for games that today might look primitive.

The graphics weren't shit back then, and even today you could say that they are simpler but also of excellent craftsmanship.

I'm not into creating art, I hate it in fact because you can't quantify the success of produced art.

So, duh, find an artist. Ok, how? This is the part where I have no fucking idea how.

You start spamming shit like "I need an artist" online? I dunno, something for another post I guess.

I guess the most healthy thing I could do is making demos that might look like shit just to get experience so that when I get to find an artist, I have practice already.

I don't know if I'm being pranked or not, but I work with my boss and he has the strangest way of doing things.
- Only use PHP
- Keep error_reporting off (for development), Site cannot function if they are on.
- 20,000 lines of functions in a single file, 50% of which was unused, mostly repeated code that could have been reduced massively.
- Zero Code Comments
- Inconsistent variable names, function names, file names -- I was literally project searching for months to find things.
- There is nothing close to a normalized SQL Database, column ID names can't even stay consistent.
- Every query is done with a mysqli wrapper to use legacy mysql functions.
- Most used function is to escape stirngs
- Type-hinting is too strict for the code.
- Most files packed with Inline CSS, JavaScript and PHP - we don't want to use an external file otherwise we'd have to open two of them.
- Do not use a package manger composer because he doesn't have it installed.. Though I told him it's easy on any platform and I'll explain it.
- He downloads a few composer packages he likes and drag/drop them into random folder.
- Uses $_GET to set values and pass them around like a message contianer.
- One file is 6000 lines which is a giant if statement with somewhere close to 7 levels deep of recursion.
- Never removes his old code that bloats things.
- Has functions from a decade ago he would like to save to use some day. Just regular, plain old, PHP functions.
- Always wants to build things from scratch, and re-using a lot of his code that is honestly a weird way of doing almost everything.
- Using CodeIntel, Mess Detectors, Error Detectors is not good or useful.
- Would not deploy to production through any tool I setup, though I was told to. Instead he wrote bash scripts that still make me nervous.
- Often tells me to make something modern/great (reinventing a wheel) and then ends up saying, "I think I'd do it this way... Referes to his code 5 years ago".
- Using isset() breaks things.
- Tens of thousands of undefined variables exist because arrays are creates like $this[][][] = 5;
- Understanding the naming of functions required me to write several documents.
- I had to use #region tags to find places in the code quicker since a router was about 2000 lines of if else statements.
- I used Todo Bookmark extensions in VSCode to mark and flag everything that's a bug.
- Gets upset if I add anything to .gitignore; I tried to tell him it ignores files we don't want, he is though it deleted them for a while.
- He would rather explain every line of code in a mammoth project that follows no human known patterns, includes files that overwrite global scope variables and wants has me do the documentation.
- Open to ideas but when I bring them up such as - This is what most standards suggest, here's a literal example of exactly what you want but easier - He will passively decide against it and end up working on tedious things not very necessary for project release dates.
- On another project I try to write code but he wants to go over every single nook and cranny and stay on the phone the entire day as I watch his screen and Im trying to code.

I would like us all to do well but I do not consider him a programmer but a script-whippersnapper. I find myself trying to to debate the most basic of things (you shouldnt 777 every file), and I need all kinds of evidence before he will do something about it. We need "security" and all kinds of buzz words but I'm scared to death of this code. After several months its a nice place to work but I am convinced I'm being pranked or my boss has very little idea what he's doing. I've worked in a lot of disasters but nothing like this.

We are building an API, I could use something open source to help with anything from validations, routing, ACL but he ends up reinventing the wheel. I have never worked so slow, hindered and baffled at how I am supposed to build anything - nothing is stable, tested, and rarely logical. I suggested many things but he would rather have small talk and reason his way into using things he made.

I could fhave this project 50% done i a Node API i two weeks, pretty fast in a PHP or Python one, but we for reasons I have no idea would rather go slow and literally "build a framework". Two knuckleheads are going to build a PHP REST framework and compete with tested, tried and true open source tools by tens of millions?

I just wanted to rant because this drives me crazy. I have so much stress my neck and shoulder seems like a nerve is pinched. I don't understand what any of this means. I've never met someone who was wrong about so many things but believed they were right. I just don't know what to say so often on call I just say, 'uhh..'. It's like nothing anyone or any authority says matters, I don't know why he asks anything he's going to do things one way, a hard way, only that he can decipher. He's an owner, he's not worried about job security.

Spent a lot of time designing a proper HTTP (dare I even say RESTful) API for our - what is until now a closed system, using a little-known/badly-supported message-over-websocket protocol to do RPC-style communications - supposedly enterprise-grade product.

I make the API spec go through several rounds of review with the rest of the dev team and customers/partners alike. After a few iterations, everybody agrees that the spec will meet the necessary requirements.

I start implementing according to spec. Because this is the first time we're actually building proper HTTP handling into the product, but we of course have to make it work at least somewhat with the RPC-style codebase, it's mostly foundational work. But still, I manage to get some initial endpoints fully implemented and working as per the spec we agreed. The first PR is created, reviews are positive, the direction is clear and what's there already works.

At this point in time, I leave on my honeymoon for two weeks. Naturally, I assume that the remaining endpoints will be completed following the outlines/example of the endpoints which I built. When I come back, the team mentions that the implementation is completed and I believe all is well.

The feature is deployed selectively to some alpha customers to start validation testing before the big rollout. It's been like that for a good month, until a few days ago when I get a question related to a PoC integration which they can't seem to get to work.

I start investigating and notice that the API hasn't been implemented according to the previously agreed upon spec at all. Not only did the team manage to implement the missing functionality in strange and some even broken ways, they also managed to refactor my previously working endpoints into being non-compliant.

Now, I'm a flexible guy. It's not because something isn't done exactly as I've imagined it that it's automatically bad. However, I know from experience that designing a good/clear/future-proof API is a tricky exercise. I've put a lot of time and effort into deliberate design decisions that made up the spec that we all reviewed repeatedly and agreed upon. The current implementation might also be fine, but I now have to go over each endpoint again and reason about whether the implementation still fulfills the requirements (both soft and hard) that we set out to meet.

I'm met with resistance, pushback and disbelief from product management and dev co-workers alike when I raise the concern that the API might actually not be production-ready (while I'm frantically rewriting my integration tests and figuring out how the actual implementation works in comparison to what was spec'ed).

Oh, and did I mention that product management wants to release this by end-of-week?!

I am calling this a premonition rant, of more rants to come.
I have a feeling in my bones.
We have a newly acquired fat cat customer with bucks to blow who we have done some digital work for already and swag bag of marketing perkiness.
I will call the CEO of this whale "The Porcupine"
The Porcupine has a business degree and industry experience, nothing to do with websites or applications.
It claims to be a visual perfectionist yet never delivers an overall coherent review.
It likes to fixate on minor brand style differences in websites and apps we have built.
The Porcupine seems to be always busy with policy and legal and other things rather than participating in their own projects.
Procrastination on feedback or reviews until the day before release is common.
Many overtime hours worked, not a sliver of thanks. The haughty attitude indicative of somebody who thinks web development is like desktop publishing.

"It's just code" in response to a crash production server change they were warned was a risk that borked all of our responsive templates and took 3 hours to fix.
Their entire brand is shades of pea green, grey and lime. No serif fonts because they are suck. Arial and Helvetica are boss.
One of my devs missed a CSS style on privacy policy hyperlink text that went times new roman and I had various account directors and our CEO on phone telling me how embarrassing it was for us to let this happen.
Anyway. They pay on time and the cost estimates for all the upcoming work are juicy.
We have shitloads going on for an upcoming hard date conference and everything is already compressing.
Therefore I can already smell doom and feel those porcupine quill getting closer to my ass as I beg their AD today if we have any feedback on the 10 or so project reviews yet?
Nope.

There was an issue whilst you were away, we had to make a small css change.. We pushed it into master but it said something about the branch being behind the tip by 50 commits or something. It's okay, we forced it up though and force pushed it to production as well but the site went down.. In the end we had to ftp it up manually but the customer is saying things that were there before now aren't there any more?

I thought you put this "release process" in so things like this wouldn't happen! I think we need to review it as it clearly isn't working.

We have a production release tomorrow and we didn't have any kind of testing other than the unit tests we wrote. 💣💥🤯🎆

Getting ready for a release to production next week. Needless to say, tensions are high. Found this in the source history. 😂

I'm getting really tired of those dumbass programmers that do not understand shit and then come to me when production breaks. (I am also a programmer, not really a DevOps engineer, but I'm the least worst at DevOps stuff, so it's my job...).

We're programming some kind of document management tool. Today we had a release, and one of the new features is to download all of your documents as a zip file, which is asynchronuously generated. When it's done, the user gets a mail with the download link to the zip file.

The feature works basically, but today it broke our production service, as somebody was running a test of it.

Turns out all the documents are loaded into memory to be zipped. So if you have 2 gigs of documents, a container with memory restrictions in that area will crash.

I asked the programmer who reported this «ops problem» to me, why he didn't just shit the files into a temp foler in order to zip them in there.

He told me that he wanted to do so, but did not know how to mock this for a unit test, and therefore went to the in-memory «solution», which was easier for him to mock.

For fuck's sake, unit tests and mocks are fucking tools, not ends in itself! I don't give a fuck about your pointless mocking code when the application crashes!

When I got to deal with such dumbasses, I'd prefer to mock those motherfuckers with a leaky bucket of liquid shit, which basically accomplishes the same task from my perspective: dripping shit all over the place and make everything suck as fuck.

I dunno about coolest, but I did sort of cement my reputation as the "database guy" in my first job because of this.

My first job was with a group maintaining a series of websites. Because of the nature of the websites, every morning we had to pull the records from one database on one network, sneaker net the data to a database on another network, and import the data via custom data import function.

However, the live site would crash after 100 or so records were imported. The dba at the live site had to script out a custom data partitioning script to do his daily duties, but it definitely messed up his productivity.

Turns out, the custom mass import function had recycled the standard import function, which was only used to import 1 record at a time, and it never closed its database connections, because it never needed to. A one line fix to production code was delivered 6 months later (because that was our release cycle) and I came up with the temporary work around, which was basically removing the connection limit. It would still crash with the work around, but only with multiple days worth of data. So basically only on Monday. Also developed the test set for the import (15k+ records).

"The tool to push new releases to the data centre blocked us last night. Saying all the nodes are 'unhealthy', resolve the issue(s) first. But then the remote team said 'we have a way around that' so we managed to get it deployed in time. We need to document the process as there were many ... 'shady' processes and steps involved lol"

- Manager explaining how the first production release on our new team went last night

... he called it a success

Yayyyyyy tonight we have a production release, and right now I'm still doing last minute coding 🤗

Dev gets told in the morning there's an emergency fix needed due to a critical issue with the app that's in production and that the fix needs to be in the release that will be cut this evening.
Dev drops everything he/she is working on, works frantically all day to get it in 2 minutes before the deadline.
Release gets cut.
Next day release gets trashed because some exec did not like the size of the font used in some obscure part of the app even though it's been this way for 6 months...

Of course the shouting episodes all happened during the era I was doing WordPress dev.

So we were a team of consultants working on this elephant-traffic website. There were a couple of systems for managing content on a more modular level, the "best" being one dubbed MF, a spaghettified monstrosity that the 2 people who joined before me had developed.

We were about to launch that shit into production, so I was watching their AWS account, being the only dev who had operational experience (and not afraid to wipe out that macos piece of shit and dev on a real os).

Anyhow, we enable the thing, and the average number of queries per page load instantly jumps from ~30 (even vanilla WP is horrible) to 1000+. Instances are overloaded and the ASG group goes up from 4 to 22. That just moves the problem elsewhere as now the database server is overwhelmed.

Me: we have to enable database caching for this thing *NOW*

Shitty authors of the monstrosity (SAM): no, our code cannot be responsible for that, it's the platform that can't handle the transition.

Me: we literally flipped a single switch here and look at the jump in all these graphs.

SAM: nono, it's fine, just add more instances

Me: ARE YOU FUCKIN SERIOUS?
Me: - goes and enables database caching without any approvals to do so, explaining to mgmt. that failure to do so would impair business revenue due to huge loading times, so they have to live with some data staleness -

SAM: Noooo, we'll show you it's not our code.
SAM: - pushes a new release of the monstrosity that makes DB queries go above 2k / page load -

...

Tho on the bright side, from that point on I focused exclusively on performance, was building a nice fragment caching framework which made the site fly regardless of what shitty code was powering it, tuned the stack to no end and learned a ton of stuff in the process which allowed me to graduate from the tar pit of WP development.

Jesus fuckin' Christ. I own a webshop together with someone else. This guy is so fuckin' stupid. Yesterday I've deployed a release to our acceptance environment. I talked with him extensively about it. This morning I texted him to check out all the new stuff.

5 minutes later he texts.me back: I would suggest changing option x. Uuh... what option x? We don't have that any more.

Dude! What the fuck! We talked extensively about acceptance testing so you know it is in our acceptance environment, not production, asshole.

And then again, he asks for the link to the acceptance, which I gave him twice already.

Are you really that stupid??

My killer PR:
I made a small feature and it was to be merged in my senior's branch.
So I made it, resolved all the conflicts and when it got merged it broke my senior's branch.
This is nothing new and it happens, so me and him sat down and got it working. After two days, his branch got merged and broke it's parent 😂, after investigation my code was the reason, got fixed and merged.
Same thing happened 3 more times, and every time my code was the culprit.
Now for staging we thoroughly tested everything and deployed it, after running for 2 hours the piece of shit broke again 😂😂😂.
A meeting was arranged for scolding the team, and after the meeting the architect comes to me and say "how did you manage to fuck things on so many levels, literally?". I handled it like a pro with an awkward laugh.

We exhaustively checked it for production. Deployed, it did not broke, we were happy. After a month of successful run, I just joked about the above incident while we were working on next release on morning coffee. That cursed thing broke on the same evening. I was like 😮

Story && rant && dev && linux

I was using linux mint for a while... more like 5 months for work, there's this Touchpad/mouse issue in it that was driving me crazy, so basically the mouse stops responding out of nowhere in the middle of my coding and I have to restart the fucking laptop to get it back. Yeah, I tried all the solutions I could find on the Internet and nothing works.

This issue likes to fucking mess with me so much, it seems to only happen when I absolutely mustn't restart the laptop or I'm working on a task and have a tight deadline and I don't have time to waste restarting my pc.

A couple of days ago, I had this major feature I needed to release to production and the time I estimated for it and shared with my team turned out to be insufficient, so I had to work extra hours from home to finish it ... while I was working, the mouse issue returned and I had to restart my pc like 20 times that day. It was fucking frustrating and It was already midnight and all you can hear are keyboard sounds and fucks flying.

I made a promise to myself that once i finish this task, I'm gonna fucking migrate to another distro, I'm fed up with linux mint's BS. I've been putting up with it for so long it's time to move on.

Yesterday I installed Manjaro and I'm happily working on it today xD.

FUCK! For fucking years, you 5 brain celled gimp cunts have been forcing horrific images into production. Are you seriously fucking brain dead, blind or just don't give a fuck because unless your eyes are painted on, even a fucking partially blind person can see that your images aren't scaled proportionately or even match the perfect one that was there in the last release.

Seriously?! At what point did you dim witted fuckwits look at that image that was stretched worse than your mouth from all the cocks in it, and think yeah.. I can put that in production... again...for the millionth time.

But ok, I'm over reacting for having to mention it again. 👍🏼

YOOO Product owner if you don't specify which freaking page something doesn't work I CANNOT DO ANYTHING FOR YOU!!!!

Weeks after our partner has been nagging us to release to production because we've been putting it off because their API sucks and wouldn't properly work.... we finally did....

except their production API doesnt even work LMAO

Two days ago I wrote the deployment instructions. 5 lines. I sent them to the devops four days before the release (two days before usual).
A colleague of mine leashed out and had me send another message to say to ignore my instructions because they "generated too much entropy" he is releasing too his application and we should create a single instruction file. Okay, I see no reason to do that nor how that helps the devops. A longer file is not easier to understand than a smaller one.
Today the devops deploy our application. They make a backup of the new files and promptly overwrite the original copies with the files from production.
I lost 3 hours today. My colleague is refusing to communicate the error properly to the devops and I have a meeting in 20 minutes. I love my job.

One of our projects migrated their file-repository to another one during a major release.

Instead of giving this task to an experienced programmer, they gave it to the head of the respective dev department due to the usual release panic.

Soo.... He wrote the migration tool. It was executed during the release. Everything seemed fine so far.

A few days later. Someone from the above project came to my team due to some "strange behaviour on the production database".
They reported that they couldn't download some of the user's documents due to unknown reasons.

After quickly analyzing the current state of the new file-repository, we concluded that the affected documents did not exist in the new repository.

Then we took a look at the so called migration tool...
Well.. After nearly 30 min. we knew the root cause for that.
They only migrated the first 4 levels of the folder structure. Due to the assumption that "we don't use deeper nesting". (Facepalm)

As the head of their department wrote it, no one seems to questioned it either. Nor did they made a code review and ended up with a tool with hard coded urls to the production db, no version control, no build tool, no ci, nothing. Breaking nearly every possible company standard.

However.. That's not it. When analyzing their migration tool we noticed another even more dangerous thing.
They mixed up the id generation of the migrated documents resulting in a random assignment between customers and documents. Which is quite bad as this contains sensitive information. E.g. passports

They offered us quite a nice amount of money to fix this until EOB. We declinded as it was simply not possible in that time, but agreed to support them with the new tool.

After some time I heard that they migrated production again. And they fucked it up again. They never talked to us after we offered them support...

The third and final migration was written by us. Not only migrated it correctly. It was also way faster. By factor 20.

In the end we haven't gained anything from this rushed project as the penalties were piling up due to this fucked up migration.
After all this time I'm not sure who is to blame. In my opinion, partly all of them.
Head of department who can't and shouldn't code.
Seniors who didn't review the code and didn't ask for help.
Release mgmt who put way too much pressure on the devs.

.... herein is a major reason I despise this time period.

Between people uniting to turn every masonic and historical site look like absolute shit, its being stoked over something new: a repeat film release from way back in 2021 of an arthurian legend....

and diverse workforce practices strike again.
can you see in this image why this film might suck and be a pussy production not meant to entertain anyone who actually questions historical or even literary precedence ?

If I haven't mentioned this recently.
I hate you half baked fucks. I mean my god. Nell Flanders has morgan freeman in it and they managed to set a decent message in it without being inaccurate !

Today during a follow-up meeting of the grand project I'm workng on...

TL: ... and I want to start working on the production environment and have it ready by next month.
Me: (interrupts) hold up! We are not ready, we have a huge backlog of technical tasks that need to be addressed and we are still not in possession of the very crucial business and functional requirements that you are supposed to provide. The acceptation environment is just set up on infra perspective but does not have anything running yet! The API we depend on is still not ready because you keep adding change tasks to it. We have a mountain of work to do to even get to a first release to integration yet and there is still the estimations on data loads and systems... your dream will not be possible until at least Q2 of 2024.
TL: stop being so negative @neatnerdprime and try to be more customer friendly. I want it by the end of the next month.
Me: remember what I said to you about moving prematurely. Remember I don't take any responsibility if things break because you rush the project. Please, reconsider!
TL: I just want it, please do it

FUCK YOU YOU SORRY EXCUSE OF A PEOPLE PERSON KNOWING JACK SHIT AND JUST LICKING THE MIDDLE MANAGEMENT ASSHOLE TO RECEIVE ATTABOY PETS ON YOUR UGLY ASS BALD HEAD AND CROOKED TEETH. YOU SHOULD FUCKING DIE IN A FURNACE AND LEAVE NO TRACE BEHIND.

When another developer has put a "Yolo" production log in de codebase and committed that to the release branch..

We basically don't unit test at work. I write some tests for my code and honest to God people complain I'm wasting time saying a test bed and manual tests are good enough. We don't write test beds for about half of our production code and rely on integration tests for the rest. We only test release builds which have been symbol stripped, I get handed a crash report with no stack trace that I'm unable to reproduce and expected to stay late to fix it for some arbitrary internal deadline.

I've since moved to R&D where basically I'm left to do my own thing so it's better.

We don't project manage. Project leads take time estimates and double them so management might cut them some slack. This doesn't matter because management made up time estimates before the project started. Last project I was on had a timeline of 3 months and took a year.

We have released broken products. Not that any of the above really matters, our software products have made about 50k revenue in 2 years. There are 6 people on software. Fortunately hardware has made about 3 mill. That said our hardware customers are getting frustrated with us as we keep fucking up, shipping broken products and missing deadlines.

I've been working there about a year and a half and will be looking for a job at the end of the current project.

I joined devRant about when I was most pissed off with my job, my rant frequency has definitely gone down since I moved over to R&D.

Don't need Netflix when you have a production deployment right before a long weekend. It has failed since last two weeks due to vulnerabilities present in one of libraries(P.S. FUCK JAVASCRIPT and Post release vulnerability scans!). You have rewritten the whole functionality from scratch twice! Security gates finally open for you, welcoming with arms wide open. So you click Deploy! DAFUQ!! FUCK MY LIFE! Deployment failed! It's only a 3 hour window to deploy! You frantically re-review your code, is it me?? Not again!! It isn't! Well, why is the deployment failing, you work against the clock. Going through configs, code, documentation! WTF is it?? Should I give up and raise a support ticket? Nope! You login to the server, sifting through logs and configs, there's a couple of other tickets with today's deadline. What are you going to do? And you get a hint! You take the hunch, change the config 5 minutes before deadline!
Get merge request approved, wait for the build, hit DEPLOY!! Nail biting 3 minutes! Your eyes fixed on the logs! Building..... Pushing instances..... Starting App..... SUCCESS!!! Finish the remaining tickets! Your long weekend still exists!

Thursday afternoon. Client gives us the go to deploy the latest release to production.
Friday late afternoon; my colleague - "wait, did we ever deploy"? Me - "uh, nope".
"Alright have a good weekend"

So we released to production today (Friday), not my decision.
All pages work fine expect for the one page which I added a new feature.
It worked fine in Chrome and Edge. But after release a customer who requested the feature said it doesn't work for him. Screenshot showed he was using IE.
Horror time.. it was evident that it has to be the changes to the JavaScript I did, but why does the whole page doesn't work.

So I started debugging. Nothing works on that page in IE11, it doesn't even load the fucking script file. Then I dared to change mode to IE10, it actually gave me an error in my script file. The bad IE has actually picked a mistake that other browsers didn't.

So, the mistake is fun part too.
I had the following jQuery (or Jake Weary) call
$.getJSON(
'/url',
{
argA: a, argB, b, argC:c
},
function (){
// did something
}
);

In second argument, I accidentally typed comma instead of colon. Chrome and Edge ran the script perfectly passing all the arguments.IE 11 failed to load script without giving any error and only IE 10 gave an error of expecting a colon.
I do not know which browser to blame.

PS I didn't try in Firefox, safari, etc.

I have had it with the new pm for my project. We've had to literally write down rules for him to follow on how we want to run our project.
- we always go over time on standup because he's asking everyone specific questions about each ticket we're working on instead of just letting us give our updates. Wrote a rule for not doing that.
- he gets overly excited to get things done to the point that he's approved a PR himself and merged a production release during off hours.
- we're a team of 3 devs but he has 7 big items he wants us to work on concurrently.
- we use jira to keep track of things but he insists on us updating a spread sheet that he made as well. We just straight up told him no but he'll bring it 3 times every week.
- he wants us to write a report of our daily progress in our jira ticket before we give the same updates on standup
- every time I give him an estimate, he tells the clients it's a promise we'll get it done by x date.
- he never pushes back and says yes to everything
- oh here's a funny one, he's "reviewed" my code several times

Idk what to do here anymore. We've literally talked to him about all of these issues. He'll change for a few days and go back to doing whatever he does.

I can't believe this shit happened in time for this week's rant!

Here it goes.

I have a table on AWS Athena which has partitions. Now, in the earlier versions of this project whenever I write something to a new partition a simple `MSCK` query worked (and keep in mind I am NOT deleting anything)!

Now, my so called Team Lead in the PR for the latest (major) release tells me to change it to an `ALTER TABLE`. I was like fine, but I did not add the s3 location to it, because it was NOT NEEDED. TL asks me to add location as well. I try to convince this person that it's not needed, but I lose. So there it is in production, all wrong.

Today I notice that the table is all fucked up. I bring this up in the stand up. The main boss asks me to look into it, which I do. Figure out what the issue is. This TL looks at it and says you need to change the location. I put my foot down.

"NO. What I need is to remove the bloody location. IT'S NOT NEEDED!"

TL's like, "Okay. Go ahead"

Two things:

1. It's your fault that there's this problem in production.
2. Why the fuck are you looking into this when I was clearly told to do so? It's not like you have nothing to do!

Ok so some of you have probably seen my previous rants about my computer science teacher and our project but I'm just going to summarize all of them and share with you more of my pain.

1. He edits in the production environment. Its a laravel project and he is creating test database migrations IN THE PRODUCTION ENVIRONMENT AND SWITCHING BACK AND FORTH FROM MASTER AND DEV.

2. He edits in vim and doesn't follow codestyle even though I printed him off a piece of paper and emailed it to him.

3. He doesn't have any ethic when it comes to more complex things like laravel homestead.

4. He doesnt want me to release features even though he takes really long to do them.

While I love vim and it is my editor of choice, some things should be done in an ide. This is really annoying me and I'm really just considering handing him the project if he can't follow basic outline.

I HATE the idea of only releasing on pre-determined schedules despite work being completed and just waiting for that day to arrive.

I'm a co-founder of a small software company. We have partnered with another particular company that also writes software. Some of our clients have access to paid content of that company's services through our application.

Every once in a while, our clients will report issues with that company's service to us, because they access it through our application. They think it's our issue.

We then pass the report on to the partner company, telling them that their stuff is broken. Their reply goes like this:

"Ok. We'll get the bug fix scheduled, and we'll release it next Thursday."

"Next Thursday? The issue is now, they can't use the service."

"That's our scheduled release date."

O.M.G.

We voluntarily walked away from our safe, cushy jobs working for other people, taking enormous pay cuts to start this company. Now, we're 6+ years in, disrupting established fat-and-happy competitors in this space. I GUARANTEE you that if we had that same attitude, we would have been absolutely obliterated early on.

We are quick. Guided by kanban boards, our suite of unit tests and integration tests is vast and kick-ass. With continuous integration and the click of a button we know if we broke something or if the piece we're working on is ready to be pushed to production, IMMEDIATELY. Our "release schedule" is when the damn thing is complete.

It isn't all bad. Our integration with them has been beneficial for both of us. I just loathe their snail's pace which negatively affects our mutual customers. It can make us look bad, and we can do nothing about it.

Blah.

Build pipelines are awesome. What's not awesome is forgetting the 'pipeline' bit.

Sure, let's have each TeamCity job in the 'pipeline' build the deployable from different places! You must manually merge your code from dev to test to release.

What's that? The functionality working in dev isn't there in production? I wonder how that happened.

TGIF & remember..

DO NO FUCKING PUSH TO PRODUCTION TODAY!
DO NOT FUCKING RELEASE A NEW VERSION OF THE CLIENT APP!
FUCK!!
have a nice weekend partners 🤗

When you make a release on friday and then spend the whole weekend fixing bugs in production.

delete random, unused column from production database
entire app blows up
re-release last change on code pipeline
everything works beautifully
¯\_(ツ)_/¯

You've got the perfect architecture lined up, a great idea about how to transition and you're about to start refactoring. Then it hits you. You've got 10 clients running your code in production and you'll have to migrate their data to the new structures, from the blob to each new microservice datastore. For each release. FML.

Being the sole contributor to a project is such an experience. After endless nights of bug fixes and brainstorming, I have breleased my app to play store.
Currently it is Alpha phase . I'm getting some positive feedback as well as some great comments from my friends . Really looking forward to release the project in production soon

How much zucchini is too much zucchini?

I know I have WAY too much...

I knew at least when 1st considering D20 zucchini breads.

then when i began to wonder if the remaining batter would work with my death star waffle iron...ill know tomorrow!

....ran out of typical pans, incl foil ones(normal and mini for easy gifting)

- gave 1 away (similar sized as in pic)

- approx. 2 lg zucchini bread loaves in fridge (gave away 2, ate a ½)

- cut up\froze enough onions\peppers\pak choi to a min. acceptable zucchini : everything else stir fry ratio... x20 servings

- similarly, green onions, pak choi, marinated sesame fried tofu bits, zucchini and miso (quick miso soup) x16

- thinly sliced enough to layer it into ~20 lg servings of lasagna.

... zucchini in pic is slightly larger than the one that made the many aforementioned and pictured loaves of zucchini bread

apparently, in a week tops, I'm gonna have at least another 3 more THAT size needing to be picked

anyone in the continental US want some zucchini bread? or, if in michigan, zucchinis?

i didnt even plant much... actually only about ½ of other years.

i am also having some serious overflows coming of (at least) grapes and watermelons.

grapes...
when i bought this place, this odd, square, surrounded by cement walkways, area, with an increasingly problematic tree (risking cable\electric lines, foundation, etc) and so dense with weeds that I learned, dandelions have a giant, bush-like form, with heights beyond 8ft tall.

i grew up hanging out in the nearby woods, noticing that weeds lost the fight vs raspberry\blackberry plants. being handicapped\lazy\experimental, w\ev, i figured id just kill it all then fill it with random berries... knew nothing about grapes so just got 4+ random types... apparently they are all fancy\expensive grapes... and reeeeeaally produce. i already had to pick ~10lbs.

watermelons-
idr if i planted normal ones and little ones or just little ones... idk how to tell without cutting them open or maybe just watching a long time to see if they stopped growing?
anyone with advice (or seeking watermelons) is welcome.

assuming (hoping) they are mini ones there's at least 2dz that are at least ping pong ball size.... and around 100 little yellow flowers still.

i totally get that my frustrating problem with produce here would be beyond welcomed by most people... but seriously... wtf do i do with a few dozen to over a hundred (hopefully mini) watermelons, so many zucchini that, despite personal daily consumption and at least a half dozen friends that love zucchini bread and\or my secretly healthy lasagna(my friends tend to be guys), but have their limits capping out, plus mine, at less than ½ whats rapidly being produced and, apparently, thousands of dollars worth of hundreds of pounds of fancy grapes???

there's an interesting old lady across the street who'll take at least what her and husband can possibly consume,.. even makes grape jam, but thats still only a few dz lbs tops.

it seems wrong to kill the plants (or even to remove a large amount of blossoms and feed them all to JSON (lil tortoise)... pretty sure he's already getting tired of them just from the few that fell off in the wind or something.

i wish i knew some farmers that do farmers market things... but that kinda seems super suspicious... 'hey mr farmer... want a large supply of expensive grapes, watermelon and zucchini, for free? you can sell them to random people, or just give them away. i dont want money or anything...' idk... seems like the beginning of one of those movies that either has evil alien plants assimilating all land mammals, or where there's some crazed medical researcher convinced that there's a massive, underrated threat without enough attention for vaccination production funds-- so they are gonna release some deadly virus supposedly to save the world.

ive been cooking too long.

ideas pl0x?

Weekend thought: What counts as stable in development?

From my experience it seems that "stable" is a relative concept. My linux server is "stable" in the sense that the packages are tested for a long period of time before release, but my home distro is a rolling release and that is also stable in my opinion. So which is it? Can it be both? Or maybe we're just lying to ourselves that anything is stable.

When I'm developing web applications I always have this rule that is the user can't enter and exit the application without a major error coming out, it isn't ready for production. Once that's out of the way, from my point of view the application is stable. But if I were to present this to a company would they think the same? Probably not.

What do you think counts as a stable production release?

In last episode of "How SystemD screwed me over", we talked about Systemd's PrivateTMP and how it stopped me from generating SSL certificates.

In today's episode - SystemD vs CGroups!

Mister Pottering and his team apparently felt that CGroups are underused (As they can be quite difficult to set up), and so decided to integrate them into SystemD by default. As well as to provide a friendlier interface to control their values.

One can read about these interactions in the manual page "systemd.resource-control"

All is cool so far. So what happened to me today?

Imagine you did a major system release upgrade of a production server, previously tested on a standalone server. This upgrade doesn't only upgrade the distribution however, it also includes the switch from SysVInit to SystemD. Still, everything went smooth before, nothing to worry now then, right? Wrong.

The test server was never properly stress-tested. This would prove to be an issue.

When the upgrade finishes, it is 4 AM. I am happy to go to bed at last. At 6 AM, however, I am woken up again as the server's webservices are unavailable, and the machine is under 100% CPU load. Weird, I check htop and see that Apache now eats up all 32 virtual cores. So I restart it, casting it off to some weird bug or something as the load returns to normal.

2 hours later, however, the same situation occurs. This time, I scour all the logs I can, and find something weird - Many mentions that Apache couldn't create a worker thread? That's weird.

Several hours of research and tinkering later, I found out the following:
1 - By default, all processes of a system that runs SystemD are part of several CGroups. One of these CGroups is the PID CGroup, meant to stop a runaway process from exhausting all PIDs/TIDs of a system.

This limit is, by default, set to a certain amount of the total available PIDs. If a process exhausts this limit, it can no longer perform operations like fork().

So now, I know the how and why, but how should I solve this? The sanest option would be to get a rough estimate of just how many threads the Apache webserver might need. This option, though, is harder, than apparent. I cannot just take the MaxRequestsWorkers number... The instance has roughly double the amount of threads already. The cause being, as I found out, the HTTP/2 module, which spawns additional threads that do not count towards this limit. So I have no idea what limit to set.

Or I could... Disable the limit for just the webserver via the TasksAccounting switch. I thought this would work. And it did seem to... Until I ran out of TIDs again - Although systemctl status apache2.service no longer reported the number of tasks or a task limit of the process, the PID CGroup stayed set to the previous limit. Later I found out that I can only really disable the Task Accounting for all the units of a given slice and its parents.

This, though, systemctl somewhat didn't make apparent (And I skimmed the manual, that part was my fault)

So... The only remaining option I had was to... Just set the limit to infinite. And that worked, at last.

It took me several hours to debug this issue. And I once again feel like uninstalling systemd again, in favor of sysvinit.

What did I learn? RTFM, carefully, everything is important, it is not enough to read *half* the paragraph of a given configuration option...

Oh, and apache + http/2 = huge TID sink.

15 mins before the Production Release, after accidentally merging the Dev branch with the Master one :)

Who the hell hardcodes their localhost ports in a web.config without updating the release config to the correct production URLs? And why doesn't our ops team pick up on this shit before clicking their fancy deploy button? And why in holy heaven do we even have a pre-production server if it isn't an exact mirror of production?

God help me, I need a drink.

After three months of development, my first contribution to the client is going live on their servers in less than 12 hours. And let me say, I shall never again be doing that much programming in one go, because the last week and a half has been a nightmare... Where to begin...

So last Monday, my code passed to our testing servers, for QA to review and give its seal of approval. But the server was acting up and wouldn't let us do much, giving us tons of timeouts and other errors, so we reported it to the sysadmin and had to put off the testing.

Now that's all fine and dandy, but last Wednesday we had to prepare the release for 4 days of regression testing on our staging servers, which meant that by Wednesday night the code had to be greenlight by QA. Tuesday the sysadmin was unable to check the problem on our testing servers, so we had to wait to Wednesday.

Wednesday comes along, I'm patching a couple things I saw, and around lunch time we deploy to the testing servers. I launch our fancy new Postman tests which pass in local, and I get a bunch of errors. Partially my codes fault, partially the testing env manipulating server responses and systems failing.

Fifteen minutes before I leave work on the day we have to leave everything ready to pass to staging, I find another bug, which is not really something I can ignore. My typing skills go to work as I'm hammering line after line of code out, trying to get it finished so we can deploy and test when I get home. Done just in time to catch the bus home...

So I get home. Run the tests. Still a couple failures due to the bug I tried to resolve. We ask for an extension till the following morning, thus delaying our deployment to staging. Eight hours later, at 1AM, after working a full 8 hours before, I push my code and leave it ready for deployment the following morning. Finally, everything works and we can get our code up to staging. Tests had to be modified to accommodate the shitty testing environment, but I'm happy that we're finally done there.

Staging server shits itself for half a day, so we end up doing regression tests a full day late, without a change in date for our upload to production (yay...).

We get to staging, I run my tests, all green, all working, so happy. I keep on working on other stuff, and the day that we were slated to upload to production, my coworkers find that throughout the development (which included a huge migration), code was removed which should not have. Team panics. Everyone is reviewing my commits (over a hundred commits) trying to see what we're missing that is required (especially legal requirements). Upload to production is delayed one day because of this. Ended up being one class missing, and a couple lines of code, which is my bad (but seriously, not bad considering I'm a Junior who was handed this project as his first task at his first job).

I swear to God, from here on out, one feature per branch and merge request. Never again shall I let this happen. I don't even know why it was allowed to happen, it breaks our branch policies. But ohel... I will now personally oppose crap like this too...

Now if you'll excuse me... I'm going to be highly unproductive and rest, because I might start balding otherwise after these weeks...

When it's production release Tuesday and you get flooded with crash logs

This is more of an advice seeking rant. I've recently been promoted to Team Leader of my team but mostly because of circumstances. The previous team leader left for a start-up and I've been somehow the acting Scrum Master of the team for the past months (although our company sucks at Scrum generally speaking) and also having the most time in the company. However I'm still the youngest I'm my team so managing the actual team feels a bit weird and also I do not consider myself experienced enough to be a Technical lead but we don't have a different position for that.

Below actions happen in the course of 2-3 months.

With all the things above considered I find myself in a dire situation, a couple of months ago there were several Blocker bugs opened from the Clients side / production env related to one feature, however after spending about a month or so on trying to investigate the issues we've come to the conclusion that it needs to be refactorised as it's way too bad and it can't be solved (as a side note this issue has also been raised by a former dev who left the company). Although it was not part of the initial upcoming version release it was "forcefully" introduced in the plan and we took out of the scope other things but was still flagged as a potential risk. But wait..there's more, this feature was part of a Java microservice (the whole microservice basically) and our team is mostly made of JS, just one guy who actually works as a Java dev (I've only done one Java course during uni but never felt attracted to it). I've not been involved in the initial planning of this EPIC, my former TL was an the Java guy. Now during this the company decides that me and my TL were needed for a side project, so both of us got "pulled out" of the team and move there but we've also had to "manage" the team at the same time. In the end it's decided that since my TL will leave and I will take leadership of the team, I get "released" from the side project to manage the team. I'm left with about 3 weeks to slam dunk the feature.. but, I'm not a great leader for my team nor do I have the knowledge to help me teammate into fixing this Java MS, I do go about the normal schedule about asking him in the daily what is he working on and if he needs any help, but I don't really get into much details as I'm neither too much in sync with the feature nor with the technical part of Java. And here we are now in the last week, I've had several calls with PSO from the clients trying to push me into giving them a deadline on when will it be fixed that it's very important for the client to get this working in the next release and so on, however I do not hold an answer to that. I've been trying to explain to them that this was flagged as a risk and I can't guarantee them anything but that didn't seem to make them any happier. On the other side I feel like this team member has been slacking it a lot, his work this week would barely sum up a couple of hours from my point of view as I've asked him to push the branch he's been working on and checked his code changes. I'm a bit anxious to confront him however as I feel I haven't been on top of his situation either, not saying I was uninvolved but I definetly could have been a better manager for him and go into more details about his daily work and so on.

All in all there has been mistakes on all levels(maybe not on PSO as they can't really be held accountable for R&D inability to deliver stuff, but they should be a little more understandable at the very least) and it got us into a shitty situation which stresses me out and makes me feel like I've started my new position with a wrong step.

I'm just wondering if anyone has been in similar situations and has any tips or words of wisdom to share. Or how do you guys feel about the whole situation, am I just over stressing it? Did I get a good analysis, was there anything I could have done better? I'm open for any kind of feedback.

My first dev job was for a .net shop. Until then, I had only worked in Java and PHP. This place didn't have the normal team structure, and I soon found that I was going to be working solo on the projects I was responsible for. I'm my first week there, I was tasked with making make revisions to an application in a new language, with a new toolkit, solo. A few weeks later was the most intense day I've had as a dev, as I put in the change control to release my update to production.

So here I am coming to the end of the week after getting The Porcupines big web project into production. Pulled a 38 hour straight from Sunday to Monday chopping wood to make the thing fly. Pulled in other programmers and content creators to get the site full of something we did not have a week before. The fun part was having the account manager right there for 30 plus hours and actually seeing what it take to save a project when the client just thinks "it's just code". Now the boss has is asking for a list of all the work out of spec as they are bitching about the extra cost. These were the clients who did not read the functional spec and raked me over the coals after release that the home page did not match the design (the home page matched the design). I warned my team this would happen. They get all swept up in the hype and We can win! frame of mind and you can bet when the bitching starts it will come back to the paperwork you did at the beginning and the change requests and productions systems reports so you can wave it in front of ungrateful clients and not end up sad. Make sure you keep notes and document all of the requests and changes from internal and external even if you do not have to. one day it will save your ass and you will be able to whip it out and be a smiling motherfucker.

I have a disagreement with my product owner (PO). Our team develops APIs in Mulesoft. We've got a release coming up and PO wants to release one of the APIs that has *some* working endpoints, but other endpoints in that same API have some open bugs.

Given that it's *unlikely* that those broken endpoints will be used, does it seem like good practice to release the API with known bugs in it to hit the deadline?

Understand that we're not just releasing the working endpoints. We're releasing all of it, bugs and all. PO's logic is that those broken endpoints won't be used therefore it's fine to send known bugs into production.

I just need some advice in dealing with this

So, today we had our first production release of our web app. Last week we changed a big part of our UI. This week we changed the design, rewrote the complete API documentation, implemented mobile support. While the release the administration center of our cloud was unreachable. Shortly after the release we made a bug fix and deployed it directly to production.
So today was a very normal prod release 😂

Last weekend I was working on a small project for a friend of mine: a dockerized webapp, plus API backend and DB. I had some problems with the installation on the vps and had to try out different images and never really did a complete setup of my usual dotfiles. Got it running on an Ubuntu distro. Everything great.
It was the first release so I still had to check that every configuration worked ok, like letsencrypt companion container, the reverse proxy and all that stuff, so I decided to clone the whole project on the server tho make the changes there and then commit them from there.
Docker compose, 10 lines of code, change the hosts and password. Boom everything working. Great... Except for the images in the webapp.

WTF? Check the repo, here they are, all ok. I try different build tactics. Nothing. Even building the app on another docker always the same. Checked browser cache, all the correct ports are open. I even though that maybe react was still using some weird websocket I didn't know, but no.
Damn, I spent 5 hours checking why the f*** the server wouldn't make it out.

Then, finally, the realization...

I didn't install the f******* git-lfs plugin and all I was working with were stupid symbolics links! Webpack never even throw an error for any of the stupid images and the browser would only show a corrupted image, when decoding the base64 string.

Literally the solution took 5 minutes.
F*** changes on production, now I do everything on a fully automated CI.

What a fucking day.
Half a day looking for functionality on github that was lost in some branch and not merged to master. All remotes was pruned. Finally restored it from some remote on production and merged to master before Saturday release. Yeah !!!!
Month saved fuckers, pick some more hardcore shit to surprise me. To strong to fail.

Some idiot fixing bugs in production and overwriting files without updating his git repo when I pushed another bugfix live.

Boss to me: "it's your job to get the fix live!"

I FUCKING HATE MONDAYS!

screw performance i'm gonna run gulp.watch in production and just git reset it to last release when someone fiddles with files on the server :(

Hotfixed a bug in production. Reintroduced in the next release. Is it just me?

Creating a real mvp prototype, and not a "mvp prototype" that ended up being a production release...

Sometimes, certain features don't work in my app if it's uploaded to the production track in Google Play.

It works in debug mode.
It works in release mode.
It works in the internal testing track.
...but it won't work in the production track? Y'know, the one that actually matters? Theoretically, there shouldn't really be a difference if the same exact APK was uploaded to the internal testing and production track, but... idk dude.

As a result, I implemented a secret way to test a feature in the production build (it's an app to remotely control OBS Studio): if the first connection you added is named "yayeet_" and you open the disclaimer, it tests the feature. Luckily, I got some of the stuff figured out, but I just thought the way I had to test it in production was dumb.

Sometimes I don't get "don't test on production".
And I'm definitely not a front-end guy, I only have debug and release in mobile development.
And I definitely often test on release, because it may be broken while debug build works fine.
You know what that means?
1. Test locally
2. Try to fix issues
3. Realize that this issues would ever appear ONLY locally
4. Move to staging and test
5. Fix issues
6. Realize that most of them are caused by workarounds for localhost
7. Move to production
8. Realize that everything is fucked up and you don't have any idea why, because `h5aqq2 was called on null"

Critical bug in production? Sorry, can't fix it right now: We've got a build running with 1 hour of build time left, 8 hours of automatic tests, 3 days of manual testing, and a partridge in a pear tree. Your fix can be in the next release.

That shitty moment when you are finally about to release your code, after about one month of developing and testing, and making sure everything is OK, imagining: "Oh we're finally releasing this feature, I have worked so hard on it, it's going to kick some ass!" but surprisingly things get fucked up on production server... I mean seriously? Stupid middleware I killed myself to get to work messed up. Where the hell have you been in staging, you stupid little bug? You happy now? My CTO giving me awkward looks and shit like: "I'm sorry but you have to come fix it, during weekend." The best way to fuck up my mood, today is the last day of week for god's sake!
I hate releasing like this. seriously SAG in this release!

Just joined a new company and can only describe the merge process as madness.....is it or am I the one that is mad?!

They have the following branches:

UAT#_Development branch
UAT#_Branch (this kicks of a build to a machine named UAT#)

Each developer has a branch with the # being a number 1 to 6 except 5 which has been reserved for UAT_Testing branch.

They are working on a massive monolith (73 projects), it has direct references to projects with no nuget packages. To build the solution requires building other solutions in a particular order, in short a total fucking mess.

Developer workflow:

Branch from master with a feature or hotfix branch

Make commits to said branch and test manually as there are no automated tests

Push the commits to their UAT#_Development branch, this branch isn't recreated each time and may have differences to all the other UAT#_Development branches.

Once happy create a pull request to merge from UAT#_Development to UAT#_Branch you can approve your own pull request, this kicks off a build and pushes it to a server that is named UAT#.

Developer reviews changes on the UAT# server.

QA team create a UAT/year/month/day branch. Then tell developers to merge their UAT#_branch branches in to the previously created branch, this has to be done in order and that is done through a flurry of emails.

Once all merges are in it then gets pushed to a UAT_Testing branch which kicks off a build, again not a single automated test, and is manually tested by the QA team. If happy they create a release branch named Release/year/month/day and push the changes into it.

A pull request from the release branch is then made to pre-live environment where upon merge a build is kicked off. If that passes testing then a pull request to live is created and the code goes out into production.

Ahhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh it's a total mess. I knew when I took on this job it would be a challenge but nothing has prepped me for the scale of the challenge!! My last place it was trunk based development, commit straight to master, build kicks off with automated testing and that just gets pushed through each of the environments, so easy, so simple!

They tell me this all came about because they previously used EntityFramework EDMX models for the database and it caused merge hell.

First. I clarify my work schedule is from 7am to 4pm. I have a personal emergency so I must leave on time today.
Now my story: Today (finally) at noon they decide to publish the iOS and Android applications. The thing with the Android application is the other Developer is with a last minute improvement (since Monday) and is not over.

It's 2:45, the iOS app has already been sent for review, but Android is not. So when the Architect says that he already talked to the client and told him that everything is ready today, I asked the Developer if Android is already? and his response was "Almost I will finist at 3pm or 3:30".
(Hmmm) I'm worried about time so I say Ok, then Android will be published tomorrow! God he needs to finish the development, and I'm going to take new screenshot, do the merge with the development branch and everything that's need for a production release. So, the Project Manager says "Hell no! It will release today!" My answer: I have to leave at 4 and there is a lot of haste to do something so delicate.

I'm still waiting for an answer in slack from her.

Then the architect very "professionally" tells the other Developer to do it himself. It's almost 6pm and they still have not done anything

They had a meeting without me for a production release that was happening tomorrow. Rescheduled the release on my wife's birthday without asking anyone. This was the response when I let this known "Happy Birthday honey, you get a night without me."

My first production support release is next week and its from midnight till 5am and be at work at 8:00am. If something goes wrong with any of the other developers from our team goes wrong I have to figure it out. And this image is exactly how I picture it goes, if any tickets fail. (🤘🏽 bring it on! )

Today is release day!
Got a whole set of new features to deploy on production!
Also, internet at the office has been dead since 6am.
I'll take a coffee break.

The production bug conundrum:
The new release that's going to fix the problem isn't ready yet, but hotfixing it means merge-hell for the new release.

So this modeler on a Dev call, I have this new shiny model, let's release this to production mid November😳 (Seriously that's how he started out the first conversation).

2 min silence, everybody looks at each other for reaction, just like a TV shows !! 🤣🤣

And the my Manager lists out the things that would be required to before we ship this out.🤐

Modeler : Oh I guess we won't be able to deliver it this year.😤

I am like what were you thinking. Everything is not just import an Excel in R and crunch numbers and write reports and show graphs. is it?

There is a real development cycle that has to do all of the above on not so pretty data, at scale reliably for 100s of clients and not just your laptop.

finally we started adopting gitflow concept and it worked pretty well for most of the projects and devops guy came in saying Gitflow needs to die in a fire because we devops is the best, all the tech companies are using it.

But...

1. gitflow can be tweaked to suit the project's need
2. gitflow can work with devops model
3. our projects are still release based

noooooooooo! gitflow needs to die in a fire! everyone needs to commit directly to master branch now and we CI directly to production!

and some dude started doing so because some random dude said it. wtf?

whats wrong with people?

Bug found in production just after release. This is why unit tests are so important!

I had a colleague, who built a bunch of smaller systems for the company I'm working in. He didn't want to waste his time building a "perfect" system (which I generally agree with, the question is just where to draw the line).

But because it took him so long to build the prototype, usually it went into production without being hardened (like basic input validations were missing. It wouldn't allow anything malicious, but instead of a validatiom error it'd just 500).

When he left, literally less then a week later, one of his systems, which was a prototype and nobody except him could maintain, because it was done in a fancy new technology, which wasn't even v1 at that time and their documentation said, it's production ready when we release v1. Anyway, that one system started crashing just few days after him leaving. Another Dev and me tried to fix it, but every time we touched it, it just got worse.

At some point, we gave up and just configured a cron job to reboot it every 12h. He could have probably fixed it, but to us it was just black magic.

Anyhow, this rent isn't about him, AFAIK all the systems still working, as long as you provide the correct input. Nor is it about the management decisions, which lead to this Frankenstein service on live support, which we had to increase, to be restarted every 8 hours, 6h, 4h, 3h, .....

It's about the service itself, which I'm looking forward to every day, when the rewrite will be done and I can nuke the whole git repository.

I was even thinking about moving all the related files onto a USB stick and putting that on 🔥, once we're done rewriting it....

Maybe next month or in 2. Hopefully before we'll have to configure the cron job to restart the service every couple minutes....

What's the most insane deployment scheme you've had to work with? One client has a release schedule that deploys all major projects once a month(!). Bugfixes get deployed once a day (systemwide), so any issue that can't be verified until it's in production has at least a days delay when iterating.

Google play console is so fucking confusing, I'm trying to get my app to open beta testing stage, I've completed all the steps and done the full roll out, but the status of the app is still pending publication (3 days), wtf does "Pending publication" means do I need to publish the app for production in order for beta testing release to work, or I have to wait for google to approve the beta release for publish?
I'm lost

Just finished deploying to production. Now to find a mate and release some endorphins.

Technical lead decides to change some transitively resolved dependencies to Maven with some configurations not existing, 2 days before release to production. Thanks to our none existing regression tests, no users can log in after releasing. Guess who can fix it ASAP (of which business thinks it means, in half an hour)

I think I just realized what my biggest gripe about our career paths that I hate the most.

This is something that has worsened over time, especially the last 2 to 3 years.

As developers, we have far too many options. Some of the most powerful apps are written with languages that have hard, and I mean HARD, guardrails in place. If the app is written in a language that does not meet this criteria usually a framework has been used to install those guardrails.

We just get our minds so wrapped around the possibilities and the opportunities in the software, that we just can't focus on the end result. We're like puppies that are excited about something and we just piss all over everything.

In my career I have met far too many developers that don't have the capacity and mental fortitude to take control of their actions. Because of this I think the only way for us to stop this corruption, that I feel we are nurturing, the solutions/services that we use need to push back on us and install those guardrails for us.

All this came from a change that Microsoft put in place that seems well intended, but introduces yet another choice and a multitude of opinions in how you release code.

It used to be a simple check box. If it was checked it was pre-release, if it was unchecked it was a production release. That's it. On or off. The simplest choice you ever needed to make on a release.

Now though, there are two check boxes. One for a pre-release and one for a latest release. You can also not check either for some "ephemeral" release? So now something as easy as on or off has been made into a difficult decision on how this works within my pipeline. Now every time I make a release I have to ask myself, "which one do I check?"

I shouldn't need to spend more than a second to identify a path forward on simple shit like this, but here we are with a third choice.

Can we just stop overcomplicating shit?

When you fixed p1 bug in the production and waiting for the release is like watching the conjuring movie all alone in the night. It makes you scare out of shit.

I'm having a weird time with my current project.There are many companies involved and we are several teams coordinating with each other. My team was initially very large, for various reasons we were divided into smaller groups and I must say that the transition has been catastrophic.

We are doing SCRUM…sort of. The customer assigns the tasks to be completed at the end of the sprint, the story points are given without full understanding of the implementation and the deadlines are tights. I always find myself rushing to the release day with code that isn't production-ready but since the customer requests it and there's no objection among my superiors (please note, i tell them the deadline is tight) I gotta rush to deliver.

The customer doesn't know what he wants, but if he does know the deadline is unreasonable, or if he has just an idea of what he wants he still demands it... somehow without specifying what kind of implementations is expecting.

The current senior project developer takes everything (any task) as an emergency, it's never possible to defer to the next sprint, it's quite demeaning.

And I'm here wondering if maybe I've missed something, if the project simply lacks method and coordination, if I have more responsibility than I think, if my project leadership is too absent but I know one thing, at the moment I'm in anxiety about the current sprint due date because there is a task that will take longer than expected.

Any advice?

yeah sure, having all 5 devs only ever working directly on the production branch and not being allowed to save anything post-release unless it's an urgent bugfix sounds like a great idea

Recently joined new Android app (product) based project & got source code of existing prod app version.

Product source code must be easy to understand so that it could be supported for long term. In contrast to that, existing source structure is much difficult to understand.

Package structure is flat only 3 packages ui, service, utils. No module based grouped classes.

No memory release is done. So on each screen launch new memory leaks keep going on & on.

Too much duplication of code. Some lazy developer in the past had not even made wrappers to avoid direct usage of core classes like Shared Preference etc. So at each place same 4-5 lines were written.

Too much if-else ladders (4-5 blocks) & unnecessary repetitions of outer if condition in inner if condition. It looks like the owner of this nested if block implementation has trust issues, like that person thought computer 'forgets' about outer if when inside inner if.

Too much misuse of broadcast receiver to track activities' state in the era of activity, apપ life cycle related Android library.

Sometimes I think why people waste soooo... much efforts in the wrong direction & why can't just use library?!!

These things are found without even deep diving into the code, I don't know how much horrific things may come out of the closet.

This same app is being used by many companies in many different fields like banking, finance, insurance, govt. agencies etc.

Sometimes I surprise how this source passed review & reached the production.

That moment just before release to production where you've tested it a thousand times and suddenly encounter a couple of bugs.

shall we begin the terraform stories?

terraforming things is nice. the vcloud director provider of Terraform is also nice..ish.

for fucks sake, why do folks at VMware release a provider for use in fucking production, that only does support barely a third of all features, including the distributed logical router with all its funkyfuck features? nsx-t is nice, but did you folks remember all of those customers, who do run the old nsx-v?
you've decided that nsx-v shall be put to sleep. okay. fine. nice.
but don't you think, that the version 3.3.ass should support all major resources of your product, including old nsx-v features like the fucking DLR?!

sorry, but a product, that only supports ⅓ of all features, that can be managed in UI, only deserves a RC label at best. calling this a 3.3.ass is bold. you can't even setup a dhcp pool for a defined network. dafuq people..?! (╯°□°）╯︵ ┻━┻

It's almost all the time but specially when there is a stupid bug i find out in production after all the efford i put on testing before release, or even worst, when the bug is not stupid, is random and just hard to tell why the fuck everything is fucked up