Me : So how's the deadlines here?
Coworker : There are none.
Me : ?
Coworker : if they are unrealistic, we push non-working code. Prod comes up with bugs, and we get a new sprint to resolve those bugs.
Me: ╰[ ⁰﹏⁰ ]╯

Web developer successfully found bugs in prod.

This is going to be a long rant, coz this is the only way to vent out my frustration against our tech head.

Yesterday, while our fucking twat tech head was playing around in company aws account, he terminated the production server. By mistake, apparently. Coz he doesn't know shit about server management. But that egoist ass won't admit and fucked the production server.

And then ran away. We developers sprang into action. Updated dns to point to staging server, setup virtual hosts, env files, point to prod database, force flush dns cache. All systems were up and running in 30 mins. And since it was staging server, it had lot of untested features and codes, and we spent rest of the day fixing the bugs.

And that tech head, who ran away hiding his tail between his legs, after he fucked the server, came back after systems were up. And started cracking jokes, that "so many features got released in 1 day" . "We cut server cost by shutting down 1 server."

We were struggling and working in full throttle to make the services running again. And that fuckity fucker was cracking jokes.

And I don't even know what excuse he gave to ceo for the downtime. I am pretty sure he would have made up some crappy excuse to hide his fucking mistake. That ass never admits his mistake. I am thinking to go to ceo today and tell the real story and get that faggot head fired or at least a strict warning.

Nice job Amazon

CEO: if we would not give new features, clients would be bored and would not pay for tool.
me: but don't you think we should fix buggy old code, that would reduce effort and time that we daily invest in prod bugs?

CEO: I'm not saying we should not fix them but we should maintain the balance which is 80-20. 80% of our work would include adding new features.
😑
Next day in morning receives email:
There is a production issue, fix it asap.

😬

WHY THE FUCK ARE YOU ASSIGNING PROD BUGS WHEN I'M ON A FUCKING VACATION ?!?

Oh wait I wrote that code...

Welp

These fucking imbecile devs left me a bunch of bugs in a 150k LOC legacy hell...
Too bad it took some 8 months for them to finally surface in prod. God damned...
Technically, this whole codebase is one giant abyss filled to the brim with bugs.

FUCKING FUCK ANGULAR!!!!
LIKE FUCK IT IN THE ARSE AND BURN THE MOTHERFUCKER WHILE LAUNCHING A MISSILE ON IT TO BE SURE!

(ﾉ≧∇≦)ﾉ ﾐ ┸━┸

So I am making something on angular and I got everything running in ng serve(development environment) , after handling all issues and showing it to my boss man he approves and asked to put it up on prod for a demo , doesn’t sound like an issue , I make the prod build on cli and BAM! 16 errors ? No issues right?, I’ll just google the issue. Googles.... there aren’t no clear solutions to it as the angular version keeps changing and nobody knows what broke it, I mean people have the issue,but like 100 reasons that can cause it,
HOLY LORD RELEASE A NEWER VERSION AFTER MENDING THE OLD ONE
But nooooooo!

Angular Dev:We fucked this one, lol what should we do boss man?
Angular boss man: lol just leave it, we need to build the new version with newer bugs,

P.S. I like angular, but it’s like a underdeveloped framework, too many issues and too many changes

You know how my 'about' here says I create features from bugs?!? Well if you didn't before, now you do.. :P

Anyhow, today the most bizzare thing happened.. Customer played 'reverse uno card' on me..

Meaning?!

They reported a feature missing on uat env, when in fact I fixed a bug they have on prod..

Not sure how I should feel about this, but it sure made my day! (: I just hope they will not open a bug report for this missing bug..

When bugs are seen by the client and boss therefore asks me "did you know about this bug?", what I'd really like to answer is:
"well shit, no! I would have solved that or at least told you about it, don't you think? what kind of fucking question is that?"

But then I just answer "no, lemme check"

We had 1 Android app to be developed for charity org for data collection for ground water level increase competition among villages.

Initial scope was very small & feasible. Around 10 forms with 3-4 fields in each to be developed in 2 months (1 for dev, 1 for testing). There was a prod version which had similar forms with no validations etc.

We had received prod source, which was total junk. No KT was given.

In existing source, spelling mistakes were there in the era of spell/grammar checking tools.

There were rural names of classes, variables in regional language in English letters & that regional language is somewhat known to some developers but even they don't know those rural names' meanings. This costed us at great length in visualizing data flow between entities. Even Google translate wasn't reliable for this language due to low Internet penetration in that language region.

OOP wasn't followed, so at 10 places exact same code exists. If error or bug needed to be fixed it had to be fixed at all those 10 places.

No foreign key relationships was there in database while actually there were logical relations among different entites.

No created, updated timestamps in records at app side to have audit trail.

Small part of that existing source was quite good with Fragments, MVP etc. while other part was ancient Activities with business logic.

We have to support Android 4.0 to 9.0 of many screen sizes & resolutions without any target devices issued to us by the client.

Then Corona lockdown happened & during that suddenly client side professionals became over efficient.

Client started adding requirements like very complex validation which has inter-entity dependencies. Then they started filing bugs from prod version on us.

Let's come to the developers' expertise,
2 developers with 8+ years of experience & they're not knowing how to resolve conflicts in git merge which were created by them only due to not following git best practice for coding like only appending new implementation in existing classes for easy auto merge etc.

They are thinking like handling click events is called development.

They don't want to think about OOP, well structured code. They don't want to re-use code mostly & when they copy paste, they think it's called re-use.

They wanted to follow old school Java development in memory scarce Android app life cycle in end user phone. They don't understand memory leaks, even though it's pin pointed by memory leak detection tools (Leak canary etc.).

Now 3.5 months are over, that competition was called off for this year due to Corona & development is still ongoing.

We are nowhere close to completion even for initial internal QA round.

On top of this, nothing is billable so it's like financial suicide.

Remember whatever said here is only 10% of what is faced.

- An Engineering lead in a half billion dollar company.

TL;DR

Got Salesforce customised by an external company, took over the code and wanted to cry.

So my company decided to get a new crm for sales with contract generation and a whole lot of fluff. Specs included an easy to implement API to connect to our in-house software and an easy to adjust contract Wizard.

After month of checking various companies the CEO finally settled on...
Drumroll
The cheapest German based company.
Turns out the only part that is based in Germany is the sales department, development is based in Poland which made for interesting times during implementation because of the language barrier that comes with non native english speakers.

We as in our development department for told that we wouldn't need to worry about the solution because it would all be developed and maintained by the new company. As we are fairly small that was more then welcome.

Fast forward to integration day. No docs for the API available, contract Wizard is hardcoded, bunch of errors and inconsistencies. Get tickets for them.
Ok we can deal with this. Just tell the ticket writer he needs to address the problems with the external. Yep, doesn't work. External fixes bugs and introduces then elsewhere.
Fml. Ok I'll take a look into the code. Ugh, Java, I hate that shit but at least I don't need to worry about all the fluff, just the code, so it's ok.
No repository present, code is developed in the Dev environment and pushed to prod. Ugly but works. Code comes with a lot of functions but only one real class called "CommonUtilities" ... even the web API is defined in there. Meanwhile my colleague throws out the need for the API because we will just directly tie to SOQL. I'll let's check what's going on here, nice you reused Lead/Account/Contact for the branch offices of companies as well...
Is not like Salesforce has a bunch of logic tied to these objects...
Nice the required implementation of an automatic holiday import is not even there, just a custom object populated with this year's data...

Tell CEO how badly all of this was handled. Nice note this dumpster fire is our new in-house project because CEO cancelled the contract with the external.

Ok we can deal with this, let's set up a repo, define the CI/CD and get the extensions for vscode. Nice now this all makes sense. Fix all bugs and reimplement the contract Wizard using custom objects that sales can change so the contract is actually easily customizable by a non dev. Implement branches as their own object to avoid opportunities, triggers and the likes to be executed for them.
Took a whole day. Why did these 2000 lines of code that was shipped to us take 3 month to implement?

Lesson learned: never trust an external to just do a good job.
New rule implemented by company to always have a Dev check in regular intervals on projects handled by externals for standards and overall logic

10k down the drain for what amounted to 2 weeks tops (one dev) if we had handled the implementation of Salesforce and all the requirements in-house

Rolled out a new application I built almost entirely by myself 2 days ago... But my dev group is understaffed and has a project manager who is literally the most clueless person I have ever met, so as a result, we don't have a functional/useful dev/test/prod framework and no standards for how to deploy apps. So my past 2 days were comprised of fixing bugs in the live system that could probably have been caught if I had the time and resources to get everything thoroughly tested. It's stable now, but damn our management for being generally idiots. Our motto appears to be "Fuck it, we'll do it live"

ideal sprint fallacy.
total days 10 , total hours(excluding breaks ) 8 hrs per day= 80 hrs per dev

code freeze day = day 8, testing+ fixing days : 8,9,10. release day : day 10
so ideal dev time = 7days/56 hr
meetings= - 1hr per day => 49 hrs per dev

- 1 day for planning i.e d1 . so dev time left . 6 days 42 hrs.
-----------
all good planning. now here comes the messups

1. last release took some time. so planning could not happen on d1. all devs are waiting. . devtime = 5 days 35 hrs.

2. during planning:
mgr: hey devx what's the status on task 1?
d: i integrated mock apis. if server has made the apis, i will test them .
mgr : server says the apis are done. whats your guestimate for the task completion?
d : max 1-2 hrs?
m : cool. i assign you 4 hrs for this. now what about task 2?
d : task told to me is done and working . however sub mgr mentioned that a new screen will be added. so that will take time
m : no we probably won't be taking the screen. what's your giestimate?
d : a few more testing on existing features. maybe 1-2 hrs ?
m: cool
another 4 hrs for u. what about task 3?
d : <same story>
m : cool. another 4 hrs for u. so a total of 12 hrs out of 35 hrs? you must be relaxed this sprint.
d : yeah i guess.
m cool.
-------
timelines.
d1: wasted i previous sprint
d2 : sprint planning
d3 : 3+ hrs of meetings, apis for task 1 weren't available sub manager randomly decided that yes we can add another screen but didn't discussed. updates on all 3 tasks : no change in status
d4 : same story. dev apis starts failing so testing comes to halt.
d5 : apis for task1 available . task 3 got additional improvement points from mgr out of random. some prod issue happens which takes 4+ hrs. update on tasks : some more work done on task 3, task 1 and 2 remains same.
d6 : task1 apis are different from mocks. additionally 2 apis start breaking and its come to know thatgrs did not explain the task properly. finally after another 3+ hrs of discussion , we come to some conclusions and resolutions
d7 : prod issue again comes. 4+ hrs goes into it . task 2 and 3 are discussed for new screen additiona that can easily take 2+ days to be created . we agree tot ake 1 and drop 2nd task's changes i finish task 2 new screens in 6 hrs , hoping that finally everything will be fine.

d8 : prod issue again comes, and changes are requested in task 2 and 3

day 9 build finally goes to tester
day 10 first few bugs come with approval for some tasks
day 11(day 1 of new sprint) final build with fixes is shared. new bugs (unrelated to tasks. basically new features disguised as bugs) are raised . we reject and release the build.

day 2 sprint planning
mgr : hey dev x, u had only 12 hrs of work in your plate. why did the build got delayed?

🥲🫡

A couple of years ago I was working on a fairly large system with a complex (by necessity) access control architecture.

As is usually the case with those projects, it's awkward for developers to repro bugs that have to do with a user's accesses in production when we are not allowed to replicate production data in test, let alone locally.

We had a bug where I ended up making myself a new row in the production database for a thing I could have access to without affecting real data to repro it safely. I identified the bug so I could repro it in dev/test and removed the row and ensured everything worked normally, whew scary.

Have you ever walked into the office one day, and everyone is hunched over in a semicircle around one person's workstation, before one turns around to look at you and says - after a pause - "... ltlian?.."

Turns out I had basically "poisoned the well" with my dummy entity in a way where production now threw 500 for everyone BUT me who had transitive access to this post-non-entity. Due to the scope of the system, it had taken about a day for this to gradually propagate in terms of caching and eventual consistencies; new entities coming in was expected, but not that they disappear.

Luckily I had a decent track record for this to be a one-off. I sometimes think about how I would explain testing in prod and making it faceplant before going home for the day, other than "I assumed it would be fine". I would fire me.

IF IT WORKS DON'T TOUCH
IT…PLEASE DON'T!! 👾👾

I just love production bugs, especially when it's Facebook

Lessions I learned so far from my first big node/npm project with tons of users:
1) If you didn't build something for a while, expect 3 hours of resolving version conflicts for every two weeks since the last build.
2) Even if the tests pass, run the containers on your own machine and make sure that the app doesn't randomly crash before deploying
3) Even if the app seemed to work on your own machine, run the tests again in an environment mimicking prod at most 15 minutes before replacing the running containers.
4) Even if all else indicates that the app will work, only ever deploy if you expect to be available within the 4 hours following a deployment.
5) Don't use shrinkwrap for anything other than locking every version down completely. A partial shrinkwrap will produce bugs that are dependent on the exact hour you built the app _and_ the shrinkwrap file, and therefore no one will ever have seen them other than you.
6) Avoid gyp, and generally try not to interface too much with anything that doesn't run on node. If parts of your solution use very different toolchains, your problems will be approximately proportional to the amount of code. And you'd be surprised just how much code you're running. (otherwise it's more logarithmic because the more code the less likely a new assumption is unique)
7) Do not update webpack or its plugins or anything they might call unless you absolutely need to
8) Containers are cool but the alpine ones are pretty much useless if you have even just one gyp module.
9) There's always another cache. To save yourself a lot of pain, include the build time in every file or its name that the browser can download, and compare these to a fresh build while debugging to assert that the bug is still present in the code you're reading
+1) Although it may look like it, SQLite is far from a simple solution because the code and the bindings aren't maintained. In fact, it'll probably be more time consuming than using a proper database.

I just submitted my professional goals this year.

As I was writing them a lot of them actually sound like things my manager should be doing to increase code quality n agility n reduce the # bugs that end up in prod...

Not sure what's better on Friday:

Fix the bugs and push it to prod so manager will be happy.

Or sit there like a dumb fuck and wait for Monday before touching the code

Just before deploying this to prod found this bug... discover this bug with bunch of print statements

i have seen bugs getting into prod but not the local language getting into prod. what language is this?

Not exactly related to the topic but the exact thing is chilling the fuck out .
I always was anxious and was completely paranoid about minor bugs in my application during prod deployments(that is when I didn't know about testing utils and so on) , till the point that I couldn't fix a minor bug in the CSS and I puked 5 times over.
It was rough times but then I got over it and it really helped me alot.
I know bugs are like really not the kind of things you'd want to see in any application but it will arise in every application :3

Me: 'here we go, code working completely as intended, tested and without bugs.'
Senior after reviewing code: 'apart from the formatting errors, I'd also do this piece of coding in a different manners'
*Comments exchange in pull request*
Me: 'well this seems more like a change the whole logic request rather than a small improvement, I'll keep it like this and resolve it like suggested on a future opportunity'

Still in prod.