One week, and it turned out to be worse than that.

I was put on a project for a COVID-19 program in America (The CARES Act). The financial team came to us on Monday morning and said they need to give away a couple thousand dollars.

No big deal. All they wanted was a single form that people could submit with some critical info. Didn't need a login/ registration flow or anything. You could have basically used Google Forms for this project.

The project landed in my lap just before lunch on Monday morning. I was a junior in a team with a senior and another junior on standby. It was going to go live the next Monday.

The scope of the project made it seem like the one week deadline wasn't too awful. We just had to send some high priority emails to get some prod servers and app keys and we were fine.

Now is the time where I pause the rant to express to you just how fine we were decidedly **not**: we were not fine.

Tuesday rolls around and what a bad Tuesday it was. It was the first of many requirement changes. There was going to need to be a review process. Instead of the team just reading submissions from the site, they needed accept and reject buttons. They needed a way to deny people for specific reasons. Meaning the employee dashboard just got a little more complicated.

Wednesday came around and yeah, we need a registration and login flow. Yikes.

Thursday came and the couple-thousand dollars turned into a tens of millions. The amount of users we expected just blew up.

Friday, and they needed a way for users to edit their submissions and re-submit if they were rejected. And we needed to send out emails for the status of their applications.

Every day, a new meeting. Every meeting, new requirements that were devastating given our timeframe.

We put in overtime. Came in on the weekend. And by Monday, we had a form that users could submit and a registration/ login flow. No reviewer dashboard. We figured we could take in user input on time and then finish the dashboard later.

Well, financial team has some qualms. They wanted a more complicated review process. They wanted roles; managers assign to assistants. Assistants review assigned items.

The deadline that we worked so hard on whizzed by without so much as a thought, much less the funeral it deserved.

Then, they wanted multiple people to review an application before it was final. Then, they needed different landing pages for a few more departments to be able to review different steps of the applications.

Ended up going live on Friday, close to a month after that faithful Monday which disrupted everything else I was working on, effective immediately.

I don't know why, but we always go live on a Friday for some reason. It must be some sort of conspiracy to force overtime out of our managers. I'm baffled.

But I worked support after the launch.

And there's a funny story about support too: we were asked to create a "submit an issue" form. Me and the other junior worked on it on a wednesday three weeks into the project. Finished it. And the next day it was scrapped and moved to another service we already had running. Poor management like that plagued the project and worked in tandem with the dynamic and ridiculous requirements to make this project hell.

Back to support.

Phone calls give me bad anxiety. But Friday, just before lunch, I was put on the support team. Sure, we have a department that makes calls and deal with users. But they can't be trained on this program: it didn't exist just a month ago, and three days ago it worked differently (the slippery requirements never stopped).

So all of Friday and then all of Saturday and all of Monday (...) I had extended panic attacks calling hundreds of people. And the team that was calling people was only two people. We had over 400 tickets in the first two days.

And fuck me, stupid me, for doing a good job. Because I was put on the call team for **another** COVID project afterwards. I knew nothing about this project. I have hated my job recently. But I'm a junior. What am I gonna say, no?

Welcome back to practiseSafeHex's new life as a manager.

Episode 2: Why automate when you can spend all day doing it by hand

This is a particularly special episode for me, as these problems are taking up so much of my time with non-sensical bullshit, that i'm delayed with everything else. Some badly require tooling or new products. Some are just unnecessary processes or annoyances that should not need to be handled by another human. So lets jump right in, in no particular order:

- Jira ... nuff said? not quite because somehow some blue moon, planets aligning, act of god style set of circumstances lined up to allow this team to somehow make Jira worse. On one hand we have a gigantic Jira project containing 7 separate sub teams, a million different labels / epics and 4.2 million possible assignees, all making sure the loading page takes as long as possible to open. But the new country we've added support for in the app gets a separate project. So we have product, backend, mobile, design, management etc on one, and mobile-country2 on another. This delightfully means a lot of duplication and copy pasting from one to the other, for literally no reason what so ever.

- Everything on Jira is found through a label. Every time something happens, a new one is created. So I need to check for "iOS", "Android", "iOS-country2", "Android-country2", "mobile-<feature>", "mobile-<feature>-issues", "mobile-<feature>-prod-issues", "mobile-<feature>-existing-issues" and "<project>-July31" ... why July31? Because some fucking moron decided to do a round of testing, and tag all the issues with the current date (despite the fact Jira does that anyway), which somehow still gets used from time to time because nobody pays attention to what they are doing. This means creating and modifying filters on a daily basis ... after spending time trying to figure out what its not in the first one.

- One of my favourite morning rituals I like to call "Jira dumpster diving". This involves me removing all the filters and reading all the tickets. Why would I do such a thing? oh remember the 9000 labels I mentioned earlier? right well its very likely that they actually won't use any of them ... or the wrong ones ... or assign to the wrong person, so I have to go find them and fix them. If I don't, i'll get yelled at, because clearly it's my fault.

- Moving on from Jira. As some of you might have seen in your companies, if you use things like TestFlight, HockeyApp, AppCenter, BuddyBuild etc. that when you release a new app version for testing, each version comes with an automated change-log, listing ticket numbers addressed ...... yeah we don't do that. No we use this shitty service, which is effectively an FTP server and a webpage, that only allows you to host the new versions. Sending out those emails is all manual ... distribution groups?? ... whats that?

- Moving back to Jira. Can't even automate the changelog with a script, because I can't even make sense of the tickets, in order to translate that to a script.

- Moving on from Jira. Me and one of the remote testers play this great game I like to call "tag team ticketing". It's so much fun. Right heres how to play, you'll need a QA and a PM.

*QA creates a ticket, and puts nothing of any use inside it, and assigns to the PM.
*PM fires it back asking for clarification.
*QA adds in what he feels is clarification (hes wrong) and assigns it back to the PM.
*PM sends detailed instructions, with examples as to what is needed and assigns it back.
*QA adds 1 of the 3 things required and assigns it back.
*PM assigns it back saying the one thing added is from the wrong day, and reminds him about the other 2 items.
*QA adds some random piece of unrelated info to the ticket instead, forgetting about the 3 things and assigns it back.

and you just continue doing this for the whole dev / release cycle hahaha. Oh you guys have no idea how much fun it is, seriously give it a go, you'll thank me later ... or kill yourselves, each to their own.

- Moving back to Jira. I decided to take an action of creating a new project for my team (the mobile team) and set it up the way we want and just ignore everything going on around us. Use proper automation, and a kanban board. Maybe only give product a slack bot interface that won't allow them to create a ticket without what we need etc. Spent 25 minutes looking for the "create new project" button before finding the link which says I need to open a ticket with support and wait ... 5 ... fucking ... long ... painful ... unnecessary ... business days.

... Heres hoping my head continues to not have a bullet hole in it by then.

Id love to talk more, but those filters ain't gonna fix themselves. So we'll have to leave it here for today. Tune in again for another episode soon.

And remember to always practiseSafeHex

My team handles infrastructure deployment and automation in the cloud for our company, so we don't exactly develop applications ourselves, but we're responsible for building deployment pipelines, provisioning cloud resources, automating their deployments, etc.

I've ranted about this before, but it fits the weekly rant so I'll do it again.

Someone deployed an autoscaling application into our production AWS account, but they set the maximum instance count to 300. The account limit was less than that. So, of course, their application gets stuck and starts scaling out infinitely. Two hundred new servers spun up in an hour before hitting the limit and then throwing errors all over the place. They send me a ticket and I login to AWS to investigate. Not only have they broken their own application, but they've also made it impossible to deploy anything else into prod. Every other autoscaling group is now unable to scale out at all. We had to submit an emergency limit increase request to AWS, spent thousands of dollars on those stupidly-large instances, and yelled at the dev team responsible. Two weeks later, THEY INCREASED THE MAX COUNT TO 500 AND IT HAPPENED AGAIN!

And the whole thing happened because a database filled up the hard drive, so it would spin up a new server, whose hard drive would be full already and thus spin up a new server, and so on into infinity.

Thats probably the only WTF moment that resulted in me actually saying "WTF?!" out loud to the person responsible, but I've had others. One dev team had their code logging to a location they couldn't access, so we got daily requests for two weeks to download and email log files to them. Another dev team refused to believe their server was crashing due to their bad code even after we showed them the logs that demonstrated their application had a massive memory leak. Another team arbitrarily decided that they were going to deploy their code at 4 AM on a Saturday and they wanted a member of my team to be available in case something went wrong. We aren't 24/7 support. We aren't even weekend support. Or any support, technically. Another team told us we had one day to do three weeks' worth of work to deploy their application because they had set a hard deadline and then didn't tell us about it until the day before. We gave them a flat "No" for that request.

I could probably keep going, but you get the gist of it.

Devs: We need access to PROD DB in order to provide support you're asking us for.

Mgmt: No, we cannot trust you with PROD DB accesses. That DB contains live data and is too sensitive for you to fuck things up

Mgmt: We'll only grant PROD DB access to DBAs and app support guys

Mgmt: <hire newbies to app support>

App_supp: `update USER set invoice_directory = 54376; commit;`

----------------
I have nothing left to say....

Week 278: Most rage-inducing work experience — I’ve got a list saved! At least from the current circle of hell. I might post a few more under this tag later…

TicketA: Do this in locations a-e.
TicketB: Do this in locations e-h.
TicketC: Do this in locations i-k.
Root: There’s actually a-x, but okay. They’re all done.
Product: You didn’t address location e in ticket B! We can’t trust you to do your tickets right. Did you even test this?
Root: Did you check TicketA? It’s in TicketA.
Product guy: It was called out in TicketB! How did you miss it?!
Product guy: (Refuses to respond or speak to me, quite literally ever again.)
Product guy to everyone in private: Don’t trust Root. Don’t give her any tickets.
Product manager to boss: Root doesn’t complete her tickets! We can’t trust her. Don’t give her our tickets.
Product manager to TC: We can’t trust Root. Don’t give her our tickets.
TC: Nobody can trust you! Not even the execs! You need to rebuild your reputation.

Root: Asks coworker a simple question.
Root: Asks again.
Root: nudges them.
Root: Asks again.
Coworker: I’ll respond before tomorrow. (And doesn’t.)
Root: Asks again.
Root: Fine. I’ll figure it out in my own.
TC: Stop making it sound like you don’t have any support from the team!

Root: Asks four people about <feature> they all built.
Everyone: idk
Root: Okay, I’ll figure it out on my own.
TC: Stop making it sound like you don’t have any support from the team!

Root: Mentions multiple meetings to discuss ticket with <Person>.
TC: You called <Person> stupid and useless in front of the whole team! Go apologize!

Root: Tells TC something. Asks a simple question.
Root: Tells TC the same thing. Asks again.
TC: (No response for days.)
TC: Tells me the exact same thing publicly like it’s a revelation and I’m stupid for not knowing.
TC: You don’t communicate well!

Root: Asks who the end user of my ticket is.
Root: Asks Boss.
Root: Asks TC.
Root: Fine, I’ll build it for both.
Root: Asks again in PR.
TC: Derides; doesn’t answer.
Root: Asks again, clearly, with explanation.
TC: Copypastes the derision, still doesn’t answer.
Root: Asks boss.
Boss: Doesn’t answer.
Boss: You need to work on your communication skills.

Root: Mentions asking question about blocker to <Person> and not hearing back. Mentions following up later.
<Person>: Gets offended. Refuses to respond for weeks thereafter.

Root: Hey boss, there’s a ticket for a minor prod issue. Is that higher priority than my current ticket?
Root: Hey, should I switch tickets?
Root: Hey?
Root: … Okay, I’ll just keep on my current one.
Boss: You need to work on your priorities.

Everyone: (Endless circlejerking and drama and tattling)

Just now saw an email from Support after I asked them to run some cleanup commands on 4 servers.

Support: u need to give us an automated job for this.

WELL FUCK YOU, Y DON'T YOU JUST DO IT URSELF.

I JUST SPENT 2HRS INVESTIGATING A PROD ISSUE WHICH GOT ESCALATED CUZ UR A BUNCH OF USELESS INCOMPETENT MONKEYS... THE LEAST YOU CAN DO IS HELP... AND WHAT I HOPE WOULD BE YOU BUILD UR OWN TOOL... BEING SUPPORT UR SUPPOSED TO HELP ME REDUCE MY WORK NOT INCREASE IT

Me passing time on the weekend

Random call from unknown number

Turns out it's the manager

M: hey , how is your weekend going ...

Me: nothing much ... Whatsup ?

M : yeah well , we wanted to push some minor adhoc fixes as some clients wanted it urgently

The Devops folks need developer support . Can you pitch in and monitor

Me : I'm not aware of what changes are going , i don't think i can provide support

M : don't worry it's minor changes , it's already tested in pre prod , you just need to be on call for 30 mins

Me : ugh okay .. guess 1 hr won't hurt

M: thanks 👍🏽

Me: *logs in

*Notices the last merged PR
+ 400 lines , implemented by junior dev and merged by manager

*Wait , how is this a *minor* release...

*Release got triggered already and the CI CD pipeline is in progress

*5 mins later

*Pipeline fails , devops sends email - test coverage below 50%

Manager immediately pitches in ...

M: hey , i see test coverage is down , can you increase it ?

Me: and how do u suppose I do that ?

M : well it's simple just write UTC for the missing lines ... Will it take time ?

Me : * ah shit here we go again

Yeah it will take time , there are around 400 lines , I am not aware of this component all together

Can you ask junior dev to pitch in and write the UTC for this

*Actually junior dev is out on a vacation with his girlfriend

M : well he's out for the weekend , but
as a senior dev , i expect you to have holistic understanding of the codebase and not give excuses ,
this is a priority fix which client are demanding we need this released ASAP

Me : * wait wat ?

---

I ended up being online for next 3 hours figuring out the code change and bumping up the UTC 🤦🏾

Every year my team runs an award ceremony during which people win “awards” for mistakes throughout the year. This years was quite good.

The integration partner award- one of our sysAdmins was talking with a partner from another company over Skype and was having some issues with azure. He intended to send me a small rant but instead sent “fucking azure can go fuck itself, won’t let me update to managed disks from a vhd built on unmanaged” to our jv partner.

Sysadmin wannabe award (mine)- ran “Sudo chmod -R 700 /“ on one of our dev systems then had to spend the next day trying to fix it 😓

The ain’t no sanity clause award - someone ran a massive update query on a prod database without a where clause

The dba wannabe award - one of our support guys was clearing out a prod dB server to make some disk space and accidentally deleted one of the databases devices bringing it down.

The open source community award - one of the devs had been messing about with an apache proxy on a prod web server and it ended up as part of a botnet

There were others but I can’t remember them all

I didn't scream.. just told him to jump off of terrace..
What ticked me?! He was a support guy..slowest mofo ever..
I was in the middle of fixing major fuckup on prod, when our VPN to client disconnected. I rushed over to support to ask if it is 'just' an expired session (which he was in charge of renewing but constantly fucked up) or if there is some other problem, so I know how to proceed..do I need to contact our sysadmins, client's support guys etc..
He
started
to
slooooooooowly
explain
I
am
not
the
only
one
with
VPN
problems
...
Was that what I asked you?! // he had an annoying habit of slooooowly talking and explaining unrelated things & personal stuff that bothered him & most of the times he chose the most time sensitive period to drone off..
So I cut him of saying, that others were probably not 'tinkering' with production and that I need this back ASAP, so if he could tell me when the session will be renewed or if there is something else problematic..
He said he will check..I didn't move.. he looked at me insurprise, you want me to check *NOW*?! Yeah, it's urgent.. He proceeded very very veeeery slooooowly, taking the support phone../* he was even eating sandwich during that, so only one hand free, typing one letter at a min */
I was finaly notified that the session expired and that he will fix it soon (meaning in 15-20mins o.O which should not take him more than 5).. and was like 'can I do sth else for you'?! Yeah, do the backflip.. you know the rest..

Gather around folks, I'll paint you a nice picture based on a true story, back from my sysadmin days. Listen up.
It's about HP and their Solaris 5.4/6 support.

- Yet another Prod Solaris dinosaur crashed
- Connected to console, found a dead system disk; for some reason it was not booting on the remaining redundant disk...
- Logged an HP vendor case. Sev1. SLA for response is 30 minutes, SLA for a fix is <24 hours
- It took them 2 days to respond to our Prod server outage due to failed system disks (responses "we are looking into this" do not count)
- it took another day for them to find an engineer who could attend the server in the DC
- The field engineer came to the DC 4 hours before the agreed time, so he had to wait (DC was 4-5 hours of driving away from HP centre)
- Turns out, he came to the wrong datacentre and was not let in even when the time came

- We had to reschedule for two days later. Prod is still down
- The engg came to the DC on time. He confirmed he had the FRU on him. Looks promising
- He entered the Hall
- He replaced the disk on the Solaris server
- It was the wrong disk he replaced. So now the server is beyond rebuild. It has to be built anew... but only after he comes back and replaces the actually faulty disk.
- He replaced that disk on the wrong Solaris server

Highlights from my week:

Prod access: Needed it for my last four tickets; just got it approved this week. No longer need it (urgently, anyway). During setup, sysops didn’t sync accounts, and didn’t know how. Left me to figure out the urls on my own. MFA not working.

Work phone: Discovered its MFA is tied to another coworker’s prod credentials. Security just made it work for both instead of fixing it.

My merchant communication ticket: I discovered sysops typo’d my cronjob so my feature hasn’t run since its release, and therefore never alerted merchants. They didn’t want to fix it outside of a standard release. Some yelling convinced them to do it anyway.

AWS ticket: wow I seriously don’t give a crap. Most boring ticket I have ever worked on. Also, the AWS guy said the project might not even be possible, so. Weee, great use of my time.

“Tiny, easy-peasy ticket”: Sounds easy (change a link based on record type). Impossible to test locally, or even view; requires environments I can’t access or deploy to. Specs don’t cover the record type, nor support creating them. Found and patched it anyway.

Completed work: Four of my tickets (two high-priority) have been sitting in code review for over a month now.

Prod release: Release team #2 didn’t release and didn’t bother telling anyone; Release team #1 tried releasing tickets that relied upon it. Good times were had.

QA: Begs for service status page; VP of engineering scoffs at it and says its practically impossible to build. I volunteered. QA cheered; VP ignored me.

Retro: Oops! Scrum master didn’t show up.

Coworker demo: dogshit code that works 1 out of 15 times; didn’t consider UX or user preferences. Today is code-freeze too, so it’s getting released like this. (Feature is using an AI service to rearrange menu options by usage and time of day…)

Micromanager response: “The UX doesn’t matter; our consumers want AI-driven models, and we can say we have delivered on that. It works, and that’s what matters. Good job on delivering!”

Yep.
So, how’s your week going?

I am a senior a DevOps engineer who took the production stack down for ~10 minutes today because of a bad code commit. I could use some encouragement! It’s a fierce world of competitive engineers and I wonder why my company doesn’t replace me. The mistake was missed by two other peer reviews... but that doesn’t stop me from feeling this way.

Have you crashed prod? Did your team support you or tear you down?

Leading a team of 10 people, 5 environments (3 non prod 2 prod) to support, 25 formal deployments per week, and all I have is one fucking repository in fucking svn.

When customer support calls with a complex issue and wants a guaranteed fix date for a defect you haven't even investigated because "they need to get an answer back to the customer TODAY."

Ever had a day that felt like you're shoveling snow from the driveway? In a blizzard? With thunderstorms & falling unicorns? Like you shovel away one m² & turn around and no footprints visible anymore? And snow built up to your neck?

Today my work day was like that.. xcept shit..shit instead of pretty & puffy snow!!

Working on things a & b, trying to not mess either one up, then comes shit x, coworker was updating production.. ofc something went wrong.. again not testing after the update..then me 'to da rescue'.. :/ hardly patch things up, so it works..in a way.. feature c still missing due to needed workarounds.. going back to a and b.. got disrupted by the same coworker who is nver listening, but always asking too much..

And when I think I finally have the b thing figured out a f-ing blocker from one of our biggest clients.. The whole system is unresponsive.. Needles to say, same guy in support for two companies (their end), so they filed the jira blocker with the wrong customer that doesn't have a SLA so no urgent emails..and then the phone calls.. and then the hell broke loose.. checking what is happening.. After frantic calls from our dba to anyone who even knows that our customer exists if they were doing sth on the db.. noup, not a single one was fucking with the prod db.. The hell! Materialised view created 10 mins ago that blocked everything..set to recreate every 10 minutes..with a query that I am guessing couldn't even select all that data in under 15.. dafaaaq?! Then we kill it..and again it is there.. We found out that customers dbas were testing something on live environment, oblivious that they mamaged to block the entire db..
FML, I'm going pokemon hunting.. :/ codename for ingress n beer..

Me: Why are we spending time building reports for Support? AFAIK they never read or use them.

Boss: Seems they expect you to do it.

Me: Then what exactly are they supposed to be doing? All the issues seem to just escalate back to us.... We should just make sure issues never get into PROD

Boss: I agree, they're always firefighting... we gave them more funding so eventually they should catch up **Me: I highly doubt it, you should just stop hiring monkeys** esp. if we prevent PROD issues.

Me: Yea... we should prevent production issues because someone always has to pay off the (technical) debt and interest rates in PROD are very high

TL;DR Dear boss, firstly, you always get someone to review anything important done by a fucking intern.
Secondly, you do not give access to your fucking client's production server to an intern.
Thirdly, you don't ask your fucking intern to test the intern's work that has not been reviewed by anyone directly on your client's fucking production server.

Last week, the boss and one of the lead devs (the only guy with some serious knowledge about systems and networking) decided to give me (an intern who barely has any work experience) the task of fixing or finding an alternate solution to allowing their support team access to their client machines. Currently they used a reverse SSH tunnel and an intermediary VH but for some reason, that was very unreliable in terms of availability. I suggested using OpenVPN and explained how it would work. Seemed to be a far better idea and they accepted. After several days of working through documentations and guides and everything, I figured out how OpenVPN works and managed to deploy a TEST server and successfully test remote access using two VMs. On seeing my tests, the boss told me that he wanted to test it on the client network. I agreed. Today he comes to me and he tells me to prepare testing for tomorrow and that the client technician is going to give me access to one of their boxes. And then he adds, "It's a working prod server. We'll see if we can make it work on that" and left. I gaped at him for a while and asked another dev guy in the room if what I heard was right. He confirmed. Turns out, the lead dev and the boss's son (who also works here) had had a huge argument since morning on the same issue and finally the dev guy had washed it off his hands and declared that if anything goes wrong from testing it on production, it's entirely the boss's own fault. That's when the boss stepped in and approached me. I ran back to his office and began to explain why prod servers don't top the list of things you can fuck around with. But he simply silenced me saying, "What can go wrong?" and added, "You shouldn't stay still. You should keep moving". Okay, like firstly what the fuck and secondly, what the fuck?.
Even though OpenVPN client is not the scariest thing to install, tomorrow's going to be fun.

We had 1 Android app to be developed for charity org for data collection for ground water level increase competition among villages.

Initial scope was very small & feasible. Around 10 forms with 3-4 fields in each to be developed in 2 months (1 for dev, 1 for testing). There was a prod version which had similar forms with no validations etc.

We had received prod source, which was total junk. No KT was given.

In existing source, spelling mistakes were there in the era of spell/grammar checking tools.

There were rural names of classes, variables in regional language in English letters & that regional language is somewhat known to some developers but even they don't know those rural names' meanings. This costed us at great length in visualizing data flow between entities. Even Google translate wasn't reliable for this language due to low Internet penetration in that language region.

OOP wasn't followed, so at 10 places exact same code exists. If error or bug needed to be fixed it had to be fixed at all those 10 places.

No foreign key relationships was there in database while actually there were logical relations among different entites.

No created, updated timestamps in records at app side to have audit trail.

Small part of that existing source was quite good with Fragments, MVP etc. while other part was ancient Activities with business logic.

We have to support Android 4.0 to 9.0 of many screen sizes & resolutions without any target devices issued to us by the client.

Then Corona lockdown happened & during that suddenly client side professionals became over efficient.

Client started adding requirements like very complex validation which has inter-entity dependencies. Then they started filing bugs from prod version on us.

Let's come to the developers' expertise,
2 developers with 8+ years of experience & they're not knowing how to resolve conflicts in git merge which were created by them only due to not following git best practice for coding like only appending new implementation in existing classes for easy auto merge etc.

They are thinking like handling click events is called development.

They don't want to think about OOP, well structured code. They don't want to re-use code mostly & when they copy paste, they think it's called re-use.

They wanted to follow old school Java development in memory scarce Android app life cycle in end user phone. They don't understand memory leaks, even though it's pin pointed by memory leak detection tools (Leak canary etc.).

Now 3.5 months are over, that competition was called off for this year due to Corona & development is still ongoing.

We are nowhere close to completion even for initial internal QA round.

On top of this, nothing is billable so it's like financial suicide.

Remember whatever said here is only 10% of what is faced.

- An Engineering lead in a half billion dollar company.

When you raise a ticket with your support team that your prod server is currently down and they mark it as a medium request, like dude... seriously?! Get it sorted ASAP paleaaaaase

So, we (I'm the backend guy and work with a UI dev) are building this product portfolio management tool for our client and they have a set of 250 users. The team has two point of contacts for the 250 users who maintain the master data, help users with data quality, tool guidance, reporting and other stuff. So one day one of these two support users come to me and say : Hey I'm not able to add new transactions coz a customer is missing.

We have the provision to create / maintain customers.

I check the production DB, application code, try creating the customer and then the transaction, everything works perfectly fine.

I ask the user for a screen sharing session, the user starts reproducing the error like this :

We have a 3 system landscape - Dev / Test and Prod

U : Logs into the test system url, creates the customer.

U : Points out the toast saying customer creation is successful.

U : opens a new tab, opens the production system, tries creating the transaction, searches for the customer and says " see !! cant find the customer here ! the master data management apps never work !! "

FML?.

Backend : *breaks feature in prod by changing api*

Me: Feature is broken in prod. Please fix. I was told that the API will not deprecate the old use case.

Backend: ... Fixing it in the backend will take time. Add support for it in the frontend.

Me: I'm not done with the new feature just yet. And it will take some time to have it reviewed and fully tested. Please fix the API.

Backend: .... Well, make a new PR and add support for it first. The new feature can come later.

Me. (-_-) Okay

Sometimes it feels like I'm a code janitor rather than a frontend intern

Update: for those of you who know cougar woman/ my self proclaimed "work mom" in my previous posts (bitch who keeps stealing my lunch), yesterday she really tipped me over the edge. So I'm fucking hauling ass on my sprint work because I had to take over another team member's tasks (because he "doesn't have time" for it being prod support but all I see him doing is shitposting at his desk) and someone from another team asked the cougar a question about something. she comes up to me all demanding like "HEY you, you got the worksapce open?"
I was like um... no I don't. I'm working on the other application.
and then she's like "Look at my laptop and show him". Okay bitch let me just drop everything I'm doing to help a random person. The fact that she commanded me instead of just asking me pissed me off. Not even a single sign of "please". I'm tired of her truly. She is a snake. Even to her friend on our team. Every time he's out she's like "hey where is _____? huh why isn't he here??? hmmm" in an instigating way to remind everyone that he might've taken off for no reason. When I was too sick to volunteer at an event she organized in the morning on Saturday, she asked me the following Monday, "oh did you drink too much? lmao. a spiteful, grudge holding bitch for sure

Doing the deployment to production, and towards the end one of the support guys looks over at me;

"So, the website in prod is throwing some errors"

Followed by another guy:

"Yeah I'm getting the same, SQL exceptions on the page"

I stare at them panicked for a moment, when one of them goes "just kidding!". Like dude, my heart just skipped several beats!

Any one here ever had something cruel happen to them during a deployment?

Realized I could of prevented an entire day of our team dealing with a prod issue that could've been quickly resolved if I had noticed the Support guy actually restarted the wrong server...

Prod Issue + Friday == BAD... My brain has already logged off as of this morning...

attention to details == 0

Assuming someone else will fix it/do things correctly == 100

Two (2) senior developers and one (1) senior tester left our team and I am left with two (2) Java legacy applications that are hard to maintain. Here is a list of things I hate about these old webapps (let's call them app A and B):

1. App A depends on 80% web services. If one web service for a product or warehouse goes down, work flow is impeded while prod support team checks with the core services team for repair

2. App B is a maven project with multiple modules dependent on libraries that are dependent on company's internal libraries. So if we want to upgrade to OpenJdk 9 and up, the project will definitely produce a lot of errors due to deprecated/unsupported codes

3. App A is dependent on Tibco and I have no experience on that

4. App B's continuous integration build tool is Jenkins and the jobs that build it has a shell script that wasn't updated during the tech upgrade enhancement. The previous developer who did the knowledge transfer to me didn't tell me about this (it should be considered a defect on her part but she already resigned)

5. App A when loaded in eclipse IDE is a pain to work with since it is only allowed to build a war file using ant. I have to lookup in quick search instead of calling shortcuts (call hierarchy) because the project wasn't compiled via eclipse.

6. It's impossible to debug app A because of #5

7. Both applications have high priority and complex enhancements and I have no other teammates to help me

8. You never know what else can go wrong anytime

saw an SSIS project once used for basically string replacement and file tasks. Asked manager permission to rewrite this Proof of Concept to something more suitable language.
manager: you can't but you have to support it.
me: but i tought it's a PoC
manager: yes it is
me: so what to support?
manager: it's been sold to a customer and in production for 6 months now
So it's a PoC which is sold and moved to prod...
I hate sales and management douches sometimes... like 98% of the time....
(already left the company)

While addressing a Senior Dev's (SD) query from another team.

SD: why is this field mandatory? Can't it be just optional? Any other work around?
Me: Is your code changes already pushed in Devo? In that case, we provide a value which will work since you are not concerned about it.
SD: Yes. It's pushed till production. And, I want to test changes in Prod.
Me: (shared some codes) and explained that this feature for testing is only available in Devo.
SD: I know that. (Shared me a ticket) I want this field to be optional. That's it.
Me: (read the entire ticket. Didn't find anything related their) Told him, I will discuss with team. And meanwhile, for Devo, you can use this value.

Next morning, I accidentally came over some other ticket raised by him only which had the correct doubts regarding request to support this field in production
Now, I don't know why did he share a wrong ticket with me.
And, how will it even help him if that field was even optional.
THAT JUST WONT WORK IN PRODUCTION.

I will discuss with my team and see what can happen.

For a back-end, Java engineer, It's really sad when you are so bored at work that you get excited at the thought of designing SQL views/procs to make prod support easier.

- Am a junior dev in an awesome team & exciting project after my apprenticeship and while having just started my part time studies
- Have restructure in company so I land in an other value stream
- Get laid off by new value stream 6 months later (now) because they have a serious budget cut
- Take time to come to terms with situation. I could finally work more on my side projects or focus a bit more on my studies. Hey actually I will have 5 months time to look for something while being paid by the company and they help me brush up my CV. Pretty neat!
- Now my former boss wants me back because of my experience in the project, but only as a production support and not as dev (because budget and they're bleeding with tickets)

Not sure if I should take the offer as it feels safe to have an income and the team is cool. However, it feels a bit like a degradation as prod support sucks in that project and I'd like to code (which wouldn't be possible then).
And as this is still my first company I'm working in, it would make sense to look for something else...

Grrr need to sleep about it... Decision-making isn't exactly my strength.

Long post, TLDR: Given a large team building large enterprise apps with many parts (mini-projects/processes), how do you reduce the bus-factor and the # of Brent's (Phoenix Project)?

# The detailed version #

We have a lot of people making changes, building in new processes to support new flows or changes in the requirements and data.

But we also have to support these except when it gets into Production there is little information to quickly understand:
- how it works
- what it does/supposed to do
- what the inputs and dependencies are

So often times, if there's an issue, I have to reverse engineer whatever logic I can find out of a huge mess.

I guess the saying goes: the only people that know how it works is whoever wrote it and God.

I'm a senior dev but i spend a lot of time digging thru source code and PROD issues to figure out why ... is broken and how to maybe fix it.

I think in Agile there's supposed to be artifacts during development but never seen em.

Personally whenever i work on a new project, I write down notes and create design diagrams so i can confirm things and have easy to use references while working.

I don't think anyone else does that. And afterwards, I don't have anywhere to put it/share it. There is no central repo for this stuff other than our Wiki but for the most part, is like a dumping ground. You have to dig for information and hoping there's something useful.

And when people leave, information is lost forever and well... we hire a lot of monkeys... so again I feel a lot of times i m trying to recover information from a corrupted hard drive...

The only way real information is transferred is thru word of mouth, special knowledge transfer sessions.

Ideally I would like anything that goes into PROD to have design docs as well as usage instructions in order for anyone to be able to quickly pick it up as needed but I'm not sure if that's realistic.

Even unit tests don't seem to help much as they just test specific functions but don't give much detail about how a whole process is supposed to work.

So, today we had our first production release of our web app. Last week we changed a big part of our UI. This week we changed the design, rewrote the complete API documentation, implemented mobile support. While the release the administration center of our cloud was unreachable. Shortly after the release we made a bug fix and deployed it directly to production.
So today was a very normal prod release 😂

This is an actual transcript...
Since it's way too long for the normal 5000 characters, hence splitting it up...

Infra Guy: mr Dev, could you please give some rational for update of jjb?
Dev: sparse checkout support is missing
Infra Guy: is this support mandatory to achive whatever you trying to do?
Dev: yes
Infra Guy: u trying to get set of specific folder for set of specific components?
Dev: yes
Infra Guy: bash script with cp or mv will not work for you?
Dev: no
Infra Guy: ?
Dev: when you have already present functionality why reinvent the wheel
Dev: jenkins has support for it
Dev: the jjb is the bottle neck
Infra Guy: getting this functionality onto our infra would have some implications
Dev: why should I write bash script if jenkins allows me to do that
Dev: what implications ??
Infra Guy: will you commit to solve all the issues caused by new jjb?
Dev: you show me the implications first
Infra Guy: like a year ago i have tried to get new jjb <commit_url>
Infra Guy: no, the implications is a grey area
Infra Guy: i cant show all of them and they may hit like in week or eve month
Dev: then why was it not tackled
Dev: and why was it kept like that
Infra Guy: few jobs got broken on something
Dev: it will crop up some time later
Dev: if jobs get broken because of syntax
Dev: then jobs can be fixed
Dev: is it not ???
Infra Guy: ofc
Infra Guy: its just a question who will fix them
Dev: follow the syntax and follow the guidelines
Dev: put up a test server and try and lets see
Dev: you have a dev server
Dev: why not try on that one and see what all jobs fails
Dev: and why they fail
Dev: rather than saying it will fail and who will fix
Dev: let them fail and then lets find why
Dev: I manually define a job
Dev: I get it done
Infra Guy: i dont think we have test server which have the same workload and same attention as our prod
Dev: unless you test how would you know ??
Dev: and just saying that it broke one with a version hence I wont do it
Infra Guy: and im not sure if thats fair for us to deal with implication of upgrading of the major components just cause bash script is not good enough for u
Dev: its pretty bad
Infra Guy: i do agree
Infra TL Guy: Dev, what Infra Guy is saying is that its not possible to upgrade without downtime
Infra Guy: no
Dev: how long a downtime are we looking at ??
Infra Guy: im saying that after this upgrade we will have deal with consequences for long time
Infra Guy-2: No this is not testing the upgrade is the huge effort as we dont have dev resources to handle each job to run
Dev: if your jjb compiles all the yaml without error
Dev: I am not sure what consequences are we talking of
Infra Guy: so you think there will be no consequences, right?
Dev: unless you take the plunge will you know ??
Dev: you have a dev server running at port 9000
Infra Guy: this servers runs nothing
Dev: that is good
Dev: there you can take the risk
Infra Guy: and the fack we have managed to put something onto api doesnt mean it works
Dev: what API ?
Infra Guy: jenkins api
Infra Guy: hmmm
Dev: what have you put on Jenkins API ??
Infra Guy: (
Dev: jjb is a CLI
Infra Guy: ((
Dev: is what I understand
Dev: not a Jenkins API
Infra Guy: (((
Dev: (((((
Infra Guy: jjb build xmls and push them onto api
Infra Guy: and its doent matter
Dev: so you mean to say upgrading a CLI is goig to upgrade your core jenkisn API
Dev: give me a break
Infra Guy: the matter is that even if have managed to build something and put it onto api
Infra Guy: doesnt mean it will work
Dev: the API consumes the xml file and creates a job
Infra Guy: right
Dev: if it confirms to the options which it understands
Dev: then everything will work
Dev: I am actually not getting your point Infra Guy
Infra Guy: i do agree mr Dev
Dev: we are beating around the bush
Infra Guy: just want to be sure that if this upgrade will break something
Infra Guy: we will have a person who will fix it
Dev: that is what CICD is supposed to let me know with valid reasons
Dev: why can't that upgrade be done
Infra Guy: it can be done
Infra Guy: i even have commit in place

My first vacation of the year! Two whole glorious weeks away from the office!

Well, you know, except for the dozen or so inevitable prod support calls I'm sure to get :(

Sure wish I was actually going somewhere and doing something, but given what year it is, just sitting on the couch and getting fat(ter) doesn't sound so bad.

And maybe work on the 10 different side projects I've got going somewhere along the way.

But mostly just sitting on my ass doing nothing, and it'll be glorious!

Couldn't find it on Google but does Node 6.9.1 support ES6 syntax and const, let, etc?

The code runs on my machine which uses 6.9.1 I think but the Code Style report raises issues saying it needs a esversion: 6. I can add it but dunno if it's gonna break in PROD

This got me fucked up. Listen yo.

So we have this issue on our modal right. The issue keeps poppin. It's a hotfix because its in prod. So my senior and I were on it. After a few hours, I showed him the part of the code that is buggy. It's 50 lines of code of nested if-else, else-if. And so we're still fighting it. He redid everything since we're using angular2 he did a subject, behavior-subject all that bs and I was still trying to understand what's the bug, because it's happening on the second click and so I did my own thing and found the cause bug and showed it to him, its this:

setTimeout( () => {}, 0)

the bootstrap-modal doesn't allow async inside it (I dont why, its in the package). So he explained to me why it's there. So I did my own thing again and find a workaround which I did, a one-line of angular property, showed it to him he didn't accept it because we'll still have to redo it with subjects and he was on it. I said ok. Went back to my previous issue. The director came in and ask for a fixed, my senior came up to me and told me to push my fix. Alright no problem. So we good now. Went back to our thing bla bla bla, then got an email that we will have a meeting, So we went, bla bla bla. The internal team wants a support for mobile, senior said no problem bla bla bla, after the meeting he approaches me and said (THIS IS WHERE IT GOT FUCKED UP) we wont be supporting bootstrap4 anymore because of the modal issue and since we're going to support mobile and BOOTSTRAP4 grid system is NONINTUITIVE we are moving to material design because the grid system is easier. I was blown away man. we have more than 100 components and just because of that modal and mobile support shit he decided to abandon bootstrap. Mater of fact its the modal its his code. I'm not expert in frontend but I looked at the material design implementation its the same thing other than the class names. OHHH LAWD!

I hate prod support grrrrrr