This guy at an internship who only wanted to use anything Microsoft.

It was fine for his own use but he also wanted it for a high security prod environment and tried to push that through.

Luckily, the (very competent) team lead refused to use closed source stuff for high security environments.

"listen (team lead to that guy), it's not going to happen. We're simply not using software from a US based company which is closed source for high security stuff.
Why? The US is one of the biggest surveillance powers in this world, we just can't be sure what's in the software if it's US based. Now you can say that that's paranoid but whether or not it is, the surveillance part is a fact, deal with it. That you want to use it, fine, but NOT. IN. HIGH. SECURITY. PROD. (or prod at all really).

He continued to try and convert colleagues to windows and other Microsoft stuff for the rest of his internship.

My mentor/guider at my last internship.

He was great at guiding, only 1-2 years older than me, brought criticism in a constructive way (only had a very tiny thing once in half a year though) and although they were forced to use windows in a few production environments, when it came to handling very sensitive data and they asked me for an opinion before him and I answered that closed source software wasn't a good idea and they'd all go against me, this guy quit his nice-guy mode and went straight to dead-serious backing me up.
I remember a specific occurrence:
Programmers in room (under him technically): so linuxxx, why not just use windows servers for this data storage?
Me: because it's closed source, you know why I'd say that that's bad for handling sensitive data
Programmers: oh come on not that again...
Me: no but really look at it from my si.....
Programmers: no stop it. You're only an intern, don't act like you know a lot about thi....
Mentor: no you shut the fuck up. We. Are. Not. Using. Proprietary. Bullshit. For. Storing. Sensitive. Data.
Linuxxx seems to know a lot more about security and privacy than you guys so you fucking listen to what he has to say.
Windows is out of the fucking question here, am I clear?

Yeah that felt awesome.

Also that time when a mysql db in prod went bad and they didn't really know what to do. Didn't have much experience but knew how to run a repair.
He called me in and asked me to have a look.
Me: *fixed it in a few minutes* so how many visitors does this thing get, few hundred a day?
Him: few million.
Me: 😵 I'm only an intern! Why did you let me access this?!
Him: because you're the one with the most Linux knowledge here and I trust you to fix it or give a shout when you simply can't.

Lastly he asked me to help out with iptables rules. I wasn't of much help but it was fun to sit there debugging iptables shit with two seniors 😊

He always gave good feedback, knew my qualities and put them to good use and kept my motivation high.

Awesome guy!

Why is this CSS not updating???

Ctrl + F5

Still!!!

Let's check in incognito..

F*CK, really!!!!

.... Ooooooh right I'm checking prod not dev.

toxic workplace; leaving

I haven't wanted to write this rant. I haven't even wanted to talk to anyone (save my gf, ofc). I've just been silently fuming.

I wrote a much longer rant going into far too much detail, but none of that is relevant, so I deleted it and wrote this shorter (believe it or not) version instead. And then added in more details because details.

------

On Tuesday, as every Tuesday, I had a conference call with the rest of the company. For various, mostly stupid reasons, the boss yelled at and insulted me for twenty minutes straight in front of everyone, telling me how i'm disorganized, forgetful, how can't manage my time, can't manage myself let alone others, how I don't have my priorities straight, etc. He told the sales team to get off the call, and then proceeded to yell and chew at me for another twenty minutes in front of the frontend contractor about basically the same things. The call was 53 minutes, and he spent 40 minutes of it telling me how terrible I've been. No exaggeration, no spin. The issues? I didn't respond to an email (it got lost in my ever-filling inbox), and I didn't push a very minor update last week (untested and straight to prod, ofc). (Side note: he's yelled at me for ~15 minutes before for being horribly disorganized and unable to keep up on Trello -- because I had a single card in the wrong column. One card, out of 60+ over two boards. Never mind that most have time estimates, project tags, details, linked to cards on his boards, columns for project/qa/released, labels for deferred, released to / rejected from qa, finished, in production, are ordered by priority, .... Yep. I'm totes disorganized.)

Anyway, I spent most of conference call writing "Go fuck yourself," "Choke on a cat and die asshole," "Shit code, low pay, and broken promises. what a prize position," etc. or flipping him off under the camera on our conference-turn-video-call (switched due to connection issues, because ofc video is more stable than audio-only in his mind).

I'm just.
so, so done.

I did nothing the rest of the day on Tuesday, and basically just played games on Wednesday. I did one small ticket -- a cert replacement since that was to expire the next day -- but the rest was just playing CrossCode. (fun game, fyi; totally recommend.)

Today? It's 3:30pm and I can't be bothered to do anything. I have an "urgent" project to finish by Monday, literally "to give [random third party sales guy] a small win". Total actual wording. I was to drop all other tasks (even the expiring cert lol) and give this guy his small win. fucking whatever. But the project deals with decent code -- it's a minor extension to the first project I did for the company (see my much earlier rants), back when I was actually applying myself and learning something (everything) new, enjoying myself, and architecting+writing my own code. So I might actually do the project, but It's been two days and I haven't even opened single file yet.

But yeah. This place is total and complete shit. Dealing with the asshole reminds me of dealing with my parents while growing up, and that's a subject I don't want to broach -- far too many toxic memories.

So, I'm quitting as soon as I find something new.
and with luck, this will be before assface hires my replacement-to-be, and who will hopefully quit as soon as s/he sees the abysmal codebase. With even more luck, the asshole king himself will get to watch his company die due to horrible mismanagement. (though ofc he'll never attribute it to himself. whatever.)

I just never want to see or think about him again.
(nor this fetid landfill of a codebase. bleh.)

With luck, this will be one of my last rants about this toxic waste dump and its king of the pile.
Fourty fucking minutes, what the fuck.

“Yeah but you’re not a *real* developer”

Fuck. you.

I wrote 80% of this code base. I do 80% of the tickets/storyboard points. I do all of the QA. My nose is to the grindstone every fucking day honing this craft and sweating my balls off like a blacksmith staring into the red hot kiln while the sores of previous mistakes scream bloody murder from the unrelenting exposure to heat. I saw this amazing industry of opportunity, freedom and self examination and wanted in no matter what it took. I glued myself to every pithy resource I could possibly get my hands on and crawled through the muck and filth of it all until I could keep myself warm with the smallest spark of my own making. I stoked that spark until it became a fire and stoked that fire until I could set entire forests ablaze. I listened to the ungrateful people keeping warm by my combustion saying it “wasn’t hot enough” or “would have been a nicer colour if they did it” or “could have warmed up just fine jogging on the spot”. I made painstaking alterations to my ignition and watched my undeserving benefactors gradually be silenced and begin to sit quietly by the heat. I jumped into that inferno daily, was reduced to ash daily and emerged reborn daily. But you are right! I didn’t get scammed out of $40k+ studying technology in an archaic institution from instructors who don’t give a shit and answering “D all of the above” for 4+ years straight therefor my opinion doesn’t mean shit. Push your bullshit to prod and watch the server come burning out of the cloud as the apocalyptic swarm of angry tickets come flooding in why don’t you? Bet they didn’t teach you that in school. You’ve never poked around inside an open source codebase in your life. They are just a mystery boxes of magic that unless someone holds your hands with finely crafted instructions containing a 50/50 picture to word ratio you throw a hissy fit. Every problem that comes up instead of working to solve it you reflexively point to the first person in the room while thinking with your pea brain how you can possibly scapegoat them into taking the fall for whatever it is that’s come up today you couldn’t possibly understand.

Not a real developer?

Fuck. You.

Going on a vacation, so notify all clients that I won't be available during two weeks.

Client: well we have this huge presentation and here's a list of stuff we absolutely need for it
Me: sure I have a look.
Me: holy shit dude! That's gona take about 2-3 days. I'm leaving soon!
Client: it's realy critical to have them in a week as it's a very important presentation! Is there any way you can make it work? If we can do anything to help, just name it.
Me: well I'll do my best (planed 1 day for such rhings)
Me: *pulling a 15h day*
Me: here, all done budy! Did a 15h but now it's done, so do that presentation!
Client: oh, nice, but it wasn't that urgent
Me: ...

Next day:
Client ssh in to the server, fucks shit up
Client: well I did a thing and now stage and prod is fucked, can you do anything?
Me: (knowing it will take 30min to fix) well... I try my best. Btw. I'll leave in a few hours and won't take my computer, so try not to fuck every thing again, okay?

One week, and it turned out to be worse than that.

I was put on a project for a COVID-19 program in America (The CARES Act). The financial team came to us on Monday morning and said they need to give away a couple thousand dollars.

No big deal. All they wanted was a single form that people could submit with some critical info. Didn't need a login/ registration flow or anything. You could have basically used Google Forms for this project.

The project landed in my lap just before lunch on Monday morning. I was a junior in a team with a senior and another junior on standby. It was going to go live the next Monday.

The scope of the project made it seem like the one week deadline wasn't too awful. We just had to send some high priority emails to get some prod servers and app keys and we were fine.

Now is the time where I pause the rant to express to you just how fine we were decidedly **not**: we were not fine.

Tuesday rolls around and what a bad Tuesday it was. It was the first of many requirement changes. There was going to need to be a review process. Instead of the team just reading submissions from the site, they needed accept and reject buttons. They needed a way to deny people for specific reasons. Meaning the employee dashboard just got a little more complicated.

Wednesday came around and yeah, we need a registration and login flow. Yikes.

Thursday came and the couple-thousand dollars turned into a tens of millions. The amount of users we expected just blew up.

Friday, and they needed a way for users to edit their submissions and re-submit if they were rejected. And we needed to send out emails for the status of their applications.

Every day, a new meeting. Every meeting, new requirements that were devastating given our timeframe.

We put in overtime. Came in on the weekend. And by Monday, we had a form that users could submit and a registration/ login flow. No reviewer dashboard. We figured we could take in user input on time and then finish the dashboard later.

Well, financial team has some qualms. They wanted a more complicated review process. They wanted roles; managers assign to assistants. Assistants review assigned items.

The deadline that we worked so hard on whizzed by without so much as a thought, much less the funeral it deserved.

Then, they wanted multiple people to review an application before it was final. Then, they needed different landing pages for a few more departments to be able to review different steps of the applications.

Ended up going live on Friday, close to a month after that faithful Monday which disrupted everything else I was working on, effective immediately.

I don't know why, but we always go live on a Friday for some reason. It must be some sort of conspiracy to force overtime out of our managers. I'm baffled.

But I worked support after the launch.

And there's a funny story about support too: we were asked to create a "submit an issue" form. Me and the other junior worked on it on a wednesday three weeks into the project. Finished it. And the next day it was scrapped and moved to another service we already had running. Poor management like that plagued the project and worked in tandem with the dynamic and ridiculous requirements to make this project hell.

Back to support.

Phone calls give me bad anxiety. But Friday, just before lunch, I was put on the support team. Sure, we have a department that makes calls and deal with users. But they can't be trained on this program: it didn't exist just a month ago, and three days ago it worked differently (the slippery requirements never stopped).

So all of Friday and then all of Saturday and all of Monday (...) I had extended panic attacks calling hundreds of people. And the team that was calling people was only two people. We had over 400 tickets in the first two days.

And fuck me, stupid me, for doing a good job. Because I was put on the call team for **another** COVID project afterwards. I knew nothing about this project. I have hated my job recently. But I'm a junior. What am I gonna say, no?

Doot doot.

My day: Eight lines of refactoring around a 10-character fix for a minor production issue. Some tests. Lots of bloody phone calls and conference calls filled with me laughing and getting talked over. Why? Read on.

My boss's day: Trying very very hard to pin random shit on me (and failing because I'm awesome and fuck him). Six hours of drama and freaking out and chewing and yelling that the whole system is broken because of that minor issue. No reading, lots of misunderstanding, lots of panic. Three-way called me specifically to bitch out another coworker in front of me. (Coworker wasn't really in the wrong.) Called a contractor to his house for testing. Finally learned that everything works perfectly in QA (duh, I fixed it hours ago). Desperately waited for me to push to prod. Didn't care enough to do production tests afterwards.

My day afterwards: hey, this Cloudinary transform feature sounds fun! Oh look, I'm done already. Boo. Ask boss for update. Tests still aren't finished. Okay, whatever. Time for bed.

what a joke.

Oh, I talked to the accountant after all of this bullshit happened. Apparently everyone that has quit in the last six years has done so specifically because of the boss. Every. single. person.

I told him it was going to happen again.

I also told him the boss is a druggie with a taste for psychedelics. (It came up in conversation. Absolutely true, too.) It's hilarious because the company lawyer is the accountant's brother.

So stupid.

Welcome back to practiseSafeHex's new life as a manager.

Episode 2: Why automate when you can spend all day doing it by hand

This is a particularly special episode for me, as these problems are taking up so much of my time with non-sensical bullshit, that i'm delayed with everything else. Some badly require tooling or new products. Some are just unnecessary processes or annoyances that should not need to be handled by another human. So lets jump right in, in no particular order:

- Jira ... nuff said? not quite because somehow some blue moon, planets aligning, act of god style set of circumstances lined up to allow this team to somehow make Jira worse. On one hand we have a gigantic Jira project containing 7 separate sub teams, a million different labels / epics and 4.2 million possible assignees, all making sure the loading page takes as long as possible to open. But the new country we've added support for in the app gets a separate project. So we have product, backend, mobile, design, management etc on one, and mobile-country2 on another. This delightfully means a lot of duplication and copy pasting from one to the other, for literally no reason what so ever.

- Everything on Jira is found through a label. Every time something happens, a new one is created. So I need to check for "iOS", "Android", "iOS-country2", "Android-country2", "mobile-<feature>", "mobile-<feature>-issues", "mobile-<feature>-prod-issues", "mobile-<feature>-existing-issues" and "<project>-July31" ... why July31? Because some fucking moron decided to do a round of testing, and tag all the issues with the current date (despite the fact Jira does that anyway), which somehow still gets used from time to time because nobody pays attention to what they are doing. This means creating and modifying filters on a daily basis ... after spending time trying to figure out what its not in the first one.

- One of my favourite morning rituals I like to call "Jira dumpster diving". This involves me removing all the filters and reading all the tickets. Why would I do such a thing? oh remember the 9000 labels I mentioned earlier? right well its very likely that they actually won't use any of them ... or the wrong ones ... or assign to the wrong person, so I have to go find them and fix them. If I don't, i'll get yelled at, because clearly it's my fault.

- Moving on from Jira. As some of you might have seen in your companies, if you use things like TestFlight, HockeyApp, AppCenter, BuddyBuild etc. that when you release a new app version for testing, each version comes with an automated change-log, listing ticket numbers addressed ...... yeah we don't do that. No we use this shitty service, which is effectively an FTP server and a webpage, that only allows you to host the new versions. Sending out those emails is all manual ... distribution groups?? ... whats that?

- Moving back to Jira. Can't even automate the changelog with a script, because I can't even make sense of the tickets, in order to translate that to a script.

- Moving on from Jira. Me and one of the remote testers play this great game I like to call "tag team ticketing". It's so much fun. Right heres how to play, you'll need a QA and a PM.

*QA creates a ticket, and puts nothing of any use inside it, and assigns to the PM.
*PM fires it back asking for clarification.
*QA adds in what he feels is clarification (hes wrong) and assigns it back to the PM.
*PM sends detailed instructions, with examples as to what is needed and assigns it back.
*QA adds 1 of the 3 things required and assigns it back.
*PM assigns it back saying the one thing added is from the wrong day, and reminds him about the other 2 items.
*QA adds some random piece of unrelated info to the ticket instead, forgetting about the 3 things and assigns it back.

and you just continue doing this for the whole dev / release cycle hahaha. Oh you guys have no idea how much fun it is, seriously give it a go, you'll thank me later ... or kill yourselves, each to their own.

- Moving back to Jira. I decided to take an action of creating a new project for my team (the mobile team) and set it up the way we want and just ignore everything going on around us. Use proper automation, and a kanban board. Maybe only give product a slack bot interface that won't allow them to create a ticket without what we need etc. Spent 25 minutes looking for the "create new project" button before finding the link which says I need to open a ticket with support and wait ... 5 ... fucking ... long ... painful ... unnecessary ... business days.

... Heres hoping my head continues to not have a bullet hole in it by then.

Id love to talk more, but those filters ain't gonna fix themselves. So we'll have to leave it here for today. Tune in again for another episode soon.

And remember to always practiseSafeHex

One of our clients deploy their own server app. So this happened after a prod deployment. (4am)

*Cellphone rings while sleeping*

Client : we need you on the conference call now. URGENT!

*Gets on conference call*
*Client explain the problem*
*Explaining to the client that the problem is in their side (https connection not working, either network or certificate problem)*
*Client doesn't believe it and pushes me for a fix that I have no control on*
*4 hours later in a heated conversation*

Client : ok problem is on our side. We used our SSL certificate from staging with production and thought it would work.

Me :

"Hey, Root, someone screwed up and now all of our prod servers are running this useless query constantly. I know I already changed your priorities six times in the past three weeks, but: Go fix it! This is higher priority! We already took some guesses at how and supplied the necessary code changes in the ticket, so this shouldn't take you long. Remember, HIGH PRIORITY!"

1. I have no idea how to reproduce it.
2. They have no idea how to reproduce it.
3. The server log doesn't include queries.
4. The application log doesn't include queries.
5. The tooling intercepts and strips out some log entries the legendary devs considered useless. (Tangent: It also now requires a tool to read the logs because log entries are now long json blobs instead of plain text.)
6. The codebase uses different loggers like everywhere, uses a custom logger by default, and often overwrites that custom logger with the default logger some levels in. gg
7. The fixes shown in the ticket are pretty lame. (I've fixed these already, and added one they missed.)
8. I'm sick and tired and burned out and just can't bring myself to care. I'm only doing this so i don't get fired.
9. Why not have the person who screwed this up fix it? Did they quit? I mean, I wouldn't blame them.

Why must everything this company does be so infuriatingly complicated?

My team handles infrastructure deployment and automation in the cloud for our company, so we don't exactly develop applications ourselves, but we're responsible for building deployment pipelines, provisioning cloud resources, automating their deployments, etc.

I've ranted about this before, but it fits the weekly rant so I'll do it again.

Someone deployed an autoscaling application into our production AWS account, but they set the maximum instance count to 300. The account limit was less than that. So, of course, their application gets stuck and starts scaling out infinitely. Two hundred new servers spun up in an hour before hitting the limit and then throwing errors all over the place. They send me a ticket and I login to AWS to investigate. Not only have they broken their own application, but they've also made it impossible to deploy anything else into prod. Every other autoscaling group is now unable to scale out at all. We had to submit an emergency limit increase request to AWS, spent thousands of dollars on those stupidly-large instances, and yelled at the dev team responsible. Two weeks later, THEY INCREASED THE MAX COUNT TO 500 AND IT HAPPENED AGAIN!

And the whole thing happened because a database filled up the hard drive, so it would spin up a new server, whose hard drive would be full already and thus spin up a new server, and so on into infinity.

Thats probably the only WTF moment that resulted in me actually saying "WTF?!" out loud to the person responsible, but I've had others. One dev team had their code logging to a location they couldn't access, so we got daily requests for two weeks to download and email log files to them. Another dev team refused to believe their server was crashing due to their bad code even after we showed them the logs that demonstrated their application had a massive memory leak. Another team arbitrarily decided that they were going to deploy their code at 4 AM on a Saturday and they wanted a member of my team to be available in case something went wrong. We aren't 24/7 support. We aren't even weekend support. Or any support, technically. Another team told us we had one day to do three weeks' worth of work to deploy their application because they had set a hard deadline and then didn't tell us about it until the day before. We gave them a flat "No" for that request.

I could probably keep going, but you get the gist of it.

First rant here. Long, but please bear with me:

So after slogging my ass off in various early stage startups for over 4 years and keeping up with the almost non-existent development process, I joined an organisation which has some of the brightest and smartest minds I have had the pleasure to work with.

Mind you, this company is the market leader in it's field and has a 50+ people in it's tech team and the quality of work is pretty impressive.

Now for this week's sprint, I was asked to develop a feature which already exists on the Android app and they want to introduce in the iOS app too. The backend APIs are all in place and all I need to do is build it with virtually no dependency. My PM asks me to start with the UI and ask the backend dev for the API list whenever I need them.This is where the story turns.

For my first API, I go to the backend dev and ask him to share the API documentation and he looks at me as if I have asked him to dance the fucking cha cha. With a straight face he tells me that, 'The organisation doesn't maintain any kind of documentation for it's APIs.' Now this really shocks me. Even in a 5 men tech teams I have worked on, we have always maintained a spec doc for the APIs and this is a company which is known for it's tech practices.

Being the new guy I compose myself and ask if they have anything for me here: Postman collection, a workflowy doc, a goddamn txt file; anything which might help me, and he laughs at my dilusion and says no.

Dejected, I ask for a way to get the APIs and I am told that there are only two ways: either I keep bothering the Android dev for the APIs(No, I don't have the access to the android repo and nor am I gonna get it) which he had worked on 4 months back or I install the prod app on my phone, and use Charles to get every fucking API which is really, really annoying.

I thought writing out this rant would make me feel better, turns out it just made me angrier. Why the fuck can't they document such an important thing!?

Root: Fleshes out missing data in some factories. Tests affected code and finds the change breaks some specs (but shouldn’t).

Root: Reaches out to spec author.

Root: Messages thundercunt (the ticket’s code reviewer) on slack about the specs and the reaching out. No response.

Root: Works on another ticket while blocked.

Root: Logs off.

Root: Talks with spec author chick in the morning. Decide to pair on specs later.

TC: Still no slack response.

Root: Gives update in standup. Mentions factories and broken specs. Mentions pairing with spec chick.

TC: Still no slack response.

Root: Pulled off tickets in favor of prod issue. Gets ignored by everyone else diagnosing prod issue. Investigates prod issue by herself. Discovers prod issue isn’t from bad code, but bad requirements — code works as requested. Communicates this with details. Gets ignored by people still diagnosing prod issue. Tries again. Gets ignored. Gives up. Works on non-blocked tickets instead.

TC: Still no slack response.

Hours later:

TC: Comments on PR telling me I broke specs (how did I not notice?), that I need to reach out to spec chick and work with her, and that I can’t resolve the ticket until it’s fixed and passes code review.

TC: Still no slack response. (21 hours later at this point)

TC: Logs off. Still no response (25 hours at this point)

———

Ignoring the prod issue for the moment…

I broke specs. No shit.
I need to talk with spec chick. No shit.
I can’t resolve the ticket. No shit!

Bitch, I told you all of this 21 fucking hours prior, and again 3 hours prior during standup. But no, I clearly “don’t communicate” and obviously have no bloody clue what I’m doing, either, so I need everything spelled out for me.

And no, I didn’t resolve the fucking ticket. Why the fuck would I if it still has pending changes? Do you even check? Ugh!

And what the fuck with that prod issue? I’m literally giving you the answer. fucking listen! Stupid cunts.

Why is it all of the women I work with are useless or freaking awful people? Don’t get me wrong, many of the men are, too, but I swear it’s every single one of the women. (Am I awful, too?)

Just. Ugh.
I can’t wait to leave this sewer of a company.

Oddly still a good day, though. Probably because I talked to recruiters and sent out my resume again.

Manager: How come the push to prod didn’t happen?

Dev: We told you at the scrum yesterday. To reiterate, our dev environment was crashing so it’s not safe to push to prod until that is fixed.

Manager: Ok well lets set a goal to fix that and push to prod happens today so that it guaranteed happens.

Dev: That was our goal yesterday and it definitely didn’t happen.

Manager: I AM AWARE OF THAT. The corrective action is that this time compliance with the goal is 100% ABSOLUTELY MANDATORY!!

Dev: We’ll do our best, can’t guarantee anything until we figure out what the nature of what is occurring on dev though.

Manager: NO. I AM THE BOSS. YOU WILL 100% ABSOLUTELY COMPLY WITH THIS. THAT IS AN ORDER. YOU WILL SUCCESSFULLY GET THIS UPDATE OUT TO PROD TODAY. ANYTHING LESS THAN THAT SHALL BE CONSIDERED INSUBORDINATION. I WANT STATUS UPDATES EVERY 15 MINUTES ON WHERE WE ARE AT WITH THIS.

Dev: …

Dev: Can I get you to send me that request in an email?

*Manager leaves the meeting*

// *****************************

Job search is ticking along. It’s tough going though because I currently make ~120k and the best offers I’ve received so far are all ~70k because “You only have 2 years experience so you couldn’t possibly have the skills to be worth 120k. You are are junior level developer and 70K is already overpaying for you. We can pay you more later™. No we will not give you that in writing”. Ah well, the hunt continues.

Friend of mine killed his MacBook with some Softdrink.

Just poured it all over his poor a1502.

He let it dry for a few days, it starts to work again.
Except the battery.

Goes on Amazon and buys a new battery.

New battery doesn't work either and so he tells me about it and I as stupid as I am couldn't resist the temptation to finally work on a MacBook like my "hero" Lois Rossmann does.

So turns out the board is good.
Cleaned it up and basically nothing happened to it.

So what's the deal with "los batlerias"?

The first got hit by liquid, the second had a broken connection to a cell.
That could have happened through my friend, installing it without testing it first, or at the seller, so it being a DOA battery.

Now away from the stupidity of my friend and the situation to the actual source for this rant.

Once something happens to a modern Managed battery, the Battery Management System (BMS) disconnects the voltage from the system and goes into an error state, staying there and not powering anything ever again.
For noobs, it's dead. Buy a new one.

But It can be reset, depending you know how to, and which passwords were set at the factory.
Yes, the common Texas instruments BQ20Zxx chips have default passwords, and apple seems to leav them at default.

The Usb to SMBus adaptors arrived a few days ago and I went to prod the BMS.

There is a very nice available for Windows called BE2works, that I used the demo of to go in and figure out stuff. The full version supports password cracking, the demo not.

After some time figuring out how Smart Battery Systems (SBS) "API" works, I got to actually enter the passwords into the battery to try get into manufacturer and full access mode.

Just to realise, they don't unlock the BMS.

So, to conclude, my friend bought a "new" battery that was most likely cut out of a used / dead macbook, which reports 3000mah as fully charged instead of the 6xxx mah that it should have, with 0 cycles and 0hours used.
And non default access.

This screams after those motherfuckers scaming the shit out of people on Amazon, with refurb, reset, and locked fucken batteries.

I could kill those people right now.

Last but not least,
My friend theoretically can't send it back because I opened the battery to fix the broken connection.
Though maybe, it'll get send back anyway, with some suprise in the package.

I login this morning

Everyone going crazy as prod is broken in a million ways. 2.5 million exceptions In 24 hours.

We talk for 1 hour and solve nothing.

I diagnose the 2 biggest sources of exceptions and explain every step I took with screen snippets to reach my conclusions

The one other competent person on my team agrees. Otherwise complete silence.

I'm told not to fix the issue because I am leaving in 6 weeks and other people have to start taking responsibility.

No one fixes the issues. Instead they leave early. Now it's the weekend and the product is fucked.

Fucking useless people. Can't wait to leave.

Colleagues sharing passwords.That was a big fat NO when I was a sysadmin - and for a good reason. But now, since I'm closer to development, it feels like no one really cares about the passwords. If I tell my colleague I'll take 10 minutes more because I can't log in, he OFFERS me his credentials. And sends them over saying "in case you need it". [the next day the same colleague was complaining his account is locked out. Oh, wonders! How on Earth...!]

But seriously, password sharing is a serious problem. I would fire the person on spot if I caught him sharing his credentials! This is the 8th deadly sin! IDC if they are for non-prod. Most people reuse their passwords in multiple systems, and even non-prod envs can bring the prod down! Or worse - install a trojan.

This is going to be a long rant, coz this is the only way to vent out my frustration against our tech head.

Yesterday, while our fucking twat tech head was playing around in company aws account, he terminated the production server. By mistake, apparently. Coz he doesn't know shit about server management. But that egoist ass won't admit and fucked the production server.

And then ran away. We developers sprang into action. Updated dns to point to staging server, setup virtual hosts, env files, point to prod database, force flush dns cache. All systems were up and running in 30 mins. And since it was staging server, it had lot of untested features and codes, and we spent rest of the day fixing the bugs.

And that tech head, who ran away hiding his tail between his legs, after he fucked the server, came back after systems were up. And started cracking jokes, that "so many features got released in 1 day" . "We cut server cost by shutting down 1 server."

We were struggling and working in full throttle to make the services running again. And that fuckity fucker was cracking jokes.

And I don't even know what excuse he gave to ceo for the downtime. I am pretty sure he would have made up some crappy excuse to hide his fucking mistake. That ass never admits his mistake. I am thinking to go to ceo today and tell the real story and get that faggot head fired or at least a strict warning.

Week 278: Most rage-inducing work experience — I’ve got a list saved! At least from the current circle of hell. I might post a few more under this tag later…

TicketA: Do this in locations a-e.
TicketB: Do this in locations e-h.
TicketC: Do this in locations i-k.
Root: There’s actually a-x, but okay. They’re all done.
Product: You didn’t address location e in ticket B! We can’t trust you to do your tickets right. Did you even test this?
Root: Did you check TicketA? It’s in TicketA.
Product guy: It was called out in TicketB! How did you miss it?!
Product guy: (Refuses to respond or speak to me, quite literally ever again.)
Product guy to everyone in private: Don’t trust Root. Don’t give her any tickets.
Product manager to boss: Root doesn’t complete her tickets! We can’t trust her. Don’t give her our tickets.
Product manager to TC: We can’t trust Root. Don’t give her our tickets.
TC: Nobody can trust you! Not even the execs! You need to rebuild your reputation.

Root: Asks coworker a simple question.
Root: Asks again.
Root: nudges them.
Root: Asks again.
Coworker: I’ll respond before tomorrow. (And doesn’t.)
Root: Asks again.
Root: Fine. I’ll figure it out in my own.
TC: Stop making it sound like you don’t have any support from the team!

Root: Asks four people about <feature> they all built.
Everyone: idk
Root: Okay, I’ll figure it out on my own.
TC: Stop making it sound like you don’t have any support from the team!

Root: Mentions multiple meetings to discuss ticket with <Person>.
TC: You called <Person> stupid and useless in front of the whole team! Go apologize!

Root: Tells TC something. Asks a simple question.
Root: Tells TC the same thing. Asks again.
TC: (No response for days.)
TC: Tells me the exact same thing publicly like it’s a revelation and I’m stupid for not knowing.
TC: You don’t communicate well!

Root: Asks who the end user of my ticket is.
Root: Asks Boss.
Root: Asks TC.
Root: Fine, I’ll build it for both.
Root: Asks again in PR.
TC: Derides; doesn’t answer.
Root: Asks again, clearly, with explanation.
TC: Copypastes the derision, still doesn’t answer.
Root: Asks boss.
Boss: Doesn’t answer.
Boss: You need to work on your communication skills.

Root: Mentions asking question about blocker to <Person> and not hearing back. Mentions following up later.
<Person>: Gets offended. Refuses to respond for weeks thereafter.

Root: Hey boss, there’s a ticket for a minor prod issue. Is that higher priority than my current ticket?
Root: Hey, should I switch tickets?
Root: Hey?
Root: … Okay, I’ll just keep on my current one.
Boss: You need to work on your priorities.

Everyone: (Endless circlejerking and drama and tattling)

The riskiest dev choice...

How about "The riskiest thing you've done as a dev"? I have a great entry for that. and I suppose it was my choice to build the feature afterall.

I was working on an instance of a small MMO at a game company I worked for. The MMO boasted multiple servers, each of them a vastly different take on the base game. We could use, extend, or outright replace anything we wanted to, leading to everything from Zelda to pokemon to an RP haven to a top-down futuristic counterstrike. The server in this particular instance was a fantasy RPG, and I was building it a new leveling and experience system with most of the trimmings. (Talents, feats/perks, etc. were in a future update.)

A bit of background, first: the game's dev setup did not have the now-standard dev/staging/prod servers; everything ran on prod, devs worked on prod, players connected and played on prod, etc. Worse yet, there was no backup system implemented -- or not really. The CTO was really the only person with sufficient access. The techy CEO did as well, but he rarely dealt with anything technical except server hardware, occasionally. And usually just to troll/punish us devs (as in "Oops ! I pulled the cat5 ! ;)"). Neither of them were the most reliable of people, either. The CTO would occasionally remote in and make backups of each server -- we assumed whenever he happened to think of it -- and would also occasionally do it when asked, but it could take him a week, sometimes even up to a month to get around to it. So the backups were only really useful for retreiving lost code and assets, not so much for player data.

The lack of reliable backups and the lack of proper testing grounds (among the plethora of other issues at the company) made for an absolutely terrible dev setup, but that's just how it was, and that's what we dealt with. We were game devs, afterall. Terrible or not, we got to make games! What more could you ask for!? It was amazing and terrible and wonderful and the worst thing ever, all at the same time. (and no, I'm not sharing the company name, but it isn't EA or Nexon, surprisingly 😅)

Anyway, back to the story! My new leveling system also needed to migrate players' existing data, so... you can see where this is going.

I did as much testing and inspection of my code as I could, copied it from a personal dev script to the server's xp system, ... and debated if I really wanted to click [Apply]. Every time I considered it, I went back to check another part or do yet more testing. I ended up taking like 40 minutes to finally click it.

And when I did... that was the scariest button press of my life. And the scariest three seconds' wait afterwards. That one click could have ruined every single player's account, permanently lost us players ...

After applying it, I immediately checked my character to see if she was broken, checked the account data for corruption or botched flags, checked for broken interactions with the other systems....

Everything ended up working out perfectly, and the players loved all of the new features. They had no idea what went into building them, and certainly had no idea of what went into applying them, or what could have gone wrong -- which is probably a good thing.

Looking back, that entire environment was so fragile, it's a wonder things didn't go horribly wrong all the time. Really, they almost never did. Apocalypses did happen, but were exceedingly rare, and were ususally fixed quickly. I guess we were all super careful simply because everything was so fragile? or the decent devs were, at least. We never trusted the lessers with access 😅 at least on the main servers where it mattered. Some of the smaller servers... well, we never really cared about those.

But I'm honestly more surprised to realize I've never had nightmares of that button click. It was certainly terrifying enough.

But yay! Complete system overhaul and migration of stored and realtime player data! on prod! With no issues! And lots of happy players! Woooooo!

Thinking back on it makes me happy 😊

Be me, new dev on a team. Taking a look through source code to get up to speed.

Dev: **thinking to self** why is there no package lock.. let me bring this up to boss man

Dev: hey boss man, you’ve got no package lock, did we forget to commit it?

Manager: no I don’t like package locks.

Dev: ...why?

Manager: they fuck up computer. The project never ran with a package lock.

Dev: ..how will you make sure that every dev has the same packages while developing?

Manager: don’t worry, I’ve done this before, we haven’t had any issues.

**couple weeks goes by**

Dev: pushes code

Manager: hey your feature is not working on my machine

Dev: it’s working on mine, and the dev servers. Let’s take a look and see

**finds out he deletes his package lock every time he does npm install, so therefore he literally has the latest of like a 50 packages with no testing**

Dev: well you see you have some packages here that updates, and have broken some of the features.

Manager: >=|, fix it.

Dev: commit a working package lock so we’re all on the same.

Manager: just set the package version to whatever works.

Dev: okay

**more weeks go by**

Manager: why are we having so many issues between devs, why are things working on some computers and not others??? We can’t be having this it’s wasting time.

Dev: **takes a look at everyone’s packages** we all have different packages.

Manager: that’s it, no one can use Mac computers. You must use these windows computers, and you must install npm v6.0 and node v15.11. Everyone must have the same system and software install to guarantee we’re all on the same page

Dev: so can we also commit package lock so we’re all having the same packages as well?

Manager: No, package locks don’t work.

**few days go by**

Manager: GUYS WHY IS THE CODE DEPLOYING TO PRODUCTION NOT WORKING. IT WAS WORKING IN DEV

DEV: **looks at packages**, when the project was built on dev on 9/1 package x was on version 1.1, when it was approved and moved to prod on 9/3 package x was now on version 1.2 which was a change that broke our code.

Manager: CHANGE THE DEPLOYMENT SCRIPTS THEN. MAKE PROD RSYNC NODE_MODULES WITH DEV

Dev: okay

Manager: just trust me, I’ve been doing this for years

Who the fuck put this man in charge.

Story #1: So I took a month of parental leave. And was planning to extend it a little longer to deal with my final exams. I was planning to spend lots of quality time with my wife and newborn son. Little did I know... It turns out that out of 5 OoO weeks I was looking forward I actually had 3 at most. The rest I've spent working remotely as I was insisted to deploy a brand new and poorly tested feature to PROD 2 days before my paternity leave. So I spent 2 weeks debugging things in PROD. Remotely. Needless to say that did suck.

Story #2: After story #1 I've learnt my lesson. This summer I took 3 weeks annual leave to renovate my apartment. I asked to not to be disturbed unless there's an emergency. And an emergency it was. One of our app users had a planned hi-load batch job lasting for 2-3 months. Hundreds of thousands of items had to be created and processed. It turns out the _processing_ algo had some flaws and was acting out. I was called out and asked to assist. I knew this sort of debugging is going to take a lot of my time so this time I put my conditions on the table: I will assist but I'll extend my leave by 1.5 the time I spend working now. They took the deal. Instead of 3 weeks I had 5 weeks of vacation!

I don't care that much about my salary. I prefer to exchange it for my time off hence I didn't ask for compensations.

Bottom line: NEVER EVER underestimate or undersell your time and effort. You are a valuable asset and if the team/client needs you on your day off -- make it count. Your time off is YOUR time. Never forget it.

!rant

We just did a massive update to our prod db environment that would implicate damn near all system in our servers....on a friday.

Luckily for us, our DB is a badass rockstar mfking hero that was planning this shit for a little over a year with the assistance of yours truly as backup following the man's lead...and even then I didn't do SHIT

My boy did great, tested everything and the switch was effortless, fast (considering that it went on during working hours) and painless.

I salute my mfking dude, if i make my own company I am stealing this mfker. Homie speaks in SQL, homie was prolly there when SQL was invented and was already speaking in sql before shit was even set in spec, homie can take a glance at a huge db and already cast his opinion before looking at the design and architecture, homie was Data Science before data science was a thing.

Homie is my man crush on the number one spot putting mfking henry cavill on second place.

Homie wakes up and pisses greatness.

Homie is the man. Hope yall have the same mfking homie as I do

Me passing time on the weekend

Random call from unknown number

Turns out it's the manager

M: hey , how is your weekend going ...

Me: nothing much ... Whatsup ?

M : yeah well , we wanted to push some minor adhoc fixes as some clients wanted it urgently

The Devops folks need developer support . Can you pitch in and monitor

Me : I'm not aware of what changes are going , i don't think i can provide support

M : don't worry it's minor changes , it's already tested in pre prod , you just need to be on call for 30 mins

Me : ugh okay .. guess 1 hr won't hurt

M: thanks 👍🏽

Me: *logs in

*Notices the last merged PR
+ 400 lines , implemented by junior dev and merged by manager

*Wait , how is this a *minor* release...

*Release got triggered already and the CI CD pipeline is in progress

*5 mins later

*Pipeline fails , devops sends email - test coverage below 50%

Manager immediately pitches in ...

M: hey , i see test coverage is down , can you increase it ?

Me: and how do u suppose I do that ?

M : well it's simple just write UTC for the missing lines ... Will it take time ?

Me : * ah shit here we go again

Yeah it will take time , there are around 400 lines , I am not aware of this component all together

Can you ask junior dev to pitch in and write the UTC for this

*Actually junior dev is out on a vacation with his girlfriend

M : well he's out for the weekend , but
as a senior dev , i expect you to have holistic understanding of the codebase and not give excuses ,
this is a priority fix which client are demanding we need this released ASAP

Me : * wait wat ?

---

I ended up being online for next 3 hours figuring out the code change and bumping up the UTC 🤦🏾

I didn't scream.. just told him to jump off of terrace..
What ticked me?! He was a support guy..slowest mofo ever..
I was in the middle of fixing major fuckup on prod, when our VPN to client disconnected. I rushed over to support to ask if it is 'just' an expired session (which he was in charge of renewing but constantly fucked up) or if there is some other problem, so I know how to proceed..do I need to contact our sysadmins, client's support guys etc..
He
started
to
slooooooooowly
explain
I
am
not
the
only
one
with
VPN
problems
...
Was that what I asked you?! // he had an annoying habit of slooooowly talking and explaining unrelated things & personal stuff that bothered him & most of the times he chose the most time sensitive period to drone off..
So I cut him of saying, that others were probably not 'tinkering' with production and that I need this back ASAP, so if he could tell me when the session will be renewed or if there is something else problematic..
He said he will check..I didn't move.. he looked at me insurprise, you want me to check *NOW*?! Yeah, it's urgent.. He proceeded very very veeeery slooooowly, taking the support phone../* he was even eating sandwich during that, so only one hand free, typing one letter at a min */
I was finaly notified that the session expired and that he will fix it soon (meaning in 15-20mins o.O which should not take him more than 5).. and was like 'can I do sth else for you'?! Yeah, do the backflip.. you know the rest..

Today is sprint demo day. As usual I'm only half paying attention since being a Platform Engineer, my work is always technically being "demoed" (shit's running ain't it? There you go, enjoy the EC2 instances.)

One team presents a new thing they built. I'm still half paying attention, half playing Rocket League on another monitor.

Then someone says

"We're storing in prod-db-3"

They have my curiosity.

"Storing x amount of data at y rate"

They now have my attention. I speak up "Do you have a plan to drop data after a certain period of time?"

They don't. I reply "Okay, then your new feature only has about 2 months to live before you exhaust the disk on prod-db-3 and we need to add more storage"

I am asked if we can add more storage preemptively.

"Sure, I say." I then direct my attention to the VP "{VP} I'll make the change request to approve the spend for additional volume on prod-db-3"

VP immediately balks and asks why this wasn't considered before. I calmly reply "I'm not sure. This is the first time I'm learning of this new feature even coming to life. Had anyone consulted with the Platform team we'd have made sure the storage availability was there."

VP asks product guy what happened.

"We didn't think we'd need platform resources for this so we never reached out for anything".

I calmly mute myself, turn my camera back off and go back to Rocket League as the VP goes off about planning and collaboration.

"CT we'll reach out to you next week about getting this all done"

*unmute, camera stays off* "Sounds good" *clears ball*

I guess this is the reason why companies are not transparent. The way everyone is ranting on GitLab and saying this is the end of them, many of us won't even notice if GitLab wasn't this transparent and directly said they removed prod db.

This kind of things happen. Even in companies. They have a very bad backup system and they should learn from it.

I hope they recover and keep up their good work.

One of the biggest reality checks you will run into when starting your first dev related job - and which they don't teach you about in school - is that a lot of the time will be spent working with other people's code, and rewriting it into "your own" is rarely an option.

You might be super into making things, but not everyone manages to maintain that same spark while taking over a 15 year old project with fundamental issues that have to be triaged "for now" because you need a hotfix on this other specific thing out in prod before lunch.

There are no gods now. They left the company years ago and nobody knows why they used the windows registry as a user repo.

TLDR - you shouldn't expect common sense from idiots who have access to databases.

I joined a startup recently. I know startups are not known for their stable architecture, but this was next level stuff.
There is one prod mongodb server.
The db has 300 collections.
200 of those 300 collections are backups/test collections.
25 collections are used to store LOGS!! They decided to store millions of logs in a nosql db because setting up a mysql server requires effort, why do that when you've already set up mongodb. Lol 😂
Each field is indexed separately in the log.
1 collection is of 2 tb and has more than 1 billion records.
Out of the 1 billion records, 1 million records are required, the rest are obsolete. Each field has an index. Apparently the asshole DBA never knew there's something called capped collection or partial indexes.

Trying to get approval to clean up the db since 3 months, but fucking bureaucracy. Extremely high server costs plus every week the db goes down since some idiot runs a query on this mammoth collection. There's one single set of credentials for everything. Everyone from applications to interns use the same creds.
And the asshole DBA left, making me in charge of handling this shit now. I am trying to fix this but am stuck to get approval from business management. Devs like these make me feel sad that they have zero respect for their work and inability to listen to people trying to improve the system.

Going to leave this place really soon. No point in working somewhere where you are expected to show up for 8 hours, irrespective of whether you even switch on your laptop.
Wish me luck folks.

Got an email earlier this week. It went something like this:

"It looks like your team still hasn't delivered the logging and monitoring solution that we asked for. Can you get it done in time for our production deployment next Friday?"

Um, wait, excuse me, WHAT?

1. You never actually asked for the thing you claim we didn't deliver. In fact, when we brought up the fact that you should probably have some monitoring set up for your servers, you said it would be handled entirely by your own team.
2. I HAVE BEEN WORKING ON THIS PROJECT FOR SIX MONTHS WHY DIDN'T YOU TELL ME YOUR DEADLINE UNTIL NOW
3. I won't even have time to start working on this until the Monday after your prod deployment date. Sorrynotsorry.

I really shouldn't be surprised though. This project has been a clusterfuck from the very beginning so this is just par for the course.

Gather around folks, I'll paint you a nice picture based on a true story, back from my sysadmin days. Listen up.
It's about HP and their Solaris 5.4/6 support.

- Yet another Prod Solaris dinosaur crashed
- Connected to console, found a dead system disk; for some reason it was not booting on the remaining redundant disk...
- Logged an HP vendor case. Sev1. SLA for response is 30 minutes, SLA for a fix is <24 hours
- It took them 2 days to respond to our Prod server outage due to failed system disks (responses "we are looking into this" do not count)
- it took another day for them to find an engineer who could attend the server in the DC
- The field engineer came to the DC 4 hours before the agreed time, so he had to wait (DC was 4-5 hours of driving away from HP centre)
- Turns out, he came to the wrong datacentre and was not let in even when the time came

- We had to reschedule for two days later. Prod is still down
- The engg came to the DC on time. He confirmed he had the FRU on him. Looks promising
- He entered the Hall
- He replaced the disk on the Solaris server
- It was the wrong disk he replaced. So now the server is beyond rebuild. It has to be built anew... but only after he comes back and replaces the actually faulty disk.
- He replaced that disk on the wrong Solaris server

Joined a new company / team to work on an iOS app that has 2 different backend environments "Dev" and "Prod". Also being referred to in iOS speak as "Debug" and "Release".

Been trying to get accounts on these backends (no sign up in app, its controlled via another process). Eventually get access to "Dev" for one of the regions, so I load up "Debug" and its not working.

This is odd, so I open the Android app and load "Dev" and it works? I then Notice Android has "Dev", "QA", "Staging" and "Prod" for every region where as iOS only has 2 of these.

So I go back to iOS and find the file for the settings and it has iOS Debug assigned a variable for the backend Dev ... which is actually pointing to QA. Because they use QA to Debug and not Dev.

... confused? join the club

Highlights from my week:

Prod access: Needed it for my last four tickets; just got it approved this week. No longer need it (urgently, anyway). During setup, sysops didn’t sync accounts, and didn’t know how. Left me to figure out the urls on my own. MFA not working.

Work phone: Discovered its MFA is tied to another coworker’s prod credentials. Security just made it work for both instead of fixing it.

My merchant communication ticket: I discovered sysops typo’d my cronjob so my feature hasn’t run since its release, and therefore never alerted merchants. They didn’t want to fix it outside of a standard release. Some yelling convinced them to do it anyway.

AWS ticket: wow I seriously don’t give a crap. Most boring ticket I have ever worked on. Also, the AWS guy said the project might not even be possible, so. Weee, great use of my time.

“Tiny, easy-peasy ticket”: Sounds easy (change a link based on record type). Impossible to test locally, or even view; requires environments I can’t access or deploy to. Specs don’t cover the record type, nor support creating them. Found and patched it anyway.

Completed work: Four of my tickets (two high-priority) have been sitting in code review for over a month now.

Prod release: Release team #2 didn’t release and didn’t bother telling anyone; Release team #1 tried releasing tickets that relied upon it. Good times were had.

QA: Begs for service status page; VP of engineering scoffs at it and says its practically impossible to build. I volunteered. QA cheered; VP ignored me.

Retro: Oops! Scrum master didn’t show up.

Coworker demo: dogshit code that works 1 out of 15 times; didn’t consider UX or user preferences. Today is code-freeze too, so it’s getting released like this. (Feature is using an AI service to rearrange menu options by usage and time of day…)

Micromanager response: “The UX doesn’t matter; our consumers want AI-driven models, and we can say we have delivered on that. It works, and that’s what matters. Good job on delivering!”

Yep.
So, how’s your week going?

Somewhere in a lonely break room
There's a guy starting to realize that eternal hell has been unleashed unto him.
It's two a.m.

It's two a.m.
The boss has gone
I'm sitting here waitin'
This desktop's slow
I am getting tired of fixin' all my coworkers' problems
Yeah there's a bug on the loose
Errors in the code
This is unreadable
Rubber ducky can't help
I cannot debug, my whole life spins into a frenzy

Help I'm slippin' into the programming zone
Git push to the prod
Set up a repo
My hard drive just crashed
All my code is gone
Where am I to go
Now that I've broke my distro
Soon you will come to know
When you need Stack Overflow
Soon you will come to know
When you need Stack Overflow

I'm falling down a spiral
Solution unkown
Disgusting legacy, ugly code
Can't get no connection
Can't get through to commit

Well the night weights heavy
On my confused mind
Where's the error on this line
When the CEO comes
He knows damn well
To keep his distance
And he says

Help I'm slippin' into the programming zone
Git push to the prod
Set up a repo
My hard drive just crashed
All my code is gone
Where am I to go
Now that I've broke my distro
Soon you will come to know
When you need Stack Overflow
Soon you will come to know
When you need Stack Overflow
When you need Stack Overflow

When you need Stack Overflow, a ha
When you need Stack Overflow
When you need Stack Overflow, a ha
When you need Stack Overflow

When you need Stack Overflow, a ha
When you need Stack Overflow
When you need Stack Overflow, a ha
When you need Stack Overflow

When will I fuckin learn that
a) customers lie
b) customers are sloppy
c) customers are wrong
d) customers do not do their work (properly)
e) customers want us to do their (dirty) work
f) possibly all of the freakinly above?! + khm....

They will fuckin aaaalwaaaays say sth is not working after the update..
And I will alwaaaays assume I fucked up something..even if I didn't touch that part of the code/data..
And almost aaaaalways it turns out that the bug they complain about is how the system worked (or didn't work) before the update and/or some fuckup from their side..

Anyhow, I rushed over, grabbed the files went testing in dev..wtf, output is different, mine is ok, theirs is..wtf is that shit?!

Transfer newly built dll to test..same shit as on prod..wtf?! How?!

I assumed they have thing A correctly linked to thing B.. ofc thing A was linked to thing C in their case and in another case (our test) to correct thing B..
I got chillies when grabbing files, that
I should have tripple checked that they didn't fuck up something on the link part, but I just assumed they know what they were doing & that they checked they linked correct files with correct content already, before being pissy that the update fucked up things.. riiiight!! :/

I wanted to find solutions to this fuckup asap so I disregarded my gut feeling..yet again!! Fuuuck!
I've spent too much time trying to find ways to fix a bug that wasn't even a real bug to begin with.. :/

Fuuuuuck!!

So yeah, always treat the customers like they are 3yrs old & have no clue what they are doing & check exactly wtf they were indeed trying to do..it will save you time & nerves..

And note to self: reread this shit daily!! And imprint it in your brain that everything is not always your fault!!

I was asked to look into a site I haven't actively developed since about 3-4 years. It should be a simple side-gig.
I was told this site has been actively developed by the person who came after me, and this person had a few other people help out as well.
The most daunting task in my head was to go through their changes and see why stuff is broken (I was told functionality had been removed, things were changed for the worse, etc etc).
I ssh into the machine and it works. For SOME reason I still have access, which is a good thing since there's literally nobody to ask for access at the moment.
I cd into the project, do a git remote get-url origin to see if they've changed the repo location. Doesn't work. There is no origin. It's "upstream" now. Ok, no biggie. git remote get-url upstream. Repo is still there. Good.
Just to check, see if there's anything untracked with git status. Nothing. Good.
What was the last thing that was worked on? git log --all --decorate --oneline --graph. Wait... Something about the commit message seems familiar. git log. .... This is *my* last commit message. The hell?
I open the repo in the browser, login with some credentials my browser had saved (again, good because I have no clue about the password). Repo hasn't gotten a commit since mine. That can't be right.
Check branches. Oh....Like a dozen new branches. Lots of commits with text that is really not helpful at all. Looks like they were trying to set up a pipeline and testing it out over and over again.
A lot of other changes including the deletion of a database config and schema changes. 0 tests. Doesn't seem like these changes were ever in production.
...
At least I don't have to rack my head trying to understand someone else's code but.... I might just have to throw everything that was done into the garbage. I'm not gonna be the one to push all these changes I don't know about to prod and see what breaks and what doesn't break
.
I feel bad for whoever worked on the codebase after me, because all their changes are now just a waste of time and space that will never be used.

* Yesterday:*
PM: Yes, so, could you please do those changes on this page tomorrow by 3 pm and push to prod?
Me: Yeah sure! Noted :) (task is to "untick" a checkbox in a page's settings on our CMS)

* the next day -- 11 am *
PM: erm yes so please can you do the changes I told you about, it's getting urgent and you didn't start it already and it stresses the hell out of me because today is friday and it needs to be up and running fine for Monday 12pm and you don't work on weekend so I'd like yo-
Me:

So we found an interesting thing at work today...

Prod servers had 300GB+ in locked (deleted) files. Some containers marked them for deletion but we think the containers kept these deleted files around.

300 GB of ‘ghost’ space being used and `du` commands were not helping to find the issue.

This is probably a more common issue than I realize, as I’m on the newer side to Linux. But we got it figured out with:

`lsof / | grep deleted`

I created our login system to be secure and reliable.

One coworker hardcoded the roles a person who is logging in receives and built a backend to just assign roles you want. He pushed this to prod...

Yeah...

These fucking imbecile devs left me a bunch of bugs in a 150k LOC legacy hell...
Too bad it took some 8 months for them to finally surface in prod. God damned...
Technically, this whole codebase is one giant abyss filled to the brim with bugs.

I'm having an existential crisis with this client.

We are spending millions of $s every year to make sure the product's performance is perfect. We are testing various scenarios, fine-tuning PLABs: the environment, application, middleware, infra,... And then we provide our recommendations to the client: "To handle load of XX parallel users focusing on YY, yy and Zy APIs, use <THIS> configuration".

And what the client does?
- take our recommendations and measure the wind speed outside
- if speed is <20m/s and milk hasn't gone bad yet, add 2x more instances of API X
- otherwise add 3xX, 1xY and give more CPUs to Z
- split the setup in half and deploy in 2 completely separate load-balanced prod environments.
- <do other "tweaking">

- bomb our team with questions "why do we have slow RTs?", "why did the env crash?", "why do we have all those errors?", "why has this been overlooked in PLABs?!?"

If you're improvising despite our recommendations, wtf are we doing here???

One day I will crack. Hopefully, not sometime soon.

On chat today.
Dude: can you run a script for me? We don't have permission.
Me: what kind of script? Who wrote it?
Dude: posts screenshot of DML select/update statement he tried to run.
Me: I'm a DBA. We don't run DML for people.
Dude: Oh. Can you give me the password?
Me: examine script and notice he tried to run it on QA DB.
Me: No. We don't memorize passwords, and this is QA; you need to check the password out of the safe. You also need a change ticket to DevOps, and they will run it for you.

At that point I ended the discussion, because running anything in QA or Prod without a change ticket gets you fired. And I like my job. Really annoyed.

Me: “Hey boss, you assigned these things to me that I’m not qualified for and have no experience in. We should really hire someone with the specialized skills in this”

Boss “I agree. It’s a role I desperately think we should have hired for a long time ago”

Me “Ok so about these tickets the-“

Boss “I need you to write up a justification for this role, what kind of work the person would be doing and what budget implications we will incur”

Me “You’re asking me to write a job description for a class of work I’ve already admitted I have no experience or qualifications doing MYSELF?”

Boss “Correct”

Me “and I’m still responsible in the meantime for getting these other tickets done still aren’t I?”

Boss “Yes”

Me “Very well. I’ll email you a recap of this discussion then so we can come back to it later when we start hiring for the role”

(and so my ass is sufficiently covered when I inevitably bring down prod and people start asking why I broke prod)

This was some time ago. A Legendary bug appeared. It worked in the dev environment, but not in the test and production environment.

It had been a week since I was working on the issue. I couldn't pinpoint the problem. We CANNOT change the code that was already there, so we needed to override the code that was written. As I was going at it, something happened.

---
Manager: "Hey, it's working now. What did you do?"
Me: *Very confused because I know I was nowhere close to finding the real source of the problem* Oh, it is? Let me check.
Also me: *Goes and check on the test and prod environment and indeed, it's already working*
Also me to the power of three: *Contemplates on life, the meaning of it, of why I am here, who's going to throw out the trash later, asking myself whether my buddies and I will be drinking tonight, only to realize that I am still on the phone with my manager*
Me again: "Oh wow, it's working."
Manager: "Great job. What were the changes in the code?"
Me: "All I did was put console logs and pushed the changes to test and prod if they were producing the same log results."
Manager: "So there were no changes whatsoever, is that what you mean?"
Me: "Yep. I've no idea why it just suddenly worked."
Manager: "Well, as long as it's working! Just remove those logs and deploy them again to the test and prod environment and add 'Test and prod fix' to the commit comment."
Me: "But what if the problem comes up again? I mean technically we haven't resolved the issue. The only change I made were like 20 lines of console logs! "
Manager: "It's working, isn't it? If it becomes a problem, we'll work it out later."
---

I did as I was told, and Lo and Behold, the problem never occurred again.

Was the system playing a joke on me? The system probably felt sorry for me and thought, "Look at this poor fucker, having such a hard time on a problem he can't even comprehend. That idiotic programmer had so many sleepless nights and yet still couldn't find the solution. Guess I gotta do my job and fix it for him. I'm the only one doing the work around here. Pathetic Homo sapiens!"

Don't get me wrong, I'm glad that it's over but..

What the fuck happened?

School gave me 3 DigitalOcean droplets to try out Kubernetes in the cloud, awesome!

Wrote an Ansible script to not only simply install docker and add users but also add kubernetes, nice!

Oh wait, error?! Well I should've known this wasn't going to be easy... ah well no problem. Let's see... Ansible is cryptic as always, it can't connect to the API server? Is it even running?

Let's ssh to the master, ah nothing is running, great. Let's try out kubeadm init and see what happens, oh gosh, my Docker version has not been validated! No problem, let's just downgrade!

How do I do that? Oh I know, change the version in the role! Wait that version doesn't exit? Let's travel to Docker's website and see what versions exist of docker-ce, oh I see, it needs a subversion, no problem.

Oh that errors too? Wait then what... Oh I need a ~ and a ubuntu and a 0 somewhere, my mistake!

Let's run it again! Fails!

Same ssh process, oh wait...

Oh god no...

Kubernetes requires 2 cores and these things only have 1...

Welp, time to ask the teachers to resize my droplet by a small amount tomorrow, hopefully I'll get a new error!

----------------------------------------------

My adventure so far with Kubernetes. I'm not installing it for any serious/prod reason, just for educational purposes. K8s seems like 'endgame' to me, like one of the 'big guys' that big enterprises use so I'm eager to throw stuff at a droplet and see what happens.

Going further down the rabbit hole tomorrow!

Wish me luck :3

(And yes, I could've figured this all out beforehand with documentation, but this is more fun in my opinion)

So I joined this financial institution back in Nov. Selling themselves as looking for a developer to code micro-services for a Spring based project and deploying on Cloud. I packed my stuff, drove and moved to the big city 3500 km away. New start in life I thought!
Turns out that micro-services code is an old outdated 20 year old JBoss code, that was ported over to Spring 10 years ago, then let to rot and fester into a giant undocumented Spaghetti code. Microservices? Forget about that. And whats worse? This code is responsible for processing thousands of transactions every month and is currently deployed in PROD. Now its your responsibility and now you have to get new features complied on the damn thing. Whats even worse? They made 4 replicas of that project with different functionalities and now you're responsible for all. Ma'am, this project needs serious refactoring, if not a total redesign/build. Nope! Not doing this! Now go work at it.
It took me 2-3 months just to wrap my mind around this thing and implement some form of working unit tests. I have to work on all that code base by myself and deliver all by myself! naturally, I was delayed in my delivery but I finally managed to deliver.
Time for relief I thought! I wont be looking at this for a while. So they assign me the next project: Automate environment sync between PROD and QA server that is manually done so far. Easy beans right? And surely enough, the automation process is simple and straightforward...except it isnt! Why? Because I am not allowed access to the user Ids and 3rd party software used in the sync process. Database and Data WareHouse data manipulation part is same story too. I ask for access and I get denied over and over again. I try to think of workarounds and I managed to do two using jenkins pipeline and local scripts. But those processes that need 3rd party software access? I cannot do anything! How am I supposed to automate job schedule import on autosys when I DONT HAVE ACCESS!! But noo! I must think of plan B! There is no plan B! Rather than thinking of workarounds, how about getting your access privileges right and get it right the first time!!
They pay relatively well but damn, you will lose your sanity as a programmer.
God, oh god, please bless me with a better job soon so I can escape this programming hell hole.
I will never work in finance again. I don't recommend it, unless you're on the tail end of your career and you want something stable & don't give a damn about proper software engineering principles anymore.

On Skype.
[tldr: #muhPrivacy;]

You know, people hiring via Skype.
Gaming, seeing family or having long range relationships.

It's become a decent tool.

Then there is the Skype employee.
Opening a court case because in his work time, evaluating Skype calls - ON FUCKING OBSERVING SKYPING PEOPLE - he has to look at too much flesh (as in porn) for his salary level.
Like : the payment category states that you gotta be classed like 1,2 salary categories higher for such work.

So the first instance did not recognise the employees case, because they said its a state thing, or even higher.
Later instance evaluated the employee was right and decreed Microsoft / the NSA (whomever direct employees they are) to properly categorize their employees.
Therefore cost relatively exploded and an algorithm to detect nudity was built.
Wich is operational way earlier than Skypes TOS renewal mid 2018.
That also bans bad language and auto bans given accounts.

Talking about social credit..
in PROC (or prod, as they're known).
And btw: complaining about Google while posting Christmas gatherings on Instagram.. You get what I mean.

Honestly, I don't recall the sources. It's been a while.

I'd really appreciate a little compendium of this for historical reasons.

They will ask: what has brought us here? What is everyone an ultimate right/left/center/agnostic/religious fascist?

And we'll have it on paper. Or papyrus,.. even stone. As I don't know how far mighty people will go for their fortune.

!dev && rant

Can we talk about banks? Those fuckers! Suposed to keep our money save and be competent... They today gave me the biggest scare of my live and I've run one an update query on a prod db without a where clause! (Okay I knew we had a backup but still pretty scarry moment!)

As a few know, besides being a dev I help to organize a small openair music festival here in Switzerland. The openair was this weekend. Every thing wen't well, until I checked our ebanking account today. There was only 2/3 of the money that should be there. A quick call to the bank and they told me, nope they never received it. As we've thrown it in a secure locker during the night, we didn't receive any receipt or something like that. It took those fuckers 3.5 hours to actually go and check the looker, just to find the remaining money in the corner of it. What the fuck people, can't you open your fucking eyes and not give me a fucking heartatack? I thought you guys are professionals!

Note locker: we get a key to open it from the outside, place our payment during the night, as soon as we close it, it falls inside a vault, so there it's a pay in only system, for lack of a better word, I called it locker.

My heart is still beating like mad, because of them.

On my project the customer has re-signed into a contract several times when they have budget to continue work. The first time they got us to build the system was a huge success story because the team was assembled quickly and we did rapid development. Initialize repo to prod in 1.5 months. The customer asked for the same dev team. Strong dev team, a PM that doesn't take shit, and pure agile. Lets call her don't-take-shit PM.

When the customer re-signed the executive decided that she didn't like don't-take-shit PM. So the project manager gets replaced by play-by-the-rules PM who will comply with stupid requests and micromanagement. He isn't a bad PM but he tries to make everyone happy. The amount of management types executive installs on the project is massive, and development team is cut down in major ways. Customer and executive shit rolls down to the development team and we can't get anything done. The customer starts to lose faith because we can't get traction. They start demanding traditional waterfall/SDLC docs. Which causes more delay in the project.

So the executive decides that the PM can take a fall for it to save face for the company. She moves play-by-the-rules PM to another project. He starts handover to a new PM that has a history of being her pushover. The customer hadn't seen him yet so now we have push-over PM.

Play-by-the-rules PM is finally out of the project and instead of moving to a different account the company decides to "lay him off because there is no work". So basically they made him take the fall for the failure while promising reassignment, and instead let him go. This is so unfair..

Meeting with push-over PM yesterday and he shows us his plan. Identical to play-by-the-rules PM's plan that got him axed.We point that out and show him the docs that were made for it. His face clearly communicates "OH SHIT WHAT DID I SIGN UP FOR?"

Ok I completely lost it... Been in a prod issue all week and basically just said the root cause is bc this team is blind....

Use the forest and trees analogy first but they still didn't seem to see it so I sent them this.

And then the chat went silent....

I got cut from a contracting job yesterday I have 3 weeks left in the contract. They said I worked well with the team, had a great work ethic but didn't think I had strong enough tech skills. In the past this would have hurt my feelings and it does a little but I think my tech skills are fairly high. There were three devs working on 66 apps with no tests, some source control but most of the code in source control was older than code deployed in prod, no automatic builds, people would wait a week before checking in code, others would check in code that would not build. Today the boss asked if I had messed with app pools on the prod iIs server because something was wrong. I said no because never remote into the server. Anyway it is the end of the world and I feel fine.

This is what happened today in our dayli:

Lead: We need to profile our software

Me: You can use the chrome devtools as remote profiler, even on prod, or make HAR files for later inspection.

Lead: Yeah but no that’s just collecting data on every tick, we need something like “has been called x times”

Me: Yeah but you can filt -

Lead: Yeah no, so back when I wrote code in Delphi...

Me: *oh god no not this again*

Lead: ... We could have clicked a button in our IDE and it would wrap the function call with the API call to profile that function ...

Me, to the secret dev group in slack: doesn’t a simple method decorator and node performance api help with that?

The people in the group: We had this topic last Friday all day...

Me: oh well *get’s coffee and ignores lead*

A guy who had the same nationality as the enterprise we were working for was promoted from JUNIOR js developer to UX/UI coordinator for the entire department just because he was 2 year older than me (26 vs 28). Literally he was a junior dev and went to that.

One day he was accusing me of writing a piece of code which led prod to downtime. I was in the office, he was in another country with our manager and technical director next to him and we were talking over internal conference system. I shown git history + his name + his code and he was saying ‘that’s not true!!!’.

I couldn’t resist and I began to yell something like ‘You fucking fuck piece of shit cocksucker...’ for 5 minutes. Since that day i was the god on my project for UI/UX side.

Even now he is in the same place on the same position...

PS: more stories to come with this guy

Central team: No, your team must be doing something wrong. Our pipeline is super-configurable and works for any situation! You just have to read the docs!

Me: Where are the docs?

Central team: Uhh, well, umm... we'll hook you up with a CI/CD coach!

Me: Okay, cool. In the mean time, can you point me at the repo where all the base scripts are?

Central team: Sure, it's here.

Me, some weeks later: Yeah, uhh, the coach can't seem to figure out how to make our Prod deployment work either.

Central team: That's impossible! It's so easy and completely configurable!

Me: Well, okay... but, here's the thing: your pipeline IS pretty "configurable", in the sense that you look for A LOT of variables...

Central team: See! We told you!

Me: ...none of which are actually documented, so they're just about useless to me...

Central team: But, but the coach...

Me: ...couldn't make heads or taisl of it either despite him literally being ON YOUR TEAM...

Central team: Then your project must just be architected wrong!

Me: Well, we're not perfect, so could be...

Central team: Right!

Me: ...but I think it's far more likely that the scripts... you know, the ACTUAL Python scripts the pipeline executes... while it took me DAYS to get through all your levels of abstraction and indirection and, well, BULLSHIT... it turns out they are incredibly NOT flexible. They do one thing, all the time, basically disregarding any flexibility in the pipeline. So, yeah, I'm thinking this is probably one of this "it's you, not me" deals.

Central team: Waaaaahhhhhhhh!!!!!!

Common misconception is that low efficiency technologies (react native, in this case) are completely useless. That is not true at all. Im building a smart home, and yes i want a mobile app, but 90% is just requests to endpoints, so its way simpler to just copy paste some buttons inside a js file than do it properly with native. Would i use it in prod? Fuck no. Would i hack together a bunch of buttons? Fuck yes.

management logic.

dev : calling api on every product scroll is a stupid idea. we shouldn't do it. what if user has 100s of products bought?
mgmt : it isn't a practical scenario. in prod, we checked the data and we rarely have customers with more than 20 products
dev : 😮🤷‍♂️

dev : this is a rare issue that only happens for very old devices from this specific manufacturer. even manufacturers have acknowledged this.
mgmt : we don't care. fix it, as per data this error has been logged for more than 12 times (from 1 user only)
dev : 😮😢

When will X be finished?!

Um..I don't know cuz thing Y is blocking me from doing X implementation and it makes no sense.. I have no idea what is wrong with it and can't tell how long it'll take me to fix, hopefully by the end of today I'll have thing Y fixed enough so I can finish thing X..

Ok, will you be able to update production today?! Just put it on production when you're done with X.

Uuuumm... Yeah... // still need to know how X will behave in cases a, b, c & b+c...

I'm writing this while waiting for abc feedback.. this is going great.. :/

I know prod is not used as prod yet, but still... fuuuuu!!

So last week I really fucked up

I had this new implementation that was supposedly to be integrating smoothly into the rest of the service. It depended on a serialized model made by a data scientist. I test it in local, in QA environment: no problem.

So, Friday, 4pm, I decide to deploy to production. I check once from the app: the service throw an error. Panic attack, my chief is at my desk, we triy to understand what went wrong. I make calls with cUrls: no problem. Everything seems fine. I recheck from the app again: no problem.
We dedice to let it in prod, as the feature work. I go get some beers with the guys, to celebrate the deploy.

Fast-forward the next morning, 11am, my phone ring: it's a colleague of my chief. "Please check Slack, a client is trying to use the feature, it's broken"

FUUUUUUUUUUUUCK!!!

Panic attack again. I go to the computer, check the errors: two types of errors. One I can fix, the other from a missing package on the machine that the data guy used.

Needless to say, I had a fairly good weekend.

Lessons learned:
- make sure Dev, QA and Prod are exactly the same (use Ansible or Container)
- never deploy on a Friday afternoon if you don't have a quick way to revert

There are comments in prod code which say "need to change after POC" or something similar in multiple places.

Also, something that was designed to check something, but the call is made in such a way that it always returns true.

Best part, all the original authors left the company before I joined this team.

Oh so day continued....

My boss just asked me before I left... You know that report we wanted automated, you said you'd get it done by today, is it done yet?

Me: well uh who dragged me into a PROD issue because no one else knows how to investigate... EVEN THOUGH I BUILT AND SHOWED U HOW TO USE THEM... SEVERAL TIMES. (no i didn't say this last but that's what went on in my head).

Oh and I figured out what the issue was...

Few years ago as a junior android dev with couple years of self taught experience of working in startups I submitted a simple android app assignment for a junior android dev role. Assignment had only like 8 requirements so I followed them to the letter. That didn't end well.

App was simple just 3 screens. Login screen with username and password input fields, login button.

Had to call a login endpoint after login button was clicked, redirecting to home screen, calling items endpoint, displaying a list of items and when an item was clicked passing item data and redirect to item details screen.

Needless to say big swinging dick senior was not impressed. UI was not perfect, I forgot to display a loading animation when fetching data, didnt handle back button properly.

I agreed with some points but other comments were clearly just nitpicking: his preferred variable naming conventions, his opinions on architecture that was not up to his standard (official google arch at the time was not up to his standard).

He also was mad that app wasn't prepared for release to googleplay (another out of the ass requirement). Like I would prepare a 3 screen app for prod release that he will forget ever existed after 20min of his review.

Lots more of nitpicking, encapsulation this encapsulation that, omg now hes shocked that there are a few warnings after the project is built.

Regardless my self confidence was destroyed at that point and after few more negative experiences I dropped android dev alltogether for a couple years and switched to game dev.

After game dev ran its course I went back to android dev and found a supportive place where I could grow.

Looking back, they were actually hiring atleast a mid level for a junior position but I was grilled as a senior. The guy literally didnt wrote any single positive thing in that review about my code even tho my senior peers said my project was decent back then, its just that I didnt handle a few edge cases and that's all.

I looked up the guy in linkedin, turns out hes a uni dropout who posts all books that he red about software dev in his education section of his linkedin profile. Found a bunch of other narcissistic stuff on his profile. Guy was a fucking idiot. Even if I worked under him it would have probably sucked.

Learned some important lessons I guess. Always get a second, 3rd and 4th opinion and dont take criticism too seriously. Always check what kind of person is providing feedback.

Rant from a previous gig I just remembered that reignited my fury lol

Suddenly, CSV exports became massively critical to our product's success. "They were always part of the plan, if we don't have them the product is a failure". Plot twist, they were NOT always part of the plan. And our backend is not at all designed for querying the combinations of data you're asking for.

Nevermind we've been entirely focused these last few months on making the new user experience as slick as possible because "our customers want cake, not meat and potatoes". Forget the fact that, in order to meet the deadlines, my team coupled the backend a little too much with the needs of the frontend because otherwise integrations took too long. We NEED fucking CSV exports of everything you can fucking imagine.

No. Fuck you. If you want it, it's gonna take at least 2 engineers and a month, and according to you we only have a few weeks of runway. No, I'm not compromising jack shit, this is the reality we live in. This is going to go nuclear in production if we don't do it right. Either give us the month and bankrupt the company, or fucking drop it.

Or...you could go cry to the frontend team for solutions. And convince them to page through ALL of the data and generate CSVs in the fucking browser. Sure, it sort of works in QA with the miniscule amount of data we have there, but how'd that work out for you in prod?

Jesus fucking christ why are you people such incompetent morons, and how the fuck did you become executives??

We had 1 Android app to be developed for charity org for data collection for ground water level increase competition among villages.

Initial scope was very small & feasible. Around 10 forms with 3-4 fields in each to be developed in 2 months (1 for dev, 1 for testing). There was a prod version which had similar forms with no validations etc.

We had received prod source, which was total junk. No KT was given.

In existing source, spelling mistakes were there in the era of spell/grammar checking tools.

There were rural names of classes, variables in regional language in English letters & that regional language is somewhat known to some developers but even they don't know those rural names' meanings. This costed us at great length in visualizing data flow between entities. Even Google translate wasn't reliable for this language due to low Internet penetration in that language region.

OOP wasn't followed, so at 10 places exact same code exists. If error or bug needed to be fixed it had to be fixed at all those 10 places.

No foreign key relationships was there in database while actually there were logical relations among different entites.

No created, updated timestamps in records at app side to have audit trail.

Small part of that existing source was quite good with Fragments, MVP etc. while other part was ancient Activities with business logic.

We have to support Android 4.0 to 9.0 of many screen sizes & resolutions without any target devices issued to us by the client.

Then Corona lockdown happened & during that suddenly client side professionals became over efficient.

Client started adding requirements like very complex validation which has inter-entity dependencies. Then they started filing bugs from prod version on us.

Let's come to the developers' expertise,
2 developers with 8+ years of experience & they're not knowing how to resolve conflicts in git merge which were created by them only due to not following git best practice for coding like only appending new implementation in existing classes for easy auto merge etc.

They are thinking like handling click events is called development.

They don't want to think about OOP, well structured code. They don't want to re-use code mostly & when they copy paste, they think it's called re-use.

They wanted to follow old school Java development in memory scarce Android app life cycle in end user phone. They don't understand memory leaks, even though it's pin pointed by memory leak detection tools (Leak canary etc.).

Now 3.5 months are over, that competition was called off for this year due to Corona & development is still ongoing.

We are nowhere close to completion even for initial internal QA round.

On top of this, nothing is billable so it's like financial suicide.

Remember whatever said here is only 10% of what is faced.

- An Engineering lead in a half billion dollar company.

Update Table_Name Set Column_Name = ‘New Value’;

Commit;

I did this on prod and my manager started screaming on me.

Is there any issue in it?

Just needing somewhere to let some steam off

Tl;dr: perfectly fine commandline system is replaced by bad ui system because it has a ui.

For a while now we have had a development k8s cluster for the dev team. Using helm as composing framework everything worked perfectly via the console. Being able to quickly test new code to existing apps, and even deploy new (and even third party apps) on a simar-to-production system was a breeze.

Introducing Rancher

We are now required to commit every helm configuration change to a git repository and merge to master (master is used on dev and prod) before even being able to test the the configuration change, as the package is not created until after the merge is completed.

Rolling out new tags now also requires a VCS change as you have to point to the docker image version within a file.

As we now have this awesome new system, the ops didn't see a reason to give us access to kubectl. So the dev team is stuck with a ui, but this should give the dev team more flexibility and independence, and more people from the team can roll releases.

Back to reality: since the new system we have hogged more time from ops than we have done in a while, everyone needs to learn a new unintuitive tool, and the funny thing, only a few people can actually accept VCS changes as it impacts dev and prod. So the entire reason this was done, so it is reachable to more people, is out the window.

FUCK WORDPRESS
FUCK WORDPRESS
FUCK WORDPRESS
FUCK WORDPRESS
FUCK WORDPRESS

Who the fuck thought "Oh lets put all caches in the options table". This is so incredible dumb. The options table has nearly 8GB and is just unusable. The query for the "real options" takes about 3s. To export this fucking shit I had to block the prod server for 20 minutes to get a db dump. I want to womit in the mouth of the fucking idiot who did this.

1 Week ago I took down the development version of an internal demo, leaving only the prod one (temporary space issue). Colleagues were told n+15 times via email, face to face, comments on tickets etc. The prod one has also been live for weeks, and again, they were told when it went up.

This just happened:

Colleague 1: practiseSafeHex can you help me, the demo doesn't seem to be working.

Me: *logs in*, *click around* ... seems fine dude. Are you using the correct URL: <prod-url>.

Colleague 1: let me try again and see.
Colleague 2: practiseSafeHex i'm with Colleague 1, were trying to use the demo, and its not working. Can you have a look, we need it.

Me: I just told him, its fine, I think you have the wrong URL <prod-url>. Can you try again.

Colleague 2: No I have the right one, can you check it.

Me: *does nothing*, yep i've had a look at it, can you try again: <prod-url>.

Colleague 2: Ah its back now, cheers.

They are with a customer now, so I won't say anything, when they get back, one of them is being castrated.

Found this line in code that's been in prod for 10 years

"int kak = object.getValue()"

There is no contextual reason for it to be called kak, which means shit in Afrikaans

A couple of years ago I was working on a fairly large system with a complex (by necessity) access control architecture.

As is usually the case with those projects, it's awkward for developers to repro bugs that have to do with a user's accesses in production when we are not allowed to replicate production data in test, let alone locally.

We had a bug where I ended up making myself a new row in the production database for a thing I could have access to without affecting real data to repro it safely. I identified the bug so I could repro it in dev/test and removed the row and ensured everything worked normally, whew scary.

Have you ever walked into the office one day, and everyone is hunched over in a semicircle around one person's workstation, before one turns around to look at you and says - after a pause - "... ltlian?.."

Turns out I had basically "poisoned the well" with my dummy entity in a way where production now threw 500 for everyone BUT me who had transitive access to this post-non-entity. Due to the scope of the system, it had taken about a day for this to gradually propagate in terms of caching and eventual consistencies; new entities coming in was expected, but not that they disappear.

Luckily I had a decent track record for this to be a one-off. I sometimes think about how I would explain testing in prod and making it faceplant before going home for the day, other than "I assumed it would be fine". I would fire me.

I hate having to deal with our IT service desk. Every time it takes enormous energy to get to the right people and make them understand that no, you are not an idiot, but you actually have a technical issue.

Sure thing they do have a few competent nice folks there too I've gotten to know over time and they indeed have to deal with a ton of dumb non-tech savvy idiots on a daily basis. However, if my job title mentions "software" and "engineer" they should at least assume I'm an idiot in tech. Or something. Every single time I need to open a ticket, even for the simplest "add x to env y", I need to quadruple check that the subject line is moron-friendly because otherwise they would take every chance to respond "nah we can't do that", "that's not us", or "sry that's not allowed". And then I would need to respond, "yes you do:) your slightly more competent colleague just did this for us 2 weeks ago".

Now you might imagine this is on even another level when the problem is complex.

One of our internal apps has been failing because one of the internal APIs managed by a service desk team responds a 500 status code randomly but only when called with a specific internal account managed by another service desk team.
(when I say "managed by", that doesn't mean they maintain it, it just mean they are the only ones who would have access to change something)

Yesterday I spent over a fucking hour writing a super precise essay detailing the issue, proving a million times it's not on our end and that they need to fix it. Now here is an insight to what beautiful "IT service" our service desk provides:

1) ticket gets assigned to a "Connectivity Engineer" lady

2) few hours later she responds and asks me to give her the app and environment IDs and grant her access to those
(naturally everything in my email was ignored including these two IDs)

3) since the app needs to be in prod for the issue, I make a copy isolating the failing part and grant her access to the original "for reference" and the copy to play with

4) few hours later I get an email from the env that some guy called P made changes to the actual app, no changes to the copy
(maybe they immediately fixed the app even though I asked them to only touch the copy)
I also check the env and the live app had been shared with another 2 people giving them editing rights:)

5) another few hours pass and the lady responds that she had been chatting with P (no mention of who tf that guy is) and that P has a suggestion that might work and I should test it, "please see screen shot" for details:

These motherfuckers sent me a fucking screenshot of the env config file where "P has edited a few parameters" that might help. The screenshot had a 16 line part of the config json with a bunch of IDs and Base64 params which HE EDITED LOCALLY.

Again, because I needed a few iterations to realise what I've just witnessed:
These idiots modified some things in the main app (not the copy) for hours. Then came to the conclusion that the config needs some IDs and params updated. They downloaded the config json. Edited it locally. Did not fucking upload it back to the main or test app. Did not test it live. Did not CC in or direct the guy with changes to me. Did not send me the modified config file. Did not even paste the new IDs into the email. But TOOK A FUCKING SCREENSHOT OF THE MODIFIED FILE AND SENT THAT SHIT TO ME. And then had the audacity to ask me to test it when they had access to it and that's literally their fucking job.

I had to compare the fucking screenshot to the live config file and manually type in the changes.

And no, it still doesn't work. And Now I have to get back to them showing it still fails the same way but I just can't deal with these people. Fuck. Was hoping by the time I write it all down it'd be better, and it does feel a bit better, but I still need to get this app fixed. And I can only do it through these... monkeys. I just can't. Talking to these people drains my life energy... I'm just sad.

ideal sprint fallacy.
total days 10 , total hours(excluding breaks ) 8 hrs per day= 80 hrs per dev

code freeze day = day 8, testing+ fixing days : 8,9,10. release day : day 10
so ideal dev time = 7days/56 hr
meetings= - 1hr per day => 49 hrs per dev

- 1 day for planning i.e d1 . so dev time left . 6 days 42 hrs.
-----------
all good planning. now here comes the messups

1. last release took some time. so planning could not happen on d1. all devs are waiting. . devtime = 5 days 35 hrs.

2. during planning:
mgr: hey devx what's the status on task 1?
d: i integrated mock apis. if server has made the apis, i will test them .
mgr : server says the apis are done. whats your guestimate for the task completion?
d : max 1-2 hrs?
m : cool. i assign you 4 hrs for this. now what about task 2?
d : task told to me is done and working . however sub mgr mentioned that a new screen will be added. so that will take time
m : no we probably won't be taking the screen. what's your giestimate?
d : a few more testing on existing features. maybe 1-2 hrs ?
m: cool
another 4 hrs for u. what about task 3?
d : <same story>
m : cool. another 4 hrs for u. so a total of 12 hrs out of 35 hrs? you must be relaxed this sprint.
d : yeah i guess.
m cool.
-------
timelines.
d1: wasted i previous sprint
d2 : sprint planning
d3 : 3+ hrs of meetings, apis for task 1 weren't available sub manager randomly decided that yes we can add another screen but didn't discussed. updates on all 3 tasks : no change in status
d4 : same story. dev apis starts failing so testing comes to halt.
d5 : apis for task1 available . task 3 got additional improvement points from mgr out of random. some prod issue happens which takes 4+ hrs. update on tasks : some more work done on task 3, task 1 and 2 remains same.
d6 : task1 apis are different from mocks. additionally 2 apis start breaking and its come to know thatgrs did not explain the task properly. finally after another 3+ hrs of discussion , we come to some conclusions and resolutions
d7 : prod issue again comes. 4+ hrs goes into it . task 2 and 3 are discussed for new screen additiona that can easily take 2+ days to be created . we agree tot ake 1 and drop 2nd task's changes i finish task 2 new screens in 6 hrs , hoping that finally everything will be fine.

d8 : prod issue again comes, and changes are requested in task 2 and 3

day 9 build finally goes to tester
day 10 first few bugs come with approval for some tasks
day 11(day 1 of new sprint) final build with fixes is shared. new bugs (unrelated to tasks. basically new features disguised as bugs) are raised . we reject and release the build.

day 2 sprint planning
mgr : hey dev x, u had only 12 hrs of work in your plate. why did the build got delayed?

🥲🫡

I hate dev politics...

PM: Hey there is a weird error happening when I upload this file on production, but it works on our test environments.

Me: After looking at this error, I don't find any issues with the code, but this variable is set when the application is first loaded, I bet it wasn't loaded correctly our last deployment and we just need to reload the application.

Senior Dev: We need to output all of the errors and figure out where this error is coming from. Dump out all the errors on everything in production!!

Me: That's dumb... the code works on test... it's not the code.. it's the application.

Senior dev: %$*^$>&÷^> $

Me: Hey I have an idea! If test works... I can go ahead and deploy last week's changes to prod and dump those errors you were talking about!!

Senior Dev: OK

Me: *runs Jenkins job the deploys the new code and restarts the application*

PM: YAY you fixed it!!

Senior Dev: Did you sump put those errors like I said.

Me: Nope didn't touch a thing... I just deployed my irrelevant changes to that error and reloaded the application.

TL;DR

Got Salesforce customised by an external company, took over the code and wanted to cry.

So my company decided to get a new crm for sales with contract generation and a whole lot of fluff. Specs included an easy to implement API to connect to our in-house software and an easy to adjust contract Wizard.

After month of checking various companies the CEO finally settled on...
Drumroll
The cheapest German based company.
Turns out the only part that is based in Germany is the sales department, development is based in Poland which made for interesting times during implementation because of the language barrier that comes with non native english speakers.

We as in our development department for told that we wouldn't need to worry about the solution because it would all be developed and maintained by the new company. As we are fairly small that was more then welcome.

Fast forward to integration day. No docs for the API available, contract Wizard is hardcoded, bunch of errors and inconsistencies. Get tickets for them.
Ok we can deal with this. Just tell the ticket writer he needs to address the problems with the external. Yep, doesn't work. External fixes bugs and introduces then elsewhere.
Fml. Ok I'll take a look into the code. Ugh, Java, I hate that shit but at least I don't need to worry about all the fluff, just the code, so it's ok.
No repository present, code is developed in the Dev environment and pushed to prod. Ugly but works. Code comes with a lot of functions but only one real class called "CommonUtilities" ... even the web API is defined in there. Meanwhile my colleague throws out the need for the API because we will just directly tie to SOQL. I'll let's check what's going on here, nice you reused Lead/Account/Contact for the branch offices of companies as well...
Is not like Salesforce has a bunch of logic tied to these objects...
Nice the required implementation of an automatic holiday import is not even there, just a custom object populated with this year's data...

Tell CEO how badly all of this was handled. Nice note this dumpster fire is our new in-house project because CEO cancelled the contract with the external.

Ok we can deal with this, let's set up a repo, define the CI/CD and get the extensions for vscode. Nice now this all makes sense. Fix all bugs and reimplement the contract Wizard using custom objects that sales can change so the contract is actually easily customizable by a non dev. Implement branches as their own object to avoid opportunities, triggers and the likes to be executed for them.
Took a whole day. Why did these 2000 lines of code that was shipped to us take 3 month to implement?

Lesson learned: never trust an external to just do a good job.
New rule implemented by company to always have a Dev check in regular intervals on projects handled by externals for standards and overall logic

10k down the drain for what amounted to 2 weeks tops (one dev) if we had handled the implementation of Salesforce and all the requirements in-house

An intern made a very bad impression on the first day.

This was before I become a developer. I was working in commercial art sales. One day, I had an appointment to onboard two new interns together.

Intern 1 shows up and I ask her for her signed confidentiality agreement. The boss had sent it out a week before and told me the interns were bringing the signed paperwork on their first day. I see the surprised look on her face and she says she forgot. She’s lucky I had access to another copy. If I didn’t, things could have gotten pretty awkward if I had to contact my boss, who was out of office. If there’s no signed agreement, I can’t onboard her and I’d have to send her home. The appointment was made with intern 1’s availability in mind, so intern 1 could have spent her time coming to the office for nothing and being turned away because of a stupid mistake she made.

While we wait for intern 2 to arrive, I try to engage in small talk with intern 1. I try to get to know her a little better and I ask “are you still in college/university?” She word vomits that she thought she had graduated, but six months later she hadn’t received her diploma and she called the school and they told her her pre-college credits had not transferred, so she’s finishing those credits now.

Oh, intern, you should have just simplified all this to “I’m finishing up my degree” or “yes, I’m still in college.” This is TMI. You don’t want to give out information about yourself that could put you in a bad light. You need to know to be discreet about yourself. You’re 22 years old. It’s really bad judgement to say this to your supervisor (me) and we’ve only known each other for ten minutes. I’m not your friend, I’m your supervisor. Honestly, I thought the explanation didn’t make sense because she would have found out about the credits when she tried to transfer them and when she applied for graduation. I didn’t prod for more details.

I did have to tell my boss about intern 1 forgetting the paperwork. It’s not something the intern would be reprimanded for, but it is something that’s not a good sign. The paperwork had been sent by the boss a week prior. It’s troublesome that an intern would forget to complete an important task that was sent by the boss. This was never a problem with prior interns.

Boss did freak out because boss thought I onboarded intern 1 without intern agreeing to the confidentiality agreement. Boss hadn’t considered an intern would forget the paperwork and didn’t tell me what to do if this did happen. I reassured boss that I had printed a new copy and had intern 1 sign the agreement.

I didn’t say anything about the word vomit. The content was troubling, but I was concerned this would be gossip and I wasn’t out to sabotage the intern.

Forgetting the paperwork and the word vomit were signs the intern wasn’t reliable. Intern had trouble taking direction even when it was written down. She’d do stupid things like invite her boyfriend to the office for hours and let BF sit at the boss’s desk—boss caught her and boss’s office is visible from our public viewing floor, so visitor did see this too. I suspected she might have an diagnosed learning disability.

In the end, intern didn’t ask for a reference letter. Boss said that if intern asked for one in the future, the answer would be no.

Intern 1 is the reason why I don’t want to be in change of interns ever again even though I’m not in art sales anymore.

TL;DR: When picking vendors to outsource work to, vet them really well.

Backstory:
Got a large redesign project that involves rebuilding a website's main navigation (accessibility reasons).
Project is too big just for our dev team to handle with our workload so we got to bring a 3rd party vendor to help us. We do this often so no big deal.
But, this time the twist was Senior Management already had retained hours with a dev shop so they want us to use them for project. Okay...

It begins:
Have our scope / discovery meeting about the changes and our expected DevOps workflow.

Devs work Local and push changes to our Github, that kicks off the build and we test on Dev, then it goes to Staging for more testing & PM review. Once ready we can push to prod, or whenever needed. All is agreed, everyone was happy.

Emailed the vendors' project manager to ask for their devs Github accounts so we can add them to the project. Got no reply for 3 days.

4th day, I get back "Who sets up the Github accounts?"

fuck me. they've never used Github before but in our scope meeting 4 days ago you said Github was fine...??

Whatever, fuck it. I'll make the accounts and add them.

Added 4 devs to the repo and setup new branch. 40min later get an email that they can't setup dev environment now, the dev doesn't know how to setup our CMS locally, "not working for some reason."

So, they ask for permission to develop on our STAGING server.. "because it's already setup"... they want to actively dev on our staging where we get PM/Senior Management approvals?

We have dev, staging, production instances and you want to dev in staging, not dev?... nay nay good sir.

This is whom senior management wants us to use, already paid for via retainer no less. They are a major dev shop and they're useless...

😢😭

Cant wait for today's progress checkup meeting. 😐😐

/rant

Tl;Dr Im the one of the few in my area that sees sftping as the prod service account shouldn't be a deployment process. And the ONLY ONE THAT CARES THAT THIS IS GONNA BREAK A BUNCH OF SHIT AT SOME POINT.

The non tl;dr:

For a whole year I've been trying to convince my area that sshing as the production service account is not the proper way to deploy and/or develop batch code. My area (my team and 3 sister teams) have no concept of using version control for our various Unix components (shell scripts and configuration files) that our CRITICAL for our teams ongoing success. Most develop in a "prodqa like" system and the remainder straight in production. Those that develop straight in prodqa have no "test" deployment so when they ssh files straight to actual production. Our area has no concept of continuous integration and automated build checking. There is no "test cases", no "systems testing" or "regression testing". No gate checks for changing production are enforced. There is a standing "approved" deployment process by the enterprise (my company is Whyyyyyyyyyy bigger than my area ) but no one uses it. In fact idk anyone in my area who knows HOW to deploy using the official deployment method. Yes, there is privileged access management on the service account. Yes the managers gets notified everytime someone accesses the privileged production account. The managers don't see fixing this as a priority. In fact I think I've only talk to ONE other person in my area who truly understands how terrible it is that we have full production change access on a daily basis. Ive brought this up so many times and so many times nothing has been done and I've tried to get it changed yet nothing has happened and I'm just SO FUCKING SICK that no one sees how big of a deal this. I mean, overall I live the area I work in, I love the people, yet this one glaring deficiency causes me so much fucking stress cause it's so fucking simple to fix.
We even have an newer enterprise deployment. Method leveraging a product called "urban code deploy" (ucd) to deploy a git repository. JUST FUCKING GIT WITH THE PROGRAM!!!!..... IT WAS RELEASED FUCKING 12 YEARS AGO......

Please..... Please..... I just want my otherwise normally awesome team to understand the importance and benefits of version control and approved/revertable deployments

Ever have one of those moments where you're running a service you built to update about a decade worth of police records, realize about halfway through that you fucked the loop and you're copying data from the first record onto every other record, and then just really wish that you had checked things better in test before running this on the prod server?

I'm sure the only reason I'm still here is because the audit log contained the original values and I'm good at pulling data out of it.

A loooong time ago...

I've started my first serious job as a developer. I was young yet enthusiastic as well as a kind of a greenhorn. First time working in a business, working with a team full of experienced full-lowered ultra-seniors which were waiting to teach me the everything about software engineering.

Kind of.

Beside one senior which was the team lead as well there were two other devs. One of them was very experienced and a pretty nice guy, I could ask him anytime and he would sit down with me a give me advice. I've learned a lot of him.

Fast forward three months (yes, three months).

I was not that full kind of greenhorn anymore and people started to give me serious tasks. I had some experience in doing deployments and stuff from my other job as a sysadmin before so I was soon known as the "deployment guy", setting up deployments for our projects the right way and monitoring as well as executing them. But as it should be in every good team we had to share our knowledge so one can be on vacation or something and another colleague was able to do the task as well.

So now we come to the other teammate. The one I was not talking about till now. And that for a reason.

He was very nice too and had a couple of years as a dev on his CV, but...yeah...like...

When I switched some production systems to Linux he had to learn something about Linux. Everytime he encountered an error message he turned around and asked me how to fix it. Even. For. The. Simplest. Error. He. Could. Google. Up.

I mean okay, when one's new to a system it's not that easy, but when you have an error message which prints out THE SOLUTION FOR THE ERROR and he asks me how to fix it...excuse me?

This happened over 30 times.
A. Week.

Later on I had to introduce him to the deployment workflow for a project, so he could eventually deploy the staging environment and the production environment by hisself.

I introduced him. Not for 10 minutes. I explained him the whole workflow and the very main techniques and tools used for like two hours. Every then and when I stopped and asked him if he had any questions. He had'nt! Wonderful!

Haha. Oh no.

So he had to do his first production deployment. I sat by his side to monitor everything. He did well. One or two questions but he did well.

The same when he did his second prod deploy. Everythings fine.

And then. It. Frikkin. Begins.

I was working on the project, did some changes to the code. Okay, deploy it to dev, time for testing.

Hm.

Error checking out git. Okay, awkward. Got to investigate...

On the dev server were some files changed. Strange. The repo was all up to date. But these changes seemed newer because they were fixing at least one bug I was working on.

This doubles the strangeness.

I want over to my colleague's desk.

I asked him about any recent changes to the codebase.

"Yeah, there was a bug you were working on right? But the ticket was open like two days so I thought I'll fix it"

What the Heck dude, this bug was not critical at all and I had other tasks which were more important. Okay, but what about the changed files?

"Oh yeah, I could not remember the exact deployment steps (hint from the author: I wrote them down into our internal Wiki, he wrote them done by hisself when introducing him and after all it's two frikkin commands), so I uploaded them via FTP"

"Uhm... that's not how we do it buddy. We have to follow the procedure to avoid..."

"The boss said it was fine so I uploaded the changes directly to the production servers. It's so much easier via FTP and not this deployment crap, sorry to say that"

You. Did. What?

I could not resist and asked the boss about this. But this had not Effect at all, was the long-time best-buddy-schmuddy-friend of the boss colleague's father.

So in the end I sat there reverting, committing and deploying.

Yep

It's soooo much harder this deployment crap.

Years later, a long time after I quit the job and moved to another company, I get to know that the colleague now is responsible for technical project management.

Hm.

Project Management.

Karma's a bitch, right?

I'm still studying computer science/programming, I still have one year to do in order to graduate (Master). I am in a work study program so I'm working for a company half of the time, and I'm studying the other half. It is important to mention that I am the only web developer of the company

When I arrived in the company 9 months ago, I was given a Vue project which had been developed by a trainee a few weeks before my arrival and I was asked to correct a few things, it was mostly about css. Then, I was ask to add a few functionalities, nothing really hard to code, and we were supposed to test the solution in a staging environment, and if everything was ok, deploy it to prod.

However, the more I did what I was asked, the more functionalities I had to implement, until I reached a point where I had to modify the API, create new routes, etc. I'm not complaining about that, that's my job and I like it. But the solution was supposed to be ready when I arrived, it was also supposed to be tested and deployed.

The problem is, the person emitting these demands (let's call him guy X) is not from the IT service, it's a future user of the website in the admin side. The demands kept going and going and going because, according to him, the solution was not in a good enough state to be deployed, it missed too many (un)necessary features. It kept going for a few months.

The best is yet to come though : guy X was obviously a superior, and HIS superior started putting pressure on me through mails, saying the app was already supposed to be in production and he was implying that I wasn't working fast enough. Luckily, my IT supervisor was aware of what was going on and knew I obviously wasn't to blame.

In the end, the solution was eagerly deployed in production, didn't go through the staging environment and was opened to the users. Now, guy X receives complaints because none of what I did was tested (it was by me, but I wasn't going to test every single little thing because I didn't have time). Some users couldn't connect or use this or that feature and I am literally drowning in mails, all from guy X, asking me to correct things because users are blocked and it's time consuming for him to do some of the things the website was doing manually.

We are here now just because things have been done in a rush, I'm still working on it and trying to fix prod problems and it's pissing me off because we HAVE a staging environment that was supposed to prevent me from working against the clock.

On a final note, what's funny is that the code I'm modifying, the pre-existing one needs to be refactored because bits and pieces are repeated sometimes 5 times where it should have been externalized and imported from another file. But I don't know when and if I will ever be able to do that.

I could have given more context but it's 4am and I'm kinda tired, sorry if I'm not clear or anything. That's my first rant

Was recently asked if our team wanted to change from Java to Kotlin so I looked over the feature lists but don't see much that's impressive.

One argument was less boilerplate code but I think they're are libraries like Lomboq that can write that stuff for you.

The other was smart type casting and type inference and to that I'm like this sounds like giving monkeys a hammer.

Our JS codebase looks like shit... And our Java app just crashed in prod.

Getting a ton of text messages this morning and thankfully I'm on vacation still...

The error is not caused by NPE... but how some or logic spammed the db..

A new language isn't going to fix this.... And a team that writes this sort of shit logic clearly shows they are incapable of learning a new one probably...

They are already script kiddies... Don't need them to become babies...

Until today, I had assumed deploying stuff to prod would NOT be one of my responsabilities in this company. Apparently that's not the case.

Had to deploy my code and pray it didn't break anything. Why is this a big deal at all?

Well you see, there is no repository. At all. No git, no svn, not even duplicate folders. No tests, no pipeline. Just a bunch of CPanels.

Had to manually copy files and folders from the development site to the production site and partially copy a database. "Just drag and drop" were the instructions I was given.

As if using CakePHP2, PHP5 and having to parse fucking Excel files wasn't bad enough, now I have to deal with one of the worst ways to deploy code.

Fuck it, I'm switching on the looking-for-job flag on linkedin.

!rant

I've always been wondering why do tech companies need everyone to have a strong grasp of algos and data structures?

I've been coding most of my life but didn't get a CS degree so ended up in IT but I kind of want to get into a tech company as my thinking is the quality of code much higher (I spend a lot of time cleaning up other people's code and prod issues over the years...), I've been learning Algo/DS but when I see those technical questions on CareerCup, I go WTF.... it's this the kind of problems you guys do every day?

I just joined a new company.
Their CI pipeline is to give root access to staging and prod servers to every developer in the company and the manually git pull each repository (8-10 repos per server) and manually set nginx and port configurations. And if this wasn't enough, all of the 30 sites they have are basically the same site and they make the changes manually for each tenant (no env file). I'm amazed at how hard some people are willing to work.

Who the fuck invented the glorified pile of shit people call laravel? Is this actually used in PROD for anything else than load testing a monitoring server by creating loads of error messages?
OOP exists for a reason, not to create bazillions of classes with static methods.
Dump that shit ffs!

In a previous job, I was trying to organize a common repository with our shitty business partner so we could both be able to contribute our part so our work would not overlap. Not like they cared anyways.

One thing I quickly noticed is those fuckers would just straight up commit untested changes on master and cripples our whole testing and prod deployment at times because we were depending on a shitty IoT service they provided us onto which we had no control whatsoever.

I told my boss, who was often complaining about them being unreliable in the first place, I would simply restrict them from merging and commiting to develop or to master without my approval. We cannot keep working like this.

He told me that we could not impose on them our work practices and that I should not try to piss them off. To be diplomatic.

I politely and professionally refused to do it, but he did change his mind in the end. He and I left not too long after. I guess he felt obliged to respond that having his job at stake but you cannot condone voluntarily shitty work.

saw an SSIS project once used for basically string replacement and file tasks. Asked manager permission to rewrite this Proof of Concept to something more suitable language.
manager: you can't but you have to support it.
me: but i tought it's a PoC
manager: yes it is
me: so what to support?
manager: it's been sold to a customer and in production for 6 months now
So it's a PoC which is sold and moved to prod...
I hate sales and management douches sometimes... like 98% of the time....
(already left the company)

I am forced to work with a client's notoriously slow SOAP api. Slow in this case is 1.5-2s per request.
The api is structured rather... creatively... at the same time. So we have to bombard it with thousands of requests to build our data base with historical SOAP data. Also the data sometimes is a couple of hours late, giving a flat line (all values at 0) until retroactively fixing the output for the same requests.
So to fill one dev data base with a year's worth of historical data (nice to have when testing a dashboard application) we hammer the api with ~20k requests (~1 million if we want to be thorough).
Best thing about that: There is no staging/test api and the prod api seems not to handle lots of requests at the same time very well...
Latest thought: Maybe we could put a varnish cache in front of the SOAP for testing. Better have wrong data, than nothing at all and we don't kill the prod clients every time we ramp up a new instance.
Also that would dramatically decrease the 4.2 hours of data pumping to about 7 minutes after the first run.

This condition over here which is never supposed to be true sometimes becomes true, no one know how or why, just change it manually in prod whenever that happens...

Finally made my node production server stable enough that I could focus on writing tests*. I start by setting up docker, mocking cognito, preparing the database and everything. Reading up on Node test suites and following a short tut to set up my first unit test. Didn't go smoothly, but it's local and there are no deadlines so who cares. 4 days later, first assert.equal(1+1, 2) passes and I'm happy.

I start writing all sorts of tests, installing everything required into "devDependancies," and getting the joy of having some tests pass on first try with all asserts set up, feels good!

I decide to make a small update to production, so I add a test, run and see it fail, implement the feature, re-run and, it passes!

I push the feature to develop, test it, and it works as intended. Merge that to master and subsequently to one of my ec2 production servers**, and lo and behold, production server is on a bootloop claiming it "Cannot find module `graphql`". But how? I didn't change any production dependencies, and my package lock json is committed so wth?

I google the issue, but can't find anything relevant. The only thing that I could guess was that some dependencies (including graphql) were referenced*** in both, prod and dev, and were omitted when installed on a prod NODE_ENV, but googling that specific issue yielded no results, and I would have thought npm would be clever enough to see that and would always install those dependencies (spoiler: it didn't for me).

With reduced production capacity (having one server down) I decided to npm uninstall all dev dependencies anyway and see what happens. Aaaaand it works.....

So now I have a working production server, but broken local tests, and I'm not sure why npm is behaving like this...

* Yes I see the irony.
** No staging because $$$, also this is a personal project.
*** I am not directly referencing the same thing twice, it's probably a subdependency somewhere.

Two (2) senior developers and one (1) senior tester left our team and I am left with two (2) Java legacy applications that are hard to maintain. Here is a list of things I hate about these old webapps (let's call them app A and B):

1. App A depends on 80% web services. If one web service for a product or warehouse goes down, work flow is impeded while prod support team checks with the core services team for repair

2. App B is a maven project with multiple modules dependent on libraries that are dependent on company's internal libraries. So if we want to upgrade to OpenJdk 9 and up, the project will definitely produce a lot of errors due to deprecated/unsupported codes

3. App A is dependent on Tibco and I have no experience on that

4. App B's continuous integration build tool is Jenkins and the jobs that build it has a shell script that wasn't updated during the tech upgrade enhancement. The previous developer who did the knowledge transfer to me didn't tell me about this (it should be considered a defect on her part but she already resigned)

5. App A when loaded in eclipse IDE is a pain to work with since it is only allowed to build a war file using ant. I have to lookup in quick search instead of calling shortcuts (call hierarchy) because the project wasn't compiled via eclipse.

6. It's impossible to debug app A because of #5

7. Both applications have high priority and complex enhancements and I have no other teammates to help me

8. You never know what else can go wrong anytime

While addressing a Senior Dev's (SD) query from another team.

SD: why is this field mandatory? Can't it be just optional? Any other work around?
Me: Is your code changes already pushed in Devo? In that case, we provide a value which will work since you are not concerned about it.
SD: Yes. It's pushed till production. And, I want to test changes in Prod.
Me: (shared some codes) and explained that this feature for testing is only available in Devo.
SD: I know that. (Shared me a ticket) I want this field to be optional. That's it.
Me: (read the entire ticket. Didn't find anything related their) Told him, I will discuss with team. And meanwhile, for Devo, you can use this value.

Next morning, I accidentally came over some other ticket raised by him only which had the correct doubts regarding request to support this field in production
Now, I don't know why did he share a wrong ticket with me.
And, how will it even help him if that field was even optional.
THAT JUST WONT WORK IN PRODUCTION.

I will discuss with my team and see what can happen.

If a team uses multiple languages and stacks (Have, JS, Python) do you think it's better to have everyone use/constantly switch between them or have dedicated developers for each language (ie. 80% main, 20% others)?

--END QUESTION, ANSWER NOW BEFOREHAND CONTINUING---
---BEGIN RANT---

My boss likes keeping the team "will rounded" so everyone does everything. One month in working in Java, the next with Node web apps. When I switch to node, it takes like a week of "wtf doesn't it work.... what changed, is it a big?" And usually end it"oh right I remember I need to ..."

And also always... "How the fuck do I write tests in {some reading framework} again?"

So feels like everyone is just a generalist and no one is a master/has time to develop mastery. I don't know if it's just me (1/3 Senior developers on the team that has to do everything) or if I'm the only one that complains... Not that it makes a difference... (Only option to really be heard is to resign but I need to somewhere else to work and finding one is hard for personal reasons)

And well this is the biggest reason I would leave the team. No time for mastery, no standardization/shared knowledge (everyone does their own thing but probably not well and no time for testing or documentation; how the fuck does whatever you wrote work, how do we use it, what the fuck did you put in prod that does ... And where the fuck did you put it cuz it's not in ANY of our repos).

I always feel one day soon it will come crashing down and I can say "I told you so" but will then it's too late and I'll be there one cleaning it up... Again

Well, this happens time to time...

I'm freelancing as a backend guy. I like to take care of all infrastructure before really starting to build anything, this mostly includes dev/staging/prod environments with some linear promotion strategy. So.. I did this API. Still on staging, proceeding with the development as planned, everything goes according the timeline.

And then.. this happens... At some point PM told frontend guy that it's time for production (without notifying me), so the frontend guy does what "anyone" would do in this case - tells PM to create DNS record for production to point to staging app.

Time passes, I'm still unaware of this. But I'm starting to see some quality entries in the DB, not the usual QA crap. I write to them that they're doing good job and continue with my tasks.

One of the tasks required some major DB change. I could've written migrations script, but since we're not in "production" yet, I just wipe the DB and recreate schema as I need it.

In 10 minutes the furious PM starts shouting that "production" is down and I need to fix it ASAP.

I'm lost, I'm asking questions, I'm slowly understanding what's happening...

So I want to grab some coffee, sat back down, wrote politely that they suck, added a finger emoji and terminated the contract.

Felt like the right thing to do as I definitely don't want to continue within the same "team".

is it necessary to have cherry picking a part of git branching/release process?

we have 3 branches : develop, release and master.

currently every dev works on feature as follows : they make a branch out of develop, write code, raise pr against develop, get it reviewed and merge back to develop. later the release feature list is generated, and we cherry pick all the release related commits to release branch, and make a prod build out of release branch. finally, the code is moved to master and rags are generated accordingly.

so the major issue with this process is feature blocking. as of now, i have identified 4 scenarios where a feature should not be released :

1. parallel team blocker : say i created a feature x for android that is supposed to go in release 1.2.1 . i got it merged to develop and it will be cherry picked to release on relase day. but on release day it is observed that feature x was not completed by the ios dev and therefore we cannot ship it for android alone.

2. backend blocker : same as above scenario, but instead of ios, this time its the backend which hasn't beem created for the feature x

3. qa blocker : when we create a feature and merge it to develop, we keep on giving builds from develop branch adter every few days. however it could be possible that qa are not able to test it all and on release day, will declare thaf these features cannot be tested and should not be moved to release

4. pm blocker: basically a pm will add all the tickets for sprint in the jira board. but which tickets should be released are decided at the very late days of sprint. so a lot of tasks get merged to develop which are not supposed to go.

so there's the problem. cherry picking is being a major part of release process and i am not liking it. we do squash and merges, so cherry picking is relatively easy, but it still feels a lot riskier.

for 1 and 2 , we sometimes do mute releases : put code in release but comment out all the activation code blocks . but if something is not qa tested or rejected by pm, we can't do a mute release.

what do you folks suggest?

Now this bank is testing in Prod 😂

So one of my colleague does not put fucking merge and commit rules in his piece of shit project, and somehow I gotta help maintaining it. I wish I was admin to put some order in this bullshit.

Some other dumb fuck decided to copy paste a slack thread containing code and commit it in the master branch because he does not understand what the fuck he is doing. Prod is now going to shit lol.

This legit just gave me cancer.

i once explored our codebase and copied every line of commented out code into a text file. that file is over 350 lines long.

Ok so first technical blog post/rant cuz I just reduced a lot of debt... Prolly gonna put this in an email to my boss (he says progress improvement is now a priority but there are some problems as listed below):

So last week, I spent a lot of time investigating db logs manually to figure out a prod issue: tiring, time consuming, and not very effective.

This week I built an app. It took a few days but having the time to design it correctly, it is very powerful.

So in order to really do process improvement, you need to have: dedicated the time, the problem solving mindset (the right people), and the understanding of what the problem is and why so you can build a good solution (time and people).

This is a true story when I was working as a application technician a couple of years ago!

Before I started working there, they had a couple of incidents with ppl with less knowledge accidently deleted stuffs in prod databases, and only a handfull of ppl get the full access to them. I started working in this team, and one day I was asked to run a snippet in one of the prod databases from a co-worker with less privilege.

Loged in, run the snippet and the server STALLED for a couple of minutes! When the snippet was finished I looked at the screen and saw the output "1724217 rows deleted". The fun part here, was that we went to a coffee break right after this, and after a couple of minutes we started to hear ppl mumbling that the network was slow as f*ck, servers didn't respond etc etc.

Well, I responded that I got a snippet that deleted 1724217 row in a table and we ran back to our computers and started to work backwards to solve this.

The best part in this story is that:

* Was not my fault! Even thou I was the one that executed it.
* The tables was deleted from a live prod server that was not heavily used!
* I asked for a life line for us in this team, that we needed a prevented output so we can "match" the actual output after we ran it in prod from the ones from the developers!

Even thou it was not my fault, this is the worst mess up I have done working in IT over 10 years. O_o

Today, I'm making a revolutionary change to our code base. I'm finally deprecating a script that lived in this goddamn repo for way too long. There's about 10 copies of the same script in 10 different directories. The script copies all the code into a "dev" folder and then runs "sed s/prod/dev/g" on all the files, and then overwites a bunch of it with some files suffixed with ".dev". Finally, after fighting the so-called architects, the devs and everyone else that seems to have gotten used to the pain this cursed dumpster fire script has caused us, my merge request is now open and ready to go to get rid of this insanity. Now we won't have to deal with as many "surprises" that happen every goddamn time we deploy to production, overwriting all our hard work by accident, and relieve some of my OCD of having the same script in 10 different places in the repository.

After a year of using mongo in prod and personal projects I have realised some things. Its really nice early on the project, especially when there are changing requirements and for small projects or proof of concepts.
But when you make commercial software things tend to get more complex and relational. Stakeholders want reporting and even a report building which a document store isn't the best at.
With most projects projects when they get big things get relational and this becomes more and more expensive to handle in terms of compute power and developer time.
I don't doubt mongo has its place, maybe as an secondary specialised data store or if the project is inherently document oriented.
Blog over.

- Am a junior dev in an awesome team & exciting project after my apprenticeship and while having just started my part time studies
- Have restructure in company so I land in an other value stream
- Get laid off by new value stream 6 months later (now) because they have a serious budget cut
- Take time to come to terms with situation. I could finally work more on my side projects or focus a bit more on my studies. Hey actually I will have 5 months time to look for something while being paid by the company and they help me brush up my CV. Pretty neat!
- Now my former boss wants me back because of my experience in the project, but only as a production support and not as dev (because budget and they're bleeding with tickets)

Not sure if I should take the offer as it feels safe to have an income and the team is cool. However, it feels a bit like a degradation as prod support sucks in that project and I'd like to code (which wouldn't be possible then).
And as this is still my first company I'm working in, it would make sense to look for something else...

Grrr need to sleep about it... Decision-making isn't exactly my strength.

Long post, TLDR: Given a large team building large enterprise apps with many parts (mini-projects/processes), how do you reduce the bus-factor and the # of Brent's (Phoenix Project)?

# The detailed version #

We have a lot of people making changes, building in new processes to support new flows or changes in the requirements and data.

But we also have to support these except when it gets into Production there is little information to quickly understand:
- how it works
- what it does/supposed to do
- what the inputs and dependencies are

So often times, if there's an issue, I have to reverse engineer whatever logic I can find out of a huge mess.

I guess the saying goes: the only people that know how it works is whoever wrote it and God.

I'm a senior dev but i spend a lot of time digging thru source code and PROD issues to figure out why ... is broken and how to maybe fix it.

I think in Agile there's supposed to be artifacts during development but never seen em.

Personally whenever i work on a new project, I write down notes and create design diagrams so i can confirm things and have easy to use references while working.

I don't think anyone else does that. And afterwards, I don't have anywhere to put it/share it. There is no central repo for this stuff other than our Wiki but for the most part, is like a dumping ground. You have to dig for information and hoping there's something useful.

And when people leave, information is lost forever and well... we hire a lot of monkeys... so again I feel a lot of times i m trying to recover information from a corrupted hard drive...

The only way real information is transferred is thru word of mouth, special knowledge transfer sessions.

Ideally I would like anything that goes into PROD to have design docs as well as usage instructions in order for anyone to be able to quickly pick it up as needed but I'm not sure if that's realistic.

Even unit tests don't seem to help much as they just test specific functions but don't give much detail about how a whole process is supposed to work.

I'm so down that i didn't see the red circle with the cross to add a rant...

Why is that? Because several month ago i began a job with all my motivation & optimistic mood.

I was so glad that a compagny payed attention to my profil that it was the best day of my life. I wanted to improve myself and learn!

At this point i did'nt know yet that i will began to work with assholes.

In this fantastic world, designers are kings and you have to do magic to adapt one of their stupid static design on web.
Because the suprem king is the client and designs are validated.
And don't even ask for an fonctionel analysis they will laught at you!

I did everything that i could do to make things work, fast and good. One time i managed the end of a project all by my self (like said once Celine Dion). I maked the work of my colegue who was on holiday because she left with unfinished work. She said to me "it's easy". She liked to say that i maked lost her time because of my questions and that i need to search the answer by myself & work more and more and more. So i worked, day & night because i didn't have enough time. And other thing is that some persons loved to say "if you don't do that someone will need to do that for you"!

I'm a junior developer and i had acces to staging and prod environements and crashed it both several time... I needed to develope in one year the experience of a senior developer.
Every thing is my fault because i need to pay attention to things that i ignore.

Today i'm not glad, i learned a few things but can't remembered it because things went o fast for me and i can't memorized everithing. All i know is that i'm just happy to still be able to get out from bed.

This is an actual transcript...
Since it's way too long for the normal 5000 characters, hence splitting it up...

Infra Guy: mr Dev, could you please give some rational for update of jjb?
Dev: sparse checkout support is missing
Infra Guy: is this support mandatory to achive whatever you trying to do?
Dev: yes
Infra Guy: u trying to get set of specific folder for set of specific components?
Dev: yes
Infra Guy: bash script with cp or mv will not work for you?
Dev: no
Infra Guy: ?
Dev: when you have already present functionality why reinvent the wheel
Dev: jenkins has support for it
Dev: the jjb is the bottle neck
Infra Guy: getting this functionality onto our infra would have some implications
Dev: why should I write bash script if jenkins allows me to do that
Dev: what implications ??
Infra Guy: will you commit to solve all the issues caused by new jjb?
Dev: you show me the implications first
Infra Guy: like a year ago i have tried to get new jjb <commit_url>
Infra Guy: no, the implications is a grey area
Infra Guy: i cant show all of them and they may hit like in week or eve month
Dev: then why was it not tackled
Dev: and why was it kept like that
Infra Guy: few jobs got broken on something
Dev: it will crop up some time later
Dev: if jobs get broken because of syntax
Dev: then jobs can be fixed
Dev: is it not ???
Infra Guy: ofc
Infra Guy: its just a question who will fix them
Dev: follow the syntax and follow the guidelines
Dev: put up a test server and try and lets see
Dev: you have a dev server
Dev: why not try on that one and see what all jobs fails
Dev: and why they fail
Dev: rather than saying it will fail and who will fix
Dev: let them fail and then lets find why
Dev: I manually define a job
Dev: I get it done
Infra Guy: i dont think we have test server which have the same workload and same attention as our prod
Dev: unless you test how would you know ??
Dev: and just saying that it broke one with a version hence I wont do it
Infra Guy: and im not sure if thats fair for us to deal with implication of upgrading of the major components just cause bash script is not good enough for u
Dev: its pretty bad
Infra Guy: i do agree
Infra TL Guy: Dev, what Infra Guy is saying is that its not possible to upgrade without downtime
Infra Guy: no
Dev: how long a downtime are we looking at ??
Infra Guy: im saying that after this upgrade we will have deal with consequences for long time
Infra Guy-2: No this is not testing the upgrade is the huge effort as we dont have dev resources to handle each job to run
Dev: if your jjb compiles all the yaml without error
Dev: I am not sure what consequences are we talking of
Infra Guy: so you think there will be no consequences, right?
Dev: unless you take the plunge will you know ??
Dev: you have a dev server running at port 9000
Infra Guy: this servers runs nothing
Dev: that is good
Dev: there you can take the risk
Infra Guy: and the fack we have managed to put something onto api doesnt mean it works
Dev: what API ?
Infra Guy: jenkins api
Infra Guy: hmmm
Dev: what have you put on Jenkins API ??
Infra Guy: (
Dev: jjb is a CLI
Infra Guy: ((
Dev: is what I understand
Dev: not a Jenkins API
Infra Guy: (((
Dev: (((((
Infra Guy: jjb build xmls and push them onto api
Infra Guy: and its doent matter
Dev: so you mean to say upgrading a CLI is goig to upgrade your core jenkisn API
Dev: give me a break
Infra Guy: the matter is that even if have managed to build something and put it onto api
Infra Guy: doesnt mean it will work
Dev: the API consumes the xml file and creates a job
Infra Guy: right
Dev: if it confirms to the options which it understands
Dev: then everything will work
Dev: I am actually not getting your point Infra Guy
Infra Guy: i do agree mr Dev
Dev: we are beating around the bush
Infra Guy: just want to be sure that if this upgrade will break something
Infra Guy: we will have a person who will fix it
Dev: that is what CICD is supposed to let me know with valid reasons
Dev: why can't that upgrade be done
Infra Guy: it can be done
Infra Guy: i even have commit in place

This place I’m consulting at just had a new Directory of Change, first policy she made for IT is mandatory 3 working days wait for any release in Test and 2 weeks for Prod. That includes application config value update in Test database. We think she might have misunderstood her job title Ms. Director of No Change..

We have our product's second release this week. And my manager was like... I don't want anyone to be absent in the coming days.
But he concluded the pep talk with an announcement that he is off work for the next 5 days.
I think he meant - idgaf even if our PROD burns down before I return !! -_-

This is a continuation of my previous rant about admob being not very informative when it comes to invalid traffic and the resulting restriction in ad delivery.

I then wanted to use admob mediation to hang in facebook ads. My app is written with Xamarin.Forms.

So first I needed to make some facebook configuration - create an account, let my app review, create some ad placements and other shit. I came to the point where I had to put in a link to my privacy policy and the link could not be accepted due to some SSL fuckup -.-'

I then found out that there is an issue with my SSL Chain. With the help of whatsmychaincert.com I solved that issue. Little side note here: I have limited knowledge of that stuff and my cousin helped me set up my homepage so I had no idea what I was doing. Did a snapshot and luckily I did not needed that as everything worked :)
This took me around half an hour just so I can paste the fucking link to activate my app in facebook developer portal.

After that I made the whole mediation configuration shit - not an issue as google documented this quite well but it took some time.

Now comes the shitty part. To use admob mediation you need adapters to the other ad network. I found a nuget package with exactly what I needed just to find out that it is outdated. So I pulled the repo and saw that this thing is an aar binding library. Never did that stuff so I read some docs again. Updated the package and consumed it in my app.

The google docs then said "Use this mediation test shit to check if you did everything correct before going prod" - aar binding nr. 2 (but I am now familiar with that :P). This thing then told me that facebook ads could not be loaded because the SDK version is outdated -.-' SDK version comes from another nuget package which is referenced by the first aar thingie. I tracked that thing back to a repo where I found out that they are indeed totally behind. So I downloaded the aar, made a binding lib and bound that to my first aar binding lib as that depends on this.

Put that all back in my app - tested mediation and fucking finally after 6 hours everything comes together! all lights are green and things work.

Sorry if this is not quite a rant but it was quite a journey and I just had to share it.

So following my previous post, the issue happened again. And actually for background what I've been telling my boss, for years, we need ELK setup and integrated into all our APIs ASAP.

I think it's a punishable crime if any program is released into prod at a tech company with out real time logging/monitoring built in?

So issue still happening, user sent us the request details. So now need to find the actual now that handles the request and look into it's logs to see the details.

Now he's doing it the hard way.... Just finished took 1hr, and the best answer her can come up with is "I think .... Maybe ..."

And if course this is based on infinite data. He stopped after finding a "probably cause"

I have a script that is like promotion ELK, downloads all looks and parsed then so I can run queries to pinpoint the exact call and which log it's in. And can see what's happening around it.

We'll see what my way find but definitely does not take more than 1hr...

Loading data maybe but that's because it needs to download the logs and parse them all...

On a side note, guess I'm Beck on devrant as I have something to rant about. Though it's the same something that I was wanting about years ago... Monkeys...

Any one running Symfony on a Docker container in production? I currently try to migrate our dev env to a docker compose setup (from a "monolith" vagrant vm). I'm atually not stuck at a Symfony specific thing, but on a, I guess Docker specific one(?), The issue is, I need to read and write with two users to one folder (in my case the /application/var/cache folder). Since I mount my whole code into the docker container (to use an IDE on the local files), I've got a volume (not mounted to the outside world) for that folder. (As far, as good). Now this folder is owned by root and root is also the user I get when I enter the container. When I then run a cli script, that writes to this folder, every thing works (as it's run by root) and the resulting entries in the cache dir are owned by root. Trouble starts when the php fpm process tries to write stuff in there too (as it's run by www-data).
If I add `USER www-data` (or create a new user foobar and add `USER foobar`) the container exits with status 0

So I guess the question is, is anyone running an Symfony app on Docker in Prod, if so how do you solve this? Or another question would be what is the best practice to do this? Sure on dev I could just `chmod 777` the whole folder or run the php-fpm process as root, but if that thing ever goes to prod, I wouldn't sleep very well...

So I'm assigned once again to fix a new someone else created and that seems to be the case whenever there's an issue...

Boss just assigns it to whoever is most likely to be able to investigate it... which is basically me. Other than the little time I can use to develop stuff, I'm usually cleaning up other people's messes.

And these other people are to busy working on new crap to properly explain how their existing code/processes/changes works.

And well the fact that anything breaks in production (that's not due to upstream one off issues) whoever does not think he needs to take responsibility for it.

So everyone else and especially me has to spend time understanding the shit they wrote and fixing it for them.

How do I tell my boss this nicely that we need clearly definitely ownership and whenever a component blows up in prod, the guy that wrote the code fixes it no matter what? Thereby incentivizing him to not write shit code in the first place and be more proactive in making sure it doesn't in the first place since he knows otherwise he's doing overtime to fix it?

Is it just me or is there really no such thing as a dev job where something doesn't blow up due to poorly tested and designed code every other day?

I work in a small team. As the senior dev I tens to focus on important tasks that shape the core of the product but some times I can’t divide my self when there are multiple tasks at hand, so I pass some tasks to the an other mid level dev.

So the task was to create an automation in order to CD (continuously deliver) an order from WHMCS of the (git versioned) product to customers UAT, PROD envs.

To get a background this is an old guy with “constricted” experience in PHP/jQuery/Joomla/Wordpress.

So when we were breaking up the tasks he told me he would like to implement this so i gave him the task as i was busy with core features.

I was like what could go wrong? I know he doesn’t know much about CI/CD but he can read right? He will google right? He will search for CI/CD solutions that do this out of the box right? He will design on paper or what ever and do small POCs right? He will design the flow first before starting the implementation right? RIGHT?

So fast forward to today I had a call with him this morning about some DB staff. And he wanted to show me his progress…

His solution is:
(parentheses is my brain)
1. Customer completes WHMCS order (perfect)
2. Web Hook 🪝 action (YES)
3. cpanel gets source and “automatic!” Init, all using pure PHP code ignoring the usage of the current framework (ok… something is missing)
4. cpanel web hooks(?) WHMCS to send email to customer with the envs initial setup page(?)
5. Customer opens link and adds setup info (ok fuck, fuck, fuck)

(Ok stay cool composed, lets ask some questions maybe he thought it all in a cool way I can’t get my mind around)

Me: So how are you gonna get the correct version from the repo to the env and init the correct schema?

Dev: I haven’t thought about it yet.

Me: Are we gonna save each version to a file system then your code is going to fetch them?

Dev: I haven’t really thought about it we will see. But look on customer init user setup I implemented a password strength validation and it also checks if the password is the same.

So after this Pokémon encounter I politely closed teams. Stood up drank some (a lot) coffee ☕️. Put out the washed laundry while reflecting on life’s good things, while listening to classical music 🎼 .

Then I sat on my office chair drank some more coffee, put some linking park starting with in that order:
“Numb” then “What I’ve Done” and ended with “In the end, it does really fucking matter”

So there is this owner team who reviewed my code recently. I don't have much context about the their system and architecture. We try to build our changes with less context and rely in owner team's knowledge for any review gap.
The guy from the owner team missed something in my review and changes went to prod, review already took more that it was expected to take. He took 1 week for small change reviews. Now, not him but with someone else's advice they had to revert.
I wrote a mail shooting to manager, the guy who reverted and the guy who reviewed, asking the reviewer guy to explain why didn't he mentioned about any issues at the time of the review.
I have tried best from my side. But all this, god!!!
Why everything I do has some kind of weird issue. I feel so bad blaming the guy, I just think that, the way I used to feel anxious he must be feeling the same, but what can I do? I don't want to take the blame I don't even see if I can and I shouldn't be. If it was a major issue it should have been raised but he didn't. I feel so bad that I am almost crying, I am feeling that like always I am going to be judged by my team that work is slow and on top of that I can't do anything for the guy I blamed it on.

I don't know, is it my mistake? but I cannot think of anyway I would have known this.

This got me fucked up. Listen yo.

So we have this issue on our modal right. The issue keeps poppin. It's a hotfix because its in prod. So my senior and I were on it. After a few hours, I showed him the part of the code that is buggy. It's 50 lines of code of nested if-else, else-if. And so we're still fighting it. He redid everything since we're using angular2 he did a subject, behavior-subject all that bs and I was still trying to understand what's the bug, because it's happening on the second click and so I did my own thing and found the cause bug and showed it to him, its this:

setTimeout( () => {}, 0)

the bootstrap-modal doesn't allow async inside it (I dont why, its in the package). So he explained to me why it's there. So I did my own thing again and find a workaround which I did, a one-line of angular property, showed it to him he didn't accept it because we'll still have to redo it with subjects and he was on it. I said ok. Went back to my previous issue. The director came in and ask for a fixed, my senior came up to me and told me to push my fix. Alright no problem. So we good now. Went back to our thing bla bla bla, then got an email that we will have a meeting, So we went, bla bla bla. The internal team wants a support for mobile, senior said no problem bla bla bla, after the meeting he approaches me and said (THIS IS WHERE IT GOT FUCKED UP) we wont be supporting bootstrap4 anymore because of the modal issue and since we're going to support mobile and BOOTSTRAP4 grid system is NONINTUITIVE we are moving to material design because the grid system is easier. I was blown away man. we have more than 100 components and just because of that modal and mobile support shit he decided to abandon bootstrap. Mater of fact its the modal its his code. I'm not expert in frontend but I looked at the material design implementation its the same thing other than the class names. OHHH LAWD!

Pushed some changes to PROD today. Go to login and check changes .. noooooope!

Still a bit new to Symfony 5... but I'm just not a fan right now. The login screen just jumps back to itself. No login failed message and prod log had a size of 0 so that was no help.

Traced this thing way down into the CSRF Authentication functions. \is_callable(...namespace) just returning null so no go on getting a token for isTokenValid() =/
ugh! This is truly the most torturous junk I've ever seen. Nothing in the logs so I decided to just use the good old ECHO'HERE' debugger.

What was the issue you might ask?... effin' yaml file
Fix for now is to set the session handler_id back to null

Fuck my company, sincerly.

So Im crunching my ass off, to make product, there is +- fuckton of changes that for example require refactoring flow of certain things, restructure of how shit work, Im +- 2nd weekend now, and most heavy features are cleared.

I work till late. constantly I have someone with stupid shit like calls, indeed Im needed for that stuff but also, that slows down progress of this project. Just sake of example friday 18:00 I had call (I work till 16:00) about new minor and frankly easy feature. Today, morning 8:30 one call, than 13:00 long call, Ive done the feature, didn't push it to alpha. yet though.

Now during that call that started 13:00 I get yelled on that all ordered features aren't on prod yet (I throw them to alpha becouse manual tests must be done as standard here).

Dude what the motherfuck. Im literally wearing my ass off to deliver your stupid product becouse I know its critical for company but it does not mean I can do it all in one fucking night.

F**k off and shut your mouth up and let me work for f**k sakes.

Ah also, stop f**king remotely micromanage me you little piece of sh*t.

Thanx for allowing me to vent out,
Peace.

Why do some employers make such a distinction between learning the tools at university and learning the same tools at the workplace?

Are they backward or old? Don't they know modern, high-quality universities have modern environments that are in fact real life?

Environments with acc-test-prod-dev with gitlab, ci/cd in Scrum teams and the works? Heck, at my uni we even worked at real companies, did internships there for months!

Come on.. to me this 'the tools you learned in school isn't the same experience as real life experience'. Right, these guys must be on some conservative backward model because there is in fact no difference.
I have worked both during my uni internship at a real company (in teams too) as well as irl at real companies and there is no difference, it's the same thing.
I don't care if I've learned to experience git + ReactJS etc during an internship through uni or at a workplace. It's all bureaucracy.

Whats the fucking purpose of our companys dev test and prod env. Dev always only has a single instance. Sometimes clustered services run as cluster on test. Producing headaches because the clustering behaviour couldnt be seen on a single instance and Prod lacks all the nice deployment tools off dev/test. Fuck thinking you could dev then test and prod without any major reconfiguration and headaches. And all because the Storage costs is RETARDEDLY expensive because the backup EVERYTHING with ridiculess overkill. That results in headaches when requesting new servers. Took an old Workstation from the shelves and made it my vm slave so at least i could reliably deploy to test.. Fuck this process

My team is pretty small right now. It's myself and two other guys. One lead, who's been here for five years. A senior who we brought on 2 weeks ago. And me, a regular app dev. The lead put his two weeks in last week and has been trying to brain dump as much as he can onto us.

I've been building a list of prioritization to compensate for when he leaves based on what he was saying was the most important. This list has gotten pretty massive after reviewing most of the processes in place.

I was hired mainly to quell new requests coming in and not to maintain our systems, so that's what I did. I didn't examine our prod code base too closely. I wish I had. It's in a sorry state. I'm pretty sure I have about 2 years of tech debt for a crew of two guys constantly working on it.

I've been trying to prioritize based on what gets the most bug fixes and change requests. These apps will see the biggest changes and will undergo the most maintenance.

Since I'm just a regular app dev it feels weird trying to come up with this and try to prioritize this and come up with a plan. It feels like someone else should have. If it needs done then I guess it needs done. I need to be able to collaborate and work with my co worker and be able to plan for what projects are coming next.

If anyone has any suggestions to tackle tech debt please make them. Or if there's any help for managing priorities in a different manner that may prove helpful I'm open. Honestly, I don't want to tackle this completely blind, it feels like a lot.

So apparently my code went to prod more or less all right. Phew, the deadline was this week-end. This project have been sitting there for month, they gave me the technical requirement and never bothered to ask the stakeholders about it. When the contract went in, they started to freak out it wasn't usable.

The thing is, this project had way more moving part and trying to threat video in the frontend is not the easiest. But now is REFACTOR TIME.

I dream of getting rid of the browser video api (too flaky), download the bitmap directly and render it in requestAnimationFrame. I call it just-in-time rendering. I think i'd need to put a decoder in aws, I did it already with ffmpeg. I could not manage to put it in streaming mode though, so it was still a bit slow, but i could decode, write and re-encode faster than the video player speed.

What do you lads think? Doable or not? I at least need to general tidy up (this codebase have grown organically without any fucking direction from above, like this project took all my time on the technical side, I did not have time to run after people to get specs), centralizing state, improve monorepo and tooling, perfs,...

Hopefully they understood i cant keep adding whatever feature they want today.

Recently joined new Android app (product) based project & got source code of existing prod app version.

Product source code must be easy to understand so that it could be supported for long term. In contrast to that, existing source structure is much difficult to understand.

Package structure is flat only 3 packages ui, service, utils. No module based grouped classes.

No memory release is done. So on each screen launch new memory leaks keep going on & on.

Too much duplication of code. Some lazy developer in the past had not even made wrappers to avoid direct usage of core classes like Shared Preference etc. So at each place same 4-5 lines were written.

Too much if-else ladders (4-5 blocks) & unnecessary repetitions of outer if condition in inner if condition. It looks like the owner of this nested if block implementation has trust issues, like that person thought computer 'forgets' about outer if when inside inner if.

Too much misuse of broadcast receiver to track activities' state in the era of activity, apપ life cycle related Android library.

Sometimes I think why people waste soooo... much efforts in the wrong direction & why can't just use library?!!

These things are found without even deep diving into the code, I don't know how much horrific things may come out of the closet.

This same app is being used by many companies in many different fields like banking, finance, insurance, govt. agencies etc.

Sometimes I surprise how this source passed review & reached the production.

So trunk-based is the new approach everyone is using, because it is so cool.
I used gitflow for the last projects with azure devops, set up the pipelines like tipically in 1 week if I had other things to do with the help of the portal clicking through things. PR-s triggered pipelines, everything worked cool.
But then trunk-based got momentum, so I worked with this client where 2 developers worked for !!!3 months!!! to setup trunk-based pipeline. It was not my money, so I did not say a thing. They were using infrastructure as code.
I am all in for automation, but seriously? Then again, another project where a DevOps team took 1 dev-month to setup the pipeline + meetings. And what do you get in the end? So that the same image goes on all environments? Like how many releases do you have for prod in a year. Lets say 24. 24 x 5 minutes of manual work for the release, that is 2 hours. So my question is why would you spend 2 hours of manual work while you can automate it merely in a month? Everyone loves to code, but using the ui on the DevOps portal saves you so much time. I don't get this. Maybe I am getting old :D

So, I am in the last stages of development of a really big project and I need to figure out a way to package future patches and updates for the client in order for them to manually update the project on prod server.

For reasons I cannot specify here, they will not use any automated process, and we need to provide regular patches and updates for the next year.

So I was thinking of using git archive to package changed files from our repo for every new commit, or series of commits, and just give them that, along with any database schema updates as sql files (again, no automation can be used).

We are talking about a large PHP + MySQL app, and cannot use automated deployment strategies.

I feel there must be a better way to do this, but this is the best I could come up with so far.

What do you people think?

Any ideeas?

Random error that was occurring in prod discord authentication for new users (only in prod):

It wasn't returning the discord ID of the user logging in (only in prod). This should not even be possible if the user authenticates. So the null value from the ID was used and returned the first user in the DB (I know, weird mongodb edge cases). It should have returned null for the user searched not the first user in the DB wtfff.

Anyways, new users logging in would be logged in as me. Bahahah

Pushed the fix. All is good in the hood.

PUT method works in dev but not in prod for code ignitor. Stuck on this one from a week. Someone HELP 🥲

(Checked htaccess and PUT is enabled)