!rant

After over 20 years as a Software Engineer, Architect, and Manager, I want to pass along some unsolicited advice to junior developers either because I grew through it, or I've had to deal with developers who behaved poorly:

1) Your ego will hurt you FAR more than your junior coding skills. Nobody expects you to be the best early in your career, so don't act like you are.

2) Working independently is a must. It's okay to ask questions, but ask sparingly. Remember, mid and senior level guys need to focus just as much as you do, so before interrupting them, exhaust your resources (Google, Stack Overflow, books, etc..)

3) Working code != good code. You are an author. Write your code so that it can be read. Accept criticism that may seem trivial such as renaming a variable or method. If someone is suggesting it, it's because they didn't know what it did without further investigation.

4) Ask for peer reviews and LISTEN to the critique. Even after 20+ years, I send my code to more junior developers and often get good corrections sent back. (remember the ego thing from tip #1?) Even if they have no critiques for me, sometimes they will see a technique I used and learn from that. Peer reviews are win-win-win.

5) When in doubt, do NOT BS your way out. Refer to someone who knows, or offer to get back to them. Often times, persons other than engineers will take what you said as gospel. If that later turns out to be wrong, a bunch of people will have to get involved to clean up the expectations.

6) Slow down in order to speed up. Always start a task by thinking about the very high level use cases, then slowly work through your logic to achieve that. Rushing to complete, even for senior engineers, usually means less-than-ideal code that somebody will have to maintain.

7) Write documentation, always! Even if your company doesn't take documentation seriously, other engineers will remember how well documented your code is, and they will appreciate you for it/think of you next time that sweet job opens up.

8) Good code is important, but good impressions are better. I have code that is the most embarrassing crap ever still in production to this day. People don't think of me as "that shitty developer who wrote that ugly ass code that one time a decade ago," They think of me as "that developer who was fun to work with and busted his ass." Because of that, I've never been unemployed for more than a day. It's critical to have a good network and good references.

9) Don't shy away from the unknown. It's easy to hope somebody else picks up that task that you don't understand, but you wont learn it if they do. The daunting, unknown tasks are the most rewarding to complete (and trust me, other devs will notice.)

10) Learning is up to you. I can't tell you the number of engineers I passed on hiring because their answer to what they know about PHP7 was: "Nothing. I haven't learned it yet because my current company is still using PHP5." This is YOUR craft. It's not up to your employer to keep you relevant in the job market, it's up to YOU. You don't always need to be a pro at the latest and greatest, but at least read the changelog. Stay abreast of current technology, security threats, etc...

These are just a few quick tips from my experience. Others may chime in with theirs, and some may dispute mine. I wish you all fruitful careers!

"You gave us bad code! We ran it and now production is DOWN! Join this bridgeline now and help us fix this!"

So, as the author of the code in question, I join the bridge... And what happens next, I will simply never forget.

First, a little backstory... Another team within our company needed some vendor client software installed and maintained across the enterprise. Multiple OSes (Linux, AIX, Solaris, HPUX, etc.), so packaging and consistent update methods were a a challenge. I wrote an entire set of utilities to install, update and generally maintain the software; intending all the time that this other team would eventually own the process and code. With this in mind, I wrote extensive documentation, and conducted a formal turnover / training season with the other team.

So, fast forward to when the other team now owns my code, has been trained on how to use it, including (perhaps most importantly) how to send out updates when the vendor released upgrades to the agent software.

Now, this other team had the responsibility of releasing their first update since I gave them the process. Very simple upgrade process, already fully automated. What could have gone so horribly wrong? Did something the vendor supplied break their client?

I asked for the log files from the upgrade process. They sent them, and they looked... wrong. Very, very wrong.

Did you run the code I gave you to do this update?

"Yes, your code is broken - fix it! Production is down! Rabble, rabble, rabble!"

So, I go into our code management tool and review the _actual_ script they ran. Sure enough, it is my code... But something is very wrong.

More than 2/3rds of my code... has been commented out. The code is "there"... but has been commented out so it is not being executed. WT-actual-F?!

I question this on the bridge line. Silence. I insist someone explain what is going on. Is this a joke? Is this some kind of work version of candid camera?
Finally someone breaks the silence and explains.

And this, my friends, is the part I will never forget.

"We wanted to look through your code before we ran the update. When we looked at it, there was some stuff we didn't understand, so we commented that stuff out."

You... you didn't... understand... my some of the code... so you... you didn't ask me about it... you didn't try to actually figure out what it did... you... commented it OUT?!

"Right, we figured it was better to only run the parts we understood... But now we ran it and everything is broken and you need to fix your code."

I cannot repeat the things I said next, even here on devRant. Let's just say that call did not go well.

So, lesson learned? If you don't know what some code does? Just comment that shit out. Then blame the original author when it doesn't work.

You just cannot make this kind of stuff up.

Big event. Massive traffic in production, so we were monitoring all night.

I was in a room with 2 devs of my team, a marketting girl, my boss and a designer... chilling.

Suddenly the production is down.

Boss: production is down, anyone can check?
Me: already on it
Dev1: it looks ok for me
Dev2: me too
Me: wait what? Impossible everything is down
Dev1: oh I refreshed the page it's not working
Me: don't stay on the page refreshing it like you are fucking monkeys. Give me useful intel or be quiet.
Market girl: is it working?
...
Guys is it working?
...
Hello?
Me: Not yet we are looking. Don't distract me.
Boss: client called us. They want it online now.
Dev1&2: he's looking
... 1 min later...
Boss: is it working?
Boss: is it working?
Boss: is it working?
Me: SHUT THE FUCK FOR FUCKING ONE SECOND. ALL OF YOU, OUT NOW. YOU ARE FUCKING MONKEYS WHO CAN'T DO SHIT. IF YOU CAN'T HELP JUST SHUT YOUR DAMN SHITHOLE. DEVS, LOOK WITH ME. MARKET GIRL PREPARE A FUCKING POST-MORTEM MAIL. BOSS GET THE CLIENT ON THE PHONE AND STALE. DO. YOUR. FUCKING. JOBS.

That's how I ended up screaming at everyone... the rest of the night went in complete silence and I fixed the issue 2min after the got quiet or busy.

I worked with a good dev at one of my previous jobs, but one of his faults was that he was a bit scattered and would sometimes forget things.

The story goes that one day we had this massive bug on our web app and we had a large portion of our dev team trying to figure it out. We thought we narrowed down the issue to a very specific part of the code, but something weird happened. No matter how often we looked at the piece of code where we all knew the problem had to be, no one could see any problem with it. And there want anything close to explaining how we could be seeing the issue we were in production.

We spent hours going through this. It was driving everyone crazy. All of a sudden, my co-worker (one referenced above) gasps “oh shit.” And we’re all like, what’s up? He proceeds to tell us that he thinks he might have been testing a line of code on one of our prod servers and left it in there by accident and never committed it into the actual codebase. Just to explain this - we had a great deploy process at this company but every so often a dev would need to test something quickly on a prod machine so we’d allow it as long as they did it and removed it quickly. It was meant for being for a select few tasks that required a prod server and was just going to be a single line to test something. Bad practice, but was fine because everyone had been extremely careful with it.

Until this guy came along. After he said he thought he might have left a line change in the code on a prod server, we had to manually go in to 12 web servers and check. Eventually, we found the one that had the change and finally, the issue at hand made sense. We never thought for a second that the committed code in the git repo that we were looking at would be inaccurate.

Needless to say, he was never allowed to touch code on a prod server ever again.

So they were having trouble with the server always being slow and maxed to 100%, so the boss told me when wait times were hitting 5+mins due to server trying to catch up, he complained at me, said if I could get the wait time to 30sec to instant he would raise my pay to 90k a year, then walked away after I agreed, I was quite serious but I don't think he thought I was, so I decided to look over the system, IDK who but they put all the calculations and processing server-side for the CA's on floor then sent the completed view to the CA, so I spent months recreating the entire system except the server only pulled the data needed then the new client would do all the processing on their computer since they weren't doing anything anyways, I did a practice run today as its one of our peak days, wait times went to barely 5secs or "instant" according to CA's, I walked into the office, slapped that hourly report down after just two hours and showed the massive increase in employees production times.

That look on his face...
That look on my face...
That look on my next check...

Bliss

Production is down, a coworker got himself locked in his own apartment so he can’t leave and another is late, the phone won’t stop ringing

And I don’t have the credentials to access the production server

Just a monday morning, everything’s fine 🔥😊🔥

Production is down

Me to Customer :What did you do?
Customer: Nothing
Me blurt out: The fuck you didn't!
Customer: ...
Me: ...(fuckfuckfuck)
Customer:... Well, I did run these scripts..
Me: (oh thank Christ)
Me: ok, I'll get right on it (Click)
Me to TeamLead: client called. Their prod is down!
TeamMate: did he say he didn't do anything?
Me:Yes
TeamMate: ..... Every fucking time...

The project where I realized I wanted to go from chemist to pro dev.

I built a flow-chemistry spectrometer with monitoring backend in Haskell.

Spectroscopy is where you add a reagent to a glass tube, it changes color, and by measuring the exact color it tells you how much of something (for example, a toxin) is present in the sample.

I had to do that a lot on factory samples, writing down measurements using pen & paper.

I'm lazy so I decided to do the logical thing: Automate it. I bought a second hand spectrometer, stripped the casing, did a shitload of glassblowing and hooked up tubes to the production pipelines, so I could get samples, mixing them in the correct ratio with reagents in continuous flows using valves.

I ended up using 2 home-crafted arduino-like boards (etching PCBs is fun!).

One to calibrate the mixture against known samples and control solenoid valves to continuously cycle through various reagents and deionized flushing water, the other to record the measurements and send them to a server running a Haskell/Yesod API.

The server collected the information into InfluxDB (A time series database), displaying all data on a graphite dashboard.

Eventually I wrote Haskell plugins for most of the chemistry processes, from pH & temperature measurements to polymer property and pigment tests (they made a lot of printer ink).

Then I was fired because they didn't need chemists anymore, and the code "could be maintained by the intern" (poor guy)...

But I did find out that I loved functional programming, chemistry automation projects, and crafting my own electronics during that time.

User: *Clicks on staging environment*

Giant Warning Dialog: YOU ARE CURRENTLY ENTERING THE STAGING ENVIRONMENT

Users: Ok

App: *Completely different colour, I’m talking bright unsightly yellow*

User: Ok

Giant Yellow and Red Flashing Banner at the Top of the Screen: WARNING YOU ARE CURRENTLY USING STAGING, THIS AREA IS FOR TESTING ONLY

User: The production environment sure is acting strange today. It’s a weird colour and I don’t recognize any of the data, it’s all just dummy filler data. I better create a ticket for the dev team to check o—….. no wait I’ll send an email CC everyone including the CEO and sound the alarm production is currently down and filled with giant warning messages.

Manager: OH MY GOD PRODUCTION IS DOWN DID YOU HEAR ABOUT THIS??? WHAT THE FUCK COULD THESE WARNING MESSAGES BE THAT’S ONLY SUPPOSED TO HAPPEN ON STAGING! THE CEO IS BREATHING DOWN MY NECK YOU NEED TO GET THIS FIXED IMMEDIATELY!!!!!!!

Dev: …

Tested on local work perfectly,
Push to production
Restart service
CPU went up to 100%, process alive but all services down.
Panic.
Try to think of what would be the cause.
With no hope, i restarted the service again.
Everything works as expected.
I still dont know what happened. :/

Hey, Root? How do you test your slow query ticket, again? I didn't bother reading the giant green "Testing notes:" box on the ticket. Yeah, could you explain it while I don't bother to listen and talk over you? Thanks.

And later:
Hey Root. I'm the DBA. Could you explain exactly what you're doing in this ticket, because i can't understand it. What are these new columns? Where is the new query? What are you doing? And why? Oh, the ticket? Yeah, I didn't bother to read it. There was too much text filled with things like implementation details, query optimization findings, overall benchmarking results, the purpose of the new columns, and i just couldn't care enough to read any of that. Yeah, I also don't know how to find the query it's running now. Yep, have complete access to the console and DB and query log. Still can't figure it out.

And later:

Hey Root. We pulled your urgent fix ticket from the release. You know, the one that SysOps and Data and even execs have been demanding? The one you finished three months ago? Yep, the problem is still taking down production every week or so, but we just can't verify that your fix is good enough. Even though the changes are pretty minimal, you've said it's 8x faster, and provided benchmark findings, we just ... don't know how to get the query it's running out of the code. or how check the query logs to find it. So. we just don't know if it's good enough.

Also, we goofed up when deploying and the testing database is gone, so now we can't test it since there are no records. Nevermind that you provided snippets to remedy exactly scenario in the ticket description you wrote three months ago.

And later:

Hey Root: Why did you take so long on this ticket? It has sat for so long now that someone else filed a ticket for it, with investigation findings. You know it's bringing down production, and it's kind of urgent. Maybe you should have prioritized it more, or written up better notes. You really need to communicate better. This is why we can't trust you to get things out.

*twitchy smile*

Why are job postings so bad?
Like, really. Why?

Here's four I found today, plus an interview with a trainwreck from last week.
(And these aren't even the worst I've found lately!)

------

Ridiculous job posting #1:
* 5 years React and React Native experience -- the initial release of React Native was in May 2013, apparently. ~5.7 years ago.
* Masters degree in computer science.
* Write clean, maintainable code with tests.
* Be social and outgoing.

So: you must have either worked at Facebook or adopted and committed to both React and React Native basically immediately after release. You must also be in academia (with a masters!), and write clean and maintainable code, which... basically doesn't happen in academia. And on top of (and really: despite) all of this, you must also be a social butterfly! Good luck ~

------

Ridiculous job posting #2:
* "We use Ruby on Rails"
* A few sentences later... "we love functional programming and write only functional code!"

Cue Inigo Montoya.

------

Ridiculous job posting #3:
* 100% remote! Work from anywhere, any time zone!
* and following that: You must have at least 4 work hours overlap with your coworkers per day.
* two company-wide meetups per quarter! In fancy places like Peru and Tibet! ... TWO PER QUARTER!?

Let me paraphrase: "We like the entire team being remote, together."

------

Ridiculous job posting #4:
* Actual title: "Developer (noun): Superhero poised to change the world (apply within)"
* Actual excerpt: "We know that headhunters are already beating down your door. All we want is the opportunity to earn our right to keep you every single day."
* Actual excerpt: "But alas. A dark and evil power is upon us. And this… ...is where you enter the story. You will be the Superman who is called upon to hammer the villains back into the abyss from whence they came."

I already applied to this company some time before (...surprisingly...) and found that the founder/boss is both an ex cowboy dev and... more than a bit of a loon. If that last part isn't obvious already? Sheesh. He should go write bad fantasy metal lyrics instead.

------

Ridiculous interview:
* Service offered for free to customers
* PHP fanboy angrily asking only PHP questions despite the stack (Node+Vue) not even freaking including PHP! To be fair, he didn't know anything but PHP... so why (and how) is he working there?
* Actual admission: No testing suite, CI, or QA in place
* Actual admission: Testing sometimes happens in production due to tight deadlines
* Actual admission: Company serves ads and sells personally-identifiable customer information (with affiliate royalties!) to cover expenses
* Actual admission: Not looking for other monetization strategies; simply trying to scale their current break-even approach.

------

I find more of these every time I look. It's insane.
Why can't people be sane and at least semi-intelligent?

Worst dev team failure I've experienced?

One of several.

Around 2012, a team of devs were tasked to convert a ASPX service to WCF that had one responsibility, returning product data (description, price, availability, etc...simple stuff)
No complex searching, just pass the ID, you get the response.

I was the original developer of the ASPX service, which API was an XML request and returned an XML response. The 'powers-that-be' decided anything XML was evil and had to be purged from the planet. If this thought bubble popped up over your head "Wait a sec...doesn't WCF transmit everything via SOAP, which is XML?", yes, but in their minds SOAP wasn't XML. That's not the worst WTF of this story.

The team, 3 developers, 2 DBAs, network administrators, several web developers, worked on the conversion for about 9 months using the Waterfall method (3~5 months was mostly in meetings and very basic prototyping) and using a test-first approach (their own flavor of TDD). The 'go live' day was to occur at 3:00AM and mandatory that nearly the entire department be on-sight (including the department VP) and available to help troubleshoot any system issues.

3:00AM - Teams start their deployments
3:05AM - Thousands and thousands of errors from all kinds of sources (web exceptions, database exceptions, server exceptions, etc), site goes down, teams roll everything back.

3:30AM - The primary developer remembered he made a last minute change to a stored procedure parameter that hadn't been pushed to production, which caused a side-affect across several layers of their stack.

4:00AM - The developer found his bug, but the manager decided it would be better if everyone went home and get a fresh look at the problem at 8:00AM (yes, he expected everyone to be back in the office at 8:00AM).

About a month later, the team scheduled another 3:00AM deployment (VP was present again), confident that introducing mocking into their testing pipeline would fix any database related errors.

3:00AM - Team starts their deployments.
3:30AM - No major errors, things seem to be going well. High fives, cheers..manager tells everyone to head home.
3:35AM - Site crashes, like white page, no response from the servers kind of crash. Resetting IIS on the servers works, but only for around 10 minutes or so.
4:00AM - Team rolls back, manager is clearly pissed at this point, "Nobody is going fucking home until we figure this out!!"
6:00AM - Diagnostics found the WCF client was causing the server to run out of resources, with a mix of clogging up server bandwidth, and a sprinkle of N+1 scaling problem. Manager lets everyone go home, but be back in the office at 8:00AM to develop a plan so this *never* happens again.

About 2 months later, a 'real' development+integration environment (previously, any+all integration tests were on the developer's machine) and the team scheduled a 6:00AM deployment, but at a much, much smaller scale with just the 3 development team members.

Why? Because the manager 'froze' changes to the ASPX service, the web team still needed various enhancements, so they bypassed the service (not using the ASPX service at all) and wrote their own SQL scripts that hit the database directly and utilized AppFabric/Velocity caching to allow the site to scale. There were only a couple client application using the ASPX service that needed to be converted, so deploying at 6:00AM gave everyone a couple of hours before users got into the office. Service deployed, worked like a champ.

A week later the VP schedules a celebration for the successful migration to WCF. Pizza, cake, the works. The 3 team members received awards (and a envelope, which probably equaled some $$$) and the entire team received a custom Benchmade pocket knife to remember this project's success. Myself and several others just stared at each other, not knowing what to say.

Later, my manager pulls several of us into a conference room
Me: "What the hell? This is one of the biggest failures I've been apart of. We got rewarded for thousands and thousands of dollars of wasted time."
<others expressed the same and expletive sediments>
Mgr: "I know..I know...but that's the story we have to stick with. If the company realizes what a fucking mess this is, we could all be fired."
Me: "What?!! All of us?!"
Mgr: "Well, shit rolls downhill. Dept-Mgr-John is ready to fire anyone he felt could make him look bad, which is why I pulled you guys in here. The other sheep out there will go along with anything he says and more than happy to throw you under the bus. Keep your head down until this blows over. Say nothing."

The network starts slowing down, transactions start to fail across the 450+ stores, the website starts to spit 500 errors what is going on?

Queue a frantic running around the office working out what was going wrong... Calls from all 3 data centres, nothing is going in or out of the network.

Notice the network admin come back to his desk, his eyebrows raise and he looks left and right before unplugging his laptop ethernet from one of the server access points

The network rushes back to life, everything is fine.

That particular network mapping tool is now banned for use on production.

Once we were going to present a web service to governmental firm. All is going well so far and my boss asks me to host the web application the day before the presentation.

I hosted it and all was good with demo production tests, but I had a bad feeling.

While it was running on our server, I also ran it locally with a reverse proxy just in case.

* Meeting starts *

* Ice broken and down to business *

"And now our developer will run the demo for you..."

* Run the demo from my laptop to double check --> 500 Internal Server Error *

Holy shit!!!

* Opens reverse proxy link on my laptop. Present demo during meeting. Demo works like a charm. *

Firm representative: "Great! Looking forward to go live."

*Our team walks out*

GM: "Good job guys"

ME:

plot twist
linuxxx is some of the most advanced and efficient AI ever developed, and actually uses devrant to scope out security-aware devs and take them down 1 production server at a time

Contrary to most people I really love to receive email related to jobs when I'm in holiday. I keep important alerts on.

It's like:
email: ***urgent, server down***
me (sipping mojito by the pool): fuck them. let's them deal with that
email: ***requirements all wrong, must develop the feature again***
me (enjoying a dinner): oh, I told them 100 times!, fuck all of them, work for me now, stupid moron.
email: I destroyed by mistake the db with an update..."
me (dancing like crazy): ahahaha I told you that support guys should not have access to production db, fuckfuck you, fix it yourself!!!

and so on..... I don't know, it just boost my pleasure during holiday.

Haven’t been on here for ages, but I felt like I needed to post this:

Warning:
This is long, and it might make you cry.

Backstory:
A couple of months back I worked for a completely clueless dude who had somehow landed a contract for a new website for a huge company. After a while he realised that he was incapable of completing the assignment. He then hired me as a subcontractor and I deleted literally everything he had done and started from scratch. He had over promised and under explained what needed to be done to me. It took many sleepless nights to get this finished with all the amendments and I had to double my pricing because he kept changing the brief.

Even after doubling my prices I still put in way too many hours of work. At one point I had enough and just ghosted the guy as I had done what he asked, and when he submitted it to them they wanted changes. He couldn’t make the changes, so I had to. He wouldn’t pay me extra though. I decided it wasn’t worth my time.

A couple of days ago I heard from him again. He had found another subcontractor to finish the changes. He still needed a few things though, so he promised me that I would get paid after fixing those things. I looked at the few things he had listed in our KANBAN and thought it was a few easy tasks.. until I opened the project..

I had my computer set up to sync with his server because he wanted everything done live and in production. So I naturally thought I would just “sync down” everything that the other subcontractor had done.

Here is where the magic started to happen.. I started the sync and went to grab a glass of water, and it was still running when I came back. I looked at the log and saw a bunch of “node_module” files syncing - around 900 folders. Funny thing is; neither the site nor server has anything to do with node..

I disregarded this and downloaded the files in a more manual fashion to a new folder. Interestingly I could see that my SCSS folders had not been touched since I stopped working on the project.. interesting, I thought to myself..

Turns out, the other subcontractor had taken my rendered and minimised CSS file, prettified it and worked from there. This meant that the around ~1500 lines of SCSS neatly organised in around 20 files was suddenly turned into a monster of a single CSS file of no less than 17300 lines.

I tried to explain to the guy that the other subcontractor had fucked up, but he said that I should be able to fix it since I was the one that made it initially. I haven’t replied. My life is too short for this.

To improve our user's "experience" I suggested to my boss to add a status page showing...well, the current status of our services. Everybody was up for it, so I go off and implement a basic version + automated monitoring backend, get lots of positive feedback, all seems fine.

Then it starts:

Boss: "Can you get it all set up by this Saturday?"
Me: "Uh, today is Wednesday and I've never set up all the stuff needed on a proper server before"
Boss: "Well, you still have a few days. Please also contact your coworker to get it all hooked up in our launcher"
Me: "I'll try, can't make any promises though"

Contact my coworker and tell him what the plan is. I had already given him access to the repo and he is positive to get it all hooked up (I doubt he ever cloned my repo, let alone ran my code)

Spend all Friday getting my stuff set up on the production server, feeling pretty good thanks to the many tutorials.

Contact the boss Friday evening:
Me: "All up and running"
Boss: "Thanks, but we decided to go with a basic HTML page instead. We can just manually edit that, should be enough.
Me: "..."

In the end my stuff was never used, the server I set up was finally taken down a month ago. The gratitude you get when not hacking together some absolute shit that causes problems when you don't add <br/> tags at the correct places to prevent an ugly overflow, cause the coworker was too lazy to implement some form of line wrap in the launcher. I'm not saying my stuff is the best of the best, but at least it was professional looking to a certain extent.

Wait what's that? You don't use version control on Production servers?
You want me to do what?
You want me to rename every file I have to replace with an underscore and the date after the extension so it looks like this?
SHIT.JAR_01262019
You've got to be fucking kidding me right!?
No?
Oh the production server is down again?
Is it because we're not using the right Jar file?
Well shit, I wonder why that's happening...

This is going to be a long rant, coz this is the only way to vent out my frustration against our tech head.

Yesterday, while our fucking twat tech head was playing around in company aws account, he terminated the production server. By mistake, apparently. Coz he doesn't know shit about server management. But that egoist ass won't admit and fucked the production server.

And then ran away. We developers sprang into action. Updated dns to point to staging server, setup virtual hosts, env files, point to prod database, force flush dns cache. All systems were up and running in 30 mins. And since it was staging server, it had lot of untested features and codes, and we spent rest of the day fixing the bugs.

And that tech head, who ran away hiding his tail between his legs, after he fucked the server, came back after systems were up. And started cracking jokes, that "so many features got released in 1 day" . "We cut server cost by shutting down 1 server."

We were struggling and working in full throttle to make the services running again. And that fuckity fucker was cracking jokes.

And I don't even know what excuse he gave to ceo for the downtime. I am pretty sure he would have made up some crappy excuse to hide his fucking mistake. That ass never admits his mistake. I am thinking to go to ceo today and tell the real story and get that faggot head fired or at least a strict warning.

Customer : c
Me : m

*Few weeks ago*
C: the server is slow, it sometimes takes 7 seconds before I see our data

(the project is 7+ years old and wasn't written by someone who is very good in SQL)

M: yeah I see that, our servers are busy with this one "process" (SQL query)
C: make it faster
M: well that's possible but it will take a few days (massive SQL spaghetti that I first have to untangle)
C: 😡 nvm then

*Yesterday*
C: server is down !
M: 🤔 *loads data from server and waits ~ 7 seconds*
M: Well what's the problem?
C: I need the data but it's so slow

WELL YOU MINDLESS IMBECILE... If something is slow it doesn't mean our god damn production server is down !
That just means that you have to give us a day or two so we can optimise the (ALSO BY YOUR REQUEST) rushed project... And save you YOUR money that YOU waste on the processing time on our server...

Quick recap of my last two weeks: 15 year old production server is basically dead, boss has taken over calls and claims credit for "resolving" outages (even though my coworker and I did the work, but ultimately the traffic died down enough to where it wasn't an issue anymore).

I go to a meeting to plan migration to a better server, boss bitches about not getting invited, I tell him I invited myself, and then he lectures about how that's not our job.

Different boss says we're migrating a schema for an application that should have been decommissioned 5+ years ago to use as a baseline. I explain what's going on, he says he understands, and proceeds to tell higher bosses it's perfect because there will be no user impact. OF COURSE THERE'S NO FRICKING IMPACT, YA DUNCE! there are no users!!!!

I merge two email threads together, since they discuss the same thing, but with different insight, and get yelled at, even though they requested it.

The two bosses I like are OOO for the next week, too, so I'm just sitting here hoping I don't say something that'll get me fired or sent to sensitivity training.

I'm just starting my on call rotation and don't know that I can do this. I cry when my phone rings, now, because I experience physical pain with how hard I cringe.

I got yelled at today by a guy because SOMEONE I DON'T KNOW assigned a ticket to him directly, rather than to the proper team (not his team). So I had to look into that, which at least had the benefit of preventing a catastrophic outage to our customers world wide, but no one will know because I don't brag at work; I'm too busy doing my job as well as most of my division/section/larger team, whatever the hell it's called. I saved us probably 25+ hours of continuous troubleshooting call from noticing something tiny that the people "smarter" than me missed.

**edit: sorry for typos; got my nails done yesterday but they feel like they're a mile long and I have to relearn how to type**

Dev: Hey, I need you to see something.
Client: ok
Dev: this is the models of those reports you need
Client: ok, wait, what is that number? This is wrong, we can't...
Dev: hey, calm down, this is not the production, it's just fake database!
Client: ah, ok.
(5 minutes of explantion)
Dev: so, what do you think?
Client: just one point, that data is very very wrong, we need to change this ASAP!

How do you call it when production breaks and a programmer has find out what went wrong/fix it as soon as possible while the client is breathing down their neck?

He's going through a rough patch.

*production is down*
Ops: At 5pm? On a Friday? *checks deploy history* God! Who did the deploy
Dev: It was a small patch, a tiny patch. It shouldn't have....
Ops: Deploy on a Friday evening?
Colleague: I didn't think it would...

Ops (on the outside) : *takes a deep breath* Its okay Dev, we can fix this. Don't worry

Me(in my mind) : for fuck sakes! Are you fucking kidding me?*** **** *** god damn it! *****

I started this new freelance project where I am building some android libraries for the client. Anyways, during meeting I was about to present my results and suddenly backend seemed to be down. I looked into the round "are your servers down?"
Team Lead: "Yeah our cto, also our only backend dev, is developing a new feature."
Me: "Okay but why is production down?",
Team Lead: "Ah dont worry we always test on production. We have a pretty solid hardware, we will even upgrade it soon!"
Me:"... How about you just separate your stage environments and have a develop environment?"
Client: "see, this is where our strength is. We dont need a develop stage we have very strong hardware and our backend is fully in PHP"

Thanks God I'm a freelancer

I recently joined this big MNC after shutting down my own startup. I was trying to automate their build process properly. They were currently using grunt and I favor gulp, so I offered to replace the build process with gulp and manage it properly.
I was almost done with it in development environment and QA was being done for production.
In the meantime I was trying to fix some random bug in a chrome extension backend. I pushed some minor changes to production which was not going to affect the main site. That was in the afternoon.
This Friday my senior rushed to me. It was like he ran six floors to reach me. He asked, did you push the new build system to production, I refused. He then went to the computer nearby and opened the code.
It was Friday and I was about to leave. But being a good developer, I asked what's the problem. He told me that one complete module is down and the developers responsible for them left for the day already and are unreachable.
I worked on that module multiple times last month, so I offered my help. He agreed and we get to work.
The problem was in the Angular front end. So we immediately knew that the build process is screwed. I accidentally kept the gulp process open for anyone, so I immediately rebuilt using grunt and deployed again, but to no success.
Then I carefully analyzed all the commits to the module to find out that I was the one who pushed the change last. That was the chrome extention. I quickly reverted the changes and deployed and the module was live again. The senior asked, how did you do that? I told the truth.
He was surprised that how come that change affect the complete site too. We identified it after an hour. It was the grunt task which includes all the files from that particular module, including chrome extension in the build process.
He mailed the QA team to put Gulp in increased priority and approved the more structural changes, including more scrutiny before deployment and backup builds.
The module was down for more than 5 hours and we got to know only after the client used it for their own process. I was supposed to be fired for this. But instead everyone appreciated my efforts to fix things.
I guess I am in a good company 😉

I resigned today 😍
No more dealing with shit architecture and experimental beta technologies being used in production which went down constantly 😎
Back to good ol' product development! 🤩
One month notice period to go

Rather than singling out one person, I wanna present what I see as incompetent/stupid/ignorant:
- no will to learn
- failure to follow the very specific instructions & later asking for help when they FUBR sth & not even knowing what they did to fuck up in the first place
- asking how to solve stuff, then ignoring the suggestions & doing sth totally against recommendations
- failure to remember most basic stuff, especially if not writing it down to look at later when needed
- failure to check logs & 'google' stuff before asking why something isn't working the way they want it
- after two weeks, asking me how feature xy works, mind you they coded it, not me
- asking me why they did something in a specific way - WTF, am I a mind reader?! Who designed that crap?! Me or you?!!
- being passive/aggressive & snarky when told to do something or being asked why isn't it done already
- not testing their shit properly
- not making backups when upgrading (production) servers
- not checking the input value, no validation.. even after many many debacles on production with null ref exceptions
- failure to admit they fucked up
- not learning from (their) mistakes

When the production server goes down

Let me tell you a story.

Our company has a homegrown monitoring solution. Keeps track of our deployments and alerts us when something is broken. Really nice for the most part, except a little issue where we get up to 25 alerts PER DAY that our PRODUCTION ENVIRONMENT IS DOWN. Including weekends.

With this many false positives, we quickly learn to ignore the alerts and miss real incidents.

So we approached this team, remember its our own tool, and told them about the problem. Turns out it is a known issue. And here's the kicker: they aren't planning on fixing it!

It gets better. Rather than fix this glaring issue, their solution is to make ANOTHER ALERT that lets us know the monitoring is misbehaving.

To recap, we can now expect to get up to 25 false positive alerts per day that our production is down, followed immediately by more alerts that the monitor is broken, which means we can ignore the previous alert.

As our PM said when he heard this: fuck that noise. We are escalating the shit out of this!

Years ago, I was part of a project that went completely off the rails. A little context: we were a services company and we had local offices in cities all over the country. My team provided 2nd-level support which means we often had the PMs call us from those via an annoying Nextel radio.

I won’t go through the details but suffice it to say this project envolved one such branch going rogue and committing actual fraud, with criminal proceedings and all. People were on the edge, and the relationship with that branch was increasingly hostile. There was also an internal power struggle in the company between some directors at that point. In other words, a clusterfuck I’ll always cherish, if by cherish you mean hate hate hate. Anywho…

One time, there was a national holiday on a Thursday and we were going to make it a long weekend. As customary, I communicated with all the PMs about contigency plans. This PM then told us that we could not take Friday off because the customer wanted us to fly over there. We were supposed to be at the customer’s site early Friday morning. That meant we would have to fly Thrusday afternoon. I wasn’t happy.

It immediatly felt arbitrary too. As I said, the relationship was not good and we suspected he was just trying to cost us our days off. I knew enough of the customer to be fairly sure they would not have requested us that Friday. Why did the customer want us then? There was nothing yet on production and if it was just to show progress, surely we could move it to Monday. At worst, can’t we make it over the phone? No, no, no, he said. The customer was adamant that we be there on Friday. Sucked to be us.

So we flew over Thursday afternoon and on Friday morning we headed to the customer’s offices only to find it closed. They too had made it a long weekend and wouldn’t be back until Monday.

Normally I would be furious over the waste of time but to be honest, both I and my colleague smiled at that. It confirmed that the PM just tried to screw us and the customer have never asked for us. We headed back to the local office.

Before coming in, we both bought ourselves some icecream. My friend stayed in the little garden in front and I went in. The PM immediatly saw me and demanded to know why I wasn’t at the customer yet. I didn’t answer. Instead I grabbed the Nextel radio and headed back out with the PM following. I then sat down on the grass and called my director. Smiling and staring at the PM, I told him about the office being closed. The PM’s face froze when my director asked to talk to him.

We sat outside under the sun, enjoying our icecreams while the PM got shouted at. It was the best icecream I’ve ever had.

This happened via mail thread today.

Boss: we need this new brilliant feature I just made up and running asap! Top priority, it has to be done well, for my reputation is on the line!

Me: *looks at the specifics* 'kay, looks easy enough, this evening max and it will be ready. I just nees some extra info about what kind of data validations (I speak no accountant) are needed, and some other details (a total of 3 questiona).

B: Sure! Remember, it needs to be perfect, as my reputation is at stake. Call me on the phone and I'll give you the details!

M: Can't you answer via mail? Thua way both me and the other devs will have clewr guidelines

B: Just call me! Why do you need it to be written down? It's faster this way!

...Fine. I'll keep asking until you're ready to give me a written answer to my questions. No way I'll take security details via phone for something you want in production this evening. No chance in Hell I'll take responsibility for "misunderstanding" what you said on the phone. Why does it always has to be like that?

That paranoia when you deploy for production and you keep checking if it isn't down every 10 minutes.

me: "Why is the QA guy manually typing JSON into a production environment?"
asshole: "That's not your responsibility."
me: "Why didn't you just migrate data? This is dangerous."
asshole: "You need to go sit down."

We have 2 layers of testing environments and production.

I tested the changes on the 1st layer, bud since it was 5min to lunch i did not test on 2nd layer which is connected to the production DB. I pushed to production and caused 5+ websites to go full retard and went to lunch.

Came back to 19emails and 3+ skype msgs about "why the fck would you do that..."

Estimated damages nearly 20k EUR and i lost some permissions for two weeks, but my great boss helped me out and cheered me up by telling stories how he took down multiple servers too

plot twist: im the team leader of our office now :)

So, a few years ago I was working at a small state government department. After we has suffered a major development infrastructure outage (another story), I was so outspoken about what a shitty job the infrastructure vendor was doing, the IT Director put me in charge of managing the environment and the vendor, even though I was actually a software architect.
Anyway, a year later, we get a new project manager, and she decides that she needs to bring in a new team of contract developers because she doesn't trust us incumbents.
They develop a new application, but won't use our test team, insisting that their "BA" can do the testing themselves.
Finally it goes into production.
And crashes on Day 1. And keeps crashing.
Its the infrastructure goes out the cry from her office, do something about it!
I check the logs, can find nothing wrong, just this application keeps crashing.
I and another dev ask for the source code so that we can see if we can help find their bug, but we are told in no uncertain terms that there is no bug, they don't need any help, and we must focus on fixing the hardware issue.
After a couple of days of this, she called a meeting, all the PMs, the whole of the other project team, and me and my mate. And she starts laying into us about how we are letting them all down.
We insist that they have a bug, they insist that they can't have a bug because "it's been tested".
This ends up in a shouting match when my mate lost his cool with her.
So, we went back to our desks, got the exe and the pdb files (yes, they had published debug info to production), and reverse engineered it back to C# source, and then started looking through it.
Around midnight, we spotted the bug.
We took it to them the next morning, and it was like "Oh". When we asked how they could have tested it, they said, ah, well, we didn't actually test that function as we didn't think it would be used much....
What happened after that?
Not a happy ending. Six months later the IT Director retires and she gets shoed in as the new IT Director and then starts a bullying campaign against the two of us until we quit.

- Be me (Dev)
- Develop website for client that has an online payment integration.
- Notice there is no one for QA and testing.
- Try to raise voice about it but get shot down by management.
- Get order to push the project to production (because the deadline has arrived, therefore the project must be ready)
- Production performance has an error margin of 15% (customers getting their card over/under charged)
- Get blamed and flamed for everything cuz well, its all the dev's fault.

FML

The first time I caused a massive error on production.

The good news was the site didn't go completely down. The bad news, however, was that it went down for 60% of our users, and because it's only partial, it got detected only after about two hours.

Everyone halted what they were doing to help investigate the issue. When it turned out that my latest commit caused the error, I was told to fix it... with the CTO and senior software architects watching.

It all happened because I deleted one too many line, an if statement, making the accompanying else statement a complete nonsense. It was a corner case code unforeseen by the QA guy.

The attached meme perfectly describes my feeling for the rest of the month following that accident.

Just pushed my first ever ticket to production! AAAND on top of that, did not take down our entire system. 😅

Production crashed literally 5 minutes after I went to bed. Collaborators couldn't fix the problem as they had to reset an API key only my account could access. They tried calling me but my phone was muted. Woke up 5 hours later only to realise it had been down all night.

Whoops.

2 weeks into the job and already had a fiesta

Apparently production went down.
I work for a huge airline company so that’s a big deal..

However, production was down for the better part of a fuckin day.

The reason? Not a single developer noticed 🤷‍♂️

1. high severity production incident was asked to look into at the end of the day.
2. needed fix in ui.
3. fixed and deployed in 1 hour.
4. issue remained. debugging began.
5. gave up at 1 AM and went to sleep.
6. woke up at 6 and after debugging for 2 hours, identified to be a back end issue.
7. worked with back end team for the fix, and 6 hours and 3 deployments later, it worked.
8. third party vendor reported they are still not receiving one parameter from us.
9. back end team realised they forgot to ask ui to send another parameter.
10. added the parameter in ui, redeployed ui.
11. build and deployment tool broke down. got it fixed. delay of 1.5 hours.
12. finally things are in place. total time 26 hours.

13. found half bottle of vodka, leftover from last weekend. *Priceless*

OK, so I was making this website for a university event. After publishing it, I start getting calls that the registration form isn't working. So I review the code and fix the problem. But when I push it to production, the problem still persists, all this while the event management is breathing down my neck because they can't get any registration! kept on trying for the whole fucking day, switched off my phone and went to sleep! The next morning, I find out, I had been pushing the old version of the file! Yes! I felt incompetent and stupid!

> Worst work culture you've experienced?

It's a tie between my first to employers.

First: A career's dead end.
Bosses hardly ever said the truth, suger-coated everything and told you just about anything to get what they wanted. E.g. a coworker of mine was sent on a business trip to another company. They had told him this is his big chance! He'd attend a project kick-off meeting, maybe become its lead permanently. When he got there, the other company was like "So you're the temporary first-level supporter? Great! Here's your headset".

And well, devs were worth nothing anyway. For every dev there were 2-3 "consultants" that wrote detailed specifications, including SQL statements and pseudocode. The dev's job was just to translate that to working code. Except for the two highest senior devs, who had perfect job security. They had cooked up a custom Ant-based build system, had forked several high-profile Java projects (e.g. Hibernate) and their code was purposely cryptic and convoluted.
You had no chance to make changes to their projects without involuntarily breaking half of it. And then you'd have to beg for a bit of their time. And doing something they didn't like? Forget it. After I suggested to introduce automated testing I was treated like a heretic. Well of course, that would have threatened their job security. Even managers had no power against them. If these two would quit half a dozen projects would simply be dead.

And finally, the pecking order. Juniors, like me back then, didn't get taught shit. We were just there for the work the seniors didn't want to do. When one of the senior devs had implemented a patch on the master branch, it was the junior's job to apply it to the other branches.

Second: A massive sweatshop, almost like a real-life caricature.
It was a big corporation. Managers acted like kings, always taking the best for themselves while leaving crumbs for the plebs (=devs, operators, etc). They had the spacious single offices, we had the open plan (so awesome for communication and teamwork! synergy effects!). When they got bored, they left meetings just like that. We... well don't even think about being late.
And of course most managers followed the "kiss up, kick down" principle. Boy, was I getting kicked because I dared to question a decision of my boss. He made my life so hard I got sick for a month, being close to burnout. The best part? I gave notice a month later, and _he_still_was_surprised_!
Plebs weren't allowed anything below perfection, bosses on the other hand... so, I got yelled at by some manager. Twice. For essentially nothing, things just bruised his fragile ego. My bosses response? "Oh he's just human". No, the plebs was expected to obey the powers that be. Something you didn't like? That just means your attitude needs adjustment. Like with the open plan offices: I criticized the noise and distraction. Well that's just my _opinion_, right? Anyone else is happily enjoying it! Why can't I just be like the others? And most people really had given up, working like on a production line.

The company itself, while big, was a big ball of small, isolated groups, sticking together by office politics. In your software you'd need to call a service made by a different team, sooner or later. Not documented, noone was ever willing to help. To actually get help, you needed to get your boss to talk to their boss. Then you'd have a chance at all.

Oh, and the red tape. Say you needed a simple cable. You know, like those for $2 on Amazon. You'd open a support ticket and a week later everyone involved had signed it off. Probably. Like your boss, the support's boss, the internal IT services' boss, and maybe some other poor sap who felt important. Or maybe not, because the justification for needing that cable wasn't specific enough. I mean, just imagine the potential damage if our employees owned a cable they shouldn't!

You know, after these two employers I actually needed therapy. Looking back now, hooooly shit... that's why I can't repeat often enough that we devs put up with way too much bullshit.

it was 12am when we are ready to launch our new web design which requires a lot of hardwork and routing processes. my team lead was the one who pushes the button to production using "cap production deploy" command. everyone in the room (including PM) was like counting down like launching a rocket to space. the feeling is great knowing that everyone was sleepy at that time. im glad it went smooth and everyone congratulates each other.

Took down banking application on production server.

I left my previous company because my tech leadership was insensitive and agressive.

However, I am in a start-up right now and CTO is a nut job.

He creates random Slack threads and keeps messaging me like crazy. The co-founders have shut him down multiple times and yet his only success metric is "number of deliveries".

The other day something broke in production and teams were discussing about resolving the bug in one of the Slack channels.

CTO literally wrote this and I wish I was making this up, "let us not look at the logs and trust our code to work fine."

I was baffled and confused. I realised me leaving my previous organisation because of such tech leadership was a stupid decision.

Crows are black everywhere.

! exactly dev

I'd ditched Windows and spent a while exploring the Linux ecosystem for content creation. And I have to say, it was not a nice experience.

As much as I respect the Linux mantra of "free as in freedom" and "you need to roll up your sleeves and figure out stuff on your own", it just isn't good enough for non-dev work. Sorry guys, but I need software that gets out of my way and at least does what it's supposed to do. I can't stand a horrible UI or delays and random crashes, which is exactly what happens with most things under Linux.

To replace my Windows workflow I used the following:
1. Windows -> elementaryOS (because Debian/Ubuntu repositories seem to have the best software support, and elementaryOS is the least horrible looking thing that supports that) and then Arch, because, well, Arch.
2. Blender + Maya -> Blender + Maya on Linux.
3. Reaper + FL Studio -> Ardour + LMMS.
4. Photoshop -> GIMP + Krita + Inkscape.
5. ZBrush -> nothing :(

As you can see, my use cases are pretty much all over the spectrum.

Firstly, installing and configuring stuff. A pleasure on Windows, an absolute pain on Linux. Everything just worked on Windows, I had to wrestle with library versions and patches and unstable audio layers (Linux audio just sucks, except for JACK) on Linux.

Out of these, Blender and Maya were the best experience. But even then, both would suffer from random crashes that just didn't happen on Windows.
Ardour is actually really nice when it works. Its use of JACK for routing makes it really really flexible, but it just isn't stable enough to depend on. LMMS is utter crap. I'm sorry, but I just hate the UI. Can't stand it.
GIMP, Krita, and Inkscape can't beat Photoshop, even when you consider them together. Adobe software workflow is just so much better and more intuitive.
Blender 3D sculpting is not bad, but it's nowhere as good as ZBrush.

Also, if you're a C++ dev like me, nothing beats Visual Studio 2017. Nothing. That IDE just blows everything else out of the water. Even VSCode. And it's not slow at all, it handled a fairly large project (PBRTv3) just fine on my Windows development VM. Yes, a VM.

So...I ditched Linux and went back to Windows, but I keep Linux as a VM for when I actually want to mess with Blender or Ardour. Or some dev stuff which Windows sucks at (which is becoming less frequent because of WSL).

Out of all the above, the only one I'd consider ready for production use would be Blender. Developers of open source software, please learn from Blender. Kickass UI and user friendly operation is extremely important, you can't make a random window with GTK buttons and text boxes and arcane config files and expect people to use it for serious work.

Also, Windows beats Linux hands down as an everyday OS. It's always been rock solid, if you take care of it properly (and that goes for any OS). Updates hardly take any time because I run it on a SSD. As for all the advertising and marketing bullshit, you can block a large amount of stuff. And for what can't be blocked, well, I just have to live with it, because the alternative is compromising on my creative output, which is too much for me.

I still run Linux on my server, though. And on my embedded devices (Pi, BeagleBone, etc.). It absolutely rocks there.

I realize that Linux software is not going to improve unless we do something about it, so I'll be contributing fixes and code (the joys of being a C++ dev, yay). Still, I feel that the platform and software as a whole is just not mature enough.

Have you ever had a problem with a partner, but you cannot prove at 100% that he has something against you? But you cannot stand him?

Well, this happened two years ago. I was working as a tester, and “John” (I won’t say his real name) was the dude who tests my tests, but in production.

I ‘m a sociable person, and I don’t mind talking with another people. Suddenly, I noticed that my co-worker started to behave a little bit... rude? Plain? I don’t know, but sometimes he didn’t answer my conversations/questions, or sometimes answered with extra-negative stuff.

“Well, his life is not easy” I said to myself. “Everybody haveproblems”, “I have to understand him and calm down”.

Two weeks later started to report really REALLY absurd production bugs, and with absurd I mean, for example, that he didn’t like the color of a button, a point next to a phrase, etc., things very very simples, but sometimes he ignored big errors.

Once I had to went out of the city for few hours, and asked to permission to go out. I had pending tests, but I left a document with specifications in case of emergency. Even passwords. Before I could finish the thing in the other city, my partner called me two times, and asking me obvious things! I had to go back to the office ealier that I had planned, so f*** angry 😡 and when I arrived to the office, John said “no, forget it, let’s solve it tomorrow” 🙃 WTF?!!!!!!

I decided not to argue with him.

Also noticed that his headaches suddenly increased, and looked so tired ☹️ I felt guilty to judge him

I felt so guilty, and even today I don’t know what to feel about that or what to think. I don’t work there anymore, but, What do you think? What would you do?

Hey Designer/Developers, I got a question for you. Yeah, you 👇🏽

When working on a project codebase that is expected to grow and evolve heavily. How do you usually split up your CSS (SASS, LESS etc) in a good way to take into account all the different device sizes?

I am not asking how it is done but more about the design of the code. This would be for a production codebase to be released.

Do you use large blocks broken down by media...

(Media width) {

~site code

}

(Other media) {

~same site code with diff sizes

}

Or do you do individual media queries inside css classes...

.className {

(Media size) {

}

(Other media) {

}

}

Or a mixture of both?

If it is a mixture of both then how do you decide which way to go about structuring the code.

I have been endeavouring to greatly improve my CSS and have done so. But this question has been bugging me. Both sides seem to be a bit sloppy and my programmer side is fighting the repeatitipve code.

Note: all code examples are gibberish and only intended for visualization.

Two years ago I started a small online business. It was not a long term investment and it literally ended up being a one man business. The idea was to provide a service to a small group of people who will benefit from my idea and to offer it to them at a very cheap price. (It being the cheapest helped its popularity a lot).

However, never once did it actually make any profit. (and i never wanted it to make a profit) I wanted it to be self sustaining business and it was.

This was a project for my University by the way, I started off in my first year because of my extensive knowledge in the particular matter, and I only sold to people on campus.

Now that its been 4 years, my batch is graduating, and so there aren't many people to spread the word about this project. It's finally the time to actually say goodbye to this project.

I leased a dedicated server two years ago, and I am finally saying goodbye to that too (can't afford to keep it live anymore). And seriously, it feels sad to shut this machine down haha, I've had so much fun playing around with the configurations (even though it was a production server).

It's clear that this downsizing will continue and I will be closing the service in the near future.

Boss: Did you get that trivial change I requested completed?
Me: No, I've been busy trying to fix a critical issue with a production app.
Boss: I don't want other people dictating how you spend your time.
Me: ...

Let it all burn down, then, I guess!

That moment of chills down your spine when you delete a few records in a huge production db and thinks "strange, this is taking longer than it should?" and suddenly realise that you forgot to include the "where"-clause in the statement...

This is the craziest shit... MY FUCKING SERVER JUST SET ON FIRE!!!

Like seriously its hot news (can't resist the puns), it's actually really bad news and I'm just in shock (it's not everyday you find out your running the hottest stack in the country :-P)... I thought it slow as fuck this morning but the office internet was also on the fritz so I carried on with my life until EVERYTHING went down (completely down - poof gone) and within 2 minutes I had a technician from the data centre telling me that something to do with fans had failed and they caught fire, melted and have become one with the hardware. WTF? The last time I went to the data centre it was so cold I pissed sitting down for 2 days because my dick vanished.

I'm just so fucking torn right now because initially I was absolutely fucking ecstatic - 1 week ago after a year of doomsday bitching about having a single point of failure and me not being a sysadmin only to have them look at me like I'm some kind of techie flat earther I finally got approval to spend around 5x more per month and migrate all our software to containerized micro services.

I'll admit this is a bit worse than I expected but thanks to last week at least I have recent off site images of the drives - because big surprise I have to set this monolithic beast back up (No small feat - its gonna be a long night) on a fresh VPS, I also have to do it on premises or the data will only finish uploading sometime next week.

Pro Tip: If your also pleading for more resources/better production environment only to be stone walled the second you mention there's a cost attached be like me - I gave them an ultimatum, either I deploy the software on a stack that's manageable or they man the fuck up and pay a sys admin (This idea got them really amped up until they checked how much decent sys admins cost).

Now I have very flexible pockets because even if I go rambo the max server costs would only be 15-20% of a sys admins paycheck even though that is 13 x more than our current costs.

We had a major core router hardware failure in our LA datacenter today and every one of our services has been down since 6am, including all production servers. We have about 15,000 sites down across our entire platform. Our manager came over and told us to just go home because we need to replace the hardware and the process is expected to take all day, and we can't do any work until then because all the production servers are down. So you could say that it's been a pretty easy Friday so far! I'm headed home to play Spider-Man

I am a senior a DevOps engineer who took the production stack down for ~10 minutes today because of a bad code commit. I could use some encouragement! It’s a fierce world of competitive engineers and I wonder why my company doesn’t replace me. The mistake was missed by two other peer reviews... but that doesn’t stop me from feeling this way.

Have you crashed prod? Did your team support you or tear you down?

It's 5 AM and I don't want to shit on anybody's party but trust me when I say most of you here complaining about legacy code don't know the meaning of the word.

As someone who maintained a PHP4 codebase with an average file length of 3000+ lines for almost 4 years, I feel you, I feel your pain and your helplessness. But I've seen it all and I've done it all and unless you've witnessed your IDE struggle to highlight the syntax, unless you had to make regular changes in a test-less SVN's working copy that **is** the production and unless you are the reason that working copy exists because you've had enough of `new_2_old_final_newest.php` naming scheme, you do not know legacy. If you still don't believe me bare in mind I said "is" as in: "this system is still in production".

But also bare hope. Because as much grief as it cost me and countless before me, today of all days, without a warning, it got green lit for userbase migration to a newer platform. And if this 20 years of generous custom features and per client implemented services can be shut down even though it brings more profit than all the other products combined, so can happen to any of your projects. 🙏

Unfortunately, I do mean *any*.

Today is a stark reminder of why i want to leave here. First we couldn't do anything because production was down which blocks dev login. Then support tells me I need to work harder because my bug count keeps going up. But what is in my bugs? Feature requests, global changes, and work that isn't mine. Gee thanks :( Why does support get to comment on my performance anyway by something as dumb as a bug count? Grrr.

Sat down with the Project Management team today to discuss a signage installation. This is how the conversation went...

Me: Right, so we need to get the hardware on-site asap so we can get this configured before it goes over to the production guys to have the facisa installed.
Them: That's fine we have plenty of time. Stop rushing things.
Me: Okay, so do we have the story board in place ready for development?
Them: Nope. Hasn't been done by the designer yet because he is in a bad mood.
Me: Okay so when does the client want this?
Them: 3 Weeks' time
Me: But it is atleast a week of dev time?
Them: Sure. But you can work late if needed...right?

This is a typical conversation between them and me. I'm the sole developer here. So done with today.

There was an issue whilst you were away, we had to make a small css change.. We pushed it into master but it said something about the branch being behind the tip by 50 commits or something. It's okay, we forced it up though and force pushed it to production as well but the site went down.. In the end we had to ftp it up manually but the customer is saying things that were there before now aren't there any more?

I thought you put this "release process" in so things like this wouldn't happen! I think we need to review it as it clearly isn't working.

Fucking shit i just had a 3 days chat with google's cloud engineer about an issue i had in a project. eventually the issue occured due to an update they made on some projects involving IAM changes that required some changes from my part in my security toles. Like wtf haven't you heard of data fixes when you roll out such changes?! I just had my production env down for 72hours for their fuckup.
At least send an email regarding it so we could set it up in time

I work on a warehouse dev team. One day this past year, I was trying to deploy a new build to a QA server. Earlier that day I had been looking at the logs on the production server and had left the ssh session open. I had been working for less than a year out of college at this point and shouldn't have had access to deploy to the production server.

Long story short I deployed my QA build to the production server and saw there were problems connection to our production database. Then my heart dropped in my chest as I realized I had just brought down our production server.

I managed to get the server back up by rolling back in about 5 minutes and no one ever knew except some people on my team.

I felt horrible for the longest time. Later in the year another guy that joined my team that has about 20 years of experience under his belt did the exact same thing, but needed help rolling it back. Needless to say, that made me feel a lot better. 😂

Definitely the worst moment of my year.

I didn't leave, I just got busy working 60 hour weeks in between studying.

I found a new method called matrix decomposition (not the known method of the same name).

Premise is that you break a semiprime down into its component numbers and magnitudes, lets say 697 for example. It becomes 600, 90, and 7.

Then you break each of those down into their prime factorizations (with exponents).
So you get something like

>>> decon(697)
offset: 3, exp: [[Decimal('2'), Decimal('3')], [Decimal('3'), Decimal('1')], [Decimal('5'), Decimal('2')]]
offset: 2, exp: [[Decimal('2'), Decimal('1')], [Decimal('3'), Decimal('2')], [Decimal('5'), Decimal('1')]]
offset: 1, exp: [[Decimal('7'), Decimal('1')]]

And it turns out that in larger numbers there are distinct patterns that act as maps at each offset (or magnitude) of the product, mapping to the respective magnitudes and digits of the factors.

For example I can pretty reliably predict from a product, where the '8's are in its factors.

Apparently theres a whole host of rules like this.
So what I've done is gone an started writing an interpreter with some pseudo-assembly I defined. This has been ongoing for maybe a month, and I've had very little time to work on it in between at my job (which I'm about to be late for here if I don't start getting ready, lol).

Anyway, long and the short of it, the plan is to generate a large data set of primes and their products, and then write a rules engine to generate sets of my custom assembly language, and then fitness test and validate them, winnowing what doesn't work.

The end product should be a function that lets me map from the digits of a product to all the digits of its factors.

It technically already works, like I've printed out a ton of products and eyeballed patterns to derive custom rules, its just not the complete set yet. And instead of spending months or years doing that I'm just gonna finish the system to automatically derive them for me. The rules I found so far have tested out successfully every time, and whether or not the engine finds those will be the test case for if the broader system is viable, but everything looks legit.

I wouldn't have persued this except when I realized the production of semiprimes *must* be non-eularian (long story), it occured to me that there must be rich internal representations mapping products to factors, that we were simply missing.

I'll go into more details in a later post, maybe not today, because I'm working till close tonight (won't be back till 3 am), but after 4 1/2 years the work is bearing fruit.

Also, its good to see you all again. I fucking missed you guys.

Production server is down.

Do I want to continue?
Y -> vacation lost (Production server is down).

N -> Ok, I will gather more packages for you to update next time.
😭😭

When I brought down production with a change I made in my less experienced days.

Hello DevRant community! It’s been a while, almost 5 years to be exact. The last time I posted here, I was a newbie, grappling with the challenges of a new job in a completely new country. Oh, how time flies!

Fast forward to today, and it’s been quite the journey. The codebase that once seemed like an indecipherable maze is now my playground. The bugs that used to keep me up at night are now my morning coffee puzzles. And the team, oh the team! We’ve moved from awkward nods to inside jokes and shared victories.

But let’s talk about the real hero here - the coffee machine. The unsung hero that has fueled late-night coding sessions and early morning stand-ups. It’s seen more heated debates than the PR comments section. If only it could talk, it would probably write its own rant about the indecisiveness of developers choosing between cappuccino and latte.

And then there are the unforgettable ‘learning opportunities’ - moments like accidentally shutting down the production server or dropping the customer database. Yes, they were panic-inducing crises of apocalyptic proportions at that time, but in hindsight, they were valuable lessons. Lessons about the importance of thorough testing, proper version control, reliable backup systems, and most importantly, owning up to our mistakes.

So here’s to the victories and failures, the bugs and fixes, the refactorings and 'wontfix’s. Here’s to the incredible journey of growth and learning. And most importantly, here’s to this amazing community that’s always been there with advice, sympathy, humor, and support.

Can’t wait to see what the next 5 years bring! 🥂

You get to work, things have broken down in the night, you have no access to production or even test environment and you have to guess why. You do the same job as somebody in other countries for less pay while everyone else has this laid back approach where the time they actually spend working is negligible. Until the sheer amount of entropy in your organization wears you down and you just become part of the problem.

That feeling when you're working with production data and you get a cold feeling running down your spine, telling yourself: you could really fuck up now.

So, in my company we where initially about 20 programmers doing two big projects.

The client (who also is the owner of the company) keep asking more and more and more things. Each 3 months we update the site but the client doesn't start the marketing or anything else, so the app don't have any users.

After two years of development, 26 micro services, one big web platform in Python (web2py, bad decision) and a hybrid mobile app the client decide to shut down the project because it was "a little bit illegal".

The second project have the same problems, but this project does have marketing, the shitty part is after two year and a lot of development now the project isn't viable because the market is gone.

The boss calls, says he have some problems and he will fire 18 persons and reduce the payment of the rest, he ask us to "hold" for the good times.

The great idea he had for earn money is rewriting a WordPress app that have 4 years in production to angular (because he, who knows why, thinks angular is the best shit out there)

I want to quit but even with the reduced payment I know he pays way more than the market average, plus I'm still student.

I didn’t turn down a dev freelance project when the client decided against going with best practices because the solution I offered was a well-established design pattern but created a need for a financial management change she didn’t like. I stupidly built what she asked for. It worked fine in the 3rd party vendor test environment but failed on production. After hours of analysis of code to ensure no changes happened to my source during test->prod deployment, and the vendor denying they had config differences between them, and the client refusing to pay, all I could do was abandon the project.

Bug I had to fix today: some elements in our React app were being swapped with other elements.

We had `<foo>bar</foo>` on the component but on the html `foo` was being swapped with some other element in our app. It's contents ("bar" in the example) were being left in place, though, so we were getting `<baz>bar</baz>`.

This would only happen when running on production mode. On development everything was fine.

Also, everything seemed fine on the React dev tools. `foo` was where it was supposed to be, but on the html it was somewhere else.

Weirdest shit I ever saw when using React. I found a way to go around it and applied that fix, but I'm still trying to track this down to the source.

The worst part was waiting for fucking webpack to finish the production bundle on every fucking change I wanted to test. I didn't miss the change-save-compile-test flow at all.

What a shit day.

Get ready for a awesome conspiracy theory/ WhatsApp forward :D i like how people are coming with new stuff every minute of their boredom . Makes you ponder:

====================================

🔥🔥🔥🔥🔥🔥

How to dominate the world quickly?

THE GREAT CHINESE STAGE

1. Create a virus and the antidote.

2. Spread the virus.

3. A demonstration of efficiency, building hospitals in a few days. After all, you were already prepared, with the projects, ordering the equipment, hiring the labor, the water and sewage network, the prefabricated building materials and stocked in an impressive volume.

4. Cause chaos in the world, starting with Europe.

5. Quickly plaster the economy of dozens of countries.

6. Stop production lines in factories in other countries.

7. Cause stock markets to fall and buy companies at a bargain price.

8. Quickly control the epidemic in your country. After all, you were already prepared.

9. Lower the price of commodities, including the price of oil you buy on a large scale.

10. Get back to producing quickly while the world is at a standstill. Buy what you negotiated cheaply in the crisis and sell more expensive what is lacking in countries that have paralyzed their industries.

After all, you read more Confucius than Karl Marx.

PS: Before laughing, read the book by Chinese colonels Qiao Liang and Wang Xiangsui, from 1999, “Unrestricted Warfare: China’s master plan to destroy America”, on Amazon, then we talk. It's all there.

🔥🔥🔥🔥🔥🔥🔥🔥

Worth pondering..
Just Think about this...

How come Russia & North Korea are totally free of Covid- 19? Because they are staunch ally of China. Not a single case reported from this 2 countries. On the other hand South Korea / United Kingdom / Italy / Spain and Asia are severely hit. How come Wuhan is suddenly free from the deadly virus?

China will say that their drastic initial measures they took was very stern and Wuhan was locked down to contain the spread to other areas. I am sure they are using the Anti dode of the virus.

Why Beijing was not hit? Why only Wuhan? Kind of interesting to ponder upon.. right? Well ..Wuhan is open for business now. America and all the above mentioned countries are devastated financially. Soon American economy will collapse as planned by China. China knows it CANNOT defeat America militarily as USA is at present

THE MOST POWERFUL country in the world. So use the virus...to cripple the economy and paralyse the nation and its Defense capabilities. I'm sure Nancy Pelosi got a part in this. . to topple Trump. Lately President Trump was always telling of how GREAT American economy was improving in all fronts. The only way to destroy his vision of making AMERICA GREAT AGAIN is to create an economic havoc. Nancy Pelosi was unable to bring down Trump thru impeachment. ....so work along with China to destroy Trump by releasing a virus. Wuhan,s epidemic was a showcase. At the peak of the virus epidemic. ..

China's President Xi Jinping...just wore a simple RM1 facemask to visit those effected areas. As President he should be covered from head to toe.....but it was not the case. He was already injected to resist any harm from the virus....that means a cure was already in place before the virus was released.

Some may ask....Bill Gates already predicted the outbreak in 2015...so the chinese agenda cannot be true. The answer is. ..YES...Bill Gates did predict. .but that prediction is based on a genuine virus outbreak. Now China is also telling that the virus was predicted well in advance. ....so that its agenda would play along well to match that prediction. China,s vision is to control the World economy by buying up stocks now from countries facing the brink of severe ECONOMIC COLLAPSE. Later China will announce that their Medical Researchers have found a cure to destroy the virus. Now China have other countries stocks in their arsenal and these countries will soon be slave to their master...CHINA.

Just Think about it ...
The Doctor Who declared this virus was also Silenced by the Chinese Authorities...

Rant considering the latest Cyber attack and the news around it.

(A recap: a lot of Windows computers were infected with ransomware (due to security hole on Windows), which demanded 300$ in bitcoins to unlock data. After 3 days the price would double, and after 7 days the data was to be deleted)
1) In our country, one of the biggest companies was attacked (car factory). The production stopped and they got for around 1 000 000€ damage in less than 24h (1300 people without work). The news said that they were attacked because they are such a big company and were charged more, as the hackers "knew who they were dealing with" - another reason being the fact that the text was in croatian (which is our neighbor country), but noone realized that it is just a simple google translate of english text - which is obviously not true. The hackers neither know nor care who is hacked, and will charge everyone the same. They only care about the payment.
2) In UK whole (or large part) of medical infrastructure went down. The main thing everyone was saying was: "Nobody's data is stolen". Which, again, is obvious. But noone said anything about data being deleted after a week, which includes pretty much whole electronic medical record of everyone and is pretty serious.
And by the way, the base of the ransomware is code which was stolen from NSA.
All that millions and millions of dollars of damage could be avoided by simply paying the small fee.
The only thing that is good is that (hopefully) the people will learn the importance of backups. And opening weird emails.
P.S. I fucking hate all that 'hacky thingys' they have all over the news.

developer makes a "missed-a-semicolon"-kind of mistake that brings your non-production infrastructure down.

manager goes crazy. rallies the whole team into a meeting to find "whom to hold accountable for this stupid mistake" ( read : whom should I blame? ).

spend 1-hour to investigate the problem. send out another developer to fix the problem.

... continue digging ...
( with every step in the software development lifecycle handbook; the only step missing was to pull the handbook itself out )

finds that the developer followed the development process well ( no hoops jumped ).
the error was missed during the code review because the reviewer didn't actually "review" the code, but reported that they had "reviewed and merged" the code

get asked why we're all spending time trying to fix a problem that occurred in a non-production environment. apparently, now it is about figuring out the root cause so that it doesn't happen in production.

we're ALL now staring at the SAME pull request. now the manager is suddenly more mad because the developer used brackets to indicate the pseudo-path where the change occurred.
"WHY WOULD YOU WASTE 30-SECONDS PUTTING ALL THOSE BRACES? YOU'RE ALREADY ON A BRANCH!"

PS : the reason I didn't quote any of the manager's words until the end was because they were screaming all along, so, I'd have to type in ALL CAPS-case. I'm a CAPS-case-hater by-default ( except for the singular use of "I" ( eye; indicating myself ) )

WTF? I mean, walk your temper off first ( I don't mean literally, right now; for now, consider it a figure of speech. I wish I could ask you to do it literally; but no, I'm not that much of a sadist just yet ). Then come back and decide what you actually want to be pissed about. Then think more; about whether you want to kill everyone else's productivity by rallying the entire team ( OK, I'm exaggerating, it's a small team of 4 people; excluding the manager ) to look at an issue that happened in a non-production environment.
At the end of the week, you're still going to come back and say we're behind schedule because we didn't get any work done.
Well, here's 4 hours of our time consumed away by you.

This manager also has a habit of saying, "getting on X's case". Even if it is a discussion ( and not a debate ). What is that supposed to mean? Did X commit such a grave crime that they need to be condemned to hell?

I miss my old organization where there was a strict no-blame policy. Their strategy was, "OK, we have an issue, let's fix it and move on."
I've gotten involved ( not caused it ) in even bigger issues ( like an almost-data-breach ) and nobody ever pointed a finger at another person.
Even though we all knew who caused the issue. Some even went beyond and defended the person. Like, "Them. No, that's not possible. They won't do such dumb mistakes. They're very thorough with their work."
No one even talked about the person behind their back either ( at least I wasn't involved in any such conversation ). Even later, after the whole issue had settled down. I don't think people brought it up later either ( though it was kind of a hush-hush need-to-know event )

Now I realize the other unsaid-advantage of the no-blame policy. You don't lose 4 hours of your so-called "quarantine productivity". We're already short on productivity. Please don't add anymore. 🙏

Have you ever been interrupted because a marketing workmate had a friend on the phone who needed advice on a WordPress hosting, and wanted your advice right now?

Because I have.

When we had a massive server failure and our production environment was down.

Seriously, what the fuck is wrong with people nowadays.

I don't understand how my managers suddenly forgot that my "down weeks" we're due to technical debt I inherited. The whole on boarding hasn't been in my favor. I've stayed at work everyday til long after work hours, digging through code, trying to get JIRA tickets done, encountering issues specific to our code base that no one would ever discover on their own without docs/help from the original dev. The whole time, I was told that they know what's going on and apologize. I constantly expressed that plenty of what we were doing was building on antipatterns. They acknowledged. When a ticket wasn't done, they always knew the very specific reason and I wasn't faulted. 6 months in, I receive a great annual review. 7 months in? I receive an email titled "Performance Discussion," detailing 4 of those incidents where a ticket was pushed back -- with inaccurate depictions of what actually went down. They actually wrote that I didn't communicate. One part of the report expressed that there were "bugs found in production due to inadequate test coverage." WTF!! Everything made it past code review and QA. What are you talking about?? In fact, the person who wrote that merged my code in each time!!!! Insane!! Anyway, Q2 is partly about cleaning up technical debt, which is a responsibility I have been vested (fantastic). I've deleted about 800 lines of code in the last 2 weeks and added plenty of doc strings. Two of the most important modules our application works from are about 1000 lines of JavaScript each without any comments/docs. I'm changing that, but I don't know if my managers truly know the significance. Someone was recently promoted to my position but manually wrote out a sorting algorithm (specified numeric indexes and all); didn't do shit to earn it but breathe. And while they get more and more praise and responsibility, I'm over here stuck trying to prove myself and live up to why I assume they hired me. It's ridiculous. I love the company, but I'm not getting any sleep and I'm stressed out. It's only been about 7 months and I've been doing everything I can. Why is this happening? What am I doing wrong? I've been developing a recurring (physical) headache and ticks. My heart/chest area sometimes feels like it's lifting weights. I sound like an idiot, pushing so hard for a company that isn't mine, but I take so much pride in being in this position, and I'm so set on proving myself this early in my career (I'm 25).

Migrating to another sub domain for the 4th time.
First they say 'oh this sub domain is just for demo and you can freely test your code on it .
Few days later they say the same thing to another team and then :

- 'dude ! Is the website down because of your code ?'

- 'The code is not complete yet ! I've been adding some features! '

-'Is it possible that you migrate to another demo sub domain? Content production team wants to upload more content on that sub domain '

-'Sure ! Why not :('

My cousin a recent graduate who just joined a company told me that he took down production today.
I said "Child it is a rite of passage as a developer, you always remember your first time".

Ahh this brings back memories of my first time I took down production. I would often look at my email that whole week to see if I was fired.

I recently tried to apply the same data analytics rationale that I use at work to my personal life. This is not a rant, it is more like an data storytelling of an actual use case I would like some input on.

I set a goal - gotta thin up a bit and calm down my ticker - and got a (almost unreasonably expensive) field expert consultant to yell at me about it for a couple hours.

I unravel the metrics - there is like a million weight-related KPIs and most say nothing at all. I have never seen an non-infrastructure measurable subject that could not be resumed to 2-5 performance metrics. I got overall weight, how well my nine-years-old business suit fits me, heart rate, and day-after relative muscle pain (it will make sense soon).

Then its data-pipeline time. I bought a cheap weight scale and smartwatch, and every morning I input the data in an app. Yes, I try to put on the suit every morning. It still does not fit.

After establishing a baseline, I tried to fit different approaches. Doing equipment-free exercises, going to the gym, dieting. None was actually feasible in the long run, but trying different approaches does highlight the impacts and the handling profile of each method.

Looking at the now-gathered data, one thing was obvious - can't do dieting because it is not doable to have a shopping list and meals for me and another for the family.
Gym is also off the table - too much overhead. I spend more time on the trip there and back than actually there.
And home exercise equipment is either super crappy or very expensive. But it is also the most reasonable approach.

So it is solutions time. I got a nice exercise bycicle (not a peloton), an yoga mat (the wife already had that one) and an exercise program that uses only those two resources. Not as efficient without dieting, not as measurable and broad as the gym, but it fits my workflow. Deploy to production!

A few months pass and the dataset grows. The signal is subtle but has support - it works! The handling, however, needs improvement, since I cannot often enough get with the exercise program. Some mornings are just after some hard days.

I start thinking about what else I can improve in the program, but it is already pretty lean and full of compromises.
So I pull an engineer and start thinking about the support systems and draft profile. What else could be draining my willpower and morning time?

Chores. Getting the kids ready for school, firing up the moka pot, setting the off-brand roomba, folding the overnight-dried clothes, cooking breakfast, doing the dishes, cleaning the toilets. All part of my morning routine. It might benefit from some automation.

Last month I got that machine our elders call "wasteful" and "useless crap lazy entitled Americans invented because they feel oh-so-insulted for simply doing something by hand like everyone always did" - a "dish-washer".
Heh, I remember how hard was to convince my mother-in-law that an remote-controled electric garage door would not make she look like an spoiled brat.
Still to early to call, but I think that the dishwasher just saved me about 25 mins every morning. It might be enough to save willpower for me to do more exercise.

This is all so reflective of all data analytics cases really are out in the wild - the analytics phase seems so small compared to the gathering and practical problem-solving all around. And yet d.a. is what tells you that you are doing the wrong thing all along. Or on what you should work next.

Fucking shit for brains authors that think the digital world is a fantasy realm where everything can happen just to aid their story. Out of boredom i watched "scorpion" today, a tv series about a group of geniusses which are a special case task force.
They got a visitor from the government saying the servers from the federal reserve bank were encrypted with ransomware. I already twitched when they said the economic system would collapse if the servers were left inoperational for a few days. Then one guy got to his desk and "hacked" the fed network to check... he then tried to remove the malware but "it changed itself when observed". But they got the magical fingerprint of the device that uploaded it. In the end some non-programmers created the malware, but it is super fast and dangerous because it runs on a quantum computer which makes it hyper fast and dangerous. They got to the quantum computer which was a glowing cube inside another cube with lasers going into it and they had to use mirrors to divert the lasers to slow down that quantum thingy. And be careful with that, otherwise it explodes. In the end the anti-malware battled the malware and won, all in a matter of minutes.

This is a multimillion hollywood production. How can a movie this abusive to computer science even air on television? Shit like this is the reason people still think the cyberworld is some instable thing that can explode any second. It's not, it's an instable thing that can break down any second. I remember "ghost in the wires" and people had surreal imaginations about the internet already. Shit like this is why people stay dumb and think everything can be done in seconds. If i ever should encounter one of these idiots i tell him i have an app that can publish his browser history by taking a picture of his phone and watch his reaction.
Time to shuw down the tv and learn vim again.

Long time ago i ranted here, but i have to write this off my chest.

I'm , as some of you know, a "DevOps" guy, but mainly system infrastructure. I'm responsible for deploying a shitload of applications in regular intervals (2 weeks) manually through the pipeline. No CI/CD yet for the vast majority of applications (only 2 applications actually have CI/CD directly into production)

Today, was such a deployment day. We must ensure things like dns and load balancer configurations and tomcat setups and many many things that have to be "standard". And that last word (standard) is where it goes horribly wrong

Every webapp "should" have a decent health , info and status page according to an agreed format.. NOPE, some dev's just do their thing. When bringing the issue up to said dev the (surprisingly standard) answer is "it's always been like that, i'm not going to change". This is a problem for YEARS and nobody, especially "managers" don't take action whatsoever. This makes verification really troublesome.

But that is not the worst part, no no no.

the worst is THIS:

"git push -a origin master"

Oh yes, this is EVERYWHERE, up to the point that, when i said "enough" and protected the master branch of hieradata (puppet CfgMgmt, is a ENC) people lots their shits... Proper gitflow however is apparently something otherworldly.

After reading this back myself there is in fact a LOT more to tell but i already had enough. I'm gonna close down this rant and see what next week comes in.

There is a positive thing though. After next week, the new quarter starts, and i have the authority to change certain aspects... And then, heads WILL roll on the floor.

My killer PR:
I made a small feature and it was to be merged in my senior's branch.
So I made it, resolved all the conflicts and when it got merged it broke my senior's branch.
This is nothing new and it happens, so me and him sat down and got it working. After two days, his branch got merged and broke it's parent 😂, after investigation my code was the reason, got fixed and merged.
Same thing happened 3 more times, and every time my code was the culprit.
Now for staging we thoroughly tested everything and deployed it, after running for 2 hours the piece of shit broke again 😂😂😂.
A meeting was arranged for scolding the team, and after the meeting the architect comes to me and say "how did you manage to fuck things on so many levels, literally?". I handled it like a pro with an awkward laugh.

We exhaustively checked it for production. Deployed, it did not broke, we were happy. After a month of successful run, I just joked about the above incident while we were working on next release on morning coffee. That cursed thing broke on the same evening. I was like 😮

I started off in a MNC company as a junior developer. I entered with candy glasses.
I didn't expect to win the lottery. Of getting abuse by superior.

I stayed for a year, at the project. Constantly being belittled by this team lead. It was awful i enter as a fresh grad. All the new tech were so new and scary at that point.

During my time there, i constantly think that developer is not my stuff.

Ultimately i reach the state of burnout. I reached out to the manager and broke down in his office.

I actually told the manager. "I hate coding"

I remember staying up to 4am just complete a piece of program. To be ready to be push to production the next day. My team lead just come screaming at me saying there is bug.

Upon receiving that message via skype. I broke, tears flow down my eyes.

After which i reach a state of burn out. I start to reach out to external parties for help to get me out of there.

Now i am recovered from the burn out. I am curious of the technology that were utilized in that project. I literally face palm. After understanding the technology it isn't so hard after all. I just didn't gear myself up with the tech.

I still do enjoy working on code.

I help maintain software that services thousands of users across millions of dollars of infrastructure.

My resolve is stronger than steel during a production outage.

Plex goes down for 1 minute at home and my toddler loses it, and I'm a fucking wreck.

Wtf is wrong with me?!

I was experimenting with a load test suite called 'Siege' to build and scale increasingly complex searches against our new site search engine. I assumed that an old iMac couldn't have generated a crushing load against the beta servers and I learned two things the day I wrote and started that script before heading to lunch:

1) Beta and Production shared MSSQL instances
2) That single iMac was more than enough to take the whole production site down...

Most intense day for me was at the very start of my career. Internship... went with product manager to client's office while PM installed new test version of our product for on-site integration testing. Shortly after deploy, client manager came over to ask why production had gone down...

Turns out that manually typing DB name as part of deployment script is not, erm, risk free. PM entered production DB name and took out a very busy call centre for a few hours. Agents in tears, customers raging on phones, etc! After we restored and got everything back up and running, he reached me the keyboard and said "You're doing it this time."

My attempt was problem free, thankfully. Earned many brownie points that day.

Twitter is down. Someone has fucked up production.

At a startup where the software was built haphazardly because the developer thought he'll lifelong be the sole maintainer. The dude antagonized me at every turn and refused to help with familiarising with his code. He eventually left majority of the work for me, and dedication to work continued to dwindle until he threw in the towel

After his departure, we surprisingly grew fond of each other, discussing code concepts at length. He was in the habit of refusing to read any of the articles I sent him, or answer open ended questions citing the claim that they require thinking and he was busy. I didn't take any of this to heart

But it accumulated and I deleted his number. I didn't bear him any ill wishes but it wasn't respectful to myself for him to remain in my space. Some day, I was looking for a point raised during our conversations and went rummaging through our chats. Going down memory lane opened scars I'd long forgotten. I was embarrassed to see the way I forgot all about it. I should never have had anything to do with someone like that

He contacted me for a favour just less than a year after I deleted his contact. I didn't even think of declining. But this evening, I randomly remembered how he saw a defect in my code, promised me that the code will fail in production and resisted all pleas to point out what it was. I don't know if I hate him for his dastardly acts. What I feel deepest is sadness/bitterness that I got to experience all that

This week we had a live production issue that our staff were catching/fixing on the fly. We're a relatively small software team without any direct external customers, so this is not too unusual.

Unfortunately, the person in charge of dealing with these issues didn't resolve it during the work week, so we were stuck with it over the weekend. Said responsible employee left at 2:30 on Friday without figuring out how we'd deal with the problem without any staff in the office to intercept problem cases. Better yet, he drove all the way back, and was there from 3:30 to 4 and promptly left again without telling the rest of the team what was going on with the production issue. We asked how it happened, what it was, etc, but didn't focus on his fix (in hindsight, a mistake).

Since it's his job, I assumed that he would let us know what was up before he left on Friday. It turns out that he never addressed the production issue at all and just decided to leave.

A junior developer and I spent two hours contacting management (who, at this time are already at home with their families) to get clearance to either shut the system off or fix it. No one wants to give it and no one that's high enough up to approve the decision is available.

In the end, we asked the weekend mechanical support team (some friends of mine) to monitor the issue and they kindly accepted.

All of this could have been avoided if my coworker had either told us his plan earlier (so we could ask about the lack of coverage), gotten approval to shut it down for the weekend, or covered his own ass before he left for the day.

Ugggh. I get that we all make mistakes, but I really hope this guy shapes up soon.

Comsci teacher just ran "php artisan migrate:fresh" on the production database

Dang. I feel like I'm just not cut out to climb any ladder.

When we discovered a production bug. I feel bad about making people working on that part look bad by not catching it.

My manager has no issue with pointing out that I should have caught it. Beating a horse while it's down.

I mean no shit. Of course I know I should've caught it. How does making me feel worse about it help.

Feels like I'll always be in a tough spot no matter where I am on the ladder.

Or I'm just fragile. I acknowledge that, too

Does anyone work on a team with multiple stacks?

For example we have batch jobs in Java but also have a JS front-end and APIs.

How do you divide the developers and the work across these projects?

Currently everyone does everything but I feel like this is inefficient and hard to develop expertise. And different people or even the same person will make the same mistakes over and over again because they don't know how to do X or they forget or overlook some quirk. When I switched Beck to JS took me like a week to get a Promises nailed down again. And this morning someone else had a production bug and couldn't figure it out. But when I looked at the code I could pretty much see where an issue could be (uncaught exception in a promise)

Also the testing frameworks are very different and there's a lot of infrastructure technical debt, things that really should've been done a long time or fixed but no one had the time or expertise to do it or notice it (until it causes a production issue and then everyone is like WTF is happening??!!!!).

I'm not the manager but I always feel that the team needs to be split along the language lines and specific people need to own these projects to review and code changes for all these common newbie errors. And also developer enough expertise to foresee problems before it becomes a production issue.

Three days ago my focus was shifted from a development role to a support role. I was shifted to replace another support guy who had used fraud to get the position. I have no experience with this role but there was decent KT and I'm catching on fine. During onboarding and KT I'm serving as the first contact for new tickets and whatnot...

Today I got a ticket with an error on our production instance that no one had ever seen before. It prevented the guy from using our service entirely. I tried to reproduce it and... I couldn't use the service either. No one could. Everything was down. I could see the sweat building on my manager's forehead.

Thankfully another member on my team has done a bit of support before, so we collaborated with each other and other teams throughout the day to figure out what's wrong and how to fix it. I'm listening to them chat remotely as we speak - so far I've been working on it 9 hours straight.

This service is used by everyone - it's a business critical service with due dates on actions and escalations to managers... Imagine if the support ticketing service for your company crashed. That means a lot of people are asking what's wrong, requiring extensions, etc. I've been answering to managers and seniors in the business throughout the day.

The best part? We figured out why the server went down, and the reason is fantastic: someone updated the server's code without telling anyone, and all they had done was remove critical parsing code. Just took it right out, pushed, redeployed. We don't know who did it or who even has access to do that. I guess I have some detective work cut out for me after we've fixed everything that was broken by that.

I miss coding already.

I really really hope that no one post this,a friend texted it to me and I wanted to share it because made my day.

Idk where it comes, so feel free if know where this came from to post it:

//FUN PART HERE
# Do not refactor, it is a bad practice. YOLO

# Not understanding why or how something works is always good. YOLO

# Do not ever test your code yourself, just ask. YOLO

# No one is going to read your code, at any point don’t comment. YOLO

# Why do it the easy way when you can reinvent the wheel? Future-proofing is for pussies. YOLO

# Do not read the documentation. YOLO

# Do not waste time with gists. YOLO

# Do not write specs. YOLO also matches to YDD (YOLO DRIVEN DEVELOPMENT)

# Do not use naming conventions. YOLO

# Paying for online tutorials is always better than just searching and reading. YOLO

# You always use production as an environment. YOLO

# Don’t describe what you’re trying to do, just ask random questions on how to do it. YOLO

# Don’t indent. YOLO

# Version control systems are for wussies. YOLO

# Developing on a system similar to the deployment system is for wussies! YOLO

# I don’t always test my code, but when I do, I do it in production. YOLO

# Real men deploy with ftp. YOLO

So YOLO Driven Development isn’t your style? Okay, here are a few more hilarious IT methodologies to get on board with.

*The Pigeon Methodology*
Boss flies in, shits all over everything, then flies away.

*ADD (Asshole Driven Development)*
An old favourite, which outlines any team where the biggest jerk makes all the big decisions. Wisdom, process and logic are not the factory default.

*NDAD (No Developers Allowed in Decisions)*
Methodology Developers of all kinds are strictly forbidden when it comes to decisions regarding entire projects, from back end design to deadlines, because middle and top management know exactly what they want, how it should be done, and how long it will take.

*FDD (Fear Driven Development)*
The analysis paralysis that can slow an entire project down, with developments afraid to make mistakes, break the build, or cause bugs. The source of a developer’s anxiety could be attributed to a failure in sharing information, or by implicating that team members are replaceable.

*CYAE (Cover Your Ass Engineering)*
As Scott Berkun so eloquently put it, the driving force behind most individual efforts is making sure that when the shit hits the fan, you are not to blame.

Just merge this pull request in production and all service gonna work fine, trust me :D

If I had a nickel every time the unit tests failed not because something was wrong in the code, but because someone had messed up the unit test I'd be able to retire early.

I just spent the better part of 10 hours hunting down a bug in some production code only for the test to be wrong because the person who wrote it had mocked the http response incorrectly.

Nothing I did to "fix" the code worked, because nothing was wrong with it...

A loooong time ago...

I've started my first serious job as a developer. I was young yet enthusiastic as well as a kind of a greenhorn. First time working in a business, working with a team full of experienced full-lowered ultra-seniors which were waiting to teach me the everything about software engineering.

Kind of.

Beside one senior which was the team lead as well there were two other devs. One of them was very experienced and a pretty nice guy, I could ask him anytime and he would sit down with me a give me advice. I've learned a lot of him.

Fast forward three months (yes, three months).

I was not that full kind of greenhorn anymore and people started to give me serious tasks. I had some experience in doing deployments and stuff from my other job as a sysadmin before so I was soon known as the "deployment guy", setting up deployments for our projects the right way and monitoring as well as executing them. But as it should be in every good team we had to share our knowledge so one can be on vacation or something and another colleague was able to do the task as well.

So now we come to the other teammate. The one I was not talking about till now. And that for a reason.

He was very nice too and had a couple of years as a dev on his CV, but...yeah...like...

When I switched some production systems to Linux he had to learn something about Linux. Everytime he encountered an error message he turned around and asked me how to fix it. Even. For. The. Simplest. Error. He. Could. Google. Up.

I mean okay, when one's new to a system it's not that easy, but when you have an error message which prints out THE SOLUTION FOR THE ERROR and he asks me how to fix it...excuse me?

This happened over 30 times.
A. Week.

Later on I had to introduce him to the deployment workflow for a project, so he could eventually deploy the staging environment and the production environment by hisself.

I introduced him. Not for 10 minutes. I explained him the whole workflow and the very main techniques and tools used for like two hours. Every then and when I stopped and asked him if he had any questions. He had'nt! Wonderful!

Haha. Oh no.

So he had to do his first production deployment. I sat by his side to monitor everything. He did well. One or two questions but he did well.

The same when he did his second prod deploy. Everythings fine.

And then. It. Frikkin. Begins.

I was working on the project, did some changes to the code. Okay, deploy it to dev, time for testing.

Hm.

Error checking out git. Okay, awkward. Got to investigate...

On the dev server were some files changed. Strange. The repo was all up to date. But these changes seemed newer because they were fixing at least one bug I was working on.

This doubles the strangeness.

I want over to my colleague's desk.

I asked him about any recent changes to the codebase.

"Yeah, there was a bug you were working on right? But the ticket was open like two days so I thought I'll fix it"

What the Heck dude, this bug was not critical at all and I had other tasks which were more important. Okay, but what about the changed files?

"Oh yeah, I could not remember the exact deployment steps (hint from the author: I wrote them down into our internal Wiki, he wrote them done by hisself when introducing him and after all it's two frikkin commands), so I uploaded them via FTP"

"Uhm... that's not how we do it buddy. We have to follow the procedure to avoid..."

"The boss said it was fine so I uploaded the changes directly to the production servers. It's so much easier via FTP and not this deployment crap, sorry to say that"

You. Did. What?

I could not resist and asked the boss about this. But this had not Effect at all, was the long-time best-buddy-schmuddy-friend of the boss colleague's father.

So in the end I sat there reverting, committing and deploying.

Yep

It's soooo much harder this deployment crap.

Years later, a long time after I quit the job and moved to another company, I get to know that the colleague now is responsible for technical project management.

Hm.

Project Management.

Karma's a bitch, right?

I have had it with the new pm for my project. We've had to literally write down rules for him to follow on how we want to run our project.
- we always go over time on standup because he's asking everyone specific questions about each ticket we're working on instead of just letting us give our updates. Wrote a rule for not doing that.
- he gets overly excited to get things done to the point that he's approved a PR himself and merged a production release during off hours.
- we're a team of 3 devs but he has 7 big items he wants us to work on concurrently.
- we use jira to keep track of things but he insists on us updating a spread sheet that he made as well. We just straight up told him no but he'll bring it 3 times every week.
- he wants us to write a report of our daily progress in our jira ticket before we give the same updates on standup
- every time I give him an estimate, he tells the clients it's a promise we'll get it done by x date.
- he never pushes back and says yes to everything
- oh here's a funny one, he's "reviewed" my code several times

Idk what to do here anymore. We've literally talked to him about all of these issues. He'll change for a few days and go back to doing whatever he does.

I'm considering quitting a job I started a few weeks ago. I'll probably try to find other work first I suppose.

I'm UK based and this is the 6th programming/DevOps role I've had and I've never seen a team that is so utterly opposed to change. This is the largest company I've worked for in a full time capacity so someone please tell me if I'm going to see the same things at other companies of similar sizes (1000 employees). Or even tell me if I'm just being too opinionated and that I simply have different priorities than others I'm working with. The only upside so far is that at least 90% of the people I've been speaking to are very friendly and aren't outwardly toxic.

My first week, I explained during the daily stand up how I had been updating the readmes of a couple of code bases as I set them up locally, updated docker files to fix a few issues, made missing env files, and I didn't mention that I had also started a soon to be very long list of major problems in the code bases. 30 minutes later I get a call from the team lead saying he'd had complaints from another dev about the changes I'd spoke about making to their work. I was told to stash my changes for a few weeks at least and not to bother committing them.

Since then I've found out that even if I had wanted to, I wouldn't have been allowed to merge in my changes. Sprints are 2 weeks long, and are planned several sprints ahead. Trying to get any tickets planned in so far has been a brick wall, and it's clear management only cares about features.

Weirdly enough but not unsurprisingly I've heard loads of complaints about the slow turn around of the dev team to get out anything, be it bug fixes or features. It's weird because when I pointed out that there's currently no centralised logging or an error management platform like bugsnag, there was zero interest. I wrote a 4 page report on the benefits and how it would help the dev team to get away from fire fighting and these hidden issues they keep running into. But I was told that it would have to be planned for next year's work, as this year everything is already planned and there's no space in the budget for the roughly $20 a month a standard bugsnag plan would take.

The reason I even had time to write up such a report is because I get given work that takes 30 minutes and I'm seemingly expected to take several days to do it. I tried asking for more work at the start but I could tell the lead was busy and was frankly just annoyed that he was having to find me work within the narrow confines of what's planned for the sprint.

So I tried to keep busy with a load of code reviews and writing reports on road mapping out how we could improve various things. It's still not much to do though. And hey when I brought up actually implementing psr12 coding standards, there currently aren't any standards and the code bases even use a mix of spaces and tab indentation in the same file, I seemingly got a positive impression at the only senior developer meeting I've been to so far. However when I wrote up a confluence doc on setting up psr12 code sniffing in the various IDEs everyone uses, and mentioned it in a daily stand up, I once again got kickback and a talking to.

It's pretty clear that they'd like me to sit down, do my assigned work, and otherwise try to look busy. While continuing with their terrible practices.

After today I think I'll have to stop trying to do code reviews too as it's clear they don't actually want code to be reviewed. A junior dev who only started writing code last year had written probably the single worst pull request I've ever seen. However it's still a perfectly reasonable thing, they're junior and that's what code reviews are for. So I went through file by file and gently suggested a cleaner or safer way to achieve things, or in a couple of the worst cases I suggested that they bring up a refactor ticket to be made as the code base was trapping them in shocking practices. I'm talking html in strings being concatenated in a class. Database migrations that use hard coded IDs from production data. Database queries that again quote arbitrary production IDs. A mix of tabs and spaces in the same file. Indentation being way off. Etc, the list goes on.
Well of course I get massive kickback from that too, not just from the team lead who they complained to but the junior was incredibly rude and basically told me to shut up because this was how it was done in this code base. For the last 2 days it's been a bit of a back and forth of me at least trying to get the guy to fix the formatting issues, and my lead has messaged me multiple times asking if it can go through code review to QA yet. I don't know why they even bother with code reviews at this point.

Life begins when you bring down production and don’t know how to fix it

We basically don't unit test at work. I write some tests for my code and honest to God people complain I'm wasting time saying a test bed and manual tests are good enough. We don't write test beds for about half of our production code and rely on integration tests for the rest. We only test release builds which have been symbol stripped, I get handed a crash report with no stack trace that I'm unable to reproduce and expected to stay late to fix it for some arbitrary internal deadline.

I've since moved to R&D where basically I'm left to do my own thing so it's better.

We don't project manage. Project leads take time estimates and double them so management might cut them some slack. This doesn't matter because management made up time estimates before the project started. Last project I was on had a timeline of 3 months and took a year.

We have released broken products. Not that any of the above really matters, our software products have made about 50k revenue in 2 years. There are 6 people on software. Fortunately hardware has made about 3 mill. That said our hardware customers are getting frustrated with us as we keep fucking up, shipping broken products and missing deadlines.

I've been working there about a year and a half and will be looking for a job at the end of the current project.

I joined devRant about when I was most pissed off with my job, my rant frequency has definitely gone down since I moved over to R&D.

So glad to be staying a new job next week. Today a junior colleague asked me what the best way to test something would be as it won't work locally. Knowing this has a good chance of taking down the server, I suggest he sounds up a server on his AWS account. My manager comes in, oh no I don't want him doing it on AWS use the production server instead. By the time stuff States hitting the fan I will be gone.

Ran the build today 4:30 and found out our grunt file is missing some pretty critical error checks without even logging a warning. A dependency was unavailable and it was pushed to production. The site was down for 30+ minutes.

We are 3-4 days away from deployment to production. We are still bug fixing. But one coworkers decided this is the time to make a fuss about the way everything is set up. He doesn't like the dev database. So he knocks it over.. and while so doing it, he doesn't inform the team. And when I ask if something else is gonna knock over? No answer! (And something broke down too..)
Now we have issues to test our bugfixes. The whole thing took me half a day finding out and made me distracted with frustration, and not just for me. Most bugs could've been done in that half a day!
I so wanna punch the guy xD but no, I gotta save face, pfff!

A newly joined developer (who was supposed to be very senior) comes and asks me how to write a test cos for some reason the person didn't know how to mock.

In Java,
(same for any other implementation which has an interface)
Writes Arraylist list =.....
Instead of List list = Arraylist...

Deployed code (another engineer from another country helped to deploy since this new senior dev didn't have access yet.

But the new senior dev didn't update relevant files in production code which brought down the site for nearly an hour. Mistake aside, the first reaction from this new senior dev is 'WHY DIDN'T THE DEV THAT WAS HELPING DIDN'T DO THE FILE UPDATE?'

This was followed by some other complaints such as our branching stragies are wrong. When in fact the new senior dev made a mistake by just making assumptions on our git branching strategies and we already advised on correct process.

Out of all these, guess this is the best part. The senior dev never tested code locally! Just wrote code, unit test and send to QA and somehow the test passed through. I learnt this when I realised this dev... has not even set up the local environment yet.

I keep saying new but this Senior dev been around like 3 months! This person is in another team within our larger team but shares same code base. I am puzzled how do you not set up your environment for 3 months. Don't you ask for help if you are stuck? I am pretty sure the env is still not setup.

Am I over reacting or is this one disgusting developer who doesn't even qualify for an intern let alone a senior dev? It's so revolting I can't even bring myself to offer help.

Nothing better than walking out of the examination classroom, opening up your phone and being flooded with emails and messages about production servers being down

It's that time of the week again

Luckily, now that everything is in place everything is 100% automated with ci/cd 😎

Last year I was asked to optimize a code in our legacy portal (yet to be replaced with the new portal). The legacy system didn't have a design phase. Straight away went to development by whatever developer available at that time.
It was seriously fucked up.

So I went and had a look at the vanilla PHP code that served data for a datatable.
** I nearly fainted **

A query was done to get data from a table without any joins.
Then for loop to display those data.
Then inside for loop, for every single column that gets data from a related table there's a fucking query.
Eg: select * from users where.... to display username. Then again select * from users where..... to display user's email, then another query for his phone number. Then another query to get service providers name, then another to get their phone number.

I think the guy who did it wrote his first hello world app with a bunch of queries and sent it to production. No one bothered to check until 4 years later when it slowed down like a friggin snail.
I'm surprised it even survived that long.

The website I'm working on went down because of a 500 HTTP error and halted my company of working

I should not code in production

One of our servers had a disk fail this week. Luckily it's 1 of 3 in a RAID5 array. And, luckily, it was our mostly-dev box and didn't have any production stuff on it, except for some support things. We scheduled a disk replacement with the hosting company, took everything down, waited. Somebody at the hosting company apparently didn't know we'd scheduled the replacement, saw the machine was down, and brought it up again. Sigh. Finally they did the replacement, got it back up, but now we're seeing an ethernet port flapping, suggested they have someone go in and make sure all the jacks are fully seated, maybe one got loose when they were doing the disk switch. Bureacracy reared up again and we got the boilerplate "if there's a hardware issue suspected please boot into rescue mode and run the tests"... sigh...

Second week sick I see how my life slowed down and how meaningless everything around is, everyone is rushing about some bullshit, name it new amazing job opportunity, black Friday great deal, super duper product idea or some most important bug on production that we need to fix asap.

All that can’t wait a week when I’m healthy?
Seriously, people lost their minds in today’s world to some bullshit.

I’m to old and to depressed to care about such idiotic things. Living my life as I want and on my own peace, don’t care everyone is running, I’m slowly walking and I like it.

It’s better to walk straight than run around like an idiot.

I can't believe this shit happened in time for this week's rant!

Here it goes.

I have a table on AWS Athena which has partitions. Now, in the earlier versions of this project whenever I write something to a new partition a simple `MSCK` query worked (and keep in mind I am NOT deleting anything)!

Now, my so called Team Lead in the PR for the latest (major) release tells me to change it to an `ALTER TABLE`. I was like fine, but I did not add the s3 location to it, because it was NOT NEEDED. TL asks me to add location as well. I try to convince this person that it's not needed, but I lose. So there it is in production, all wrong.

Today I notice that the table is all fucked up. I bring this up in the stand up. The main boss asks me to look into it, which I do. Figure out what the issue is. This TL looks at it and says you need to change the location. I put my foot down.

"NO. What I need is to remove the bloody location. IT'S NOT NEEDED!"

TL's like, "Okay. Go ahead"

Two things:

1. It's your fault that there's this problem in production.
2. Why the fuck are you looking into this when I was clearly told to do so? It's not like you have nothing to do!

Finally made my node production server stable enough that I could focus on writing tests*. I start by setting up docker, mocking cognito, preparing the database and everything. Reading up on Node test suites and following a short tut to set up my first unit test. Didn't go smoothly, but it's local and there are no deadlines so who cares. 4 days later, first assert.equal(1+1, 2) passes and I'm happy.

I start writing all sorts of tests, installing everything required into "devDependancies," and getting the joy of having some tests pass on first try with all asserts set up, feels good!

I decide to make a small update to production, so I add a test, run and see it fail, implement the feature, re-run and, it passes!

I push the feature to develop, test it, and it works as intended. Merge that to master and subsequently to one of my ec2 production servers**, and lo and behold, production server is on a bootloop claiming it "Cannot find module `graphql`". But how? I didn't change any production dependencies, and my package lock json is committed so wth?

I google the issue, but can't find anything relevant. The only thing that I could guess was that some dependencies (including graphql) were referenced*** in both, prod and dev, and were omitted when installed on a prod NODE_ENV, but googling that specific issue yielded no results, and I would have thought npm would be clever enough to see that and would always install those dependencies (spoiler: it didn't for me).

With reduced production capacity (having one server down) I decided to npm uninstall all dev dependencies anyway and see what happens. Aaaaand it works.....

So now I have a working production server, but broken local tests, and I'm not sure why npm is behaving like this...

* Yes I see the irony.
** No staging because $$$, also this is a personal project.
*** I am not directly referencing the same thing twice, it's probably a subdependency somewhere.

I'm in a team of 3 in a small to medium sized company (over 50 engineers). We all work as full stack engineers.. but I think the definition of full stack here is getting super bloated. Let me give u an example. My team hold a few production apps, and we just launched a new one. The whole team (the 3 of us) are fully responsible on it from planning, design, database model, api, frontend (a react page spa), an extra client. Ok, so all this seems normal to a full stack dev.

Now, we also handle provisioning infra in aws using terraform, doing deployments, building a CI/CD pipeline using jenkins, monitoring, writing tests, building an analytics dashboard.

Recently our tech writer also left, so now we are also handling writing feature releases.

Few days ago, we also had a meeting where they sort of discussed that the maintenance of the engineering shared services, e.g. jenkins servers, (and about 2-3 other services) will now be split between teams in a shared board, previously this was handled only be team leads, but now they want to delegate it down.

And ofcourse not to mention supporting the app itself and updating bug tickets with findings.

I feel like my daily responsiblities are becoming the job responsibilities of at least 3 jobs.

Is this what full stack engineering looks like in your company? Do u handle everything from app design, building, cloud, ops, analytics etc..

This weeks question fits me well, as I am still unsure about the full details of how the fuck this all came together and was about to just rant about it anyway.

Ever since this companies network equipment and cabling has been updated, a lot of vital tools went down and bug out every now and then, at seemingly random times.

The codebase is a horrible mess to begin with and random things execute at random times and at random places spread all over different resources that get random hooks from random physical values etc.

Turns out (or at least what it so far seems like) all of them somehow sync their clock and other variables based on how many (valid-?) requests it gets per measured time and similar oddities, so when the network equipment got updated, that meant that multiple processes now could reach each other much faster and therefore threw off thousands of values and internal clocks.

There's a total of like 600 systems that are all "separate" from each other but all need to communicate in-sync for the production chain to properly work. Thankfully I didn't sign anything yet, so might actually just redirect them to somebody else, I am not ready to age 20 years, even for the amount that would pay.

I was working under the title of mid-level developer when I felt I was clearly still a junior and was tasked with making changes to the core search functionality of the company website. After it was 'reviewed' by 3 people more senior than I and merged to production it took the majority of the site down.

I got the blame and the reviewers weren't even questioned 😒

I left for a week and someone deployed my code to production after being completely tested by qa on a Friday night. I get into the office after being gone for a week and am told that production has been down for many customers for several days. In a panic I start troubleshooting my code with the "What the fuck did I do wrong" face.

Development and qa were in a frenzy to figure out what happened, several developers were trying to figure out what went wrong by tracing through the source code for days, fucking days!

In that while time Noone thought to roll back the code. So, I was in a bind and thought "might as well get a box". Before that I looked at the deployment instructions: only the dll's were pushed, no db or resource file changes were pushed. In 20 minutes after I got back: no more problems for any customers and everything is working fine.

SMDH.

At least I found this picture of turtles wearing raspberries.

When I started doing frontend development, I was quite shocked with how people managed to cowboy code their way into building fully functional products with a decent paying client base.

I am talking about fully function SaaS with payment gateway and all, but no version control beyond full backup copies, and spaghetti code everywhere you can literally bring the website down trying to change the homepage design.

... and the startups that managed to do better, some of them forgot the .git on production exposing their entire source code *facepalm*

Every meeting that contains one or more of the following points:

- "I don't think it belongs in the meeting, but"
- "Didn't get the meeting notes"
- "When's the food coming?"
- "I know we've said no technical discussion, but..."
- "Why is he so strict, this is no fun meeting at all :("
- "I think it's unfair to include risk assessment, you blame US before XY is finished"
- "The admins / the Team XY / ZX didn't talk with us, so we don't talk with him / her / them..."
- "Why are we here?"
- "Why is it so bad when production is down?"
- "I didn't know we do security / audit checks... Why hasn't anyone told us?"
- "Not happening. I'm against it"
- "I don't want to work with XY - he doesn't do it like I want it"

...

I could add thousand more things here.

I had countless meetings where I really thought that I was an alien who got broadcasted in a comedy reality TV soap...

Woke up in the middle of the night thinking about work and how the team seems to be always a few steps away from the next production issue and well always busy with urgent work too so that the crap that produces more and more tech debt never get cleaned or fixed...

And now it's grown so big... The bad habits are just sparking more bad habits and well the only person (boss) able to correct course still hasn't realized for the last 4 years... Constantly thinking things will get better after the next sprint. Hell we don't even use proper sprint planning... even I can't keep up anymore and can never get any long term high value/low immediate return work done...

So I guess I'm having a work overload, nervous breakdown before even going back to work...

I have an urge to tell all this to his boss and have him give him a wake-up slap or maybe bring in a more experienced/veteran manager to set the ship right but my boss personally is a very nice guy so don't want to rat him out...

So not really sure now what to do other than maybe just stay in my lane and put up the blinders? And let the whole forest around be burn down... Though I still gotta bear the heat till it all dies down by itself...

Can't say when that is though...

TL;DR The "senior dev", that the client hired on their end, is acting as a middleman between me and the project requestors. Taking the credit for my work.

I've already bitch about this before. I've been in a crusade to defend the production server from this fraud for a long time now.

But most recently he has removed me from all meetings with the actual project owner. I create the solutions, then he goes through them to understand it a bit. He proceeds to present it to the project owner in a way that almost blatantly says that he made it.

I'm sick and tired of working with this asshole. He is literally useless, worse he is slowing things down and breaking others.

I'm just gonna begin countering this...

Let's see what's on the menu today:

* Web Application Catastrophe Special *

Includes, but not limited to:
- Orphaned server processes in the configuration management cluster
- Microservice back-end architecture with no API documentation
- Poorly implemented cache microservice with no documentation
- Stale data causing everything to be shown as down in production, despite everything running fine

Cost: 1 developer's sanity

Spent all morning debugging legacy code that I need to migrate.

Most of the time is just waiting for it to load --pieces of data-- entire tables from the database and then filter out the records it doesn't want using some app logic.

WHAT SORT OF MONKEY WRITES CODE LIKE THIS? HOW WAS THIS EVEN ALLOWED INTO PRODUCTION...

I have to open Notepad to write down my chain of thoughts, steps, and things to check once the next breakpoints are hit so that I don't forget them.

So in theory I'm being paid all morning to sit around and do nothing.

That sounds great but I'm falling asleep... Shoulda worked from home...

What was I saying again...yea...

DON'T HIRE MONKEYS!!! THEY WRITE SHIT CODE THAT WASTES EVERYONE'S TIME EVEN AFTER THEY LEAVE...

I'm going to lunch now... Hopefully Notepad has enough into for me to remember what I was doing...

I am so sick of this place. Production deployment shouldn't be such a massive cluster fuck of different departments and sometimes redundant configurations. Knowing who to talk to about a problem shouldn't change daily, and it shouldn't be hard to figure out.

My jvm doesn't appear to be running in production. I hunt down the fucker who is supposed to be handling this problem today...but he's out of office and no one knows who his stand in is...GODDAMNIT!

Well, this happens time to time...

I'm freelancing as a backend guy. I like to take care of all infrastructure before really starting to build anything, this mostly includes dev/staging/prod environments with some linear promotion strategy. So.. I did this API. Still on staging, proceeding with the development as planned, everything goes according the timeline.

And then.. this happens... At some point PM told frontend guy that it's time for production (without notifying me), so the frontend guy does what "anyone" would do in this case - tells PM to create DNS record for production to point to staging app.

Time passes, I'm still unaware of this. But I'm starting to see some quality entries in the DB, not the usual QA crap. I write to them that they're doing good job and continue with my tasks.

One of the tasks required some major DB change. I could've written migrations script, but since we're not in "production" yet, I just wipe the DB and recreate schema as I need it.

In 10 minutes the furious PM starts shouting that "production" is down and I need to fix it ASAP.

I'm lost, I'm asking questions, I'm slowly understanding what's happening...

So I want to grab some coffee, sat back down, wrote politely that they suck, added a finger emoji and terminated the contract.

Felt like the right thing to do as I definitely don't want to continue within the same "team".

I was working for a project with one of the project managers. Despite several discussions, he was not ready to have provisioned for procurement of couple of extra drives for database backups. Also because it's always how they worked, developers were allowed to make changes to the production databases directly.

Since I knew it was going to be burning some day, despite his negligence, I ran a script to take full database backups every night, compress, and remove old backups all to do in the drives we had on server. Sat it automated using scheduler.

One day it happened that one of the junior developers deleted one major table taking whole production down. Next thing you know everyone went crazy. Since I felt bad for the managers and users, I was able to restore database using backup from last night.

You know who jumped in first before senior management to take credit of all this and got some nice kudos..that project manager. Also, you know who got burned..it would not be a rant if I did not got schooled for not following on the wisdom of project manager.

Anyways, we are still not taking database backups (as per project manager)

Spaghetti codes makes a person feel frustrated, non-productive even slightly depressed, and even do refractoring takes time of production and slows things down in the short term there's nothing like that "new feeling " after you successfully re-organize and re-structure a project.

The skies seem more blue the motivation re-appears and new ideas make the project feel worth it of your time again

Out of the frying pan, into the fire:
So in my first job, I thought it's just us operating so crazy: meddling with arcane C/C++ code from the 80's, shooting our code to production without testing, fixing hundred of customers data base entries by hand, letting an intern alter some core component (to have more logging) and directly push it to prod...
Silly me.
I mean I suspected, that maybe it's not only this tiny little company acting wild, that also the bigger companies with all their ISO certified processes, agile blabla, professional tooling whatsoever - will also have their skeleton in the closet,.. like some obscure assembler part buried in the heart of your code base nobody dares to touch...
How Pieter Hintjens asked about the state of the industry and all the fads so bluntly put it:
"It's all bullshit."
But we are humans, so we better jump on the bandwagon if we want to keep our jobs... and somehow try to keep that trashy house of cards from crashing down.

36 or sth..
Trying to fix production for xy client without them noticing someone fucked up everything with the previous deploy.. (wasn't me)..
Anyhow, managed to deploy my changes plus fix for the previous fuckup.. in the morning it all worked as it should.
Why it took me so long? Because why bother writing down what changeset was used for deploy.. It's much more fun to guess.. Multiple times.. Anyhow, I managed to figure approximate code for that deploy & merge my changes & fix everything.. + later found out looooads of uncommited changes on the guys computer.. :/ So yeah, never trust a bunneh!!

Production goes down because there's a memory leak due to scale.

When you say it in one sentence, it sounds too easy. Being developers we know how it all goes. It starts with an alert ping, then one server instance goes down, then the next. First you start debugging from your code, then the application servers, then the web servers and by that time, you're already on the tips of your toes. Then you realize that the application and application servers have been gradually losing memory over a period of time. If the application is one that don't get re-deployed ever so often, the complexity grows faster. No anomaly / change detection monitor can detect a gradual decrease of memory over a period of months.

I actually learnt this last year but here I go in case someone else steps into this shit.

Being a remote work team, every other colleague of mine had some kind of OS X device but I was working this Ubuntu machine.

Turns out we were testing some Ruby time objects up to a nanosecond precision (I think that's the language defaults since no further specification was given) and all tests were green in everyone's machine except mine. I always had some kind of inconsistency between times.

After not few hours of debugging and beating any hard enough surface with our heads, we discovered this: Ruby's time precision is up to nanoseconds on Linux (but just us on OS X) indeed but when we stored that into PostgreSQL (its time precision is up to microseconds) and retrieved it back it had already got its precision cut down; hence, when compared with a non processed value there was a difference. THIS JUST DOES NOT HAPPEN IN OS X.

We ended up relying on microseconds. You know, the production application runs on Ubuntu too. Fuck this shit.

Hope it helps :)

P.s.: I'm talking about default configs, if anyone knows another workaround to this or why is this the case please share.

Long post, TLDR: Given a large team building large enterprise apps with many parts (mini-projects/processes), how do you reduce the bus-factor and the # of Brent's (Phoenix Project)?

# The detailed version #

We have a lot of people making changes, building in new processes to support new flows or changes in the requirements and data.

But we also have to support these except when it gets into Production there is little information to quickly understand:
- how it works
- what it does/supposed to do
- what the inputs and dependencies are

So often times, if there's an issue, I have to reverse engineer whatever logic I can find out of a huge mess.

I guess the saying goes: the only people that know how it works is whoever wrote it and God.

I'm a senior dev but i spend a lot of time digging thru source code and PROD issues to figure out why ... is broken and how to maybe fix it.

I think in Agile there's supposed to be artifacts during development but never seen em.

Personally whenever i work on a new project, I write down notes and create design diagrams so i can confirm things and have easy to use references while working.

I don't think anyone else does that. And afterwards, I don't have anywhere to put it/share it. There is no central repo for this stuff other than our Wiki but for the most part, is like a dumping ground. You have to dig for information and hoping there's something useful.

And when people leave, information is lost forever and well... we hire a lot of monkeys... so again I feel a lot of times i m trying to recover information from a corrupted hard drive...

The only way real information is transferred is thru word of mouth, special knowledge transfer sessions.

Ideally I would like anything that goes into PROD to have design docs as well as usage instructions in order for anyone to be able to quickly pick it up as needed but I'm not sure if that's realistic.

Even unit tests don't seem to help much as they just test specific functions but don't give much detail about how a whole process is supposed to work.

Name two production service, metrics and logging included, after a famous woman and an armored vehicle.

Dude, no. When those services go down in the middle of the night some poor soul on call duty will have to handle it without the faintest idea wtf is going on.

This has been bothering me for a while. I have an old freelance client of mine I’ve created an web site for (his company) it was small one so I took the complete payment before deployment and I needed no contract. I deployed the complete version of the site on my server, bought the domain for his company under my name and it has been running for a year now.
Lately he had asked me to give admin privileges to his son (cs student 1y) to upload some photos of their new building. I noticed he ruined several functions on the site in doing so, but I was never paid to support that just the hosting for a year.
When I was making the design I made a simple but pretty logo as a placeholder for the site which went in production since they never gave me company logo. All good, no contract small cash all delivered, everyone happy.
Up until few days when I saw my f**king logo cut out from the site as 250px jpeg and made as a huge banner on the company building..
From my pov I would’ve never given permission to use that since its not something i’m proud of and would suggest to make a better one for a fee. I see this as stolen/unauthorized use of intellectual property. But the laws are super shitty in our country so at this point I am stuck at taking their site, domain a hostage until they pay for the logo they used or take it down or taking legal actions.. we never signed anything about that logo.

So here's what I'm putting up with for the last 6 months, clients..

A client proposed to me a project he had in mind. Project is pretty solid, could have a bright future. Since they didn't have the money to spend, we agreed on a % of the income they will earn from the project. So, let's say I get 20% of the income in exchange for building the application. I didn't receive any down payment or payment of any kind.

Just for info, project is a Web application/portal and it is ~80% done at the moment. Client provided a logo and a wireframe/ideas/pictures how he sees the project. I built everything, from DB to Frontend. Also, project is completely custom made, no CMS or anything. Project will make profit by subscription base, every user of the project pays.

For various reasons, we did not yet sign a contract. So, what is my issue...

Client sent me his proposal of the contract, said it's solid stuff, just sign it. In the contract, it stated that he owns the application in full, can sell it, etc. and I get % of the price. There were also other sneaky parts about me having all the responsibility but owning nothing. I naturally declined and took a lawyer to construct a normal contract.

My proposal was/is, I own the application(source code) in full. They are obligated to pay the monthly percentage and can use the application normally and make profit. At any time, application can be bought by the client if they pay for the development. So, basically, they are getting the application to use "for free" with no initial payment/investment. And this is a long term deal, they can use is as this as long as they want. Also, if they go bankrupt at any time, no penalty or payment is needed, the risk is mine.

The client refused and what he claims is the following...

His share in the project is 80%, mine is 20%. If project is to be sold, I get 20% of the price. So, meaning, if we go to production tomorrow, if I want to buy his share, I have to buy 80% of the application I built entirely. Also he is convinced that by "telling me" what to built he's owning everything. In his words, he dictated me the notes and I'm just playing the violin.

I am having trouble explaining to him that he is getting the application to use and make profit basically for free and cannot and does not own the source code unless he buys it off. We are going in circles, I send him the contract to review, he changes it and returns it back. Also, he removes the parts where it is clearly states what he provided and what was done by me.

So, we kind off agreed on the authorship but in the case we break the contract he wants to be able to use the application for 3 more years.

Was anyone here in a similar situation? How do you handle this kind of situations?

It was a shit idea to try and get async to work in my first ever real programm meant to be used in production.
Unit tests are a bitch to make.
And now everything just kinda falls down on its own.

At least i learned a ton and got a lot of firsts out of the way.

Working on production issue,
Kind of nervous checking logs and so on...
Ops manager and PO who were looking over my shoulder this whole time start shooting the breeze.

I know what they were trying to do. They are trying to create a relaxed environment.

But the issue is that the talk is very distracting. If you want to shoot the breeze please go somewhere else.

Anyway just did that, asked them to leave. They weren't happy about it. But I really needed the silence.

Was working on a high priority security feature. We had an unreasonable timeline to get all of the work done. If we didn’t get the changes onto production before our deadline we faced the possibility of our entire suit being taken offline. Other parts of the company had already been shut down until the remediations could be made -so we knew the company execs weren’t bluffing.

I was the sole developer on the project. I designed it, implemented it, and organized the efforts to get it through the rest of the dev cycle. After about 3 month of work it was all up and bug free (after a few bugs had been found and squashed). I was exhausted, and ended up taking about a week and a half off to recharge.

The project consisted of restructuring our customized frontend control binding (asp.net -custom content controls), integrations with several services to replace portions of our data consumption and storage logic, and an enormous lift and shift that touched over 6k files.

When you touch this much code in such a short period of time it’s difficult to code review, to not introduce bugs, and _to not stop thinking about what potential problems your changes may be causing in the background_.

Working from development environment. New API module works like a charm.

Migrated to production. Whole CRM breakes down.

Migraine intensifies.

Need some advice here.

So hello everyone! I recently moved abroad for work, for the sake of the experience and the excitement of learning how developers in Latin America tackle specific problems. To my surprise, the dev team is actually composed solely of Europeans and Americans.

I work for a relatively new startup with an ambitious goal. I love the drive everyone has, but my major gripe is with my team lead. He's adverse to any change, and any and all proposals made to improve quality of throughput are shot down in flames. Our stack is a horrendous mess patched together with band-aids, nothing is documented, there are NO unit tests for our backend and the same goes for our frontend. The team has been working on a database/application migration for about a month now, which I find ridiculous because the entire situation could have been avoided by following very rudimentary DevOps practices (which I'm shunned for mentioning). I should also add that for whatever reason containerization and microservices are also taboo, which I find hillarious because of our currently convoluted setup with elastic beanstalk and the the constant complaints between our development environment and production environments differing too much.

I've been tasked with managing a Wordpress site for the past 3 weeks, hardly what I would consider exciting. I've written 6 pages in the past two weeks so our marketing team can move off of squarespace to save some money and allow us more control. Due to the shit show that is our "custom theme" I had to write these pages in a manner that completely disregard existing style rules by disabling them entirely on these pages. Now, ironically they would like to change the blog's base theme but this would invertedly cause other pages created before I arrived to simply not work, which means I would have to rewrite them.

Before I took the role of writing an entire theme from scratch and updating these existing pages to work adequately, I proposed moving to a headless wordpress setup. In which case we could share assets in a much more streamline manner between our application and wordpress site and unify our styles. I was shot down almost immediately. Due to a grave misunderstanding of how wordpress works, no one else on the team seems to understand just how easy it is to fetch data from wordpress's api.

In any event, I also had a tech meeting today with developers from partner companies and realized no one knew what the fuck they were talking about. The greater majority of these self proclaimed senior developers are actually considered junior developers in the United States. I actually recoiled at the thought that I may have made a great mistake leaving the United States to look a great tech gig.

I mean no disrespect to Latin America, or any European countries, I've met some really incredible developers from Russia, the Ukraine, Italy, etc. in the past and I'm certainly not trying to make any blanket statements. I just want to know what everyone thinks, if I should maybe move back to the states and header over to the bay/NY. I'm from the greater Boston area, where some really great stuff is going on but I guess I also wanted a change of scenery.

Oddly enough, i have simultaneously been less busy and more productive since working 66% remotely.

I find myself with more time that feels "wasted" or not busy, but my metrics show that I have more production, better results, and far nicer documentation. A bunch of us also sat down and did a bunch of coursework on really putting together a domain script library for one click onboarding of new servers or new client setups. We spun up a bunch of new virtual environments that literally solved headaches that had existed for years that never got dealt with because of too many other tickets.

Some of our web clients freaked out at us because the business is moving away from doing maintenance of legacy web work (small to midsize businesses). But it didn't matter. Rather than respond with a "make them happy," the response was "well, we will get rid of them as clients. We need to focus our energy on the essential service sectors we support."

Hell, we even got an automated test that has been broken apparently since 2018 to work again.

Granted, the incoming workload has slowed down. But it's still interesting to me to see that despite the slowdown, there isn't any concern; its still paying the bills and we are getting rid of technical debt everywhere. Tbh, this has really been a good reality check.

This one time last year a colleague found out that some data went missing and suggested to recover the data from a backup. When trying to create a new database instance in the Google Cloud Platform (if everything works it's amazing!) it failed.
Not knowing why this happened, I tried to revert that backup to the production database, after creating a backup using the GCP. Needless to say that failed as well, resulting in a corrupted database instance where I couldn't access the created backups anymore.
This all went at around 10pm and the only users of our product are currently in the same timezone and use it from around 7.30AM until 6PM so no one besides our team knew the server was down.
After a long night chatting Google's support team the database was successfully recovered and the only harm done was sleep depravation for me and a colleague.

Apparently there was a bug in the GCP. It was resolved in two hours and the last time a breaking bug was in that piece was more than seventy days earlier.

I did at least learn to create local backups as well, instead of relying on the tools of the same product...

Best: the moment I saw the corrupted database spin up again and not losing my job because of it.

"And would any of ya folks be kind enough to write down a small script doin that?"

Yeah, sure... No! That would require us to make testing API keys, write connection logic, test it, debug it, test it again, et.c.

Just because it's simple doesn't mean it takes 0-time. I've learned that by listening to an idea of yours previously and sent months being your personal closed-source programmer for a huge-ass project you later abandoned before production. Go away.

A very long rant.. but I'm looking to share some experiences, maybe a different perspective.. huge changes at the company.

So my company is starting our microservices journey (we have a 359 retail websites at this moment)

First question was: What to build first?

The first thing we had to do was to decide what we wanted to build as our first microservice. We went looking for a microservice that can be used read only, consumers could easily implement without overhauling production software and is isolated from other processes.

We’ve ended up with building a catalog service as our first microservice. That catalog service provides consumers of the microservice information of our catalog and its most essential information about items in the catalog.

By starting with building the catalog service the team could focus on building the microservice without any time pressure. The initial functionalities of the catalog service were being created to replace existing functionality which were working fine.

Because we choose such an isolated functionality we were able to introduce the new catalog service into production step by step. Instead of replacing the search functionality of the webshops using a big-bang approach, we choose A/B split testing to measure our changes and gradually increase the load of the microservice.

Next step: Choosing a datastore

The search engine that was in production when we started this project was making user of Solr. Due to the use of Lucene it was performing very well as a search engine, but from engineering perspective it lacked some functionalities. It came short if you wanted to run it in a cluster environment, configuring it was hard and not user friendly and last but not least, development of Solr seemed to be grinded to a halt.

Elasticsearch started entering the scene as a competitor for Solr and brought interesting features. Still using Lucene, which we were happy with, it was build with clustering in mind and being provided out of the box. Managing Elasticsearch was easy since there are REST APIs for configuration and as a fallback there are YAML configurations available.

We decided to use Elasticsearch since it provides us the strengths and capabilities of Lucene with the added joy of easy configuration, clustering and a lively community driving the project.

Even bigger challenge? Which programming language will we use

The team responsible for developing this first microservice consists out of a group web developers. So when looking for a programming language for the microservice, we went searching for a language close to their hearts and expertise. At that time a typical web developer at least had knowledge of PHP and Javascript.

What we’ve noticed during researching various languages is that almost all actions done by the catalog service will boil down to the following paradigm:

- Execute a HTTP call to fetch some JSON

- Transform JSON to a desired output

- Respond with the transformed JSON

Actions that easily can be done in a parallel and asynchronous manner and mainly consists out of transforming JSON from the source to a desired output. The programming language used for the catalog service should hold strong qualifications for those kind of actions.

Another thing to notice is that some functionalities that will be built using the catalog service will result into a high level of concurrent requests. For example the type-ahead functionality will trigger several requests to the catalog service per usage of a user.

To us, PHP and .NET at that time weren’t sufficient enough to us for building the catalog service based on the requirements we’ve set. Eventually we’ve decided to use Node.js which is better suited for the things we are looking for as described earlier. Node.js provides a non-blocking I/O model and being event driven helps us developing a high performance microservice.

The leap to start programming Node.js is relatively small since it basically is Javascript. A language that is familiar for the developers around that time. While Node.js is displaying some new concepts it is relatively easy for a developer to start using it.

The beauty of microservices and the isolation it provides, is that you can choose the best tool for that particular microservice. Not all microservices will be developed using Node.js and Elasticsearch. All kinds of combinations might arise and this is what makes the microservices architecture so flexible.

Even when Node.js or Elasticsearch turns out to be a bad choice for the catalog service it is relatively easy to switch that choice for magic ‘X’ or component ‘Z’. By focussing on creating a solid API the components that are driving that API don’t matter that much. It should do what you ask of it and when it is lacking you just replace it.

Many more headaches to come later this year ;)

After redhats totally dick move, who would actually stay with openshift paas? Theyll just take v2 down with a month notice and think no one is affected. They could do that to v3 too one day, they clearly dont think about the customers. I guess openshift isnt for production after all, only for fun.

I have 3 jobs but one salary... I look after some financial enterprise system as one of my jobs ... In short am an integration support specialist, earlier today system was down ussd gateway... Some client pinged me on fucking skype asking for integration support on uat i ignored cause i was busy fixing production issues , the nigga escalated me to the CTio, sneakey bastard.. Some clients are shitty. Hope they make money

First let me start this rant by saying: Don't use SharePoint lists as your primary data store if you can avoid it. You're gonna have a bad time.

My coworkers and I work on a system where we need to pull tons of data down from a SharePoint site and run various algorithms and operations on it. Generate reports, that sort of thing. This is all done in the browser using a Typescript React SPFX webpart. Basically using SharePoint as a DB/DAL.

Because of the sheer amount of data we end up pulling down (our system in production is the single source of truth for one of the largest companies in Canada, and they're currently building a pipeline as we speak), in order to maintain a reasonable speed while using it, we have some pretty intense caching logic implemented, logic that ensures we get new items when new items are detected, and merges changes to already exisiting objects. It's pretty brilliant, and that's before we even consider the custom paging that my coworker implemented in order to get around the IndexedDB max size of 100MB.

Well that's all well and good, and works great in production, but it is a horror to work with. Because EVERYTHING we touch on the server is cached locally, it can be IMPOSSIBLE to detect data anomalies, be they local or server side -.- You don't know how many hours I have completely WASTED fixing a "bug" that didn't really exist... Just incorrect data in the cache

See this? Dont ever fucking tell me to use mongodb. Half the time this website is fucking broken and down. Imagine you have a production ready app that is now also fucked because this piece of shit is fucked. I always viewed anyone using mongodb as DUMB fking shitass script kiddy with a small dick helping bill gates to gangrape some kids at the epstein island

Wait. Why does this work? It doesn't copy any of the frontend code into the deploy location.

I'm not sure how this works, but it does. Crap, there goes my morning tracking down this wretched spaghetti deploy code.

At least I understand how it works in production. Shit, why is it different between production and our integ servers ,that isn't good. Maybe I can just refactor it.

That was all on Monday. It's now Wednesday and I'm still fucking refactoring something that wasn't actually broken. It just didn't make sense.

Maybe I should just revert my last three days of work on this branch and move on. No! It's too late, I've invested way too much time into this project...

... and I'm almost done, just a few more commits right?

Recently many of us may have seen that viral image of a BSOD in a Ford car, saying the vehicle cannot be driven due to an update failure.
I haven't been able to verify the story in established news sources, so I won't be further commenting on it, specifically.
But the prospects of the very concept are quite... concerning.

Deploying updates and patches to software can be reasonably called *the software industry*. We almost have no V0 software in production nowadays, anywhere (except for some types of firmware).
Thus, as car and other devices become more and more reliant on larger software rather than much shorter onboard firmware, infrastructure for online updates becomes mandatory.
And large scale, major updates for deployed software on many different runtime environments can be messy even on the most stable situations and connections (even k8s makes available rolling updates with tests on cloud infrastructure, so the whole thing won't come crashing down).
Thereby, an update mess on automotive-OS software is a given, we just have to wait for it.

When it comes... it will be a mess. Auto manufacturers will adopt a "move fast and break things" approach, because those who don't will appear to be outcompeted by those who deploy lots of shiny things, very often.
It will lead to mass outages on otherwise dependable transportation - private transportation.
Car owners, the demographic that most strongly overlaps with every other powerful demographic, will put significant pressure on governments to do something about it.
Governments (and I might be wrong here) will likely adapt existing recall implementation laws to apply to automotive OS software updates.
That means having to go to the auto shop every time there is a software update.
If Windows may be used as a reference for update frequency, that means several times per day.
A more reasonable expectation would be once per month.
Still completely impossible for large groups of rural car owners.
That means industry instability due to regulation and shifting demographics, and that could as well affect the rest of the software industry (because laws are pesky like that, rules that apply to cars could easily be used to reign in cloud computing software).

Thus... Please, someone tells me I overlooked something or that I am underestimating the adaptability of the powers at play, because it seems like a storm is on the horizon, straight ahead.

They tried to mark him, they even tried compacting him and his children, but this old generation instance is not going down without a fight. He’s in a big heap of trouble, and he’s running out of time. You better count your references, because this summer’s stop-the-world event will have you staying at work all night: Memory Leak: Production is in your code NOW

So I've worked a bit too hard today, some thing with production being down...

Does anyone have some sort of relax/wind down pattern they don't mind sharing?

My current one is watching youtube the rest of the evening, but I am interested what everyone else does to calm their minds after a long workday

Dealing with government bureaucracy today. Prepare for pure anger.

First of all, what fucking dipshit site does testing and maintenance in production without letting users know? Bitch I'm getting an invalid date error when I use your own stupid date selector and I had to waste the office lady's time asking about it because you couldn't be arsed to either test your shit properly or actually take it down if it's broken. Who made that stupid ass decision and why the fuck did nobody question it. Fuck you.

Gah, I just received this Ubuntu 18.04 VM with 8 cores and 8 gigs of ram, and since it'll be a production server both serving public and "private" networks (yes, shout at me, but projects won't be about hosting sensitive information, I wouldn't put all that on one server), and I'm struggling between my options.

Docker, or not docker?

The server's main use is to host our growing blog and install Varnish, which will hog some ram after a while. I use Laradock for my dev projets, it's really easy to develop with it, but I am unsure if it fits a production environment with performance, security and traffic load in mind :(

I read Docker has stability issues (in 2016-2017), and can bring the machine down with it, I don't know if I should just install the software (nginx, apache, percona/mysql/maria) without "containerizing" it and go for it

I'm lost xD

A dev decided to overwrite the master branch with his code saying its better. That it fixes the major bugs that all of us couldn't solve.

Against my better judgement of firing him, I decided to test it.

Firing up the testing site, we made test databases to use and we went to house.

In the middle of testing, I noticed the test DBs weren't being changed. While everyone was still testing, I looked at the code. It wasn't made to test on any databases, it was specifically designed for the actual production server.

However the damage was done. In a secret dashboard in the code, someone sent instructions to drop the tables, effectively ruining the production server.

We had the dev go to an offline backup site that only went online every 10 minutes a day to make new backups. So we shut down the production server, setup a maintenance page. I get my ass chewed out again, and we were sitting ducks.

I don't think the dev had enough punishment, so I grabbed his laptop and made a full backup of his data, and locked the SSD in a safe.

I downloaded a Windows 98 and put it on a flash drive. And installed it all on his SSD. The dev is now a proud (pirate) owner of Windows 98.

He came back and started balling on his desk. We all looked at him with a pity, but he deserved it.

I'll give him the drive on Monday.
Do you think he learned his lesson?

the servers are down and I'm the only one working in the office because I have a lot to do that doesn't require pushing to production for some time...fml