There was a problem with a server we were staging on, and I was providing DevOps help remote.

As a joke I said, "haha if you run `sudo rm -rf / --no-preserve-root` everything will be fixed!"

They ran it. RIP server-kun 2016-2018 💨

Oh, man, I just realized I haven't ranted one of my best stories on here!

So, here goes!

A few years back the company I work for was contacted by an older client regarding a new project.

The guy was now pitching to build the website for the Parliament of another country (not gonna name it, NDAs and stuff), and was planning on outsourcing the development, as he had no team and he was only aiming on taking care of the client service/project management side of the project.

Out of principle (and also to preserve our mental integrity), we have purposely avoided working with government bodies of any kind, in any country, but he was a friend of our CEO and pleaded until we singed on board.

Now, the project itself was way bigger than we expected, as the wanted more of an internal CRM, centralized document archive, event management, internal planning, multiple interfaced, role based access restricted monster of an administration interface, complete with regular user website, also packed with all kind of features, dashboards and so on.

Long story short, a lot bigger than what we were expecting based on the initial brief.

The development period was hell. New features were coming in on a weekly basis. Already implemented functionality was constantly being changed or redefined. No requests we ever made about clarifications and/or materials or information were ever answered on time.

They also somehow bullied the guy that brought us the project into also including the data migration from the old website into the new one we were building and we somehow ended up having to extract meaningful, formatted, sanitized content parsing static HTML files and connecting them to download-able files (almost every page in the old website had files available to download) we needed to also include in a sane way.

Now, don't think the files were simple URL paths we can trace to a folder/file path, oh no!!! The links were some form of hash combination that had to be exploded and tested against some king of database relationship tables that only had hashed indexes relating to other tables, that also only had hashed indexes relating to some other tables that kept a database of the website pages HTML file naming. So what we had to do is identify the files based on a combination of hashed indexes and re-hashed HTML file names that in the end would give us a filename for a real file that we had to then search for inside a list of over 20 folders not related to one another.

So we did this. Created a script that processed the hell out of over 10000 HTML files, database entries and files and re-indexed and re-named all this shit into a meaningful database of sane data and well organized files.

So, with this we were nearing the finish line for the project, which by now exceeded the estimated time by over to times.

We test everything, retest it all again for good measure, pack everything up for deployment, simulate on a staging environment, give the final client access to the staging version, get them to accept that all requirements are met, finish writing the documentation for the codebase, write detailed deployment procedure, include some automation and testing tools also for good measure, recommend production setup, hardware specs, software versions, server side optimization like caching, load balancing and all that we could think would ever be useful, all with more documentation and instructions.

As the project was built on PHP/MySQL (as requested), we recommended a Linux environment for production. Oh, I forgot to tell you that over the development period they kept asking us to also include steps for Windows procedures along with our regular documentation. Was a bit strange, but we added it in there just so we can finish and close the damn project.

So, we send them all the above and go get drunk as fuck in celebration of getting rid of them once and for all...

Next day: hung over, I get to the office, open my laptop and see on new email. I only had the one new mail, so I open it to see what it's about.

Lo and behold! The fuckers over in the other country that called themselves "IT guys", and were the ones making all the changes and additions to our requirements, were not capable enough to follow step by step instructions in order to deploy the project on their servers!!!

[Continues in the comments]

Roughly 180 days, 5 months and 29 days, 4,320 hours, 259,200 minutes, I devoted myself to a client project. I missed family outings with my daughter and my wife. People started asking my wife if we had broken up. My daughter became accustomed to daddy not being around and playing with her. Sometimes only sleeping 4 hours, I would figure out solutions to problems in my sleep and force myself to wake and put them into action. My relationship with my wife became very fragile and unstable. I knew I had to change but I just needed a little bit more time to complete this client project.

Finally, the project was ending there was light at the end of the tunnel. I “git add –-all && git status” everything looked good. I then “git commit -m “v1.0 release candidate && git push beanstalk master”

I deployed the app to the staging server where I performed my deployment steps. Everything was good. I signed-up as a new user, I upload a bunch different files types with different sizes, completed my profile and logged out. I emailed the client to arrange a time to speak remotely.

“Hello” says the client “How are you” I replied. “Great, lets begin” urged the client. I recited the apps url out to the client. The client creates a new account and tries to upload a file. The app spews a bunch of error messages on the screen.

The client says

“Merlin – I do not think you really applied yourself to this project. The first test we do and it fails. If you do not have the time to do my project properly please just say so now, so I can find somebody else who can”

I FREAKED THE FUCKOUT on the client!!!!!!! and nearly hung up. My wife was right next to and she was absolutely gobsmacked. I sat back and thought to myself “These fuckers don’t get it”. All that suffering for nothing!

Thanks for reading my rant….

BTW: I did finish the project, the client was amazed on how the app worked and it is has become an indispensable tool for their employees.

Dev: "Ah, I finally fixed that code I was working on the other day and got it pushed to staging!"

Almond: "Ah, great! What was the issue in the end?"

Dev: "It was an odd one - it wasn't actually my code that was the issue, there was a bunch of other code getting in the way."

Almond: "How do you mean?"

Dev: "It kept complaining about something called a "unit test" failing - so after a while I found the right unit tests, deleted them, and now it works great!"

Almond: "..."

Somehow I feel like I personally owe Linus for git.

17:50 Colleague whispers "fuck" and the entire project we've worked on for the last half year responds with 404.

17:55 A quick diagnosis shows that she wrote "rm - rf ../" instead of "./" when she threw out her staging dir an thereby deleted everything.

17:58 git pull, everything is back.

18:15 everything is configured and we're up and running again.

**Alternative Timeline without Version control **

17:58 We start looking through Backup folders

18:20 We're fairly confident to have found the most up to date Backup in /var/backup/newback/v2/june/new/released/ and start copying back into the project directory.

19:30 Some files are missing we start patching shit up.

19:40 I realize how much work went down the drain and start strangling my colleague. The Api seems to do the most important things again.

20:00 My colleagues dead body is hidden and I'm 80% confident that the tasks depending on us should run.

Next day: They didn't run. Every nightly build failed, nobody can do anything useful.

A week later : Shits starting to work again, all lost files are replaced. Replacement for dead colleague still missing though.

It's moments like this that make you really appreciate the luxurys we have nowadays...

User: *Clicks on staging environment*

Giant Warning Dialog: YOU ARE CURRENTLY ENTERING THE STAGING ENVIRONMENT

Users: Ok

App: *Completely different colour, I’m talking bright unsightly yellow*

User: Ok

Giant Yellow and Red Flashing Banner at the Top of the Screen: WARNING YOU ARE CURRENTLY USING STAGING, THIS AREA IS FOR TESTING ONLY

User: The production environment sure is acting strange today. It’s a weird colour and I don’t recognize any of the data, it’s all just dummy filler data. I better create a ticket for the dev team to check o—….. no wait I’ll send an email CC everyone including the CEO and sound the alarm production is currently down and filled with giant warning messages.

Manager: OH MY GOD PRODUCTION IS DOWN DID YOU HEAR ABOUT THIS??? WHAT THE FUCK COULD THESE WARNING MESSAGES BE THAT’S ONLY SUPPOSED TO HAPPEN ON STAGING! THE CEO IS BREATHING DOWN MY NECK YOU NEED TO GET THIS FIXED IMMEDIATELY!!!!!!!

Dev: …

!rant
!!git

Who here uses `master` for development?

My boss (api guy) tried to convince me that was normal practice. I gently told him that it sounded crazy and very very bad.

Here's the dev path I'm enforcing on my repos:

(feature branches) -> dev -> qa* -> master -> production*

*: the build server auto-pulls from these branches, and pushes any passing builds to staging/production.

Everyone works on their own feature branches, and when they're happy with their work, they merge it into `dev`. `dev`, therefore, is for feature integration testing. After everything is working well on `dev`, it gets merged into `qa` for the testers to fawn over and beat with sticks. Anything that passes QA gets merged into `master`, where it sits until we're ready to release it. When that time comes (it's usually right away, but not always), `master` gets merged into `production`.

This way, `master` is always stable and contains the newest code, so it's perfect for forking/etc. Is this standard practice, or should I be doing something different?

Also, api guy encourages something he calls "running a racetrack" -- each dev has their own branch (their initials) and they push to that throughout the day. everyone else pulls from it regularly and pushes to their own branch. When anyone's happy with their code, they push from their (updated) branch to `qa` (I insisted on `dev` instead.)

Supposedly this drastically reduces the number of merge conflicts when pushing to an upstream branch due to having a more recent ancestor node?

I don't quite follow that, but it seems to me that merging/pushing throughout the day would just make them happen sooner? idk.

What are your thoughts?

Took yesterday off to sort out a new passport.
Today on the stand:

Manager: "So we've been trying to get app X running on a dev environment for client X but we couldn't expose it to them"

Me: "Well yeah it's a dev environment if you want to give them one give them access to staging"

Manager: "Oh well we're still going to give them access to dev because they asked for it. It's due for 10am but we couldn't get it to run. You have to get it running since we edited the config files"

*accessing dev environment*
half of config files is missing, random files committed to the repo, SSL certs manually edited, eth0 down and found swan vpn installed.

never taking a day off again.

At one of my former jobs, I had a four-day-week. I remember once being called on my free Friday by an agitated colleague of mine arguing that I crashed the entire application on the staging environment and I shall fix it that very day.

I refused. It was my free day after all and I had made plans. Yet I told him: OK, I take a look at it in Sunday and see what all the fuzz is all about. Because I honestly could fathom what big issue I could have caused.

On that Sunday, I realized that the feature I implemented worked as expected. And it took me two minutes to realize the problem: It was a minor thing, as it so often is: If the user was not logged in, instead of a user object, null got passed somewhere and boom -- 500 error screen. Some older feature broke due to some of my changes and I never noticed it as while I was developing I was always in a logged in state and I never bothered to test that feature as I assumed it working. Only my boss was not logged in when testing on the stage environment, and so he ran into it.

So what really pushed my buttons was:

It was not a bug. It was a regression.

Why is that distinction important?

My boss tried to guilt me into admitting that I did not deliver quality software. Yet he was the one explicitly forbidding me to write tests for that software. Well, this is what you get then! You pay in the long run by strange bugs, hotfixes, and annoyed developers. I salute you! :/

Yet I did not fix the bug right away. I could have. It would have just taken me just another two minutes again. Yet for once, instead of doing it quickly, I did it right: I, albeit unfamiliar with writing tests, searched for a way to write a test for that case. It came not easy for me as I was not accustomed to writing tests, and the solution I came up with a functional test not that ideal, as it required certain content to be in the database. But in the end, it worked good enough: I had a failing test. And then I made it pass again. That made the whole ordeal worthwhile to me. (Also the realization that that very Sunday, alone in that office, was one of the most productive since a long while really made me reflect my job choice.)

At the following Monday I just entered the office for the stand-up to declare that I fixed the regression and that I won't take responsibility for that crash on the staging environment. If you don't let me write test, don't expect me to test the entire application again and again. I don't want to ensure that the existing software doesn't break. That's what tests are for. Don't try to blame me for not having tests on critical infrastructure. And that's all I did on Monday. I have a policy to not do long hours, and when I do due to an "emergency", I will get my free time back another day. And so I went home that Monday right after the stand-up.

Do I even need to spell it out that I made a requirement for my next job to have a culture that requires testing? I did, and never looked back and I grew a lot as a developer.

I have familiarized myself with both the wonderful world of unit and acceptance testing. And deploying suddenly becomes cheap and easy. Sure, there sometimes are problems. But almost always they are related to infrastructure and not the underlying code base. (And yeah, sometimes you have randomly failing tests, but that's for another rant.)

One of our clients deploy their own server app. So this happened after a prod deployment. (4am)

*Cellphone rings while sleeping*

Client : we need you on the conference call now. URGENT!

*Gets on conference call*
*Client explain the problem*
*Explaining to the client that the problem is in their side (https connection not working, either network or certificate problem)*
*Client doesn't believe it and pushes me for a fix that I have no control on*
*4 hours later in a heated conversation*

Client : ok problem is on our side. We used our SSL certificate from staging with production and thought it would work.

Me :

Private chat pops up. (- separator for new message)

Hello
- (1 min)
Can you help me?
- (2-3 mins)
Please it's urgeeeent!!!!!
- (1 min)
Come on you're online, I see the green dot.
- (5 mins)
Ok then I won't be able to work. Will write this down in the ticket.

- (15 mins) - new private chat pops up
Hi, we need to talk.
- (3 mins)
Regarding ticket XY, why aren't you responding? It's really urgent.
- (5 mins)
Please notify me as soon as you're available, it's really important!!!

- (20 mins, new private chat opens)
Hi mate, I think the devs are up to mischief. Said you're not reachable, I'll try to poke them with the stun gun.

- (60 mins, message in the official and only endorsed support room)
@all We broke staging, <Me> never responds and <Team mate who tried to use the stun gun> wasn't helpful either.
We really need this now!!!!!!!

- 30 mins later... la me:
@all I was in a meeting with the stakeholders as we had an priority meeting... What was so important that you not only ignored the rule of not messaging privately and even ignored <team mate>s instructions?

- 5 mins later, answer
no need to be so unfriendly.... We broke staging as we had to test stuff out for next week's sprint review [something which is still 3 days away or sth like that]. We really need to take a look in the team at it and for that we must have staging working now!!!!
- (La me)
If you need it urgent now, you didn't plan ahead. And if you didn't plan ahead, you have to wait for others. The sprint review and all other important days are planned ahead for a reason.
- (Silence)

- (20 mins later, private chat, team lead)
Will you finally fix staging now?

- La me
If it could wait 3 hours now and you / your team ignored all netiquette, it can wait till next day, too. We had this discussion more than once, I don't think I need to explain this further.

(Silence)

All in all, the joys of communication...

Now the fun stuff is when this not only happens with 1 team, but many teams....

Having 35 - 40 private chats and chat window looking like a christmas tree thx to the immeasurable amount of notifications and colors... Yay...

Did I mention that I hate the ego some programmers have -.

With the other members of the team refusing to learn git and making changes directly to the staging server i get to write the commit messages for everyone.

Log:
UPDATE: *informative details *

UPDATE: mark made some changes

UPDATE: colin made the same changes as mark but different

UPDATE: andrew undid all colins updates to change one link and I had to add them back in, thank gawd I commit the night before

BUGFIX: andrew keeps changing the database host to localhost and uploading it without changing it back

UPDATE: we all hate andrew

Well. My life is over. I got punchy and accidentally deleted a client’s entire site. Backups and all. Wasn’t paying attention to which specific delete feature I was using and POOF. It’s all gone.

The worst part? I was actually just trying to delete a staging site where I was trying out some stuff to try to win a site redesign contract. Now it looks like I’ll be paying THEM for damages and/or building them a new site for FREE.

I’m so done.

Dev created a new service for our application that opened SQL connections. The catch? They never closed said connections. How this passed QA/review I've no idea, but after a few minutes on Staging the service would crash with "connection pool overflow".

I raised a urgent defect, and the devs "fix"? To up the maxConnectionPool to Int.Max!! Argh!

This is going to be a long rant, coz this is the only way to vent out my frustration against our tech head.

Yesterday, while our fucking twat tech head was playing around in company aws account, he terminated the production server. By mistake, apparently. Coz he doesn't know shit about server management. But that egoist ass won't admit and fucked the production server.

And then ran away. We developers sprang into action. Updated dns to point to staging server, setup virtual hosts, env files, point to prod database, force flush dns cache. All systems were up and running in 30 mins. And since it was staging server, it had lot of untested features and codes, and we spent rest of the day fixing the bugs.

And that tech head, who ran away hiding his tail between his legs, after he fucked the server, came back after systems were up. And started cracking jokes, that "so many features got released in 1 day" . "We cut server cost by shutting down 1 server."

We were struggling and working in full throttle to make the services running again. And that fuckity fucker was cracking jokes.

And I don't even know what excuse he gave to ceo for the downtime. I am pretty sure he would have made up some crappy excuse to hide his fucking mistake. That ass never admits his mistake. I am thinking to go to ceo today and tell the real story and get that faggot head fired or at least a strict warning.

Always the same story:

Marketing: hey I'm gonna do a demo to a customer. They were asking for feature XYZ. That's ready on thr staging server right? Do you think I could use the staging server for the demo?

Devs: well feature XYZ is not 100% done. Basically just feature X is done, and it still has a few bugs. The deadline ain't for another month, since we gotta finish ABC first. I guess you could use the staging, but it has a lot of bugs.

Marketing: perfect!

*after presentation*

Marketing: the staging had so many bugs! Why didn't you tell me?! It was so embarrassing showing it to new customers! Anyway, they loved the new feature. We need it to be ready ASAP.

Devs: What?! That's gonna mess up with our schedule. You know what? Fine, but feature ABC will have to wait another month.

Marketing: Well, it'd be ideal if we could do both...

Devs: Pay for more devs or dor extra hours.

Marketing: Just do XYZ. It's a pity that you'll have to push back ABC but it's fine, XYZ is more important.

(I might ask, if it was so important, why didn't you notice so in the meeting where we had decided that ABC would be prioritized?)

*tons of working hours later*

Devs: There, we finished XYZ.

Marketing: Yay! Wow, this month we'll have two major features done: ABC and XYZ!

Devs: No, ABC is not done yet.

Marketing: What? But the deadline was this week.

Devs: It was, but then you decided to prioritize XYZ and we said we had to push back ABC to get XYZ ready, and you agreed.

Marketing: Did we? Fine. But do it quick.

Marketing and their mood swings.

The riskiest dev choice...

How about "The riskiest thing you've done as a dev"? I have a great entry for that. and I suppose it was my choice to build the feature afterall.

I was working on an instance of a small MMO at a game company I worked for. The MMO boasted multiple servers, each of them a vastly different take on the base game. We could use, extend, or outright replace anything we wanted to, leading to everything from Zelda to pokemon to an RP haven to a top-down futuristic counterstrike. The server in this particular instance was a fantasy RPG, and I was building it a new leveling and experience system with most of the trimmings. (Talents, feats/perks, etc. were in a future update.)

A bit of background, first: the game's dev setup did not have the now-standard dev/staging/prod servers; everything ran on prod, devs worked on prod, players connected and played on prod, etc. Worse yet, there was no backup system implemented -- or not really. The CTO was really the only person with sufficient access. The techy CEO did as well, but he rarely dealt with anything technical except server hardware, occasionally. And usually just to troll/punish us devs (as in "Oops ! I pulled the cat5 ! ;)"). Neither of them were the most reliable of people, either. The CTO would occasionally remote in and make backups of each server -- we assumed whenever he happened to think of it -- and would also occasionally do it when asked, but it could take him a week, sometimes even up to a month to get around to it. So the backups were only really useful for retreiving lost code and assets, not so much for player data.

The lack of reliable backups and the lack of proper testing grounds (among the plethora of other issues at the company) made for an absolutely terrible dev setup, but that's just how it was, and that's what we dealt with. We were game devs, afterall. Terrible or not, we got to make games! What more could you ask for!? It was amazing and terrible and wonderful and the worst thing ever, all at the same time. (and no, I'm not sharing the company name, but it isn't EA or Nexon, surprisingly 😅)

Anyway, back to the story! My new leveling system also needed to migrate players' existing data, so... you can see where this is going.

I did as much testing and inspection of my code as I could, copied it from a personal dev script to the server's xp system, ... and debated if I really wanted to click [Apply]. Every time I considered it, I went back to check another part or do yet more testing. I ended up taking like 40 minutes to finally click it.

And when I did... that was the scariest button press of my life. And the scariest three seconds' wait afterwards. That one click could have ruined every single player's account, permanently lost us players ...

After applying it, I immediately checked my character to see if she was broken, checked the account data for corruption or botched flags, checked for broken interactions with the other systems....

Everything ended up working out perfectly, and the players loved all of the new features. They had no idea what went into building them, and certainly had no idea of what went into applying them, or what could have gone wrong -- which is probably a good thing.

Looking back, that entire environment was so fragile, it's a wonder things didn't go horribly wrong all the time. Really, they almost never did. Apocalypses did happen, but were exceedingly rare, and were ususally fixed quickly. I guess we were all super careful simply because everything was so fragile? or the decent devs were, at least. We never trusted the lessers with access 😅 at least on the main servers where it mattered. Some of the smaller servers... well, we never really cared about those.

But I'm honestly more surprised to realize I've never had nightmares of that button click. It was certainly terrifying enough.

But yay! Complete system overhaul and migration of stored and realtime player data! on prod! With no issues! And lots of happy players! Woooooo!

Thinking back on it makes me happy 😊

Day 1:

Me: 'Hi'

Middleware guy: 'Raise a Jira. We have always been so accommodating. Contact your manager.'

*Jeez*

===

Day 2:

Me: 'Could you please start the server in dev environment? I am a new joiner. I don't have access. Here is a jira.'

Middleware guy: 'Deadlines may be for you. It is not for me. Wait until tomorrow.'

*Damn, did he get a divorce recently?*

===

Day 5: *An urgent delivery asap* 'Hi, could you please do the configuration of the new app in staging?'

Middleware guy: 'So, here is the split up...
Task 1
Task 2
Task 3
Task 4
Task 5 & 6
Your app will be configured by tomorrow first half hopefully.. Oh and you can escalate if it is too late..
'
*What a b...*

===

Day 8:

Me: *Doing late sit for pushing a task* 'Hi, we have an issue. The server is not starting. Could it be due to..'

MW guy in 'second' shift: 'Oh, we don't extend support on unusual hours'

Me: 'But this is second shift.'

MW guy: 'Yeah, but I have to go home early now...'

====

Day 10:

Team Lead: 'Any innovative solutions?'

Me: 'Let's go SERVERLESS!' :D

It's enough. I have to quit my job.

December last year I've started working for a company doing finance. Since it was a serious-sounding field, I tought I'd be better off than with my previous employer. Which was kinda the family-agency where you can do pretty much anything you want without any real concequences, nor structures. I liked it, but the professionalism was missing.

Turns out, they do operate more professionally, but the intern mood and commitment is awful. They all pretty much bash on eachother. And the root cause of this and why it will stay like this is simply the Project Lead.

The plan was that I was positioned as glue between Design/UX and Backend to then make the best Frontend for the situation. Since that is somewhat new and has the most potential to get better. Beside, this is what the customer sees everyday.

After just two months, an retrospective and a hell lot of communication with co-workers, I've decided that there is no other way other than to leave.

I had a weekly productivity of 60h+ (work and private, sometimes up to 80h). I had no problems with that, I was happy to work, but since working in this company, my weekly productivity dropped to 25~30h. Not only can I not work for a whole proper work-week, this time still includes private projects. So in hindsight, I efficiently work less than 20h for my actual job.

The Product lead just wants feature on top of feature, our customers don't want to pay concepts, but also won't give us exact specifications on what they want.
Refactoring is forbidden since we get to many issues/bugs on a daily basis so we won't get time.
An re-design is forbidden because that would mean that all Screens have to be re-designed.
The product should be responsive, but none of the components feel finished on Desktop - don't talk about mobile, it doesn't exist.
The Designer next to me has to make 200+ Screens for Desktop and Mobile JUST so we can change the primary colors for an potential new customer, nothing more. Remember that we don't have responsiveness? Guess what, that should be purposely included on the Designs (and it looks awful).
I may hate PHP, but I can still work with it. But not here, this is worse then any ecommerce. I have to fix legacy backend code that has no test coverage. But I haven't touched php for 4 years, letalone wrote sql (I hate it). There should be no reason whatsoever to let me do this kind of work, as FRONTEND ARCHITECT.

After an (short) analysis of the Frontend, I conclude that it is required to be rewritten to 90%. There have been no performance checks for the Client/UI, therefor not only the components behave badly, but the whole system is slow as FUCK! Back in my days I wrote jQuery, but even that shit was faster than the architecuture of this React Multi-instance app. Nothing is shared, most of the AppState correlate to other instances.

The Backend. Oh boy. Not only do we use an shitty outated open-source project with tons of XSS possibillities as base, no we clone that shit and COPY OUR SOURCES ON TOP. But since these people also don't want to write SQL, they tought using Symfony as base on top of the base would be an good idea.
Generally speaking (and done right), this is true. but not then there will be no time and not properly checked. As I said I'm working on Legacy code. And the more I look into it, the more Bugs I find. Nothing too bad, but it's still a bad sign why the webservices are buggy in general. And therefor, the buggyness has to travel into the frontend.

And now the last goodies:
- Composer itself is commited to the repo (the fucking .phar!)
- Deployments never work and every release is done manually
- We commit an "_TRASH" folder
- There is an secret ongoing refactoring in the root of the Project called "_REFACTORING" (right, no branches)
- I cannot test locally, nor have just the Frontend locally connected to the Staging webservices
- I am required to upload my sources I write to an in-house server that get's shared with the other coworkers
- This is the only Linux server here and all of the permissions are fucked up
- We don't have versions, nor builds, we use the current Date as build number, but nothing simple to read, nonono. It's has to be an german Date, with only numbers and has always to end with "00"
- They take security "super serious" but disable the abillity to unlock your device with your fingerprint sensor ON PURPOSE

My brain hurts, maybe I'll post more on this shit fucking cuntfuck company. Sorry to be rude, but this triggers me sooo much!

This will be a long rant.

Met this person through my brother's friend. He wanted a cheap static website with a few products images for his handicraft store. I agreed as it would not take much time. He paid me in full upfront.

I registered the domain and created all the page templates with dummy contents. I uploaded the site under a staging sub-domain and asked him for the contents, but he said he was a bit busy and would give them to me later on.

After almost 2 years, out of nowhere, he got back to me and told me he didn't want that old website anymore but he wanted a new personal website. What shocked me was that he told me that he would only pay me for the domain and space but won't pay me for the website as his pervious work was never complete. I denied straight forward and told him I would keep the 80% amt for my work done but he kept telling me I hadn't done anything except place a few images in some html files. Hahaha, After a few in and out mails I told him I didn't wanted his money and would refund all of it, but this shitty guy had a messed of mind. He started posting messages of my fb wall about how i was a fraud and I had taken his money. Things got intense. I had decided to sue this cheap minded asshole but after my brother's friend convinced me not to and apologized on his behalf. He told me not to worry about it and move on.

Haven't heard from him till now.
Thank you for reading till this far.

Series of events between me (Mi) and dude in office (DIO).

Instance 1
DIO: There is not psql installed on staging.
Mi: Install it.
DIO: YUM is not working.
Mi: *tries yum it works* It is
DIO: Oh. Didn't work earlier.
Mi: *blank* Make sure you install 9.6
DIO: Cannot find psql
Mi: *types psql, it is already installed*
DIO: Oh, didn't work earlier.

Instance 2
DIO: Made this change to the API, the endpoint is not returning the right value
Mi: *restarts server, shit starts working*
DIO: I am pretty sure I did that, don't know what happened.

Instance 3
DIO: Cannot alter role to give login to this db user.
MI: *runs alter role db_user with login* works
DIO: Don't know why it wasn't working before.

Instance 4
DIO: I have been stuck on this test for the past 1 day, cannot get the API to return the right data while the Rest Endpoint works fine.
Mi: You are hitting the wrong endpoint in the test.
DIO: Oh, I put an extra 's'
Mi: BTW you are testing Spring-Boot with that test and nothing else.
DIO: Yes but what if Spring Boot has a bug?
Mi: ok.

Our web department was deploying a fairly large sales campaign (equivalent to a ‘Black Friday’ for us), and the day before, at 4:00PM, one of the devs emails us and asks “Hey, just a heads up, the main sales page takes almost 30 seconds to load. Any chance you could find out why? Thanks!”
We click the URL they sent, and sure enough, 30 seconds on the dot.
Our department manager almost fell out of his chair (a few ‘F’ bombs were thrown).
DBAs sit next door, so he shouts…
Mgr: ”Hey, did you know the new sales page is taking 30 seconds to open!?”
DBA: “Yea, but it’s not the database. Are you just now hearing about this? They have had performance problems for over week now. Our traces show it’s something on their end.”
Mgr: “-bleep- no!”
Mgr tries to get a hold of anyone …no one is answering the phone..so he leaves to find someone…anyone with authority.
4:15 he comes back..
Mgr: “-beep- All the web managers were in a meeting. I had to interrupt and ask if they knew about the performance problem.”
Me: “Oh crap. I assume they didn’t know or they wouldn’t be in a meeting.”
Mgr: “-bleep- no! No one knew. Apparently the only ones who knew were the 3 developers and the DBA!”
Me: “Uh…what exactly do they want us to do?”
Mgr: “The –bleep- if I know!”
Me: “Are there any load tests we could use for the staging servers? Maybe it’s only the developer servers.”
DBA: “No, just those 3 developers testing. They could reproduce the slowness on staging, so no need for the load tests.”
Mgr: “Oh my –bleep-ing God!”
4:30 ..one of the vice presidents comes into our area…
VP: “So, do we know what the problem is? John tells me you guys are fixing the problem.”
Mgr: “No, we just heard about the problem half hour ago. DBAs said the database side is fine and the traces look like the bottleneck is on web side of things.”
VP: “Hmm, no, John said the problem is the caching. Aren’t you responsible for that?”
Mgr: “Uh…um…yea, but I don’t think anyone knows what the problem is yet.”
VP: “Well, get the caching problem fixed as soon as possible. Our sales numbers this year hinge on the deployment tomorrow.”
- VP leaves -
Me: “I looked at the cache, it’s fine. Their traffic is barely a blip. How much do you want to bet they have a bug or a mistyped url in their javascript? A consistent 30 second load time is suspiciously indicative of a timeout somewhere.”
Mgr: “I was thinking the same thing. I’ll have networking run a trace.”

4:45 Networking run their trace, and sure enough, there was some relative path of ‘something’ pointing to a local resource not on development, it was waiting/timing out after 30 seconds. Fixed the path and page loaded instantaneously. Network admin walks over..
NetworkAdmin: “We had no idea they were having problems. If they told us last week, we could have identified the issue. Did anyone else think 30 second load time was a bit suspicious?”
4:50 VP walks in (“John” is the web team manager)..
VP: “John said the caching issue is fixed. Great job everyone.”
Mgr: “It wasn’t the caching, it was a mistyped resource or something in a javascript file.”
VP: “But the caching is fixed? Right? John said it was caching. Anyway, great job everyone. We’re going to have a great day tomorrow!”
VP leaves
NetworkAdmin: “Ouch…you feel that?”
Me: “Feel what?”
NetworkAdmin: “That bus John just threw us under.”
Mgr: “Yea, but I think John just saved 3 jobs. Remember that.”

A few months ago, I decided to let go some old clients with bad behavior and/or bad projects, since I noticed this was affecting my mental health (lowering my self esteem, make me feel depressed, anxious, etc).

I was exhausted of doing miracles in projects without git, build files, staging enviroments (yes... you can imagine), and receive all sort of curses when sudenly something stopped to work.

I set some requirements to work with any new project/client: 1) project needs to be under version control, 2) it must have staging enviroment, 3) I must work with build files.

As I still have contracts running, I'm communicating this to clients as soon as I finish my obligations.

Today, one of these clients told me they are leaving to work with another developer.

Reason: They said my new requirements are unreasonable and they prefer doing the old way.

rant¡

Client: "Can you add some logos to the homepage?"
Me: "Sure, I've just added it, take a look at staging."
Client: "Great, we can move it to live"

** 5 days later after it being moved to live and telling them I'm going on holiday. **

Client: "EMERGENCY change logo now, we need to change x logo. These logos look crappy too. How did this happen?"

Fuck this. I'm not employed by you, you don't have any right to speak to me like that. Especially after working tirelessly for your company.

**sent email back explaining how to upload files**

Told them not to contact me unless it's technically related.

That time I had to explain the manager (who decided and committed the deadlines) what a URL was and why we need to use a server for staging the web site.

So one year ago I was working at this company from the US, me being in Europe, which automatically implies there is several hours of timezone difference.

The eng. manager decided we would have a release tomorrow (decision was made one month earlier), and stuff was being prepped up to make that happen.

In the US the workday was about lunch time and in EU it was one hour before finishing. The manager gets us in a meeting and asks me and another dude to do some testing that would take several hours to do. This testing could have been done several days or weeks earlier.

40 minutes after that meeting I get a private message from the PM asking for the status of the test...

Me: aaa.. well I started it and will continue tomorrow

Manager: wait what? we have launch tomorrow, this testing has to be done by tomorrow

Me: it's the end of the workday here, I got personal errands that I have to attend to

Manager: uhm ok ... I see...

I was just messaging something in the public chat right before calling it a day and the manager writes "thanks for the input, your day is over now", completely out of context to the conversation I was having with whomever.

There was no question of "can you stay extra hours and do this?", there was no "hey, I know your day is over we will pay you premium hours with this amount as according to our contract, could you do this now as we have release tomorrow?" ..no ..just .. "do it!". I automatically assumed that ..hey, maybe he wants to do this during and after the live launch (and yes I do admit my mistake of not asking just to be clear, but I assumed the manager knows that there is a timezone difference ..like it's a no brainer).

I can not tell you the heat sensation I had after that last reply from the manager ... it was completely uncalled for, and unreasonable.

I mean why not make a pre-launch phase where you put stuff on the staging server, and perform all the necessary tests and then when you get all the green lights from testing you then proceed with the actual deploy? ...no ... mention this like right at the end of the day before the launch....

And another thing that scratched my neuronal cortex is, how does he know exactly how long the tests would take?

Suddenly it hits me.
It’s 01:20 here but i get it.
It’s ALL a budget thing.

No dedicated tester means less expenses.

No personal parkspot?
No expenses!

And no good staging or testing environment? Less expenses!

Meanwhile every developer can setup, work on, and maintain about 20 websites on their shitty local Windows machine, that doesn’t even have a proper SSD installed, and we are setting impossible deadlines to figure out who will sink and who will swim.

Ow, here is a SSD.. Figure out the installation yourself because we have no IT knowledge or budget for people that do.

You want a challenge? How about 40 other people that are distracting you all day long.

Meanwhile everybody has to improve their skills in js, react, html5, ccs3, angular, .net and razor so money can made faster.

It would be nice if you could build apps as well.

You had a question? Sorry, no time. Expect some feedback 14 days later.

You finished the site?
Great!

But here are 101 bugs to solve before next week.

All hail their crazy company!

Unaware that this had been occurring for while, DBA manager walks into our cube area:

DBAMgr-Scott: "DBA-Kelly told me you still having problems connecting to the new staging servers?"
Dev-Carl: "Yea, still getting access denied. Same problem we've been having for a couple of weeks"
DBAMgr-Scott: "Damn it, I hate you. I got to have Kelly working with data warehouse project. I guess I've got to start working on fixing this problem."
Dev-Carl: "Ha ha..sorry. I've checked everything. Its definitely something on the sql server side."
DBAMgr-Scott: "I guess my day is shot. I've got to talk to the network admin, when I get back, lets put our heads together and figure this out."
<Scott leaves>
Me: "A permissions issue on staging? All my stuff is working fine and been working fine for a long while."
Dev-Carl: "Yea, there is nothing different about any of the other environments."
Me: "That doesn't sound right. What's the error?"
Dev-Carl: "Permissions"
Me: "No, the actual exception, never mind, I'll look it up in Splunk."
<in about 30 seconds, I find the actual exception, Win32Exception: Access is denied in OpenSqlFileStream, a little google-fu and .. >
Me: "Is the service using Windows authentication or SQL authentication?"
Dev-Carl: "SQL authentication."
Me: "Switch it to windows authentication"
<Dev-Carl changes authentication...service works like a charm>
Dev-Carl: "OMG, it worked! We've been working on this problem for almost two weeks and it only took you 30 seconds."
Me: "Now that it works, and the service had been working, what changed?"
Dev-Carl: "Oh..look at that, Dev-Jake changed the connection string two weeks ago. Weird. Thanks for your help."
<My brain is screaming "YOU NEVER THOUGHT TO LOOK FOR WHAT CHANGED!!!"
Me: "I'm happy I could help."

Wow, the new React app looks so amazing. Lets test is on different browsers.
Chrome ✓
Firefox ✓
Opera ✓
... ✓
IE 11 ...
1. I am on Mac
2. Let's install virtual box
3. Let's install Windows 7 in virtual box
4. Open IE8
5. Open IE11 download page.
6. IE8 crashes
7. Download and install Google chrome using IE
8. Restart Windows
9. Open staging url on IE11
10. **cking blank screen welcomes you.

“In order to release on production you merge on production”

Makes sense.

“In order to release on staging you merge on release”

Wait, what?

“The CI for the staging release is called release to production”

What? No, stop!

“You also need to create a tag called pre-production-deployment, so that staging can work”

STOP FFS

“It’s very easy, you just need to read the documentation!”

How do people do not realise they are spitting out bs?

"LeT's uSe gRaPhQL!" They said.

"It EliMinAtEs cOmpLeX aNd vErBoSe REST coDe!" they said.

Me sitting here for hours waiting for the backend team to fix major regressions every time they push the smallest "updates" to staging... 🤡

Call me a boomer but I can't help but feeling graphQL makes things MORE complex than REST... either that or the backend devs have no idea what they are doing

Ooof.

In a meeting with my client today, about issues with their staging and production environments.

They pull in the lead dev working on the project. He's a 🤡 who freelanced for my previous company where I was CTO.

I fired him for being plain bad.

Today he doesn't recognize me and proceeds to patronize me in server administration...

The same 🤡 that checks production secrets into git, builds projects directly in the production vm.

Buckle up... Deploys *both* staging and production to the *same* vm...

Doesn't even assign a static IP to the VM and is puzzled when its IP has changed after a relaunch...

Stores long term aws credentials instead of using instance roles.

Claims there are "memory leaks", in a js project. (There may be memory misuse by project or its dependencies, an actual memory leak in v8 that somehow only he finds...? Don't think so.)

Didn't even set up pm2 in systemd so his services didn't even relaunch after a reboot...

You know, I'm keeping my mouth shut and make the clown work all weekend to fix his own hubris.

A colleague named Sam was really pissed off today at an out sourcing firm from India.

My Boss outsourced an application to India based firm. Sam was the one handling the project after the handover. Sam coded a feature 2 weeks ago and moved to staging server for approval. After the sign off from the lead developer of the outsourcing firm, he moved the feature to production. For the past 2 days the application was crashing over and over again so Sam went to check and found out that the feature he coded was causing the issue. When he pulled the feature to his computer and had a look at the code, it wasn’t his code. The code he wrote was commented out and the lead developer of the outsourcing firm wrote new code.

When Sam emailed to him regarding this he replied that he re-wrote his code to fix issues with the feature. Sam and outsourcing firm lead developer had heated argument about this. It’s turns out that the outsourcing developer re-wrote the code without anyone’s approval and on production server.

The lead developer of the outsourcing firm was fired.

TM: Hey, do you have a moment?

Me: not really, I'm already overtime and have enough work for the whole year.

TM: Yeah, we know. Just a quick meeting to discuss something awkward.

Me: Hmkay.

...

Later that day:

TM: Yeah. To make it quick - we're confused and bit dissatisfied with how project X turned out. The staging server is blazing fast, but the devs machines seem to be extremely slow... Some devs complained.

Me: No wonder. I said from the beginning that the devs shouldn't do X and Y, and that the dev machines need to be redone after staging is done - as we need to gather hands on experience first, cause no one could explain to me what resources the project actually needed.

TM: Oh. I wasn't aware of that.

Me: I guessed so. You were on vacation at the beginning and I didn't had the time to lead another team...

TM: Yeah... So the dev machines get replaced?

Me: They _could_ be replaced, but the devs would need to reset up their environment, as I and won't transfer the environment of the dev user.

TM: Ah... So they would have to retransfer their personal modifications, if they made any?

Me: Yes. As always, the basic setup just provides the necessary services, settings etc. - stuff like remote IDE settings on the machine, configuration etc is left out and we don't transfer it as it is usually too much of a hassle and risky, as every dev does have his / her own preferences, and we don't want to support every possible configuration out there.

TM: Just out of curiosity... Staging was ready like... Last year?

Me: Beginning of December, yes.

TM: Sigh.

Me: The jolly of having a kinder garten full of toys that no kid wants to clean up...

TM: No comment. The kinder garten Kids might make me a Pinata otherwise.

Me: If only they'd fill us with chocolate first instead of just beating us.

...

Tales of lazy devs, to be continued...

Current work project is microservices architecture out of 4 - 8 components.

It is fully Infrastructure as a Code automatized. I just change somewhere code, git pushing

And it automatically invokes Gitlab CI, terraform, ansible, kubernetes helm charts.
Auto checking itself with unit and integration tests in autoredeployed staging env. Then it saves tested results to docker registry and asks for one button verificating click to be rereleased to prod.

I just go for drink or eat food. While all the stuff is happening.

And I am proud that all the infrastructure, backend and frontend I made on my own.

I don't need to remember how to Deploy it. It is all automatized

My fellow coworker dev just sent a staging site link to our CFO to review some new functionality...

CFO clicks on our corporate HQ link in coworker's email signature instead and berates coworker for sending him the wrong link...

This person is somewhat largely in charge of making decisions that affect our digital marketing, website budgets, and strategy.

I can't make this shit up.

My department is focused solely on web development. Of course we are part of the major portion of I.T

The entire I.T department got acknowledged for a very important piece of software. That I wrote.
The ceremony in which we were being recognized did not listed MY department, no, they listed the ENTIRETY of I.T.

Thing is, if this product was not delivered, then I was told that the blame would be MINE (I am speaking as the head of my department) but apparently if it succeeded (which it did) it is to be attributed to people that were not even involved in the project.

My employees tried calming me down when I got upset, one of them stated that it was not even our department's effort, but mine alone. And yes, I was the one that developed the solution. By myself, with complete testing, staging, the whole works. Everything, developed by me. BUT my employees held the entire department down while I was behind close doors developing this solution.
I was fucking upset, more so because my director sent an email thanking the entire I.T department for this "win"
I asked him through or messaging service if he could point out to me who else was involved, since I did not know of anyone else that did absolutely anything in this process other than myself and my guys.

Maybe the output of my program was parsed by another I.T department and something happened from it, maybe the money generated by the application (obscene amounts of it btw) were used to add more to the infrastructure etc, who knows, but as far as I know, you cannot say "if this fails it is on you" just for them to later on thank people that were not involved in the project.

This is why I would gladly move on to a different field. I don't want to be patted on the back constantly, I know how fucking good I am at what I do. But if I do something amazing I do not want to see those efforts being given to someone else.

The dev world is usually a thankless industry, but if thanks are given, then I want the sole credit.

If I am winning or loosing I want the whole fucking credit and you can be any more gangstah than that.

Joined a new company / team to work on an iOS app that has 2 different backend environments "Dev" and "Prod". Also being referred to in iOS speak as "Debug" and "Release".

Been trying to get accounts on these backends (no sign up in app, its controlled via another process). Eventually get access to "Dev" for one of the regions, so I load up "Debug" and its not working.

This is odd, so I open the Android app and load "Dev" and it works? I then Notice Android has "Dev", "QA", "Staging" and "Prod" for every region where as iOS only has 2 of these.

So I go back to iOS and find the file for the settings and it has iOS Debug assigned a variable for the backend Dev ... which is actually pointing to QA. Because they use QA to Debug and not Dev.

... confused? join the club

"Who needs a staging server, test suites and continuous integration anyways haha"
-company i just joined

Nothing screams panic like accidentally deploying an older, broken staging build, that also run outdated database migrations on start, straight to production

It was my internship and I've end up working on a law company specializing on Australian construction laws they're working on a website that will take care of all the paperworks for the contractors. They have a dev team who's working on it but they don't have a web designer. I was accepted for the job as an intern/web designer/tester. I was so happy that I've got a really cool internship as a designer but that's only for a second.

The hell starts on day one. They've told me that they're using agile workflow and that they need to make the website responsive. It was based on bootstrap and gosh their code was so broken. HTML tags overlay on each other, some are unclosed. I've tried to fix the problems and did a great job at that. Made the front page responsive and all laid out. When I went to the next php file it has a different header.php and footer.php and same problems apply and we're not even touching the worst.

They didn't use any version management and they're cowboying everything. Now that the website is on the staging server they use Cpanel text editor to edit the code! My headache started to pileup.

The Australian client asked me to provide icons and fix the colors of the website. Also the typography looks great already. I've fixed almost all the problems and I'm satisfied with the design when suddenly a new co-worker from a famous and expensive college was absorbed by the company. He worked as the marketing specialist who has no experience at web design at all. He told me to do this and that and the whole website changed. He bullied me for my skills in design (I'm an intern) and just took over the whole design. Everyone even the boss listen to him as if everything he say is right. He's skilled at design but not web design. He made the website look like a freakin movie poster.

All my works are for nothing, I got headache for nothing and I've got hated for nothing.

It was the day when I finished my internship. It was a long 3 months. After a month I've heard from my co-interns that the whole dev team was fired including the marketing specialist. Also the whole website is scrapped and has been rebuilt by a single guy who used WordPress which he did in only a month.

"we will also consider skills in other production languages, such as Haskell, Clojure, OCaml, Rust and C++"

ah, of course, the classic "production" languages. wouldn't want to be using those other "developing" or "staging" languages... 🥱

i swear, the 🤡 count increases by the hour.

Fuck. I can't take this shit anymore.

There was a project where we had to implement third-party system for government agency processes management. For some reason, probably because my work is cheap for my boss, the task was assigned to me. Just as a reminder, I'm a .NET Dev. Zero experience in server management. Zero experience in external services implementation.

Anyway, system producent, also an government agency, got angry, becasue they can only earn money on implementation. They have to give the software to other agencies for free. Because of that I've got client program, incomplete documentation and broken scripts for database creation. It took me 2 months to get it all to work but at the end client was happy, my boss got paid and I've got 500 PLN (~130 USD) bonus.

Everything was fine for a while, but after a month server has started freezing everyday, some time before 7 am. The only way I found to make it work again was to restore snapshot made everyday at 10 pm. For a month I was waking up earlier and restored snapshot, and after that my boss took it upon himself. I tried few times to find a bug and fix it, but to no effect. Even person with much more experience with it tried to help but also couldn't find anything.

My solution? Copy all the data and configuration, create new machine, copy everything and check if the problem persists. If not, kill old server. Client won't even notice. But nooooooooo... It would cost my boss a bit of money and I'd need to work on it and he can't let it be, because I'm the only developer working on his flagship product. He'd rather wake up everyday and restore snapshot. Okay, as you wish.

And today, finally, everything went downhill. Snapshot wasn't created, server froze, backup can't be created. Nothing can be done. Client is furious, because they have had reported this problem and a few times restoration was too late and they couldn't work. No one knows how to fix it, I'm not working today (I'm still studying and am available only 2 days) and situation is really shitty.

BUT SURE. ITS BETTER TO RESTORE SERVER EVERYDAY THAN JUST FUCKING FIX IT.

Oh, also, there's no staging or any other real backup. We have snapshots for each day and that's that. Boss' order. Why do I even care...

If you're going to request CRITICAL changes to thousands of records in the database, and approve it through testing which is done on an exact replica of production, then tell me it was done incorrectly after the fact it has been implemented and you didn't actually review the changes made to the data or business logic that you requested then you are an idiot. Our staging environment is there to ensure all the changes are accurate you useless human. Its the data you provided, I didn't just magically pull it from thin air to make yours and my job a pain the ass.

So I work as a "Web Development Lead"
Which means I lead (frontend,backend and infrastructure teams)
Also I am in charge of infrastructure or devops or whatever you call it, which means I handle production issues, dev and staging environments,...etc

and I am a team of one, and today I asked for a day off because it's my wife's birthday

and suddenly everyone is blocked, everything is on fire, and the phone is not stopping ringing, I had to go out of the cinema theater to answer the non-stopping calls

I AM ASKING FOR A SINGLE DAY, A FUCKING DAY, EVEN IF SOMEONE IS BLOCKED SO WHAT IT'S NOT EVEN A DAY I ONLY NEED 6 HOURS

IS TO TOO MUCH FUCKING TO ASK

Out of necessity (or rather: lack of support) I've been neglecting my test suite for the past ~month. Now that one of the beta versions of RSpec has better Rails 6 support, I can finally get back to writing tests. Yay!

I just merged staging into my testing branch, and it's now 344 commits ahead of origin! eep.

So, I've got lots of tests to write. yay.

A discussion about writing tests for frontend applications.

Context: my frontend coworkers don't write tests, at all. Yeah, really. Our testing process is very manual. We test manually when developing. We test manually when reviewing code. After merging, the application is deployed to a staging server and the design team does a QA Sprint. Lots of manual testing and some bugs still crawl by.

So I decided to start pushing my coworkers to start writing tests. One of the reasons I constantly hear them say to not write tests in the frontend is: "It's not worth the time, because design keeps changing, which means we have to take time to fix the tests. Time that we usually don't have."

I've been thinking about this a lot and it seems to me that this is more related to bad tests than to tests in general.

Tests should not break with design changes (small changes at least). They should test funcionality, not how things look. A form should not break if the submit button's style changes, so why should its tests fail? I also think that tests help save time, as they prevent some back and forth because of bugs.

Writing good tests is the hard part. Tests that cover what's really important and aren't frail and break with things that shouldn't break them. What (and how) should we test? And what shouldn't be tested?

Writing them fast is another hard thing. Are you doing it right if they take more time to write than the actual code?

What do you think about this? Do you write tests for your frontend applications? What do you test? How much time do you spend writing tests? What are your testing tools/frameworks?

Manager: We need to fix this QA backlog. I’m going to share the workload of doing QA.

Dev: Please don—

*Dev email notification getting spammed with approvals*

Dev: …Are you even pulling the code down to test it locally?

Manager: There’s no time for that! We have to get this PR backlog pushed through! I’m just looking at the code to see if it looks good and approving based on that.

*Later that day*

Manager: HEY NONE OF THE FEATURES ON STAGING MEET THE REQUIREMENTS AT ALL. THIS IS A BUGGY MESS, WHAT HAPPENED GUYS??

Dev: …

I'll admit - I come from a WordPress background of almost 9 years in the making. I guess I can justify it because of all of the sites I created using it, it was the best that it could be on WP. Fast, efficient, custom - none of that off-the-shelf themeforest crap. I created everything custom. I actually knew what was going on behind the scenes of WP.

And then a buddy of mine and I had an idea for a new company/software project. I was smart enough to know that WP was not the foundation for this, so I did some NodeJS/Express tutorials. Started learning React, and really getting into the Javascript world.

And now I'm wondering WHY IN THE ABSOLUTE FUCK I ever bothered trying to become an expert in WP. It's the largest use of PHP in the fucking world and it doesn't even have native composer support. And by the time you actually get your project set up using composer you have to add a fucking mirror of the wordpress.org plugin repo to get anything to work. It's 2018 and you'd think that WP and composer would have all of this shit figured out by now.

And don't get me started on git - as soon as you have more than 1 person working on a WP site, I hope you have hourly backups of your DB because someones work will get overwritten. So you all either need to work on the same staging area of work around each other by pushing/pulling the DB and schedule your workflows.

I guess WP CLI and the REST API are a step in the right direction, but the foundation of everything is just so fucked up.

I don't feel like I've wasted my web dev career, but I definitely wish I had started down this path a lot earlier. I guess you don't know what you don't know. Thanks for reading!

Started work: 9:30am
Finished work: 9:30pm

Finally finished my "basically done" ticket from two days ago, though I'm still not sure if it's behaving correctly. It deals with the Apple wallet and iPhone notifications, and all I have is an emulator, so. 😕 Things work as expected maybe half the time? and idfk why. I'm going to leave the actual testing to QA since they have actual iPhones and it'll run on staging on actual servers with actual connectivity and actual pushes, so maybe then things will actually make sense. Until then? So done.

Started drinking: 9:30pm
Finished drinking: TBD

<media lyrics="I don't know what the fuck just happened but I don't really care, imma get the fuck up outta here. Fuck this shit i'm out" />

Name of the branches in a git repository are like

dev
dev-branch
dev-hotfix
dev-test
dev-merge
dev-app
dev-com
dev-com
dev-ref
dev-staging
dev-username1
dev-username2
dev-username3
.
.
.
.
.
.
Master

Have a couple I want to air today.

First was at my first gig as a dev, 4-5 months out of school. I was the only dev at a startup where the owner was a computer illiterate psycopath with serious temper tantrums. We're talking slamming doors, shouting at you while you are on the phone with customers, the works...

Anyways, what happened was that we needed to do an update in our database to correct some data on a few order lines regarding a specific product. Guess who forgot the fucking where-clause... Did I mention this boss was a cheap ass, dollar stupid, penny wise asshole that refused to have anything but the cheapest hosting? No backups, no test/dev/staging environment, no local copies... Yeah, live devving in prod, fucking all customers with a missing semi-colon (or where clause).

Amazingly, his sheer incompetance saved my ass, because even if I explained it, he didn't get it, and just wanted it fixed as best we could.

The second time was at a different company where we were delivering managed network services for a few municipalities. I was working netops at that time, mostly Cisco branded stuff, from Voice-over-IP and wifi to switches and some routing.

One day I was rolling out a new wireless network, and had to add the VLAN to the core switch on the correct port. VLAN's, for those who don't know, are virtual networks you can use to run several separated networks on the same cable.

To add a VLAN on a Cisco switch one uses the command:

switchport access vlan add XYZ

My mistake was omitting the 'add', which Cisco switches happily accept without warning. That command however can be quite disruptive as it replaces all of the excisting VLAN's with the new one.

Not a big deal on a distribution switch supplying an office floor or something, but on a fucking core switch in the datacenter this meant 20K user had no internet, no access to the applications in the DS, no access to Active Directory etc. Oh and my remote access to that switch also went down the drain...

Luckily a colleague of mine was on site with a console cable and access to config backups. Shit was over within 15 minutes. My boss at that time was thankfully a pragmatic guy who just responded "Well, at least you won't make that mistake again" when we debriefed him after the dust settled.

I used to work on a production management team, whose job was, among other things, safeguarding access to production. Dev teams would send us requests all the time to, "run a quick SQL script."

Invariably, the SQL would include, "SELECT * FROM db_config."

We would push the tickets back, and the devs would call us, enraged. I learned pretty quickly that they didn't have any real interest in dev, test, or staging environments, and just wanted to do everything in prod, and see if it works.

But they would give up their protests pretty fast when I offered to let them speak to a manager when they were upset I wouldn't run their SQL.

The frustration when you try to find out why your local code changes don't propagate to the browser and you suddenly notice that the address bar points to staging. #neverlearnfrommistakes

We developed an application and deployed on production (but not launched)
And business team already created lot of garbage or dummy data. Reporting systems are huge pile of bars, stats and shit.

Now, has to destroy and clean production.

Already advise them to do experiments or testing dev or staging.

Damn. First time in my career experienced this. Has to delete production.

Giant, month-and-a-half-long-ticket.
After learning six or so complicated areas of the system and updating them all to work with the new changes, make them all play nicely, etc. I finally got everything working. 95% spec coverage, though no ui tests because I haven't gotten selenium working. whatever, everything's done and works.

Second dev bases her ticket off of mine and continues working. Work elsewhere continues and there's an official release, so we both merge in master. I run tests, everything passes, and go back to working on other tickets.

She finishes her ticket.
We do end-to-end testing, and everything works perfectly. Time for a demo!

She merges in master again, and pushes her branch to two staging servers. (idk why two.)

Demo starts.
We connect to the staging servers, and... none of the UI changes exist; they aren't running the correct code!

So she runs it locally for a demo instead. Two features in my ticket no longer work. She throws me under the bus. She throws me under the bus again by criticising a rake task I scrapped because she wanted to do it. Then again because I didn't update my branch to master and push it before the demo, despite having no reason to. and despite the demo being of her branch.

Then she continues to show off and brag about how she's like the "legend" (senior dev) she envies. QAbuys it.

I'm having an emotion, and it's called anger.

I once ran a batch import job to stress test with much more data than usual in staging... so I thought.
I sshd into prod by accident. History search can be a bastard at times...

The moment I realized what I was doing was crazy. I was shivering and thought I get fired if anything went wrong.

fortunately I just duplicated the original data for the test, and the system was built to ignore unnecessary updates... so the data was correct and nothing went wrong.

Not an active stupid dev choice but still something I will remember for a while.

Me: Hi, I finished X and Y, will finish Z tomorrow and be able to push it to the staging server for you to test!
*Client 15 hours later*
Client: I don't see it, where is it?
Me: It's not there yet, I need to finish Z first, it's right now on the development server, not the testing server
Client: Yes, but I don't see it?
...

Don't you just love it when something works in your development environment, your staging environment and then randomly breaks on production? :^)

But it worked perfectly in QA and UAT... Then we go to staging

If you just git add . by instinct, you're already dead inside

Instead, consider checking out the diffs of your changes before staging them, and then stage the files or directories individually

Of course I'm saying this to complain about my colleagues who stage and commit things they shouldn't, it probably doesn't apply to small side projects, but staging individually is probably a good habit to have

When I arrived at my new job last year they were working on developing a large site using a live server with all the devs ftp-ing every change in the build process to test it. 😵

It was not uncommon to hear 'is anybody in the style sheet?' being shouted across the office!

Needless to say, I had to fix many bugs multiple times when my fixes were overwritten!

I accidentally started a reindex on a collection that had 14 million records in the middle of the day. Caused an outage in a major portion of our applications for about 3 hours. Worst thing was that once I pressed enter, I realized that it was for the production database, and not the staging database like I intended. I immediately went to go tell the dev ops lead, and he basically said, "whelp, let's just sit back and watch the world burn. Not much we can do about it"

My co-worker, still studying but working as a "senior dev", just decided that we don't need a test/staging environment anymore. We just "validate" (we also don't use the word "test" anymore) newly created features in production.

Makes absolutely sense...

Thank god I have a new job from february on!

Ugh, I hate when customers refuse to pay for staging systems but then randomly complain about stuff that's WIP and not working as expected yet ...

"Yes, sir, this house doesn't have a roof right now because the basement isn't done yet."

So after 6 months of asking for production API token we've finally received it. It got physically delivered by a courier, passed as a text file on a CD. We didn't have a CD drive. Now we do. Because security. Only it turned out to be encrypted with our old public key so they had to redo the whole process. With our current public key. That they couldn't just download, because security, and demanded it to be passed in the fucking same way first. Luckily our hardware guy anticipated this and the CD drives he got can burn as well. So another two weeks passed and finally we got a visit from the courier again. But wait! The file was signed by two people and the signatures weren't trusted, both fingerprints I had to verify by phone, because security, and one of them was on vacation... until today when they finally called back and I could overwrite that fucking token and push to staging environment before the final push to prod.

Only for some reason I couldn't commit. Because the production token was exactly the same as the fucking test token so there was *nothing to commit!*

BECAUSE FUCKING SECURITY!

I just auto charged myself because I forgot I tested my 7 day trial in production last week

🤦‍♂️🤡🤦‍♂️🤡🤦‍♂️🤡🤦‍♂️🤡🤦‍♂️🤡🤦‍♂️🤡🤦‍♂️🤡🤦‍♂️🤡🤦‍♂️🤡🤦‍♂️🤡🤦‍♂️🤡🤦‍♂️🤡🤦‍♂️🤡🤦‍♂️🤡🤦‍♂️🤡🤦‍♂️🤡🤦‍♂️🤡🤦‍♂️🤡🤦‍♂️🤡🤦‍♂️🤡🤦‍♂️🤡🤦‍♂️🤡

I just... don't have the motivation to code. This thing that once gave me chills and joy for hours now feels tedious.

I still love programming. My depression is starting to win, that's all. Tearing up trying to write this.

Oh and yeah, my coworker just knocked out the entire staging Oracle database, so there's that.

Don't you just love it when a customer reports a bug in their live system and it's really urgent to correct it; then you go out of your way to fix it ASAP and deploy it to the staging system for them to verify. Three weeks later the customer has still not tested the bugfix...

Staging

Production

Exact replicas according to git

Of course, production is the one that doesn't work

FUCK I FUCKING HATE IT WHEN THIS HAPPENS SO FUCKING ANNOYING AND FRUSTRATING

//rant

So I'm a BI consultant, been doing this for about 6 years now, and I'm pretty good at the data stuffs. Now I had to complete a project for a client where we call a web service and it had to be done in .NET. I wrote a console app in C# that called the WS, dumped the data then a stored proc processed the staging tables into final tables that our visualization tool can consume.

It works, it's done.

Mind you I'm not a pure .NET developer.

And now that it's completed and working this fucking .NET dude that works for my client is basically giving me an attitude talking about "why wasn't it done as a Windows service? Blah, blah" Like WTF!!??? I get that he's the C# BSD but like chill bruh!!

It's annoying as fuck having to work on projects that are not your area of EXPERTISE and then be ridiculed by other elitist assholes about it.

Doesn't happen much, but fuck it's something I hate about dev. FYI, if it was the opposite I would just be asking questions for understanding, not being a sarcastic prick.

//rant done

A lot of PM-rants (for good reason) but how about a reversed one:

Lead Dev: "Yes, but do we really need a dev and staging server? Can't we just try it out and see if it works?"

I created some test entities specifically for our staging site. Written in all capitalized letters in the BIG TITLE of the entity I included DO NOT DELETE. This is very clearly visible in the CMS. What's the first thing the content managers do?

You guessed it.

I guess if plain English doesn't work, I'll have to use Kindergarten rules and put a custom lock on them so they can never be deleted.

Muad'Dib fullstackchris can already predict the future, in a few weeks: "hey!!!! fullstackchris, I can't delete these test entities!!!!! whats wrong with the system?!?!"

sigh...

It's Friday morning, I haven't had my coffee yet and already stumbled upon 2 bugs in staging. How did QA miss this on development?

Oh wait, we don't have QA!

List of shit my superior said and wrote in the project:

1. Prefer to write "pure" SQL statement rather than ORM to handle basic CRUD ops.

2. Mixing frontend and backend data transformation.

3. Dump validation, data transformation, DB update in one fucking single function.

4. Calculate the datetime manually instead of using library like momentjs or Carbon.

5. No version control until I requested it. Even with vcs, I still have to fucking FTP into the staging and upload file one by one because they don't use SSH (wtf you tell me you don't know basic unix command?)

6. Don't care about efficiency, just loop through thousands of record for every columns in the table. An O(n) ops becomes O(n * m)

7. 6MB for loading a fucking webpage are you kidding me?

Now you telling me you want to make it into AJAX so it'll response faster? #kthxbye

Fuck. My boss left for a meeting with the client to show him the new frontend. 5 minutes after he left I saw that on the staging server half of the page entries were gone for the non-default localizations. FUCK.

It took me 20 minutes to restore the fuckery. I hope the client didn't see it...

I love one particular old game. It's called Port Royale (the first one). Why? Because the game crashes a lot. Players know that, devs knew that. It's so old and unknown to people who haven't played that devs don't even fix it. But, but... why do you write it here?

This game tought me autosaving! Yeah, they have autosaving in [5, 10, 15] minute intervals, but the game is so fast, that even a little change you do will cripple your whole economy. Not to mention the saving mechanism is partially broken (or that's what the log says, fml). By broken I mean it tries to autosave, but sometimes it crashes the whole thing, just because it can. A game with special effects - crashing in _intervals_!

Because of this lovely game I have a habit of saving and staging (or even commiting). Maybe they should be proud for making such a bug. Saved me once again a minute ago when I managed to crash Emacs with Python. :D

So I've been doing work for a big ass real estate firm. Managing work of their 12 websites. A month ago they sent a prototype built with React.js which their design agency did. What we did was creating a backend API and an administration area and hooked the prototype to make it dynamic.
Everything was working quite great but around 2-3 days ago their guy who had been looking over the work of this React.js website came up and said that it isn't working in Internet Explorer 11. Bastard didn't mention this earlier throughout the whole month when we had provided him access to a staging server. I coldly replied that the prototype was provided by their design agency. But today the fucker put it all on us and said that it is something we implemented that caused this issue. I had to do some research but I finally managed to run the website in IE11. Bastard tried to steal my weekend.
Fuck IE11, fuck such pig clients, fuck Microsoft. IE, just DIE already you syphilis infected piece of whore's ass!

>Adds new feature
>New feature works fine on dev
>New feature works fine on staging
>New feature doesn't work on live
>You can't easily figure out what is wrong because you need to wait an hour for it each time :|

If they followed my suggestion and went straight to debugging the server issues they would have been solved it from week 1 and everyone would have thought the migration had a minor performance hiccup. In fact, we have already done such at least twice before and nobody batted an eye.

Instead they self-labelled the migration a failure on first error, setting the stage for apologizing to the client, and put themselves on the spot for a whole staging / production signoff, replication / backup worfklow, almost a blue-green "seamless" deployment reminiscent of DigitalOcean.

Well they're not DigitalOcean, and anyone who has spent any time understanding users knows they will not participate in "new system" tests long enough to find or report issues.

So of course the migration stretched out to almost three months up until the whole reason for the migration - the rapidly escalating risk of the old provider disappearing - hit like a freight train and now they have to go through the problem of debugging the server like I told them to on week 1. Only this time they've set the client mindset against it, lost any chance of reverting, have had grave risk for data loss, and are under pressure to debug other people's code in real-time.

This is why I don't trust devs to do ops. A dev's first solution to any problem is to throw tech at it.

One of the other teams have been hogging staging for forever. My dev lead:

Client: The new page template you sent us looks different on production compared to the other pages that use the same component.

Me: Oh, that's strange since the styling is at the component level. Hmm, let me dig in to it.

Start poking around trying to figure out what I managed to screw up only to find that it looks exactly the same on local and staging. Eventually find another style sheet the client is importing on the production site to change some of the styles.

You know, a change that isn't anywhere to be found in the repo, and no one ever asked for anything to be changed. Their "Dev" decided he would hack in a fix instead of shooting me an email.

Apparently he tried changing the SCSS file but the changes weren't showing up. He changed the minified stylesheet but his changes were overwritten on the next deployment..... #howdoesSASSwork?!

Same client as my last rant so I'm not sure why I'm surprised by this. Oh well, I'll take that hourly rate.

Beware: Here lies a cautionary tale about shared hosting, backups, and -goes without saying- WordPress.

1. Got a call from a client saying their site presented an issue with a third-party add-on. The vendor asked us to grant him access to our staging copy.

2. Their staging copy, apparently, never got duplicated correctly because, for security reasons, their in-house dev changed the name of the wp-content folder. That broke their staging algo. So no staging site.

3. In order to recreate the staging site, we had to reset everything back to WP defaults. Including, for some reason, absolute paths inside the database. A huge fucking database. Because WordPress.

4. Made the changes directly in a downloaded sql file. Shared hosting, obviously, had an upload limit smaller to the actual database.

5. Spent half an hour trying to upload table by table to no avail.

6. In-house uploads a new, fixed database with the help of the shared hosting provider.

7. Database has the wrong path. Again.

8. In-house performs massive Find and Replace through phpMyAdmin on the production server.

9. Obviously, MySQL crashes instantly and the site gets blocked for over 3 hours for exceeding shared hosting limits.

10. Hosting provider refuses to accept this was caused by such a stupid act and says site needs to be checked because queries are too slow.

11. We are gouging our eyeballs as we see an in-house vs. hosting fight unfold. So we decide to watch a whole Netflix documentary in between.

12. Finally, the hosting folds and enables access to the site, which is obvi not working because, you know, wrong paths.

13. Documentary finishes. We log in again, click restore from backup. Go to bed. Client phones to bless us. Client’s in-house dev probably looking for a cardboard box to pack his stuff first thing in the morning. \_(ツ)_/¯

Code review day 1:

Me: Where are these three templates being used? I don't see any class that uses them.

Him: Oh, yeah. My bad. I'll remove them.

(Time passes)

Him: I don't know what's going on with my branch. Please ignore the PR for now.

(Time passes)

Him: Is there any way for me to get the files from staging. I don't know what I did to my branch.

Me: Hang on. I'll push a copy of the one I have in a new branch.

He declines his PR and stops by to say he's going home but he'll open a new PR tomorrow and remove the unused files.

Code review day 2:

Him: Hey, do you know what happened to my PR? It looks like it disappeared.

Me: You declined it yesterday. Said you'd make a new one today without those extra files.

Him: Ooooh!

That moment when you develop on localhost and wonder why the page does not refresh while in fact you're looking at the staging.

i was staging the fucker site to deploy and get on with it, but now the requirements changed, and major redesign is needed
I'm payed by hour but this shit still infuriates me

I just fucking hate compiling this fucking C# (ASP.NET) code and then transferring to staging server. Fuck you.....no no no listen to me fuck you and fuck this shit.

My killer PR:
I made a small feature and it was to be merged in my senior's branch.
So I made it, resolved all the conflicts and when it got merged it broke my senior's branch.
This is nothing new and it happens, so me and him sat down and got it working. After two days, his branch got merged and broke it's parent 😂, after investigation my code was the reason, got fixed and merged.
Same thing happened 3 more times, and every time my code was the culprit.
Now for staging we thoroughly tested everything and deployed it, after running for 2 hours the piece of shit broke again 😂😂😂.
A meeting was arranged for scolding the team, and after the meeting the architect comes to me and say "how did you manage to fuck things on so many levels, literally?". I handled it like a pro with an awkward laugh.

We exhaustively checked it for production. Deployed, it did not broke, we were happy. After a month of successful run, I just joked about the above incident while we were working on next release on morning coffee. That cursed thing broke on the same evening. I was like 😮

I ended today a meeting.

Was annoyed.

Did it bastard administrator of hell style.

I shut down the necessary staging machine for presentation after disabling the network service.

Of course the machine died unexpectedly and sadly we had to postpone this important meeting.

In reality this meeting was as important as hay rolling down a street.

I hate everything at the moment.

It took AWS about a month to figure out why their load balancer was screwing up content length for requests from our site. Multiple times the ticket was closed due to inactivity because they took so long to investigate. Turns out there's a bug with how AWS load balancers scale, and when they are below a certain traffic threshold they truncate extremely long content. Their solution was to edit the balancer behind the scenes to always be scaled up, and then tell us to never delete it.

So then every time we needed to set up a staging environment we had to contact support so they'd edit the balancer. Which always took ages since most of the support agents didn't understand the convoluted issue and had to forward it on to more technically inclined staff, who then had to investigate fresh every time.

This was ridiculously annoying, so I spent months writing an automated solution to spin up staging new environments on the spot, this made use of a haproxy server which had to edit rules on the fly so that the AWS balancer could be circumnavigated. It was a better system then the old way anyway, but all the same an irritating issue to be forced to deal with.

All around a very shitty experience. This was a few years ago now and I'm not employed there any more, but I hope AWS fixed this since then.

Me: Alright Derwent, don't fuck up this database update. There's no undo button and no way to import a database backup so you gotta be extra careful or you're going to have to spend hours writing a whole bunch of regular expressions and sql statements to sift through an 11mb database dump and figure out how to restore 59 thousand records to the correct state. Let's practice this transition on a staging server first and make sure we get it right
Me: I got you fam *presses the wrong button*

TL;DR: When picking vendors to outsource work to, vet them really well.

Backstory:
Got a large redesign project that involves rebuilding a website's main navigation (accessibility reasons).
Project is too big just for our dev team to handle with our workload so we got to bring a 3rd party vendor to help us. We do this often so no big deal.
But, this time the twist was Senior Management already had retained hours with a dev shop so they want us to use them for project. Okay...

It begins:
Have our scope / discovery meeting about the changes and our expected DevOps workflow.

Devs work Local and push changes to our Github, that kicks off the build and we test on Dev, then it goes to Staging for more testing & PM review. Once ready we can push to prod, or whenever needed. All is agreed, everyone was happy.

Emailed the vendors' project manager to ask for their devs Github accounts so we can add them to the project. Got no reply for 3 days.

4th day, I get back "Who sets up the Github accounts?"

fuck me. they've never used Github before but in our scope meeting 4 days ago you said Github was fine...??

Whatever, fuck it. I'll make the accounts and add them.

Added 4 devs to the repo and setup new branch. 40min later get an email that they can't setup dev environment now, the dev doesn't know how to setup our CMS locally, "not working for some reason."

So, they ask for permission to develop on our STAGING server.. "because it's already setup"... they want to actively dev on our staging where we get PM/Senior Management approvals?

We have dev, staging, production instances and you want to dev in staging, not dev?... nay nay good sir.

This is whom senior management wants us to use, already paid for via retainer no less. They are a major dev shop and they're useless...

😢😭

Cant wait for today's progress checkup meeting. 😐😐

/rant

I am so fucking drunk but still I ended up completing a task which is a feature request.... I have no idea how I did it... But it works in staging... That's what I remember.... Will see tomorrow.... I am also the tester in the hierarchy which starts and ends with me... I don't know whether this qualifies as a rant

Who the hell deploys PROD environments in a STAGING cluster.....?

Who the hell deploys PROD environments in a PROD cluster and (deliberately) configures them to use a STAGING environment's DB cluster....?

Boss: "You hardcoded the redirect uri in the code (Early on during development and forgot about it, because apple OAuth is a piece of shit), but don't worry I fixed it by hardcoding the uri with the production host into the config file where clearly all settings are fetched from the OS Environment variables at runtime. This will surely fix the problem in staging we have, no need to thank me"

It's been broken in production for two weeks, and you want to tell me it's "drop everything urgent" now? Is it important? Yes. Is it urgent? No.

Also, how the fuck did you not know this was broken for two weeks, and what the hell do you want me to do about it now? It's your codebase, it passed all the tests in staging AFTER I made my changes to the infrastructure.

How the hell some devs are releasing to production before passing to QA, Alpha testing, staging at least handle all type of errors and hide your API Keys.

Got to work today to find out our staging server HDD has gone poof... Guess who's gonna have to reconfigure it...

Here: this is BDD, a Linter, and Git. CI your shit to staging before you CI to production. Never deploy on a Friday.

A few years ago I worked at company specialized in Magento(eCommerce) and Magento was changing their licensing model. At the time they had 3 Versions. Community(free), Pro and Enterprise.

They decided to ditch Pro and either make all migrate to Enterprise(with a discount) or go community which wasn’t really compatible. So some shops were in need of a more or less complete rewrite.

My hdd crashed literally the day before but hey no big deal everything is 99% done and on staging. So I had a Trainee at that Time and thought the last few crappy things could be done in pair programming so he can learn a few things.

But fuck him! That motherfucker! He managed to WIPE the staging server and no that was at a time without gut and no SVN. That dipshit just deleted 2 months of work because he thought it was a good idea to SYNC his empty project to the staging system.

Oh god I nearly stabbed him. He did that shit out of his own mind even though I told him a dozen times what would happen... we had to do the whole thing again with me sitting next to him watching every stroke he made.

Guess he learned something while inward silently raging the next weeks.

I mistyped cap (from Capistrano) so often that I made an alias for it. Now bash history is full of... Crap. Literally.

My boss seen it once.... Luckily, he's already seen so many of my shenanigans that he didn't even care...

So yeah, I made an alias that changed one single character so I don't need to learn how to type.

crap staging deploy

How to manage updates on production as web agency

1: Update the GitHub private repo with your changes
2: Write an email to the customer sysadmin and ask to sync in staging and later production
3: Wait 2 weeks and do another ping about it
4: The sysadmin later will do that
5: In the meantime there is a security update of your CMS and a lot of plugins to update
6: Try to update from the CMS panel and there are no write permission
7: Cry a lot and write another email to go back to point 2

First rant here..

So earlier this week, on a php Laravel project, I created a set of nice new features.

The code is tested, locally all fine, I push to Github, circleCi kicks in and double checks myself, still everything green. (Just for a not, its a private project so only I work on it.)

I go ahead and merge, deploy to staging and continue on my next ticket, which is a very small one.

I call it the day, next day I pick back up where I left, test locally, all green, push... then circleCi says no.

I spend 2 days debugging, trying to figure out what is wrong without advance. I just push develop branch again, guess what also failing.

I leave it for the day as I already spend enough time on it.

This morning, I simply do a composer update, push and everything miraculously starts working.. even if there were no changes in the working branches.

Im so mad right now, and this is going in my "try this before you debug a ci" book..

Records Person: Can you look at this member renewal issue for system A? It’s happening on the website you maintain. Here are some recent errors to debug.

Me (web developer): I can’t reproduce the error your reporting. Is there something I’m missing? And is there an example for the staging environment?

RP: There’s another team that will manually reconcile the records in system A if they don’t match what’s in system B. So this gives users two active memberships when it should only be one.

Me: 😑 So you already know the issue is human intervention messing with the records and causing the renewal issue. This is not a website issue. It’s a data issue.

I'm still studying computer science/programming, I still have one year to do in order to graduate (Master). I am in a work study program so I'm working for a company half of the time, and I'm studying the other half. It is important to mention that I am the only web developer of the company

When I arrived in the company 9 months ago, I was given a Vue project which had been developed by a trainee a few weeks before my arrival and I was asked to correct a few things, it was mostly about css. Then, I was ask to add a few functionalities, nothing really hard to code, and we were supposed to test the solution in a staging environment, and if everything was ok, deploy it to prod.

However, the more I did what I was asked, the more functionalities I had to implement, until I reached a point where I had to modify the API, create new routes, etc. I'm not complaining about that, that's my job and I like it. But the solution was supposed to be ready when I arrived, it was also supposed to be tested and deployed.

The problem is, the person emitting these demands (let's call him guy X) is not from the IT service, it's a future user of the website in the admin side. The demands kept going and going and going because, according to him, the solution was not in a good enough state to be deployed, it missed too many (un)necessary features. It kept going for a few months.

The best is yet to come though : guy X was obviously a superior, and HIS superior started putting pressure on me through mails, saying the app was already supposed to be in production and he was implying that I wasn't working fast enough. Luckily, my IT supervisor was aware of what was going on and knew I obviously wasn't to blame.

In the end, the solution was eagerly deployed in production, didn't go through the staging environment and was opened to the users. Now, guy X receives complaints because none of what I did was tested (it was by me, but I wasn't going to test every single little thing because I didn't have time). Some users couldn't connect or use this or that feature and I am literally drowning in mails, all from guy X, asking me to correct things because users are blocked and it's time consuming for him to do some of the things the website was doing manually.

We are here now just because things have been done in a rush, I'm still working on it and trying to fix prod problems and it's pissing me off because we HAVE a staging environment that was supposed to prevent me from working against the clock.

On a final note, what's funny is that the code I'm modifying, the pre-existing one needs to be refactored because bits and pieces are repeated sometimes 5 times where it should have been externalized and imported from another file. But I don't know when and if I will ever be able to do that.

I could have given more context but it's 4am and I'm kinda tired, sorry if I'm not clear or anything. That's my first rant

We're doing a huge demo in half an hour for a governmental branch, and for some fucking stupid reason I decide to tinker with the deployment setup. And yes, all our staging environments went down, including the environment set up for the demo. Managed to get it up and running again though 😅

A loooong time ago...

I've started my first serious job as a developer. I was young yet enthusiastic as well as a kind of a greenhorn. First time working in a business, working with a team full of experienced full-lowered ultra-seniors which were waiting to teach me the everything about software engineering.

Kind of.

Beside one senior which was the team lead as well there were two other devs. One of them was very experienced and a pretty nice guy, I could ask him anytime and he would sit down with me a give me advice. I've learned a lot of him.

Fast forward three months (yes, three months).

I was not that full kind of greenhorn anymore and people started to give me serious tasks. I had some experience in doing deployments and stuff from my other job as a sysadmin before so I was soon known as the "deployment guy", setting up deployments for our projects the right way and monitoring as well as executing them. But as it should be in every good team we had to share our knowledge so one can be on vacation or something and another colleague was able to do the task as well.

So now we come to the other teammate. The one I was not talking about till now. And that for a reason.

He was very nice too and had a couple of years as a dev on his CV, but...yeah...like...

When I switched some production systems to Linux he had to learn something about Linux. Everytime he encountered an error message he turned around and asked me how to fix it. Even. For. The. Simplest. Error. He. Could. Google. Up.

I mean okay, when one's new to a system it's not that easy, but when you have an error message which prints out THE SOLUTION FOR THE ERROR and he asks me how to fix it...excuse me?

This happened over 30 times.
A. Week.

Later on I had to introduce him to the deployment workflow for a project, so he could eventually deploy the staging environment and the production environment by hisself.

I introduced him. Not for 10 minutes. I explained him the whole workflow and the very main techniques and tools used for like two hours. Every then and when I stopped and asked him if he had any questions. He had'nt! Wonderful!

Haha. Oh no.

So he had to do his first production deployment. I sat by his side to monitor everything. He did well. One or two questions but he did well.

The same when he did his second prod deploy. Everythings fine.

And then. It. Frikkin. Begins.

I was working on the project, did some changes to the code. Okay, deploy it to dev, time for testing.

Hm.

Error checking out git. Okay, awkward. Got to investigate...

On the dev server were some files changed. Strange. The repo was all up to date. But these changes seemed newer because they were fixing at least one bug I was working on.

This doubles the strangeness.

I want over to my colleague's desk.

I asked him about any recent changes to the codebase.

"Yeah, there was a bug you were working on right? But the ticket was open like two days so I thought I'll fix it"

What the Heck dude, this bug was not critical at all and I had other tasks which were more important. Okay, but what about the changed files?

"Oh yeah, I could not remember the exact deployment steps (hint from the author: I wrote them down into our internal Wiki, he wrote them done by hisself when introducing him and after all it's two frikkin commands), so I uploaded them via FTP"

"Uhm... that's not how we do it buddy. We have to follow the procedure to avoid..."

"The boss said it was fine so I uploaded the changes directly to the production servers. It's so much easier via FTP and not this deployment crap, sorry to say that"

You. Did. What?

I could not resist and asked the boss about this. But this had not Effect at all, was the long-time best-buddy-schmuddy-friend of the boss colleague's father.

So in the end I sat there reverting, committing and deploying.

Yep

It's soooo much harder this deployment crap.

Years later, a long time after I quit the job and moved to another company, I get to know that the colleague now is responsible for technical project management.

Hm.

Project Management.

Karma's a bitch, right?

I've been working on the ecommerce website from hell for over a year now. I should have heard the alarm bells when the studio who were running the project took a month to pay my deposit but still expected me to start working, but I explained that I wouldn't start without some form of security and they were cool with it, so I carried on.

It started off as a simple build with simple products, no product variations etc and a few links on the designs which appeared to lead to external links, and checkout and cart pages were nowhere to be seen. It wasn't a big money job so I just build them in as plain and straightforward as I could, in line with how the rest of the site looked. They then changed their mind about how they wanted these to look, and added loads of functionality to the site throughout the build, so by the end of the line, the scope of work had completely changed. I also had loads of disagreements in terms of design and useability, as their designs straight-up weren't going to function otherwise, plus every round of changes meant that I had to prolong the job further and fit it around work for other clients.

Fastforward a few more months and I get sent a really angry email with some of the client's complaints, including one that raised an issue with the user journey, and the finger of blame was pointed at me. The user journey had been a part of the designs from the start, and this was never raised as an issue for A WHOLE YEAR. They then said that it had to go live on Monday (three days after they sent email with these huge new structural changes). I told them I could no longer work on the project but was happy to waive the rest of my fee (3/4 of the total fee, when I had essentially completed the site, minus 2 minor bugs), so they could find another developer in the limited time they had. At first they refused to hire another developer, claiming that it would be too expensive, which made no sense, as for a few minor fixes and out of scope additions he could get paid a wage that would have otherwise paid for the majority of the work I had done on the site. I stood my ground and finally they found someone, so I sent over all of the files and database to their new developer and asked him to give me a heads up when I could remove the staging site from my server. The next day, I received an email from the studio asking me to fix some bugs the developer was requesting I fix so he could carry on with the site. They were basically asking me to work more, for free, to enable him to walk off with the majority of the money and do less work. They also forwarded a suuuuuper shitty, condescending email from him, listing all the things he thought was wrong with the site (he even listed 'no favicon' although they'd never supplied a graphic for this). He also wrote a paragraph at the bottom EXPLAINING MY JOB TO ME and telling me:

I get the feeling you like to write Javascript, while being one of the easiest languages to learn, it can also be one of the hardest to master. While I applaud you for writing Vanilla JS, it looks like you have a general problem with structuring your application.

Not sure if I'm being oversensitive here but it felt so patronising, and i couldn't even go for an angry walk to get it out my system because of social distancing lol.

Let a girl quarantine in peace!!!!!!

So in the project I’m working on we were about to do a push to live, no major functionality just minor adjustments and nice to have stuff. One of the things I did was a reminder, nothing special just sends an email out if something hasn’t been done for 3 days and then sends an email every day following. Push to live and every thing goes fine with no issues. Day 1 there are no issues. Day 2 there are no issues. Day 3 and I’m inundated with people telling me that the emails are getting sent to practically everyone, shit. What have I done? What have I missed?
So I start looking at the live database hoping for a data problem, no such luck. I look at my code looking for something blatantly obvious but nothing. I start replicating the data but I can’t reproduce this bug and it’s annoying the hell out of me. I checked one of the emails that the client sent to us more thoroughly and seen that it was sent at 07:01. This is odd as our webjob runs at 1am so I start looking at environmental factors and started looking at release management, more out of hope than expectation. I check the staging environment and see that the webjob ran at 7:00. Coincidence I thought, the webjob gets packaged on the release pipeline and everything in the database was dummy data anyway but I’d better check anyway. The database was an exact copy of the live database, turns out a “senior developer” wanted to sanity check everything by running live data through the code so he copied the database over. It was fine for the first couple of days but the data was now 3 days out of date triggering my email code and I get hit with the shit storm. I’ve never met such an incompetent developer in my fucking life, functions 700 lines long, classes that are over 20000 lines, repetition every where and the only design patterns he’s used is when he picks up a child’s colouring book. I can live with the fact that he writes code like someone on their first day of University But copying a database because he wants to “visualise” the fucking data is absolutely farcical. No wonder the project is fucked with a “developer” (in the loosest possible use of the word) is at the helm.

Old story, happened some way back. I worked part-time for a small web development company that did between other things something called SharePoint development, basically .net webforms with shit glitter on top of it.

The most weird part of it, was the fact that we were working on vms that hosted the app, it was our dev, test and staging environment, as well as were we showed the client the polished turd.

Did I say that it was on a vm? Well it was on a remote vm, that each of use had access to it, through our domain accounts, and they couldn't configure the windows server to accept more than two or three users at once to be connected.

That was our test enviroment and dev enviroment, sooo showing the app to the client meant for the rest of us to not write any code because it might crash or get stuck.

The app was accessible and discoverable by url and through google search from outside, I dont think that should have been allowed.

The most disastrous part was that we had NO source versioning whatsoever, just plain old copy and paste in different folders.
Deploying to client meant remoting to the clients host or whatever it was, and manually copying the source files

If someone wanted to debug the application you had to shout, and you also could hear it, in the office: "I'm debugging!" or "I'm deploying!". Because we were on the same machine, there was only one process with the server and it meant that if you debug or deployed it would block it for the others.

Should I talk about code quality? Maybe not.

Head of IT: kindly check what the issue with the following transaction

Me: Is this on production or on staging?

Head of IT: YES

Me: 🖕🏾 🖕🏾

One of those mysterious bugs that only happens on production. Want to solve it on your laptop, it's not reproducible. Staging server? Nope. Production?

HOW DARE YOU TOUCH THE LIVE SYSTEM?!?

Get a ticket for a low priority bug, reported internally. Fix the issue mentioned in the bug.

Moves to QA environment, the original bug reporter tests and *passes* the ticket.

Moves to Staging environment, same exact individual then *fails* the testing. Cites totally new/unrelated changes that need to be made.

Apparently our the workflow is -

Code->QA->Staging->Requirements

Makes sense! :)

Me: "Oh right I still had to fix that"
Me: *fixes thing*
Me: *checks on staging to make sure it all works*
Thing: *works absolutely fine*
Me: "Cool, let's promote it"
Me: *updates production stack*
Me: *checks on production to make sure it works*
Thing: *still broken*

dsfhjh dsaghdsaupot nvhudfot vnhgue

I've now worked on both monolithic solutions and microapps/microservices. I gotta say I'm not sold on the new approach. There's so much overhead! You don't have to know your way around one solution -- no, now you need to know your way around 100 solutions. Debugging? Yeah, good luck with that. You don't have to provision one environment for dev, test, staging, and prod. No, now you need 100 environments per... environment. Now, you need a dedicated fulltime devops person. Now devs can check in breaking changes because their code compiles fine in that one tiny microapp. The extra costs go on and on and on. I get the theoretical benefits but holy crap you pay for it dearly. Going back to monolithic is so satisfying. You just address the bug or new feature head on without the ceremony and complexity. You know you're not crapping on other people's day (compilation-wise) because the entire solution compiles.

...and yeah, I'm getting old. So get off the lawn! ;)

Long story ahead

Background:
I recently started a job in a smallish startup doing web development in a mostly js stack as an entry-junior engineer/dev. I’m the only person actively working on our internal tools as my Lead Engineer (the only other in house dev) is working on other stuff.

Now I was given a two week sprint to rebuild a portion of our legacy internal app from angular 1.2 with material-ui looking components with no psd’s or cut-outs of any kind to a React and bootstrap ui for the front end and convert our .net API routes into Node.js ones. I had to build the API routes, SQL queries (as there were plenty of changes and reiterations that I had to go through to get the exact data I needed to display), and front end. I worked from 9am until 11pm every day for those two weeks including weekends as our company has a huge show this upcoming week.

I finish up this past sunday and push to our staging environment. The UI is 5.5/10 as we’re changing all of our styling to bootstrap and I’m no ui expert. The api has tests and works flawlessly (tm).

So we go into code review and everything is working as expected until one tab that I made erred out and was written down as a “Needs to be fixed.”

This fix was just a null value handler that took three minutes and a push back to staging, but that wasnt before a stupendous amount of shit being flung my way for the ui not looking great and that one bug was a huge deal and that he couldnt believe it slipped through my fingers.

Honestly, I’m feeling really unmotivated to do anything else. I overworked myself for that only to be shit on for one mistake and my ui being lack-luster with no guides.

Am I being a baby about this or is this something to learn from?

I just joined a new company.
Their CI pipeline is to give root access to staging and prod servers to every developer in the company and the manually git pull each repository (8-10 repos per server) and manually set nginx and port configurations. And if this wasn't enough, all of the 30 sites they have are basically the same site and they make the changes manually for each tenant (no env file). I'm amazed at how hard some people are willing to work.

I deployed one of our staging websites to a free plan because the site is rarely used. Project Manager sends the stakeholders the new url. There will be a lot of 🤦‍♀️🤦‍♂️🤦 all around. Some of it’s my fault. A lot of it is just WTF.

Stakeholder: We still need the staging site because we don’t want to test in the live site…

PM: Okay. We didn’t say we were deleting the site. We are just moving it to a new and better hosting platform, so we’re letting you know the url has changed.

Stakeholder: This url is for the front facing page. How do I access the backend? [they mean the admin interface]

Me: The only thing that’s changed is the url for the staging website. So domain-A/account is now domain-B/account.

I thought that was a pretty straightforward way of explaining things, that even a non technical person would get it. They took the /account example as the literal login url.

Stakeholder: I forgot the password for our admin login and I submitted a password reset, but I realize I don’t know if I have access to the admin email. Or if it’s even a real email account.

WTF

I look back at the email chain and I realize that I gave the PM the wrong url.

Also, WTF x 2. How did this stakeholder not realize they were looking at the wrong website?? There are definitely noticeable style and content differences. And why would you have an admin login that uses a fake email??

Me: My apologies. I sent over the incorrect url. My instructions are mostly the same. All that’s changed is the domain.

Stakeholder’s assistant: [DMs me] How do we access the backend?

WTF…are they seriously playing this game and demanding I type out the url for them?! 🤬 I’m not playing this game and I just copy and paste the example that I already sent over.

They figure it out eventually. Apparently, they never used /account to login before They used /admin/index… but that would still bring them to /account, but with ?redirect=/admin/index appended to the url if they weren’t logged in. Again, WTF.

I know I made mistakes in this whole thing, but damn. I can’t even. I’m pretty sure this whole incident is fueling my boss’s push to stop supporting this particular website anymore so I can focus on sites that actually bring in revenue…and have stakeholders that aren’t looney and condescending like this.

One dev starts db migrations for staging...
Four devs stop working cause their tables no longer exist...
Best Friday Ever

#devRantBreak

We take over development of a live customer facing system and PM agrees date for our first code deployment with client CIO
Me: The dev and staging environments don't have any test data currently as the old agency screwed it up
PM: Well you better load some
Me: There isn't any... It'll take 10 days to copy prod db due to hosting provider SLAs, leaving 1 week for SIT, UAT and performance testing (assuming they don't screw up)
PM: Well the date is set, 1 week will be enough for testing

Debugging WebRTC is pure hell.

For starters, it's JavaScript, so you know this isn't gonna end well. Second, it's still in kinda beta phase for some browsers so you gotta add polyfills. Let's talk compatibility now. During normal days, yeah, I could ask for a couple of computers in the office, each using a different browser. But, covid. One browser mishbehaves and doesn't wanna share the camera with the other browser, so I can't really test a connection with the only 1 computer I have. I can't take my partner's computer all day to debug.

Solution: ask the marketing department or even the execs to video chat with you to test it on a staging server. So I push my changes to the server, wait for them to build, call my lab rat, check all the bugs, clean the code, push the changes back up. No fancy breakpoints. I'm doing the old style like my great uncle did. Oh wait no, he was pretty intelligent, but my lab rat isn't. They probably don't know what a console is. So no baby I'm not only talking about console logging the problems, I'm talking `alert` the heck out of the bugs - okay no, I'll just display the objects in the middle of the screen. The screen is my console.

When I first started reading about Angular 4 I must admit I was a bit excited. It seemed like it fit the company enterprise requirements. The improvements it offered on paper looked quite good for our use case. HOWEVER... After writing Angular 4 for two weeks I'm seriously doubting I made the right decision. Testing is a dependency hell and there are two ways to build and structure your application. The webpack way and the SystemJS way. The grunt way and the angular-cli way. For fuck sake Google. And the documentation is somewhat half supporting one thing, half supporting another. So when you're using angular-cli with webpack, you're pretty much screwed when we're talking about documentation. It has now taken me almost 50 hours to write a pretty basic Angular app, made it compliant with our staging environment and writing a Makefile for it, since I haven't been able to find any same way to provide custom arguments when building it with the angular-cli --aot option. So fuck you Google. Luckily I've found a way to modularize it so much that I'll be able to reuse the core in the future. So I guess I got that thing going for me, which is nice... -.-' *sigh*

The most scary thing happens to me is that I wrote a code in staging without any bugs and breaks in production... fuck

accidentally merged staging with master :-/

Somewhere half of the project, after just placing a new version on staging I get an email from the customer:

" This is really going the wrom way! Everything is so big, and what happened to the original design? The one I agreed to! I really dont like this, please go back to the way it was!!!"

I replied asking him what exactly wasnt as the original design (We really followed the prototype we had showed him earlier)

So the next day I get this email:

"Whoops, turns out I was still zoomed at 150%. I guess it was a bit of an overreaction.."

Couldnt believe this :')

When staging is down the devs party

story of a release
v2.1.0 major changes went live : new features, bug fixes, optimisations. also included releases for 2 associated libraries
release process tasks:
- do code
- update test cases
- test sample app
- test on another sample app
- get code reviewed and approved by senior ( who takes his own sweet time to review and never approves on first try)
- get code reviewed again
- merge to develop after 20 mins( coz CICD pipeline won't finish and allow merging before that)
- merge to master after 20 mins( coz CICD again)
- realise that you forgot to update dates in markdown files as you thought the release will be on 10th sept and release is happennig on 12th sept coz of sweet senior's code fucking/reviewing time
- again raise a branch to develop
- again get it a review approval by sr (who hopefully gives a merge approval in less time now)
- again get it merged to develop after waiting for 20 mins
- again get it merged to master after waiting for 20 mins
- create a clean build aar file
- publish to sonatype staging
- publish to sonatype release
- wait for 30 mins to show while having your brain fucked with tension
- create a release doc with all the changes
- update the documentation on a wyswig based crappy docs website
- send a message to slack channels
- done

===========

why am i telling you this? coz i just found a bug in a code that i shipped in that release which still got in after all the above shitty processes. its a change of a 3 lines of code, but i will need to do all the steps again. even though i am going through the same shitty steps for another library version upgrade that depends on this library 😭😭

AND I AM THE ONE WHO CAUGHT IT. it went unnoticed because both of those shitty samples did not tested this case. now i can keep mum about it and release another buggy build that depends on it and let the chaos do its work, or i can get the blame and ship a rectification asap. i won't get any reward or good impression for the 2nd, and a time bomb like situation will get created if i go with 1st :/

FML :/

My stress ball has been stolen!

I came in to work to an email alerting me to a bug in production. I copied the site to staging to work on the issue but I was unable to replicate bug. My rubber duck wasn't helping so I went to go bounce my ball off the wall when I realized I don't have a stress ball anymore.

I spent 7 hours working on the bug without a stress ball before finally fixing it. And now I'm ready to deal with the theft the old-fashioned way.

Finally made my node production server stable enough that I could focus on writing tests*. I start by setting up docker, mocking cognito, preparing the database and everything. Reading up on Node test suites and following a short tut to set up my first unit test. Didn't go smoothly, but it's local and there are no deadlines so who cares. 4 days later, first assert.equal(1+1, 2) passes and I'm happy.

I start writing all sorts of tests, installing everything required into "devDependancies," and getting the joy of having some tests pass on first try with all asserts set up, feels good!

I decide to make a small update to production, so I add a test, run and see it fail, implement the feature, re-run and, it passes!

I push the feature to develop, test it, and it works as intended. Merge that to master and subsequently to one of my ec2 production servers**, and lo and behold, production server is on a bootloop claiming it "Cannot find module `graphql`". But how? I didn't change any production dependencies, and my package lock json is committed so wth?

I google the issue, but can't find anything relevant. The only thing that I could guess was that some dependencies (including graphql) were referenced*** in both, prod and dev, and were omitted when installed on a prod NODE_ENV, but googling that specific issue yielded no results, and I would have thought npm would be clever enough to see that and would always install those dependencies (spoiler: it didn't for me).

With reduced production capacity (having one server down) I decided to npm uninstall all dev dependencies anyway and see what happens. Aaaaand it works.....

So now I have a working production server, but broken local tests, and I'm not sure why npm is behaving like this...

* Yes I see the irony.
** No staging because $$$, also this is a personal project.
*** I am not directly referencing the same thing twice, it's probably a subdependency somewhere.

When your staging/test server dies just as the boss wants a big update completed. Straight to production it is!

So I'm currently "assigned" a task in which I need to fix a slow query problem, which isn't a big deal. The biggest problem is that the original team of this project haven't got any means to develop things on your local machine. Looking at their docs and scripts, it seems like everything is deployed to a dev server. But whilst looking for details for this server, I found out that the network team have decommissioned the server!

So my dilemma right now is that I can't test any of my fixes on anywhere besides staging, or possibly production! Inheriting projects is the bloody worst!

Marketing team built out some changes in the staging environment using the CMS, didn't test it, submitted a ticket for cloning with the note that they only changed the content of one page. I check and it works fine, complete the clone. Two weeks go by and I get a ticket saying one of the pages isn't working, I check and it doesn't work because it only exist in staging. Turns out they were hoping to sneak one by me and deploy something that they were trying to get printed for shipping that day in their original request. So now I have to spend the next hour running test, getting approvals, and deploying that page. I need to finish my CI/CD for this site.

Small chaotic startup that never grew up (15 years atm).
Hosts/maintains a number of apps/sites for various customers.

At some point, someone decides that a CMS would be usefull to maintain the content across all products. Forgoing all sense, reason and the very notion of "additional maintenance and dev" it is decided that one should be built in-house.

Fast forward a number of years.

Ops performs routine maintenance on prod-servers. A java-patch accidently knocks out one of the pillars a 3rd party lib the CMS uses for storing images. CMS basically burst in to flames causing a.... significant incident.

Enter yours truly to fix the mess.
Spend a few days replacing the affected 3rd party lib. Run tests on CMS in test and staging environments. Apply java-patch. All seems fine.

When speaking to frontenders and app-devs, a significant hurdle present itself:
All test/staging instances of all websites/apps/etc ALL USE PRODUCTION CMS. Hardcoded. No way around.

There is -no- way to properly test and verify the functionality of any changes made to the home-brewed CMS.
My patch did indeed work in the end.

But did the company learn anything? Did they listen to my reasoning, pleading or even anguished screams for sanity?

No.

Just spent 4-5 hours debugging a failing staging deployment that was using Bower, among many other things.

Turns out bower has a feature where if any library in bower.json has been deleted, it just hangs, and silently waits for nothing.

Fuck bower, and fuck me for using it in 2014!

trying to do a git merge by command line, because my coworker insists on developing directly on the staging server using ssh

I am forced to work with a client's notoriously slow SOAP api. Slow in this case is 1.5-2s per request.
The api is structured rather... creatively... at the same time. So we have to bombard it with thousands of requests to build our data base with historical SOAP data. Also the data sometimes is a couple of hours late, giving a flat line (all values at 0) until retroactively fixing the output for the same requests.
So to fill one dev data base with a year's worth of historical data (nice to have when testing a dashboard application) we hammer the api with ~20k requests (~1 million if we want to be thorough).
Best thing about that: There is no staging/test api and the prod api seems not to handle lots of requests at the same time very well...
Latest thought: Maybe we could put a varnish cache in front of the SOAP for testing. Better have wrong data, than nothing at all and we don't kill the prod clients every time we ramp up a new instance.
Also that would dramatically decrease the 4.2 hours of data pumping to about 7 minutes after the first run.

Trying to setup a staging server boss says just use AWS, system admin doesn't like that thought because it doesn't involve him so waited all day for 3 VM's and still not ready... All I need is 3 blank CoreOS VMs nothing fancy like even doing the cloud config no. Anyone else has colleagues scared of moving out of the private high maintenance servers in the basement?

other team: accidentally deploys app to staging with incorrect config file
app: gets 401 response from our API due to bad credentials
other team: opens ticket claiming our API is broken and whining that it's holding up their release
me: (⌐■_■)

Fucking hate it when I have to ask someone to clean the DB of an app am working on, ON STAGING environment!! 😖

Give me the fucking access will you!!

To my dev manager: hold on... you are annoyed that one of our high profile customers found a bug in our software after upgrading to production because they didn’t test that feature in staging?? Did I hear that right? Yeah, let’s blame our customers for bugs that we introduced and that we clearly didn’t test during QA... that’s what we’re all about over here...

A developer couldn't get a application performance monitoring (APM) tool to trace his application. They claimed that their libraries and their configurations were alright and that the APM tool was non-performant.

The developer then argues with sysadmin that the APM tool can't trace the application and that there's nothing wrong with the application or the configurations. When sysadmin questions whether the developer got the tool to work anywhere, they say, "No" and head off to make it work at least in one place. They come back saying that it works on their development environment (which is their local machine). Sysadmin claims that the system configurations on the server instances cannot be matched by the development environment and there could be a lot more factors to be considered for the problem. The sysadmin asks to prove it on a server instance on one of the test environments and then they'd agree that it is a problem with the tool. They also argue that this is not the only application that uses the APM tool and the tool happily traces other applications with no issues.

The developer tries the same configuration on a staging instance and fails. In order to make it work, they silently uninstall the existing version of the APM tool and then compiles an unstable branch of the tool. It finally works with this version.

They go back to the sysadmin and show that it works on the staging environment, but does not on production. After banging their head on the wall for a while, the sysadmin figure that the tool had been swapped out for the unstable branch that was manually compiled. When questioned, the developer responds, "It works with this version on staging, so deploy the same version on production"

WTF? You don't deploy an unstable branch to production. Just because you can't make it work on the stable branch doesn't mean that it is the problem with the tool itself. There's a big difference between a stable branch and a non-stable branch. How would you feel if the sysadmin retorted by asking you to deploy the staging branch of your application to production?

The typical university project as a group:

- Discuss/Make somebody create the items in Jira: 2 days
- Implementation: 2 hours
- Tests: What are tests? Do I write them in my main method?
- Review/Someone sees the code: 9 hours
- Staging/Production: No one is ever going to use the "software"

Well, seems like my boss realized what he did (see my last rant) and wrote a passive aggressive commit message about himself to make up for it

Too bad staging is still broken 🤣🤣🤣🤣

Well, this happens time to time...

I'm freelancing as a backend guy. I like to take care of all infrastructure before really starting to build anything, this mostly includes dev/staging/prod environments with some linear promotion strategy. So.. I did this API. Still on staging, proceeding with the development as planned, everything goes according the timeline.

And then.. this happens... At some point PM told frontend guy that it's time for production (without notifying me), so the frontend guy does what "anyone" would do in this case - tells PM to create DNS record for production to point to staging app.

Time passes, I'm still unaware of this. But I'm starting to see some quality entries in the DB, not the usual QA crap. I write to them that they're doing good job and continue with my tasks.

One of the tasks required some major DB change. I could've written migrations script, but since we're not in "production" yet, I just wipe the DB and recreate schema as I need it.

In 10 minutes the furious PM starts shouting that "production" is down and I need to fix it ASAP.

I'm lost, I'm asking questions, I'm slowly understanding what's happening...

So I want to grab some coffee, sat back down, wrote politely that they suck, added a finger emoji and terminated the contract.

Felt like the right thing to do as I definitely don't want to continue within the same "team".

Helping to fix legacy code on a staging server. No version control (at least not that I am aware of). Besides rare code comments, no way to see the author, time, or even purpose of customizations that have been made. No fun!

Why is live user data in the staging environment?! 😬😳

Got a new devops "manger" today we had an argument for 20 mins about why Staging/UAT was needed and why we could not just by pass it and get to prod quicker .. WHAT?! I am dumbfounded I do not have words to express the emotions I am having right now.

A top food chain client wants a feature Fx
and has a deadline on Friday.
We are still working on it and already estimated hours and set deployment on Monday.
(No deployments on Friday)
And the business/sales guy comes up with new deadline to submit it at Friday morning.
And was only discussing with one of my team member already working on it. And i knew there is more hours required for testing and need to deployment pre deployment phase (staging of dev)
I was over hearing the conversation between them and I got pissed off and jumped in and said Not Possible at all.

He tries to argues about giving something to him. I said we can give it to you but will not garauntee anything. Now project manager jumps in. PM and my team already know that we will be delivering on Monday.

He arguing that if the Fx is not ready then I will call client developer to office to test it directly on my team members laptop.
I said, No way. We are not ready yet and havent finished yet. Major work will be on Thursday and on Friday we will be testing till end of the day.

PM explains him blah blah stuff.
He calms down and says no worries we will check the status on Friday afternoon amd roll out something to Client.
PM, developer and I looked each other and I said, sure will deploy but will not garauntee anything. He goes back to his desk.

Seriously.
WE ALREADY ESTIMATED F* MAN HOURS AND WILL BE READY ON MONDAY MEANS MONDAY DONT F* BUILD MORE PRESSURE ON US. F* SALES

> push code on staging and promote on production works fine
>.come home
> pull from master and code doesn't work locally
> pull from the feature branch, same issue
> fml

I'd be so fucking disappointed in myself if I hadn't pushed it and merged it.

When it comes to config files for any kind of application, I tend to make sure that I understand what each config does, and that for each environment, they have the correct settings.

However some of my co-workers don't bother understanding what the configs mean and so I have people copying and pasting config details from development environments into staging and production configs. They think that just changing hostnames is more than enough.

So they ended up wasting a good 3 hours trying to figure out why sessions were always invalidating and cached objects were not caching. I had a look at the config and realised their mistake in like 3 mins.

After three months of development, my first contribution to the client is going live on their servers in less than 12 hours. And let me say, I shall never again be doing that much programming in one go, because the last week and a half has been a nightmare... Where to begin...

So last Monday, my code passed to our testing servers, for QA to review and give its seal of approval. But the server was acting up and wouldn't let us do much, giving us tons of timeouts and other errors, so we reported it to the sysadmin and had to put off the testing.

Now that's all fine and dandy, but last Wednesday we had to prepare the release for 4 days of regression testing on our staging servers, which meant that by Wednesday night the code had to be greenlight by QA. Tuesday the sysadmin was unable to check the problem on our testing servers, so we had to wait to Wednesday.

Wednesday comes along, I'm patching a couple things I saw, and around lunch time we deploy to the testing servers. I launch our fancy new Postman tests which pass in local, and I get a bunch of errors. Partially my codes fault, partially the testing env manipulating server responses and systems failing.

Fifteen minutes before I leave work on the day we have to leave everything ready to pass to staging, I find another bug, which is not really something I can ignore. My typing skills go to work as I'm hammering line after line of code out, trying to get it finished so we can deploy and test when I get home. Done just in time to catch the bus home...

So I get home. Run the tests. Still a couple failures due to the bug I tried to resolve. We ask for an extension till the following morning, thus delaying our deployment to staging. Eight hours later, at 1AM, after working a full 8 hours before, I push my code and leave it ready for deployment the following morning. Finally, everything works and we can get our code up to staging. Tests had to be modified to accommodate the shitty testing environment, but I'm happy that we're finally done there.

Staging server shits itself for half a day, so we end up doing regression tests a full day late, without a change in date for our upload to production (yay...).

We get to staging, I run my tests, all green, all working, so happy. I keep on working on other stuff, and the day that we were slated to upload to production, my coworkers find that throughout the development (which included a huge migration), code was removed which should not have. Team panics. Everyone is reviewing my commits (over a hundred commits) trying to see what we're missing that is required (especially legal requirements). Upload to production is delayed one day because of this. Ended up being one class missing, and a couple lines of code, which is my bad (but seriously, not bad considering I'm a Junior who was handed this project as his first task at his first job).

I swear to God, from here on out, one feature per branch and merge request. Never again shall I let this happen. I don't even know why it was allowed to happen, it breaks our branch policies. But ohel... I will now personally oppose crap like this too...

Now if you'll excuse me... I'm going to be highly unproductive and rest, because I might start balding otherwise after these weeks...

I'm losing my mind.

It turns out that the API test environment we are using to test a new payment method is their development/staging environment. A couple of hours ago, they fucked up something, and we have been stuck because of that.

client: "can you build out a staging server for us? here's all the code, everything you need"
me: "awesome, looking good, i have almost everything i need, just give me the credentials for the server, and I'll get started installing all the infrastructure"
client: "ok, try these!"
me: "doesn't work"
client: "this one?"
me: "doesn't work..."
client: "how about this one?"
me: "STILL NOT WORKING!!!"

imagine you want someone to do stuff on your server and you don't even know the root SSH password.... smh

why is this always a problem, use fucking 1password or something its 40 bucks a year, secure, and you can organize alllll your passwords. don't be a fucking boomer and write them on a piece of paper, or worse, apparently like my client, never know it or have it in the first place.

Had a client whom was using the staging system on my server as cdn, remote computing, etc... because his prod server was a cheap vhost while the vm was a beast compared to it. I shut it down without telling. I just got a call that his site is now slow a f and full of errors.

I kindly told him that there was a recent security breach called dirty cow. Then I told him that I shut the vm down because it would mean security risk for him since there are no patches available yet and only Power on again with there was work for me to do.

If you want resources pay for them

"You shouldn't mark things as done if they aren't. It's only done when I can see it on the server and demo it."

Well, I just demoed it to you, you prick. The fact that it's not running on a server is because that AWS endpoint we have there is no where near being able to be called "staging" even, mainly because the other dev on the team hasn't committed their work in 8 days, let alone push it to said server. Data models have changed, APIs have changed, hell, the god forsaken Sahara desert is now green and blooming as far as I'm concerned.

So instead of trying to look smart to your boss, why don't you ask first you obnoxious waste of organic matter. Stop breathing our oxygen for once. There are more useful things to do with it.