Worst thing you've seen another dev do? Long one, but has a happy ending.

Classic 'Dev deploys to production at 5:00PM on a Friday, and goes home.' story.

The web department was managed under the the Marketing department, so they were not required to adhere to any type of coding standards and for months we fought with them on logging. Pre-Splunk, we rolled our own logging/alerting solution and they hated being the #1 reason for phone calls/texts/emails every night.

Wanting to "get it done", 'Tony' decided to bypass the default logging and send himself an email if an exception occurred in his code.

At 5:00PM on a Friday, deploys, goes home.

Around 11:00AM on Sunday (a lot folks are still in church at this time), the VP of IS gets a call from the CEO (who does not go to church) about unable to log into his email. VP has to leave church..drive home and find out he cannot remote access the exchange server. He starts making other phone calls..forcing the entire networking department to drive in and get email back up (you can imagine not a group of happy people)

After some network-admin voodoo, by 12:00, they discover/fix the issue (know it was Tony's email that was the problem)

We find out Monday that not only did Tony deploy at 5:00 on a Friday, the deployment wasn't approved, had features no one asked for, wasn't checked into version control, and the exception during checkout cost the company over $50,000 in lost sales.

Was Tony fired? Noooo. The web is our cash cow and Tony was considered a top web developer (and he knew that), Tony decided to blame logging. While in the discovery meeting, Tony told the bosses that it wasn't his fault logging was so buggy and caused so many phone calls/texts/emails every night, if he had been trained properly, this problem could have been avoided.

Well, since I was responsible for logging, I was next in the hot seat.

For almost 30 minutes I listened to every terrible thing I had done to Tony ever since he started. I was a terrible mentor, I was mean, I was degrading, etc..etc.
Me: "Where is this coming from? I barely know Tony. We're not even in the same building. I met him once when he started, maybe saw him a couple of times in meetings."
Andrew: "Aren't you responsible for this logging fiasco?"
Me: "Good Lord no, why am I here?"
Andrew: "I'll rephrase so you'll understand, aren't you are responsible for the proper training of how developers log errors in their code? This disaster is clearly a consequence of your failure. What do you have to say for yourself?"
Me: "Nothing. Developers are responsible for their own choices. Tony made the choice to bypass our logging and send errors to himself, causing Exchange to lockup and losing sales."
Andrew: "A choice he made because he was not properly informed of the consequences? Again, that is a failure in the proper use of logging, and why you are here."
Me: "I'm done with this. Does John know I'm in here? How about you get John and you talk to him like that."
'John' was the department head at the time.
Andrew:"John, have you spoken to Tony?"
John: "Yes, and I'm very sorry and very disappointed. This won't happen again."
Me: "Um...What?"
John: "You know what. Did you even fucking talk to Tony? You just sit in your ivory tower and think your actions don't matter?"
Me: "Whoa!! What are you talking about!? My responsibility for logging stops with the work instructions. After that if Tony decides to do something else, that is on him."
John: "That is not how Tony tells it. He said he's been struggling with your logging system everyday since he's started and you've done nothing to help. This behavior ends today. We're a fucking team. Get off your damn high horse and help the little guy every once in a while."
Me: "I don't know what Tony has been telling you, but I barely know the guy. If he has been having trouble with the one line of code to log, this is the first I've heard of it."
John: "Like I said, this ends today. You are going to come up with a proper training class and learn to get out and talk to other people."

Over the next couple of weeks I become a powerpoint wizard and 'train' anyone/everyone on the proper use of logging. The one line of code to log. One line of code.

A friend 'Scott' sits close to Tony (I mean I do get out and know people) told me that Tony poured out the crocodile tears. Like cried and cried, apologizing, calling me everything but a kitchen sink,...etc. It was so bad, his manager 'Sally' was crying, her boss 'Andrew', was red in the face, when 'John' heard 'Sally' was crying, you can imagine the high levels of alpha-male 'gotta look like I'm protecting the females' hormones flowing.

Took almost another year, Tony released a change on a Friday, went home, web site crashed (losses were in the thousands of $ per minute this time), and Tony was not let back into the building on Monday (one of the best days of my life).

Sometime in mid 2013 or 2014 as a junior dev I woke up to a call from my company's CEO. He informed me that the legacy system they use for order processing is down nationwide that nobody can add new orders until it's fixed and that I needed to fix it. I had been working there 6 months and was hired along with a senior dev to begin developing a web app to replace this legacy system. The senior dev had left the company two weeks earlier for a better offer so it was put on me to figure it out. I was very frank with the CEO and told him I didn't know if I could fix it and suggested he try to call the company they hired to create it. I didn't even know where the source code was let alone what the design paradigm was or whether or not there was any documentation. He said he would try figuring out who created it and give them a call and asked "As a developer you shouldn't you be able to fix this?" I just told him it wasn't that simple and left it at that.

I get to work and the CEO has discovered that the company who created the software no longer exists and I tell him he may need to find a company to consult on this if I can find the source code and if I can't find the code he might be screwed.
I found the source code in a random IT shared folder there is no source control, no documentation, no unit tests, no test environment, and it looks like nobody had touched it since 2005 or about 8 years.

Despite being completely unfamiliar with the code and the design paradigm I was able to figure out that they were validating customer addresses against an old Google geocoding API that was shutdown the day before and the lack of response was killing the application. I fixed the issue and warned the CEO before deployment that I wasn't able to test but he said to go ahead and thankfully all went well.

Most satisfying bug I've fixed?

Fixed a n+1 issue with a web service retrieving price information. I initially wrote the service, but it was taken over by a couple of 'world class' monday-morning-quarterbacks.
The "Worst code I've ever seen" ... "I can't believe this crap compiles" types that never met anyone else's code that was any good.

After a few months (yes months) and heavy refactoring, the service still returned price information for a product. Pass the service a list of product numbers, service returns the price, availability, etc, that was it.

After a very proud and boisterous deployment, over the next couple of days the service seemed to get slower and slower. DBAs started to complain that the service was causing unusually high wait times, locks, and CPU spikes causing problems for other applications. The usual finger pointing began which ended up with "If PaperTrail had written the service 'correctly' the first time, we wouldn't be in this mess."

Only mattered that I initially wrote the service and no one seemed to care about the two geniuses that took months changing the code.

The dev manager was able to justify a complete re-write of the service using 'proper development methodologies' including budgeting devs, DBAs, server resources, etc..etc. with a projected year+ completion date.

My 'BS Meter' goes off, so I open up the code, maybe 5 minutes...tada...found it. The corresponding stored procedure accepts a list of product numbers and a price type (1=Retail, 2=Dealer, and so on). If you pass 0, the stored procedure returns all the prices.
Code basically looked like this..

public List<Prices> GetPrices(List<Product> products, int priceTypeId)
{
foreach (var item in products)
{
List<int> productIdsParameter = new List<int>();
productIdsParameter.Add(item.ProductID);
List<Price> prices = dataProvider.GetPrices(productIdsParameter, 0);
foreach (var price in prices)
{
if (price.PriceTypeID == priceTypeId)
{
prices = dataProvider.GetPrices(productIdsParameter, price.PriceTypeID);
return prices;
}
* Omitting the other 'WTF?' code to handle the zero price type
}
}
}

I removed the double stored procedure call, updated the method signature to only accept the list of product numbers (which it was before the 'major refactor'), deployed the service to dev (the issue was reproducible in our dev environment) and had the DBA monitor.

The two devs and the manager are grumbling and mocking the changes (they never looked, they assumed I wrote some threading monstrosity) then the DBA walks up..

DBA: "We're good. You hit the database pretty hard and the CPU never moved. Execution plans, locks, all good to go."
<dba starts to walk away>
DevMgr: "No fucking way! Putting that code in a thread wouldn't have fix it"
Me: "Um, I didn't use threads"
Dev1: "You had to. There was no way you made that code run faster without threads"
Dev2: "It runs fine in dev, but there is no way that level of threading will work in production with thousands of requests. I've got unit tests that prove our design is perfect."
Me: "I looked at what the code was doing and removed what it shouldn't be doing. That's it."
DBA: "If the database is happy with the changes, I'm happy. Good job. Get that service deployed tomorrow and lets move on"
Me: "You'll remove the recommendation for a complete re-write of the service?"
DevMgr: "Hell no! The re-write moves forward. This, whatever you did, changes nothing."
DBA: "Hell yes it does!! I've got too much on my plate already to play babysitter with you assholes. I'm done and no one on my team will waste any more time on this. Am I clear?"

Seeing the dev manager face turn red and the other two devs look completely dumbfounded was the most satisfying bug I've fixed.

Our web department was deploying a fairly large sales campaign (equivalent to a ‘Black Friday’ for us), and the day before, at 4:00PM, one of the devs emails us and asks “Hey, just a heads up, the main sales page takes almost 30 seconds to load. Any chance you could find out why? Thanks!”
We click the URL they sent, and sure enough, 30 seconds on the dot.
Our department manager almost fell out of his chair (a few ‘F’ bombs were thrown).
DBAs sit next door, so he shouts…
Mgr: ”Hey, did you know the new sales page is taking 30 seconds to open!?”
DBA: “Yea, but it’s not the database. Are you just now hearing about this? They have had performance problems for over week now. Our traces show it’s something on their end.”
Mgr: “-bleep- no!”
Mgr tries to get a hold of anyone …no one is answering the phone..so he leaves to find someone…anyone with authority.
4:15 he comes back..
Mgr: “-beep- All the web managers were in a meeting. I had to interrupt and ask if they knew about the performance problem.”
Me: “Oh crap. I assume they didn’t know or they wouldn’t be in a meeting.”
Mgr: “-bleep- no! No one knew. Apparently the only ones who knew were the 3 developers and the DBA!”
Me: “Uh…what exactly do they want us to do?”
Mgr: “The –bleep- if I know!”
Me: “Are there any load tests we could use for the staging servers? Maybe it’s only the developer servers.”
DBA: “No, just those 3 developers testing. They could reproduce the slowness on staging, so no need for the load tests.”
Mgr: “Oh my –bleep-ing God!”
4:30 ..one of the vice presidents comes into our area…
VP: “So, do we know what the problem is? John tells me you guys are fixing the problem.”
Mgr: “No, we just heard about the problem half hour ago. DBAs said the database side is fine and the traces look like the bottleneck is on web side of things.”
VP: “Hmm, no, John said the problem is the caching. Aren’t you responsible for that?”
Mgr: “Uh…um…yea, but I don’t think anyone knows what the problem is yet.”
VP: “Well, get the caching problem fixed as soon as possible. Our sales numbers this year hinge on the deployment tomorrow.”
- VP leaves -
Me: “I looked at the cache, it’s fine. Their traffic is barely a blip. How much do you want to bet they have a bug or a mistyped url in their javascript? A consistent 30 second load time is suspiciously indicative of a timeout somewhere.”
Mgr: “I was thinking the same thing. I’ll have networking run a trace.”

4:45 Networking run their trace, and sure enough, there was some relative path of ‘something’ pointing to a local resource not on development, it was waiting/timing out after 30 seconds. Fixed the path and page loaded instantaneously. Network admin walks over..
NetworkAdmin: “We had no idea they were having problems. If they told us last week, we could have identified the issue. Did anyone else think 30 second load time was a bit suspicious?”
4:50 VP walks in (“John” is the web team manager)..
VP: “John said the caching issue is fixed. Great job everyone.”
Mgr: “It wasn’t the caching, it was a mistyped resource or something in a javascript file.”
VP: “But the caching is fixed? Right? John said it was caching. Anyway, great job everyone. We’re going to have a great day tomorrow!”
VP leaves
NetworkAdmin: “Ouch…you feel that?”
Me: “Feel what?”
NetworkAdmin: “That bus John just threw us under.”
Mgr: “Yea, but I think John just saved 3 jobs. Remember that.”

So yesterday I deployed a build on our release environment and i had added a new rest api end-point which I needed to test.. A heads up though, its written in java spring and the entire flow consisted of too many calls/returns from various other java & python services.. Also to make things worse, the entire deployment is a really cumbersome process as you need to copy the build from one box to another..

After like almost 4-5 hours of debugging, adding logs left right & center, crazy upload speeds (yaa this is sarcastic) and frustation at its peak, I found the issue..

There was an if condition that was checking for equality between an enum constant & an enum in a request aaaannnnnddd

*Drum roll

THE CONSTANT ENUM BELONGED TO THE WRONG PACKAGE HENCE ALWAYS EVALUATING TO FALSE... ALSO, BOTH THE ENUMS IN THE DIFFERENT PACKAGES ARE IDENTICAL... FUCCKKKKKKK MY LIFE

😑🔫

!rant

So I am building a website and already finished it and ready to be deployed.
Got server access and found that there are many other websites files and folder.

Note : multiple copies of website that I am working on.

So I raise my concern with client about which folder is actually handling the website. She contact the old dev guy and got the webroot for that website. She told me to delete everything and clean it up and make tge current websites available.

I realize that there are many other websites may rely on those files and folder and inform about the consequences. And finally told her I will not touch any files except for the website I am working on.

So, I deployed the website and works good as expected. After 3 weeks clients comes back with an issue that one of her websites is notworking and trying to blame me for it.

Fyi : before my deploymemt she says that she will talk to hosting providers to clean their webspace. I am not sure on that yet.

So I tried to do auditing and found that after 2 days frm my deployment some has already wiped everything from those other websites.
I showed my audits and its someone did it.
You should contact your hosting provider or the other dev who she contacted for wiping.

From the begining I am telling that lady to take backups. She said no need.

The reason I didnt took it because I am not working on those website and obviously client isnt going to pay me for those back.

There is still more to go waiting for her resonses.

Me: Hey Guys we've been working on this application(project 1) for 4 months and i think we're almost done.

Owner of Company(Not My Boss): CooCook4Choo we moving you to project 2, forget about the previous one.

2 months go by, project is completed.

Boss: I've got another project for you
Me: Awesome!

1 month later...

PM: We're moving you back to project 1
Me: Why?
PM: Our senior dev resigned, we only have junior Devs and we need a lot of help before deployment next month.
Me: Why am i moving back to a project i was taken off of
PM: Where an agile company and you will be moved off many projects
Me: **Fuuuuuuuuuuck!!!* Ok i'll need documentation of everything that happened in the past three months, the current issue, what the current sprint revolves around and A demo of what has been added.
PM: Relax, I've got a lot of work myself, you will get them soon.

2 days later, still don't have what i need, PM is on vacation.

Me: Guess i don't have any work to do.

I was on vacation when my employer’s new fiscal year started. My manager let me take vacation because it’s not like anything critical was going to happen. Well, joke was on us because we didn’t foresee the stupidity of others…

I had to update a few product codes in the website’s web config and deploy those changes. I was only going to be logged in for 30 minutes to complete that.

I get messaged by one of our database admins. He was doing testing and was unable to complete a payment on the website. That was strange. There was a change pushed by our offsite dev agency, but that was all frontend changes (just updating text) and wouldn’t affect payments.

We don’t want to enlist the dev agency for debugging work, especially when it’s not likely that it’s a code issue. But I was on vacation and I couldn’t stay online past the time I had budgeted for. So my employer enlists the dev agency for help. It’s going to be costly because the agency is in Lithuania, it was past their business hours, and it was emergency support.

Dev agency looks at error logs. There are Apple Pay errors, but that doesn’t explain why non Apple Pay transactions aren’t going through. They roll back my deployment and theirs, but no change. They tell my employer to contact our payment processor.

My manager and the Product Manager contact Payroll, who is the stakeholder for our payment gateways. Payroll contacts our payment gateway and finds out a service called Decision Manager was recently configured for our account. Decision Manager was declining all payments. Payroll was not the person who had Decision Manager installed and our account using this service was news to her.

Payroll works with our payment processor to get payments working again. The damage is pretty severe. Online payments were down for at least 12 hours. Our call center had logged reports from customers the night before.

At our post mortem, we had to find out who ok’d Decision Manager without telling anyone. Luckily, it was quick work. The first stakeholder up was for the Fundraising Dept. She said it wasn’t her or anyone on her team. Our VP of Analytics broke it to her that our payment processor gave us the name of the person who ok’d Decision Manager and it was someone on the Fundraising team. Fundraising then starts backtracking and says that oh yes she knew about it but transactions were still working after the Decision Manager had been configured. WTAF.

Everyone is dumbfounded by this. How could you make a big change to our payment processor and not tell anyone? How did our payment processor allow you to make this change when you’re not the account admin (you’re just a user)?

Our company head had to give an awkward speech about communication and how it’s important. The web team can’t figure out issues if you don’t tell us what you did. The company head was pissed because it was a shitty way to start off the new fiscal year. Our bill for the dev agency must have been over $1000 for debugging work that wasn’t helpful.

Amazingly, no one was fired.

I think I want to quit my first applicantion developer job 6 months in because of just how bad the code and deployment and.. Just everything, is.

I'm a C#/.net developer. Currently I'm working on some asp.net and sql stuff for this company.

We have no code standards. Our project manager is somewhere between useless and determinental. Our clients are unreasonable (its the government, so im a bit stifled on what I can say.) and expect absurd things from us. We have 0 automated tests and before I arrived all our infrastructure wasn't correct to our documentation... And we barely had any documentation to begin with.

The code is another horror story. It's out sourced C# asp.net, js and SQL code.. And to very bad programmers in India, no offense to the good ones, I know you exist. Its all spagheti. And half of it isn't spelled correctly.

We have a single, massive constant class that probably has over 2000 constants, I don't care to count. Our SQL projects are a mess with tons of quick fix scripts to run pre and post publishing. Our folder structure makes no sense (We have root/js and root/js1 to make you cringe.) our javascript is majoritly on the asp.net pages themselves inline, so we don't even have minification most of the time.

It's... God awful. The result of a billion and one quick fixes that nobody documented. The configuration alone has to have the same value put multiple times. And now our senior developer is getting the outsourced department to work on moving every SINGLE NORMAL STRING INTO THE DATABASE. That's right. Rather then putting them into some local resource file or anything sane, our website will now be drawing every single standard string from the database. Our SENIOR DEVELOPER thinks this is a good idea. I don't need to go into detail about how slow this is. Want to do it on boot? Fine. But they do it every time the page loads. It's absurd.

Our sql database design is an absolute atrocity. You have to join several tables together just to get anything done. Half of our SP's are failing all the time because nobody really understands the design. Its gloriously awful its like.. The epitome of failed database designs.

But rather then taking a step back and dealing with all the issues, we keep adding new features and other ones get left in the dust. Hell, we don't even have complete browser support yet. There were things on the website that were still running SILVERLIGHT. In 2019. I don't even know how to feel about it.

I brought up our insane technical debt to our PM who told me that we don't have time to worry about things like technical debt. They also wouldn't spend the time to teach me anything, saying they would rather outsource everything then take the time to teach me. So i did. I learned a huge chunk of it myself.

But calling this a developer job was a sick, twisted joke. All our lives revolve around bugnet. Our work is our BN's. So every issue the client emails about becomes BN's. I haven't developed anything. All I've done is clean up others mess.

Except for the one time they did have me develop something. And I did it right and took my time. And then they told me it took too long, forced me to release before it was ready, even though I had never worked on what I was doing before. And it worked. I did it.

They then told me it likely wouldn't even be used anyway. I wasn't very happy at all.

I then discovered quickly the horrors of wanting to make changes on production. In order to make changes to it, we have to... Get this

Write a huge document explaining why. Not to our management. To the customer. The customer wants us to 'request' to fix our application.

I feel like I am literally against a wall. A huge massive wall. I can't get constent from my PM to fix the shitty code they have as a result of outsourcing. I can't make changes without the customer asking why I would work on something that doesn't add something new for them. And I can't ask for any sort of help, and half of the people I have to ask help from don't even speak english very well so it makes it double hard to understand anything.

But what can I do? If I leave my job it leaves a lasting stain on my record that I am unsure if I can shake off.

... Well, thats my tl;dr rant. Im a junior, so maybe idk what the hell im talking about.

1. high severity production incident was asked to look into at the end of the day.
2. needed fix in ui.
3. fixed and deployed in 1 hour.
4. issue remained. debugging began.
5. gave up at 1 AM and went to sleep.
6. woke up at 6 and after debugging for 2 hours, identified to be a back end issue.
7. worked with back end team for the fix, and 6 hours and 3 deployments later, it worked.
8. third party vendor reported they are still not receiving one parameter from us.
9. back end team realised they forgot to ask ui to send another parameter.
10. added the parameter in ui, redeployed ui.
11. build and deployment tool broke down. got it fixed. delay of 1.5 hours.
12. finally things are in place. total time 26 hours.

13. found half bottle of vodka, leftover from last weekend. *Priceless*

The only thing more dangerous than an alcoholic short-term-memory-challenged non-technical throw-you-under-the-bus IT director with self-esteem issues that are sporadically punctuated by delusions of superiority is one who fears for his job. Submitted for your inspection: a besotted mass of near-human brain function who not only has a 50 person IT department to run, but has also been questioned by the business owners as to what he actually does. So he has decided to show them. He has purchased a vendor product to replace a core in-house developed application used to facilitate creating the product the business sells. The purchased software only covers about 40 percent of the in-house application's functionality, so he is contracting with the vendor to perform custom development on the purchased product (at a cost likely to be just shy of six-figures) so that about 90 percent of existing functionality will be covered. He has asked one of his developers (me) to scale down the existing software to cover the functionality gaps the purchased software creates. There is no deployment plan that will allow the business to transition from the current software to the new vendor​-supplied one without significantly hurting the ability of the business to function. When anyone raises this issue he dismisses it with sage musings such as, "I know it will be painful, but we'll just have to give the users really good support." Because he has no idea what any of his staff actually does, he is expecting one of his developers (again, unfortunately, me) to work with the vendor so that the Frankensoftware will perform as effectively as the current software (essentially as a project manager since there will be no in-house coding involved). Lastly, he refuses to assign someone to be responsible for the software: taking care of maintenance, configuration, and issue resolutions after it has been rolled out. When I pointedly tell him I will not be doing that (because this is purchased software and I am not a system admin or desktop engineer) he tells me, "Let me think about this." The worst part is that this is only one of four software replacement initiatives he is injecting himself into so he can prove his worth to the business owners. And by doing so he is systematically making every software development initiative akin to living in Dante's Eighth Circle. I am at the point where I want to burn my eye out with a hot poker, pour salt into the wound, and howl to the heavens in unbearable agony for a month, so when these projects come to fruition, and I am suffering the wrath of the business owners, I can look back on that moment I lost my eye and think "good times."

Client: THIS IS CRITICAL, SOME DATA HAS BEEN DELETED, WHAT ZE FUUK HAPPENED, UNDO THIS FAST
Us: so after carefully reviewing the code, related resources and the network traffic we conclude that was never sent in the first place.
*closes issue*

I'm glad we got such a meaningful bug report on the same day a production system started failing, one big deployment that that was like a boss with 3 phases, an unnecessary long meeting and an app developer that that wanted me to break HTTP standards.

Why am I sad, depressed, demotivated, you ask?
Because I was asked to create-react-app with nodemailer, it worked well on heroku, YAYYY MEE, "

"NOTHING GOES WRONG IN DEPLOYMENT FUCK YEAH"

Little did I know that was a "demo" for the business people, My superior / manager/boss wants me to deploy on 1and1 service provider,

> Okay 1 and 1 service provider does provide Nodej, so it shouldn't be hard.
> Turns out it is a Windows hosting server IIS 10 without URL Rewrite.
> *INTERNAL SCREAMING*
I went up to him to talk about this issue and requested to let me talk to 1 and 1, and get this sorted

> But bro, if we cannot fix it, I think they also cannot fix, probably.

*INTERNAL SCREAMING AT PEAK*

I just want URL Rewrite installed on IIS10 so that I can move on to the next project.
A little background for this project
> No support from him during development.

> I personally used HD Images, because why not?

> Website seems slow because of HD Images, and now he complains about it.

You fucking (managers) want a website to be scalable and fast and yet you choose to focus on B U S I N E S S instead of support the real guy.

I'm fucking sick and tired, it took me 24 hours figure out the issue because there is nothing on 1 and 1 support/ forum/help center.

Another 24 hours to try and fix, yet no luck.

I'm gonna finally point the domain name to heroku. Fuck, I'm so fucking done

Strap in...

- Previous employer
- 3rd party partner firm
- integration link between both over SOAP
- Both sides riddled with poor code and messed up political structures (partner firm CEO is an investor in my employer)
- Doing a deployment to update to https (I know)
- Keep http endpoint live
- Other side starts shitting itself
- Diagnose
- Not us
- feelsgoodman.tiff
- Get angry email
- Explain not us
- Back and forth
- Tell client it’s “irrelevant” on https issue, it’s their side that’s gone wrong
- Get angry reply with boss cc’d about how nothing is “irrelevant” for the client
- We all had to have a make up meeting and meal
- Client was calm and reasonable, all agreed we just snapped and it wouldn’t happen again
- 2 weeks later
- Their system shits itself again and suddenly we’re on the hook
- BA on my team (smarmy little bastard) constantly fucking me off
- Get so close to actually screaming and hitting him

So yeah. I don’t tend to hold that a job is more important to me than my dignity.

I have and will never hold my tongue for the sake of a job, I’m not gonna put up with people shouting / belittling / backstabbing etc.

procrastinating by getting drunk since 11:00 AM, and writing specs for my (hypothetical) language/os/platform.

feeling righteous retribution because the client made me be stressed for 3 hours due to an issue that THEY caused but for 3 hours the only info I had was "there's a critical blocker issue and we're convinced YOU caused it"

well... no... i did NOT cause the fact that you UPGRADED PHP DURING THE WEEKEND BEFORE MONDAY'S PRESENTATION TO CLIENT (while waiting for an urgent commit from me).

seriously.

also, germans. i've heard many times from other people that they're... basically racist towards us (slavic nations), thinking of us as untermensch, coal-miner peons, but I didn't realize their passive-aggressive covertly smug demanding attitude is due to this, I just assumed it's a reaction to me being incompetent.

so yesterday when we finished the call (in preparation for which I tried to switch to their "client demonstration" branch since that's where the error was, and I wanted a headstart on fixing it, ended up in a place that my today's whole-day task should be "rebuilding the DB into working condition", because there's about 10 "core" sql scripts in two different folders, which need to be run (in a very specific order, of course, which readme tells you, but what it tells you has been outdated at least for 3 months, of course), and

...THE MAIN CORE SCRIPT THAT IS THE FIRST TO RUN, THAT CREATES THE DB schema, HAS THREE SYNTAX-LEVEL typos which fail it mid-way...

...the joys of continuous deployment via scripts, I guess? I would love to challenge any person from them to screenshare to me, manual deployment of the current version from zero, and I would be willing to give the person 20% of my monthly salary if they would be able to do it within 20 minutes.

but... well...
the point is, i should be doing not entirely bullshit stuff.
but yesterday's 6 hours of being in "at full attention because it seems we fucked up" totally convinced me, that today I'm taking a break.

So I'm gonna go buy another 3 beers and continue writing the specs of my dream language/os/platform.

Editing java files on server for deployment.

Deployed. Found major issue.
Don't have time. Editing source code on server changed if else
Rebuild and deploy.
Now push from server to repo and pull locally...
Pure sin.

Long time ago i ranted here, but i have to write this off my chest.

I'm , as some of you know, a "DevOps" guy, but mainly system infrastructure. I'm responsible for deploying a shitload of applications in regular intervals (2 weeks) manually through the pipeline. No CI/CD yet for the vast majority of applications (only 2 applications actually have CI/CD directly into production)

Today, was such a deployment day. We must ensure things like dns and load balancer configurations and tomcat setups and many many things that have to be "standard". And that last word (standard) is where it goes horribly wrong

Every webapp "should" have a decent health , info and status page according to an agreed format.. NOPE, some dev's just do their thing. When bringing the issue up to said dev the (surprisingly standard) answer is "it's always been like that, i'm not going to change". This is a problem for YEARS and nobody, especially "managers" don't take action whatsoever. This makes verification really troublesome.

But that is not the worst part, no no no.

the worst is THIS:

"git push -a origin master"

Oh yes, this is EVERYWHERE, up to the point that, when i said "enough" and protected the master branch of hieradata (puppet CfgMgmt, is a ENC) people lots their shits... Proper gitflow however is apparently something otherworldly.

After reading this back myself there is in fact a LOT more to tell but i already had enough. I'm gonna close down this rant and see what next week comes in.

There is a positive thing though. After next week, the new quarter starts, and i have the authority to change certain aspects... And then, heads WILL roll on the floor.

Just launched a successful deployment today. Took 4 months to build everything. It's probably not a big deal but I'm just really happy today that everything worked right away, and almost no deployment hiccups at all.

Only one issue popped up, but come to find out it's a particular thing about the Prod environment and nothing to do with my code.

Gonna go celebrate now, before more work comes in. (hey, can you refactor this for me?)

Email from a department mgr regarding a sharepoint site we inherited (lots of custom javascript, XLS, etc, stuff we didn't write)

Dan: "The department filter isn't showing up when I select the 'Logistics and Support' department. Was this caused by the changes you guys made? Its causing a major disruption in our processes and need it fixed ASAP."
Me: "Those changes went out almost two months ago and all the filters were working fine, at least that is what you told me when you tested it."
Dan: "I thought so, but its not working. It has probably been broken ever since you made those changes so I filed a corrective action ticket against your department for not following the documented deployment and testing processes"
Me: "Really? We've been over this. Its your department that is responsible for that sharepoint site. Previous developers hacked javacript together to make it all work, but I'm sure its something simple."
Dan: "Great. I'll start putting together a root-cause analysis to determine which of your processes we need to address."

Start looking at the javascript and found the issue..

if (dept === "Logistics & Support") {
$('deptFilter').show();
}
else {
$('deptFilter').hide();
}

Me: 'Found the issue. Did you rename the logistics department?'
Dan: 'No'
Me: 'To show or hide the filter, the code was looking for "Logistics & Support", someone changed the title to "Logistics and Support"'
Dan: "Well...I guess I did that yesterday...but I didn't change the name, just that stupid character. That shouldn't make any difference."
Me: "I can fix that right now. Are you going to need more information for your root cause analysis?"
Dan: "No, I think we're good. Thanks."

So, it's been a while since I've been working on my current project and I've never had the "luck" to touch the legacy project wrote in PHP, until this week when I got my first issue.

And damn, this goddamn issue. It was a bug, a very strange bug, that only happens in production and that nobody has any idea what was happening, so yeah, I didn't have anyone to ask and I got less time than usual ( because Thanksgiving ).

And thus, I have no starting point, no previous knowledge on PHP and less time! I expected a very fun week 😀 and it was beyond my expectations.

First I tried to understand what might be causing the issue, but there wasn't any real clue to star with, so no choice, time to read the flow on the code and see what are they're doing and using ( 1k line files, yay, legacy ). Luckily I got some clues, we're using a cookie and a php session variable for the session, ok, let's star with the session variable. Where it's that been initialize ? Well, spoiler alert, I shouldn't start with that, because my search end up in the login method of the API that set a that variable and for some reason in the front end app it was always false and that lead me to think that some of the new backend functions were failing, but after checking the logs I got no luck.

Ok, maybe the cookie it's the issue, I should try open the previous website on the brow...redirect to new project login, What? Why ? I ask around and it's a new feature push on Monday, ok I got Chrome Dev tools I can see which value of the cookie it's been set and THERE IT WAS it has a wrong domain! After 2 days ( I resume a lot of my pain ) I got what I've been looking for, so now I should be able to fix the bug. Then where is the cookie initialized ? In the first file the server hits whenever you tried to enter any page of the app, ok, I found the method, but it's using a function that process the domain and sets it correctly? wtf ? Then how in heaven do I get the incorrect domain ? Hello? Ok, relax, you still have one more day to fix this, let's take it easy.

Then, at the end of the Wednesday, nope I still have no clue how this is happening. I talked with the Devops guy and he explain me how this redirection happens and with what it depends on, I followed the PHP code through and nothing, everything should works fine, sigh. Ok I still have 2 days, because I'm not from US and I'm not in US, so I still have time, but the Sprint is messed up already, so whatever I'm gonna had done this bug anyhow.

Thursday ! I got sick, yay, what else could happen this week. Somehow I managed to work a little and star thinking in what external issue could affect the processing, maybe the redirection was bringing a wrong direction, let's talk with the Devops guy again, and he answer me that the redirection it was being made by PHP code, IN A FILE THAT DOESN'T EXIST IN THE REPOSITORY, amazing, it's just amazing. Then he explained me why this file might be missing and how it's the deployment of this app ( btw the Devops guy it's really cool and I will invite him a beer ) . After that I checked the file and I see a random session_star in the first line of the code, without any configuration, eureka ! There was the cause and I only need to ask someone If that line it's necessary anymore, but oh they're on holiday, damn, well I'll wait till Monday to ask them. But once and for all that bug was done for ! 🎉

What do I learn ? PHP and that I don't want any more tickets of PHP 😆.

When the CTO/CEO of your "startup" is always AFK and it takes weeks to get anything approved by them (or even secure a meeting with them) and they have almost-exclusive access to production and the admin account for all third party services.

Want to create a new messaging channel? Too bad! What about a new repository for that cool idea you had, or that new microservice you're expected to build. Expect to be blocked for at least a week.

When they also hold themselves solely responsible for security and operations, they've built their own proprietary framework that handles all the authentication, database models and microservice communications.

Speaking of which, there's more than six microservices per developer!

Oh there's a bug or limitation in the framework? Too bad. It's a black box that nobody else in the company can touch. Good luck with the two week lead time on getting anything changed there. Oh and there's no dedicated issue tracker. Have you heard of email?

When the systems and processes in place were designed for "consistency" and "scalability" in mind you can be certain that everything is consistently broken at scale. Each microservice offers:

1. Anemic & non-idempotent CRUD APIs (Can't believe it's not a Database Table™) because the consumer should do all the work.
2. Race Conditions, because transactions are "not portable" (but not to worry, all the code is written as if it were running single threaded on a single machine).
3. Fault Intolerance, just a single failure in a chain of layered microservice calls will leave the requested operation in a partially applied and corrupted state. Ger ready for manual intervention.
4. Completely Redundant Documentation, our web documentation is automatically generated and is always of the form //[FieldName] of the [ObjectName].
5. Happy Path Support, only the intended use cases and fields work, we added a bunch of others because YouAreGoingToNeedIt™ but it won't work when you do need it. The only record of this happy path is the code itself.

Consider this, you're been building a new microservice, you've carefully followed all the unwritten highly specific technical implementation standards enforced by the CTO/CEO (that your aware of). You've decided to write some unit tests, well um.. didn't you know? There's nothing scalable and consistent about running the system locally! That's not built-in to the framework. So just use curl to test your service whilst it is deployed or connected to the development environment. Then you can open a PR and once it has been approved it will be included in the next full deployment (at least a week later).

Most new 'services' feel like the are about one to five days of writing straightforward code followed by weeks to months of integration hell, testing and blocked dependencies.

When confronted/advised about these issues the response from the CTO/CEO
varies:

(A) "yes but it's an edge case, the cloud is highly available and reliable, our software doesn't crash frequently".
(B) "yes, that's why I'm thinking about adding [idempotency] to the framework to address that when I'm not so busy" two weeks go by...
(C) "yes, but we are still doing better than all of our competitors".
(D) "oh, but you can just [highly specific sequence of undocumented steps, that probably won't work when you try it].
(E) "yes, let's setup a meeting to go through this in more detail" *doesn't show up to the meeting*.
(F) "oh, but our customers are really happy with our level of [Documentation]".

Sometimes it can feel like a bit of a cult, as all of the project managers (and some of the developers) see the CTO/CEO as a sort of 'programming god' because they are never blocked on anything they work on, they're able to bypass all the limitations and obstacles they've placed in front of the 'ordinary' developers.

There's been several instances where the CTO/CEO will suddenly make widespread changes to the codebase (to enforce some 'standard') without having to go through the same review process as everybody else, these changes will usually break something like the automatic build process or something in the dev environment and its up to the developers to pick up the pieces. I think developers find it intimidating to identify issues in the CTO/CEO's code because it's implicitly defined due to their status as the "gold standard".

It's certainly frustrating but I hope this story serves as a bit of a foil to those who wish they had a more technical CTO/CEO in their organisation. Does anybody else have a similar experience or is this situation an absolute one of a kind?

Have u guys ever wonder, all those devs we rant about (mostly senior developer), how it feels like to be them? Today I realized, I am most probably becoming like one.
I joined devops 7 month back(around one and half year in industry). Right now, I am 2nd senior member in project. I have done deployment on multiple environments more than 100 times. But till today, I never knew how the deployment is being done. I knew to trigger job but I never knew how it worked. Today when a junior asked me, then I learn ansible, then I understand whole deployment process.(and remember I am 2nd senior most with 7 month in project)
Sometime I wonder, till now I always had good rating and most responsible title. But how much is that because of my technical knowledge? Sometime it feels like I have very good luck. But man, it's very depressing. Sometime it feels like my junior don't get enough limelight because I am in their way although they have good knowledge but they lack the though process for now. Most of the time my senior present me as role model to juniors, and it's very embarrassing for me(this will not continue on as I talked to my seniors) . I did work on good projects from time I joined company. And never had any issue and always deliver what needed. But I still can't write code in Java to take input or do for each on array in javascript without seeing stackoverflow once.
Now I fear that someday I will write piece shit of code and whole efficiency of project will go down cause of me. Atleast, the person who will get to fix it will get a chance to have good rant here. I tried open source projects to understand how to write good code but I always have hard time understanding new-projects which I never worked on.
Then there is reputation on Indian devs. This is my another Fear. That someday cause of me, my fellow devs will get bad reputation as well.
This coming year, my goal is to fill up all the holes but I don't know why my fingers are crossed.
Sorry, I had to bring this out somewhere. And please ignore my grammatical mistakes.

junior developer raises an issue saying that there's an application deployment error on one of their dev clusters.

sysadmin asks them to go back and look at the error logs and come back with the problem.

they come back saying, "No space left on device"

sysadmin takes a look at server. finds this :

Posted in DevOps discussion board (teams channel):

“Program x isn’t behaving the same way that it does on production. Can you please take a look?”

..a little background: we have a deployment scheduled for today and this issue was found during regression testing.

The issue found is that when a file is clicked on it disappears from the screen, and then isn’t opened…

The file is not on prem, and doesn’t get uploaded to a server that our DevOps team owns…

So why on earth would this development team be asking DevOps to look into a bug that is most likely a code related issue? 😆

Is this a common occurrence for anyone else?
A Bug is found, and the first thought is that the code isn’t the issue?

I work at a small company (4 devs, CTO, a senior, me: mid level, and a new junior dev). Junior and I handle the client projects and the Senior and CTO handle the overall platform and server deployments and such. Our senior dev just gave his 2 weeks notice. I was told they are not replacing him and now ALL of his tasks have been pushed onto me on top of all my already full plate. My issue is, although I am excited to learn about the upper management and deployment stuff, they (CTO and CEO) just dumped all these tasks onto me without even asking if I wanted the added responsibility and also told me there is no monetary bonus for taking it all on. Am I right in being a little mad that I was not even asked if I wanted it and it was just assumed I would handle it all without any bonus or monetary promotion?

Was working on setting up a ci/CD pipeline. The ci part with automated testing and deployment to a on-premise docker registry worked already, so I thought "hey I could try to actually run one of those fresh containers" so I tried it with the usual docker run command.. "Manifest not found" suddenly appeared, it confused me a bit since I used the same url I used for publishing... So I googled around only to find NOTHING that is even remotely connected to my issue. "Eh let's let the guy that runs that registry fix it" was what I thought and called it a day. The next day I was eager to try it again and checked the urls case by case only to notice that I wrote secret-project-backend-client instead of secret-project-api.. I tried it with the new name and it worked!
Never felt so retarded in my life....

Yesterday, I had to set up a demo environment for a project, we are working on.
Everything was okay, frontend loaded, connection to backend is working, database is connected.
10 minutes before I wanted to leave for my well deserved weekend, PO came over: "I can't play any video, I uploaded"
Okay, couldn't be a big issue, it worked when I added this functionality 3 weeks ago, just before my holidays.
A bit under pressure, my girlfriend Was already waiting downstairs, I inspected the database and realized that a table Was not properly filled.
Checked the backend and everything seems fine, so checked the requests from the frontend and realized that the request was almost empty.
So some code, building the request body had to be wrong.
Already 10 minutes late, with a lightly annoyed girlfriend waiting for me, I found the issue but couldn't recognize that I wrote these few lines. A quick check of the git history showed, that my colleage changed my code during my holidays, so I just reverted everything.
After commit and deployment, I called my colleage and told him that I just reverted his changes.
"But now my feature is not working anymore, I had to change it like this!" he answered. I just responded that we will talk about that on monday and look at it together. While I hurried down the stairs, I was thinking why the hell somebody just changes stuff without checking if it affects other functionalities?
This should be basic knowledge for every dev, that if you change existing, working code to make it work with your feature, you have to ensure to not brake anything.
If you can't do that, then create a new function to handle your shit.

In the end, my girlfriend had to wait 30 minutes, because of 4 lines of codes, someone just changed without thinking what else could happen...

I had a pretty good year! I've gone from being a totally unknown passionate web dev to a respected full stack dev. This will be a bit lengthy rant...
Best:
- Got my first full time employment dev role at a company after being self-taught for 8+ years at the start of the year. Finally got someone to take the risk of hiring someone who's "untested" and only done small and odd jobs professionally. This kickstarted my career, super grateful for that!
- Started my own programming consulting company.
- Gained enough confidence to apply to other jobs, snatched a few consulting jobs, nailed the interviews even though I never practiced any leet code.
- Currently work as a 99% remote dev (only meet up in person during the initialization of some projects.) I never thought working remotely could actually work this well. I am able to stay productive and actually focus on the work instead of living up to the 9-5 standard. If I want to go for a walk to think I can do that, I can be as social and asocial as I want. I like to sleep in and work during the night with a cup of tea in the dark and it's not an issue! I really like the freedom and I feel like I've never been more productive.
- Ended up with very happy customers and now got a steady amount of jobs rolling in and contracts are being extended.
- I learned a lot, specialized in graph databases, no more db modelling hell. Loving it!
- Got a job where I can use my favorite tools and actually create something from scratch which includes a lot of different fields. I am really happy I can use all my skills and learn new things along the way, like data analysis, databricks, hadoop, data ingesting, centralised auth like promerium and centralised logging.
- I also learned how important softskills are, I've learned to understand my clients needs and how to both communicate both as a developer and an entrepeneur.
Worst:
- First job had a manager which just gave me the specifications solo project and didn't check in or meet me for 8 weeks with vague specifications. Turns out the manager was super biased on how to write code and wanted to micromanage every aspect while still being totally absent. They got mad that I had used AJAX for requests as that was a "waste of time".
- I learned the harsh reality of working as a contractor in the US from a foreign country. Worked on an "indefinite" contract, suddenly got a 2 day notification to sum up my work (not related to my performance) after being there for 7+ months.
- I really don't like the current industry standard when it comes to developing websites (I mostly work in node.js), I like working with static websites (with static website generators like what the Svelte.js driver) and use a REST API for dynamic content. When working on the backend there's a library for everything and I've wasted so many hours this year to fix bugs and create workarounds related to dependencies. You need to dive into a rabbit hole for every tool and do something which may work or break something later. I've had so many issues with CICD and deployment to the cloud. There's a library for everything but there's so many that it's impossible to learn about the edge cases of everything. Doesn't help that everything is abstracted away, which works 90% of the time but I use 15 times the time to debug things when a bug appears. I work against a black box which may or may not have an up to date documentation and it's so complex that it will require you to yell incantations from the F#$K
era and sacrifice a goat for it to work properly.
- Learned that a lot of companies call their complex services "microservices". Ah yes, the microservice with 20 endpoints which all do completely unrelated tasks?

To add a bit more context to my last rant.

The following situation happened today and similar situations are at the moment common as fuck.

Situation started roughly 1 1/2 months ago as a deployment failed.

Seemed to be a DNS problem for the devs, so my basic assumption was that they checked their shit.

As I was and I am currently more than swamped, told them it had to wait if it is an DNS issue...

Well.

Backstabbing product manager complained to upper management as it took so long.

Backstabbing manager even went so far to propose alternative solutions - think of switching product to work around issue and throwing away a year of development of a 5 man team...

So additional to my work I had to deescalate and prevent complete nonsense.

Today I finally found time for the problem.

After 2-3 hours of turning every stone inside the DNS setup, cloudflare, loadbalancers, etc...

Well. Devs. Don't trust them.

Turned out the devs misconfigured the environment entirely.

Its not so obvious in this product as it is rather complicated, though the devs documentation explicitly mentioned that if one overrides the configuration for e.g. several languages, one has to make sure to set two env variables for TLS mode...

There was only one set.

:(

8 fucking weeks of backstabbing and blaming others while they could have just read their own fucking documentation and fixed that shit in 5 minutes.

How do you do your CI/CD pipeline? Sorry if this is a dumb question. Just wondering how the tests and deployment usually runs. Is it on a per team basis? Is it the whole release getting deployed to Test many times per day? What happens if too many automated tests fail or there is not enough coverage, does it abort the deployment? If so, how can every team get delayed by every issue - is that actually a good policy?

My pipeline is very slow and requires a team of 12 people working in shifts to complete it. I’m not an expert but I know it does a lot of steps and never completes without manual intervention. I would like to help but I’m not sure how bad it is.

Hello, my first time here. I got to know this website/app from my PM because I need to vent it somewhere other than him according to my PM.

So, here goes my first rant. The date is today (Monday). The rant subject is our new tester. Some context on the guy. He started in our office 8 weeks ago and his title is senior tester with some years in testing. Me and my team with the exception of our PM are new hires and for me, this is my first job after graduation.

After a grueling month of pushing for new modules and bug fixes from our monthly UAT from the client (yes, this will be a future rant one day), about 2/3 of the team is on vacation paired with a long weekend. So, a very few ppl in the team including me and my PM came for today.

I usually came quite early, around 8 am as I commute with public transportation. As soon as I have my breakfast and just getting ready to open my dev laptop, he came to me with a bug. This is like under an hour I came to office. I'm ok with anything related to the project as today was deployment day to test server for our monthly UAT. So, I check the bug and it wasn't my module but the PIC is not there and I familiar with the code thus I fixing the module.

Then, not even 15 mins later, while fixing this module, he came to me with another bug. I'm still the only one who in office that can fix it thus have to do it too. Finished the both bugs, pushed and je retested it. Fortunately, my PM and another colleague came. But, for some reason, he only comes to me for the bug fixes.

The annoying thing for me is that he comes to me every time he found an obstacle, bug or glitch. At this rate, by hourly. Thus, this cycle of impromptu going around fixing-on-the-go for the project begins, for me. Then, my PM asks him abt our past issue log given by the client UAT. Another annoying part is he never checks the clients feedback to see if the result can be produced again. The time he checks it is when ppl ask abt it and test it 1 by 1. Then he came to me again with why x person marked it as done. Like hell I know why they marked it done, you the one who need to check with them. Thus, I called/messaged the PIC for x modules abt the issue and then they explain it. I have to explain it again to him abt it and then he makes the summary report for the feedback. This goes until lunch.

I thought the bug fixes is over and I can deploy it after lunch. I thought wrong and I kinda regret coming back early from lunch which I thought I can rest for a while with the debacle over morning. Nope, straight he comes to me after I sit down for 10 mins and until almost work hour is done, he came to me with small bugs and issues like previously, hourly. By then I think I crushed like ~10 bugs/issues and I'm knackered. I complained to the PM many times and the PM also said to him many times but he still does it again and again. Even the PM also ranted to me abt his behavior. The attitude of not compiling an issue log for the day and not testing the system to verify what the client feedbacks are valid or not is grinding my gears more and more. Not hating the guy even though his personality is quite unique but this is totally grinding ppl's gears atm. As of now, it's midnight and I finally deployed the system to the testing server. This totally drains my mental health and it's just Monday. May god have mercy on me.

Owh, the other colleague that come today? He was doing pretty much the same thing but he was resolving a major issue which is why the tester came to me.

What's the most insane deployment scheme you've had to work with? One client has a release schedule that deploys all major projects once a month(!). Bugfixes get deployed once a day (systemwide), so any issue that can't be verified until it's in production has at least a days delay when iterating.

Do you guys have people in your office that just REFUSE to cooperate, or people who tell you they'll cooperate, but then they literally do anything except for cooperate?

I'm having trouble with the latter; I've been trying to get one of our less experienced members to work on our deployment. He's successfully configured at least 4 other deployments, and this one is the EXACT SAME as the other ones. The issue is that the person who is im control of this particular master console is someone higher up than me, but they don't know how to delegate. Thus everything that they touch becomes their own little pet project that no one else can dare touch, because they'll "mess it up" (not do it the right way according to his limited bible of best practices).

So now I'm stuck here, trying to convince HIS BOSS to get him access, but i even HE cant get him to do it! Now I'm sitting here waiting, getting more and more fed up with this guy, because like i said, it's his MO: im on two other projects with him, and they're all moving at a GLACIER'S pace.

Seriously, if you dont have the time for a project, but it on the backburner, dont start it and make your other projects suffer.

Interruptions from production team because of a tiny issue that’s already been fixed waiting for deployment

Received 'Thank you' e-mail after Test Deployment and UAT !!

Usually i receive that e-mail after production and some issue findings!

Fuck heroku

Keep refusing to verify my credit card information.

Weird field on their form , that nobody really know what they are asking for.

Billing address line 1
Billing address line 2
State/province (I have provided my country before, so I don't know what to insert here)

Please help