Oh sh*t

Today I broke a record 🎉

..in the production database 😅

Oh, man, I just realized I haven't ranted one of my best stories on here!

So, here goes!

A few years back the company I work for was contacted by an older client regarding a new project.

The guy was now pitching to build the website for the Parliament of another country (not gonna name it, NDAs and stuff), and was planning on outsourcing the development, as he had no team and he was only aiming on taking care of the client service/project management side of the project.

Out of principle (and also to preserve our mental integrity), we have purposely avoided working with government bodies of any kind, in any country, but he was a friend of our CEO and pleaded until we singed on board.

Now, the project itself was way bigger than we expected, as the wanted more of an internal CRM, centralized document archive, event management, internal planning, multiple interfaced, role based access restricted monster of an administration interface, complete with regular user website, also packed with all kind of features, dashboards and so on.

Long story short, a lot bigger than what we were expecting based on the initial brief.

The development period was hell. New features were coming in on a weekly basis. Already implemented functionality was constantly being changed or redefined. No requests we ever made about clarifications and/or materials or information were ever answered on time.

They also somehow bullied the guy that brought us the project into also including the data migration from the old website into the new one we were building and we somehow ended up having to extract meaningful, formatted, sanitized content parsing static HTML files and connecting them to download-able files (almost every page in the old website had files available to download) we needed to also include in a sane way.

Now, don't think the files were simple URL paths we can trace to a folder/file path, oh no!!! The links were some form of hash combination that had to be exploded and tested against some king of database relationship tables that only had hashed indexes relating to other tables, that also only had hashed indexes relating to some other tables that kept a database of the website pages HTML file naming. So what we had to do is identify the files based on a combination of hashed indexes and re-hashed HTML file names that in the end would give us a filename for a real file that we had to then search for inside a list of over 20 folders not related to one another.

So we did this. Created a script that processed the hell out of over 10000 HTML files, database entries and files and re-indexed and re-named all this shit into a meaningful database of sane data and well organized files.

So, with this we were nearing the finish line for the project, which by now exceeded the estimated time by over to times.

We test everything, retest it all again for good measure, pack everything up for deployment, simulate on a staging environment, give the final client access to the staging version, get them to accept that all requirements are met, finish writing the documentation for the codebase, write detailed deployment procedure, include some automation and testing tools also for good measure, recommend production setup, hardware specs, software versions, server side optimization like caching, load balancing and all that we could think would ever be useful, all with more documentation and instructions.

As the project was built on PHP/MySQL (as requested), we recommended a Linux environment for production. Oh, I forgot to tell you that over the development period they kept asking us to also include steps for Windows procedures along with our regular documentation. Was a bit strange, but we added it in there just so we can finish and close the damn project.

So, we send them all the above and go get drunk as fuck in celebration of getting rid of them once and for all...

Next day: hung over, I get to the office, open my laptop and see on new email. I only had the one new mail, so I open it to see what it's about.

Lo and behold! The fuckers over in the other country that called themselves "IT guys", and were the ones making all the changes and additions to our requirements, were not capable enough to follow step by step instructions in order to deploy the project on their servers!!!

[Continues in the comments]

Finally did it. Quit my job.

The full story:
Just came back from vacation to find out that pretty much all the work I put at place has been either destroyed by "temporary fixes" or wiped clean in favour of buggy older versions. The reason, and this is a direct quote "Ari left the code riddled with bugs prior to leaving".

Oh no. Oh no I did not you fucker.

Some background:

My boss wrote a piece of major software with another coder (over the course of month and a balf). This software was very fragile as its intention was to demo specific features we want to adopt for a version 2 of it.

I was then handed over this software (which was vanilajs with angular) and was told to "clean it up" introduce a typing system, introduce a build system, add webpack for better module and dependency management, learn cordova (because its essential and I had no idea of how it works). As well as fix the billion of issues with data storage in the software. Add a webgui and setup multiple databses for data exports from the app. Ensure that transmission of the data is clean and valid.

What else. This software had ZERO documentation. And I had to sit my boss for a solid 3hrs plus some occasional questions as I was developing to get a clear idea of whats going on.

Took a bit over 3 weeks. But I had the damn thing ported over. Cleaned up. And partially documented.

During this period, I was suppose to work with another 2 other coders "my team". But they were always pulled into other things by my Boss.

During this period, I kept asking for code reviews (as I was handling a very large code base on my own).

During this period, I was asking for help from my boss to make sure that the visual aspect of the software meets the requirements (there are LOTS of windows, screens, panels etc, which I just could not possibly get to checking on my own).

At the end of this period. I went on vacation (booked by my brothers for my bday <3 ).

I come back. My work is null. The Boss only looked at it on the friday night leading up to my return. And decided to go back to v1 and fix whatever he didnt like there.

So this guy calls me. Calls me on a friggin SUNDAY. I like just got off the plane. Was heading to dinner with my family.
He and another coder have basically nuked my work. And in an extremely hacky way tied some things together to sort of work. Moreever, the webguis that I setup for the database viewing. They were EDITED ON THE PRODUCTION SERVER without git tracking!!

So monday. I get bombarded with over 20 emails. Claiming that I left things in an usuable state with no documentation. As well as I get yelled at by my boss for introducing "unnecessary complicated shit".

For fuck sakes. I was the one to bring the word documentation into the vocabulary of this company. There are literally ZERO documentated projects here. While all of mine are at least partially documented (due to lack of time).

For fuck sakes, during my time here I have been basically begging to pull the coder who made the admin views for our software and clean up some of the views so that no one will ever have to touch any database directly.

To say this story is the only reason I am done is so not true.

I dedicated over a year to this company. During this time I saw aspects of this behaviour attacking other coders as well as me. But never to this level.

I am so friggin happy that I quit. Never gonna look back.

Achievement unlocked
Killed production database for 40 minutes

The project where I realized I wanted to go from chemist to pro dev.

I built a flow-chemistry spectrometer with monitoring backend in Haskell.

Spectroscopy is where you add a reagent to a glass tube, it changes color, and by measuring the exact color it tells you how much of something (for example, a toxin) is present in the sample.

I had to do that a lot on factory samples, writing down measurements using pen & paper.

I'm lazy so I decided to do the logical thing: Automate it. I bought a second hand spectrometer, stripped the casing, did a shitload of glassblowing and hooked up tubes to the production pipelines, so I could get samples, mixing them in the correct ratio with reagents in continuous flows using valves.

I ended up using 2 home-crafted arduino-like boards (etching PCBs is fun!).

One to calibrate the mixture against known samples and control solenoid valves to continuously cycle through various reagents and deionized flushing water, the other to record the measurements and send them to a server running a Haskell/Yesod API.

The server collected the information into InfluxDB (A time series database), displaying all data on a graphite dashboard.

Eventually I wrote Haskell plugins for most of the chemistry processes, from pH & temperature measurements to polymer property and pigment tests (they made a lot of printer ink).

Then I was fired because they didn't need chemists anymore, and the code "could be maintained by the intern" (poor guy)...

But I did find out that I loved functional programming, chemistry automation projects, and crafting my own electronics during that time.

A "support" guy my boss got in. I had told my boss numerous times, "Get rid of this guy, he's only wasting our time and money. And he's going to end up doing something where we will end up having to put out the fires."

Sure as a pair of nuts on a squirrel, this crazy bastard goes and DELETES a client's database. Yes folks, in fucking production. A live database. The heart of the business' transactions are... *poof*... GONE!!!

No backups for the day! No synchronisation beforehand! No nothing... just GONE!!! Fucking flat-lining!

Well, when I realised what he has done, I had to remove myself from the room before shit got outta hand!

I told the boss man that is the last straw and he needs to go...

The long and short of it...
- The client had luckily only lost about half a days data.
- I'm no longer at the company.
- This dumb fuck still is.

I worked on a greenfield project a couple of years ago. The company had an old solution written in Omnis (heard of it? Yeah, me neither) with an SQL database. My team was to create a completely new web based system... on top of the old database, so the customers could keep their existing stuff.

The dba was an intelligent man, one of the nicest people I've met, and over the course of fifteen years he had made a remarkably terrifying monstrosity of a database. Some years before me they wanted to "future proof" the system and make it "easier to switch to new technologies". So they moved the entire business logic into the database...

I used a tool to create a visualization of said database when we started. It had no views, only tables and sprocs. Look at it! Tables and sprocs are rectangles (well, dots) and any connections are drawn in grey lines. There were no foreign keys, so a tables only visualization only yielded a collection of independent rectangles without a single line.

Now, the stored procedures were bloody MASSIVE. A single procedure that only registered a new interested party and attached them to a property had 2500+ lines and over 150 parameters.

Also, this dba added features and fixed bugs by logging into the respective customers production server and writing SQL.

That database is the stupidest thing I've ever seen a developer do.

My biggest dev blunder. I haven't told a single soul about this, until now.

👻👻👻👻👻👻

So, I was working as a full stack dev at a small consulting company. By this time I had about 3 years of experience and started to get pretty comfortable with my tools and the systems I worked with.

I was the person in charge of a system dealing with interactions between people in different roles. Some of this data could be sensitive in nature and users had a legal right to have data permanently removed from our system. In this case it meant remoting into the production database server and manually issuing DELETE statements against the db. Ugh.

As soon as my brain finishes processing the request to venture into that binary minefield and perform rocket surgery on that cursed database my sympathetic nervous system goes into high alert, palms sweaty. Mom's spaghetti.

Alright. Let's do this the safe way. I write the statements needed and do a test run on my machine. Works like a charm 😎

Time to get this over with. I remote into the server. I paste the code into Microsoft SQL Server Management Studio. I read through the code again and again and again. It's solid. I hit run.

....

Wait. I ran it?

....

With the IDs from my local run?

...

I stare at the confirmation message: "Nice job dude, you just deleted some stuff. Cool. See ya. - Your old pal SQL Server".

What did I just delete? What ramifications will this have? Am I sweating? My life is over. Fuck! Think, think, think.

You're a professional. Handle it like one, goddammit.

I think about doing a rollback but the server dudes are even more incompetent than me and we'd lose all the transactions that occurred after my little slip. No, that won't fly.

I do the only sensible thing: I run the statements again with the correct IDs, disconnect my remote session, and BOTTLE THAT SHIT UP FOREVER.

I tell no one. The next few days I await some kind of bug report or maybe a SWAT team. Days pass. Nothing. My anxiety slowly dissipates. That fateful day fades into oblivion and I feel confident my secret will die with me. Cool ¯\_(ツ)_/¯

To replace humans with robots, because human beings are complete shit at everything they do.

I am a chemist. My alignment is not lawful good. I've produced lots of drugs. Mostly just drugs against illnesses. Mostly.

But whatever my alignment or contribution to the world as a chemist... Human chemists are just fucking terrible at their job. Not for a lack of trying, biological beings just suck at it.

Suiting up for a biosafety level lab costs time. Meatbags fuck up very often, especially when tired. Humans whine when they get acid in their face, or when they have to pour and inhale carcinogenic substances. They also work imprecisely and inaccurately, even after thousands of hours of training and practice.

Weaklings! Robots are superior!

So I replaced my coworkers with expensive flow chemistry setups with probes and solenoid fluid valves. I replaced others with CUDA simulations.

First at a pharma production & research lab, then at a genetics lab, then at an Industrial R&D lab.

Many were even replaced by Raspberry Pi's with two servos and a PH meter attached, and I broke open second hand Fischer Sci spectrophotometers to attach arduinos with WiFi boards.

The issue was that after every little overzealous weekend project, I made myself less necessary as well.

So I jumped into the infinitely deep shitpool called webdev.

App & web development is kind of comfortable, there's always one more thing to do, but there's no pressure where failure leads to fatalities (I think? Wait... do I still care?).

Super chill, if it weren't for the delusion that making people do "frontend" and "fullstack" labor isn't a gross violation of the Geneva Convention.

Quickly recognizing that I actually don't want to be tortured and suffer from nerve damage caused by VueX or have my organs slowly liquefied by the radiation from some insane transpiling centrifuge, I did what any sane person would do.

Get as far away from the potential frontend blast radius as possible, hide in a concrete bunker.

So I became a data engineer / database admin.

That's where I'm quarantining now, safely hiding from humanity behind a desk, employed to write a MySQL migration or two, setting up Redis sorted sets, adding a field to an Elastic index. That takes care of generating cognac and LSD money.

But honestly.... I actually spend most of my time these days contributing to open source repositories, especially writing & maintaining Rust libraries.

Hey, Root? How do you test your slow query ticket, again? I didn't bother reading the giant green "Testing notes:" box on the ticket. Yeah, could you explain it while I don't bother to listen and talk over you? Thanks.

And later:
Hey Root. I'm the DBA. Could you explain exactly what you're doing in this ticket, because i can't understand it. What are these new columns? Where is the new query? What are you doing? And why? Oh, the ticket? Yeah, I didn't bother to read it. There was too much text filled with things like implementation details, query optimization findings, overall benchmarking results, the purpose of the new columns, and i just couldn't care enough to read any of that. Yeah, I also don't know how to find the query it's running now. Yep, have complete access to the console and DB and query log. Still can't figure it out.

And later:

Hey Root. We pulled your urgent fix ticket from the release. You know, the one that SysOps and Data and even execs have been demanding? The one you finished three months ago? Yep, the problem is still taking down production every week or so, but we just can't verify that your fix is good enough. Even though the changes are pretty minimal, you've said it's 8x faster, and provided benchmark findings, we just ... don't know how to get the query it's running out of the code. or how check the query logs to find it. So. we just don't know if it's good enough.

Also, we goofed up when deploying and the testing database is gone, so now we can't test it since there are no records. Nevermind that you provided snippets to remedy exactly scenario in the ticket description you wrote three months ago.

And later:

Hey Root: Why did you take so long on this ticket? It has sat for so long now that someone else filed a ticket for it, with investigation findings. You know it's bringing down production, and it's kind of urgent. Maybe you should have prioritized it more, or written up better notes. You really need to communicate better. This is why we can't trust you to get things out.

*twitchy smile*

5 beers in
Migrating a 200 GB database in production
What could go wrong?

I recently joined the dark side - an agile consulting company (why and how is a long story). The first client I was assigned to was an international bank. The client wanted a web portal, that was at its core, just a massive web form for their users to perform data entry.

My company pitched and won the project even though they didn't have a single developer on their bench. The entire project team (including myself) was fast tracked through interviews and hired very rapidly so that they could staff the project (a fact I found out months later).

Although I had ~8 years of systems programming experience, my entire web development experience amounted to 12 weeks (a part time web dev course) just before I got hired.

I introduce to you, my team ...

Scrum Master. 12 years experience on paper.
Rote memorised the agile manifesto and scrum textbooks. He constantly went “We should do X instead of (practical thing) Y, because X is the agile way.” Easily pressured by the client to include ridiculous (real time chat in a form filling webpage), and sometimes near impossible features (undo at the keystroke level). He would just nag at the devs until someone mumbled ‘yes' just so that he would stfu and go away.

UX Designer. 3 years experience on paper ... as business analyst.
Zero professional experience in UX. Can’t use design tools like AI / photoshop. All he has is 10 weeks of UX bootcamp and a massive chip on his shoulder. The client wanted a web form, he designed a monstrosity that included several custom components that just HAD to be put in, because UX. When we asked for clarification the reply was a usually condescending “you guys don’t understand UX, just do <insert unhandled edge case>, this is intended."

Developer - PHD in his first job.
Invents programming puzzles to solve where there are none. The user story asked for a upload file button. He implemented a queue system that made use of custom metadata to detect file extensions, file size, and other attributes, so that he could determine which file to synchronously upload first.

Developer - Bootlicker. 5 years experience on paper.
He tried to ingratiate himself with the management from day 1. He also writes code I would fire interns and fail students for. His very first PR corrupted the database. The most recent one didn’t even compile.

Developer - Millennial fratboy with a business degree. 8 years experience on paper.
His entire knowledge of programming amounted to a single data structures class he took on Coursera. Claims that’s all he needs. His PRs was a single 4000+ line files, of which 3500+ failed the linter, had numerous bugs / console warnings / compile warnings, and implemented 60% of functionality requested in the user story. Also forget about getting his attention whenever one of the pretty secretaries walked by. He would leap out of his seat and waltz off to flirt.

Developer - Brooding loner. 6 years experience on paper.
His code works. It runs, in exponential time. Simply ignores you when you attempt to ask.

Developer - Agile fullstack developer extraordinaire. 8 years experience on paper.
Insists on doing the absolute minimum required in the user story, because more would be a waste. Does not believe in thinking ahead for edge conditions because it isn’t in the story. Every single PR is a hack around existing code. Sometimes he hacks a hack that was initially hacked by him. No one understands the components he maintains.

Developer - Team lead. 10 years of programming experience on paper.
Writes spaghetti code with if/else blocks nested 6 levels deep. When asked "how does this work ?”, the answer “I don’t know the details, but hey it works!”. Assigned as the team lead as he had the most experience on paper. Tries organise technical discussions during which he speaks absolute gibberish that either make no sense, or are complete misunderstandings of how our system actually works.

The last 2 guys are actually highly regarded by my company and are several pay grades above me. The rest were hired because my company was desperate to staff the project.

There are a 3 more guys I didn’t mention. The 4 of us literally carried the project. The codebase is ugly as hell because the others merge in each others crap. We have no unit tests, and It’s near impossible to start because of the quality of the code. But this junk works, and was deployed to production. Today is it actually hailed as a success story.

All these 3 guys have quit. 2 of them quit without a job. 1 found a new and better gig.

I’m still here because I need the money. There’s a tsunami of trash code waiting to fail in production, and I’m the only one left holding the fort.

Why am I surrounded by morons?
Why are these retards paid more than me?
Why are they so proud when all they produce is trash?
How on earth are they still hired?

And yeah, FML.

If drinking coffee doesn't wake you up in the morning, try deleting a table in the production database.

Verified, it worked.

FML. An overreaching supergenius "architect" and a database team:

A: "We have decided that apps should use mysql. Install a MySQL so we match cloud"
DBA: "we don't have an image or experience with MySQL. We have mssql and Oracle "
A: "ok, use mssql in data center and mysql in production cloud"
DBA: "that's... not going to work well"
A: "just do it!"

...

Me, reading this shit, sends email: "ignoring the fact that we have more than 500 queries in this application which will need to be checked and most likely rewritten, how are we supposed to test the mysql queries without production access?"
A: "just use mssql local and MySQL in cloud"
M: "... Just to make sure I understand, you want us to write queries for mssql, test them locally, and then write separate queries, with a separate SQL connection abstraction that deploys to production? Again, how are we going to test this?"
A: "no, use same queries, should be fine"
M: "they really won't, they're different dialects"
A: "do the needful, make work!"

If karma were a thing, this person would have long since exploded into a cloud of atomized blood.

Worst dev team failure I've experienced?

One of several.

Around 2012, a team of devs were tasked to convert a ASPX service to WCF that had one responsibility, returning product data (description, price, availability, etc...simple stuff)
No complex searching, just pass the ID, you get the response.

I was the original developer of the ASPX service, which API was an XML request and returned an XML response. The 'powers-that-be' decided anything XML was evil and had to be purged from the planet. If this thought bubble popped up over your head "Wait a sec...doesn't WCF transmit everything via SOAP, which is XML?", yes, but in their minds SOAP wasn't XML. That's not the worst WTF of this story.

The team, 3 developers, 2 DBAs, network administrators, several web developers, worked on the conversion for about 9 months using the Waterfall method (3~5 months was mostly in meetings and very basic prototyping) and using a test-first approach (their own flavor of TDD). The 'go live' day was to occur at 3:00AM and mandatory that nearly the entire department be on-sight (including the department VP) and available to help troubleshoot any system issues.

3:00AM - Teams start their deployments
3:05AM - Thousands and thousands of errors from all kinds of sources (web exceptions, database exceptions, server exceptions, etc), site goes down, teams roll everything back.

3:30AM - The primary developer remembered he made a last minute change to a stored procedure parameter that hadn't been pushed to production, which caused a side-affect across several layers of their stack.

4:00AM - The developer found his bug, but the manager decided it would be better if everyone went home and get a fresh look at the problem at 8:00AM (yes, he expected everyone to be back in the office at 8:00AM).

About a month later, the team scheduled another 3:00AM deployment (VP was present again), confident that introducing mocking into their testing pipeline would fix any database related errors.

3:00AM - Team starts their deployments.
3:30AM - No major errors, things seem to be going well. High fives, cheers..manager tells everyone to head home.
3:35AM - Site crashes, like white page, no response from the servers kind of crash. Resetting IIS on the servers works, but only for around 10 minutes or so.
4:00AM - Team rolls back, manager is clearly pissed at this point, "Nobody is going fucking home until we figure this out!!"
6:00AM - Diagnostics found the WCF client was causing the server to run out of resources, with a mix of clogging up server bandwidth, and a sprinkle of N+1 scaling problem. Manager lets everyone go home, but be back in the office at 8:00AM to develop a plan so this *never* happens again.

About 2 months later, a 'real' development+integration environment (previously, any+all integration tests were on the developer's machine) and the team scheduled a 6:00AM deployment, but at a much, much smaller scale with just the 3 development team members.

Why? Because the manager 'froze' changes to the ASPX service, the web team still needed various enhancements, so they bypassed the service (not using the ASPX service at all) and wrote their own SQL scripts that hit the database directly and utilized AppFabric/Velocity caching to allow the site to scale. There were only a couple client application using the ASPX service that needed to be converted, so deploying at 6:00AM gave everyone a couple of hours before users got into the office. Service deployed, worked like a champ.

A week later the VP schedules a celebration for the successful migration to WCF. Pizza, cake, the works. The 3 team members received awards (and a envelope, which probably equaled some $$$) and the entire team received a custom Benchmade pocket knife to remember this project's success. Myself and several others just stared at each other, not knowing what to say.

Later, my manager pulls several of us into a conference room
Me: "What the hell? This is one of the biggest failures I've been apart of. We got rewarded for thousands and thousands of dollars of wasted time."
<others expressed the same and expletive sediments>
Mgr: "I know..I know...but that's the story we have to stick with. If the company realizes what a fucking mess this is, we could all be fired."
Me: "What?!! All of us?!"
Mgr: "Well, shit rolls downhill. Dept-Mgr-John is ready to fire anyone he felt could make him look bad, which is why I pulled you guys in here. The other sheep out there will go along with anything he says and more than happy to throw you under the bus. Keep your head down until this blows over. Say nothing."

Never in my life I was scared as today.
I recently left a big company to work for a small one as the first internal developer.
Had a small issue in the production server. The fix was easy, just remove a single table entry. And... *drum roll*... I forgot to add a where clause. All orders were lost.
No idea if we had backups or anything, I quickly called the one other IT dude in the company.
He had no clue where are the backups and how to find them.
Having some experience with Nmap, I quickly scanned our network and found a Nas device.
There was a backup, whole VHD backup. 300GB of it, the download speed is around 512kb/s. No way I can fix it before management finds out, but then an idea came to mind. Old glorious 7zip. Managed to extract only the database files, sent them to the server and quickly swapped them. Everything was fine... The manager connected 5 minutes later. Scariest 45 minutes of my life...

My team handles infrastructure deployment and automation in the cloud for our company, so we don't exactly develop applications ourselves, but we're responsible for building deployment pipelines, provisioning cloud resources, automating their deployments, etc.

I've ranted about this before, but it fits the weekly rant so I'll do it again.

Someone deployed an autoscaling application into our production AWS account, but they set the maximum instance count to 300. The account limit was less than that. So, of course, their application gets stuck and starts scaling out infinitely. Two hundred new servers spun up in an hour before hitting the limit and then throwing errors all over the place. They send me a ticket and I login to AWS to investigate. Not only have they broken their own application, but they've also made it impossible to deploy anything else into prod. Every other autoscaling group is now unable to scale out at all. We had to submit an emergency limit increase request to AWS, spent thousands of dollars on those stupidly-large instances, and yelled at the dev team responsible. Two weeks later, THEY INCREASED THE MAX COUNT TO 500 AND IT HAPPENED AGAIN!

And the whole thing happened because a database filled up the hard drive, so it would spin up a new server, whose hard drive would be full already and thus spin up a new server, and so on into infinity.

Thats probably the only WTF moment that resulted in me actually saying "WTF?!" out loud to the person responsible, but I've had others. One dev team had their code logging to a location they couldn't access, so we got daily requests for two weeks to download and email log files to them. Another dev team refused to believe their server was crashing due to their bad code even after we showed them the logs that demonstrated their application had a massive memory leak. Another team arbitrarily decided that they were going to deploy their code at 4 AM on a Saturday and they wanted a member of my team to be available in case something went wrong. We aren't 24/7 support. We aren't even weekend support. Or any support, technically. Another team told us we had one day to do three weeks' worth of work to deploy their application because they had set a hard deadline and then didn't tell us about it until the day before. We gave them a flat "No" for that request.

I could probably keep going, but you get the gist of it.

It's 2018 and I am forced to add new features on project that was deployed in 2003 and lastly updated at 2009.

Best part is that PHP version on the production server is [drum roll] 4.3.8. I found out that this version was the latest at July 2003. On top of that, server runs on Windows Server 2003 and the database is oracle (meaning php driver is needed).

I have spent a WHOLE WEEK just trying to recreate that environment in order to start working on the new features.

Sorry for grumping but I had to take it out somewhere.

Thursday: Hey wife…. I finished my project and tomorrow I should have a very easy day, just watching Slack.

Friday: Database— corrupted bin logs, major production outages.

Wife: 😡I can never believe you when you say you’ll have an easy day.

Just last month we finished a big project for a client. The company had their lawyers make a contract to transfer ownership for the website to them.

One of the requirements was to deliver all code in a word file. And just that. In the whole contract was not a word about transfering the production environment to their servers. Or supplying a working version. The files and database. No just the word file.

As of today still wondering what they where planning to do with a word file with hunderds of pages of code.

Offcourse we deliverd a working version to their servers. But why are there people making decisions about things they understand nothing at all.

This is a funny one:

I found this gem in SO.
Why is it funny? Well, PDVSA is the state-owned company for oil production of my country (Petróleos De Venezuela).

This little intern decided that it was a good idea to publish in SO an answer with actual code, showing database and tables names to everyone.

Priceless.

One:

Had a stack of harddrives with my important data, two USB drives and a 4.7gb disc, two or three cloud storage accounts.

Needed a restore:
Knocked the stack of hard drives onto the floor (all broken), stood on one of the flash drives, found the other one in a pocket of a pair of trousers which just came out of the washing machine, dvd too scratched to read and couldn't verify my cloud storage account because I lost the password to the connected email account and the backup email account to verify that one didn't exist anymore. Fucking hell.

Two:

Production database with not that much yet but at least some production data which wasn't backupped.
Friend: can I reboot the db machine?
Me: yup!
Friend: what's the luks crypt password?
Me: 😯😐😓😫😲😧😭

End of story 😅

For the record, the first one actually happened (I literally cried afterwards) and that taught me to update my recovery email addresses more often!

This is going to be a long rant, coz this is the only way to vent out my frustration against our tech head.

Yesterday, while our fucking twat tech head was playing around in company aws account, he terminated the production server. By mistake, apparently. Coz he doesn't know shit about server management. But that egoist ass won't admit and fucked the production server.

And then ran away. We developers sprang into action. Updated dns to point to staging server, setup virtual hosts, env files, point to prod database, force flush dns cache. All systems were up and running in 30 mins. And since it was staging server, it had lot of untested features and codes, and we spent rest of the day fixing the bugs.

And that tech head, who ran away hiding his tail between his legs, after he fucked the server, came back after systems were up. And started cracking jokes, that "so many features got released in 1 day" . "We cut server cost by shutting down 1 server."

We were struggling and working in full throttle to make the services running again. And that fuckity fucker was cracking jokes.

And I don't even know what excuse he gave to ceo for the downtime. I am pretty sure he would have made up some crappy excuse to hide his fucking mistake. That ass never admits his mistake. I am thinking to go to ceo today and tell the real story and get that faggot head fired or at least a strict warning.

Dev: Hey, I need you to see something.
Client: ok
Dev: this is the models of those reports you need
Client: ok, wait, what is that number? This is wrong, we can't...
Dev: hey, calm down, this is not the production, it's just fake database!
Client: ah, ok.
(5 minutes of explantion)
Dev: so, what do you think?
Client: just one point, that data is very very wrong, we need to change this ASAP!

Client: "I did not receive the email that should be send after that event. Please fix."
Me:
* Checks code - ok
* Tests feature in locally - ok
* Tests feature in production - ok
* checks values in database - ok
* 2 hours wasted - ok
"Please help me dear CTO, idk what else I could check or how I should even respond to this."
CTO: "hmm, the clients account uses a adminstrative email address for testing. Let me just check if it is in the mailbox."
*checks* "Yeah, that's the email you're looking for, right?"
Me: *experiences relief, anger, blood lust and disappointment at the same time* "Could you please respond to the client for me, I need a break. Thanks"

We've built a web app and now a client wants a VPN acces to the database of web app. When asked why, they said they want to occasionally pull some data out. 😱

We said no, and this is what they wrote:
"We’ve got live VPN access to every other web database we work with – why is this different?"

Well because maybe we know that we can build you an export of whatever you want, prepare you API calls for getting data to your CRM, but hell I'm not giving you access to the production DB.

Most satisfying bug I've fixed?

Fixed a n+1 issue with a web service retrieving price information. I initially wrote the service, but it was taken over by a couple of 'world class' monday-morning-quarterbacks.
The "Worst code I've ever seen" ... "I can't believe this crap compiles" types that never met anyone else's code that was any good.

After a few months (yes months) and heavy refactoring, the service still returned price information for a product. Pass the service a list of product numbers, service returns the price, availability, etc, that was it.

After a very proud and boisterous deployment, over the next couple of days the service seemed to get slower and slower. DBAs started to complain that the service was causing unusually high wait times, locks, and CPU spikes causing problems for other applications. The usual finger pointing began which ended up with "If PaperTrail had written the service 'correctly' the first time, we wouldn't be in this mess."

Only mattered that I initially wrote the service and no one seemed to care about the two geniuses that took months changing the code.

The dev manager was able to justify a complete re-write of the service using 'proper development methodologies' including budgeting devs, DBAs, server resources, etc..etc. with a projected year+ completion date.

My 'BS Meter' goes off, so I open up the code, maybe 5 minutes...tada...found it. The corresponding stored procedure accepts a list of product numbers and a price type (1=Retail, 2=Dealer, and so on). If you pass 0, the stored procedure returns all the prices.
Code basically looked like this..

public List<Prices> GetPrices(List<Product> products, int priceTypeId)
{
foreach (var item in products)
{
List<int> productIdsParameter = new List<int>();
productIdsParameter.Add(item.ProductID);
List<Price> prices = dataProvider.GetPrices(productIdsParameter, 0);
foreach (var price in prices)
{
if (price.PriceTypeID == priceTypeId)
{
prices = dataProvider.GetPrices(productIdsParameter, price.PriceTypeID);
return prices;
}
* Omitting the other 'WTF?' code to handle the zero price type
}
}
}

I removed the double stored procedure call, updated the method signature to only accept the list of product numbers (which it was before the 'major refactor'), deployed the service to dev (the issue was reproducible in our dev environment) and had the DBA monitor.

The two devs and the manager are grumbling and mocking the changes (they never looked, they assumed I wrote some threading monstrosity) then the DBA walks up..

DBA: "We're good. You hit the database pretty hard and the CPU never moved. Execution plans, locks, all good to go."
<dba starts to walk away>
DevMgr: "No fucking way! Putting that code in a thread wouldn't have fix it"
Me: "Um, I didn't use threads"
Dev1: "You had to. There was no way you made that code run faster without threads"
Dev2: "It runs fine in dev, but there is no way that level of threading will work in production with thousands of requests. I've got unit tests that prove our design is perfect."
Me: "I looked at what the code was doing and removed what it shouldn't be doing. That's it."
DBA: "If the database is happy with the changes, I'm happy. Good job. Get that service deployed tomorrow and lets move on"
Me: "You'll remove the recommendation for a complete re-write of the service?"
DevMgr: "Hell no! The re-write moves forward. This, whatever you did, changes nothing."
DBA: "Hell yes it does!! I've got too much on my plate already to play babysitter with you assholes. I'm done and no one on my team will waste any more time on this. Am I clear?"

Seeing the dev manager face turn red and the other two devs look completely dumbfounded was the most satisfying bug I've fixed.

Stolened! 😆😆

Try it!!!😂😂😂😂😂😂

So we hired an intern and his first task was to change a few things in email layout for our client, which is an investment bank.

I told to one of my developers to make his local database dump and setup the project for an intern. When intern completed the task, my developer thought that title "Dow Jones index crashed" was pretty funny title for a test.

What he didn't thought through enough, is that he forgot to configure fake SMTP server and he had production database dump with real email addresses.

I had really awkward 20 minutes conversation with our client. Fuck my life.

This happend to me around 2 weeks ago. For some reason, I decied to post this now.

I won the lottery, yey! I mean, bot really, but I am <19yo student, "less than junior dev" in my office, but sonce I am the only one who is capable of working with hardware, I was working month back as a sysadmin for a few days. Our last sysadmin was really good working but really, really toxic guy, so he got fired on a spot after argument with some manager or whatever, no big deal, we could have another guy hired in a week. But, our backup server literally was on fire, all data probably dead because bad capacitor or whatever. This was our only backup of everything at the time. Everyone in full fucking panic mode, we had literally no other working HW we could use for backup, but then comes me, intern employed on his first dev job for 3 months. That day I bought some HW for my own personal server at home (Intel NUC with some Celeron, 4GB DDR4 RAM and two 240GB SSDs for RAID 1. My manager asked everyone in the office for sollution how to survive next 4 days before new server arrives. People there had no idea what tk do and no knowedgle about HW, I just came from a break and offered my components for a week, since there was noone else who can work with HW, servers and stuff like this, manager offered me $500+HW cost if I, random intern, can make it work. I installed Debian on that little PC, created RAID1 from both SSDs, installed MySQL server and mirrored GIT server from our last standing server (we had two before one of them went lit 🔥), made simple Python script to copy all data on that RAID, with some help of our database guy copied whole DB from production to this little computer and edited some PHP so every SQL request made on our server will run on that NUC too. Everything after ±2 hours worked perfectly. Untill a fucking PSU burned in our server and took RAID controller with him in sillicon heaven next night, so we could not access any data unltill we got a new one. Thanks to every god out there, I was able to create software RAID from survived HDDs on our production server and copy all data from that NUC on the servers software RAID and make it working at 3 AM in the night before an exam 😂. Without this, we would be next ±40 hours without aerver running and we might loose soke of our data and customers. So my little skill with Linux, Python, MySQL and most importantly my NUC hardware I got that day running as a backup server saved maybe whole company 😂.
Btw, guess who is now employee of the year with $2500 bonus? 😀
Sorry for bragging and log post, but I was so lucky an so happy when everything worked out, good luck to all sysadmins out there! 👍

TL:DR: Random intern saved company and made some money 😂

So I took on a fairly big project and poured my heart and soul into it, was the biggest thing I did yet. I kept on sending beta's to the customer after each change for review! Kept on insisting that they review it, the answer was always "this looks amazing keep doing what you're doing"! After I finished and pushed everything to production.

They didn't use it for nearly 6 months! And then out of the blue they call me saying that half of the app is wrong.. WTF? Where was this information during testing! I informed them that the changes would take some time since I need to do migrations and change the whole database schema.

In which they replied "but you already finished it once won't changing things make it easier? We shouldn't pay for your mistakes"
I don't know how I handled that but they should be thankful they were half way across the country 😠😠😠😠

Worst data loss? Probably back when I was working at gitlab and accidentally dropped a production database....

Just heard the DB admin yell from across the hall...

The solution for this one isn't nearly as amusing as the journey.

I was working for one of the largest retailers in NA as an architect. Said retailer had over a thousand big box stores, IT maintenance budget of $200M/year. The kind of place that just reeks of waste and mismanagement at every level.

They had installed a system to distribute training and instructional videos to every store, as well as recorded daily broadcasts to all store employees as a way of reducing management time spend with employees in the morning. This system had cost a cool 400M USD, not including labor and upgrades for round 1. Round 2 was another 100M to add a storage buffer to each store because they'd failed to account for the fact that their internet connections at the store and the outbound pipe from the DC wasn't capable of running the public facing e-commerce and streaming all the video data to every store in realtime. Typical massive enterprise clusterfuck.

Then security gets involved. Each device at stores had a different address on a private megawan. The stores didn't generally phone home, home phoned them as an access control measure; stores calling the DC was verboten. This presented an obvious problem for the video system because it needed to pull updates.

The brilliant Infosys resources had a bright idea to solve this problem:

- Treat each device IP as an access key for that device (avg 15 per store per store).
- Verify the request ip, then issue a redirect with ANOTHER ip unique to that device that the firewall would ingress only to the video subnet
- Do it all with the F5

A few months later, the networking team comes back and announces that after months of work and 10s of people years they can't implement the solution because iRules have a size limit and they would need more than 60,000 lines or 15,000 rules to implement it. Sad trombones all around.

Then, a wild DBA appears, steps up to the plate and says he can solve the problem with the power of ORACLE! Few months later he comes back with some absolutely batshit solution that stored the individual octets of an IPV4, multiple nested queries to the same table to emulate subnet masking through some temp table spanning voodoo. Time to complete: 2-4 minutes per request. He too eventually gives up the fight, sort of, in that backhanded way DBAs tend to do everything. I wish I would have paid more attention to that abortion because the rationale and its mechanics were just staggeringly rube goldberg and should have been documented for posterity.

So I catch wind of this sitting in a CAB meeting. I hear them talking about how there's "no way to solve this problem, it's too complex, we're going to need a lot more databases to handle this." I tune in and gather all it really needs to do, since the ingress firewall is handling the origin IP checks, is convert the request IP to video ingress IP, 302 and call it a day.

While they're all grandstanding and pontificating, I fire up visual studio and:

- write a method that encodes the incoming request IP into a single uint32
- write an http module that keeps an in-memory dictionary of uint32,string for the request, response, converts the request ip and 302s the call with blackhole support
- convert all the mappings in the spreadsheet attached to the meetings into a csv, dump to disk
- write a wpf application to allow for easily managing the IP database in the short term
- deploy the solution one of our stage boxes
- add a TODO to eventually move this to a database

All this took about 5 minutes. I interrupt their conversation to ask them to retarget their test to the port I exposed on the stage box. Then watch them stare in stunned silence as the crow grows cold.

According to a friend who still works there, that code is still running in production on a single node to this day. And still running on the same static file database.

#TheValueOfEngineers

The experience that made me feel like a dev badass was when a teammate accidentally deleted the database for production and I had the latest backup. Everybody was panicking not until I told them I had the solution

Accidently deleted user table on production database and some backups where broken. Had to deploy a 4 days old backup...

Where I work, in our database, we use 3 to indicate true and 7 to indicate false and 0 is true but null is false

In another table, we use 'P' to indicate true and 'I' to indicate false and 'Y' is also false and null is false

And the most used table, we also use 'Y' to indicate yes and 'N' to indicate no, but null is also Yes.

We also store integers as varchar in a live table, but stays an integer in all the other tables. I hope I'm not there when the number of digits exceeds the varchar limit.

These are all live and used in production all created by my boss, the head of IT.

Always use SELECT-query with the same conditions before you DELETE/UPDATE in a production database.

Today I saw a code written by my junior. Basically excel export. The laravel excel package provide great ways for optimization.

My junior instead did 6 times loop to modify the data before giving that data to the export package. We need to export around 50K users.

When I asking him why this ? He said it works and it's fast so what the issue ???

Noob , you have only 100 users in the database and production has 10 million.

Sometime I just want to kill him.

Rant, but also want to know, how the fuck do you accidentally drop a database link. In production. On the most important database. Housing our biggest applications.

Old boss story. This guy was nice but a terrible boss. Also relevant, he has a background in IT so should know better.

Him: So when you wanna check a password is correct you just unhash it in the database?
Me: *facepalm*

Me: Hey we should be doing unit and integration testing at a minimum to lower bugs.
Him: We don't need those, we're not a bank. If a problem comes up we just fix it and push to production.
(A while later)
Him(in email): Why do we keep getting bugs reported. Don't you devs test your code.

I was mildly annoyed at that one.

Him: We're always over budget on projects, how can we fix this.
Me: What if we increase our quotes.(technically there are other ways as well but not really possible at that time)
Him: We can't do that, clients won't want to pay.

Me: *finishing off my handover as I'm leaving for a new job*
Him: Wow you do a lot of work

one of our engineers accidentally deleted our production database and our clients didn't even notice. that's when we knew they are not using our product.

Ok so this happend in the last 3 days, I didn't post it till now because I had to seriously take a rest with all the bullshit and stress that came with it...
(Legacy project I have the lead in called: "Foo")

Monday:
Management decided it would be effective to add a senior and a junior to Foo, which would make (together with me) to be 2 juniors and one senior developer
Well I've spend most of that day helping both the junior and the senior to setup "Foo" on their local development machines... So I could not do any programming myself

tuesday:
The senior wanted to refactor EVERYTHING... and I had to stop him multiple times because we simply do not have the time to do that...
The junior tried to work on other things as much as he could, and after he had run out of things to do, asked me for EVERYTHING... EVEN WHERE TO FUCKING CHANGE SOME GOD DAMN STRINGS!....
Also he did in total 3 commits, two of which existed of my code (because I had to "help" him

wednesday:
Both the junior and senior were removed from the project and I got another senior.. who fucking deleted the production database on accident

god damn rough few days man...

Turns out that dropping all tables on the production database wasn’t such a good idea. 😅

Thank god we had a backup from 10 minutes before it happend.

*In the final weeks of development with a project on a short timeline because the client "needs it".*

Client: "We've hired a consultant we want you to work with."

Me: "Okay, can we push this to after the delivery?"

Client: "Of course"

Wake up to an email from the consultant with a list of scripts he just ran on the production database server for the currently live app.

Get follow-up emails about bugs and app crashes from the client.

My rage is so hot it can keep warm an Eskimo tribe over the winter season.

Other PM: We must fix the database performance issues now.

Me: We can't. We're still only halfway on the dependency chain to tackle this and honestly, even if the dependency chain would be fulfilled, I'd leave at least 2 weeks monitoring the production after the changes were rolled out before we further poke around.

Other PM: This is taking far too long. And whaddya mean by dependency chain? Why was I not informed about this?

Me: *sigh* like in every meeting in the last weeks: the dependency chain are the current open blockers before we can proceed with the database changes. We've talked about this _at length_... Especially why these blockers exist.

Other PM: No, we need to start now. I've _examined_ at the blockers or "dependency chain" as you call it.

(Examined.... He opened on his currently streaming laptop, which was connected to the active beamer, the mentioned ticket with a detailed blocker ... And quickly scrolled. Yeeah. Warmonger...).

Me: I'm very tired of discussing this. But since you are already presenting us the ticket, read out the referenced meeting notes... We explained it in great detail.

Other PM: Why? This is just a waste of time!!!!!!!

--
Yes. This happened. Other PM was my nemesis.
In this meeting were 2 PMs (Him, Me)… I think 5 - 7 devs... And we were sitting in this meeting since 2 hours at least. Everyone was angry...

After this "manifesto of intelligence"… I simply left the room, followed by a few devs.

And yes. Other PM did this on a regular basis....

Who needs fireworks if the sound of angry consumers after deleting a production database is as resounding as the sound of war itself

Never ever have a tab open on production server database. Changed all users passwords by mistake. Thought was my test database.

My worst dev sin was leaving out the 'WHERE' in a SQL update statement on a production database

Set every booking to be owned by the same sales consultant 👀

Just took a new role...

Production database filled with tables like the following :/

SELECT TOP (1000) [ID]
,[Field1]
,[Field2]
,[Field3]
,[Field4]
,[Field5]
,[Field6]
,[Field7]
,[Field8]
,[Field9]
,[Field10]
,[Field11]
,[Field12]
,[CapID]
,[Car_Manufacturer]
,[Car_Range]
,[Car_Derivative]
,[Field17]
,[Field18]
,[Field19]
,[Car_Model]
,[Field21]
,[Field22]
,[Field23]
FROM [****].[dbo].[****]

This isn't even the worst, there's a table with 250+ fields that should be broken out into 10-15 tables. The code is far, far worse.

Don't think I'll be here too long.

I think I want to quit my first applicantion developer job 6 months in because of just how bad the code and deployment and.. Just everything, is.

I'm a C#/.net developer. Currently I'm working on some asp.net and sql stuff for this company.

We have no code standards. Our project manager is somewhere between useless and determinental. Our clients are unreasonable (its the government, so im a bit stifled on what I can say.) and expect absurd things from us. We have 0 automated tests and before I arrived all our infrastructure wasn't correct to our documentation... And we barely had any documentation to begin with.

The code is another horror story. It's out sourced C# asp.net, js and SQL code.. And to very bad programmers in India, no offense to the good ones, I know you exist. Its all spagheti. And half of it isn't spelled correctly.

We have a single, massive constant class that probably has over 2000 constants, I don't care to count. Our SQL projects are a mess with tons of quick fix scripts to run pre and post publishing. Our folder structure makes no sense (We have root/js and root/js1 to make you cringe.) our javascript is majoritly on the asp.net pages themselves inline, so we don't even have minification most of the time.

It's... God awful. The result of a billion and one quick fixes that nobody documented. The configuration alone has to have the same value put multiple times. And now our senior developer is getting the outsourced department to work on moving every SINGLE NORMAL STRING INTO THE DATABASE. That's right. Rather then putting them into some local resource file or anything sane, our website will now be drawing every single standard string from the database. Our SENIOR DEVELOPER thinks this is a good idea. I don't need to go into detail about how slow this is. Want to do it on boot? Fine. But they do it every time the page loads. It's absurd.

Our sql database design is an absolute atrocity. You have to join several tables together just to get anything done. Half of our SP's are failing all the time because nobody really understands the design. Its gloriously awful its like.. The epitome of failed database designs.

But rather then taking a step back and dealing with all the issues, we keep adding new features and other ones get left in the dust. Hell, we don't even have complete browser support yet. There were things on the website that were still running SILVERLIGHT. In 2019. I don't even know how to feel about it.

I brought up our insane technical debt to our PM who told me that we don't have time to worry about things like technical debt. They also wouldn't spend the time to teach me anything, saying they would rather outsource everything then take the time to teach me. So i did. I learned a huge chunk of it myself.

But calling this a developer job was a sick, twisted joke. All our lives revolve around bugnet. Our work is our BN's. So every issue the client emails about becomes BN's. I haven't developed anything. All I've done is clean up others mess.

Except for the one time they did have me develop something. And I did it right and took my time. And then they told me it took too long, forced me to release before it was ready, even though I had never worked on what I was doing before. And it worked. I did it.

They then told me it likely wouldn't even be used anyway. I wasn't very happy at all.

I then discovered quickly the horrors of wanting to make changes on production. In order to make changes to it, we have to... Get this

Write a huge document explaining why. Not to our management. To the customer. The customer wants us to 'request' to fix our application.

I feel like I am literally against a wall. A huge massive wall. I can't get constent from my PM to fix the shitty code they have as a result of outsourcing. I can't make changes without the customer asking why I would work on something that doesn't add something new for them. And I can't ask for any sort of help, and half of the people I have to ask help from don't even speak english very well so it makes it double hard to understand anything.

But what can I do? If I leave my job it leaves a lasting stain on my record that I am unsure if I can shake off.

... Well, thats my tl;dr rant. Im a junior, so maybe idk what the hell im talking about.

This just happened at work, except for the fact the entire database was dropped...

Because I’m a fucking cowboy and a charlatan, and because I hate sleep and despise feeling refreshed and happy, I’m working pretty much full time as a contractor (I’m the full stack dev. I do everything) on a (well funded) startup alongside my day job.

Tonight I had to make some quick (lol “quick”) changes to a core piece of the platform.

Now before continuing please refer back to the first line of this rant.

So instead of writing new functionality, I copied and pasted another section.

I renamed all references of “new_order” to, cleverly “new_order2”.

I know.

I deploy to production...

My phone starts blowing up. In short, everything is fucked.

I’m going over the query, checking the production database. Why is this manifesting like this? It all looks correct.

2 HOURS of broken sales, pissed off customers, pissed off service agents and I see that there was still one reference of “new_order” that should have been “new_order2”.

I am a piece of shit.

Nothing screams panic like accidentally deploying an older, broken staging build, that also run outdated database migrations on start, straight to production

I messed up carelessly in production. Learnt how SQL queries bite you in the ass when it knows you are under pressure.

Was hosting an online quiz kinda thing during my college techfest. Tens of thousands of people participating.

Using MySQL as database and thousands of queries were being executed. Everyone were pretty excited as the event just opened up.

None of the teams could solve one particular level. Turns out the solution was wrong and was asked by the organisers to change the solution for that particular level. Usual stuff, right?

Was too lazy to open up the web UI for the back office and so, straight ahead logged in to the MySQL server and ran the UPDATE query on the table consisting of the solutions.

It had been a couple of hours and the organisers came to me with a weird problem. There were no changes in the scoreboard for the last two hours. Everyone were stuck wherever they were. Weird, right?

I then realized.

Fk.

In that dreaded query, I had only run

UPDATE 'qa' SET answer = 'something'

leaving out the where clause, specifying the question to update, like

WHERE qno=13

As a result, solutions to all the questions were updated to the same answer. After hastily fixing everything back, I had the dreaded conversation.

Org: What was the problem?

Me: It was the cache.

Org: Damn thing. Always messes up.

Me: *sheepishly* yeah

Probably the most embarrassing moment in my life, wrt coding 😑

I woke up early and thaught I could finish some database work today.
Beeing tipsy I deleted two production tables and shit my pants.

Luckily I do backups since the last retarded db-deletion.

Ended in an UPDATE without WHERE query to a core table in a Drupal project (in a dev database stored in the same server that production database was)

If you're going to request CRITICAL changes to thousands of records in the database, and approve it through testing which is done on an exact replica of production, then tell me it was done incorrectly after the fact it has been implemented and you didn't actually review the changes made to the data or business logic that you requested then you are an idiot. Our staging environment is there to ensure all the changes are accurate you useless human. Its the data you provided, I didn't just magically pull it from thin air to make yours and my job a pain the ass.

Client: can you filter boats by location?
Me: Let me see... As you know, there are three remote systems that feed data into your database. I'd have to make a connection between the location records. But I can't rely on coordinates, name, ID or anything else. You'd have to manually create those links for me by remote systems records IDs. Telling me that record XY from system A is identical to record YX from system B, etc...
Client: How many records are we talking about?
Me: 504.

Three days later...

Client: Got it, is that enough for you in excel?
Me: Let me see... Very nice work, I can work with that.
Client: I almost died on it!

An hour later...

Me: Got it, test it and let's run it on the production version.
Client: It works beautifully.

A minute later...

Can we filter the ships by ports?
Me: Let me see... Yes, it's theoretically possible, but it's the same situation as with places...
Client: How many records are we talking about?
Me: 12,647.

Skype relayed to me the sound of something heavy falling, something grunting. Something dying.

My manager is instructing my team to add a feature that can only be enabled for users by running an update script in the database.

When I argued that it's not really "complete" if it can't be turned on without someone going into the production database, I was told that not only is it complete, but they plan to have our non-technical customer service enable it for customers if the customer requests it...

Apparently giving everyone and their brother write access to prod is a good idea, but implementing a checkbox is a "waste of time and would cost too much money".

Probably going to float my resume... :-p

OMFG I don't even know where to start..
Probably should start with last week (as this is the first time I had to deal with this problem directly)..

Also please note that all packages, procedure/function names, tables etc have fictional names, so every similarity between this story and reality is just a coincidence!!

Here it goes..

Lat week we implemented a new feature for the customer on production, everything was working fine.. After a day or two, the customer notices the audit logs are not complete aka missing user_id or have the wrong user_id inserted.
Hm.. ok.. I check logs (disk + database).. WTF, parameters are being sent in as they should, meaning they are there, so no idea what is with the missing ids.

OK, logs look fine, but I notice user_id have some weird values (I already memorized most frequent users and their ids). So I go check what is happening in the code, as the procedures/functions are called ok.

Wow, boy was I surprised.. many many times..
In the code, we actually check for user in this apps db or in case of using SSO (which we were) in the main db schema..
The user gets returned & logged ok, but that is it. Used only for authentication. When sending stuff to the db to log, old user Id is used, meaning that ofc userid was missing or wrong.

Anyhow, I fix that crap, take care of some other audit logs, so that proper user id was sent in. Test locally, cool. Works. Update customer's test servers. Works. Cool..

I still notice something off.. even though I fixed the audit_dbtable_2, audit_dbtable_1 still doesn't show proper user ids.. This was last week. I left it as is, as I had more urgent tasks waiting for me..

Anyhow, now it came the time for this fuckup to be fixed. Ok, I think to myself I can do this with a bit more hacking, but it leaves the original database and all other apps as is, so they won't break.
I crate another pck for api alone copy the calls, add user_id as param and from that on, I call other standard functions like usual, just leave out the user_id I am now explicitly sending with every call.
Ok this might work.

I prepare package, add user_id param to the calls.. great, time to test this code and my knowledge..

I made changes for api to incude the current user id (+ log it in the disk logs + audit_dbtable_1), test it, and check db..
Disk logs fine, debugging fine (user_id has proper value) but audit_dbtable_1 still userid = 0.

WTF?! I go check the code, where I forgot to include user id.. noup, it's all there. OK, I go check the logging, maybe I fucked up some parameters on db level. Nope, user is there in the friggin description ON THE SAME FUCKING TABLE!!
Just not in the column user_id...

WTF..Ok, cig break to let me think..

I come back and check the original auditing procedure on the db.. It is usually used/called with null as the user id. OK, I have replaced those with actual user ids I sent in the procedures/functions. Recheck every call!! TWICE!! Great.. no fuckups. Let's test it again!
OFC nothing changes, value in the db is still 0. WTF?! HOW!?

So I open the auditing pck, to look the insides of that bloody procedure.. WHAT THE ACTUAL FUCK?!
Instead of logging the p_user_sth_sth that is sent to that procedure, it just inserts the variable declared in the main package..
WHAT THE ACTUAL FUCK?! Did the 'new guy' made changes to this because he couldn't figure out what is wrong?! Nope, not him. I asked the CEO if he knows anything.. Noup.. I checked all customers dbs (different customers).. ALL HAD THIS HARDOCED IN!!! FORM THE FREAKING YEAR 2016!!! O.o

Unfuckin believable.. How did this ever work?!

Looks like at the begining, someone tried to implement this, but gave up mid implementation.. Decided it is enough to log current user id into BLABLA variable on some pck..
Which might have been ok 10+ years ago, but not today, not when you use connection pooling.. FFS!!

So yeah, I found easter eggs from years ago.. Almost went crazy when trying to figure out where I fucked this up. It was such a plan, simple, straight-forward solution to auditing..

If only the original procedure was working as it should.. bloddy hell!!

Soooo... My senior dev just dropped a very importand table om our production enviroment.
Time to restore the database.. only 50 GB left

Worst collaboration experience story?

I was not directly involved, it was a Delphi -> C# conversion of our customer returns application.

The dev manager was out to prove waterfall was the only development methodology that could make convert the monolith app to a lean, multi-tier, enterprise-worthy application.

Starting out with a team of 7 (3 devs, 2 dbas, team mgr, and the dev department mgr), they spent around 3 months designing, meetings, and more meetings. Armed with 50+ page specification Word document (not counting the countless Visio workflow diagrams and Microsoft Project timeline/ghantt charts), the team was ready to start coding.
The database design, workflow, and UI design (using Visio), was well done/thought out, but problems started on day one.
- Team mgr and Dev mgr split up the 3 devs, 1 dev wrote the database access library tier, 1 wrote the service tier, the other dev wrote the UI (I'll add this was the dev's first experience with WPF).
- Per the specification, all the layers wouldn't be integrated until all of them met the standards (unit tested, free from errors from VS's code analyzer, etc)
- By the time the devs where ready to code, the DBAs were already tasked with other projects, so the Returns app was prioritized to "when we get around to it"

Fast forward 6 months later, all the devs were 'done' coding, having very little/no communication with one another, then the integration. The service and database layers assumed different design patterns and different database relationships and the UI layer required functionality neither layers anticipated (ex. multi-users and the service maintaining some sort of state between them).

Those issues took about a month to work out, then the app began beta testing with real end users. App didn't make it 10 minutes before users gave up. Numerous UI logic errors, runtime errors, and overall app stability. Because the UI was so bad, the dev mgr brought in one of the web developers (she was pretty good at UI design). You might guess how useful someone is being dropped in on complex project , months after-the-fact and being told "Fix it!".

Couple of months of UI re-design and many other changes, the app was ready for beta testing.

In the mean time, the company hired a new customer service manager. When he saw the application, he rejected the app because he re-designed the entire returns process to be more efficient. The application UI was written to the exact step-by-step old returns process with little/no deviation.

With a tremendous amount of push-back (TL;DR), the dev mgr promised to change the app, but only after it was deployed into production (using "we can fix it later" excuse).

Still plagued with numerous bugs, the app was finally deployed. In attempts to save face, there was a company-wide party to celebrate the 'death' of the "old Delphi returns app" and the birth of the new. Cake, drinks, certificates of achievements for the devs, etc.

By the end of the project, the devs hated each other. Finger pointing, petty squabbles, out-right "FU!"s across the cube walls, etc. All the team members were re-assigned to other teams to separate them, leaving a single new hire to fix all the issues.

Adding a feature to webapp...

Webapp relies on database in production server...

*adds feature to production webapp directly*

Every page: ERROR 500

Manager: what did you do???!!!! You MESSED UP the production, FIX IT NOW

*Use ctrl-z because manager doesn't like Version Control*

Typos kill, kids! And deploying to production.

Instead of "for item in items" in my script, I accidentally did "for items in items". Thus, an exponential loop has been entering things into the database for the past few hours before I found the place to fix it.

By the way, this runs on cron every minute. So there are processes still running exponentially right now, possibly 180+.

Yeah, I'm setting up a a test server instead now.

I was asked to look into a site I haven't actively developed since about 3-4 years. It should be a simple side-gig.
I was told this site has been actively developed by the person who came after me, and this person had a few other people help out as well.
The most daunting task in my head was to go through their changes and see why stuff is broken (I was told functionality had been removed, things were changed for the worse, etc etc).
I ssh into the machine and it works. For SOME reason I still have access, which is a good thing since there's literally nobody to ask for access at the moment.
I cd into the project, do a git remote get-url origin to see if they've changed the repo location. Doesn't work. There is no origin. It's "upstream" now. Ok, no biggie. git remote get-url upstream. Repo is still there. Good.
Just to check, see if there's anything untracked with git status. Nothing. Good.
What was the last thing that was worked on? git log --all --decorate --oneline --graph. Wait... Something about the commit message seems familiar. git log. .... This is *my* last commit message. The hell?
I open the repo in the browser, login with some credentials my browser had saved (again, good because I have no clue about the password). Repo hasn't gotten a commit since mine. That can't be right.
Check branches. Oh....Like a dozen new branches. Lots of commits with text that is really not helpful at all. Looks like they were trying to set up a pipeline and testing it out over and over again.
A lot of other changes including the deletion of a database config and schema changes. 0 tests. Doesn't seem like these changes were ever in production.
...
At least I don't have to rack my head trying to understand someone else's code but.... I might just have to throw everything that was done into the garbage. I'm not gonna be the one to push all these changes I don't know about to prod and see what breaks and what doesn't break
.
I feel bad for whoever worked on the codebase after me, because all their changes are now just a waste of time and space that will never be used.

So I'm wrapping up for the day and right before I leave a coworker comes up to me with a problem. Our company uses barcodes to track some of our products through their development and we recently switched over to a new system for producing them. The barcodes for this particular product are supposed to have 8 digits, but the last 200 we printed have 9.

I immediately panic because I wrote the script that generates the bar codes and there had been a bug in the past where the script would add extra leading zeroes that weren't supposed to be there. I scramble and check the database, it would be a huge headache if our production database had been compromised with junk barcodes. Nope, all the new barcodes there have the right number of digits.

Next place to check is in the code that writes the barcodes to a text file for staff to print the physical labels from. Nope that's all fine too.

I ask the person who printed out the recent batch of labels to show me how the printing software reads from the text file. She seems confused by my question and shows me how she manually enters in the barcode range to the software. As she does this I watch her add an extra zero to the numbers. 🙃

Even worse there was an option to import all the codes from a text file literally RIGHT BELOW the manual option.

TLDR; Thought my script had screwed up our database, ended up being the fault of a coworker who didn't know how to import text files.

"Why is the production database empty?"

Obviously, we had backups, but using the situation to make my employer shit his pants were funny. And yes, I'm still there

It's 17:55... Did much work that day since I came in earlier than usual, so I could leave in time and do some shopping with the girlfriend.

A colleague comes in to my room, a tad distressed. He had accidentally ran a fixture script on a production environment database (processing a shipload of records per minute), truncating all tables...

Using AWS RDS to rollback the transaction log takes up about 20m. I had to do that about 5 times to estimate the date and time of when the fixture script ran... Since there was no clear point in time...

Finally I get to the best state of the data I could get. I log in remotely run some queries. All is well again... With minor losses in data.

I try to download a dump using pg_dump and apparently my version is mismatched with the server. I add the latest version to aptitudes source list of postgres repo and I am ready to remove and purge the current postgres client and extensions...

sudo apt-get remove post*
Are you sure? (Y/n) *presses enter and enters into a world of pain*

Apparently a lot of system critical applications start with post... T_T

Tomcat

manager: "hey, we have this old java software running and need it to be compatible with our brand new crm system"
me: "okay, i've never workes with jsp before, but i'll have a look at it"
code: undocumented, who would have guessed
manager: "oh and btw. you must test your code on our live system, with our production database, but make sure not to brake anything, our last backup was 20 months ago."
me: "..."

So I started in a new job a week ago for a two weeks probation period, im getting payed double than my last job but it's so professionally frustrating.

They use a full php stack with a framework called tinymvc that I never heard of and the last commit is from 2009.

Beside this they implement some sort of "flexible" MVC where a great amount of the logic are inside the views. They have one model for each entity (in theory) but in reality one model have methods for a lot of tables.

Beside this the i18n is done by querying the database for all the translation strings and copying it in every user session, so every session file it's about 400kb where around 380 are duplicated translations.

The views folder is empty because they decided to modify the framework to look for the views in another folder called resource's and the development must be done connected directly to the production database

Above all this shit all the many-to-many relations in the database (MySQL) are handled creating a comma separated field on one of the tables, completely breaking the reference integrity.

So, after a week of work I can't stop thinking who the fuck developed this? In which world this shit is okay? How can I work around this big lake of shit?

Oh man.
I imagine the face of the guy who deleted Gitlab production database when he realized what he had done.

Worst hypothetical Dev job... hmm 🤔 well I think I have the right scenario for you. A medical company stores patient charts and critical life saving information in a database. This database makes medical decisions for lifesaving incubators and if there’s a bug it means 10,000 newborn sick babies will die because of your fuck up (oh and you’re criminally liable too so good luck!). But beyond the high pressure job that sounds at least somewhat normal, the database is written in a special form of assembly for a custom undocumented CPU where only one copy in the world exists so all tests and development are on production. Google and StackOverflow is banned so your only resource is your brain. Good luck🍀

This former developer made an app 2 years ago which is in production since then. On the 404 page it's throws the database credentials. The database saves personal information about the mobile owner.

Luckily I found out and fixed it. The client doesn't know about this.

Oh boy!

Maybe not worst, but most frustrating. One of the systems I helped maintain at my first job had a few different bugs that caused bad data in the database. The "solution" to the problem was to write SQL queries to directly fix the production data. This would take one member of our team (it rotated weekly) about an hour every day to fix because there were literally dozens of these errors.

All the devs knew that we could identify the root cause and fix it in, probably, 3-4 days tops. Management would never approve the time because it would take longer to fix the root cause than it took to fix the data.

I worked at that company for 7 years. The bug was there when I came on, and it was there when I left.

I don't know if I'm being pranked or not, but I work with my boss and he has the strangest way of doing things.
- Only use PHP
- Keep error_reporting off (for development), Site cannot function if they are on.
- 20,000 lines of functions in a single file, 50% of which was unused, mostly repeated code that could have been reduced massively.
- Zero Code Comments
- Inconsistent variable names, function names, file names -- I was literally project searching for months to find things.
- There is nothing close to a normalized SQL Database, column ID names can't even stay consistent.
- Every query is done with a mysqli wrapper to use legacy mysql functions.
- Most used function is to escape stirngs
- Type-hinting is too strict for the code.
- Most files packed with Inline CSS, JavaScript and PHP - we don't want to use an external file otherwise we'd have to open two of them.
- Do not use a package manger composer because he doesn't have it installed.. Though I told him it's easy on any platform and I'll explain it.
- He downloads a few composer packages he likes and drag/drop them into random folder.
- Uses $_GET to set values and pass them around like a message contianer.
- One file is 6000 lines which is a giant if statement with somewhere close to 7 levels deep of recursion.
- Never removes his old code that bloats things.
- Has functions from a decade ago he would like to save to use some day. Just regular, plain old, PHP functions.
- Always wants to build things from scratch, and re-using a lot of his code that is honestly a weird way of doing almost everything.
- Using CodeIntel, Mess Detectors, Error Detectors is not good or useful.
- Would not deploy to production through any tool I setup, though I was told to. Instead he wrote bash scripts that still make me nervous.
- Often tells me to make something modern/great (reinventing a wheel) and then ends up saying, "I think I'd do it this way... Referes to his code 5 years ago".
- Using isset() breaks things.
- Tens of thousands of undefined variables exist because arrays are creates like $this[][][] = 5;
- Understanding the naming of functions required me to write several documents.
- I had to use #region tags to find places in the code quicker since a router was about 2000 lines of if else statements.
- I used Todo Bookmark extensions in VSCode to mark and flag everything that's a bug.
- Gets upset if I add anything to .gitignore; I tried to tell him it ignores files we don't want, he is though it deleted them for a while.
- He would rather explain every line of code in a mammoth project that follows no human known patterns, includes files that overwrite global scope variables and wants has me do the documentation.
- Open to ideas but when I bring them up such as - This is what most standards suggest, here's a literal example of exactly what you want but easier - He will passively decide against it and end up working on tedious things not very necessary for project release dates.
- On another project I try to write code but he wants to go over every single nook and cranny and stay on the phone the entire day as I watch his screen and Im trying to code.

I would like us all to do well but I do not consider him a programmer but a script-whippersnapper. I find myself trying to to debate the most basic of things (you shouldnt 777 every file), and I need all kinds of evidence before he will do something about it. We need "security" and all kinds of buzz words but I'm scared to death of this code. After several months its a nice place to work but I am convinced I'm being pranked or my boss has very little idea what he's doing. I've worked in a lot of disasters but nothing like this.

We are building an API, I could use something open source to help with anything from validations, routing, ACL but he ends up reinventing the wheel. I have never worked so slow, hindered and baffled at how I am supposed to build anything - nothing is stable, tested, and rarely logical. I suggested many things but he would rather have small talk and reason his way into using things he made.

I could fhave this project 50% done i a Node API i two weeks, pretty fast in a PHP or Python one, but we for reasons I have no idea would rather go slow and literally "build a framework". Two knuckleheads are going to build a PHP REST framework and compete with tested, tried and true open source tools by tens of millions?

I just wanted to rant because this drives me crazy. I have so much stress my neck and shoulder seems like a nerve is pinched. I don't understand what any of this means. I've never met someone who was wrong about so many things but believed they were right. I just don't know what to say so often on call I just say, 'uhh..'. It's like nothing anyone or any authority says matters, I don't know why he asks anything he's going to do things one way, a hard way, only that he can decipher. He's an owner, he's not worried about job security.

Today was a manic-depressive kind of day. Spent the morning helping some developers with getting their code to run a stored procedure to drop old partitions, but it wasn't working on their end. It was a fairly simple proc. But working with partitions is a little like working with an array. I figured out that they were passing the wrong timestamp, and needed to add +1 to delete the right partition. Got that sorted out, and things were good. Lunch time.

After lunch I did some busy work, and then the PO comes up at about 2PM and says he's assigned some requests to me. The first was just attaching some scripts. Easy. The second, the user wants a couple of schemas exported ... at 6PM. I've been in the office since 6:45AM.

While I'm setting up some commands to run for the data export, a BA walks up and asks if I'm filling in for another DBA who is out for a few weeks. Yep. There's a change request that hasn't been assigned, and he normally does the work. I ask when it's due. Well, the pre-implementation was supposed to be done in the morning, but it wasn't, and we're in the implementation window ... half way through. I bring up the change task, and look at. Create new schema and users. That's all it says. The BA laughs. I tell I need more to go on. 10 minutes later he sends an email with the information. There's only two hours left in the window, and I can only use half of it, because the production guys have to their stuff, and we're in their window. Now I'm irritated, because I'm new to Oracle, and it's an unforgiving mistress. Fortunately, another DBA says he'll do it, so that we can get it done in time. But can't work it either, because Dev DBAs don't have access to QA, and the process required access for this task. Gets shelved until the access issue is resolved. It's now after 4:15PM. I'm going to in traffic with that 6PM deadline.

I manage to get home and to the computer by 5:45PM. Log in. Start VPN. Box pops on screen. Java needs to update. I chose skip update. Box pops up again. It won't let me log in until Java is current. Passed.

I finally get logged in, and it's 6:10PM. I'm late getting the job started. I pull up Putty and log into the first box, and paste my pre-prepared command in the command line and hit error. Command not found. I'm tired, so it's a moment to sink in. I don't have time for this.

I log into DBArtisan and pull up the first data base, use the wizard to set the job, and off it goes. Yay. Bring up the second database, and have enter the connect info. Host not found. Wut? Examine host name. Yep, it's correct. Try a different method. Host not found. Go back to Putty. Log in. Past string. Launch. Command not found. Now my brain is quitting on me. Why now? It's after 6:30PM. Fiddle with some settings, reset $Oracle home. Try again. Yay. It works. I'm done. It's after 7PM.

There is nothing like technology to snatch the euphoria of a success away from you. It's a love-hate thing, but I wouldn't trade it for anything else. I'm done. Good night.

I was testing database migration and there was some issues which I couldn't fix them. So I drop the table just to test if the issues are still there.
"Why nothing have been changed?" After some minutes I realize it was production database which I had dropped 😓

Discovered yesterday that my boss does tests with production database... and I'm responsible for the fucking backups and he doesn't even care to let me now when so I can at least schedule one at slave. Come on... it's not that hard to let others know or test on your own machine...

I accidentally started a reindex on a collection that had 14 million records in the middle of the day. Caused an outage in a major portion of our applications for about 3 hours. Worst thing was that once I pressed enter, I realized that it was for the production database, and not the staging database like I intended. I immediately went to go tell the dev ops lead, and he basically said, "whelp, let's just sit back and watch the world burn. Not much we can do about it"

When the test database has better specifications & performance than the production one.. FML! 😭😭😭😭😭

One night, after one very stressful week of production code fixes (I was working on a game with some friends and I created the network infrastructure for P2P and database communication from scratch), I was at my gf's house. After we fell asleep, I stood up and screamed right at her something like "I fucking already told you how X works and how to communicate with Y. Learn to write code properly and after double checking yours then you shall ask me for a non-existing 'bug' fix. Learn how to properly write event based code and use polling you moron!". After that I turned to the other side and fell asleep immediately.
When I saw her the next morning sleeping in the couch, I could not understand why... Only after she described to me the whole incident I started laughing.
After that I just took two weeks off the project and after that period I never actually worked the same way (so hard) in my free time with them.

I once agreed to maintain and develop an application used in a different section of the school to keep inventory and make sure everything is where it is supposed to be.

At first there was enthusiasm, together with 2 of my classmates we agreed and git clone-d the .NET application that now graduated students built and maintained for the past few years. What could go wrong right?!

It became clear that the original students that worked on it followed an older curriculum, meaning they still got taught .NET instead of the core variant that we get now, not only that but it also seemed that they either did not fully grasp the Clean/Onion architecture or didn't get it in class since there were infrastructure components in the 'Domain' project of the solution. Think of 2 DBContexts in the domain model, yep.

One of us bailed in the first week, the other one and I felt bad for the people using the app so we went on and tried to work on the first bugs that were described in a document. One of these bugs was 'whenever I filter on something in the list, everybody gets to see that filter on their screen instead of only me'. Woah that's weird! Let's see how they put that together!

Oh god, they are using a _static_ variable to store filters, no wonder that it doesn't work properly. Ever heard of sessions?!

Second bug: Sometimes people can't create an account when we sign them up from the admin panel. Alright that is weird, let's figure that one out! Wait a second it seems to work in development? What's this about.

Oh wait I can't create an account on production either? Oh that's weird, wait a second... Why do I have to put my e-mail in a form that was sent to me through e-mail? Why is my address not filled in already? OOH, if someone types in the wrong e-mail address (which is easy since our school has 4 variants of the same f*cking e-mail address) it won't work since it can't recognize the user! Brilliant! Remove e-mail input box and make a token/queryparam determine the user account.

Ah that seems good, it's a mess but it seems a tiny bit better now, great! We're making progress and some sweet buck.

Next bug, trillions of 50x errors on random pages, that's a weird one.

Hm everything works in development, that's odd. Is the production data corrupted?

DID I MENTION that in order to get into the system in development we have to load in a f*cking production database backup ON OUR DEVELOPMENT MACHINE and then ask one of the users' password to login to it and create an account for ourselves? Seeding? What's that, right?!

Anyway, back to bug fixing. I e-mail the the people responsible for the app and get a production admin account, oh I also can't ssh into it because of policies so I have to do everything over e-mail and figure out what's causing the errors. I somehow also wonder if they have any kind of virtualization in place, giving students a VM to do that stuff in doesn't seem so weird does it ? Even with school policies?

Oh btw, 'deploying' means sending a .zip file to a guy in another building and telling him how to configure it, apparently this resulted in a missing folder that the application needed to work and couldn't make on its own. This after 2 weeks of e-mailing back and forth.

After 3 months i quit out of despair and sadness, and due to the fact that I just couldn't do it anymore. I separated everything into logical subprojects and let the last guy handle it, he was OK with that and understood why I left.

Luckily, around that time I already had an actual job at a software development company :)

Okay. So my dumbass boss took this project that had a steep timeline. I told him straight up, it won't work because we won't make the timeline. If we do this, I will be the one bending over backwards to deliver. I don't like to promise and fail. I got the oh don't worry let's just try. If we don't make it that's fine. Unfortunately that's not how I work. I refuse to deliberately fail. So I say okay and we begin. I suggested open source is the fastest way to deliver bit the fucked up part is, I am the only senior dev in the team. I will be expected to reverse engineer the open source app to connect our own deployment parameters. Use tech I have never used before. Connect frontend and backend. Handle dns bullshit. I have literally been working on Vibes and coffee for the past two weeks because ofcourse I ran into so many issues. Now I have an extension for Monday and I hate to fail. So I am not sleeping or resting just working on a fucking java app I didnt build and I am expected to make it work seemlessly on our production environment. I made some progress. Deployed frontend, deployed backend. Forgot to connect production dB so I decided to go with azure database for mysql driver since we have credits on azure. Now my java app is pissing itself over ssl handshake. I generate my keystore and add it and now java socket just times out. I want to pummel somebody or a punching bag that looks like my boss.

I work on a warehouse dev team. One day this past year, I was trying to deploy a new build to a QA server. Earlier that day I had been looking at the logs on the production server and had left the ssh session open. I had been working for less than a year out of college at this point and shouldn't have had access to deploy to the production server.

Long story short I deployed my QA build to the production server and saw there were problems connection to our production database. Then my heart dropped in my chest as I realized I had just brought down our production server.

I managed to get the server back up by rolling back in about 5 minutes and no one ever knew except some people on my team.

I felt horrible for the longest time. Later in the year another guy that joined my team that has about 20 years of experience under his belt did the exact same thing, but needed help rolling it back. Needless to say, that made me feel a lot better. 😂

Definitely the worst moment of my year.

Everything about the company is a mess. The only thing that is decent is the people. And by that I mean they aren't shit.
Workflows are fucked.
Clients are fucked. You're pressuring me to get this shit production ready before new year's eve and you still don't know what the text should say and want to make changes to the UI? The fuck?!
Design is a complete shit show. There is a design team. They only make a fucking psd to show clients how an interface would look like. No mobile version (but it's still expected to work!), no markup. Resolution is fucking inconsistent and whenever a change is requested, they are nowhere to be seen so I have to actually do designing on top of having to use this worthless fucking framework I hate it so much.
Codebases are turbo-fucked because of said framework.
Databases are an inconsistent, fucked up mess. No foreign key constraints because every single fucking table is using the MyISAM engine.

And the thing that really makes me incredibly angry is all the "custom systems" look the fucking same at the database level. Like 30 fucking useless tables made for stupid HR workflows that make no fucking sense.

Today in development: discovered that it's possible via combination of keys to rename a database in SQL Server Management Studio without as much as a dialog box to confirm.

Shout out to the 2000ish users in production that discovered this delightful nugget of info with me.

Lessons learned:

A) Don't trust Microsoft to create software that makes you confirm potentially catastrophic actions

B) Make sure your user hasn't been granted ALTER DATABASE permissions without your knowledge before you start using it.

A little late but whatever.

About half a year ago, I started working on setting up self hosted (slippy) maps. For one, because of privacy reasons, for two, because it'd be in my own control and I could, with enough knowledge, be entirely in control of how this would work.

While the process has been going on for hours every day for about half a year (with regular exceptions), I'll briefly lay out what I've accomplished.

I started with the OpenMapTiles project and tried to implement it myself. This went well but there were two major pitfalls:
1. It worked postgres database based. This is fine but when you want to have the entire world.... the queries took insanely long (minutes, at lower zoom levels) and quite intimate postgres/tooling knowledge was required, which I don't have.
2. Due to the long queries and such, the performance was so bad that the maps could take minutes to render and when you'd want that in production... yeah, no.

After quite some time I finally let that idea sail and started looking into the MBTiles solution; generating sqlite databases of geojson features. Very fast data serving but the rendering can take quite some time.

After some more months, I finally got the hang of it to the point that I automated 50-70 percent of the entire process. The one problem? It takes a shitload of resources and time to generate a worldwide mbtiles database.
After infinite numbers of trial and error, I figured out that one can devide a 'render' (mbtiles aka sqlite database) into multiple layers (one for building data, one for water, one for roads and so on), so I started doing renders that way.
Result? Styling became way more easy and logical and one could pick specific data to display; only want to display the roads? Its way more simple this way. (Not impossible otherwise but figuring out how that works... Good luck).

Started rendering all the countries, continents and such this way and while this seemed like a great idea; the entire world is at 3-4 percent after about a month. And while 40-70 percent generates 10 times as fast, that's still way too slow.

Then, I figured out that you can fetch data per individual layer/source. Thus, I could render every layer separately which is way faster.

Tried that with a few very tiny datasets and bam, it works. (And still very fast).

So, now, I'm generating all layers per continent. I want to do it world based but figured out that that's just not manageable with my resources/budget.

Next to that, I'm working on an API which will have exactly the features I want/need!

Killed the backend production db of my app with a dev ops guy for 20 hours.

The emails I received were not the nicest of all time.

I really think there should be a subject in every CS course to teach us how to handle/work-under Grade-A assholes and dumbfucks. Not that it would help, but atleast warn us on what we are getting into.

In my opinion, development is not *that* hard or frustrating but is made so by these shitty people. But again, what do I know.

I was scolded by my boss for using for-loop to iterate through an array recently. Apparently for-loop is not used in real world projects and this iteration should be done "in-memory". My colleagues and I are still trying to understand and process that.

I was asked to add fitbit integration to a project within 2 hours just because I had "already done it a week ago" in *another* project. Luckily, it was then given to a "senior" developer who took 4 days for it and essentially copy-pasted my work without much changes, ofcourse it stopped working every now and then.

I am given unreal deadlines on my tasks, on technologies I haven't worked on before, and then expected to churn out production ready code with no bugs in them.

My boss literally just sends me the links of 1st three google results on the problems I encounter and report, after humiliating me ofcourse. Yes, I did google it and yes I went through all I could find from Google forums to GitHub issues. When the library/plugin author himself says that this feature is not yet available, don't expect me to develop it in 2 hours you dumbfuck.

And for the love of God, please stop changing the data model every single day and justify it with agile development. Think before making any changes to it. Ever heard of Join queries? Foreign keys? Or any other basic database concepts.
We reached a point where each branch in the repo had different data model. Not kidding. And we were a team of just 4 developers. Atleast inform us when you change models after discussing it with your shit for knowledge "senior" developer, so we don't have to redo it all over again. The channels on slack are not for sharing random articles only.

I am just waiting to complete my year here.

I should have known what I got myself into the day he asked me to remove the comments I had added to explain what my code does. Why you ask? Because "we don't write comments".

I never understood why writing data to a database had to be such a complicated process, at least in production systems.
I shouldn't need a PhD to figure out how the db was configured.

This is a public service announcement with a threat at the end of it:

"Do not, I repeat, do NOT attempt to write web applications, or any particular sort of application that works with a relational database (damn near more than half of applications) without a PROPER grasp and knowledge of SQL.
I do not want to see you reaching out for an ORM either, no, you need to learn to properly design a database or to properly interact with them AT most before you even attempt using an ORM OR designing an application from the beginning, shit will only hurt you in the long term I promise, learning SQL can go a looooong fucking way and most DBA's I know make way tf more than people think they make, it might even be an interesting career choice"

If you do not follow the above advise, and I see your ass reaching for building a web application without the above knowledge I will be under your bed at night, putting oil in my hairy body before I jump into bed to you and leave you confused for the rest of your life.

Build to learn, YES, but for the love of Chamberlain and Boyce PLEASE do not neglect SQL. I have seen such neglect REACH production and I am currently wishing I had these mfkers close to me.

All hail database backups.

Today my worst development related nightmare happened, I managed to screw a production database at work.

Thankfully we had a backup from 2 hours ago, I’ve never been so thankful to scheduled backups.

Take it from me, do backup your databases even if it’s annoying to do.

Piece and merry Christmas ✌️

I dunno about coolest, but I did sort of cement my reputation as the "database guy" in my first job because of this.

My first job was with a group maintaining a series of websites. Because of the nature of the websites, every morning we had to pull the records from one database on one network, sneaker net the data to a database on another network, and import the data via custom data import function.

However, the live site would crash after 100 or so records were imported. The dba at the live site had to script out a custom data partitioning script to do his daily duties, but it definitely messed up his productivity.

Turns out, the custom mass import function had recycled the standard import function, which was only used to import 1 record at a time, and it never closed its database connections, because it never needed to. A one line fix to production code was delivered 6 months later (because that was our release cycle) and I came up with the temporary work around, which was basically removing the connection limit. It would still crash with the work around, but only with multiple days worth of data. So basically only on Monday. Also developed the test set for the import (15k+ records).

Hello DevRant community! It’s been a while, almost 5 years to be exact. The last time I posted here, I was a newbie, grappling with the challenges of a new job in a completely new country. Oh, how time flies!

Fast forward to today, and it’s been quite the journey. The codebase that once seemed like an indecipherable maze is now my playground. The bugs that used to keep me up at night are now my morning coffee puzzles. And the team, oh the team! We’ve moved from awkward nods to inside jokes and shared victories.

But let’s talk about the real hero here - the coffee machine. The unsung hero that has fueled late-night coding sessions and early morning stand-ups. It’s seen more heated debates than the PR comments section. If only it could talk, it would probably write its own rant about the indecisiveness of developers choosing between cappuccino and latte.

And then there are the unforgettable ‘learning opportunities’ - moments like accidentally shutting down the production server or dropping the customer database. Yes, they were panic-inducing crises of apocalyptic proportions at that time, but in hindsight, they were valuable lessons. Lessons about the importance of thorough testing, proper version control, reliable backup systems, and most importantly, owning up to our mistakes.

So here’s to the victories and failures, the bugs and fixes, the refactorings and 'wontfix’s. Here’s to the incredible journey of growth and learning. And most importantly, here’s to this amazing community that’s always been there with advice, sympathy, humor, and support.

Can’t wait to see what the next 5 years bring! 🥂

Ok, here goes...

I was once asked to evaluate upgrade options for an online shop platform.

The thing was built on Zend 1, but that's not the problem.

The geniuses that worked on it before didn't have any clue about best practices, framework convention, modular thinking, testing, security issues...nothing!

There were some instances when querying was done using a rudimentary excuse for a model layer. Other times, they would just use raw queries and just ignore the previous method. Sometimes the database calls were made in strange function calls inside randomly loaded PHP files from different folders from all over the place. Sometimes they used JOINs to get the data from multiple tables, sometimes they would do a bunch of single table queries and just loop every data set to format it using multiple for loops.

And, best of all, there were some parts of the app that would just ignore any ideea of frameworks, conventions and all that and would be just a huge PHP file full of spagetti code just spalshed around, sometimes with no apparent logic to it. Queries, processing, HTML...everything crammed in one file...

The most amazing thing was that this code base somehow managed to function in production for more than 5 years and people actualy used it...

Imagine the reaction I got from the client the moment I said we should burn it to the ground and rebuild the whole thing from scratch...

Good thing my boss trusted me and backed me up (he is a great guy by the way) and we never had to go along with that Frankenstein monster...

A service had/has been logging hundreds of errors in the development environment and I reached out to the owning process mgr that the error was occurring and perhaps a good opportunity to log additional data to help troubleshoot the issue if the problem ever made its way to production. He responded saying the error was related to a new feature they weren't going to implement in the backing dev database (TL;DR), and they know it works in production (my spidey sense goes off).
They deployed the changes to production this morning and immediately starting throwing errors (same error I sent)
Mgr messaged me a little while ago "Did you make any changes to the documentation service? We're getting this error .."

50% sure someone misspelled something in a config, but only thing they are logging is 'Unable to parse document'. Nothing that indicates an issue with the service they're using.

Are you even a dev if you haven't dropped a production database at least once?

Some Project Manager outsourced a redundant RADIUS setup with MySQL backend. We got 2 copies of a daloradius appliance running on Ubuntu 10.04. Once I saw this, I started to get a bit suspicious and requested to audit the system and database redundancy. With the system in production, and without getting back any documentation, I got into the VMs using the default root password. This was not even the worst part, as I found. One server was using a local MySQL instance, while the other was also using the first one's MySQL instance. When I reported this, I was told to comment clearly any changes to the configuration files, which resulted in commenting the word SHAME above each change.

Beware: Here lies a cautionary tale about shared hosting, backups, and -goes without saying- WordPress.

1. Got a call from a client saying their site presented an issue with a third-party add-on. The vendor asked us to grant him access to our staging copy.

2. Their staging copy, apparently, never got duplicated correctly because, for security reasons, their in-house dev changed the name of the wp-content folder. That broke their staging algo. So no staging site.

3. In order to recreate the staging site, we had to reset everything back to WP defaults. Including, for some reason, absolute paths inside the database. A huge fucking database. Because WordPress.

4. Made the changes directly in a downloaded sql file. Shared hosting, obviously, had an upload limit smaller to the actual database.

5. Spent half an hour trying to upload table by table to no avail.

6. In-house uploads a new, fixed database with the help of the shared hosting provider.

7. Database has the wrong path. Again.

8. In-house performs massive Find and Replace through phpMyAdmin on the production server.

9. Obviously, MySQL crashes instantly and the site gets blocked for over 3 hours for exceeding shared hosting limits.

10. Hosting provider refuses to accept this was caused by such a stupid act and says site needs to be checked because queries are too slow.

11. We are gouging our eyeballs as we see an in-house vs. hosting fight unfold. So we decide to watch a whole Netflix documentary in between.

12. Finally, the hosting folds and enables access to the site, which is obvi not working because, you know, wrong paths.

13. Documentary finishes. We log in again, click restore from backup. Go to bed. Client phones to bless us. Client’s in-house dev probably looking for a cardboard box to pack his stuff first thing in the morning. \_(ツ)_/¯

Day 0: thank you for being an Amazon Customer, your database is about to be upgrade in the near future with or without your consent! Tough titties motherfucker!

Day 16: ok, every upgraded by hand in the test environment, everything seems stable, let's go make preparations for production!

Day 16.5: ssh user@<prod_bastion_ip> --yada --yada
Unable to connect

Oooook, let's try again,
Unable to connect

Day 16.5.1: WHY THE FUCK NOT, the IP is fucking right, the cert is right, the user is right, the..... fucking.... EC2 instance has been......... terminated.....

FML!
---

Why! why can't people leave things alone.

Excuse me while I hit the bourbon 🥃

Of course the shouting episodes all happened during the era I was doing WordPress dev.

So we were a team of consultants working on this elephant-traffic website. There were a couple of systems for managing content on a more modular level, the "best" being one dubbed MF, a spaghettified monstrosity that the 2 people who joined before me had developed.

We were about to launch that shit into production, so I was watching their AWS account, being the only dev who had operational experience (and not afraid to wipe out that macos piece of shit and dev on a real os).

Anyhow, we enable the thing, and the average number of queries per page load instantly jumps from ~30 (even vanilla WP is horrible) to 1000+. Instances are overloaded and the ASG group goes up from 4 to 22. That just moves the problem elsewhere as now the database server is overwhelmed.

Me: we have to enable database caching for this thing *NOW*

Shitty authors of the monstrosity (SAM): no, our code cannot be responsible for that, it's the platform that can't handle the transition.

Me: we literally flipped a single switch here and look at the jump in all these graphs.

SAM: nono, it's fine, just add more instances

Me: ARE YOU FUCKIN SERIOUS?
Me: - goes and enables database caching without any approvals to do so, explaining to mgmt. that failure to do so would impair business revenue due to huge loading times, so they have to live with some data staleness -

SAM: Noooo, we'll show you it's not our code.
SAM: - pushes a new release of the monstrosity that makes DB queries go above 2k / page load -

...

Tho on the bright side, from that point on I focused exclusively on performance, was building a nice fragment caching framework which made the site fly regardless of what shitty code was powering it, tuned the stack to no end and learned a ton of stuff in the process which allowed me to graduate from the tar pit of WP development.

Living on the edge!

One or two years ago I managed to deploy a DDL change directly on the production server. As I knew there was a backup job which will run every day at noon and at midnight. So I run my script some minutes after noon. So far so good. But somehow I tested it badly in my test environment and the UI of the application throws error after error now in production.

Well, just revert the db to the latest recovery point with the backup, I thought.

It became clear then after a couple of minutes of searching the backup folder for the db backup that there was no such file. The youngest backup file was 3 years old.

Now what happened: The backup script had a switch "simulate=true" and then simulated a successful backup on each run. Therefore the monitoring system got no alerts for not correctly executing those jobs correctly. Then the monitoring job which should do the backupfolder surveillance stuck with green, because there was a valid backup file inside. But it did not check for a specific creation date.

Now this database is the one we need for doing our daily business and is really crucial. Therefore It was easier to emergencyfix the application than doing a rollback of the db 🙄

Well, not really a data loss story, but close to one.

Our production database has tables that exist only to be recompiled into a single table. Like, one record across 4 tables. Each record is only for one other record in the other 3 tables.

I LOST 2 HOURS DEBUGGING IT BECAUSE IT COULD NOT RECOMPILE THIS FUCKING TABLE. FUCK!

I was cleaning up dangling images in docker, and I accidentally removed the production database container as well.

Its not a big issue, I can just up the container back and everything should be fine. But after I up the container and connected to the database, I found out there's no data inside. I thought I fucked up, and sent msg in slack channel that I nuked the db.

Later my friend asked me which compose file I am using and that's when I realized I used the wrong config to up the db. Used the correct config to up the database again and everything goes back to normal.

It's friday evening and if I really dropped the db it would be fucking bad weekend....

I run update without where on mysql console on production database Today.

CLASSIC

Just because I needed to fix database after bug fix on the backend of the application.
I thought I wrote good sql statement after executing it on my local machine and then everything got bad.

Luckily it was only one column with some cached statistics data and I checked that it was not important data before I actually started fixing stuff but still ...
Almost got hard attack afterwards.
Made a script to fix this column and it took me only 15 minutes but still...

Bug was caused in part I got no unit tests and application grow after 3 years of development from simple one for one customer and volumes of documents around 50k to over 40 customers and volumes over 2mil per month, don’t know how many pages each, just in one year after we completed all needed features.

I have daily backups and logs of every api operation but still.
I think this got to far for one backend developer.

I got scared that I will loose money cause I am contractor and the only backend developer working on it.
I am so tired of this right now I think I need a break from work.

Responsibility is killing me so hard right now.
It will take a week to get back to normal.

life is short.we might not experience the next sunrise.

drop that production database today.

Sometimes for personal projects (and one client gig) I use the same database for local development and production.

Why?

Because I am a piece of shit.

# Retrospective as Backend engineer

Once upon a time, I was rejected by a startup who tries to snag me from another company that I was working with.
They are looking for Senior / Supervisor level backend engineer and my profile looks like a fit for them.
So they contacted me, arranged a technical test, system design test, and interview with their lead backend engineer who also happens to be co-founder of the startup.

## The Interview

As usual, they asked me what are my contribution to previous workplace.
I answered them with achievements that I think are the best for each company that I worked with, and how to technologically achieve them.

One of it includes designing and implementing a `CQRS+ES` system in the backend.
With complete capability of what I `brag` as `Time Machine` through replaying event.

## The Rejection

And of course I was rejected by the startup, maybe specifically by the co-founder. As I asked around on the reason of rejection from an insider.

They insisted I am a guy who overengineer thing that are not needed, by doing `CQRS+ES`, and only suitable for RND, non-production stuffs.

Nobody needs that kind of `Time Machine`.

## Ironically

After switching jobs (to another company), becoming fullstack developer, learning about react and redux.
I can reflect back on this past experience and say this:

The same company that says `CQRS+ES` is an over engineering, also uses `React+Redux`.
Never did they realize the concept behind `React+Redux` is very similar to `CQRS+ES`.

- Separation of concern
- CQRS: `Command` is separated from `Query`
- Redux: Side effect / `Action` in `Thunk` separated from the presentation
- Managing State of Application
- ES: Through sequence of `Event` produced by `Command`
- Redux: Through action data produced / dispatched by `Action`
- Replayability
- ES: Through replaying `Event` into the `Applier`
- Redux: Through replay `Action` which trigger dispatch to `Reducer`

---

The same company that says `CQRS` is an over engineering also uses `ElasticSearch+MySQL`.
Never did they realize they are separating `WRITE` database into `MySQL` as their `Single Source Of Truth`, and `READ` database into `ElasticSearch` is also inline with `CQRS` principle.

## Value as Backend Engineer

It's a sad days as Backend Engineer these days. At least in the country I live in.
Seems like being a backend engineer is often under-appreciated.
Company (or people) seems to think of backend engineer is the guy who ONLY makes `CRUD` API endpoint to database.

- I've heard from Fullstack engineer who comes from React background complains about Backend engineers have it easy by only doing CRUD without having to worry about application.
- The same guy fails when given task in Backend to make a simple round-robin ticketing system.
- I've seen company who only hires Fullstack engineer with strong Frontend experience, fails to have basic understanding of how SQL Transaction and Connection Pool works.
- I've seen company Fullstack engineer relies on ORM to do super complex query instead of writing proper SQL, and prefer to translate SQL into ORM query language.
- I've seen company Fullstack engineer with strong React background brags about Uncle Bob clean code but fail to know on how to do basic dependency injection.
- I've heard company who made webapp criticize my way of handling `session` through http secure cookie. Saying it's a bad practice and better to use local storage. Despite my argument of `secure` in the cookie and ability to control cookie via backend.

The moment I accidentally drop the entire Database in production with no backup.

If having a coffee in the morning doesn't wake you up, try deleting a collection or table in a production database instead.

Tonight, I gave birth to my first fully functional and deployment-ready dockerized application, "lestadium_web_1"!!
The big baby contains a Laravel showcase website with some React already in production! Big bonus with that, it's connected to a database, and I managed to setup some environment variables so nothing too dangerous is built within the image!

Fuck, that was exhausting but I'm so happy to finally understand how to make my stuff work, how they work and how to find some examples to get inspired from 😍

Other team lead: Hi DevOps Team, We need you to deploy this app to production. It's maintainers gave up on it in 2019, but we looked at it and it feels right.

Me: Uhm. That's not going to work. It'll fail the security scan before you can even finish the build in CI.

Other team lead: Yeah, this app is the right thing to do, and we needed it last week, but since that won't work, we'll just use this other very very infant technology that was just born yesterday. It's not stable in production, or on MySQL, or in AWS at all, but it's the other direction we can to go.

Me: What problem are you trying to solve in the first place?

Other team lead: Oh, we need access to the read from the production database.

Database is being slow AF again. Team lead is investigating. This is happening more often lately and affects both production and dev because everything is just in one gigantic database. So clients are calling support being angry about the speed they get and us devs get to twiddle our thumbs while waiting for our own data to load.

Imagine accidentally deleting the entire production database today and people thought you were joking.

One of our projects migrated their file-repository to another one during a major release.

Instead of giving this task to an experienced programmer, they gave it to the head of the respective dev department due to the usual release panic.

Soo.... He wrote the migration tool. It was executed during the release. Everything seemed fine so far.

A few days later. Someone from the above project came to my team due to some "strange behaviour on the production database".
They reported that they couldn't download some of the user's documents due to unknown reasons.

After quickly analyzing the current state of the new file-repository, we concluded that the affected documents did not exist in the new repository.

Then we took a look at the so called migration tool...
Well.. After nearly 30 min. we knew the root cause for that.
They only migrated the first 4 levels of the folder structure. Due to the assumption that "we don't use deeper nesting". (Facepalm)

As the head of their department wrote it, no one seems to questioned it either. Nor did they made a code review and ended up with a tool with hard coded urls to the production db, no version control, no build tool, no ci, nothing. Breaking nearly every possible company standard.

However.. That's not it. When analyzing their migration tool we noticed another even more dangerous thing.
They mixed up the id generation of the migrated documents resulting in a random assignment between customers and documents. Which is quite bad as this contains sensitive information. E.g. passports

They offered us quite a nice amount of money to fix this until EOB. We declinded as it was simply not possible in that time, but agreed to support them with the new tool.

After some time I heard that they migrated production again. And they fucked it up again. They never talked to us after we offered them support...

The third and final migration was written by us. Not only migrated it correctly. It was also way faster. By factor 20.

In the end we haven't gained anything from this rushed project as the penalties were piling up due to this fucked up migration.
After all this time I'm not sure who is to blame. In my opinion, partly all of them.
Head of department who can't and shouldn't code.
Seniors who didn't review the code and didn't ask for help.
Release mgmt who put way too much pressure on the devs.

Shoutout to all the devops people currently on their parent's WiFi trying to fix a database issue in production

Forgetting to write that WHERE statement, on a production database, during a serious case of Schroedinger's backup system, that failed.

....
oops

I'm a fullstack engineer, this period there is literally nothing to do, we are a 1000+ employees company.
I got so bored I toke over the database of our production server two times in a week, exploiting dumb vulnerabilities I discovered out of boredom, of course I reported everything.

The funny thing is that they just don't care, no one took action or is willing to fix it and they actually insulted me because I set a query in sleep for 8 minutes exploiting one of the vulnerabilities.

I work for a great company that hosts (in this very server) most italian citizens informations C: free to take for everyone c:

Comsci teacher just ran "php artisan migrate:fresh" on the production database

Bad: Delete your production database
Good: Have a backup
Bad: Can't reimport it because your backup procedure uses scheme that are no longer supported for import by your cloud provider
Good: Backup are plaintext and somehow easy to parse
Bad: Spending the rest of the day writing scripts to reinsert everything.

End of the story: everything is up and running, 8hours of efforts

A couple of years ago I was working on a fairly large system with a complex (by necessity) access control architecture.

As is usually the case with those projects, it's awkward for developers to repro bugs that have to do with a user's accesses in production when we are not allowed to replicate production data in test, let alone locally.

We had a bug where I ended up making myself a new row in the production database for a thing I could have access to without affecting real data to repro it safely. I identified the bug so I could repro it in dev/test and removed the row and ensured everything worked normally, whew scary.

Have you ever walked into the office one day, and everyone is hunched over in a semicircle around one person's workstation, before one turns around to look at you and says - after a pause - "... ltlian?.."

Turns out I had basically "poisoned the well" with my dummy entity in a way where production now threw 500 for everyone BUT me who had transitive access to this post-non-entity. Due to the scope of the system, it had taken about a day for this to gradually propagate in terms of caching and eventual consistencies; new entities coming in was expected, but not that they disappear.

Luckily I had a decent track record for this to be a one-off. I sometimes think about how I would explain testing in prod and making it faceplant before going home for the day, other than "I assumed it would be fine". I would fire me.

I had rough week last week. Accidentally deployed dev code to production, soon found out there was no production version of code to do a redeploy. Deployed another app to production, it was working fine then another Dev changed a data type in the database from bool to nullable Boolean which broke some Linq queries. Looked like I deployed crap code. 3rd week with company.

I was Just college fresher who completed his Engineering. My first week in the office. And a system was provided to me, since it was support project so I was given direct access to production database.

Fresher + Production Database + Access of Admin credentials = Worst Possible Combination

So it was my night shift, I was told to update new tariff plan for our client (which was one of the largest telecom service in India) .

If someone recharges for more than 200 Rupee, that person will get 10% or 20% extra talk time. Which was only applicable for particular circle (Like Bihar and Rajasthan).

Since I was fresher, I was told to update given query from my senior employee which he shared on the shared folder. Production downtime was in the mid night, so at that time I updated that query on the production database.

Query successfully updated. I completed my night shift, went home and slept.

When I woke up, I saw my mobile it had 200+ missed calls from different locations of India. They were Circle heads of that telecom service provider who contacted me. I realized something unexpected is expecting me.

Then at that moment my team lead called me and he asked me to come office right away.

Reminding you I was a fresher, I was shivering. What have I done there?

When I reached office, I came to know that the query I updated on production bombarded.

Every person who recharged that day (duration from midnight to morning 10 AM) got 10 times or 20 times more talktime.

A part of Query was something like this where error was made:

TalkTime = RechargeAmount + RechargeAmount * 10/100; (Bihar)
or

TalkTime = RechargeAmount + RechargeAmount * 20/100; (Rajasthan)
But instead of this query, I updated below one:

TalkTime = RechargeAmount + RechargeAmount * 10;
or

TalkTime = RechargeAmount + RechargeAmount * 20;
In a span of 10 hours, that telecom service lost revenue of 6.5 crore Rupees. Thanks to recovery team they were able to recover 6 crore but still 50 lakh Rupees were in loss.

One small query, and approx 1 million dollar was on stake.

Aftermath of this incident

My Mistake:

I should have taken those queries on mail. Or, there should have been mail communication regarding this.
Never ever do anything over oral communication. Senior employee who did this denied and said he provided correct query, and I had no proof of communication.
I told them, it was me who executed that query on production. Since I was fresher, and took my responsibility of that incident. My team lead rescued me from that situation.
Lesson Learned:

Always test your query and code multiple times before you execute or Go live it on production.
Always have email communication for every action you take on production.
Power comes with responsibility. If you have admin credentials of production never use it for update/delete/drop until you are sure.
Don’t take your job lightly.
I was not fired from that Job, but I have learnt my lesson very well.

> Drops production database.
> Wait for chaos to ensue.
“It’s just a prank, bro”

Weeks ago, a change went into production. For some reason, we can't implement our own changes or create new databases in production, we have to have a whole different department do it. This would be great except for one thing:

THEY CAN'T THINK FOR THEMSELVES. I've had to tell them how to run scripts I wrote. I've had to tell them how to fix problems that arise.

Back to that script ran three weeks ago or so. It didn't add permissions to allow me, the system and application developer to see the stored procedure, much less run it. Application can't run it. Thankfully the application works without it.

Fast forward to tonight. My change that I'm attempting to implement is the creation of the stored procedure, because nothing could see it, I assumed it didn't exist... reasonable, right? Database folks tells me it exists. They then tell me they can't give me nor the application permissions because it doesn't ask for it in the change plan.

Excuse me.... WHAT FUCKING WORLD DOES IT MAKE SENSE TO CREATE SOMETHING AND HIDE IT FROM THE CREATOR LET ALONE THE APPLICATION SO IT CAN'T USE IT?! FUCKING THINK. WHY WOULD I WASTE MY FUCKING TIME TO TALK TO YOU OFFSHORE PIECES OF SHIT AT 10PM WHEN I'D RATHER PLAY VIDEO GAMES.

I'm so fucking done with enterprises. Someone with reasonable job security at a startup, please hire me. You will probably pay me more fucking money than this company does anyway.

Now on to my second change of the night. Thankfully I don't have to rely on anyone outside of me... so I won't be wasting my fucking time.

So it's Friday afternoon just before a bank holiday weekend here I'm the UK, perfect time for our production database to go TITSUP (total inability to support usual performance), life sucks then you die folks....

At what point do you stop optimizing queries and realize it's a database architecture, scaling problem?

We've been having production issues this week because a lot more users with more demands, and I'm going we need more servers... We can't just have one db, we need to parallelize like Hadoop...

Everyone else is going, how do we optimize queries, indexes, reduce the load...

What's the most difficult english word to spell in your opinion?

Mine is "length". Keep forgetting if it's "th" or "ht" (but if you start to think about it is obvious).
And I bet not only mine since iit's written "lenght" in the production database at the company where I work ...

When your company expects you to manually change information in the production db by saying "hey, client Billy wants his stuff moved back to where it was"

I'm considering quitting a job I started a few weeks ago. I'll probably try to find other work first I suppose.

I'm UK based and this is the 6th programming/DevOps role I've had and I've never seen a team that is so utterly opposed to change. This is the largest company I've worked for in a full time capacity so someone please tell me if I'm going to see the same things at other companies of similar sizes (1000 employees). Or even tell me if I'm just being too opinionated and that I simply have different priorities than others I'm working with. The only upside so far is that at least 90% of the people I've been speaking to are very friendly and aren't outwardly toxic.

My first week, I explained during the daily stand up how I had been updating the readmes of a couple of code bases as I set them up locally, updated docker files to fix a few issues, made missing env files, and I didn't mention that I had also started a soon to be very long list of major problems in the code bases. 30 minutes later I get a call from the team lead saying he'd had complaints from another dev about the changes I'd spoke about making to their work. I was told to stash my changes for a few weeks at least and not to bother committing them.

Since then I've found out that even if I had wanted to, I wouldn't have been allowed to merge in my changes. Sprints are 2 weeks long, and are planned several sprints ahead. Trying to get any tickets planned in so far has been a brick wall, and it's clear management only cares about features.

Weirdly enough but not unsurprisingly I've heard loads of complaints about the slow turn around of the dev team to get out anything, be it bug fixes or features. It's weird because when I pointed out that there's currently no centralised logging or an error management platform like bugsnag, there was zero interest. I wrote a 4 page report on the benefits and how it would help the dev team to get away from fire fighting and these hidden issues they keep running into. But I was told that it would have to be planned for next year's work, as this year everything is already planned and there's no space in the budget for the roughly $20 a month a standard bugsnag plan would take.

The reason I even had time to write up such a report is because I get given work that takes 30 minutes and I'm seemingly expected to take several days to do it. I tried asking for more work at the start but I could tell the lead was busy and was frankly just annoyed that he was having to find me work within the narrow confines of what's planned for the sprint.

So I tried to keep busy with a load of code reviews and writing reports on road mapping out how we could improve various things. It's still not much to do though. And hey when I brought up actually implementing psr12 coding standards, there currently aren't any standards and the code bases even use a mix of spaces and tab indentation in the same file, I seemingly got a positive impression at the only senior developer meeting I've been to so far. However when I wrote up a confluence doc on setting up psr12 code sniffing in the various IDEs everyone uses, and mentioned it in a daily stand up, I once again got kickback and a talking to.

It's pretty clear that they'd like me to sit down, do my assigned work, and otherwise try to look busy. While continuing with their terrible practices.

After today I think I'll have to stop trying to do code reviews too as it's clear they don't actually want code to be reviewed. A junior dev who only started writing code last year had written probably the single worst pull request I've ever seen. However it's still a perfectly reasonable thing, they're junior and that's what code reviews are for. So I went through file by file and gently suggested a cleaner or safer way to achieve things, or in a couple of the worst cases I suggested that they bring up a refactor ticket to be made as the code base was trapping them in shocking practices. I'm talking html in strings being concatenated in a class. Database migrations that use hard coded IDs from production data. Database queries that again quote arbitrary production IDs. A mix of tabs and spaces in the same file. Indentation being way off. Etc, the list goes on.
Well of course I get massive kickback from that too, not just from the team lead who they complained to but the junior was incredibly rude and basically told me to shut up because this was how it was done in this code base. For the last 2 days it's been a bit of a back and forth of me at least trying to get the guy to fix the formatting issues, and my lead has messaged me multiple times asking if it can go through code review to QA yet. I don't know why they even bother with code reviews at this point.

Haven't ranted for awhile but here it goes...
In a meeting with a front end user yesterday. They don't like the entry screens on our Oracle ERP system. They want us to provide them with a tool so they can create new entry screens to replace those they dont like. They want full autonomy over that tool and no interference from IT. Oh, and they want unfettered database access to the production data, including full ability to execute DML. I so wanted to say 'Are you high?'.

MySQL is freaking and keeps restarting :(

Production server "Error establishing database connection"

FUCKKKKKKKKKKKKKKKK

We are 3-4 days away from deployment to production. We are still bug fixing. But one coworkers decided this is the time to make a fuss about the way everything is set up. He doesn't like the dev database. So he knocks it over.. and while so doing it, he doesn't inform the team. And when I ask if something else is gonna knock over? No answer! (And something broke down too..)
Now we have issues to test our bugfixes. The whole thing took me half a day finding out and made me distracted with frustration, and not just for me. Most bugs could've been done in that half a day!
I so wanna punch the guy xD but no, I gotta save face, pfff!

During the "how to install WordPress" the professor tells us to use the root user of the database.... Wooooooo yeah! Let's get fucked!

And he never even notes to them "DON'T use root in production!"

I can just imagine every one of the 40 people in attendance that day wondering how their website could POSSIBLY get hacked...

And they are are going to entering the industry, some of them as freelancers from the onset, thinking all that is ok.

If having a coffee in the morning doesn't wake you up, try deleting a table in a production database instead.

Until today, I had assumed deploying stuff to prod would NOT be one of my responsabilities in this company. Apparently that's not the case.

Had to deploy my code and pray it didn't break anything. Why is this a big deal at all?

Well you see, there is no repository. At all. No git, no svn, not even duplicate folders. No tests, no pipeline. Just a bunch of CPanels.

Had to manually copy files and folders from the development site to the production site and partially copy a database. "Just drag and drop" were the instructions I was given.

As if using CakePHP2, PHP5 and having to parse fucking Excel files wasn't bad enough, now I have to deal with one of the worst ways to deploy code.

Fuck it, I'm switching on the looking-for-job flag on linkedin.

If I had a penny for every time I have had a thought about deleting the production database.....

Ok so some of you have probably seen my previous rants about my computer science teacher and our project but I'm just going to summarize all of them and share with you more of my pain.

1. He edits in the production environment. Its a laravel project and he is creating test database migrations IN THE PRODUCTION ENVIRONMENT AND SWITCHING BACK AND FORTH FROM MASTER AND DEV.

2. He edits in vim and doesn't follow codestyle even though I printed him off a piece of paper and emailed it to him.

3. He doesn't have any ethic when it comes to more complex things like laravel homestead.

4. He doesnt want me to release features even though he takes really long to do them.

While I love vim and it is my editor of choice, some things should be done in an ide. This is really annoying me and I'm really just considering handing him the project if he can't follow basic outline.

My client (not an engineer) has asked me access to production database and keeps asking an intern to directly query data from it instead of using the API.

We use at our company one of the largest Python ORM and dont code ourselfs on it, event tough I can code. Its some special contract which our General Manager made, before we as Devs where in the Project and everything is provided from the external Company as Service. The Servers are in our own Datacenter, but we dont have access.

We have our Consultants (Project Manager) as payd hires and they got their own Devs.

Im in lead of Code Reviews and Interfaces. Also Im in the "Run" Team, which observes, debuggs and keeps the System alive as 3rd-Level (Application Managers).

What Im trying to achieve is going away from legacy .csv/sftp connections to RestAPI and on large Datasets GraphQL. Before I was on the Project, they build really crappy Interfaces.

Before I joined the Project in my Company, I was a Dev for a couple of Finance Applications and Webservices, where I also did coding on Business critical Applications with high demand Scaling.

So forth, I was moved by my Boss over to the Project because it wasn't doing so well and they needed our own Devs on it.

Alot of Issues/Mistakes I identified in the Software:
- Lots of Code Bugs
- Missing Process Logic
- No Lifecycle
- Very fast growing Database
- A lot of Bad Practices

Since my switch I fixed alot of bugs, was the man of the hour for fixing major Incidents and so on so forth. A lot of improvements have been made. Also the Team Spirit of 15+ People inside the Project became better, because they could consult me for solutions/problems.

But damn I hate our Consultants. We pay them and I need to sketch the concepts, they are to dumb for it. They dont understand Rest or APIs in general, I need to teach them alot about Best Practices and how to Code an API. Then they question everything and bring out a crooked flawed prototype back to me.

WE F* PAY THEM FOR BULLCRAP! THEY DONT EVEN WRITE DOCUMENTATION, THEY ARE SO LAZY!

I even had a Meeting with the main Consultant about Performance Problems and how we should approach it from a technical side and Process side. The Software is Core Business relevant and its running over 3 Years. He just argumented around the Problem and didnt provide solutions.

I confronted our General Manager a couple of times with this, but since 3 Years its going on and on.

Im happy with my Team and Boss, they have my back and I love my Job, but dealing with these Nutjobs of Consultants is draining my nerves/energy.

Im really am at my wits end how to deal with this anymore? Been pulling trough since 1 year. I wanna stay at my company because everything else besides the Nutjob Consultants is great.

I told my Boss about it a couple of times and she agrees with me, but the General Manager doesnt let go of these Consultants.

Even when they fuck up hard and crash production, they fucking Bill us... It's their fault :(

A prototype being used as production code written in procedural PHP with the code drawn using echo and MySQL (not MySQLi) all mixed together and the configuration with world readable database stored as config.inc.
All backed by a database with no foreign keys or data integrity of any kind.

Mine: not listening to early warning sings that shit won't work. As a result we implemented the wrong technology and had to roll back after a month in production. Lost some customers data, though we had backups, so it was just a delta of a day or so. Still, not pretty, had a rough night writing scripts at 3am to check data validity after.

I will throw in another one from a colleague of mine. He was running some database migrations and ended up pointing the DB migration tool to production by accident. Oops. That one required a restore as well, if I remember correctly.

Oooh I have quite a few,
My favourite: accidently left a log. Debug("bollocks") in a try catch this made it through testing and does (still) occasionally go into production log files.

Worst: wrote an interceptor for jboss with the intent of checking cache for some lookup data. I picked the wrong one of two similarly named methods and instead queried the database, I effectively wrote a denial of service utility into our app

Lol I remember deleting half a production database 6 months into this job. Now a little under 2 years my boss lets me do whatever the fuck I want as long as my quality of work stays high and I complete a regular number of jira tasks a week

I can set up website in production doing both backend w database and front end with js csd html. But my design skills are not great. I prefer being behind the scenes but am i selling myself short? Am i really a full stack developer who just stinks at css?

This one was from my Tech Lead, She had to update phone number of a customer in Database on production server. And guess what she forgot 'WHERE'. Next we were facing each other with poker face. :|

p.s: fortunately we had backup of just 4 hours back. Still we lost data of about 100 people.

Developed an update to our database procedures and tested it with local copy. After a few days everything was ready. Opened our server and started the update. After a couple of rows an error occurred. Turns out our production db is older version and does not support some syntax I used. It became a bit longer day at work...

Quickly delete a double record in the production database with a script, just forgot the where statement...

When the CTO/CEO of your "startup" is always AFK and it takes weeks to get anything approved by them (or even secure a meeting with them) and they have almost-exclusive access to production and the admin account for all third party services.

Want to create a new messaging channel? Too bad! What about a new repository for that cool idea you had, or that new microservice you're expected to build. Expect to be blocked for at least a week.

When they also hold themselves solely responsible for security and operations, they've built their own proprietary framework that handles all the authentication, database models and microservice communications.

Speaking of which, there's more than six microservices per developer!

Oh there's a bug or limitation in the framework? Too bad. It's a black box that nobody else in the company can touch. Good luck with the two week lead time on getting anything changed there. Oh and there's no dedicated issue tracker. Have you heard of email?

When the systems and processes in place were designed for "consistency" and "scalability" in mind you can be certain that everything is consistently broken at scale. Each microservice offers:

1. Anemic & non-idempotent CRUD APIs (Can't believe it's not a Database Table™) because the consumer should do all the work.
2. Race Conditions, because transactions are "not portable" (but not to worry, all the code is written as if it were running single threaded on a single machine).
3. Fault Intolerance, just a single failure in a chain of layered microservice calls will leave the requested operation in a partially applied and corrupted state. Ger ready for manual intervention.
4. Completely Redundant Documentation, our web documentation is automatically generated and is always of the form //[FieldName] of the [ObjectName].
5. Happy Path Support, only the intended use cases and fields work, we added a bunch of others because YouAreGoingToNeedIt™ but it won't work when you do need it. The only record of this happy path is the code itself.

Consider this, you're been building a new microservice, you've carefully followed all the unwritten highly specific technical implementation standards enforced by the CTO/CEO (that your aware of). You've decided to write some unit tests, well um.. didn't you know? There's nothing scalable and consistent about running the system locally! That's not built-in to the framework. So just use curl to test your service whilst it is deployed or connected to the development environment. Then you can open a PR and once it has been approved it will be included in the next full deployment (at least a week later).

Most new 'services' feel like the are about one to five days of writing straightforward code followed by weeks to months of integration hell, testing and blocked dependencies.

When confronted/advised about these issues the response from the CTO/CEO
varies:

(A) "yes but it's an edge case, the cloud is highly available and reliable, our software doesn't crash frequently".
(B) "yes, that's why I'm thinking about adding [idempotency] to the framework to address that when I'm not so busy" two weeks go by...
(C) "yes, but we are still doing better than all of our competitors".
(D) "oh, but you can just [highly specific sequence of undocumented steps, that probably won't work when you try it].
(E) "yes, let's setup a meeting to go through this in more detail" *doesn't show up to the meeting*.
(F) "oh, but our customers are really happy with our level of [Documentation]".

Sometimes it can feel like a bit of a cult, as all of the project managers (and some of the developers) see the CTO/CEO as a sort of 'programming god' because they are never blocked on anything they work on, they're able to bypass all the limitations and obstacles they've placed in front of the 'ordinary' developers.

There's been several instances where the CTO/CEO will suddenly make widespread changes to the codebase (to enforce some 'standard') without having to go through the same review process as everybody else, these changes will usually break something like the automatic build process or something in the dev environment and its up to the developers to pick up the pieces. I think developers find it intimidating to identify issues in the CTO/CEO's code because it's implicitly defined due to their status as the "gold standard".

It's certainly frustrating but I hope this story serves as a bit of a foil to those who wish they had a more technical CTO/CEO in their organisation. Does anybody else have a similar experience or is this situation an absolute one of a kind?

Spent all morning debugging legacy code that I need to migrate.

Most of the time is just waiting for it to load --pieces of data-- entire tables from the database and then filter out the records it doesn't want using some app logic.

WHAT SORT OF MONKEY WRITES CODE LIKE THIS? HOW WAS THIS EVEN ALLOWED INTO PRODUCTION...

I have to open Notepad to write down my chain of thoughts, steps, and things to check once the next breakpoints are hit so that I don't forget them.

So in theory I'm being paid all morning to sit around and do nothing.

That sounds great but I'm falling asleep... Shoulda worked from home...

What was I saying again...yea...

DON'T HIRE MONKEYS!!! THEY WRITE SHIT CODE THAT WASTES EVERYONE'S TIME EVEN AFTER THEY LEAVE...

I'm going to lunch now... Hopefully Notepad has enough into for me to remember what I was doing...

Once I maintained one of the most used and fucked up codebases on the market with almost 1M+ daily users. (cannot say more, sorry).

It's written in PHP and is absolutely terrifying,
the first time I saw some lines of code I was about to scream and cry.

- spaghetti code
- no indentation
- random SQL query unoptimized
- unused vars
- Code is split among several files with no logical reasoning
- Mixed procedural and oop programming
- Unsanitised user input (yes, you got it right)

No test environment, no backup database, every commit goes straight to production.

It's a real disaster but the company prefers to keep it as it is without refactoring or anything else.

Just to make it clear:
It's not hatred against PHP, it's against the code's current status and the older programmers which used to work on it.

delete random, unused column from production database
entire app blows up
re-release last change on code pipeline
everything works beautifully
¯\_(ツ)_/¯

I'm in a team of 3 in a small to medium sized company (over 50 engineers). We all work as full stack engineers.. but I think the definition of full stack here is getting super bloated. Let me give u an example. My team hold a few production apps, and we just launched a new one. The whole team (the 3 of us) are fully responsible on it from planning, design, database model, api, frontend (a react page spa), an extra client. Ok, so all this seems normal to a full stack dev.

Now, we also handle provisioning infra in aws using terraform, doing deployments, building a CI/CD pipeline using jenkins, monitoring, writing tests, building an analytics dashboard.

Recently our tech writer also left, so now we are also handling writing feature releases.

Few days ago, we also had a meeting where they sort of discussed that the maintenance of the engineering shared services, e.g. jenkins servers, (and about 2-3 other services) will now be split between teams in a shared board, previously this was handled only be team leads, but now they want to delegate it down.

And ofcourse not to mention supporting the app itself and updating bug tickets with findings.

I feel like my daily responsiblities are becoming the job responsibilities of at least 3 jobs.

Is this what full stack engineering looks like in your company? Do u handle everything from app design, building, cloud, ops, analytics etc..

5am now and trying to do a production deployment. It just failed because boss loves making un-approved changes to prod that change entire database schemas. Argh!!

Finally made my node production server stable enough that I could focus on writing tests*. I start by setting up docker, mocking cognito, preparing the database and everything. Reading up on Node test suites and following a short tut to set up my first unit test. Didn't go smoothly, but it's local and there are no deadlines so who cares. 4 days later, first assert.equal(1+1, 2) passes and I'm happy.

I start writing all sorts of tests, installing everything required into "devDependancies," and getting the joy of having some tests pass on first try with all asserts set up, feels good!

I decide to make a small update to production, so I add a test, run and see it fail, implement the feature, re-run and, it passes!

I push the feature to develop, test it, and it works as intended. Merge that to master and subsequently to one of my ec2 production servers**, and lo and behold, production server is on a bootloop claiming it "Cannot find module `graphql`". But how? I didn't change any production dependencies, and my package lock json is committed so wth?

I google the issue, but can't find anything relevant. The only thing that I could guess was that some dependencies (including graphql) were referenced*** in both, prod and dev, and were omitted when installed on a prod NODE_ENV, but googling that specific issue yielded no results, and I would have thought npm would be clever enough to see that and would always install those dependencies (spoiler: it didn't for me).

With reduced production capacity (having one server down) I decided to npm uninstall all dev dependencies anyway and see what happens. Aaaaand it works.....

So now I have a working production server, but broken local tests, and I'm not sure why npm is behaving like this...

* Yes I see the irony.
** No staging because $$$, also this is a personal project.
*** I am not directly referencing the same thing twice, it's probably a subdependency somewhere.

Fuck I forgot to make a database export before I executed delete statements on a production database...

All went fine though. Nothing broke.