Oh sh*t

Today I broke a record 🎉

..in the production database 😅

Oh, man, I just realized I haven't ranted one of my best stories on here!

So, here goes!

A few years back the company I work for was contacted by an older client regarding a new project.

The guy was now pitching to build the website for the Parliament of another country (not gonna name it, NDAs and stuff), and was planning on outsourcing the development, as he had no team and he was only aiming on taking care of the client service/project management side of the project.

Out of principle (and also to preserve our mental integrity), we have purposely avoided working with government bodies of any kind, in any country, but he was a friend of our CEO and pleaded until we singed on board.

Now, the project itself was way bigger than we expected, as the wanted more of an internal CRM, centralized document archive, event management, internal planning, multiple interfaced, role based access restricted monster of an administration interface, complete with regular user website, also packed with all kind of features, dashboards and so on.

Long story short, a lot bigger than what we were expecting based on the initial brief.

The development period was hell. New features were coming in on a weekly basis. Already implemented functionality was constantly being changed or redefined. No requests we ever made about clarifications and/or materials or information were ever answered on time.

They also somehow bullied the guy that brought us the project into also including the data migration from the old website into the new one we were building and we somehow ended up having to extract meaningful, formatted, sanitized content parsing static HTML files and connecting them to download-able files (almost every page in the old website had files available to download) we needed to also include in a sane way.

Now, don't think the files were simple URL paths we can trace to a folder/file path, oh no!!! The links were some form of hash combination that had to be exploded and tested against some king of database relationship tables that only had hashed indexes relating to other tables, that also only had hashed indexes relating to some other tables that kept a database of the website pages HTML file naming. So what we had to do is identify the files based on a combination of hashed indexes and re-hashed HTML file names that in the end would give us a filename for a real file that we had to then search for inside a list of over 20 folders not related to one another.

So we did this. Created a script that processed the hell out of over 10000 HTML files, database entries and files and re-indexed and re-named all this shit into a meaningful database of sane data and well organized files.

So, with this we were nearing the finish line for the project, which by now exceeded the estimated time by over to times.

We test everything, retest it all again for good measure, pack everything up for deployment, simulate on a staging environment, give the final client access to the staging version, get them to accept that all requirements are met, finish writing the documentation for the codebase, write detailed deployment procedure, include some automation and testing tools also for good measure, recommend production setup, hardware specs, software versions, server side optimization like caching, load balancing and all that we could think would ever be useful, all with more documentation and instructions.

As the project was built on PHP/MySQL (as requested), we recommended a Linux environment for production. Oh, I forgot to tell you that over the development period they kept asking us to also include steps for Windows procedures along with our regular documentation. Was a bit strange, but we added it in there just so we can finish and close the damn project.

So, we send them all the above and go get drunk as fuck in celebration of getting rid of them once and for all...

Next day: hung over, I get to the office, open my laptop and see on new email. I only had the one new mail, so I open it to see what it's about.

Lo and behold! The fuckers over in the other country that called themselves "IT guys", and were the ones making all the changes and additions to our requirements, were not capable enough to follow step by step instructions in order to deploy the project on their servers!!!

[Continues in the comments]

Finally did it. Quit my job.

The full story:
Just came back from vacation to find out that pretty much all the work I put at place has been either destroyed by "temporary fixes" or wiped clean in favour of buggy older versions. The reason, and this is a direct quote "Ari left the code riddled with bugs prior to leaving".

Oh no. Oh no I did not you fucker.

Some background:

My boss wrote a piece of major software with another coder (over the course of month and a balf). This software was very fragile as its intention was to demo specific features we want to adopt for a version 2 of it.

I was then handed over this software (which was vanilajs with angular) and was told to "clean it up" introduce a typing system, introduce a build system, add webpack for better module and dependency management, learn cordova (because its essential and I had no idea of how it works). As well as fix the billion of issues with data storage in the software. Add a webgui and setup multiple databses for data exports from the app. Ensure that transmission of the data is clean and valid.

What else. This software had ZERO documentation. And I had to sit my boss for a solid 3hrs plus some occasional questions as I was developing to get a clear idea of whats going on.

Took a bit over 3 weeks. But I had the damn thing ported over. Cleaned up. And partially documented.

During this period, I was suppose to work with another 2 other coders "my team". But they were always pulled into other things by my Boss.

During this period, I kept asking for code reviews (as I was handling a very large code base on my own).

During this period, I was asking for help from my boss to make sure that the visual aspect of the software meets the requirements (there are LOTS of windows, screens, panels etc, which I just could not possibly get to checking on my own).

At the end of this period. I went on vacation (booked by my brothers for my bday <3 ).

I come back. My work is null. The Boss only looked at it on the friday night leading up to my return. And decided to go back to v1 and fix whatever he didnt like there.

So this guy calls me. Calls me on a friggin SUNDAY. I like just got off the plane. Was heading to dinner with my family.
He and another coder have basically nuked my work. And in an extremely hacky way tied some things together to sort of work. Moreever, the webguis that I setup for the database viewing. They were EDITED ON THE PRODUCTION SERVER without git tracking!!

So monday. I get bombarded with over 20 emails. Claiming that I left things in an usuable state with no documentation. As well as I get yelled at by my boss for introducing "unnecessary complicated shit".

For fuck sakes. I was the one to bring the word documentation into the vocabulary of this company. There are literally ZERO documentated projects here. While all of mine are at least partially documented (due to lack of time).

For fuck sakes, during my time here I have been basically begging to pull the coder who made the admin views for our software and clean up some of the views so that no one will ever have to touch any database directly.

To say this story is the only reason I am done is so not true.

I dedicated over a year to this company. During this time I saw aspects of this behaviour attacking other coders as well as me. But never to this level.

I am so friggin happy that I quit. Never gonna look back.

The project where I realized I wanted to go from chemist to pro dev.

I built a flow-chemistry spectrometer with monitoring backend in Haskell.

Spectroscopy is where you add a reagent to a glass tube, it changes color, and by measuring the exact color it tells you how much of something (for example, a toxin) is present in the sample.

I had to do that a lot on factory samples, writing down measurements using pen & paper.

I'm lazy so I decided to do the logical thing: Automate it. I bought a second hand spectrometer, stripped the casing, did a shitload of glassblowing and hooked up tubes to the production pipelines, so I could get samples, mixing them in the correct ratio with reagents in continuous flows using valves.

I ended up using 2 home-crafted arduino-like boards (etching PCBs is fun!).

One to calibrate the mixture against known samples and control solenoid valves to continuously cycle through various reagents and deionized flushing water, the other to record the measurements and send them to a server running a Haskell/Yesod API.

The server collected the information into InfluxDB (A time series database), displaying all data on a graphite dashboard.

Eventually I wrote Haskell plugins for most of the chemistry processes, from pH & temperature measurements to polymer property and pigment tests (they made a lot of printer ink).

Then I was fired because they didn't need chemists anymore, and the code "could be maintained by the intern" (poor guy)...

But I did find out that I loved functional programming, chemistry automation projects, and crafting my own electronics during that time.

A "support" guy my boss got in. I had told my boss numerous times, "Get rid of this guy, he's only wasting our time and money. And he's going to end up doing something where we will end up having to put out the fires."

Sure as a pair of nuts on a squirrel, this crazy bastard goes and DELETES a client's database. Yes folks, in fucking production. A live database. The heart of the business' transactions are... *poof*... GONE!!!

No backups for the day! No synchronisation beforehand! No nothing... just GONE!!! Fucking flat-lining!

Well, when I realised what he has done, I had to remove myself from the room before shit got outta hand!

I told the boss man that is the last straw and he needs to go...

The long and short of it...
- The client had luckily only lost about half a days data.
- I'm no longer at the company.
- This dumb fuck still is.

I worked on a greenfield project a couple of years ago. The company had an old solution written in Omnis (heard of it? Yeah, me neither) with an SQL database. My team was to create a completely new web based system... on top of the old database, so the customers could keep their existing stuff.

The dba was an intelligent man, one of the nicest people I've met, and over the course of fifteen years he had made a remarkably terrifying monstrosity of a database. Some years before me they wanted to "future proof" the system and make it "easier to switch to new technologies". So they moved the entire business logic into the database...

I used a tool to create a visualization of said database when we started. It had no views, only tables and sprocs. Look at it! Tables and sprocs are rectangles (well, dots) and any connections are drawn in grey lines. There were no foreign keys, so a tables only visualization only yielded a collection of independent rectangles without a single line.

Now, the stored procedures were bloody MASSIVE. A single procedure that only registered a new interested party and attached them to a property had 2500+ lines and over 150 parameters.

Also, this dba added features and fixed bugs by logging into the respective customers production server and writing SQL.

That database is the stupidest thing I've ever seen a developer do.

So lot of people ask me here what is Testing?

Well, I have decided to post about it instead of answering every request individually as it is tiresome. So here we go...

- Why to test?
Testing is breaking down a piece of code so that all the possible logical and functional errors can be detected while in development phase, so that the disaster can be prevented in production.

- How to test a piece of code/software?
There are various types of testing like unit, sanity/smoke, functional, regression, performance, security, penetration to name a few. Each has it's own significance and to know more you can Google/DDG them.

- What to test?
We can test any and everything. You can test code, UI, UX, speed of application, database flow, logical flow, functional flow, et cetera.

Testing is an attitude, not a skill. Cannot be taught but only developed by one ownself.

To be good at testing, one needs to have a good design and business sense, apart from technical and functional knowledge.

Many will say, to test one needs to think outside the box. Fuck that. There is no box. Everything is a sandbox and do whatever the fuck you want to, to break things.

Most importantly, thinking from other person's perspective is critical as this helps you think what can an user do with your software. Trust me, humans are idiots and can literally fuck your software doggy style and make you wonder why your code isn't working?

To build an idiot proof software, we need to think beyond human levels.

Never think the obvious. NEVER.

Always try what you think can never happen. System will break, I promise.

Finding loopholes is way easier than you think. Fixing them is surely a challenge.

Moreover, to survive in corporate world leaning the processes is important. The SDLC cycle is what makes a software great.

And today I realised that no role is stupid. Testing is looked down upon in my country. Possibly, IT is the only field where testing has least value as compared to other places like food testing or automobile crash testing et cetera.

And I, myself used to think support is a stupid job requiring no skills. But today I talked to a friend in support and realised that they have more technical skills than a tester.

Everyone is important and we shouldn't look down upon any person or role. Nobody is superior, nobody is inferior. We all are equal.

Together we shall work as a team and a great teamwork can achieve wonders.

Hope this helps. Thak you for reading and do provide your feedback as it shall help me improve.

P.S.: last week somebody tagged me to a new comer's post seeking advice for testing. I don't recollect that so please tag them here again. Thanks.

Edit: I feel like and I might have, missed out some points so please excuse me for that.

My biggest dev blunder. I haven't told a single soul about this, until now.

👻👻👻👻👻👻

So, I was working as a full stack dev at a small consulting company. By this time I had about 3 years of experience and started to get pretty comfortable with my tools and the systems I worked with.

I was the person in charge of a system dealing with interactions between people in different roles. Some of this data could be sensitive in nature and users had a legal right to have data permanently removed from our system. In this case it meant remoting into the production database server and manually issuing DELETE statements against the db. Ugh.

As soon as my brain finishes processing the request to venture into that binary minefield and perform rocket surgery on that cursed database my sympathetic nervous system goes into high alert, palms sweaty. Mom's spaghetti.

Alright. Let's do this the safe way. I write the statements needed and do a test run on my machine. Works like a charm 😎

Time to get this over with. I remote into the server. I paste the code into Microsoft SQL Server Management Studio. I read through the code again and again and again. It's solid. I hit run.

....

Wait. I ran it?

....

With the IDs from my local run?

...

I stare at the confirmation message: "Nice job dude, you just deleted some stuff. Cool. See ya. - Your old pal SQL Server".

What did I just delete? What ramifications will this have? Am I sweating? My life is over. Fuck! Think, think, think.

You're a professional. Handle it like one, goddammit.

I think about doing a rollback but the server dudes are even more incompetent than me and we'd lose all the transactions that occurred after my little slip. No, that won't fly.

I do the only sensible thing: I run the statements again with the correct IDs, disconnect my remote session, and BOTTLE THAT SHIT UP FOREVER.

I tell no one. The next few days I await some kind of bug report or maybe a SWAT team. Days pass. Nothing. My anxiety slowly dissipates. That fateful day fades into oblivion and I feel confident my secret will die with me. Cool ¯\_(ツ)_/¯

If drinking coffee doesn't wake you up in the morning, try deleting a table in the production database.

Verified, it worked.

5 beers in
Migrating a 200 GB database in production
What could go wrong?

I recently joined the dark side - an agile consulting company (why and how is a long story). The first client I was assigned to was an international bank. The client wanted a web portal, that was at its core, just a massive web form for their users to perform data entry.

My company pitched and won the project even though they didn't have a single developer on their bench. The entire project team (including myself) was fast tracked through interviews and hired very rapidly so that they could staff the project (a fact I found out months later).

Although I had ~8 years of systems programming experience, my entire web development experience amounted to 12 weeks (a part time web dev course) just before I got hired.

I introduce to you, my team ...

Scrum Master. 12 years experience on paper.
Rote memorised the agile manifesto and scrum textbooks. He constantly went “We should do X instead of (practical thing) Y, because X is the agile way.” Easily pressured by the client to include ridiculous (real time chat in a form filling webpage), and sometimes near impossible features (undo at the keystroke level). He would just nag at the devs until someone mumbled ‘yes' just so that he would stfu and go away.

UX Designer. 3 years experience on paper ... as business analyst.
Zero professional experience in UX. Can’t use design tools like AI / photoshop. All he has is 10 weeks of UX bootcamp and a massive chip on his shoulder. The client wanted a web form, he designed a monstrosity that included several custom components that just HAD to be put in, because UX. When we asked for clarification the reply was a usually condescending “you guys don’t understand UX, just do <insert unhandled edge case>, this is intended."

Developer - PHD in his first job.
Invents programming puzzles to solve where there are none. The user story asked for a upload file button. He implemented a queue system that made use of custom metadata to detect file extensions, file size, and other attributes, so that he could determine which file to synchronously upload first.

Developer - Bootlicker. 5 years experience on paper.
He tried to ingratiate himself with the management from day 1. He also writes code I would fire interns and fail students for. His very first PR corrupted the database. The most recent one didn’t even compile.

Developer - Millennial fratboy with a business degree. 8 years experience on paper.
His entire knowledge of programming amounted to a single data structures class he took on Coursera. Claims that’s all he needs. His PRs was a single 4000+ line files, of which 3500+ failed the linter, had numerous bugs / console warnings / compile warnings, and implemented 60% of functionality requested in the user story. Also forget about getting his attention whenever one of the pretty secretaries walked by. He would leap out of his seat and waltz off to flirt.

Developer - Brooding loner. 6 years experience on paper.
His code works. It runs, in exponential time. Simply ignores you when you attempt to ask.

Developer - Agile fullstack developer extraordinaire. 8 years experience on paper.
Insists on doing the absolute minimum required in the user story, because more would be a waste. Does not believe in thinking ahead for edge conditions because it isn’t in the story. Every single PR is a hack around existing code. Sometimes he hacks a hack that was initially hacked by him. No one understands the components he maintains.

Developer - Team lead. 10 years of programming experience on paper.
Writes spaghetti code with if/else blocks nested 6 levels deep. When asked "how does this work ?”, the answer “I don’t know the details, but hey it works!”. Assigned as the team lead as he had the most experience on paper. Tries organise technical discussions during which he speaks absolute gibberish that either make no sense, or are complete misunderstandings of how our system actually works.

The last 2 guys are actually highly regarded by my company and are several pay grades above me. The rest were hired because my company was desperate to staff the project.

There are a 3 more guys I didn’t mention. The 4 of us literally carried the project. The codebase is ugly as hell because the others merge in each others crap. We have no unit tests, and It’s near impossible to start because of the quality of the code. But this junk works, and was deployed to production. Today is it actually hailed as a success story.

All these 3 guys have quit. 2 of them quit without a job. 1 found a new and better gig.

I’m still here because I need the money. There’s a tsunami of trash code waiting to fail in production, and I’m the only one left holding the fort.

Why am I surrounded by morons?
Why are these retards paid more than me?
Why are they so proud when all they produce is trash?
How on earth are they still hired?

And yeah, FML.

Never in my life I was scared as today.
I recently left a big company to work for a small one as the first internal developer.
Had a small issue in the production server. The fix was easy, just remove a single table entry. And... *drum roll*... I forgot to add a where clause. All orders were lost.
No idea if we had backups or anything, I quickly called the one other IT dude in the company.
He had no clue where are the backups and how to find them.
Having some experience with Nmap, I quickly scanned our network and found a Nas device.
There was a backup, whole VHD backup. 300GB of it, the download speed is around 512kb/s. No way I can fix it before management finds out, but then an idea came to mind. Old glorious 7zip. Managed to extract only the database files, sent them to the server and quickly swapped them. Everything was fine... The manager connected 5 minutes later. Scariest 45 minutes of my life...

My team handles infrastructure deployment and automation in the cloud for our company, so we don't exactly develop applications ourselves, but we're responsible for building deployment pipelines, provisioning cloud resources, automating their deployments, etc.

I've ranted about this before, but it fits the weekly rant so I'll do it again.

Someone deployed an autoscaling application into our production AWS account, but they set the maximum instance count to 300. The account limit was less than that. So, of course, their application gets stuck and starts scaling out infinitely. Two hundred new servers spun up in an hour before hitting the limit and then throwing errors all over the place. They send me a ticket and I login to AWS to investigate. Not only have they broken their own application, but they've also made it impossible to deploy anything else into prod. Every other autoscaling group is now unable to scale out at all. We had to submit an emergency limit increase request to AWS, spent thousands of dollars on those stupidly-large instances, and yelled at the dev team responsible. Two weeks later, THEY INCREASED THE MAX COUNT TO 500 AND IT HAPPENED AGAIN!

And the whole thing happened because a database filled up the hard drive, so it would spin up a new server, whose hard drive would be full already and thus spin up a new server, and so on into infinity.

Thats probably the only WTF moment that resulted in me actually saying "WTF?!" out loud to the person responsible, but I've had others. One dev team had their code logging to a location they couldn't access, so we got daily requests for two weeks to download and email log files to them. Another dev team refused to believe their server was crashing due to their bad code even after we showed them the logs that demonstrated their application had a massive memory leak. Another team arbitrarily decided that they were going to deploy their code at 4 AM on a Saturday and they wanted a member of my team to be available in case something went wrong. We aren't 24/7 support. We aren't even weekend support. Or any support, technically. Another team told us we had one day to do three weeks' worth of work to deploy their application because they had set a hard deadline and then didn't tell us about it until the day before. We gave them a flat "No" for that request.

I could probably keep going, but you get the gist of it.

It's 2018 and I am forced to add new features on project that was deployed in 2003 and lastly updated at 2009.

Best part is that PHP version on the production server is [drum roll] 4.3.8. I found out that this version was the latest at July 2003. On top of that, server runs on Windows Server 2003 and the database is oracle (meaning php driver is needed).

I have spent a WHOLE WEEK just trying to recreate that environment in order to start working on the new features.

Sorry for grumping but I had to take it out somewhere.

This is a funny one:

I found this gem in SO.
Why is it funny? Well, PDVSA is the state-owned company for oil production of my country (Petróleos De Venezuela).

This little intern decided that it was a good idea to publish in SO an answer with actual code, showing database and tables names to everyone.

Priceless.

Just last month we finished a big project for a client. The company had their lawyers make a contract to transfer ownership for the website to them.

One of the requirements was to deliver all code in a word file. And just that. In the whole contract was not a word about transfering the production environment to their servers. Or supplying a working version. The files and database. No just the word file.

As of today still wondering what they where planning to do with a word file with hunderds of pages of code.

Offcourse we deliverd a working version to their servers. But why are there people making decisions about things they understand nothing at all.

This is going to be a long rant, coz this is the only way to vent out my frustration against our tech head.

Yesterday, while our fucking twat tech head was playing around in company aws account, he terminated the production server. By mistake, apparently. Coz he doesn't know shit about server management. But that egoist ass won't admit and fucked the production server.

And then ran away. We developers sprang into action. Updated dns to point to staging server, setup virtual hosts, env files, point to prod database, force flush dns cache. All systems were up and running in 30 mins. And since it was staging server, it had lot of untested features and codes, and we spent rest of the day fixing the bugs.

And that tech head, who ran away hiding his tail between his legs, after he fucked the server, came back after systems were up. And started cracking jokes, that "so many features got released in 1 day" . "We cut server cost by shutting down 1 server."

We were struggling and working in full throttle to make the services running again. And that fuckity fucker was cracking jokes.

And I don't even know what excuse he gave to ceo for the downtime. I am pretty sure he would have made up some crappy excuse to hide his fucking mistake. That ass never admits his mistake. I am thinking to go to ceo today and tell the real story and get that faggot head fired or at least a strict warning.

Dev: Hey, I need you to see something.
Client: ok
Dev: this is the models of those reports you need
Client: ok, wait, what is that number? This is wrong, we can't...
Dev: hey, calm down, this is not the production, it's just fake database!
Client: ah, ok.
(5 minutes of explantion)
Dev: so, what do you think?
Client: just one point, that data is very very wrong, we need to change this ASAP!

One:

Had a stack of harddrives with my important data, two USB drives and a 4.7gb disc, two or three cloud storage accounts.

Needed a restore:
Knocked the stack of hard drives onto the floor (all broken), stood on one of the flash drives, found the other one in a pocket of a pair of trousers which just came out of the washing machine, dvd too scratched to read and couldn't verify my cloud storage account because I lost the password to the connected email account and the backup email account to verify that one didn't exist anymore. Fucking hell.

Two:

Production database with not that much yet but at least some production data which wasn't backupped.
Friend: can I reboot the db machine?
Me: yup!
Friend: what's the luks crypt password?
Me: 😯😐😓😫😲😧😭

End of story 😅

For the record, the first one actually happened (I literally cried afterwards) and that taught me to update my recovery email addresses more often!

We've built a web app and now a client wants a VPN acces to the database of web app. When asked why, they said they want to occasionally pull some data out. 😱

We said no, and this is what they wrote:
"We’ve got live VPN access to every other web database we work with – why is this different?"

Well because maybe we know that we can build you an export of whatever you want, prepare you API calls for getting data to your CRM, but hell I'm not giving you access to the production DB.

Most satisfying bug I've fixed?

Fixed a n+1 issue with a web service retrieving price information. I initially wrote the service, but it was taken over by a couple of 'world class' monday-morning-quarterbacks.
The "Worst code I've ever seen" ... "I can't believe this crap compiles" types that never met anyone else's code that was any good.

After a few months (yes months) and heavy refactoring, the service still returned price information for a product. Pass the service a list of product numbers, service returns the price, availability, etc, that was it.

After a very proud and boisterous deployment, over the next couple of days the service seemed to get slower and slower. DBAs started to complain that the service was causing unusually high wait times, locks, and CPU spikes causing problems for other applications. The usual finger pointing began which ended up with "If PaperTrail had written the service 'correctly' the first time, we wouldn't be in this mess."

Only mattered that I initially wrote the service and no one seemed to care about the two geniuses that took months changing the code.

The dev manager was able to justify a complete re-write of the service using 'proper development methodologies' including budgeting devs, DBAs, server resources, etc..etc. with a projected year+ completion date.

My 'BS Meter' goes off, so I open up the code, maybe 5 minutes...tada...found it. The corresponding stored procedure accepts a list of product numbers and a price type (1=Retail, 2=Dealer, and so on). If you pass 0, the stored procedure returns all the prices.
Code basically looked like this..

public List<Prices> GetPrices(List<Product> products, int priceTypeId)
{
foreach (var item in products)
{
List<int> productIdsParameter = new List<int>();
productIdsParameter.Add(item.ProductID);
List<Price> prices = dataProvider.GetPrices(productIdsParameter, 0);
foreach (var price in prices)
{
if (price.PriceTypeID == priceTypeId)
{
prices = dataProvider.GetPrices(productIdsParameter, price.PriceTypeID);
return prices;
}
* Omitting the other 'WTF?' code to handle the zero price type
}
}
}

I removed the double stored procedure call, updated the method signature to only accept the list of product numbers (which it was before the 'major refactor'), deployed the service to dev (the issue was reproducible in our dev environment) and had the DBA monitor.

The two devs and the manager are grumbling and mocking the changes (they never looked, they assumed I wrote some threading monstrosity) then the DBA walks up..

DBA: "We're good. You hit the database pretty hard and the CPU never moved. Execution plans, locks, all good to go."
<dba starts to walk away>
DevMgr: "No fucking way! Putting that code in a thread wouldn't have fix it"
Me: "Um, I didn't use threads"
Dev1: "You had to. There was no way you made that code run faster without threads"
Dev2: "It runs fine in dev, but there is no way that level of threading will work in production with thousands of requests. I've got unit tests that prove our design is perfect."
Me: "I looked at what the code was doing and removed what it shouldn't be doing. That's it."
DBA: "If the database is happy with the changes, I'm happy. Good job. Get that service deployed tomorrow and lets move on"
Me: "You'll remove the recommendation for a complete re-write of the service?"
DevMgr: "Hell no! The re-write moves forward. This, whatever you did, changes nothing."
DBA: "Hell yes it does!! I've got too much on my plate already to play babysitter with you assholes. I'm done and no one on my team will waste any more time on this. Am I clear?"

Seeing the dev manager face turn red and the other two devs look completely dumbfounded was the most satisfying bug I've fixed.

So I took on a fairly big project and poured my heart and soul into it, was the biggest thing I did yet. I kept on sending beta's to the customer after each change for review! Kept on insisting that they review it, the answer was always "this looks amazing keep doing what you're doing"! After I finished and pushed everything to production.

They didn't use it for nearly 6 months! And then out of the blue they call me saying that half of the app is wrong.. WTF? Where was this information during testing! I informed them that the changes would take some time since I need to do migrations and change the whole database schema.

In which they replied "but you already finished it once won't changing things make it easier? We shouldn't pay for your mistakes"
I don't know how I handled that but they should be thankful they were half way across the country 😠😠😠😠

Try it!!!😂😂😂😂😂😂

Stolened! 😆😆

So we hired an intern and his first task was to change a few things in email layout for our client, which is an investment bank.

I told to one of my developers to make his local database dump and setup the project for an intern. When intern completed the task, my developer thought that title "Dow Jones index crashed" was pretty funny title for a test.

What he didn't thought through enough, is that he forgot to configure fake SMTP server and he had production database dump with real email addresses.

I had really awkward 20 minutes conversation with our client. Fuck my life.

Worst data loss? Probably back when I was working at gitlab and accidentally dropped a production database....

This happend to me around 2 weeks ago. For some reason, I decied to post this now.

I won the lottery, yey! I mean, bot really, but I am <19yo student, "less than junior dev" in my office, but sonce I am the only one who is capable of working with hardware, I was working month back as a sysadmin for a few days. Our last sysadmin was really good working but really, really toxic guy, so he got fired on a spot after argument with some manager or whatever, no big deal, we could have another guy hired in a week. But, our backup server literally was on fire, all data probably dead because bad capacitor or whatever. This was our only backup of everything at the time. Everyone in full fucking panic mode, we had literally no other working HW we could use for backup, but then comes me, intern employed on his first dev job for 3 months. That day I bought some HW for my own personal server at home (Intel NUC with some Celeron, 4GB DDR4 RAM and two 240GB SSDs for RAID 1. My manager asked everyone in the office for sollution how to survive next 4 days before new server arrives. People there had no idea what tk do and no knowedgle about HW, I just came from a break and offered my components for a week, since there was noone else who can work with HW, servers and stuff like this, manager offered me $500+HW cost if I, random intern, can make it work. I installed Debian on that little PC, created RAID1 from both SSDs, installed MySQL server and mirrored GIT server from our last standing server (we had two before one of them went lit 🔥), made simple Python script to copy all data on that RAID, with some help of our database guy copied whole DB from production to this little computer and edited some PHP so every SQL request made on our server will run on that NUC too. Everything after ±2 hours worked perfectly. Untill a fucking PSU burned in our server and took RAID controller with him in sillicon heaven next night, so we could not access any data unltill we got a new one. Thanks to every god out there, I was able to create software RAID from survived HDDs on our production server and copy all data from that NUC on the servers software RAID and make it working at 3 AM in the night before an exam 😂. Without this, we would be next ±40 hours without aerver running and we might loose soke of our data and customers. So my little skill with Linux, Python, MySQL and most importantly my NUC hardware I got that day running as a backup server saved maybe whole company 😂.
Btw, guess who is now employee of the year with $2500 bonus? 😀
Sorry for bragging and log post, but I was so lucky an so happy when everything worked out, good luck to all sysadmins out there! 👍

TL:DR: Random intern saved company and made some money 😂

Where I work, in our database, we use 3 to indicate true and 7 to indicate false and 0 is true but null is false

In another table, we use 'P' to indicate true and 'I' to indicate false and 'Y' is also false and null is false

And the most used table, we also use 'Y' to indicate yes and 'N' to indicate no, but null is also Yes.

We also store integers as varchar in a live table, but stays an integer in all the other tables. I hope I'm not there when the number of digits exceeds the varchar limit.

These are all live and used in production all created by my boss, the head of IT.

The experience that made me feel like a dev badass was when a teammate accidentally deleted the database for production and I had the latest backup. Everybody was panicking not until I told them I had the solution

Accidently deleted user table on production database and some backups where broken. Had to deploy a 4 days old backup...

Rant, but also want to know, how the fuck do you accidentally drop a database link. In production. On the most important database. Housing our biggest applications.

Old boss story. This guy was nice but a terrible boss. Also relevant, he has a background in IT so should know better.

Him: So when you wanna check a password is correct you just unhash it in the database?
Me: *facepalm*

Me: Hey we should be doing unit and integration testing at a minimum to lower bugs.
Him: We don't need those, we're not a bank. If a problem comes up we just fix it and push to production.
(A while later)
Him(in email): Why do we keep getting bugs reported. Don't you devs test your code.

I was mildly annoyed at that one.

Him: We're always over budget on projects, how can we fix this.
Me: What if we increase our quotes.(technically there are other ways as well but not really possible at that time)
Him: We can't do that, clients won't want to pay.

Me: *finishing off my handover as I'm leaving for a new job*
Him: Wow you do a lot of work

Always use SELECT-query with the same conditions before you DELETE/UPDATE in a production database.

one of our engineers accidentally deleted our production database and our clients didn't even notice. that's when we knew they are not using our product.

Who needs fireworks if the sound of angry consumers after deleting a production database is as resounding as the sound of war itself

*In the final weeks of development with a project on a short timeline because the client "needs it".*

Client: "We've hired a consultant we want you to work with."

Me: "Okay, can we push this to after the delivery?"

Client: "Of course"

Wake up to an email from the consultant with a list of scripts he just ran on the production database server for the currently live app.

Get follow-up emails about bugs and app crashes from the client.

My rage is so hot it can keep warm an Eskimo tribe over the winter season.

Never ever have a tab open on production server database. Changed all users passwords by mistake. Thought was my test database.

Turns out that dropping all tables on the production database wasn’t such a good idea. 😅

Thank god we had a backup from 10 minutes before it happend.

Ok so this happend in the last 3 days, I didn't post it till now because I had to seriously take a rest with all the bullshit and stress that came with it...
(Legacy project I have the lead in called: "Foo")

Monday:
Management decided it would be effective to add a senior and a junior to Foo, which would make (together with me) to be 2 juniors and one senior developer
Well I've spend most of that day helping both the junior and the senior to setup "Foo" on their local development machines... So I could not do any programming myself

tuesday:
The senior wanted to refactor EVERYTHING... and I had to stop him multiple times because we simply do not have the time to do that...
The junior tried to work on other things as much as he could, and after he had run out of things to do, asked me for EVERYTHING... EVEN WHERE TO FUCKING CHANGE SOME GOD DAMN STRINGS!....
Also he did in total 3 commits, two of which existed of my code (because I had to "help" him

wednesday:
Both the junior and senior were removed from the project and I got another senior.. who fucking deleted the production database on accident

god damn rough few days man...

Just took a new role...

Production database filled with tables like the following :/

SELECT TOP (1000) [ID]
,[Field1]
,[Field2]
,[Field3]
,[Field4]
,[Field5]
,[Field6]
,[Field7]
,[Field8]
,[Field9]
,[Field10]
,[Field11]
,[Field12]
,[CapID]
,[Car_Manufacturer]
,[Car_Range]
,[Car_Derivative]
,[Field17]
,[Field18]
,[Field19]
,[Car_Model]
,[Field21]
,[Field22]
,[Field23]
FROM [****].[dbo].[****]

This isn't even the worst, there's a table with 250+ fields that should be broken out into 10-15 tables. The code is far, far worse.

Don't think I'll be here too long.

My worst dev sin was leaving out the 'WHERE' in a SQL update statement on a production database

Set every booking to be owned by the same sales consultant 👀

This just happened at work, except for the fact the entire database was dropped...

I woke up early and thaught I could finish some database work today.
Beeing tipsy I deleted two production tables and shit my pants.

Luckily I do backups since the last retarded db-deletion.

I think I want to quit my first applicantion developer job 6 months in because of just how bad the code and deployment and.. Just everything, is.

I'm a C#/.net developer. Currently I'm working on some asp.net and sql stuff for this company.

We have no code standards. Our project manager is somewhere between useless and determinental. Our clients are unreasonable (its the government, so im a bit stifled on what I can say.) and expect absurd things from us. We have 0 automated tests and before I arrived all our infrastructure wasn't correct to our documentation... And we barely had any documentation to begin with.

The code is another horror story. It's out sourced C# asp.net, js and SQL code.. And to very bad programmers in India, no offense to the good ones, I know you exist. Its all spagheti. And half of it isn't spelled correctly.

We have a single, massive constant class that probably has over 2000 constants, I don't care to count. Our SQL projects are a mess with tons of quick fix scripts to run pre and post publishing. Our folder structure makes no sense (We have root/js and root/js1 to make you cringe.) our javascript is majoritly on the asp.net pages themselves inline, so we don't even have minification most of the time.

It's... God awful. The result of a billion and one quick fixes that nobody documented. The configuration alone has to have the same value put multiple times. And now our senior developer is getting the outsourced department to work on moving every SINGLE NORMAL STRING INTO THE DATABASE. That's right. Rather then putting them into some local resource file or anything sane, our website will now be drawing every single standard string from the database. Our SENIOR DEVELOPER thinks this is a good idea. I don't need to go into detail about how slow this is. Want to do it on boot? Fine. But they do it every time the page loads. It's absurd.

Our sql database design is an absolute atrocity. You have to join several tables together just to get anything done. Half of our SP's are failing all the time because nobody really understands the design. Its gloriously awful its like.. The epitome of failed database designs.

But rather then taking a step back and dealing with all the issues, we keep adding new features and other ones get left in the dust. Hell, we don't even have complete browser support yet. There were things on the website that were still running SILVERLIGHT. In 2019. I don't even know how to feel about it.

I brought up our insane technical debt to our PM who told me that we don't have time to worry about things like technical debt. They also wouldn't spend the time to teach me anything, saying they would rather outsource everything then take the time to teach me. So i did. I learned a huge chunk of it myself.

But calling this a developer job was a sick, twisted joke. All our lives revolve around bugnet. Our work is our BN's. So every issue the client emails about becomes BN's. I haven't developed anything. All I've done is clean up others mess.

Except for the one time they did have me develop something. And I did it right and took my time. And then they told me it took too long, forced me to release before it was ready, even though I had never worked on what I was doing before. And it worked. I did it.

They then told me it likely wouldn't even be used anyway. I wasn't very happy at all.

I then discovered quickly the horrors of wanting to make changes on production. In order to make changes to it, we have to... Get this

Write a huge document explaining why. Not to our management. To the customer. The customer wants us to 'request' to fix our application.

I feel like I am literally against a wall. A huge massive wall. I can't get constent from my PM to fix the shitty code they have as a result of outsourcing. I can't make changes without the customer asking why I would work on something that doesn't add something new for them. And I can't ask for any sort of help, and half of the people I have to ask help from don't even speak english very well so it makes it double hard to understand anything.

But what can I do? If I leave my job it leaves a lasting stain on my record that I am unsure if I can shake off.

... Well, thats my tl;dr rant. Im a junior, so maybe idk what the hell im talking about.

Because I’m a fucking cowboy and a charlatan, and because I hate sleep and despise feeling refreshed and happy, I’m working pretty much full time as a contractor (I’m the full stack dev. I do everything) on a (well funded) startup alongside my day job.

Tonight I had to make some quick (lol “quick”) changes to a core piece of the platform.

Now before continuing please refer back to the first line of this rant.

So instead of writing new functionality, I copied and pasted another section.

I renamed all references of “new_order” to, cleverly “new_order2”.

I know.

I deploy to production...

My phone starts blowing up. In short, everything is fucked.

I’m going over the query, checking the production database. Why is this manifesting like this? It all looks correct.

2 HOURS of broken sales, pissed off customers, pissed off service agents and I see that there was still one reference of “new_order” that should have been “new_order2”.

I am a piece of shit.

I messed up carelessly in production. Learnt how SQL queries bite you in the ass when it knows you are under pressure.

Was hosting an online quiz kinda thing during my college techfest. Tens of thousands of people participating.

Using MySQL as database and thousands of queries were being executed. Everyone were pretty excited as the event just opened up.

None of the teams could solve one particular level. Turns out the solution was wrong and was asked by the organisers to change the solution for that particular level. Usual stuff, right?

Was too lazy to open up the web UI for the back office and so, straight ahead logged in to the MySQL server and ran the UPDATE query on the table consisting of the solutions.

It had been a couple of hours and the organisers came to me with a weird problem. There were no changes in the scoreboard for the last two hours. Everyone were stuck wherever they were. Weird, right?

I then realized.

Fk.

In that dreaded query, I had only run

UPDATE 'qa' SET answer = 'something'

leaving out the where clause, specifying the question to update, like

WHERE qno=13

As a result, solutions to all the questions were updated to the same answer. After hastily fixing everything back, I had the dreaded conversation.

Org: What was the problem?

Me: It was the cache.

Org: Damn thing. Always messes up.

Me: *sheepishly* yeah

Probably the most embarrassing moment in my life, wrt coding 😑

Ended in an UPDATE without WHERE query to a core table in a Drupal project (in a dev database stored in the same server that production database was)

If you're going to request CRITICAL changes to thousands of records in the database, and approve it through testing which is done on an exact replica of production, then tell me it was done incorrectly after the fact it has been implemented and you didn't actually review the changes made to the data or business logic that you requested then you are an idiot. Our staging environment is there to ensure all the changes are accurate you useless human. Its the data you provided, I didn't just magically pull it from thin air to make yours and my job a pain the ass.

Adding a feature to webapp...

Webapp relies on database in production server...

*adds feature to production webapp directly*

Every page: ERROR 500

Manager: what did you do???!!!! You MESSED UP the production, FIX IT NOW

*Use ctrl-z because manager doesn't like Version Control*

Soooo... My senior dev just dropped a very importand table om our production enviroment.
Time to restore the database.. only 50 GB left

My manager is instructing my team to add a feature that can only be enabled for users by running an update script in the database.

When I argued that it's not really "complete" if it can't be turned on without someone going into the production database, I was told that not only is it complete, but they plan to have our non-technical customer service enable it for customers if the customer requests it...

Apparently giving everyone and their brother write access to prod is a good idea, but implementing a checkbox is a "waste of time and would cost too much money".

Probably going to float my resume... :-p

OMFG I don't even know where to start..
Probably should start with last week (as this is the first time I had to deal with this problem directly)..

Also please note that all packages, procedure/function names, tables etc have fictional names, so every similarity between this story and reality is just a coincidence!!

Here it goes..

Lat week we implemented a new feature for the customer on production, everything was working fine.. After a day or two, the customer notices the audit logs are not complete aka missing user_id or have the wrong user_id inserted.
Hm.. ok.. I check logs (disk + database).. WTF, parameters are being sent in as they should, meaning they are there, so no idea what is with the missing ids.

OK, logs look fine, but I notice user_id have some weird values (I already memorized most frequent users and their ids). So I go check what is happening in the code, as the procedures/functions are called ok.

Wow, boy was I surprised.. many many times..
In the code, we actually check for user in this apps db or in case of using SSO (which we were) in the main db schema..
The user gets returned & logged ok, but that is it. Used only for authentication. When sending stuff to the db to log, old user Id is used, meaning that ofc userid was missing or wrong.

Anyhow, I fix that crap, take care of some other audit logs, so that proper user id was sent in. Test locally, cool. Works. Update customer's test servers. Works. Cool..

I still notice something off.. even though I fixed the audit_dbtable_2, audit_dbtable_1 still doesn't show proper user ids.. This was last week. I left it as is, as I had more urgent tasks waiting for me..

Anyhow, now it came the time for this fuckup to be fixed. Ok, I think to myself I can do this with a bit more hacking, but it leaves the original database and all other apps as is, so they won't break.
I crate another pck for api alone copy the calls, add user_id as param and from that on, I call other standard functions like usual, just leave out the user_id I am now explicitly sending with every call.
Ok this might work.

I prepare package, add user_id param to the calls.. great, time to test this code and my knowledge..

I made changes for api to incude the current user id (+ log it in the disk logs + audit_dbtable_1), test it, and check db..
Disk logs fine, debugging fine (user_id has proper value) but audit_dbtable_1 still userid = 0.

WTF?! I go check the code, where I forgot to include user id.. noup, it's all there. OK, I go check the logging, maybe I fucked up some parameters on db level. Nope, user is there in the friggin description ON THE SAME FUCKING TABLE!!
Just not in the column user_id...

WTF..Ok, cig break to let me think..

I come back and check the original auditing procedure on the db.. It is usually used/called with null as the user id. OK, I have replaced those with actual user ids I sent in the procedures/functions. Recheck every call!! TWICE!! Great.. no fuckups. Let's test it again!
OFC nothing changes, value in the db is still 0. WTF?! HOW!?

So I open the auditing pck, to look the insides of that bloody procedure.. WHAT THE ACTUAL FUCK?!
Instead of logging the p_user_sth_sth that is sent to that procedure, it just inserts the variable declared in the main package..
WHAT THE ACTUAL FUCK?! Did the 'new guy' made changes to this because he couldn't figure out what is wrong?! Nope, not him. I asked the CEO if he knows anything.. Noup.. I checked all customers dbs (different customers).. ALL HAD THIS HARDOCED IN!!! FORM THE FREAKING YEAR 2016!!! O.o

Unfuckin believable.. How did this ever work?!

Looks like at the begining, someone tried to implement this, but gave up mid implementation.. Decided it is enough to log current user id into BLABLA variable on some pck..
Which might have been ok 10+ years ago, but not today, not when you use connection pooling.. FFS!!

So yeah, I found easter eggs from years ago.. Almost went crazy when trying to figure out where I fucked this up. It was such a plan, simple, straight-forward solution to auditing..

If only the original procedure was working as it should.. bloddy hell!!

Oh man.
I imagine the face of the guy who deleted Gitlab production database when he realized what he had done.

So I started in a new job a week ago for a two weeks probation period, im getting payed double than my last job but it's so professionally frustrating.

They use a full php stack with a framework called tinymvc that I never heard of and the last commit is from 2009.

Beside this they implement some sort of "flexible" MVC where a great amount of the logic are inside the views. They have one model for each entity (in theory) but in reality one model have methods for a lot of tables.

Beside this the i18n is done by querying the database for all the translation strings and copying it in every user session, so every session file it's about 400kb where around 380 are duplicated translations.

The views folder is empty because they decided to modify the framework to look for the views in another folder called resource's and the development must be done connected directly to the production database

Above all this shit all the many-to-many relations in the database (MySQL) are handled creating a comma separated field on one of the tables, completely breaking the reference integrity.

So, after a week of work I can't stop thinking who the fuck developed this? In which world this shit is okay? How can I work around this big lake of shit?

Typos kill, kids! And deploying to production.

Instead of "for item in items" in my script, I accidentally did "for items in items". Thus, an exponential loop has been entering things into the database for the past few hours before I found the place to fix it.

By the way, this runs on cron every minute. So there are processes still running exponentially right now, possibly 180+.

Yeah, I'm setting up a a test server instead now.

So I'm wrapping up for the day and right before I leave a coworker comes up to me with a problem. Our company uses barcodes to track some of our products through their development and we recently switched over to a new system for producing them. The barcodes for this particular product are supposed to have 8 digits, but the last 200 we printed have 9.

I immediately panic because I wrote the script that generates the bar codes and there had been a bug in the past where the script would add extra leading zeroes that weren't supposed to be there. I scramble and check the database, it would be a huge headache if our production database had been compromised with junk barcodes. Nope, all the new barcodes there have the right number of digits.

Next place to check is in the code that writes the barcodes to a text file for staff to print the physical labels from. Nope that's all fine too.

I ask the person who printed out the recent batch of labels to show me how the printing software reads from the text file. She seems confused by my question and shows me how she manually enters in the barcode range to the software. As she does this I watch her add an extra zero to the numbers. 🙃

Even worse there was an option to import all the codes from a text file literally RIGHT BELOW the manual option.

TLDR; Thought my script had screwed up our database, ended up being the fault of a coworker who didn't know how to import text files.

Tomcat

manager: "hey, we have this old java software running and need it to be compatible with our brand new crm system"
me: "okay, i've never workes with jsp before, but i'll have a look at it"
code: undocumented, who would have guessed
manager: "oh and btw. you must test your code on our live system, with our production database, but make sure not to brake anything, our last backup was 20 months ago."
me: "..."

This former developer made an app 2 years ago which is in production since then. On the 404 page it's throws the database credentials. The database saves personal information about the mobile owner.

Luckily I found out and fixed it. The client doesn't know about this.

Oh boy!

It's 17:55... Did much work that day since I came in earlier than usual, so I could leave in time and do some shopping with the girlfriend.

A colleague comes in to my room, a tad distressed. He had accidentally ran a fixture script on a production environment database (processing a shipload of records per minute), truncating all tables...

Using AWS RDS to rollback the transaction log takes up about 20m. I had to do that about 5 times to estimate the date and time of when the fixture script ran... Since there was no clear point in time...

Finally I get to the best state of the data I could get. I log in remotely run some queries. All is well again... With minor losses in data.

I try to download a dump using pg_dump and apparently my version is mismatched with the server. I add the latest version to aptitudes source list of postgres repo and I am ready to remove and purge the current postgres client and extensions...

sudo apt-get remove post*
Are you sure? (Y/n) *presses enter and enters into a world of pain*

Apparently a lot of system critical applications start with post... T_T

I was testing database migration and there was some issues which I couldn't fix them. So I drop the table just to test if the issues are still there.
"Why nothing have been changed?" After some minutes I realize it was production database which I had dropped 😓

Today was a manic-depressive kind of day. Spent the morning helping some developers with getting their code to run a stored procedure to drop old partitions, but it wasn't working on their end. It was a fairly simple proc. But working with partitions is a little like working with an array. I figured out that they were passing the wrong timestamp, and needed to add +1 to delete the right partition. Got that sorted out, and things were good. Lunch time.

After lunch I did some busy work, and then the PO comes up at about 2PM and says he's assigned some requests to me. The first was just attaching some scripts. Easy. The second, the user wants a couple of schemas exported ... at 6PM. I've been in the office since 6:45AM.

While I'm setting up some commands to run for the data export, a BA walks up and asks if I'm filling in for another DBA who is out for a few weeks. Yep. There's a change request that hasn't been assigned, and he normally does the work. I ask when it's due. Well, the pre-implementation was supposed to be done in the morning, but it wasn't, and we're in the implementation window ... half way through. I bring up the change task, and look at. Create new schema and users. That's all it says. The BA laughs. I tell I need more to go on. 10 minutes later he sends an email with the information. There's only two hours left in the window, and I can only use half of it, because the production guys have to their stuff, and we're in their window. Now I'm irritated, because I'm new to Oracle, and it's an unforgiving mistress. Fortunately, another DBA says he'll do it, so that we can get it done in time. But can't work it either, because Dev DBAs don't have access to QA, and the process required access for this task. Gets shelved until the access issue is resolved. It's now after 4:15PM. I'm going to in traffic with that 6PM deadline.

I manage to get home and to the computer by 5:45PM. Log in. Start VPN. Box pops on screen. Java needs to update. I chose skip update. Box pops up again. It won't let me log in until Java is current. Passed.

I finally get logged in, and it's 6:10PM. I'm late getting the job started. I pull up Putty and log into the first box, and paste my pre-prepared command in the command line and hit error. Command not found. I'm tired, so it's a moment to sink in. I don't have time for this.

I log into DBArtisan and pull up the first data base, use the wizard to set the job, and off it goes. Yay. Bring up the second database, and have enter the connect info. Host not found. Wut? Examine host name. Yep, it's correct. Try a different method. Host not found. Go back to Putty. Log in. Past string. Launch. Command not found. Now my brain is quitting on me. Why now? It's after 6:30PM. Fiddle with some settings, reset $Oracle home. Try again. Yay. It works. I'm done. It's after 7PM.

There is nothing like technology to snatch the euphoria of a success away from you. It's a love-hate thing, but I wouldn't trade it for anything else. I'm done. Good night.

Maybe not worst, but most frustrating. One of the systems I helped maintain at my first job had a few different bugs that caused bad data in the database. The "solution" to the problem was to write SQL queries to directly fix the production data. This would take one member of our team (it rotated weekly) about an hour every day to fix because there were literally dozens of these errors.

All the devs knew that we could identify the root cause and fix it in, probably, 3-4 days tops. Management would never approve the time because it would take longer to fix the root cause than it took to fix the data.

I worked at that company for 7 years. The bug was there when I came on, and it was there when I left.

Discovered yesterday that my boss does tests with production database... and I'm responsible for the fucking backups and he doesn't even care to let me now when so I can at least schedule one at slave. Come on... it's not that hard to let others know or test on your own machine...

I accidentally started a reindex on a collection that had 14 million records in the middle of the day. Caused an outage in a major portion of our applications for about 3 hours. Worst thing was that once I pressed enter, I realized that it was for the production database, and not the staging database like I intended. I immediately went to go tell the dev ops lead, and he basically said, "whelp, let's just sit back and watch the world burn. Not much we can do about it"

Today in development: discovered that it's possible via combination of keys to rename a database in SQL Server Management Studio without as much as a dialog box to confirm.

Shout out to the 2000ish users in production that discovered this delightful nugget of info with me.

Lessons learned:

A) Don't trust Microsoft to create software that makes you confirm potentially catastrophic actions

B) Make sure your user hasn't been granted ALTER DATABASE permissions without your knowledge before you start using it.

I never understood why writing data to a database had to be such a complicated process, at least in production systems.
I shouldn't need a PhD to figure out how the db was configured.

One night, after one very stressful week of production code fixes (I was working on a game with some friends and I created the network infrastructure for P2P and database communication from scratch), I was at my gf's house. After we fell asleep, I stood up and screamed right at her something like "I fucking already told you how X works and how to communicate with Y. Learn to write code properly and after double checking yours then you shall ask me for a non-existing 'bug' fix. Learn how to properly write event based code and use polling you moron!". After that I turned to the other side and fell asleep immediately.
When I saw her the next morning sleeping in the couch, I could not understand why... Only after she described to me the whole incident I started laughing.
After that I just took two weeks off the project and after that period I never actually worked the same way (so hard) in my free time with them.

I work on a warehouse dev team. One day this past year, I was trying to deploy a new build to a QA server. Earlier that day I had been looking at the logs on the production server and had left the ssh session open. I had been working for less than a year out of college at this point and shouldn't have had access to deploy to the production server.

Long story short I deployed my QA build to the production server and saw there were problems connection to our production database. Then my heart dropped in my chest as I realized I had just brought down our production server.

I managed to get the server back up by rolling back in about 5 minutes and no one ever knew except some people on my team.

I felt horrible for the longest time. Later in the year another guy that joined my team that has about 20 years of experience under his belt did the exact same thing, but needed help rolling it back. Needless to say, that made me feel a lot better. 😂

Definitely the worst moment of my year.

I once agreed to maintain and develop an application used in a different section of the school to keep inventory and make sure everything is where it is supposed to be.

At first there was enthusiasm, together with 2 of my classmates we agreed and git clone-d the .NET application that now graduated students built and maintained for the past few years. What could go wrong right?!

It became clear that the original students that worked on it followed an older curriculum, meaning they still got taught .NET instead of the core variant that we get now, not only that but it also seemed that they either did not fully grasp the Clean/Onion architecture or didn't get it in class since there were infrastructure components in the 'Domain' project of the solution. Think of 2 DBContexts in the domain model, yep.

One of us bailed in the first week, the other one and I felt bad for the people using the app so we went on and tried to work on the first bugs that were described in a document. One of these bugs was 'whenever I filter on something in the list, everybody gets to see that filter on their screen instead of only me'. Woah that's weird! Let's see how they put that together!

Oh god, they are using a _static_ variable to store filters, no wonder that it doesn't work properly. Ever heard of sessions?!

Second bug: Sometimes people can't create an account when we sign them up from the admin panel. Alright that is weird, let's figure that one out! Wait a second it seems to work in development? What's this about.

Oh wait I can't create an account on production either? Oh that's weird, wait a second... Why do I have to put my e-mail in a form that was sent to me through e-mail? Why is my address not filled in already? OOH, if someone types in the wrong e-mail address (which is easy since our school has 4 variants of the same f*cking e-mail address) it won't work since it can't recognize the user! Brilliant! Remove e-mail input box and make a token/queryparam determine the user account.

Ah that seems good, it's a mess but it seems a tiny bit better now, great! We're making progress and some sweet buck.

Next bug, trillions of 50x errors on random pages, that's a weird one.

Hm everything works in development, that's odd. Is the production data corrupted?

DID I MENTION that in order to get into the system in development we have to load in a f*cking production database backup ON OUR DEVELOPMENT MACHINE and then ask one of the users' password to login to it and create an account for ourselves? Seeding? What's that, right?!

Anyway, back to bug fixing. I e-mail the the people responsible for the app and get a production admin account, oh I also can't ssh into it because of policies so I have to do everything over e-mail and figure out what's causing the errors. I somehow also wonder if they have any kind of virtualization in place, giving students a VM to do that stuff in doesn't seem so weird does it ? Even with school policies?

Oh btw, 'deploying' means sending a .zip file to a guy in another building and telling him how to configure it, apparently this resulted in a missing folder that the application needed to work and couldn't make on its own. This after 2 weeks of e-mailing back and forth.

After 3 months i quit out of despair and sadness, and due to the fact that I just couldn't do it anymore. I separated everything into logical subprojects and let the last guy handle it, he was OK with that and understood why I left.

Luckily, around that time I already had an actual job at a software development company :)

When the test database has better specifications & performance than the production one.. FML! 😭😭😭😭😭

I really think there should be a subject in every CS course to teach us how to handle/work-under Grade-A assholes and dumbfucks. Not that it would help, but atleast warn us on what we are getting into.

In my opinion, development is not *that* hard or frustrating but is made so by these shitty people. But again, what do I know.

I was scolded by my boss for using for-loop to iterate through an array recently. Apparently for-loop is not used in real world projects and this iteration should be done "in-memory". My colleagues and I are still trying to understand and process that.

I was asked to add fitbit integration to a project within 2 hours just because I had "already done it a week ago" in *another* project. Luckily, it was then given to a "senior" developer who took 4 days for it and essentially copy-pasted my work without much changes, ofcourse it stopped working every now and then.

I am given unreal deadlines on my tasks, on technologies I haven't worked on before, and then expected to churn out production ready code with no bugs in them.

My boss literally just sends me the links of 1st three google results on the problems I encounter and report, after humiliating me ofcourse. Yes, I did google it and yes I went through all I could find from Google forums to GitHub issues. When the library/plugin author himself says that this feature is not yet available, don't expect me to develop it in 2 hours you dumbfuck.

And for the love of God, please stop changing the data model every single day and justify it with agile development. Think before making any changes to it. Ever heard of Join queries? Foreign keys? Or any other basic database concepts.
We reached a point where each branch in the repo had different data model. Not kidding. And we were a team of just 4 developers. Atleast inform us when you change models after discussing it with your shit for knowledge "senior" developer, so we don't have to redo it all over again. The channels on slack are not for sharing random articles only.

I am just waiting to complete my year here.

I should have known what I got myself into the day he asked me to remove the comments I had added to explain what my code does. Why you ask? Because "we don't write comments".

Some Project Manager outsourced a redundant RADIUS setup with MySQL backend. We got 2 copies of a daloradius appliance running on Ubuntu 10.04. Once I saw this, I started to get a bit suspicious and requested to audit the system and database redundancy. With the system in production, and without getting back any documentation, I got into the VMs using the default root password. This was not even the worst part, as I found. One server was using a local MySQL instance, while the other was also using the first one's MySQL instance. When I reported this, I was told to comment clearly any changes to the configuration files, which resulted in commenting the word SHAME above each change.

I dunno about coolest, but I did sort of cement my reputation as the "database guy" in my first job because of this.

My first job was with a group maintaining a series of websites. Because of the nature of the websites, every morning we had to pull the records from one database on one network, sneaker net the data to a database on another network, and import the data via custom data import function.

However, the live site would crash after 100 or so records were imported. The dba at the live site had to script out a custom data partitioning script to do his daily duties, but it definitely messed up his productivity.

Turns out, the custom mass import function had recycled the standard import function, which was only used to import 1 record at a time, and it never closed its database connections, because it never needed to. A one line fix to production code was delivered 6 months later (because that was our release cycle) and I came up with the temporary work around, which was basically removing the connection limit. It would still crash with the work around, but only with multiple days worth of data. So basically only on Monday. Also developed the test set for the import (15k+ records).

Ok, here goes...

I was once asked to evaluate upgrade options for an online shop platform.

The thing was built on Zend 1, but that's not the problem.

The geniuses that worked on it before didn't have any clue about best practices, framework convention, modular thinking, testing, security issues...nothing!

There were some instances when querying was done using a rudimentary excuse for a model layer. Other times, they would just use raw queries and just ignore the previous method. Sometimes the database calls were made in strange function calls inside randomly loaded PHP files from different folders from all over the place. Sometimes they used JOINs to get the data from multiple tables, sometimes they would do a bunch of single table queries and just loop every data set to format it using multiple for loops.

And, best of all, there were some parts of the app that would just ignore any ideea of frameworks, conventions and all that and would be just a huge PHP file full of spagetti code just spalshed around, sometimes with no apparent logic to it. Queries, processing, HTML...everything crammed in one file...

The most amazing thing was that this code base somehow managed to function in production for more than 5 years and people actualy used it...

Imagine the reaction I got from the client the moment I said we should burn it to the ground and rebuild the whole thing from scratch...

Good thing my boss trusted me and backed me up (he is a great guy by the way) and we never had to go along with that Frankenstein monster...

Our production database has tables that exist only to be recompiled into a single table. Like, one record across 4 tables. Each record is only for one other record in the other 3 tables.

I LOST 2 HOURS DEBUGGING IT BECAUSE IT COULD NOT RECOMPILE THIS FUCKING TABLE. FUCK!

I'm gonna have to assume that the rants here that have nothing to do with a dev's life (usually someone's account of their sorry love life) are taken care of by the OP professionally outside of this platform, like I did.

Because if not, it's honestly sad and it can wear on our nerves.

Look, if you're gonna rant about your asshole cyka ex or this girl who never noticed you, and then put it in the rant category, at least find something in it remotely related to software development. Idk man, maybe talk about how your problem with your cheating gf made you delete the production database by accident...? Or how your medievalist boyfriend argued with you via email about your love for Arduinos then suddenly asked you to debug his Python code?

Or tag it "random" or "undefined" for the love of Mary Mother of God. Dev rants in the rant category are wonderful because I get to read about dev no-nos and fuckups man. We need our intellectual fix of dev rants man.

Of course the shouting episodes all happened during the era I was doing WordPress dev.

So we were a team of consultants working on this elephant-traffic website. There were a couple of systems for managing content on a more modular level, the "best" being one dubbed MF, a spaghettified monstrosity that the 2 people who joined before me had developed.

We were about to launch that shit into production, so I was watching their AWS account, being the only dev who had operational experience (and not afraid to wipe out that macos piece of shit and dev on a real os).

Anyhow, we enable the thing, and the average number of queries per page load instantly jumps from ~30 (even vanilla WP is horrible) to 1000+. Instances are overloaded and the ASG group goes up from 4 to 22. That just moves the problem elsewhere as now the database server is overwhelmed.

Me: we have to enable database caching for this thing *NOW*

Shitty authors of the monstrosity (SAM): no, our code cannot be responsible for that, it's the platform that can't handle the transition.

Me: we literally flipped a single switch here and look at the jump in all these graphs.

SAM: nono, it's fine, just add more instances

Me: ARE YOU FUCKIN SERIOUS?
Me: - goes and enables database caching without any approvals to do so, explaining to mgmt. that failure to do so would impair business revenue due to huge loading times, so they have to live with some data staleness -

SAM: Noooo, we'll show you it's not our code.
SAM: - pushes a new release of the monstrosity that makes DB queries go above 2k / page load -

...

Tho on the bright side, from that point on I focused exclusively on performance, was building a nice fragment caching framework which made the site fly regardless of what shitty code was powering it, tuned the stack to no end and learned a ton of stuff in the process which allowed me to graduate from the tar pit of WP development.

Living on the edge!

One or two years ago I managed to deploy a DDL change directly on the production server. As I knew there was a backup job which will run every day at noon and at midnight. So I run my script some minutes after noon. So far so good. But somehow I tested it badly in my test environment and the UI of the application throws error after error now in production.

Well, just revert the db to the latest recovery point with the backup, I thought.

It became clear then after a couple of minutes of searching the backup folder for the db backup that there was no such file. The youngest backup file was 3 years old.

Now what happened: The backup script had a switch "simulate=true" and then simulated a successful backup on each run. Therefore the monitoring system got no alerts for not correctly executing those jobs correctly. Then the monitoring job which should do the backupfolder surveillance stuck with green, because there was a valid backup file inside. But it did not check for a specific creation date.

Now this database is the one we need for doing our daily business and is really crucial. Therefore It was easier to emergencyfix the application than doing a rollback of the db 🙄

Well, not really a data loss story, but close to one.

Beware: Here lies a cautionary tale about shared hosting, backups, and -goes without saying- WordPress.

1. Got a call from a client saying their site presented an issue with a third-party add-on. The vendor asked us to grant him access to our staging copy.

2. Their staging copy, apparently, never got duplicated correctly because, for security reasons, their in-house dev changed the name of the wp-content folder. That broke their staging algo. So no staging site.

3. In order to recreate the staging site, we had to reset everything back to WP defaults. Including, for some reason, absolute paths inside the database. A huge fucking database. Because WordPress.

4. Made the changes directly in a downloaded sql file. Shared hosting, obviously, had an upload limit smaller to the actual database.

5. Spent half an hour trying to upload table by table to no avail.

6. In-house uploads a new, fixed database with the help of the shared hosting provider.

7. Database has the wrong path. Again.

8. In-house performs massive Find and Replace through phpMyAdmin on the production server.

9. Obviously, MySQL crashes instantly and the site gets blocked for over 3 hours for exceeding shared hosting limits.

10. Hosting provider refuses to accept this was caused by such a stupid act and says site needs to be checked because queries are too slow.

11. We are gouging our eyeballs as we see an in-house vs. hosting fight unfold. So we decide to watch a whole Netflix documentary in between.

12. Finally, the hosting folds and enables access to the site, which is obvi not working because, you know, wrong paths.

13. Documentary finishes. We log in again, click restore from backup. Go to bed. Client phones to bless us. Client’s in-house dev probably looking for a cardboard box to pack his stuff first thing in the morning. \_(ツ)_/¯

Sometimes for personal projects (and one client gig) I use the same database for local development and production.

Why?

Because I am a piece of shit.

I run update without where on mysql console on production database Today.

CLASSIC

Just because I needed to fix database after bug fix on the backend of the application.
I thought I wrote good sql statement after executing it on my local machine and then everything got bad.

Luckily it was only one column with some cached statistics data and I checked that it was not important data before I actually started fixing stuff but still ...
Almost got hard attack afterwards.
Made a script to fix this column and it took me only 15 minutes but still...

Bug was caused in part I got no unit tests and application grow after 3 years of development from simple one for one customer and volumes of documents around 50k to over 40 customers and volumes over 2mil per month, don’t know how many pages each, just in one year after we completed all needed features.

I have daily backups and logs of every api operation but still.
I think this got to far for one backend developer.

I got scared that I will loose money cause I am contractor and the only backend developer working on it.
I am so tired of this right now I think I need a break from work.

Responsibility is killing me so hard right now.
It will take a week to get back to normal.

MySQL is freaking and keeps restarting :(

Production server "Error establishing database connection"

FUCKKKKKKKKKKKKKKKK

Tonight, I gave birth to my first fully functional and deployment-ready dockerized application, "lestadium_web_1"!!
The big baby contains a Laravel showcase website with some React already in production! Big bonus with that, it's connected to a database, and I managed to setup some environment variables so nothing too dangerous is built within the image!

Fuck, that was exhausting but I'm so happy to finally understand how to make my stuff work, how they work and how to find some examples to get inspired from 😍

Imagine accidentally deleting the entire production database today and people thought you were joking.

If having a coffee in the morning doesn't wake you up, try deleting a collection or table in a production database instead.

Database is being slow AF again. Team lead is investigating. This is happening more often lately and affects both production and dev because everything is just in one gigantic database. So clients are calling support being angry about the speed they get and us devs get to twiddle our thumbs while waiting for our own data to load.

At what point do you stop optimizing queries and realize it's a database architecture, scaling problem?

We've been having production issues this week because a lot more users with more demands, and I'm going we need more servers... We can't just have one db, we need to parallelize like Hadoop...

Everyone else is going, how do we optimize queries, indexes, reduce the load...

I had rough week last week. Accidentally deployed dev code to production, soon found out there was no production version of code to do a redeploy. Deployed another app to production, it was working fine then another Dev changed a data type in the database from bool to nullable Boolean which broke some Linq queries. Looked like I deployed crap code. 3rd week with company.

Forgetting to write that WHERE statement, on a production database, during a serious case of Schroedinger's backup system, that failed.

....
oops

During the "how to install WordPress" the professor tells us to use the root user of the database.... Wooooooo yeah! Let's get fucked!

And he never even notes to them "DON'T use root in production!"

I can just imagine every one of the 40 people in attendance that day wondering how their website could POSSIBLY get hacked...

And they are are going to entering the industry, some of them as freelancers from the onset, thinking all that is ok.

If I had a penny for every time I have had a thought about deleting the production database.....

My client (not an engineer) has asked me access to production database and keeps asking an intern to directly query data from it instead of using the API.

One of our projects migrated their file-repository to another one during a major release.

Instead of giving this task to an experienced programmer, they gave it to the head of the respective dev department due to the usual release panic.

Soo.... He wrote the migration tool. It was executed during the release. Everything seemed fine so far.

A few days later. Someone from the above project came to my team due to some "strange behaviour on the production database".
They reported that they couldn't download some of the user's documents due to unknown reasons.

After quickly analyzing the current state of the new file-repository, we concluded that the affected documents did not exist in the new repository.

Then we took a look at the so called migration tool...
Well.. After nearly 30 min. we knew the root cause for that.
They only migrated the first 4 levels of the folder structure. Due to the assumption that "we don't use deeper nesting". (Facepalm)

As the head of their department wrote it, no one seems to questioned it either. Nor did they made a code review and ended up with a tool with hard coded urls to the production db, no version control, no build tool, no ci, nothing. Breaking nearly every possible company standard.

However.. That's not it. When analyzing their migration tool we noticed another even more dangerous thing.
They mixed up the id generation of the migrated documents resulting in a random assignment between customers and documents. Which is quite bad as this contains sensitive information. E.g. passports

They offered us quite a nice amount of money to fix this until EOB. We declinded as it was simply not possible in that time, but agreed to support them with the new tool.

After some time I heard that they migrated production again. And they fucked it up again. They never talked to us after we offered them support...

The third and final migration was written by us. Not only migrated it correctly. It was also way faster. By factor 20.

In the end we haven't gained anything from this rushed project as the penalties were piling up due to this fucked up migration.
After all this time I'm not sure who is to blame. In my opinion, partly all of them.
Head of department who can't and shouldn't code.
Seniors who didn't review the code and didn't ask for help.
Release mgmt who put way too much pressure on the devs.

Bad: Delete your production database
Good: Have a backup
Bad: Can't reimport it because your backup procedure uses scheme that are no longer supported for import by your cloud provider
Good: Backup are plaintext and somehow easy to parse
Bad: Spending the rest of the day writing scripts to reinsert everything.

End of the story: everything is up and running, 8hours of efforts

Comsci teacher just ran "php artisan migrate:fresh" on the production database

Weeks ago, a change went into production. For some reason, we can't implement our own changes or create new databases in production, we have to have a whole different department do it. This would be great except for one thing:

THEY CAN'T THINK FOR THEMSELVES. I've had to tell them how to run scripts I wrote. I've had to tell them how to fix problems that arise.

Back to that script ran three weeks ago or so. It didn't add permissions to allow me, the system and application developer to see the stored procedure, much less run it. Application can't run it. Thankfully the application works without it.

Fast forward to tonight. My change that I'm attempting to implement is the creation of the stored procedure, because nothing could see it, I assumed it didn't exist... reasonable, right? Database folks tells me it exists. They then tell me they can't give me nor the application permissions because it doesn't ask for it in the change plan.

Excuse me.... WHAT FUCKING WORLD DOES IT MAKE SENSE TO CREATE SOMETHING AND HIDE IT FROM THE CREATOR LET ALONE THE APPLICATION SO IT CAN'T USE IT?! FUCKING THINK. WHY WOULD I WASTE MY FUCKING TIME TO TALK TO YOU OFFSHORE PIECES OF SHIT AT 10PM WHEN I'D RATHER PLAY VIDEO GAMES.

I'm so fucking done with enterprises. Someone with reasonable job security at a startup, please hire me. You will probably pay me more fucking money than this company does anyway.

Now on to my second change of the night. Thankfully I don't have to rely on anyone outside of me... so I won't be wasting my fucking time.

Shoutout to all the devops people currently on their parent's WiFi trying to fix a database issue in production

So it's Friday afternoon just before a bank holiday weekend here I'm the UK, perfect time for our production database to go TITSUP (total inability to support usual performance), life sucks then you die folks....

When your company expects you to manually change information in the production db by saying "hey, client Billy wants his stuff moved back to where it was"

Haven't ranted for awhile but here it goes...
In a meeting with a front end user yesterday. They don't like the entry screens on our Oracle ERP system. They want us to provide them with a tool so they can create new entry screens to replace those they dont like. They want full autonomy over that tool and no interference from IT. Oh, and they want unfettered database access to the production data, including full ability to execute DML. I so wanted to say 'Are you high?'.

If having a coffee in the morning doesn't wake you up, try deleting a table in a production database instead.

Mine: not listening to early warning sings that shit won't work. As a result we implemented the wrong technology and had to roll back after a month in production. Lost some customers data, though we had backups, so it was just a delta of a day or so. Still, not pretty, had a rough night writing scripts at 3am to check data validity after.

I will throw in another one from a colleague of mine. He was running some database migrations and ended up pointing the DB migration tool to production by accident. Oops. That one required a restore as well, if I remember correctly.

Developed an update to our database procedures and tested it with local copy. After a few days everything was ready. Opened our server and started the update. After a couple of rows an error occurred. Turns out our production db is older version and does not support some syntax I used. It became a bit longer day at work...

What's your workspace setup?

Curious because it took awhile and a lot of experimenting/thinking to get mine setup the way it is, but now I can't even think properly unless I have things setup that way after booting up in the morning.

Here goes:

Workspace 1: General stuff, personal email. social media, random research for non work related things, etc

Workspace 2: My main project local development, includes terminals, database, browser research for bugs, debugging software, error logs, etc.

Workspace 3: My main project, production workspace, consoles, browser, etc related to production server, you get the idea

Workspace 4: local dev on my side project

I found it crucial to setup workspace 2 and 3, it has helped me avoid countless stupid errors, like, for example, accidentally working on production terminal and wanting to rip my hair out wondering why the fuck _____ isn't working, then realizing, oh shit, i'm on production, not local. Huge brainspace bandwidth saver when I setup like this.

How about you?

delete random, unused column from production database
entire app blows up
re-release last change on code pipeline
everything works beautifully
¯\_(ツ)_/¯

5am now and trying to do a production deployment. It just failed because boss loves making un-approved changes to prod that change entire database schemas. Argh!!

Spent all morning debugging legacy code that I need to migrate.

Most of the time is just waiting for it to load --pieces of data-- entire tables from the database and then filter out the records it doesn't want using some app logic.

WHAT SORT OF MONKEY WRITES CODE LIKE THIS? HOW WAS THIS EVEN ALLOWED INTO PRODUCTION...

I have to open Notepad to write down my chain of thoughts, steps, and things to check once the next breakpoints are hit so that I don't forget them.

So in theory I'm being paid all morning to sit around and do nothing.

That sounds great but I'm falling asleep... Shoulda worked from home...

What was I saying again...yea...

DON'T HIRE MONKEYS!!! THEY WRITE SHIT CODE THAT WASTES EVERYONE'S TIME EVEN AFTER THEY LEAVE...

I'm going to lunch now... Hopefully Notepad has enough into for me to remember what I was doing...

This one was from my Tech Lead, She had to update phone number of a customer in Database on production server. And guess what she forgot 'WHERE'. Next we were facing each other with poker face. :|

p.s: fortunately we had backup of just 4 hours back. Still we lost data of about 100 people.

Oooh I have quite a few,
My favourite: accidently left a log. Debug("bollocks") in a try catch this made it through testing and does (still) occasionally go into production log files.

Worst: wrote an interceptor for jboss with the intent of checking cache for some lookup data. I picked the wrong one of two similarly named methods and instead queried the database, I effectively wrote a denial of service utility into our app

Quickly delete a double record in the production database with a script, just forgot the where statement...

A prototype being used as production code written in procedural PHP with the code drawn using echo and MySQL (not MySQLi) all mixed together and the configuration with world readable database stored as config.inc.
All backed by a database with no foreign keys or data integrity of any kind.

Had to do a WordPress installation move from development to production server. Exported database, changed hostname, imported database, settings missing.
Guess who's manually applied all settings and backed up the > 5GB old site.

I really hate hoster that don't allow SSH tunneling for SQL access 😩

(First rant. Just felt like I might put my struggles into a rant and it might just feel better.)

First rant here...
Hand full of devs have to create a huge web platform that can shovel a lot of data around in about two months which is impossible...
Project lead has left major decisions in the hands of interns like database we want to use because no question can.be answered by that person. Inexperienced intern has chosen a fucking nosql database for highly relational datasets... why? Because new tech...
Development began and a bunch of problems arised... database was accessable from internet from day one. Random crashes because out of memory exceptions. Every possible feature had a description of at most 10 words... and no standards where enforced on anything.

Now that finaaaally we switch to sql after almost a year of prototypical production everybody keeps coding on new features so i have to port all the crap to the new database...

best part: a bunch of clients on different op systems have to be ported as well!

Even better part: i have to do that cause everybody else has practically no experience in any field...

And now the joke: i got hired for gui/desktop application development

Am i a wizard now?

Ok so some of you have probably seen my previous rants about my computer science teacher and our project but I'm just going to summarize all of them and share with you more of my pain.

1. He edits in the production environment. Its a laravel project and he is creating test database migrations IN THE PRODUCTION ENVIRONMENT AND SWITCHING BACK AND FORTH FROM MASTER AND DEV.

2. He edits in vim and doesn't follow codestyle even though I printed him off a piece of paper and emailed it to him.

3. He doesn't have any ethic when it comes to more complex things like laravel homestead.

4. He doesnt want me to release features even though he takes really long to do them.

While I love vim and it is my editor of choice, some things should be done in an ide. This is really annoying me and I'm really just considering handing him the project if he can't follow basic outline.

I was working for a project with one of the project managers. Despite several discussions, he was not ready to have provisioned for procurement of couple of extra drives for database backups. Also because it's always how they worked, developers were allowed to make changes to the production databases directly.

Since I knew it was going to be burning some day, despite his negligence, I ran a script to take full database backups every night, compress, and remove old backups all to do in the drives we had on server. Sat it automated using scheduler.

One day it happened that one of the junior developers deleted one major table taking whole production down. Next thing you know everyone went crazy. Since I felt bad for the managers and users, I was able to restore database using backup from last night.

You know who jumped in first before senior management to take credit of all this and got some nice kudos..that project manager. Also, you know who got burned..it would not be a rant if I did not got schooled for not following on the wisdom of project manager.

Anyways, we are still not taking database backups (as per project manager)

I can set up website in production doing both backend w database and front end with js csd html. But my design skills are not great. I prefer being behind the scenes but am i selling myself short? Am i really a full stack developer who just stinks at css?

Today at work I started doing 1 month old task with production problem.

First of all why now ?
Because I already fixed all the other urgent production problems I had during last month, done about 4 deployments of those super urgent errors.
Now I can start with not trivial one that are pending for quite time.
I am the only backend developer in this project ...

This is a dtp application and the problem is that we are not verifying if we got all fonts embedded in customer provided pdf files.

We are generating high quality images of those pdf for printing just fine from the beginning but now we need valid PDF with all fonts embedded in it. ( don’t ask me why I am only a hammer in this process )

After running simple test using python script against database it turned out we have over 500 broken PDF files without fonts.

So I guess I have just one sentence to say about it.

Fuck you PDF format for not being strict and allowing this shit.

Fire every single teacher who runs from self education. Someone who keeps themselves up to date should be keeping students up to date.

Secondly, assign each class a major project which starts from 4th semester with one of the faculty acting as the Project Manager. Allow each student to choose their area of interest and work on that module. This will help develop team work and teach how not to rm rf production server or db:drop production database ^.^

This is why code reviews are important.

Instead of loading a relevant dataset from the database once, the developer was querying the database for every field, every time the method interacted with it.

What should have been one call for 200k records ended up as 50+ calls for 200k records for every one of 300+ users.

The whole production application server was locked.

Okay, so I'm developing a system for a global rent-a-car broker. Basically website + a bunch of third party APIs + analytics, it's been running in production for over 4 years now.

Anyway, we had to connect our system to an external rental insurance API, nothing too complicated, got it to production in a month and it seemed to work okay, except the insurance provider claimed they're not getting any analytics data, which was weird, because there were no errors with API calls, and customers had no problem with the insurance.

After going back and forth for a month, we finally figured what's going on: after each API request, the insurance provider expected us to send the exact same data to their analytics API, because for whatever dumb reason they were unable to internally log requests in their analytics database.

tl;dr: we're doing 2 API calls with the exact same data to different endpoints, because a large rent-a-car insurance provider can't log their own analytics data.

Im on it, all my rants refere to it:
780 columns table, pushing unfinished changes to master and recently the new hire pushed a 700mb database bkp to production... I had to learn how to clean that shit on git and rebuild the repository...

How can someone not realize they are pushing 700mb to gitlab?
That shit must have taken ages... I realized because git pull was taking too long...

Purging my database before going into production.

Ah, I remember the user asdfdfdf with the password asdfASDF123 so well. Good guy.

When a safety check you added to a cron script triggers early in the morning and you know you might be walking into a shitstorm.

Still better than not having that check and replicating an incomplete database to production.

When your IT VP starts speaking blasphemy:

"Team,
We all know what’s going on with the API. Next week we may see 6x order volumes.

We need to do everything possible to minimize the load on our prod database server.

Here are some guidelines we’re implementing immediately:

· I’m revoking most direct production SQL access. (even read only). You should be running analysis queries and data pulls out of the replication server anyway.
· No User Management activities are allowed between 9AM and 9PM EST. If you’re going to run a large amount of updates, please coordinate with a DBA to have someone monitoring.
· No checklist setup/maintenance activities are allowed at all. If this causes business impact please let me know.
· If you see are doing anything in [App Name] that’s running long, kill it and get a DBA involved.

Please keep the communication level high and stay vigilant in protecting our prod environment!"

RIP most of what I do at work.

This friend of mine accidentally deleted client data in their production database, DBAs gotta work paid overtime to fix the mess...at least he only got away with a written warning. Dude coulda been fired on the spot

This one time last year a colleague found out that some data went missing and suggested to recover the data from a backup. When trying to create a new database instance in the Google Cloud Platform (if everything works it's amazing!) it failed.
Not knowing why this happened, I tried to revert that backup to the production database, after creating a backup using the GCP. Needless to say that failed as well, resulting in a corrupted database instance where I couldn't access the created backups anymore.
This all went at around 10pm and the only users of our product are currently in the same timezone and use it from around 7.30AM until 6PM so no one besides our team knew the server was down.
After a long night chatting Google's support team the database was successfully recovered and the only harm done was sleep depravation for me and a colleague.

Apparently there was a bug in the GCP. It was resolved in two hours and the last time a breaking bug was in that piece was more than seventy days earlier.

I did at least learn to create local backups as well, instead of relying on the tools of the same product...

Best: the moment I saw the corrupted database spin up again and not losing my job because of it.

Need some advice here.

So hello everyone! I recently moved abroad for work, for the sake of the experience and the excitement of learning how developers in Latin America tackle specific problems. To my surprise, the dev team is actually composed solely of Europeans and Americans.

I work for a relatively new startup with an ambitious goal. I love the drive everyone has, but my major gripe is with my team lead. He's adverse to any change, and any and all proposals made to improve quality of throughput are shot down in flames. Our stack is a horrendous mess patched together with band-aids, nothing is documented, there are NO unit tests for our backend and the same goes for our frontend. The team has been working on a database/application migration for about a month now, which I find ridiculous because the entire situation could have been avoided by following very rudimentary DevOps practices (which I'm shunned for mentioning). I should also add that for whatever reason containerization and microservices are also taboo, which I find hillarious because of our currently convoluted setup with elastic beanstalk and the the constant complaints between our development environment and production environments differing too much.

I've been tasked with managing a Wordpress site for the past 3 weeks, hardly what I would consider exciting. I've written 6 pages in the past two weeks so our marketing team can move off of squarespace to save some money and allow us more control. Due to the shit show that is our "custom theme" I had to write these pages in a manner that completely disregard existing style rules by disabling them entirely on these pages. Now, ironically they would like to change the blog's base theme but this would invertedly cause other pages created before I arrived to simply not work, which means I would have to rewrite them.

Before I took the role of writing an entire theme from scratch and updating these existing pages to work adequately, I proposed moving to a headless wordpress setup. In which case we could share assets in a much more streamline manner between our application and wordpress site and unify our styles. I was shot down almost immediately. Due to a grave misunderstanding of how wordpress works, no one else on the team seems to understand just how easy it is to fetch data from wordpress's api.

In any event, I also had a tech meeting today with developers from partner companies and realized no one knew what the fuck they were talking about. The greater majority of these self proclaimed senior developers are actually considered junior developers in the United States. I actually recoiled at the thought that I may have made a great mistake leaving the United States to look a great tech gig.

I mean no disrespect to Latin America, or any European countries, I've met some really incredible developers from Russia, the Ukraine, Italy, etc. in the past and I'm certainly not trying to make any blanket statements. I just want to know what everyone thinks, if I should maybe move back to the states and header over to the bay/NY. I'm from the greater Boston area, where some really great stuff is going on but I guess I also wanted a change of scenery.

Soo, lets talk about deadlines and how often I come in trouble when there is one near. This week I have (had) three. So I had a lot of problems this week.

But, after all today was a really good day.

I deleted an production database, fucked up my laravel migrations and tomorrow we have to deploy another application. The app is still in development and the site isn´t finished yet.

Friday I need to launch another webapplication. Because I don't want to deploy webapps and apps on friday and I go snowboarding next week, I somehow convinced my client to launch it next week. He agreed with me, so that's a burden off my shoulder. And I'll have an extra day for fixing the last bugs.

Today I worked really hard and almost finished the last application. So I think after all I'll somehow manage it.

Made a mysql innodb database go do a fancy shuffle like schrödingers cat, with many tables appearing as "ghosts", in production.

still not sure what went wrong.
i noted it was extremely slow and thought restarting the service might Help it...

Sooooo I came in to work yesterday and the first thing I see is that our client can't log on to the cms I set up for her a month ago. I go log in with my admin credentials and check the audit logs.

It says the last person to access it was me, the date and time exactly when we first deployed it to production.

One month ago.

I fired a calm email to our project managers (who've yet to even read the client complaint!) to check with ops if the cms production database had been touched by the ops team responsible for the sql servers. Because it was definitely not a code issue, and the audit logs never lie.

Later in the day, the audit log updated itself with additional entries - apparently someone in ops had the foresight to back up the database - but it was still missing a good couple weeks of content, meaning the backup db was not recent.

Fucking idiots.

So yesterday at a client location, our support guy called me and said this thing is trimming the characters whenever I save it. It was a ckeditor in our application, so basic troubleshooting was to check the system configuration for that page and the ckeditor configuration.
Checked the system configuration, ckeditor configuration, found nothing.
Out of curiosity, checked the schema for the table in which the data is stored, so one of the idiots took the backup of original table and appended it with the date time on which it was backed up. And created a new table with field data type of varchar with a 255 limit.
This was in UAT server as well as Production server. Changed the field type to text again in UAT. Asked to team to get the same thing done on Production server as well.

How do you quickly sync your local dev database with production database?

First year on the job. Was already good at writing software, but bad at practices and administration. One such software was being tested live, while still in development. I was developing on the production database... .
Yeah.
I was working on an edit feature of sales records, in a table that already contained hundreds of subsidized sales of very expensive products. Based on that, the supplier had to compensate the shops with half the price of every item.
I forgot to add a where clause to the update. Lost all sales data. On production.
Asked the admin if there are backups and he says yes, checks to discover that the backup script failed for the last week (since it became live)

Whole thing was incredibly stupid. I made a ton of stupid mistakes, and so did the other people involved. The loss was around 1 year of my income. Luckily the client decided to brush it off as losses and claim some tax benefits and it all ended well.

Just joined a new company and can only describe the merge process as madness.....is it or am I the one that is mad?!

They have the following branches:

UAT#_Development branch
UAT#_Branch (this kicks of a build to a machine named UAT#)

Each developer has a branch with the # being a number 1 to 6 except 5 which has been reserved for UAT_Testing branch.

They are working on a massive monolith (73 projects), it has direct references to projects with no nuget packages. To build the solution requires building other solutions in a particular order, in short a total fucking mess.

Developer workflow:

Branch from master with a feature or hotfix branch

Make commits to said branch and test manually as there are no automated tests

Push the commits to their UAT#_Development branch, this branch isn't recreated each time and may have differences to all the other UAT#_Development branches.

Once happy create a pull request to merge from UAT#_Development to UAT#_Branch you can approve your own pull request, this kicks off a build and pushes it to a server that is named UAT#.

Developer reviews changes on the UAT# server.

QA team create a UAT/year/month/day branch. Then tell developers to merge their UAT#_branch branches in to the previously created branch, this has to be done in order and that is done through a flurry of emails.

Once all merges are in it then gets pushed to a UAT_Testing branch which kicks off a build, again not a single automated test, and is manually tested by the QA team. If happy they create a release branch named Release/year/month/day and push the changes into it.

A pull request from the release branch is then made to pre-live environment where upon merge a build is kicked off. If that passes testing then a pull request to live is created and the code goes out into production.

Ahhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh it's a total mess. I knew when I took on this job it would be a challenge but nothing has prepped me for the scale of the challenge!! My last place it was trunk based development, commit straight to master, build kicks off with automated testing and that just gets pushed through each of the environments, so easy, so simple!

They tell me this all came about because they previously used EntityFramework EDMX models for the database and it caused merge hell.

Sometime ago I had to debug a override to a Joomla component made by the trainne.

So the component work just fine on development but not in production. I made the transition myself so I knew the code was the same.

This is my first rant, quite long, hope you like it.

So I thought something went wrong in the database, I searched and didn't find a thing.

Then I decided to check the code... hold yourself now

The component had to support internalization ..so instead of using the proper function to see what the current language is, he made a if with the URL hard coded..

Like http:://dummy.com/en/

:)

The PO asked if there were a couple of scripts he could run to get counts of specific things in the production database.

after a few days and iterations I'm now getting suggestions from him on how to create and drop temp tables

The highlighted lines are a part of a flask app I'm writing in Python2(not python3 because I'm a bit too lazy to fix few dependency errors). All functions work as expected and all templates are rendered individually, and routes are all defined. check_date checks for invalid dates like 32Jan, 2018. It returns 0 if date is valid. add_data basically returns 0 if it decides to add user data into the database(db).
The problem is that line 60 renders but lines 54,57 don't. Any ideas as to what might be going wrong.?
PS: I'm building this app for learning and not for a production environment...

Risk is part of my everyday life.

I take the risk everyday when opening IDE and changing line of code that can either break database or crash other systems that are depending on one I am developing. ( not instantly but in some time in the future )

So....

Many years ago I was updating some application server production code while being drunk.
Everything went fine except me waking up in the morning and didn’t remember how I did it.

... what I learned from my developers life except that heavy drinking and updating servers is not the best idea ?

First, don’t give a fuck, do your job and ask questions even if the person in front of you said that understood everything and you think you understood all of shit.
Second, if you think you know what to do think twice.
Third, having any backup, any tests and any documentation is always better then having nothing.

And the most important.
The most risky in every business are people around you, so always have good people around and there would be no risk at all or you won’t even think about it.

✌🏽 ❤️

Since this has been trending recently, here's my six word horror story -

"I accidentally deleted the production database"

I had a discussion with my colleagues about my bachelor thesis.

Together we created within the last 18 month a REST-API where we use LDAP/LMDB as database (tree structured storage). Of course our data is relational and of course we have a high redundancy there. It's a 170 call API and I highly doubt that it's actually conforming REST.
Ensuring DB integrity is done in the backend and coding style there is "If we change it at one place, let's make sure to also change it everywhere else", so you get a good impression how much of spaghetti code we have there.

Now I proposed to code a solution in my bachelor thesis where we use a relational database (we even have an administrated Oracle DB with high availability) and have a write-only layer to also store the data in LDAP but my colleagues said that "it would add too much complexity to the system".
Instead I should write the relational layer myself and fetch the data somehow from the existing LDAP tree.

What the actual fuck, spaghetti code is what makes the system really unnecessarily complex so that no one will understand that code in 2 years.

Congratulations, you just created legacy code that went into production in 2018 while not accepting the opportunity to let that legacy code get eliminated.
Now good luck with running and maintaining that system and it's inconsistencies.

I had a fun one for one of myself but then I heard a "technical architect" at my company used the same login and password for a project's production and development databases.

I then had to deal with the crisis of a new dev on that project blowing the entire production database to smithereens because of said decision.

How the fuck does someone not check from Middleware server to database? Going to production and we find that was never checked.... It's 1130pm and now I have to wake people up on a Friday night...

I have frequent dreams of accidentally dropping a production database thinking that it's local.

Put a stored procedure into production which exclusively only used synonyms to another server’s database, and didn’t actually use any of its own DB’s data..
Didn’t want to add all the scaffolding of a second connection to the app. Oh well!

Not writing enough utility programs for those occasions where you have no option but directly modify database tables in production.

Right now?

Partial

Because it's given me a measurable performance gain in an app that (with the production database loaded) takes hours to complete it's task.

So after the CIO pretty much does a table flip to the division about causing near daily customer impact we now have seven business day cool period for Changes and 13 cross functional teams stood up with 1, 15, 30, 60, and 90 day deliverables to an executive audience.

Guess what my team did today?
Offline a major Production database during the middle of the day, thinking they were in Development. Didn’t notice for 40 minutes.

We have a CRM running on an EC2 instance. We need to clone it so we can test a tool on the replica. We tried cloning it directly, sharing the AMI and creating a new instance through it but it always redirects changes to the original production server. The database is on the instance only and static files are stored in S3. Can someone guide me or share some resourses on how to do this.

So i've not slept all night, come into work and my first mistake 1 hour in is migrating data from another project into a production database now i have redundant data and redundant tables, well done! fml