Join devRant
Do all the things like
++ or -- rants, post your own rants, comment on others' rants and build your customized dev avatar
Sign Up
Pipeless API
From the creators of devRant, Pipeless lets you power real-time personalized recommendations and activity feeds using a simple API
Learn More
Search - "database mistake"
-
In a user-interface design meeting over a regulatory compliance implementation:
User: “We’ll need to input a city.”
Dev: “Should we validate that city against the state, zip code, and country?”
User: “You are going to make me enter all that data? Ugh…then make it a drop-down. I select the city and the state, zip code auto-fill. I don’t want to make a mistake typing any of that data in.”
Me: “I don’t think a drop-down of every city in the US is feasible.”
Manage: “Why? There cannot be that many. Drop-down is fine. What about the button? We have a few icons to choose from…”
Me: “Uh..yea…there are thousands of cities in the US. Way too much data to for anyone to realistically scroll through”
Dev: “They won’t have to scroll, I’ll filter the list when they start typing.”
Me: “That’s not really the issue and if they are typing the city anyway, just let them type it in.”
User: “What if I mistype Ch1cago? We could inadvertently be out of compliance. The system should never open the company up for federal lawsuits”
Me: “If we’re hiring individuals responsible for legal compliance who can’t spell Chicago, we should be sued by the federal government. We should validate the data the best we can, but it is ultimately your department’s responsibility for data accuracy.”
Manager: “Now now…it’s all our responsibility. What is wrong with a few thousand item drop-down?”
Me: “Um, memory, network bandwidth, database storage, who maintains this list of cities? A lot of time and resources could be saved by simply paying attention.”
Manager: “Memory? Well, memory is cheap. If the workstation needs more memory, we’ll add more”
Dev: “Creating a drop-down is easy and selecting thousands of rows from the database should be fast enough. If the selection is slow, I’ll put it in a thread.”
DBA: “Table won’t be that big and won’t take up much disk space. We’ll need to setup stored procedures, and data import jobs from somewhere to maintain the data. New cities, name changes, ect. ”
Manager: “And if the network starts becoming too slow, we’ll have the Networking dept. open up the valves.”
Me: “Am I the only one seeing all the moving parts we’re introducing just to keep someone from misspelling ‘Chicago’? I’ll admit I’m wrong or maybe I’m not looking at the problem correctly. The point of redesigning the compliance system is to make it simpler, not more complex.”
Manager: “I’m missing the point to why we’re still talking about this. Decision has been made. Drop-down of all cities in the US. Moving on to the button’s icon ..”
Me: “Where is the list of cities going to come from?”
<few seconds of silence>
Dev: “Post office I guess.”
Me: “You guess?…OK…Who is going to manage this list of cities? The manager responsible for regulations?”
User: “Thousands of cities? Oh no …no one is our area has time for that. The system should do it”
Me: “OK, the system. That falls on the DBA. Are you going to be responsible for keeping the data accurate? What is going to audit the cities to make sure the names are properly named and associated with the correct state?”
DBA: “Uh..I don’t know…um…I can set up a job to run every night”
Me: “A job to do what? Validate the data against what?”
Manager: “Do you have a point? No one said it would be easy and all of those details can be answered later.”
Me: “Almost done, and this should be easy. How many cities do we currently have to maintain compliance?”
User: “Maybe 4 or 5. Not many. Regulations are mostly on a state level.”
Me: “When was the last time we created a new city compliance?”
User: “Maybe, 8 years ago. It was before I started.”
Me: “So we’re creating all this complexity for data that, realistically, probably won’t ever change?”
User: “Oh crap, you’re right. What the hell was I thinking…Scratch the drop-down idea. I doubt we’re have a new city regulation anytime soon and how hard is it to type in a city?”
Manager: “OK, are we done wasting everyone’s time on this? No drop-down of cities...next …Let’s get back to the button’s icon …”
Simplicity 1, complexity 0.16 -
I’m so mad I’m fighting back anger tears. This is a long rant and I apologize but I’m so freaking mad.
So a few weeks ago I was asked by my lead staff person to do a data analysis project for the director of our dept. It was a pretty big project, spanning thousands of users. I was excited because I love this sort of thing and I really don’t have anything else to do. Well I don’t have access to the dataset, so I had to get it from my lead and he said he’d do it when he had a chance. Three days later he hadn’t given it to me yet. I approach him and he follows me to my desk, gives me his login and password to login to the secure freaking database, then has me clone it and put it on my computer.
So, I start working on it. It took me about six hours to clean the database, 2 to set up the parameters and plan of attack, and two or three to visualize the data. I realized about halfway through that my lead wasn’t sure about the parameters of the analysis, and I mentioned to him that the director had asked for more information than what he was having me do. He tells me he will speak with director.
So, our director is never there, so I give my lead about a week to speak with her, in the mean time I finish the project to the specifications that the director gave. I even included notes about information that I would need to make more accurate predictions, to draw conclusions, etc. It was really well documented.
Finally, exasperated, and with the project finished but just sitting on my computer for a week, I approached my director on a Saturday when I was working overtime. She confirmed that I needed to what she said in the project specs (duh), and also mentioned she needed a bigger data set than what I was working with if we had one. She told me to speak to my lead on Monday about this, but said that my work looked great.
Monday came and my lead wasn’t there so I spoke with my supervisor and she said that what I was using was the entire dataset, and that my work looked great and I could just send it off. So, at this point 2/3 of my bosses have seen the project, reviewed it, told me it was great, and confirmed that I was doing the right thing.
I sent it off to the director to disseminate to the appropriate people. Again, she looked at it and said it was great.
A week later (today) one of the people that the project was sent to approaches me and tells me that i did a great job and thank you so much for blah blah blah. She then asks me if the dataset I used included blahblah, and I said no, that I used what was given to me but that I’d be happy to go in and fix it if given the necessary data.
She tells me, “yeah the director was under the impression that these numbers were all about blahblah, so I think there was some kind of misunderstanding.” And then implied that I would not be the one fixing the mistake.
I’m being taken off of the project for two reasons: 1. it took to long to get the project out in the first place,
2. It didn’t even answer the questions that they needed answered.
I fucking told them in the notes and ALL THROUGH THE VISUALIZATIONS that I needed additional data to compare these things I’m so fucking mad. I’m so mad.15 -
This is going to be a long rant, coz this is the only way to vent out my frustration against our tech head.
Yesterday, while our fucking twat tech head was playing around in company aws account, he terminated the production server. By mistake, apparently. Coz he doesn't know shit about server management. But that egoist ass won't admit and fucked the production server.
And then ran away. We developers sprang into action. Updated dns to point to staging server, setup virtual hosts, env files, point to prod database, force flush dns cache. All systems were up and running in 30 mins. And since it was staging server, it had lot of untested features and codes, and we spent rest of the day fixing the bugs.
And that tech head, who ran away hiding his tail between his legs, after he fucked the server, came back after systems were up. And started cracking jokes, that "so many features got released in 1 day" . "We cut server cost by shutting down 1 server."
We were struggling and working in full throttle to make the services running again. And that fuckity fucker was cracking jokes.
And I don't even know what excuse he gave to ceo for the downtime. I am pretty sure he would have made up some crappy excuse to hide his fucking mistake. That ass never admits his mistake. I am thinking to go to ceo today and tell the real story and get that faggot head fired or at least a strict warning.4 -
Short version:
Dear devRantdairy,
today I was stupid.
The End.
Full version:
I am working on some messaging system, trying to use less as possible overhead sending data. Therefore there of course are asynchronous calls and some templating. But that's just the setting of the rant: I designed an architecture to save conversations in a database. Working with transactions in pdo I wrote a query wich in my eyes should have worked well. But the result just didn't appear in the table. So I started debugging data. Recreated the table. Rewrote the query. Went to bed. Woke up. Further tryed to make this work. And in the end I realized I just forgot to commit the transaction.
How dumb can you be? There's way too much time gone for that mistake. Is there a hole? I want do dig myself.9 -
Prologue
My dad has an acquaintance - let's call him Tom. Tom is an gynecologist, one of the best in Poznań, where I live. He's a great guy but absolutely can not into tech of any kind besides his iPhone and basic PC usage. For about a year now I've been doing small jobs for him - build a new PC for his office, fix printer, fix wifi, etc. He has made a big mistake few years ago by trusting a guy, let's call him Shitface, with crating him software for work. It's supposed to be pretty simple piece of code in which you can create and modify patient file, create prescription from drugs database and such things. This program is probably one of the worst pierces of code I've ever seen and Shitface should burn for that. Worse, this guy is pretentious asshole lacking even basic IT knowledge. His code is garbage and it's taking him few months to make small changes like text wrapping. But wait, there's more. Everything is hardcoded so every PC using this software must have installed user controls for which he doesn't have license and static IP address on network card.
Part 1
Tom asked me to build him a new PC that will be acting like a server for Shitface's program. He needs it in Kalisz (around 150 km from my place). I Agred (pun intended) and after Tom brought me his old computer I've bought parts and built a new one. I have also copied everything of value and everything took me around three hours.
Part 2
Everything was ready but Shitface's program. I didn't know much about it's configuration so when I've noticed that it's not working even on the old PC I got a bit worried. Nevertheless I started breaking everything I know about it and after next three hours I've got it somewhat working. Seeing that there's still some problems with database connection (from Windows' Event Viewer) I wrote quick SMS to Shitface asking what can be wrong. He replied that he won't be able to help me any way until Monday (day after deadline). I got pissed and very courteously asked him for source code because some of libraries used in this project has license that requires either purchase of commercial license or making code open source. He replied within few minutes that he'll be able to connect remotely within next 10 minutes. He was trying to make it work for the next hour but he succeeded. It was night before deadline so I wrapped everything up and went to bed thinking that it won't take me more than an hour to get this new PC up and running in the office. Boy was I wrong.
Also, curious about his code, I've checked source and he is using beautiful ponglish (mixed Polish and English) with mistakes he couldn't even bother to fix. For people from Poland, here's an example:
TerminarzeController.DeleteTerminarzShematyDlaLekarza
Part 3
So I drove to Kalisz and started working on making everything work. Almost everything was ready so after half an hour I was done. But I wanted to check twice if it's all good because driving so far second time would be a pain. So I started up Shitface's program, logged in, tried to open ANYTHING and... KABUM. UNHANDLED EXCEPTION. WTF. I checked trace and for fuck sake something was missing. Keep in mind that then I didn't know he's using some third party control for Windows Forms that needs to be installed on client PC. After next fifteen minutes of googling I've found a solution. I just had to install this third party software and everything will work. But... It had to be exactly this version and it was old. Very old. So old that producent already removed all traces of its existence from their web page and I couldn't find it anywhere. I tried installing never version and copying files from old PC but it didn't work. After few hours of searching for a solution I called Mr Shitface asking him for this control installation file. He told me that he has it but will be able to send it my way in the evening. Resigned I asked for this new PC to be left turned on and drove home. When he sent me necessary files I remotely installed them and everything started working correctly.
So, to sum it up. Searching for parts and building new PC, installing OS and all necessary software, updating everything and configuring it for Tom taste took me around what, 1/3 of time I spent on installing Mr Shitface's stupid program which Tom is not even happy with. Gotta say it was one of worst experiences I had in recent months. Hope I won't have to see this shit again.
Epilogue
Fortunately everything seems to work correctly. Tom hasn't called me yet with any problems. Mission accomplished. I wanna kill very specific someone. With. A. Spoon.1 -
At my old job, me and a colleague were tasked with designing a new backup system. It had integrations for database systems, remote file storage and other goodies.
Once we were done, we ran our tests, and sure enough. The files and folder from A were in fact present at B and properly encrypted. So we deployed it.
The next day, after the backup routine had run over night, I got to work and noone was able to log in. They were all puzzled.
I accessed a root account to find the issue. Apparantly, we had made a mistake!
All files on A were present at B... But they were no longer present at A.
We had issued 'move' instead of 'copy' on all the backups. So all of peoples files and even the shared drives have had everything moved to remote storage :D
We spent 4 hours getting everything back in place, starting with the files of the people who were in the office that day.
Boss took it pretty well at least, but not my proudest moment.
*Stay tuned for the story of how I accidentally leaked our Amazon Web Services API key on stack overflow*
/facepalm5 -
There was a time I made an update on one of our client's e-commerce website sign-up page. The update caused a bug that allowed new users to create an account without actually creating an account.
The code block meant to save user credentials (i.e email address and password) to the database was commented out for some reasons I still can't remember to this day. After registration new users had their session created just as normal but in reality they have no recorded account on the platform. This shit went on like this for a whole week affecting over 350 new customers before the devil sent me a DM.
I got a call from my boss on that weekend that some users who had made purchases recently can't access their account from a different device and cannot also update their password. Nobody likes duty calls on a weekend, I grudgingly and sluggishly opened up my PC to create a quick fix but when I saw what the problem was I shut down my PC immediately, I ran into the shower like I was being chased by a ghost, I kept screaming "what tha fuck! what tha fuck!!" cus I knew hell was about to break loose.
At that moment everything seemed off as if I could feel everything, I felt the water dripping down my spine, I could hear the tiniest of sound. I thought about the 350 new customers the client just lost, I imagined the raving anger on the face of my boss, I thought about how dumb my colleagues would think I was for such a stupid long running bug.
I wondered through all possible solutions that could save me from this embarrassment.
-- "If this shitty client would have just allowed us verify users email before usage things wouldn't have gotten to this extent"
-- "Should I call the customers to get their email address using their provided telephone?... No they'd think I'm a scammer"
-- "Should I tell my boss the database was hacked? Pffft hack my a**",
-- "Should I create a page for the affected users to re-verify their email address and password? No, some sessions may have expired"
-- "Or maybe this the best time to quit this f*ckn job!"
... Different thoughts from all four corners of the bathroom made it a really long bath. Finally, I decided it was best I told my boss what had happened. So I fixed the code, called my boss the next day and explained the situation on ground to him and yes he was furious. "What a silly mistake..!" he raged and raged. See me in my office by Monday.
That night felt longer than usual, I couldn't sleep properly. I felt pity for the client and I blamed it all on myself... yeah the "silly mistake", I could have been more careful.
Monday came boss wasn't at the office, Tuesday, Wednesday, Thursday, Friday not available. Next week he was around and when we both met the discussion was about a different project. I tried briefing him about last week incident, he seems not to recall and demands we focus on the current project.
However, over three hundred and fifty customers swept under the carpet courtesy of me. I still felt the guilt of that f*ck up till this day.1 -
Post Anger Rant (Beware, Long rant ahead)
So there is this project we have been working for months, most of the devs involved are jr students so I was leading them in the architecture and what to do and they were doing it, the progress was slow but safe and fun.
On the team there was this guy, someone I trusted and in who I had special interest for his skills, so I let him own the github repo.
So the day of the first demo I pull the backend changes ( I had been working on front end ) and I realize that the code was different, so I started using my super awesome forensic skills to find what happened,and when I say different I mean a totally different architecture different database connections, different service pirts, basically other project, so during my criminal investigation I found out this guy I trusted had never really worked with us, from the beginning he went solo working on his own project and changing everything because of some tutorial he found on the internet, so I decided to reset to the previous version just to find out that he had already deployed the code and that a lot of fixes that we should have were only on his version.
So I went and confront him telling him that he did wrong and he had to learn team work and that I was trying to teach them good practices and he waits and asks me "so, my code was wrong?" Seriously what da hell dude? I'm talking about team work and all you can think about is your code.
Finally he admitted his mistake and repented (I think), but seriously how arrogant must you be to ignore a whole team, specially when on your first real project.undefined pichardo long rant up vote me will support soon pichardo for president screw him team work8 -
Never ever have a tab open on production server database. Changed all users passwords by mistake. Thought was my test database.2
-
the team gave access to the database to another team, and one of their guys did a delete cascade by mistake thinking it was theirs... everyone was scrambling for days trying to get a backup, thankfully there was one3
-
So one rant reminded me of a situation I whent through like 10 years ago...
I'm not a dev but I do small programs from time to time...
One time I was hired to pass a phone book list from paper to a ms Access 97 database...
On my old laptop I could only add 3 to 5 records cause MS access doesn't clean after itself and would crash...
So I made an app (in vb6) , to easily make records, was fast, light and well tabbed.
But now I needed a form to edit the last record when I made a mistake...
Then I wanted a form to check all the records I made.
Well that gave me an idea and presented the software to the client... A cheesy price was agreed for my first freelance sell...
After a month making it perfect and knowing the problems the client would had I made a admin form to merge all the databases and check for each record if it would exist.... I knew the client would have problems to merge hundreds of databases....
When it was done... The client told me he didn't need the software anymore.... So I gave it to a friend to use as an client dabatase software... It was perfect for him.
One month later the client called me because he couldn't merge the databases...
I told him I was already working in a company. That my software was ready to solve his problem, but I got mad and deleted everything...
He had to pay almost 20 times more for a software company to make the same software but worst... Mine would merge and check all the databases in a folder... Their's had to pick one by one and didn't check for duplicates... So he had to pay even more for another program to delete duplicates...
That's why I didn't follow programming as a freelance... Lots of regrets today...
Could be working at home, instead had a burn out this week cause of overwork...
Sorry for the long rant.2 -
Hello DevRant! Time for my first rant. As the saying goes: Everyone should put his own house in order first. So why not rant about myself before i rant about others...
Dear past self next time be less an idiot and enter a database into the persistence.xml wich actually exisist so i don't waste a fucking hour or something like that, for looking what's wrong with my JDBC driver....
Learning something new can suck sometimes if you realize that you made a dumb mistake which prevents you from progress. -
The worst mistake I made was using json files as a simple database.
The project was to be delivered the day before it was even ordered, so I had to take shortcuts.
I totally underestimated the popularity of the project, so all that file reading and writing made the server slow and sluggish. It also being submitted on a shared host, made it even worse.
In the end the host shut it down. I don't recall, but I guess I had to add that database support after all, the project being that popular...4 -
During the first few months of my first professional development role, I had a really odd bug on a live WordPress site that I couldn't replicate locally, despite having the same code and dependency setup. Using WordPress was a mistake but not the one I'm writing about.
I decided to copy live site and its database. Then I thought it best to delete all the users from the copy of the database (I'm not sure why I thought I should do that) and I did so via the WordPress admin UI.
What I wasn't aware of was there was a custom function to email the user before they get deleted.
I got inundated with hundreds of confused/angry/hysterical users about their accounts being deleted, even though they hadn't actually been, and a telling off from the boss.1 -
Have a couple I want to air today.
First was at my first gig as a dev, 4-5 months out of school. I was the only dev at a startup where the owner was a computer illiterate psycopath with serious temper tantrums. We're talking slamming doors, shouting at you while you are on the phone with customers, the works...
Anyways, what happened was that we needed to do an update in our database to correct some data on a few order lines regarding a specific product. Guess who forgot the fucking where-clause... Did I mention this boss was a cheap ass, dollar stupid, penny wise asshole that refused to have anything but the cheapest hosting? No backups, no test/dev/staging environment, no local copies... Yeah, live devving in prod, fucking all customers with a missing semi-colon (or where clause).
Amazingly, his sheer incompetance saved my ass, because even if I explained it, he didn't get it, and just wanted it fixed as best we could.
The second time was at a different company where we were delivering managed network services for a few municipalities. I was working netops at that time, mostly Cisco branded stuff, from Voice-over-IP and wifi to switches and some routing.
One day I was rolling out a new wireless network, and had to add the VLAN to the core switch on the correct port. VLAN's, for those who don't know, are virtual networks you can use to run several separated networks on the same cable.
To add a VLAN on a Cisco switch one uses the command:
switchport access vlan add XYZ
My mistake was omitting the 'add', which Cisco switches happily accept without warning. That command however can be quite disruptive as it replaces all of the excisting VLAN's with the new one.
Not a big deal on a distribution switch supplying an office floor or something, but on a fucking core switch in the datacenter this meant 20K user had no internet, no access to the applications in the DS, no access to Active Directory etc. Oh and my remote access to that switch also went down the drain...
Luckily a colleague of mine was on site with a console cable and access to config backups. Shit was over within 15 minutes. My boss at that time was thankfully a pragmatic guy who just responded "Well, at least you won't make that mistake again" when we debriefed him after the dust settled. -
My company thinks "DevOps Engineer" means
(Senior AWS Solutions Architect) + (Lead DevOps Engineer) + (CISO) + (Helpdesk) + (Sys Admin) + (Team Lead) + (Database Adminstrator)
My mistake was taking on all those duties without demanding 7x the pay.10 -
One time, I accidentally deleted all the data from one of the client's database table (invoices) because i forgot to comment the line from the script.
Good thing there were some backups and I was able to restore everything without anyone noticing. I was so fucking dumb at that time and I felt all the blood rush to my head at that moment when I did a SELECT * and NOTHING was displayed on the screen.
Rookie mistake.11 -
I have started using the input lockout when supporting colleagues.
The phrase "don't touch anything" must be code for "close all my windows"
Last one actually rebooted her machine in the middle of a (manager requested) intervention...
I got some very strange looks in the office to my "good afternoon, I'm calling to inform you that you will need to speak to HR to reinstate your credentials as I have accidentally marked you as a leaver while doing some database maintenance, I'll transfer you now, please hold" phone call, especially as we don't actually have access to do that lol
I put her on hold to myself while I finished then advised that "I'm sorry HR are busy, but I've managed to undo the mistake anyway, my apologies"
Kept her away from the machine so it was lawful evil right?1 -
1) API Server crash due to Uncaught Error and none of us realised it until 30 mins later (no email alerts, no mechanism in place to notify devs): expensive mistake
2) api server and database server on one frickin vps, with no caching mechanisms or memoisation on frontend: Worst nightmare of my life5 -
Last Week Friday:
PM: We'll be taking you off the one project on to another, we'll send the details later.
Me: Cool
*Hours Later*
PM: Ok cool, so you'll be looking at a script that one of our Pillar heads has scripted. You need to make sure it works and that it can run on the server.
Me: *I always thought this guy was useless now i get to see what he can do* Cool, just send the documentation and i'll take a look at it over the weekend. Just tell me when you've sent it.
PM: Cool.
Project Head: I'll inform you when i send the files and how to run them.
Me: *I know how to set up a database locally, i'm not an idiot* Cool.
Whole Weekend I don't get a single message.
Monday Morning:
Project Head(PH): Have you taken a look at it yet?
Me: Taken a look at what?
PH: The Database and the Script
Me: i didn't get any message over the weekend.
PH: I sent it yesterday, it should be in your inbox.
Me: There's Nothing. Sending anything on a Sunday is expecting me not to see it, especially at 10pm. Besides i can't retrieve any of the files in the attachment(Outlook tripping), rather send it in a zip file or upload it to onedrive.
PH sends the link. I get the files, set up the DB, glance at the script.
Me: This is actually interesting.
PH: You know what it does?
Me: My SQL knowledge is below average but i can read and understand it pretty well. So your dynamically copying the database from the server to the warehouse, cool.
It's not going to work though.
PH: Check first.
I check it
Me: Doesn't work, but it sort of works.
PH: What do you mean?
Me: Some tables are populated but some aren't,, how and there's a shit tone of errors.
PH: So i does copy the data over.
Me: Some of the data.
PH: test it on the Server
Me: Not a good idea.
PH: Just try it.
PM: In the mean time i'll send you some documentation i need you to review and edit.
Me: *Idiots* Cool.
Tuesday:
Me: Have you checked it on the server yet?
PH: Not yet, busy.
Me: Where's the documentation again?
PM: I'll send it it a moment.
Me: In the mean time i'll write some script to fix that script that's definitely not going to work.
Wednesday:
Boss: I heard you done with the script
Me: It's not done, but we'll be testing it on the server later.
Boss: Then why are you running it on the server?
Me: Ask the PH and PM.
Boss: What are you doing now?
Me: Well i'm supposed to do documentation *looks at PM* but i haven't recieved any yet, so I've been writing a script to fix the copy script.
PH: Ok we'll test when the boss leaves, after all the meetings.
PM: here's the documentation.
Me: Thanks
I start on documentation.
PH: It didn't work.
Me: I know.
PH: Fix it.
Thursday:
Meeting.
PM: What you doing?
Me: Fixing the script,
PM: Do the documentation first
Me: Cool.
End of the day:
PH: Why you doing the documentation? The script has highest priority.
Me: Ask the PM.
Friday(Today):
Boss: can we talk.
Me: Sure.
Boss: I though you said the script was done?
Me: i said it sort of works, just doesn't do the job 100%.
Boss: Monday i was told it's done.
Me: i only looked through it Monday to understand it, i done nothing before Tuesday. though i have been trying to create a script to fix it.
Boss: Your working really slow hey.
Me: *It's been a week, and stupid people are in charge* I was doing what i was told.
Boss: Cool.(His Upset)
Stupid FUCKEN people, make stupid FUCKEN decisions. But Hey, the boss only see's the final result. I am a human being, even i make mistakes. But there's a huge gap between stupidity and a mistake. -
I was Just college fresher who completed his Engineering. My first week in the office. And a system was provided to me, since it was support project so I was given direct access to production database.
Fresher + Production Database + Access of Admin credentials = Worst Possible Combination
So it was my night shift, I was told to update new tariff plan for our client (which was one of the largest telecom service in India) .
If someone recharges for more than 200 Rupee, that person will get 10% or 20% extra talk time. Which was only applicable for particular circle (Like Bihar and Rajasthan).
Since I was fresher, I was told to update given query from my senior employee which he shared on the shared folder. Production downtime was in the mid night, so at that time I updated that query on the production database.
Query successfully updated. I completed my night shift, went home and slept.
When I woke up, I saw my mobile it had 200+ missed calls from different locations of India. They were Circle heads of that telecom service provider who contacted me. I realized something unexpected is expecting me.
Then at that moment my team lead called me and he asked me to come office right away.
Reminding you I was a fresher, I was shivering. What have I done there?
When I reached office, I came to know that the query I updated on production bombarded.
Every person who recharged that day (duration from midnight to morning 10 AM) got 10 times or 20 times more talktime.
A part of Query was something like this where error was made:
TalkTime = RechargeAmount + RechargeAmount * 10/100; (Bihar)
or
TalkTime = RechargeAmount + RechargeAmount * 20/100; (Rajasthan)
But instead of this query, I updated below one:
TalkTime = RechargeAmount + RechargeAmount * 10;
or
TalkTime = RechargeAmount + RechargeAmount * 20;
In a span of 10 hours, that telecom service lost revenue of 6.5 crore Rupees. Thanks to recovery team they were able to recover 6 crore but still 50 lakh Rupees were in loss.
One small query, and approx 1 million dollar was on stake.
Aftermath of this incident
My Mistake:
I should have taken those queries on mail. Or, there should have been mail communication regarding this.
Never ever do anything over oral communication. Senior employee who did this denied and said he provided correct query, and I had no proof of communication.
I told them, it was me who executed that query on production. Since I was fresher, and took my responsibility of that incident. My team lead rescued me from that situation.
Lesson Learned:
Always test your query and code multiple times before you execute or Go live it on production.
Always have email communication for every action you take on production.
Power comes with responsibility. If you have admin credentials of production never use it for update/delete/drop until you are sure.
Don’t take your job lightly.
I was not fired from that Job, but I have learnt my lesson very well. -
worst mistake? plausible
I just chose _system_ as my username, I got a feeling it will be a prime target in a case of database intrusion :/ -
Been working on a new project for the last couple of weeks. New client with a big name, probably lots of money for the company I work for, plus a nice bonus for myself.
But our technical referent....... Goddammit. PhD in computer science, and he probably. approved our project outline. 3 days in development, the basic features of the applications are there for him to see (yay. Agile.), and guess what? We need to change the user roles hierarchy we had agreed on. Oh, and that shouldn't be treated as extra development, it's obviously a bug! Also, these features he never talked about and never have been in the project? That's also a bug! That thing I couldn't start working on before yesterday because I was still waiting the specs from him? It should've been ready a week ago, it's a bug that it's not there! Also, he notes how he could've developes it within 40 minutes and offered to sens us the code to implement directly in our application, or he may even do so himself.... Ah, I forgot to say, he has no idea on what language we are developing the app. He said he didn't care many times so far.
But the best part? Yesterday he signales an outstanding bug: some data has been changed without anyone interacting. It was a bug! And it was costing them moneeeeey (on a dev server)! Ok, let's dig in, it may really be a bug this time, I did update the code and... Wait, what? Someone actually did update a new file? ...Oh my Anubis. HE did replace the file a few minutes before and tried to make it look like a bug! ..May as well double check. So, 15 minutes later I answer to his e-mail, saying that 4 files have been compromised by a user account with admin privileges (not mentioning I knee it was him)... And 3 minutes later he answered me. It was a message full of anger, saying (oh Lord) it was a bug! If a user can upload a new file, it's the application's fault for not blocking him (except, users ARE supposed to upload files, and admins have been requestes to be able to circumvent any kind of restriction)! Then he added how lucky I was, becausw "the issue resolved itself and the data was back, and we shouldn't waste any more yime.on thos". Let's check the logs again.... It'a true! HE UPLOADED THE ORIGINAL FILES BACK! He... He has no idea that logs do exist? A fucking PhD in computer science? He still believes no one knows it was him....... But... Why did he do that? It couldn't have been a mistake. Was he trying to troll me? Or... Or is he really that dense?
I was laughing my ass of there. But there's more! He actually phones my boss (who knew what had happened) to insult me! And to threaten not dwell on that issue anymore because "it's making them lose money". We were both speechless....
There's no way he's a PhD. Yet it's a legit piece of paper the one he has. Funny thing is, he actually manages to launch a couple of sort-of-nationally-popular webservices, and takes every opportunity to remember us how he built them from scratch and so he know what he's saying... But digging through google, you can easily find how he actually outsurced the development to Chinese companies while he "watched over their work" until he bought the code
Wait... Big ego, a decent amount of money... I'm starting to guess how he got his PhD. I also get why he's a "freelance consultant" and none of the place he worked for ever hired him again (couldn't even cover his own tracks)....
But I can't get his definition of "bug".
If it doesn't work as intended, it's a bug (ok)
If something he never communicated is not implemented, it's a bug (what.)
If development has been slowed because he failed to provide specs, it's a bug (uh?)
If he changes his own mind and wants to change a process, it's a bug it doesn't already work that way (ffs.)
If he doesn't understand or like something, it's a bug (i hopw he dies by sonic diarrhoea)
I'm just glad my boss isn't falling for him... If anything, we have enough info to accuse him of sabotage and delaying my work....
Ah, right. He also didn't get how to publish our application we needes access to the server he wantes us to deploy it on. Also, he doesn't understand why we have acces to the app's database and admin users created on the webapp don't. These are bugs (seriously his own words). Outstanding ones.
Just..... Ffs.
Also, sorry for the typos.5 -
"Hello,
it turns out we probably caused the downtime ourselves. I didn't know dropping 170 databases and deleting 80 entire projects at once could do that"
Gave me a hearty chuckle. Especially as the client adamantly refused to have SSDs installed for the DB to run on top.
Just imagining the poor read-write heads jerking back and forth in vain attempts to find all the data to delete... So yes, dropping 170 databases at once does in fact take a database server down to its knees, as deleting is all the drives will be doing for a while.
At least it wasn't my or my colleague's mistake this time.4 -
Worst mistake I have made is accidentally wiping a database table by messing up a SQL script. This was caught after it had been deployed to around 30 customer sites, most of which had no technical staff on site, and we didn't have remote access to. I bought our support team cookies for fixing that one.1
-
Here's something that should be an unspoken rule in the dev space:
DON'T MAKE A LESSON OF YOUR TEAMMATE'S MISTAKES (OR WHAT YOU CONSIDER A MISTAKE).
More context:
I defined a column as `mediumText` in a Laravel app. What the asshole team lead did after seeing that in my PR, is taking that snippet and blast that out in the company dev slack channel on full display, saying - "Guys we shouldn't take more space in the database than we need to. If the value for the column would be < 255 characters please take VARCHAR not MEDIUMTEXT".
I mean, what an asshole !!! You could've told me that privately and I'd have happily changed it. The reason I did so, is to be safeguard myself from the future, in case the value length increases or decreases. Last thing I want to see in the logs is "Column size is too short for column {COLUMN}, value truncated".
But nah nah, let's take a screenshot of that snippet and `teach` everyone else what should and shouldn't be done according to your `standards`.
Fucking piece of shit team lead.16 -
Never doubt myself.
Never questioned my skills...
Because I have few skills.
I'm no dev, an amateur programmer who learned in school (best part, learned logic programming) and stopped programming for years because I had no future without an engineer University course... How mistaked I was.
So I know that I'll spend more time on google on every project I start.
Still doesn't stop me... Until I find out that I can't do what I want (like the time I made all the UI of a web app in JavaScript to use in electron and then found out that I couldn't use a file database, sq lite on that case... one month almost wasted... Almost, kept the UI as a mock up. Did the same mistake two years latter, only to remember like one week latter why I didn't use JS the last time. Doing it in python+Kivy now) I'll just keep pushing, and trying, and learning.
Never stop, never quit, only death is impossible (for now). -
The "Progress 4GL", now "OpenEdge" doesn't only allow you to abbreviate reserved words, it goes even further.
It allows you to abbreviate database field names.
Yup. Make a spelling mistake and you are messing with a different field.
Forget to declare a variable and there's a field in scope beginning with the same letters? It'll cheerfully assume you mean that field.
This is truly the WORST, most bone-headedly STUPID decision ever taken in the entire history of programming languages.
I defy anyone to offer anything worse than this. -
Need some advice here.
So hello everyone! I recently moved abroad for work, for the sake of the experience and the excitement of learning how developers in Latin America tackle specific problems. To my surprise, the dev team is actually composed solely of Europeans and Americans.
I work for a relatively new startup with an ambitious goal. I love the drive everyone has, but my major gripe is with my team lead. He's adverse to any change, and any and all proposals made to improve quality of throughput are shot down in flames. Our stack is a horrendous mess patched together with band-aids, nothing is documented, there are NO unit tests for our backend and the same goes for our frontend. The team has been working on a database/application migration for about a month now, which I find ridiculous because the entire situation could have been avoided by following very rudimentary DevOps practices (which I'm shunned for mentioning). I should also add that for whatever reason containerization and microservices are also taboo, which I find hillarious because of our currently convoluted setup with elastic beanstalk and the the constant complaints between our development environment and production environments differing too much.
I've been tasked with managing a Wordpress site for the past 3 weeks, hardly what I would consider exciting. I've written 6 pages in the past two weeks so our marketing team can move off of squarespace to save some money and allow us more control. Due to the shit show that is our "custom theme" I had to write these pages in a manner that completely disregard existing style rules by disabling them entirely on these pages. Now, ironically they would like to change the blog's base theme but this would invertedly cause other pages created before I arrived to simply not work, which means I would have to rewrite them.
Before I took the role of writing an entire theme from scratch and updating these existing pages to work adequately, I proposed moving to a headless wordpress setup. In which case we could share assets in a much more streamline manner between our application and wordpress site and unify our styles. I was shot down almost immediately. Due to a grave misunderstanding of how wordpress works, no one else on the team seems to understand just how easy it is to fetch data from wordpress's api.
In any event, I also had a tech meeting today with developers from partner companies and realized no one knew what the fuck they were talking about. The greater majority of these self proclaimed senior developers are actually considered junior developers in the United States. I actually recoiled at the thought that I may have made a great mistake leaving the United States to look a great tech gig.
I mean no disrespect to Latin America, or any European countries, I've met some really incredible developers from Russia, the Ukraine, Italy, etc. in the past and I'm certainly not trying to make any blanket statements. I just want to know what everyone thinks, if I should maybe move back to the states and header over to the bay/NY. I'm from the greater Boston area, where some really great stuff is going on but I guess I also wanted a change of scenery.2 -
Dumb mistake from when I was still working:
My work laptop’s SSD went haywire, and I/O would spike every 10 minutes or so for ~50 ms. The hardware guy said he could replace the SSD right away, or I could endure it for a few weeks and get a new laptop instead. Obviously, I agreed to wait. The stutter noticeably affected screen rendering, but I didn’t notice any other issues. Little did I know that every time it happened, all input was ignored (as in: not queued). Normally it wouldn’t matter, because hitting a random ~50 ms window is hard. How-the-f×ck-ever…
A few days later — without getting into “why” — I was forced to apply a patch in production. So I opened an SSH session to prod in one terminal, spun up a dev environment in another, copied the database schema from prod to dev, and made sure to test everything. No issues, so I jumped to prod, applied the patch, restarted services, jumped back to dev, and cleaned up the now-unnecessary database. Only to discover that my “jumped back to dev” keystroke didn’t register.16 -
I wasn't happy with one of our UI views for editing a database query that consisted of about 50 fields ("editing" being the operative term here, not just viewing. It had to be two-way). Everything was hardcoded and defined manually, with each block of ~10 lines being repeated and mostly identical apart from the occasional double inline field and name of the variable. It had "just ended up that way" over time due to the variable names in the UI being different than the names of the variables that came from the API.
I decided to overhaul it all where I defined the different input components and which fields should be included, then made a function which would generate the page based on these definitions. It was about 500 lines of modularized functions and classes where the class for the actual view was about 50 lines- compared to the 1400+ lines of the previous version.
But, it didn't work. It should, but it "just didn't". There was no error. All I got was a blank, solid white page. I could make a drastic change or try something completely different and I would get the same error, same blank page. API fetch succeeded, value assignments succeeded, the object exists, but if you iterated it it was... empty.
I started getting really discouraged that I had made it too abstract. Maybe I actually made it more complex and unreadable than before. Maybe just hardcoding it all was the better solution after all. Maybe I had gone against KISS and overdesigned it.
I was up pretty late and everyone had gone home. When the last guy left there was that mood where "yeah if I can't make this work we'll just use the current version...".
Turns out I had tried iterating over a property of the set of fields to render, rather than the entire collection. In the old method the variables were a member of an object, but now they were its own object, a change I had made to isolate the set of values which were to be viewed/edited and make them easier to pass back and forth. This member existed since I hadn't cleaned it out yet, but it was empty.
I had been banging my head against this for a whole day and I was ready to admit I had made a mistake and wasted my time before discarding it all, but then I backspaced this one property and the interface went from empty to rendering perfectly and with all functionality intact. I swear god rays were coming out of my screen. -
I started running a Database benchmark yesterday morning, with my system configuration, expected time to complete was 36hrs(arround), so I left it and made sure no one disturbs (I stuck a note in the monitor) because it was on common system in the lab.
Then I went to my other work.
Evening ,I came to check the progress, my monitor was switched off, I thought its in power saving mode!
Fuck, I bend down and see the CPU is off!
Wtf!! Who shut it down ,even after the note.
Then I saw the electric outlet was off!
Then after wards asking ppl in the lab, they told ,the cleaning person was cleaning the switches, so yeah she could have by mistake!
* I facepalmed *
So again, I set it up with frustration!
Today morning ,I came to see the progress
FML, from no where ,
" It's in Windows automatic repair loop! "
It's been 3hrs, trying to get out of that loop without loosing the data.
1TB of data is there, took 1month to setup all the things
Fuck Microsoft for adding these kind idiotic stuff in windows.
Is there a spirit in the lab not allowing me to do benchmark? -
So I was working on a web scraper to basically download all listings with detailed info from a e-shop to my database for some analysis.
And I completely forgot throttling which is quite important when writing such things in node.js.
It's funny how in other languages you try to figure out how to make your application faster and in node you're trying to make it slower 😄
Anyhow, I apparently hit the poor site with 5000+ simultaneous requests, all of which hit their database (to gather product info). Suffice to say, the site got visibly slow 🤣
Thankfully I print out where each request is made so I quickly realised my mistake and killed the process.
Now I hope no-one comes knocking on my door lol
The adventures of being a node.js dev1 -
Arg. I messed up bad today. Updated 17k records in database by mistake and had to rollback and apologize...
I won't rush on a task for rest of the day1 -
How is it possible?
I installed a treeview module. I followed all the documentation and the module is showing. Perfect!
Me: create 3 roots,
Module: all is ok.
Me: 😊create 2 nodes,
Module: everything is fine.
Me: 😁3rd node
Module: ... Kaput. tree displays like the nodes are in random order.
Me: 🤨Check the database, and fix the set.
Module: Aah much better.
Me: 😃Try to change an int.
Module: Noooooo! Big mistake!!!
Me: 🤔Ok, ok, ok, rollback! 😧
Module: still in random situation
Me: 😶 and now what? -
"Oh, sorry I didn't write you back! I checked 3 hours ago, and we only add the data once in our database before sending the notification to your endpoint, so everything is fine! Check if you run the same functio twice, it's an easy mistake!"
You. Fucking. Moron. You send the data 2 or 3 times (at random) every fucking time. I have nginx logs showing that, and I've fucking shown them to you TWICE. I don't fucking care if your DB is fine, check how many fucking times you POST the damn data. We're already 2 days behind schedule because you can't be arsed to check your own damn code. Ffs. How can you even be a senior developer?! -
I hate group project so much.
I yet again successfully stirred up a big drama in my project group. For project, I proposed a CDN cache system for a post only database server. Super simple. I wanted to see what ideas other people come up with. So I said I am not good at the content and the idea is dumb. Oh man, what a horrible mistake. One group member wants to build a chat app with distributed storage. We implemented get/put for a terribly designed key value store and now they want to build a freaking chat app on top of a more stupid kV store using golang standard lib. I don't think any of those fools understand the challenges that comes with the distributed storage.
I sent a video explaining part of crdt. "That's way too complicated. Why are you making everything complicated."
Those fools leave too much details for course stuff's interpretation and says
"course stuff will only grade the project according to the proposal. It's in the project description".
I asked why don't they just take baby steps and just go with their underlying terribly designed kV store.
"Messaging app is more interesting and designing kV store with generic API is just as difficult"
😂 Fucking egos
Then I successfully pissed off all group members with relatively respectful words then pissed off myself and joined another group.1 -
I wanted to know what is the worst mistake you make on database.
I have actually implements the logic of token access control on database and not on business logic layer.
The database have a login procedure which accept username and password. That login procedure actually hash the password and try to authenticate user.
If it is a correct user , it generate a token. In other to use other procedure on database , you must provide a token. By using that token , the procedure know who is it and what permission is granted to that user.4 -
Did a backup.
Did a bulk import to a service (netbox, great tool for network documentation by the way).
Fucked it up.
Back to backup.
Fixed bug.
Fucking gerbils on ecstasy (devs) annoying the fuck out of me.
Lobotomizing meetings.
Yuck coffee. Cold, stale and over extracted.
Gerbils planning a mutiny. Killed them with a big cheese wheel (lots of stupid boring tasks as they seem to be completely missing stuff to do).
Deleted backup.
Redid import.
Small typo.
Find the mistake...
I just love it when a small task becomes a huge fuckup due to too many disturbances.
Luckily SQL UPDATE went well....
Nothing's more fun than to patch a live database.
Yeah. I didn't wanna do a new backup. :)
The "I don't give a crap mood" is sometimes a great way to accelerate tasks... -
Persisterising derived values. Often a necessary evil for optimisation or privacy while conflicting with concerns such as auditing.
Password hashing is the common example of a case considered necessary to cover security concerns.
Also often a mistake to store derived values. Some times it can be annoying. Sometimes it can be data loss. Derived values often require careful maintenance otherwise the actual comments in your database for a page is 10 but the stored value for the page record is 9. This becomes very important when dealing with money where eventual consistency might not be enough.
Annoying is when given a and b then c = a + b only b and c are stored so you often have to run things backwards.
Given any processing pipeline such as A -> B -> C with A being original and C final then you technically only need C. This applies to anything.
However, not all steps stay or deflate. Sum of values is an example of deflate. Mapping values is an example of stay. Combining all possible value pairs is inflate, IE, N * N and tends to represent the true termination point for a pipeline as to what can be persisted.
I've quite often seen people exclude original. Some amount of lossy can be alright if it's genuine noise and one way if serving some purpose.
If A is O(N) and C reduces to O(1) then it can seem to make sense to store only C until someone also wants B -> D as well. Technically speaking A is all you ever need to persist to cater to all dependencies.
I've seen every kind of mess with processing chains. People persisting the inflations while still being lossy. Giant chains linear chains where instead items should rely on a common ancestor. Things being applied to only be unapplied. Yes ABCBDBEBCF etc then truncating A happens.
Extreme care needs to be taken with data and future proofing. Excess data you can remove. Missing code can be added. Data however once its gone its gone and your bug is forever.
This doesn't seem to enter the minds of many developers who don't reconcile their execution or processing graphs with entry points, exist points, edge direction, size, persistence, etc.2 -
So i've not slept all night, come into work and my first mistake 1 hour in is migrating data from another project into a production database now i have redundant data and redundant tables, well done! fml
-
Disclaimer: I love open source and I adore the owasp for what they do.
BUT owasp zap has to be the most overly complicated, badly documented tool in existence. As long as one stays within its most basic functions everything is fine, setting it up as a proxy and even issuing a root cert for our test devices worked wonderfully simple.
Then I made the mistake to try to actually do anything with the data we pulled and had to dive into the scripting console.
The documentation basically consists only of "This thing exists", it provides a msg object with no information what it contains or how it's structured, has no code completion and, here comes the kicker, if the script is run and has an error it gets flagged and can't be reenabled after the error is fixed. So I'm currently at forwarder48.groovy trying to simply store the request on a database for possible diagnostics.
So right now I already know that I'll spend most of my vacation next week trying to decipher the source, document it, fix that damn "flagged as error" bullshit and jump through a billion hoops trying to get a pull request through.2