Do all the things like ++ or -- rants, post your own rants, comment on others' rants and build your customized dev avatarSign Up
Get a devDuck
Rubber duck debugging has never been so cute! Get your favorite coding language devDuckBuy Now
Search - "dba"
*Now that's what I call a Hacker*
MOTHER OF ALL AUTOMATIONS
This seems a long post. but you will definitely +1 the post after reading this.
xxx: OK, so, our build engineer has left for another company. The dude was literally living inside the terminal. You know, that type of a guy who loves Vim, creates diagrams in Dot and writes wiki-posts in Markdown... If something - anything - requires more than 90 seconds of his time, he writes a script to automate that.
xxx: So we're sitting here, looking through his, uhm, "legacy"
xxx: You're gonna love this
xxx: smack-my-bitch-up.sh - sends a text message "late at work" to his wife (apparently). Automatically picks reasons from an array of strings, randomly. Runs inside a cron-job. The job fires if there are active SSH-sessions on the server after 9pm with his login.
xxx: kumar-asshole.sh - scans the inbox for emails from "Kumar" (a DBA at our clients). Looks for keywords like "help", "trouble", "sorry" etc. If keywords are found - the script SSHes into the clients server and rolls back the staging database to the latest backup. Then sends a reply "no worries mate, be careful next time".
xxx: hangover.sh - another cron-job that is set to specific dates. Sends automated emails like "not feeling well/gonna work from home" etc. Adds a random "reason" from another predefined array of strings. Fires if there are no interactive sessions on the server at 8:45am.
xxx: (and the oscar goes to) fuckingcoffee.sh - this one waits exactly 17 seconds (!), then opens an SSH session to our coffee-machine (we had no frikin idea the coffee machine is on the network, runs linux and has SSHD up and running) and sends some weird gibberish to it. Looks binary. Turns out this thing starts brewing a mid-sized half-caf latte and waits another 24 (!) seconds before pouring it into a cup. The timing is exactly how long it takes to walk to the machine from the dudes desk.
xxx: holy sh*t I'm keeping those
The bash scripts weren't bogus, you can find his scripts on the this github URL:
Yesterday, in a meeting with project stakeholders and a dev was demoing his software when an un-handled exception occurred, causing the app to crash.
Dev: “Oh..that’s weird. Doesn’t do that on my machine. Better look at the log”
- Dev looks at the log and sees the exception was a divide by zero error.
Dev: “Ohhh…yea…the average price calculation, it’s a bug in the database.”
<I burst out laughing>
Me: “That’s funny.”
<Dev manager was not laughing>
DevMgr: “What’s funny about bugs in the database?”
Me: “Divide by zero exceptions are not an indication of a data error, it’s a bug in the code.”
Dev: “Uhh…how so? The price factor is zero, which comes from a table, so that’s a bug in the database”
Me: “Jim, will you have sales with a price factor of zero?”
StakeholderJim: “Yea, for add-on items that we’re not putting on sale. Hats, gloves, things like that.”
Dev: “Steve, did anyone tell you the factor could be zero?”
DBA-Steve: “Uh...no…just that the value couldn’t be null. You guys can put whatever you want.”
DevMgr: “So, how will you fix this bug?”
DBA-Steve: “Bug? …oh…um…I guess I could default the value to 1.”
Dev: “What if the user types in a zero? Can you switch it to a 1?”
Me: “Or you check the factor value before you try to divide. That will fix the exception and Steve won’t have to do anything.”
<awkward couple of seconds of silence>
DevMgr: “Lets wrap this up. Steve, go ahead and make the necessary database changes to make sure the factor is never zero.”
StakeholderJim: “That doesn’t sound right. Add-on items should never have a factor. A value of 1 could screw up the average.”
Dev: “Don’t worry, we’ll know the difference.”
<everyone seems happy and leaves the meeting>
I completely lost any sort of brain power to say anything after Dev said that. All the little voices kept saying were ‘WTF? WTF just happened? No really…W T F just happened!?’ over and over. I still have no idea on how to articulate to anyone with any sort of sense about what happened. Thanks DevRant for letting me rant.16
Boss: "Could you join the new DevOps team for a week or two, for some coaching?"
Me: "I'd rather watch you masturbate furiously in a corner of the office while you cry over your ex boyfriend"
Boss: "Yeah... that's why I ask you. You are the only one brave enough to watch"
Me: *Sigh* "But I don't know shit about what DevOps does, I'm a DBA. I've told you the difference a million times. Can't we just douse it in gasoline and set it on fire?"
Me: "Not the team, the servers..."
Boss, imitating Gimli: "And my ex!"
Me: "I get why he left you"
Boss: "It's funny, he was actually better with computers than me, maybe even better than you. He hated me for starting this company, told me I was just chasing money instead of ideals. He just isn't grown up enough to see that there is more to the world than computer games, brewing beer, maker festivals and gay bars, that you need to take responsibility... Maybe it just never works out between managers and geeks..."
Me: "Indeed. The difference in competence is too large"
Boss: "Ugh. You are like straight version of him... but will you at least take a look?"
Me: "Fine, unzip your pants..."
Boss: "No, not that... you need to teach DevOps this docking thing, with the parallel stuff, and the horizontal growth"
Me: "Damn I really hope we're talking about servers now... Do you mean Docker?"
Boss: "That's it. They want to learn how to dock on the Windows servers. They reserved two 4xlarge on AWS. Is that enough for docking?"
Me: "You know what. I'm going back to hug my DB designs, and wash my brain with some queries. Then I'll return here to burn everything to the ground. There is no hope for you left"
Boss: "That's what he said"
Me: "You're using that meme wrong"
Boss: "OK. So what if you just stay on DB management, and I'll just give you the budget to recruit a new DevOps lead and pay for training?"
Me: "That would work"
Boss: "Why are you grinning?"
Me: "Because I have your ex's phone number"20
I was once working on a project that dealt with incredibly sensitive financial data.
We needed a client’s database to do a migration.
They wouldn’t send it over the internet because it was too big and they didn’t think it would be secure.
They opt to send it in the post on an encrypted usb drive.
(Fair enough thinks I)
USB drive arrives.
Is indeed encrypted.
MFW there’s a post it note in the envelope with the password on.
MFW this is a billion dollar multinational petrochem company.
MFW this same company’s ‘sysadmin’ and ‘dba’ once complained because a SQL script I sent them didn’t work - they’d pasted it twice and couldn’t work this out from the fucking “table already created” error message management studio was throwing at them.3
I worked on a greenfield project a couple of years ago. The company had an old solution written in Omnis (heard of it? Yeah, me neither) with an SQL database. My team was to create a completely new web based system... on top of the old database, so the customers could keep their existing stuff.
The dba was an intelligent man, one of the nicest people I've met, and over the course of fifteen years he had made a remarkably terrifying monstrosity of a database. Some years before me they wanted to "future proof" the system and make it "easier to switch to new technologies". So they moved the entire business logic into the database...
I used a tool to create a visualization of said database when we started. It had no views, only tables and sprocs. Look at it! Tables and sprocs are rectangles (well, dots) and any connections are drawn in grey lines. There were no foreign keys, so a tables only visualization only yielded a collection of independent rectangles without a single line.
Now, the stored procedures were bloody MASSIVE. A single procedure that only registered a new interested party and attached them to a property had 2500+ lines and over 150 parameters.
Also, this dba added features and fixed bugs by logging into the respective customers production server and writing SQL.
That database is the stupidest thing I've ever seen a developer do.37
In a user-interface design meeting over a regulatory compliance implementation:
User: “We’ll need to input a city.”
Dev: “Should we validate that city against the state, zip code, and country?”
User: “You are going to make me enter all that data? Ugh…then make it a drop-down. I select the city and the state, zip code auto-fill. I don’t want to make a mistake typing any of that data in.”
Me: “I don’t think a drop-down of every city in the US is feasible.”
Manage: “Why? There cannot be that many. Drop-down is fine. What about the button? We have a few icons to choose from…”
Me: “Uh..yea…there are thousands of cities in the US. Way too much data to for anyone to realistically scroll through”
Dev: “They won’t have to scroll, I’ll filter the list when they start typing.”
Me: “That’s not really the issue and if they are typing the city anyway, just let them type it in.”
User: “What if I mistype Ch1cago? We could inadvertently be out of compliance. The system should never open the company up for federal lawsuits”
Me: “If we’re hiring individuals responsible for legal compliance who can’t spell Chicago, we should be sued by the federal government. We should validate the data the best we can, but it is ultimately your department’s responsibility for data accuracy.”
Manager: “Now now…it’s all our responsibility. What is wrong with a few thousand item drop-down?”
Me: “Um, memory, network bandwidth, database storage, who maintains this list of cities? A lot of time and resources could be saved by simply paying attention.”
Manager: “Memory? Well, memory is cheap. If the workstation needs more memory, we’ll add more”
Dev: “Creating a drop-down is easy and selecting thousands of rows from the database should be fast enough. If the selection is slow, I’ll put it in a thread.”
DBA: “Table won’t be that big and won’t take up much disk space. We’ll need to setup stored procedures, and data import jobs from somewhere to maintain the data. New cities, name changes, ect. ”
Manager: “And if the network starts becoming too slow, we’ll have the Networking dept. open up the valves.”
Me: “Am I the only one seeing all the moving parts we’re introducing just to keep someone from misspelling ‘Chicago’? I’ll admit I’m wrong or maybe I’m not looking at the problem correctly. The point of redesigning the compliance system is to make it simpler, not more complex.”
Manager: “I’m missing the point to why we’re still talking about this. Decision has been made. Drop-down of all cities in the US. Moving on to the button’s icon ..”
Me: “Where is the list of cities going to come from?”
<few seconds of silence>
Dev: “Post office I guess.”
Me: “You guess?…OK…Who is going to manage this list of cities? The manager responsible for regulations?”
User: “Thousands of cities? Oh no …no one is our area has time for that. The system should do it”
Me: “OK, the system. That falls on the DBA. Are you going to be responsible for keeping the data accurate? What is going to audit the cities to make sure the names are properly named and associated with the correct state?”
DBA: “Uh..I don’t know…um…I can set up a job to run every night”
Me: “A job to do what? Validate the data against what?”
Manager: “Do you have a point? No one said it would be easy and all of those details can be answered later.”
Me: “Almost done, and this should be easy. How many cities do we currently have to maintain compliance?”
User: “Maybe 4 or 5. Not many. Regulations are mostly on a state level.”
Me: “When was the last time we created a new city compliance?”
User: “Maybe, 8 years ago. It was before I started.”
Me: “So we’re creating all this complexity for data that, realistically, probably won’t ever change?”
User: “Oh crap, you’re right. What the hell was I thinking…Scratch the drop-down idea. I doubt we’re have a new city regulation anytime soon and how hard is it to type in a city?”
Manager: “OK, are we done wasting everyone’s time on this? No drop-down of cities...next …Let’s get back to the button’s icon …”
Simplicity 1, complexity 0.17
--- GitHub 24-hour outage post mortem ---
As many of you will remember; Github fell over earlier this month and cracked its head on the counter top on the way down. For more or less a full 24 hours the repo-wrangling behemoth had inconsistent data being presented to users, slow response times and failing requests during common user actions such as reporting issues and questioning your career choice in code reviews.
It's been revealed in a post-mortem of the incident (link at the end of the article) that DB replication was the root cause of the chaos after a failing 100G network link was being replaced during routine maintenance. I don't pretend to be a rockstar-ninja-wizard DBA but after speaking with colleagues who went a shade whiter when the term "replication" was used - It's hard to predict where a design decision will bite back and leave you untanging the web of lies and misinformation reported by the databases for weeks if not months after everything's gone a tad sideways.
When the link was yanked out of the east coast DC undergoing maintenance - Github's "Orchestrator" software did exactly what it was meant to do; It hit the "ohshi" button and failed over to another DC that wasn't reporting any issues. The hitch in the master plan was that when connectivity came back up at the east coast DC, Orchestrator was unable to (un)fail-over back to the east coast DC due to each cluster containing data the other didn't have.
At this point it's reasonable to assume that pants were turning funny colours - Monitoring systems across the board started squealing, firing off messages to engineers demanding they rouse from the land of nod and snap back to reality, that was a bit more "on-fire" than usual. A quick call to Orchestrator's API returned a result set that only contained database servers from the west coast - none of the east coast servers had responded.
Come 11pm UTC (about 10 minutes after the initial pant re-colouring) engineers realised they were well and truly backed into a corner, the site was flipped into "Yellow" status and internal mechanisms for deployments were locked out. 5 minutes later an Incident Co-ordinator was dragged from their lair by the status change and almost immediately flipped the site into "Red" status, a move i can only hope was accompanied by all the lights going red and klaxons sounding.
Even more engineers were roused from their slumber to help with the recovery effort, By this point hair was turning grey in real time - The fail-over DB cluster had been processing user data for nearly 40 minutes, every second that passed made the inevitable untangling process exponentially more difficult. Not long after this Github made the call to pause webhooks and Github Pages builds in an attempt to prevent further data loss, causing disruption to those of us using Github as a way of kicking off our deployment processes (myself included, I had to SSH in and run a git pull myself like some kind of savage).
Glossing over several more "And then things were still broken" sections of the post mortem; Clever engineers with their heads screwed on the right way successfully executed what i can only imagine was a large, complex and risky plan to untangle the mess and restore functionality. Github was picked up off the kitchen floor and promptly placed in a comfy chair with a sweet tea to recover. The enormous backlog of webhooks and Pages builds was caught up with and everything was more or less back to normal.
It goes to show that even the best laid plan rarely survives first contact with the enemy, In this case a failing 100G network link somewhere inside an east coast data center.
Link to the post mortem: https://blog.github.com/2018-10-30-...10
This week I started a new job.
My role changed from "Full-stack-web-developer-sysadmin-DBA-helpdesk-strange-person-fixing-stuff-around" to "Back-end Developer".
Moreover, it's a full remote position (so difficult to find in Italy!), so:
1. I can wake up 1.5 hours later;
2. I don't have to waste anymore 2 hours every fucking day driving in traffic to reach the workplace;
3. I can use my fucking bathroom;
4. I can drink hot tea in August without being criticized. 😀
I'm fucking happy!13
I'm convinced code addiction is a real problem and can lead to mental illness.
Dev: "Thanks for helping me with the splunk API. Already spent two weeks and was spinning my wheels."
Me: "I sent you the example over a month ago, I guess you could have used it to save time."
Dev: "I didn't understand it. I tried getting help from NetworkAdmin-Dan, SystemAdmin-Jake, they didn't understand what you sent me either."
Me: "I thought it was pretty simple. Pass it a query, get results back. That's it"
Dev: "The results were not in a standard JSON format. I was so confused."
Me: "Yea, it's sort-of JSON. Splunk streams the result as individual JSON records. You only have to deserialize each record into your object. I sent you the code sample."
Dev: "Your code didn't work. Dan and Jake were confused too. The data I have to process uses a very different result set. I guess I could have used it if you wrote the class more generically and had unit tests."
<oh frack...he's been going behind my back and telling people smack about my code again>
Me: "My code wouldn't have worked for you, because I'm serializing the objects I need and I do have unit tests, but they are only for the internal logic."
Dev:"I don't know, it confused me. Once I figured out the JSON problem and wrote unit tests, I really started to make progress. I used a tuple for this ... functional parameters for that...added a custom event for ... Took me a few weeks, but it's all covered by unit tests."
Me: "Wow. The way you explained the project was; get data from splunk and populate data in SQLServer. With the code I sent you, sounded like a 15 minute project."
Dev: "Oooh nooo...its waaay more complicated than that. I have this very complex splunk query, which I don't understand, and then I have to perform all this parsing, update a database...which I have no idea how it works. Its really...really complicated."
Me: "The splunk query returns what..4 fields...and DBA-Joe provided the upsert stored procedure..sounds like a 15 minute project."
Dev: "Maybe for you...we're all not super geniuses that crank out code. I hope to be at your level some day."
<frack you ... condescending a-hole ...you've got the same seniority here as I do>
Me: "No seriously, the code I sent would have got you 90% done. Write your deserializer for those 4 fields, execute the stored procedure, and call it a day. I don't think the effort justifies the outcome. Isn't the data for a report they'll only run every few months?"
Dev: "Yea, but Mgr-Nick wanted unit tests and I have to follow orders. I tried to explain the situation, but you know how he is."
<fracking liar..Nick doesn't know the difference between a unit test and breathalyzer test. I know exactly what you told Nick>
Dev: "Thanks again for your help. Gotta get back to it. I put a due date of April for this project and time's running out."
APRIL?!! Good Lord he's going to drag this intern-level project for another month!
After he left, I dug around and found the splunk query, the upsert stored proc, and yep, in about 15 minutes I was done.1
TL;DR: I resigned from my full-remote job and I came back to the old office.
Some months ago I celebrated the achievement of a full-remote position.
I was tired of driving every day to reach the office (especially in summer, because I had an old car without air conditioner). Moreover I did many different things (full-stack development, system administration, DBA, helpdesk) without a well-defined task list, and I thought that was bad. Everybody on the Internet seems happy to work remotely, and being an introvert I thought I would have no problem.
However I ignored 2 things when I left:
1. I was doing something that was important to me (my software helped scientists);
2. everybody loved me.
So, I started my new remote job as a back-end developer, for a company following the Scrum methodology.
My days became very monotonous: wake up, write some code, move a ticket on the Jira board, clean the house, go to bed. All in pajamas. Repeat.
I had very few interactions with my colleagues, mostly on Slack or audio-only calls. I never seen most of them. I didn't feel engaged in what I was working on. I did it just for the money.
After some months I started feeling very depressed and I realized that I left a job suited for me for a random job, just because it was a remote position.
So, one day I called my old boss: «Hey, would you like to have me back?»
I asked for a loan to buy a new car, then I send my resignation letter.
When I started again my first day I felt really touched, because basically everybody said to me: «Hey, welcome back! I'm happy you are here again.»
The guy that did some changes to my code also said: «It has been a pleasure to maintain your code. I learned a lot.»
😲 I didn't expect that.
Please, don't follow the myth that working remotely is all we need.6
Just resigned from my company.
I’m “sadpy”, or “had”, half happy half sad.
They gave me a lot, I gave them a lot.
I feel sorry because now they won’t have a jolly anymore.
I used to do many kind of activities, from DBA to HW installation, from penetration tests to security analysis, from scripting to programming.
A full-stack all-do.
The new company wants me for my skills in electronics as well, I’ll be their all-do for IoT 😍
I’m quite happy for my new role, wish me luck 😊9
Most satisfying bug I've fixed?
Fixed a n+1 issue with a web service retrieving price information. I initially wrote the service, but it was taken over by a couple of 'world class' monday-morning-quarterbacks.
The "Worst code I've ever seen" ... "I can't believe this crap compiles" types that never met anyone else's code that was any good.
After a few months (yes months) and heavy refactoring, the service still returned price information for a product. Pass the service a list of product numbers, service returns the price, availability, etc, that was it.
After a very proud and boisterous deployment, over the next couple of days the service seemed to get slower and slower. DBAs started to complain that the service was causing unusually high wait times, locks, and CPU spikes causing problems for other applications. The usual finger pointing began which ended up with "If PaperTrail had written the service 'correctly' the first time, we wouldn't be in this mess."
Only mattered that I initially wrote the service and no one seemed to care about the two geniuses that took months changing the code.
The dev manager was able to justify a complete re-write of the service using 'proper development methodologies' including budgeting devs, DBAs, server resources, etc..etc. with a projected year+ completion date.
My 'BS Meter' goes off, so I open up the code, maybe 5 minutes...tada...found it. The corresponding stored procedure accepts a list of product numbers and a price type (1=Retail, 2=Dealer, and so on). If you pass 0, the stored procedure returns all the prices.
Code basically looked like this..
public List<Prices> GetPrices(List<Product> products, int priceTypeId)
foreach (var item in products)
List<int> productIdsParameter = new List<int>();
List<Price> prices = dataProvider.GetPrices(productIdsParameter, 0);
foreach (var price in prices)
if (price.PriceTypeID == priceTypeId)
prices = dataProvider.GetPrices(productIdsParameter, price.PriceTypeID);
* Omitting the other 'WTF?' code to handle the zero price type
I removed the double stored procedure call, updated the method signature to only accept the list of product numbers (which it was before the 'major refactor'), deployed the service to dev (the issue was reproducible in our dev environment) and had the DBA monitor.
The two devs and the manager are grumbling and mocking the changes (they never looked, they assumed I wrote some threading monstrosity) then the DBA walks up..
DBA: "We're good. You hit the database pretty hard and the CPU never moved. Execution plans, locks, all good to go."
<dba starts to walk away>
DevMgr: "No fucking way! Putting that code in a thread wouldn't have fix it"
Me: "Um, I didn't use threads"
Dev1: "You had to. There was no way you made that code run faster without threads"
Dev2: "It runs fine in dev, but there is no way that level of threading will work in production with thousands of requests. I've got unit tests that prove our design is perfect."
Me: "I looked at what the code was doing and removed what it shouldn't be doing. That's it."
DBA: "If the database is happy with the changes, I'm happy. Good job. Get that service deployed tomorrow and lets move on"
Me: "You'll remove the recommendation for a complete re-write of the service?"
DevMgr: "Hell no! The re-write moves forward. This, whatever you did, changes nothing."
DBA: "Hell yes it does!! I've got too much on my plate already to play babysitter with you assholes. I'm done and no one on my team will waste any more time on this. Am I clear?"
Seeing the dev manager face turn red and the other two devs look completely dumbfounded was the most satisfying bug I've fixed.5
Our web department was deploying a fairly large sales campaign (equivalent to a ‘Black Friday’ for us), and the day before, at 4:00PM, one of the devs emails us and asks “Hey, just a heads up, the main sales page takes almost 30 seconds to load. Any chance you could find out why? Thanks!”
We click the URL they sent, and sure enough, 30 seconds on the dot.
Our department manager almost fell out of his chair (a few ‘F’ bombs were thrown).
DBAs sit next door, so he shouts…
Mgr: ”Hey, did you know the new sales page is taking 30 seconds to open!?”
DBA: “Yea, but it’s not the database. Are you just now hearing about this? They have had performance problems for over week now. Our traces show it’s something on their end.”
Mgr: “-bleep- no!”
Mgr tries to get a hold of anyone …no one is answering the phone..so he leaves to find someone…anyone with authority.
4:15 he comes back..
Mgr: “-beep- All the web managers were in a meeting. I had to interrupt and ask if they knew about the performance problem.”
Me: “Oh crap. I assume they didn’t know or they wouldn’t be in a meeting.”
Mgr: “-bleep- no! No one knew. Apparently the only ones who knew were the 3 developers and the DBA!”
Me: “Uh…what exactly do they want us to do?”
Mgr: “The –bleep- if I know!”
Me: “Are there any load tests we could use for the staging servers? Maybe it’s only the developer servers.”
DBA: “No, just those 3 developers testing. They could reproduce the slowness on staging, so no need for the load tests.”
Mgr: “Oh my –bleep-ing God!”
4:30 ..one of the vice presidents comes into our area…
VP: “So, do we know what the problem is? John tells me you guys are fixing the problem.”
Mgr: “No, we just heard about the problem half hour ago. DBAs said the database side is fine and the traces look like the bottleneck is on web side of things.”
VP: “Hmm, no, John said the problem is the caching. Aren’t you responsible for that?”
Mgr: “Uh…um…yea, but I don’t think anyone knows what the problem is yet.”
VP: “Well, get the caching problem fixed as soon as possible. Our sales numbers this year hinge on the deployment tomorrow.”
- VP leaves -
Mgr: “I was thinking the same thing. I’ll have networking run a trace.”
4:45 Networking run their trace, and sure enough, there was some relative path of ‘something’ pointing to a local resource not on development, it was waiting/timing out after 30 seconds. Fixed the path and page loaded instantaneously. Network admin walks over..
NetworkAdmin: “We had no idea they were having problems. If they told us last week, we could have identified the issue. Did anyone else think 30 second load time was a bit suspicious?”
4:50 VP walks in (“John” is the web team manager)..
VP: “John said the caching issue is fixed. Great job everyone.”
VP: “But the caching is fixed? Right? John said it was caching. Anyway, great job everyone. We’re going to have a great day tomorrow!”
NetworkAdmin: “Ouch…you feel that?”
Me: “Feel what?”
NetworkAdmin: “That bus John just threw us under.”
Mgr: “Yea, but I think John just saved 3 jobs. Remember that.”4
Every year my team runs an award ceremony during which people win “awards” for mistakes throughout the year. This years was quite good.
The integration partner award- one of our sysAdmins was talking with a partner from another company over Skype and was having some issues with azure. He intended to send me a small rant but instead sent “fucking azure can go fuck itself, won’t let me update to managed disks from a vhd built on unmanaged” to our jv partner.
Sysadmin wannabe award (mine)- ran “Sudo chmod -R 700 /“ on one of our dev systems then had to spend the next day trying to fix it 😓
The ain’t no sanity clause award - someone ran a massive update query on a prod database without a where clause
The dba wannabe award - one of our support guys was clearing out a prod dB server to make some disk space and accidentally deleted one of the databases devices bringing it down.
The open source community award - one of the devs had been messing about with an apache proxy on a prod web server and it ended up as part of a botnet
There were others but I can’t remember them all5
The website for our biggest client went down and the server went haywire. Though for this client we don’t provide any infrastructure, so we called their it partner to start figuring this out.
They started blaming us, asking is if we had upgraded the website or changed any PHP settings, which all were a firm no from us. So they told us they had competent people working on the matter.
TL;DR their people isn’t competent and I ended up fixing the issue.
Hours go by, nothing happens, client calls us and we call the it partner, nothing, they don’t understand anything. Told us they can’t find any logs etc.
So we setup a conference call with our CXO, me, another dev and a few people from the it partner.
At this point I’m just asking them if they’ve looked at this and this, no good answer, I fetch a long ethernet cable from my desk, pull it to the CXO’s office and hook up my laptop to start looking into things myself.
IT partner still can’t find anything wrong. I tail the httpd error log and see thousands upon thousands of warning messages about mysql being loaded twice, but that’s not the issue here.
Check top and see there’s 257 instances of httpd, whereas 256 is spawned by httpd, mysql is using 600% cpu and whenever I try to connect to mysql through cli it throws me a too many connections error.
I heard the IT partner talking about a ddos attack, so I asked them to pull it off the public network and only give us access through our vpn. They do that, reboot server, same problems.
Finally we get the it partner to rollback the vm to earlier last night. Everything works great, 30 min later, it crashes again. At this point I’m getting tired and frustrated, this isn’t my job, I thought they had competent people working on this.
I noticed that the db had a few corrupted tables, and ask the it partner to get a dba to look at it. No prevail.
5’o’clock is here, we decide to give the vm rollback another try, but first we go home, get some dinner and resume at 6pm. I had told them I wanted to be in on this call, and said let me try this time.
They spend ages doing the rollback, and then for some reason they have to reconfigure the network and shit. Once it booted, I told their tech to stop mysqld and httpd immediately and prevent it from start at boot.
I can now look at the logs that is leading to this issue. I noticed our debug flag was on and had generated a 30gb log file. Tail it and see it’s what I’d expect, warmings and warnings, And all other logs for mysql and apache is huge, so the drive is full. Just gotta delete it.
I quietly start apache and mysql, see the website is working fine, shut it down and just take a copy of the var/lib/mysql directory and etc directory just go have backups.
Starting to connect a few dots, but I wasn’t exactly sure if it was right. Had the full drive caused mysql to corrupt itself? Only one way to find out. Start apache and mysql back up, and just wait and see. Meanwhile I fixed that mysql being loaded twice. Some genius had put load mysql.so at the top and bottom of php ini.
While waiting on the server to crash again, I’m talking to the it support guy, who told me they haven’t updated anything on the server except security patches now and then, and they didn’t have anyone familiar with this setup. No shit, it’s running php 5.3 -.-
Website up and running 1.5 later, mission accomplished.6
Yesterday I spent some time on the meta site for dba.stackexchange.com and found this one guy with 1 rep raging about how his questions aren't getting answered and how is answers are the best etc...
"I have 17 years of experience as a dba, blah, blah, blah, my answers are correct, blah, blah"
He got pretty destroyed by the mods and other users about how shit his answers were and how they weren't factually correct etc...
This just continues to show that no matter how much experience you have you won't always be right.
Same goes for my senior at work, he has 10 years more experience than me (I have 2) and he still asks for my point of view and help without being a dick about it.
I hope we'll all keep being nice people unlike that Stackexchange guy...4
So here is my week 72 as a reviewer. But first, let me ask y'all. Am I weird to think that you should finish coding the thing and testing the thing before kicking it out to review? Cuz that's how I do it. And that is the process at my work place.
So anyway, I was doing this review. And it was very wrong. Like really, really wrong. We give a thorough intro to our product (perhaps too thorough) so this guy should have known every test case he had to cover. Or at least, if he was unsure, asked. It was all documented.
Anyway, he kicks out this peer review. First thing I notice, it is not following the standard. Fair enough, we didn't give him the coding standard. BUT HE DIDN'T EVEN MAINTAIN THE FORMAT OF THE ORIGINAL FILE. HE JUST DID HIS OWN THING!!! So I email him the coding standard and make a comment in the review. He denies the finding. No reason. Just turns it down. Strike 1.
Then, I'm going through and he didn't even cover all of the core cases. I found several core cases that he missed. And every edge case. Make a not of it. He fixes only the couple of examples I gave him. Strike 2.
Guy decided to redesign a major chunk of our interfaces. Our interfaces are not just used by us (hence interfaces). We designed them the way they were for a reason. It was not a fun design process. Myself, the architect, one of our customers, and the guy that did the implementation all told him to roll back his change. Especially since it wasn't in the scope of what he was doing. He wouldn't. Strike 3.
I go to the lead and bring him in. He has a talk with him. All of the sudden he is putting out multiple builds per finding. Like most times I will put out like 2 to 4 for the whole peer review. No he kicks out minimum one per finding and chokes the review queue. Strike 4.
Strike 5: he tells me, a former DBA, that I didn't know what I was talking about when I told him to move something into a new table, even after I told him that "while in database terms it doesn't make sense, this is for product robustness".
Strike 6: he was just a condescending asshole. Bragging about how he did this job and that job over his career. His longest position held was about 18 months. Bragged about working at my company and being some hotshot at the company: only worked here for 8 months and that was 5 years ago.
You know. I have never really wanted to fight someone after about undergrad. But he came close.9
*me trying to add one more field to a SQL request*
*asks coworker* Hey, do you know how I'd get this field? You have to go through 4 tables just to find it because of the way the tables are related.
*coworker looks for a bit* Oh. That's crazy. We should ask the DBA.
*we ask our 60 year old DBA for some help.*
He walks us through a minefield of inner and outer joins and selects and we finally arrive at a solution. Kudos to our DBA, who in the eyes of us young devs is a wizard.3
Got in trouble today during a Data integrity meeting, everyone kept talking about data massaging. "Massage this data", "massage that data table", "insert massaged data".
Finally I just blurted out, "yeah massage it all you want but how do I get a data happy ending?"
I thought it was hilarious. The other DBA and backend devs thought it was jokes, my manager... Not so much.
Apparently, I need to keep "thoughts and comments about data happy endings to myself moving forward".
TLDR - you shouldn't expect common sense from idiots who have access to databases.
I joined a startup recently. I know startups are not known for their stable architecture, but this was next level stuff.
There is one prod mongodb server.
The db has 300 collections.
200 of those 300 collections are backups/test collections.
25 collections are used to store LOGS!! They decided to store millions of logs in a nosql db because setting up a mysql server requires effort, why do that when you've already set up mongodb. Lol 😂
Each field is indexed separately in the log.
1 collection is of 2 tb and has more than 1 billion records.
Out of the 1 billion records, 1 million records are required, the rest are obsolete. Each field has an index. Apparently the asshole DBA never knew there's something called capped collection or partial indexes.
Trying to get approval to clean up the db since 3 months, but fucking bureaucracy. Extremely high server costs plus every week the db goes down since some idiot runs a query on this mammoth collection. There's one single set of credentials for everything. Everyone from applications to interns use the same creds.
And the asshole DBA left, making me in charge of handling this shit now. I am trying to fix this but am stuck to get approval from business management. Devs like these make me feel sad that they have zero respect for their work and inability to listen to people trying to improve the system.
Going to leave this place really soon. No point in working somewhere where you are expected to show up for 8 hours, irrespective of whether you even switch on your laptop.
Wish me luck folks.4
I think the worst work culture you can experience is nepotism and corruption in hierarchy. What do I mean? Well, this happened (and I think is still happening) in my last job. It was a huge logistic/delivery company. I was an intern, working as assistant developer of the only developer of the site. There was also a guy that was the technician, his assistant, a DBA and that's it.
Well, my partner and I were working on a system that managed almost all the operations of the company in this city.
Well, I supplied the dev two weeks when he was on vacation. I knew almost all the system. what happened? the manager from other city came with another Dev, and I'm not saying that I was an expert or something like that, but that dev from the other city was an incompetent. He couldn't even make a small GUI change without messing it all...
Guess what? The company paid him weekly round tickets to come and go from his city to ours (two hours of flight).
I was too disappointed I started searching another job. A week after getting my degree, I left my job and started in the one I am now. Before leaving, I asked my boss if there was a realistic chance to grow up. He answered no. To be honest, that didn't surprised me :/
The thing that makes me angry about this is that a lot of companies give chances to people that come from other cities, even if they don't know anything >:v
Oh, I almost forgot it: The last five months I was working there, they quit our office and send us to trailer-offices :/1
Dev: I think I just deleted data I shouldn't have. Can you do a rollback?
Me: No, I'm a DBA, I don't touch data. Did you commit?
Dev: Yes, I committed.
Me: Your data is gone.
Dev: But don't you guys do backups?
Me: We backup the Prod servers. We only back up Dev on request. Did you request a backup in Dev?
Dev: The Testers are going to be pissed.2
On chat today.
Dude: can you run a script for me? We don't have permission.
Me: what kind of script? Who wrote it?
Dude: posts screenshot of DML select/update statement he tried to run.
Me: I'm a DBA. We don't run DML for people.
Dude: Oh. Can you give me the password?
Me: examine script and notice he tried to run it on QA DB.
Me: No. We don't memorize passwords, and this is QA; you need to check the password out of the safe. You also need a change ticket to DevOps, and they will run it for you.
At that point I ended the discussion, because running anything in QA or Prod without a change ticket gets you fired. And I like my job. Really annoyed.3
Never mess with a motivated developer. I will make your life difficult in return.
Me: we need server logs and stats daily for analysis
DBA: to get those, you need to open a ticket
Me: can't you just give me SFTP access and permissions to query the stats from the DB?
*Writes an Excel Template file that I basically just need to copy and paste from to create a ticket*
This process should not take me more than 2mins 👍😁😋🙂😙😙😙😙😙😙😙😙
For them.... 😈😈😈😈😈😈😈😈😈😈😈10
I find a poor tester copy/pasting data from the test environment to the live one, as he accidentally broke it. I ask the DBA, " why isn't syncing SQL records part of the deployment pipeline?"
"You're front end. This is my job. Go do your job."
"... but it's an easy query, and you're exposing us to human error."
"You need to go sit down."2
I just had the most surreal email discussion I think I've ever had...
I spent over two hours going back-and-forth over email with an enterprise DBA, trying to convince them I needed a primary key for a table. They created the table without a primary key (or any unique constraints... or indexes... but that's another discussion). I asked them to add one. Then had to justify why.
If you ever find yourself justifying why you need a primary key on a table in an RDBMS, that's the day you find yourself asking "is this real life?"
I want the last two hours of my life back. And a handful of Advil.1
Top 10 database engines on db-engine... I never knew the top 3 were so far ahead of the game, if that site is anything to go by.15
Nooooooooo! My wife told one of her friends that I'm a DBA. The response. "Oh. That's good to know. I've been having this problem with my cell phone." No! Why? Aughhh!
I JUST HAD ONE OF THOSE DAYS THAT MAKES ONE WANT TO BANG TWO BRICKS ON HEAD SND END THE PAIN THE STORY STARTS YESTETDAY WITH ISSUES AFTER A MIGRSTION AND THEY ASK ME TO HELP TROUBLESHOOT EVEN THOUGH I'M A DEV DBA AND THE ISSUE IS IN QA/SAT AND I HELP ANYWAY AND THEY CAN'T FIND A VIEW AND SO I LOOK EVERYWHERE AND CAN'T DOING IT EITHER AND IT DIDN'T EXIST IN PROD OR DEV SO I TELL THEM IT'S NOT THERE, AND THEY ARE LIKE, CAN YOU RETRIEVE IT FOR US AND I'M LIKE FROM WHERE? I DON'T KEEP VIEWS IN MY BUTT AND YOU GUYS ARE SMOKING CRACK AND THE GIVE ME THEIR QUERY WHICH CONTAIN THE VIEE ANYWAY AND THEY SAY CAN YOU RUN IT AND IT RUNS AND WORKS AND THEY CAN'T MAKE IT WORK AND IT WORKS BECAUSE IT DOESN'T CALL THE VIEW THEY HAVE ME SO NO PROBLEM THERE SO I FINALLY ASK THEM ARE YOU POINTING TO THE CORRECT DATABASE AND THEY'RE LIKE OH MAN WE TOLD YOU THE WRONG DATABASE AND SO I LOOK AT THE RIGHT DATABASE AND FIND THAT THE GRANTS ARE MISSING AND YEAH THANK YOU FOR TAKING EIGHT HOURS OF MY LIFE BECAUSE WE WERE IN THE WRONG DB YOU GAVE ME AND I HOPE THE FLAG OF A THOUSAND CAMELS INVEST YOUR ARMPITS AND THE CHIGGERS OF A THOUSAND SOUTHERN LAWNS INGEST YOUR SOCKS AND UNDERWEAR. YAAAAAA!!!!20
SQL Rule 1. Always assume there are external processes that might affect your data. (for instance, triggers).
SQL Rule 2. In Denormalised data, never execute logic on dependant table values, always copy from the parent.
SQL Rule 3. When Denormalised data schemas are created the DBA knows what they are doing.
SQL Rule 3.1. If DBA knows what they is doing then according to Rule 1 there is no problem with adding in some triggers to maintain data clones as they are created.
SQL Rule 4. If you don't like or agree with triggers, deal with it. They are a first class tool in a first class RDBMS. In a multi-app or service environment there may be many other external processes massaging your data
SQL Rule 5. If all previous rules are not broken and the system has been running efficiently for many years DO NOT complain that there are triggers in the database that are doing and have been doing the same process that you just butchered (by violating Rule 1 and 2) in your makeshift "hello world, look what I can do from my phone" angular BS when the rest of the users are still relying on the existing runtime app.
SQL Rule 6. If you turn my triggers off, you sure as hell better turn them back on!1
DBA bitches about one of our developers SQL... implement changes... deploy... now it takes 12 minutes... up from 40 seconds... good job!7
The team headed to Arby's for lunch today. Our DBA walks up to the counter.
arby's guy: "Hey, what can I get you."
dba: "Hello. Yes, I'd like the largest french dip you are legally allowed to serve me."
Absolutely slayed everyone in-line and at the counter.3
In order to reduce support costs, manager instructed his team to remove all logging/reporting of errors in the company’s CRM application.
Team’s support tickets went down 80%, manager received an award for his efforts, but mysteriously, DBA/support workload increased, bad/missing data,
increased support tickets in other areas of the business (shipping, etc. that relied on correct data from the CRM) and other side-affectual behavior.
Even after pointing this out this correlation, showing before/after code, no one believed the two were related and I was accused of not being a ‘team player’.
“You and the other teams need to learn from his example!”. As ‘punishment’ was I was moved to the team managing the CRM application.1
Hey, DBA guy! Security blocked this one port for the database. Can you change the database to a different port?
Me: No, I actually like working here.2
Today was a manic-depressive kind of day. Spent the morning helping some developers with getting their code to run a stored procedure to drop old partitions, but it wasn't working on their end. It was a fairly simple proc. But working with partitions is a little like working with an array. I figured out that they were passing the wrong timestamp, and needed to add +1 to delete the right partition. Got that sorted out, and things were good. Lunch time.
After lunch I did some busy work, and then the PO comes up at about 2PM and says he's assigned some requests to me. The first was just attaching some scripts. Easy. The second, the user wants a couple of schemas exported ... at 6PM. I've been in the office since 6:45AM.
While I'm setting up some commands to run for the data export, a BA walks up and asks if I'm filling in for another DBA who is out for a few weeks. Yep. There's a change request that hasn't been assigned, and he normally does the work. I ask when it's due. Well, the pre-implementation was supposed to be done in the morning, but it wasn't, and we're in the implementation window ... half way through. I bring up the change task, and look at. Create new schema and users. That's all it says. The BA laughs. I tell I need more to go on. 10 minutes later he sends an email with the information. There's only two hours left in the window, and I can only use half of it, because the production guys have to their stuff, and we're in their window. Now I'm irritated, because I'm new to Oracle, and it's an unforgiving mistress. Fortunately, another DBA says he'll do it, so that we can get it done in time. But can't work it either, because Dev DBAs don't have access to QA, and the process required access for this task. Gets shelved until the access issue is resolved. It's now after 4:15PM. I'm going to in traffic with that 6PM deadline.
I manage to get home and to the computer by 5:45PM. Log in. Start VPN. Box pops on screen. Java needs to update. I chose skip update. Box pops up again. It won't let me log in until Java is current. Passed.
I finally get logged in, and it's 6:10PM. I'm late getting the job started. I pull up Putty and log into the first box, and paste my pre-prepared command in the command line and hit error. Command not found. I'm tired, so it's a moment to sink in. I don't have time for this.
I log into DBArtisan and pull up the first data base, use the wizard to set the job, and off it goes. Yay. Bring up the second database, and have enter the connect info. Host not found. Wut? Examine host name. Yep, it's correct. Try a different method. Host not found. Go back to Putty. Log in. Past string. Launch. Command not found. Now my brain is quitting on me. Why now? It's after 6:30PM. Fiddle with some settings, reset $Oracle home. Try again. Yay. It works. I'm done. It's after 7PM.
There is nothing like technology to snatch the euphoria of a success away from you. It's a love-hate thing, but I wouldn't trade it for anything else. I'm done. Good night.3
I decided to quit my job. I was supposedly hired as an Android developer, but during the past few months I found myself doing everything except Android development: SQL scritps, frontend web development, backend web development, RESTful API's, DBA, release engineering... There's nothig wrong about being versatile, it's actually fun, but I wasn't doing what I really wanted to do and, most importantly, the manager didn't appreciate the fact that I was doing things that I'm not supposed to do, and not only that, I was doing it just as good as some full time web devs who do nothing but web development in the company. I'm pissed off. They probably believe the next Android dev they hire will do all the shit I was doing, accepting the same pay. Fuck them.3
This was not exactly the worst work culture because the employees, it was because the upper level of the organization chart on the IT department.
I'm not quite sure how to translate the exact positions of that chart, but lets say that there is a General Manager, a couple of Area Managers (Infrastructure, Development), some Area Supervisors (2 or 3, by each area), and the grunts (that were us). Anyway, anything on the "Manager" was the source of all the toxicity on the department.
First and foremost, there was a lack of training for almost any employee. We were expected to know everything since day-1. Yes, the new employees had a (very) brief explanation about the technologies/languages were used, but they were expected to perform as a senior employee almost since the moment they cross the door. And forget about having some KT (Knowledge Transfer) sessions, they were none existent and if they existed, were only to solve a very immediate issue (now imagine what happened when someone quit*).
The general culture that they have to always say "yes" to the client/customer to almost anything without consulting to the development teams if that what was being asked to do was doable, or even feasible. And forget about doing a proper documentation about that change/development, as "that was needed yesterday and it needs to be done to be implemented tomorrow" (you know what I mean). This contributes to the previous point, as we didn't have enough time to train someone new because we had this absurd deadlines.
And because they cannot/wanted to say "NO", there were days when they came with an amount of new requirements that needed to be done and it didn't matter that we had other things to do. And the worst was that, until a couple of years (more or less), there was almost impossible to gather the correct requirements from the client/user, as they (managers) "had already" that requirement, and as they "know better" what the user wants, it was their vision what was being described on the requirements, not the users'...
And all that caused that, in a common basis, didn't have enough time to do all this stuff (mainly because the User Support) causing that we needed to do overtime, which almost always went unpaid (because a very ambiguous clause of the contract, and that we were "non-union workers"**). And this is my favorite point of this list, because, almost any overtime went unpaid, so basically we were expected to be working for free after the end of the work day (lets say, after the 17:00). Leaving "early" was almost a sin for the managers, as they always expected that we give more time to work that the indicated on the contract, and if not, they could raise a report to HR because the ambiguous clause allowed them to do it (among other childish things that they do).
Finally, the jewel of the crown, is that they never, but never acknowledge that they made a mistake. Never. That was impossible! If something failed on the things/systems/applications that they had assigned*** it was always our fault.
- "A report for the Finance Department is giving wrong information? It's the DBA's fault**** because although he manages that report, he couldn't imagine that I have an undocumented service (that runs before the creation the report) crashed because I modified a hidden and undocumented temporal table and forgot to update that service."
But, well, at least that's on the past. And although those aren't all the things that made that workplace so toxic, for me those were the most prominent ones.
* Well, here we I live it's very common to don't say anything about leaving the company until the very last day. Yes, I know that there are people that leave their "2-days notice", but it's not common (IMHO, of course). And yes, there are some of us that give a 1 or 2-weeks notice, but still it's not a common practice.
** I don't know how to translate this... We have a concept called "trusted employee", which is mainly used to describe any administrative employee, and that commonly is expected to give the 110% of what the contract says (unpaid overtimes, extra stuff to do, etc) and sadly it's an accepted condition (for whatever reasons). I chose "non-union workers" because in comparison with an union worker, we have less protections (besides the legal ways) regarding what I've described before. Curiously, there are also "operative workers", that doesn't belong to an union, but they have (sometimes) better protections that the administrative ones.
*** Yes, they were in charge of several systems, because they didn't trust us to handle/maintain them. And I'm sure that they still don't trust in their developers.
**** One of the managers, and the DBA are the only ones that handle some stuff (specially the one that involves "money"). The thing that allows to use the DBA as scapegoat is that such manager have more privileges and permissions than the DBA, as he was the previous DBA2
I actually never felt the need to scream at a co-worker so let's talk about that time a co-worker screamed at me instead.
tl;dr : some asshole boss screamed and threatened me because someone else's project was shit and didn't work.
Context: I was in my third year of school internship (graded) and my experience is C, C++, C#, Python all in systems programming, no web.
I was working as an intern for a shit company that was selling a shit software to hospitals (though not medically critical, thank God) the only tech guy on site was the DBA (cool guy) the product was maintained by a single dev in VB from his house, the dude never showed up to work (you'll understand why) and an other intern who couldn't dev shit.
I was working with the DBA on an software making statistical analysis from DB exports, worked nice, no problems here if we forget the lack of specs or boundaries (except must work in ieShit).
The other intern was working on something else (don't ask me what it is) I just remember it was in GWT before the community revived it. His webapp was requesting the company http server for a file instead of having one of it's java servlet to fetch it (both apps ran on sane server) which caused a lot of shit especially CORS error. That guy left (end of contract) and leaves his shit as is, boss asked me to deploy the app, I fiddle with it to see if it works and when I find out it doesn't then that asshole starts screaming at me in front of every other employee present, starts threatening to burn me in the tech world and have me thrown out of my school for no goddamn reason than the other dude's project doesn't work.
After the screaming I leave and warn my school immediately.
I guess that's why the other dev never came to work.
I had three weeks of internship left, that I did from home and worked probably less than 2 hours a day so suck it asshole.
Still had a good grade because I was reviewed by the DBA and he was happy with the work I did.
It was only later that I realized that what he did was categorizing as harassment (at least in France) and decided that never again this would happen without a response from my lawyer.1
Ever had a day that felt like you're shoveling snow from the driveway? In a blizzard? With thunderstorms & falling unicorns? Like you shovel away one m² & turn around and no footprints visible anymore? And snow built up to your neck?
Today my work day was like that.. xcept shit..shit instead of pretty & puffy snow!!
Working on things a & b, trying to not mess either one up, then comes shit x, coworker was updating production.. ofc something went wrong.. again not testing after the update..then me 'to da rescue'.. :/ hardly patch things up, so it works..in a way.. feature c still missing due to needed workarounds.. going back to a and b.. got disrupted by the same coworker who is nver listening, but always asking too much..
And when I think I finally have the b thing figured out a f-ing blocker from one of our biggest clients.. The whole system is unresponsive.. Needles to say, same guy in support for two companies (their end), so they filed the jira blocker with the wrong customer that doesn't have a SLA so no urgent emails..and then the phone calls.. and then the hell broke loose.. checking what is happening.. After frantic calls from our dba to anyone who even knows that our customer exists if they were doing sth on the db.. noup, not a single one was fucking with the prod db.. The hell! Materialised view created 10 mins ago that blocked everything..set to recreate every 10 minutes..with a query that I am guessing couldn't even select all that data in under 15.. dafaaaq?! Then we kill it..and again it is there.. We found out that customers dbas were testing something on live environment, oblivious that they mamaged to block the entire db..
FML, I'm going pokemon hunting.. :/ codename for ingress n beer..3
10 year anniversary 'celebration' for a couple of employees (one dev, one a DBA) and the VP of the department was saying kind words about them, talking about the 'good old days'.
VP to the DBA: "I apologize, when you started, you walked into my database architecture. I didn't know that much back then and never thought about the architecture much beyond a few years. Its amazing my design has lasted over 20 years and triple digit business growth..blah blah blah"
Inner voice: "Mother F-er!...My database was designed IN SPITE of your meddling and demanding to create 1,500 field tables. Shut the F up you egotistical bastard!"
I can't even count how many times I had to stop him from, for example, adding a 'ProductID' field to a Customer table.
Me: "Why did you add a product id field?"
VP: "How else will we know what product the customer wants to buy?"
Me: "You mean like a wish list? What if the customer wants more than one product?"
VP: "Oh, that’s easy, we'll create more fields when that happens. ProductID2. Microsoft made it really easy to add fields."
Me: "We already have a wish list table schema. Customer can have as many wish lists and as many products as they want."
VP: "I don't understand. All I want is a field for me to store the product I'm buying. I don't know why you make this so hard, its just one more field."
Now the VP is bragging all the success was due to his expertise?! Gaaaaahhhh!
I quelled my rage with ample quantities of donuts, juice, and chocolate milk.
I dunno about coolest, but I did sort of cement my reputation as the "database guy" in my first job because of this.
My first job was with a group maintaining a series of websites. Because of the nature of the websites, every morning we had to pull the records from one database on one network, sneaker net the data to a database on another network, and import the data via custom data import function.
However, the live site would crash after 100 or so records were imported. The dba at the live site had to script out a custom data partitioning script to do his daily duties, but it definitely messed up his productivity.
Turns out, the custom mass import function had recycled the standard import function, which was only used to import 1 record at a time, and it never closed its database connections, because it never needed to. A one line fix to production code was delivered 6 months later (because that was our release cycle) and I came up with the temporary work around, which was basically removing the connection limit. It would still crash with the work around, but only with multiple days worth of data. So basically only on Monday. Also developed the test set for the import (15k+ records).
rantPercentage := .25 * RANT_AVG
tldr := "Looking for a new project/job/mentor after a problem with my 'job'"
body := `
I've been working for a while now with a smaller minecraft network (hold up now, this is serious, don't walk away yet) for free. It was an amazing opportunity for me. I had the chance to work in a team on a common goal. They had equipment that I otherwise wouldn't have access to, and people who were serious about getting things done, unlike mostly all others. We had almost everything a normal business had- multiple departments, lots of people that sometimes worked through the night, proper version control on software, etc. While others were paid for their work, I chose not to be; I was doing this completely for experience. I want to be ahead for college and for a job as much as possible, so I've dumped most of my free time into this. I was a junior developer, head of security, DBA, and sysadmin. The biggest java and kotlin projects I have ever made, and the ones I was most proud of went to this network. I challenged myself in everything I did, and improved in programming tenfold since I started. I just recently spent three days on their server, setting it up properly, because someone thought managing a control panel was too much work and we need to switch to SSH. So I worked on this server alone for three days, every minute of my free time, setting it all up, and man, I thought it was a thing of beauty. It all made sense and was so simple to manage servers my grandma could do it. Made multiple improvements- iptables was configured, ssh keys were used instead of passwords, ACL was used to manage users' permissions for finer access control to the files, to name a few. I had planned on setting up fail2ban, MySQL and Postgres databases, a website, a couple Go programs to make creating servers even easier, backups to an external server with cron, the works. So after spending in excess of 45 hours on this project (learning tons along the way), I had about 13 servers up and running in an organized fashion, with startup scripts and permissions all done. This was the best setup yet. I went to sleep, got up in the morning, and found out that they had reinstalled everything again without saying anything, wiping out all my work, and had stayed up all night setting up a control panel to get 3 servers running, which they're still working on, and may get it done in a couple more days. So all my work was wasted. A part of me is fine with that I guess, sure it wasted a ton of time on my part but I still learned a lot. But the fact that they just deleted it all without warning and decided to change to another system entirely because it was too much work to learn the new way, after making me set everything up alone without help, having to deal with multiple people breathing down my neck and trying to get people to respond so I could get my work done, annoys the hell out of me. So I decided to take a break from them.
Now I'm looking for a new way to improve in everything I do. I want to get better at java, kotlin, golang, sql, everything related to system administration, database administration, back end, and maybe even a little frontend. I want to be the best developer I can be. The challenge of learning something new is actually fun. I just need a new project, or place to help. Unfortunately, most internships start after college, so that isn't an option, and being a janitor at a small business won't help me much unless I look over other peoples' shoulders when they're working. Open source projects would be interesting, but I don't know if I'd be able to ask anyone for help or opinions on anything. The perfect situation would be working for someone, and having a mentor that really knows their stuff to help me become better. Working on personal projects only gets me so far so fast; it's mostly a cycle of doing something a bunch of different ways because I don't learn about an alternative way to do it until I'm mostly done. Also, if I worked with people in an actual place, I'd get a feel for the environment and for how all the systems worked together. Finally, it'd show me how everything is done properly (hopefully) and how software development in the real world is. A real project, in a real team would be a Godsend for me. I'm not asking for one here, obviously, I just want to know- is this possible for me? I know people my age aren't often hired for this, but I really want to learn and improve. I don't have a degree, I'm self taught in everything. I've been using java for two years, kotlin for a half, golang for less. I know it's unlikely. Just.. how can I try to get this kind of situation, if possible? Thanks.
Let me explain:
Tl;dr : someone fucked up, I took shit, it was a gwt project. In a sense I don't hate GWT because of the framework itself but because how I was introduced and forced to "work" with it.
Was working as a paid intern at a small company there were 3 devs 2 interns and one senior employee that only worked from home handling the shit ton of legacy VB6 code he wrote over several year and a boss with no technical knowledge. (Other unimportant people as well)
I was working with their DBA (cool dude) because I was writing statistic and report generating software.
The other intern was tasked of doing a gwt app that was supposed to use a input file.
Rather than asking the user to upload it with a file picker (I guess they exist in gwt I didn't got to dig in the framework) he was trying to load the file with a http request directed at the same host the app was running on.
It did not work.
Then his contract was other and the app was left in an unfinished state.
The boss then tried to have the app deployed, the remaining dev dodged the bullet invoking some bullshit because he was clearly incapable of doing it.
So it fell on me, couldn't deploy the app because it was not even close to working.
Tried to fix things and make it work.
Turns out he thought it would take me 3h to deploy when I clearly explained that the other guy didn't finish the app.
Boss got mad, threatened to ruin my studies and my future career.
Couldn't because my uni had my back.
Didn't want to see me anymore.
Couldn't break my contract.
Told me to work from home for the end of my internship.
I got 3 weeks early vacation and got paid, fuck him, fuck GWT, fuck his company.
Still got well marked for the internship as my supervisor was the DBA who was happy with my work.
Don't let your intern unsupervised, don't let your main dev work from home when you don't know shit, don't piss me off and send me work from home.1
me to dba: do you have any recommendations of sql or query improvements? dba: no, just let us know once you're done.
after sending them explain plans, new queries and asking for reviews with no response from them, i applied the changes in dev.
after applying changes.
dba: you should involve us in any development. we need to collaborate.
me: please check your emails over the past 3 weeks.
Project requirements include a database. I don't have permissions to create a new database on the server, so I go to the person that spins up new servers and deals with group policy. They rustle some papers around, looking aggravated, throws up hands and says, "I guess I'm the DBA now..." Well, what the fuck am I supposed to do shithead? Ask the web team to do it? We don't have a DBA. My boss has been gone all week and, really, this isn't a hard task. You check a fucking box.
Whatever, I'll remember this when they need a favor from me.6
Management: "You are the DBA and BI head right?"
Management: "Awesome, so that means you can develop a quick software for me to use? It's really easy!"
Me: "Awesome, here are two options since its easy, you go make it yourself or..."
Management: "Is that sarcasm?"
Me: "...or you can ask the accountant to make it"
IT Director: "you deserved that bob."
I went for an interview as a MS SQL DBA. They gave me a technical knowledge test which I passed then started asking about me, All was going well until they started telling me what the company did.
They harvested people's data from websites online questionnaires and collated it into saleable data.
This was too much for me and I'm not certain that it isn't a grey area legally. They were told that they were scum of the earth and they could go fuck themselves.4
Deployment to production last night, and the whole thing is halted because the DBA didn't find it necessary to show up. 2 hours of calling everyone possible and nothing. Thanks DBA, for your prompt and timely handling of the deployment... 😑
So they say they're missing some data in the database and that the architect thinks it is the code not sending the msg to log these into the database.
Now instead of giving me what wasn't logged or even OPENING A TICKET DIRECTED AT ME, or at least somewhere I can see it or an email they want to have a code review with 5 people (1 of which is q&a and another a DBA).
They're also putting me on standby for this issue. Even though the architect says he will not disclose info until he knows what the issue is in ~an hour.
Is this normal..? I'm finding it this very odd. Up until now they'd make a ticket and I'd research the issue and fix it or explain why it doesn't work. I don't think we've had issues doing it that way. Feels so... Undermining.2
Sysadmin's nemesis: a DBA. Especially an oracle DBA. There's no other kind of tech worker I've seen who's more opposed to best practices.
How about for devs?4
Completely fucked up replication of MySQL servers.
Remote: 2 different database Servers
--> made sense.
Except the misconfiguration. Or better: No configuration at all.
So how to solve the massiv delays and make everything even more crazy?
2 remote servers - 2 readonly slaves for reading data remote (master - slave)
2 local (internal) servers.
Remote - Local Master Master.
Unfucking this cluster fuck was a real nightmare.
It had to be done at night, cause everything needed to be ripped apart.
And the servers were the backend of a warehouse with supply chain and multiple selling channels (Amazon, eBay etcetera).
So. It had to run the next day at 05.00 clock so the incoming orders could be packaged / prepared for shipping.
That was fun. Not.
And the clusterfuck died spectaculously on my first work day - the old DBA was gone (fired....)
When you cache index a faster query but your co-worker from other part of the world clears it.... It's been six times now dude2
How can a shitty student information system that already costs $20k/yr have an optional shitty SOAP API module, that only allows read access to records, that has an initial setup fee of $5k plus $5k/yr?!5
I am a backend dev in a mostly web based enterprise, i haven't touched c# for a month and a half so....
i am thinking in becoming a dba1
Was a dba for a while. Mostly because I was the only one who knew SQL. Was working with an experienced dev doing front end work with no experience with front end work.
One day he calls over the cube wall "hey the database is broken" so I trudge over there, and see he messed up the call to the BAL from his code behind page. Later, he calls over the cube wall again. Same thing. 3 to 4 times a day. For a week. Finally my default became "no it isn't" and I continued working.
Then when it finally was a database problem, he had this smug look of satisfaction. Yes I'm the idiot.
I hate ot when your client has access to new relic and they panic and start creating critical issues when the linux db server is is 80% memory..
After a 2h conference call the client said he was going to contact a 3rd party dba because he didn't believe it was normal.
On this episode of “My Work Life” we will once again find that the code is fine, the DBAs just did something brainless again.3
When a senior engineer who was essentially a DBA on his last project is made tech lead on an iOS team. He had no iOS background before this and now he thinks hes a friggin architect.1
Just had a recruiter that actually got my attention. Said they need an Oracle DBA that is interested in being trained for DevOps/AWS/Docker, and wanted to know if I was interested. That was a hard one, but I already have great pay, benefits, and perks that I don't want to give up, and they can't guarantee to replace. Now, if I was tired of what I'm doing ...5
Created a hidden role in the Postgres database, with some tricky constraints, which only a DBA can find and remove, to give myself, disguised as the postgres account, a role which no one else in the system can have.
Called it "pharaoh". Thug life ;2
"Just try it on production"
- a coworker DBA shortly before he desperately had to figure out how to restore to the latest backup.
DBA tells us 2 weeks in advance about database upgrade, but we developers forgot as manager insists in managing tasks via emails.
Monday morning brings the news that our web app isn't functioning properly. DBA responded saying he informed us in advance.
Manager blames it on the DBA that he informed us too much in advance.
I can take my share of blame but why the poor DBA for doing his job.1
When your IT VP starts speaking blasphemy:
We all know what’s going on with the API. Next week we may see 6x order volumes.
We need to do everything possible to minimize the load on our prod database server.
Here are some guidelines we’re implementing immediately:
· I’m revoking most direct production SQL access. (even read only). You should be running analysis queries and data pulls out of the replication server anyway.
· No User Management activities are allowed between 9AM and 9PM EST. If you’re going to run a large amount of updates, please coordinate with a DBA to have someone monitoring.
· No checklist setup/maintenance activities are allowed at all. If this causes business impact please let me know.
· If you see are doing anything in [App Name] that’s running long, kill it and get a DBA involved.
Please keep the communication level high and stay vigilant in protecting our prod environment!"
RIP most of what I do at work.3
I once saw a DBA using two for loops in PL/SQL to join two tables. I wonder if he knew about the "alternative" way to do that...2
Before my vacation I’d been chatting with one of our dbas about an etl tool we needed for a customer we’d already signed all the contracts with saying we would provide one for a historical database of old data. They had been looking at one from SAP but in typical fashion a license was worth more than the actual contract.
Anyway long story short on the weekend before I went back to work I rattled together a little python proof of concept using a couple of azure databases and when I went back demo’d it to the pm and dba they loved it and we built on the poc to have a working loader which saved us about £30k by not buying the SAP product and just wrote our own.
Ha! Our Ops Support DBA Manager just asked (tongue in cheek) "if we are now supporting MS Access, too?" To which of course, the answer is no. Business user who install Access on their desktop and use it for business, get to provide their own support. As their Dev DBA, I'll be more than happy to help them migrate their data to SQL Server, Oracle, or Teradata, depending on the Use Case for the data. But, no, we don't support Access. Ever.
>how long do you think this migration will take?
>i don't know the database structures are entirely different
>so how long?
When your team can't change from the oracle database because the dba team will be out jobs and database links will be lost (people havent heard of apis)... :feelsoracleman:
dev = fighter
ops = cleris
network = wizard
design = bard
dba = warlock
infosec = rogue
it is not mine but i think its great analogy. and because i always choose the warlock1
Do you have a db specialist in teams? I often see frontend & backend ranting but no db Devs. Isn't that role so crucial as one lost index can cause massive performance drops?5
Idea for future week question, maybe not a rant 'per se', but interesting endeed: 'Describe the moment when you knew you wanted to be a programmer/developer/DBA..."
Our approval process for creating new database tables is ridiculous. After your table has been created it goes through two other men, both of whom have been woking with SQL for centuries and could actually just create the damn thing themselves and leave me the fuck out of it! Our DBA isn’t really much of a DBA...7
Developers don’t have access to the test database so I had to put in a request for a dba to add two columns and it took over two hours for them to complete it.5
Today the DBA-team (needed to keep our Oracle mess running) decided to change datatypes in tables that has been in production for 20 years. Without a headsup.
Just took us 5 hours to debug good ol' visual basic code......
Why would you do such a thing!?
Been told ill be worming with Oracle DB'S today, as someone who's never touched on Oracle stuff before. Any advice?4
So at the end of our sprint retrospective we (me and one other dev) apologize to the project owner for the fact that QA had to happen on dev for a few days because the database changes were not properly applied by the DBA. It was a one time thing, a little weird but not a big deal.
A few hours later we get a call from a new DBA saying that the project owner said that we wanted the old DBA off our project. So now we sound like assholes ruining a guy's self confidence and made working with him awkward as hell.
Thanks, project owner.
When the DBA restores the live db to staging and all your connections fail on passwords. Every fucking day!1
Just made the leap from a company with and I.T. Department of 8 people to a company with an I.T. Department of 5000 people. At the old company, I was a SysAdmin, DBA, Helpdesk, Programmer, etc. 7/8 were programmers. We all had our hands in pretty much every aspect of the companies technology. At the new company I am just a programmer. I only write code (well, I will soon. There is a LOT I have to learn about how this place operates). I worked at the old companies for almost 3 years (2 as an Intern, 10 months as Salaried Full-Time) so it's weird having most of those aspects of my old job stripped away, yet at the same time it's somewhat freeing in the sense that I am now only responsible for 1 slice of the pie, not the whole damn thing. Anyway, hope everyone is having a great monday!1
My team has a Database Admin 2 position open on the Arvest Career site. We are looking for someone with Data Warehousing/Data Integration background with SQL Server, ETL, SSIS, or equivalent. Also looking for a physical DBA with background in SQL Server, performance tuning, partitioning, DR/HA, Database migrations, dB refresh, dB restore, building out clusters.
That moment when you explain to your manager what the term "full-stack" means and next thing you know you've gone from being a "backend c#" to azure migration expert/dba/designer/api architect/network engineer/php wordpress developer... let's go back to when you'd never heard of full-stack
My uncle is a dba so gets it, my dad sort of gets it because he also works in the transport sector, my mom doesn't Really understand but knows it's computers. Everyone else thinks it's it support
Today, my boss comes up to me to tell me that the CR he had me set up was not created by the release manager because there wasn't a QA test ticket or DBA review ticket. Look, if you're gonna shoot shit down without any bullets loaded, at least have the safety off because you had 3 hours on Friday to tell me that before leaving it to Monday morning when my head is hotter than the roast I'm sippin' on.
Work email conversation with a COTS app support tech:
Tech: Could you provide me the following from Sql Server? (instructions followed)
Me: I am not the DBA, but I've sent them the request.
Tech: Could you send me the following from Sql Server while we wait? (Instructions followed)
Me: As I said, I don't have permissions to access what you want.
Tech: Oh, I see now, you most likely don't have proper access to be able to retrieve the information.
Me: ..yeah. Thanks.
Spent the day helping someone troubleshoot why they can't print from their Samsung tablet, because somehow as a DBA I "know how to do that stuff." Never figured out why it wasn't working.
My previous job was Engineer ( Ops part of DevOps), supporting the devs with VMs, configurations, dev and test environments, CI maintenance, DBA, DB-dev and such, it was sexy.
In my current company, I have no technical role, but today's task: build a small webpage in sharepoint in HTML.
And the perv part is: it's still bettet than having no Technical task at all...2
The application I work on starts throwing timeout errors for about every third user. Lead developer cannot figure out what happened. DBA is out of town and cannot be reached. I do a quick Google search and run the stored procedure sp_updatestats. Timeouts stop and there is a big performance boost on the application. Everyone congratulates me on fixing the problem, and now I'm reading up on MS SQL Server Statistics and wondering about what other magical tools everyone else knows about that me and my team are clueless on...
No matter how many times I educate them, my colleagues still end up confused between writing conditions in join statement and in where statement.
They think they're both same :'(
Things I say at work:
"...and the words 'ghetto' and 'database architecture' don't ever belong in the same sentence."