Do all the things like ++ or -- rants, post your own rants, comment on others' rants and build your customized dev avatarSign Up
From the creators of devRant, Pipeless lets you power real-time personalized recommendations and activity feeds using a simple APILearn More
Search - "dba"
*Now that's what I call a Hacker*
MOTHER OF ALL AUTOMATIONS
This seems a long post. but you will definitely +1 the post after reading this.
xxx: OK, so, our build engineer has left for another company. The dude was literally living inside the terminal. You know, that type of a guy who loves Vim, creates diagrams in Dot and writes wiki-posts in Markdown... If something - anything - requires more than 90 seconds of his time, he writes a script to automate that.
xxx: So we're sitting here, looking through his, uhm, "legacy"
xxx: You're gonna love this
xxx: smack-my-bitch-up.sh - sends a text message "late at work" to his wife (apparently). Automatically picks reasons from an array of strings, randomly. Runs inside a cron-job. The job fires if there are active SSH-sessions on the server after 9pm with his login.
xxx: kumar-asshole.sh - scans the inbox for emails from "Kumar" (a DBA at our clients). Looks for keywords like "help", "trouble", "sorry" etc. If keywords are found - the script SSHes into the clients server and rolls back the staging database to the latest backup. Then sends a reply "no worries mate, be careful next time".
xxx: hangover.sh - another cron-job that is set to specific dates. Sends automated emails like "not feeling well/gonna work from home" etc. Adds a random "reason" from another predefined array of strings. Fires if there are no interactive sessions on the server at 8:45am.
xxx: (and the oscar goes to) fuckingcoffee.sh - this one waits exactly 17 seconds (!), then opens an SSH session to our coffee-machine (we had no frikin idea the coffee machine is on the network, runs linux and has SSHD up and running) and sends some weird gibberish to it. Looks binary. Turns out this thing starts brewing a mid-sized half-caf latte and waits another 24 (!) seconds before pouring it into a cup. The timing is exactly how long it takes to walk to the machine from the dudes desk.
xxx: holy sh*t I'm keeping those
The bash scripts weren't bogus, you can find his scripts on the this github URL:
Yesterday: Senior dev messages out a screenshot of someone using an extension method I wrote (he didn’t know I wrote it)..
SeniorDev: “OMG…that has to be the stupidest thing I ever saw.”
Me: “Stupid? Why?”
SeniorDev: “Why are they having to check the value from the database to see if it’s DBNull and if it is, return null. The database value is already null. So stupid.”
Me: “DBNull is not null, it has a value. When you call the .ToString, it returns an empty string.”
SeniorDev: ”No it doesn’t, it returns null.”
<oh no he didn’t….the smack down begins>
Me: “Really? Are you sure?”
SeniorDev: “Yes! And if the developer bothered to write any unit tests, he would have known.”
Me: “Unit tests? Why do you assume there aren’t any unit tests? Did you look?”
<at this moment, couple other devs take off their head phones and turn around>
SeniorDev:”Well…uh…I just assumed there aren’t because this is an obvious use case. If there was a test, it would have failed.”
Me: “Well, let’s take a look..”
<open up the test project…navigate to the specific use case>
Me: “Yep, there it is. DBNull.Value.ToString does not return a Null value.”
SeniorDev: “Huh? Must be a new feature of C#. Anyway, if the developers wrote their code correctly, they wouldn’t have to use those extension methods. It’s a mess.”
<trying really hard not drop the F-Bomb or two>
Me: “Couple of years ago the DBAs changed the data access standard so any nullable values would always default to null. So no empty strings, zeros, negative values to indicate a non-value. Downside was now the developers couldn’t assume the value returned the expected data type. What they ended up writing was a lot of code to check the value if it was DBNull. Lots of variations of ‘if …’ , ternary operators, some creative lamda expressions, which led to unexpected behavior in the user interface. Developers blamed the DBAs, DBAs blamed the developers. Remember, Tom and DBA-Sam almost got into a fist fight over it.”
SeniorDev: “Oh…yea…but that’s a management problem, not a programming problem.”
Me: “Probably, but since the developers starting using the extension methods, bug tickets related to mis-matched data has nearly disappeared. When was the last time you saw DBA-Sam complain about the developers?”
SeniorDev: “I guess not for a while, but it’s still no excuse.”
Me: “Excuse? Excuse for what?”
<couple of awkward seconds of silence>
SeniorDev: “Hey, did you guys see the video of the guy punching the kangaroo? It’s hilarious…here, check this out.. ”
Pin shoulders the mat…1 2 3….I win.6
Yesterday, in a meeting with project stakeholders and a dev was demoing his software when an un-handled exception occurred, causing the app to crash.
Dev: “Oh..that’s weird. Doesn’t do that on my machine. Better look at the log”
- Dev looks at the log and sees the exception was a divide by zero error.
Dev: “Ohhh…yea…the average price calculation, it’s a bug in the database.”
<I burst out laughing>
Me: “That’s funny.”
<Dev manager was not laughing>
DevMgr: “What’s funny about bugs in the database?”
Me: “Divide by zero exceptions are not an indication of a data error, it’s a bug in the code.”
Dev: “Uhh…how so? The price factor is zero, which comes from a table, so that’s a bug in the database”
Me: “Jim, will you have sales with a price factor of zero?”
StakeholderJim: “Yea, for add-on items that we’re not putting on sale. Hats, gloves, things like that.”
Dev: “Steve, did anyone tell you the factor could be zero?”
DBA-Steve: “Uh...no…just that the value couldn’t be null. You guys can put whatever you want.”
DevMgr: “So, how will you fix this bug?”
DBA-Steve: “Bug? …oh…um…I guess I could default the value to 1.”
Dev: “What if the user types in a zero? Can you switch it to a 1?”
Me: “Or you check the factor value before you try to divide. That will fix the exception and Steve won’t have to do anything.”
<awkward couple of seconds of silence>
DevMgr: “Lets wrap this up. Steve, go ahead and make the necessary database changes to make sure the factor is never zero.”
StakeholderJim: “That doesn’t sound right. Add-on items should never have a factor. A value of 1 could screw up the average.”
Dev: “Don’t worry, we’ll know the difference.”
<everyone seems happy and leaves the meeting>
I completely lost any sort of brain power to say anything after Dev said that. All the little voices kept saying were ‘WTF? WTF just happened? No really…W T F just happened!?’ over and over. I still have no idea on how to articulate to anyone with any sort of sense about what happened. Thanks DevRant for letting me rant.15
Boss: "Could you join the new DevOps team for a week or two, for some coaching?"
Me: "I'd rather watch you masturbate furiously in a corner of the office while you cry over your ex boyfriend"
Boss: "Yeah... that's why I ask you. You are the only one brave enough to watch"
Me: *Sigh* "But I don't know shit about what DevOps does, I'm a DBA. I've told you the difference a million times. Can't we just douse it in gasoline and set it on fire?"
Me: "Not the team, the servers..."
Boss, imitating Gimli: "And my ex!"
Me: "I get why he left you"
Boss: "It's funny, he was actually better with computers than me, maybe even better than you. He hated me for starting this company, told me I was just chasing money instead of ideals. He just isn't grown up enough to see that there is more to the world than computer games, brewing beer, maker festivals and gay bars, that you need to take responsibility... Maybe it just never works out between managers and geeks..."
Me: "Indeed. The difference in competence is too large"
Boss: "Ugh. You are like straight version of him... but will you at least take a look?"
Me: "Fine, unzip your pants..."
Boss: "No, not that... you need to teach DevOps this docking thing, with the parallel stuff, and the horizontal growth"
Me: "Damn I really hope we're talking about servers now... Do you mean Docker?"
Boss: "That's it. They want to learn how to dock on the Windows servers. They reserved two 4xlarge on AWS. Is that enough for docking?"
Me: "You know what. I'm going back to hug my DB designs, and wash my brain with some queries. Then I'll return here to burn everything to the ground. There is no hope for you left"
Boss: "That's what he said"
Me: "You're using that meme wrong"
Boss: "OK. So what if you just stay on DB management, and I'll just give you the budget to recruit a new DevOps lead and pay for training?"
Me: "That would work"
Boss: "Why are you grinning?"
Me: "Because I have your ex's phone number"18
The way 90% of the population wears their face masks really explains a lot about their approach to using software, apps & websites as well.
I feel like giving up.
I am not a developer for the salary, or just to solve analytical puzzles. Those are motivators, but my main drive is to make the world more comfortable and enjoyable, better optimized, build ethical services which bring happiness into people's lives. I want to improve society, even if it's just a tiny bit.
But if users invest absolutely zero percent of their limited brain capacity into understanding a product that already has a super-clean design and responds with helpful validation messages...
...why the fuck bother.
I used to think of the gap between technology and tech-incompetent people as an optimization problem.
As something which could be fixed by spending a fortune on UX research. Write tests, hire QA employees, decrease tech debt, create a bold but unified & simple design.
But the technologically incompetent just get more entitled with every small thing you simplify.
It's never fucking fool-proof enough.
Why can't I upload a 220MB PDF as profile picture? Why doesn't the app install on my 9 year old Android Froyo phone? Why can't I sign up if my phone number contains a ￼ U+FFFC? Why does this page load so slowly from my rural concrete bunker in East Ukraine? WHY DO I HAVE PNEUMONIA, HOW DID I GET INFECTED EVEN THOUGH I WAS WEARING A MOUTH MASK ON MY FOREHEAD?
This is why I ran away from Frontend, to Backend, to DBA.
If I could remove myself further from the end user, I would.
At least I still have a full glass of tawny port and a huge database which needs to be normalized & migrated.
Fuck humans, I'm going to hug a server.30
I was once working on a project that dealt with incredibly sensitive financial data.
We needed a client’s database to do a migration.
They wouldn’t send it over the internet because it was too big and they didn’t think it would be secure.
They opt to send it in the post on an encrypted usb drive.
(Fair enough thinks I)
USB drive arrives.
Is indeed encrypted.
MFW there’s a post it note in the envelope with the password on.
MFW this is a billion dollar multinational petrochem company.
MFW this same company’s ‘sysadmin’ and ‘dba’ once complained because a SQL script I sent them didn’t work - they’d pasted it twice and couldn’t work this out from the fucking “table already created” error message management studio was throwing at them.3
I worked on a greenfield project a couple of years ago. The company had an old solution written in Omnis (heard of it? Yeah, me neither) with an SQL database. My team was to create a completely new web based system... on top of the old database, so the customers could keep their existing stuff.
The dba was an intelligent man, one of the nicest people I've met, and over the course of fifteen years he had made a remarkably terrifying monstrosity of a database. Some years before me they wanted to "future proof" the system and make it "easier to switch to new technologies". So they moved the entire business logic into the database...
I used a tool to create a visualization of said database when we started. It had no views, only tables and sprocs. Look at it! Tables and sprocs are rectangles (well, dots) and any connections are drawn in grey lines. There were no foreign keys, so a tables only visualization only yielded a collection of independent rectangles without a single line.
Now, the stored procedures were bloody MASSIVE. A single procedure that only registered a new interested party and attached them to a property had 2500+ lines and over 150 parameters.
Also, this dba added features and fixed bugs by logging into the respective customers production server and writing SQL.
That database is the stupidest thing I've ever seen a developer do.35
This is how I feel most of my client proposal start:
* It's simple, I'd like to re-invent <the wheel>.
* All I want to do is use <rocketship engine> on <old typewriter>.
* I'm too cheap to hire a full-time < DBA, DevOps engineer, development team>. Can I pay you pennies?
* I'm poor and broke, can you do this for free?
* I'd like to <commit illegal act> and be <legal compliant standard>.
* I heard it was possible to <fly 30 people to the moon> using <Ford Model-T>. Please do this for us.
* I <sold my house>, but now <I'm locked out by the new owners>. Please help.11
In a user-interface design meeting over a regulatory compliance implementation:
User: “We’ll need to input a city.”
Dev: “Should we validate that city against the state, zip code, and country?”
User: “You are going to make me enter all that data? Ugh…then make it a drop-down. I select the city and the state, zip code auto-fill. I don’t want to make a mistake typing any of that data in.”
Me: “I don’t think a drop-down of every city in the US is feasible.”
Manage: “Why? There cannot be that many. Drop-down is fine. What about the button? We have a few icons to choose from…”
Me: “Uh..yea…there are thousands of cities in the US. Way too much data to for anyone to realistically scroll through”
Dev: “They won’t have to scroll, I’ll filter the list when they start typing.”
Me: “That’s not really the issue and if they are typing the city anyway, just let them type it in.”
User: “What if I mistype Ch1cago? We could inadvertently be out of compliance. The system should never open the company up for federal lawsuits”
Me: “If we’re hiring individuals responsible for legal compliance who can’t spell Chicago, we should be sued by the federal government. We should validate the data the best we can, but it is ultimately your department’s responsibility for data accuracy.”
Manager: “Now now…it’s all our responsibility. What is wrong with a few thousand item drop-down?”
Me: “Um, memory, network bandwidth, database storage, who maintains this list of cities? A lot of time and resources could be saved by simply paying attention.”
Manager: “Memory? Well, memory is cheap. If the workstation needs more memory, we’ll add more”
Dev: “Creating a drop-down is easy and selecting thousands of rows from the database should be fast enough. If the selection is slow, I’ll put it in a thread.”
DBA: “Table won’t be that big and won’t take up much disk space. We’ll need to setup stored procedures, and data import jobs from somewhere to maintain the data. New cities, name changes, ect. ”
Manager: “And if the network starts becoming too slow, we’ll have the Networking dept. open up the valves.”
Me: “Am I the only one seeing all the moving parts we’re introducing just to keep someone from misspelling ‘Chicago’? I’ll admit I’m wrong or maybe I’m not looking at the problem correctly. The point of redesigning the compliance system is to make it simpler, not more complex.”
Manager: “I’m missing the point to why we’re still talking about this. Decision has been made. Drop-down of all cities in the US. Moving on to the button’s icon ..”
Me: “Where is the list of cities going to come from?”
<few seconds of silence>
Dev: “Post office I guess.”
Me: “You guess?…OK…Who is going to manage this list of cities? The manager responsible for regulations?”
User: “Thousands of cities? Oh no …no one is our area has time for that. The system should do it”
Me: “OK, the system. That falls on the DBA. Are you going to be responsible for keeping the data accurate? What is going to audit the cities to make sure the names are properly named and associated with the correct state?”
DBA: “Uh..I don’t know…um…I can set up a job to run every night”
Me: “A job to do what? Validate the data against what?”
Manager: “Do you have a point? No one said it would be easy and all of those details can be answered later.”
Me: “Almost done, and this should be easy. How many cities do we currently have to maintain compliance?”
User: “Maybe 4 or 5. Not many. Regulations are mostly on a state level.”
Me: “When was the last time we created a new city compliance?”
User: “Maybe, 8 years ago. It was before I started.”
Me: “So we’re creating all this complexity for data that, realistically, probably won’t ever change?”
User: “Oh crap, you’re right. What the hell was I thinking…Scratch the drop-down idea. I doubt we’re have a new city regulation anytime soon and how hard is it to type in a city?”
Manager: “OK, are we done wasting everyone’s time on this? No drop-down of cities...next …Let’s get back to the button’s icon …”
Simplicity 1, complexity 0.17
TL;DR: I resigned from my full-remote job and I came back to the old office.
Some months ago I celebrated the achievement of a full-remote position.
I was tired of driving every day to reach the office (especially in summer, because I had an old car without air conditioner). Moreover I did many different things (full-stack development, system administration, DBA, helpdesk) without a well-defined task list, and I thought that was bad. Everybody on the Internet seems happy to work remotely, and being an introvert I thought I would have no problem.
However I ignored 2 things when I left:
1. I was doing something that was important to me (my software helped scientists);
2. everybody loved me.
So, I started my new remote job as a back-end developer, for a company following the Scrum methodology.
My days became very monotonous: wake up, write some code, move a ticket on the Jira board, clean the house, go to bed. All in pajamas. Repeat.
I had very few interactions with my colleagues, mostly on Slack or audio-only calls. I never seen most of them. I didn't feel engaged in what I was working on. I did it just for the money.
After some months I started feeling very depressed and I realized that I left a job suited for me for a random job, just because it was a remote position.
So, one day I called my old boss: «Hey, would you like to have me back?»
I asked for a loan to buy a new car, then I send my resignation letter.
When I started again my first day I felt really touched, because basically everybody said to me: «Hey, welcome back! I'm happy you are here again.»
The guy that did some changes to my code also said: «It has been a pleasure to maintain your code. I learned a lot.»
😲 I didn't expect that.
Please, don't follow the myth that working remotely is all we need.12
FML. An overreaching supergenius "architect" and a database team:
A: "We have decided that apps should use mysql. Install a MySQL so we match cloud"
DBA: "we don't have an image or experience with MySQL. We have mssql and Oracle "
A: "ok, use mssql in data center and mysql in production cloud"
DBA: "that's... not going to work well"
A: "just do it!"
Me, reading this shit, sends email: "ignoring the fact that we have more than 500 queries in this application which will need to be checked and most likely rewritten, how are we supposed to test the mysql queries without production access?"
A: "just use mssql local and MySQL in cloud"
M: "... Just to make sure I understand, you want us to write queries for mssql, test them locally, and then write separate queries, with a separate SQL connection abstraction that deploys to production? Again, how are we going to test this?"
A: "no, use same queries, should be fine"
M: "they really won't, they're different dialects"
A: "do the needful, make work!"
If karma were a thing, this person would have long since exploded into a cloud of atomized blood.20
--- GitHub 24-hour outage post mortem ---
As many of you will remember; Github fell over earlier this month and cracked its head on the counter top on the way down. For more or less a full 24 hours the repo-wrangling behemoth had inconsistent data being presented to users, slow response times and failing requests during common user actions such as reporting issues and questioning your career choice in code reviews.
It's been revealed in a post-mortem of the incident (link at the end of the article) that DB replication was the root cause of the chaos after a failing 100G network link was being replaced during routine maintenance. I don't pretend to be a rockstar-ninja-wizard DBA but after speaking with colleagues who went a shade whiter when the term "replication" was used - It's hard to predict where a design decision will bite back and leave you untanging the web of lies and misinformation reported by the databases for weeks if not months after everything's gone a tad sideways.
When the link was yanked out of the east coast DC undergoing maintenance - Github's "Orchestrator" software did exactly what it was meant to do; It hit the "ohshi" button and failed over to another DC that wasn't reporting any issues. The hitch in the master plan was that when connectivity came back up at the east coast DC, Orchestrator was unable to (un)fail-over back to the east coast DC due to each cluster containing data the other didn't have.
At this point it's reasonable to assume that pants were turning funny colours - Monitoring systems across the board started squealing, firing off messages to engineers demanding they rouse from the land of nod and snap back to reality, that was a bit more "on-fire" than usual. A quick call to Orchestrator's API returned a result set that only contained database servers from the west coast - none of the east coast servers had responded.
Come 11pm UTC (about 10 minutes after the initial pant re-colouring) engineers realised they were well and truly backed into a corner, the site was flipped into "Yellow" status and internal mechanisms for deployments were locked out. 5 minutes later an Incident Co-ordinator was dragged from their lair by the status change and almost immediately flipped the site into "Red" status, a move i can only hope was accompanied by all the lights going red and klaxons sounding.
Even more engineers were roused from their slumber to help with the recovery effort, By this point hair was turning grey in real time - The fail-over DB cluster had been processing user data for nearly 40 minutes, every second that passed made the inevitable untangling process exponentially more difficult. Not long after this Github made the call to pause webhooks and Github Pages builds in an attempt to prevent further data loss, causing disruption to those of us using Github as a way of kicking off our deployment processes (myself included, I had to SSH in and run a git pull myself like some kind of savage).
Glossing over several more "And then things were still broken" sections of the post mortem; Clever engineers with their heads screwed on the right way successfully executed what i can only imagine was a large, complex and risky plan to untangle the mess and restore functionality. Github was picked up off the kitchen floor and promptly placed in a comfy chair with a sweet tea to recover. The enormous backlog of webhooks and Pages builds was caught up with and everything was more or less back to normal.
It goes to show that even the best laid plan rarely survives first contact with the enemy, In this case a failing 100G network link somewhere inside an east coast data center.
Link to the post mortem: https://blog.github.com/2018-10-30-...7
I'm convinced code addiction is a real problem and can lead to mental illness.
Dev: "Thanks for helping me with the splunk API. Already spent two weeks and was spinning my wheels."
Me: "I sent you the example over a month ago, I guess you could have used it to save time."
Dev: "I didn't understand it. I tried getting help from NetworkAdmin-Dan, SystemAdmin-Jake, they didn't understand what you sent me either."
Me: "I thought it was pretty simple. Pass it a query, get results back. That's it"
Dev: "The results were not in a standard JSON format. I was so confused."
Me: "Yea, it's sort-of JSON. Splunk streams the result as individual JSON records. You only have to deserialize each record into your object. I sent you the code sample."
Dev: "Your code didn't work. Dan and Jake were confused too. The data I have to process uses a very different result set. I guess I could have used it if you wrote the class more generically and had unit tests."
<oh frack...he's been going behind my back and telling people smack about my code again>
Me: "My code wouldn't have worked for you, because I'm serializing the objects I need and I do have unit tests, but they are only for the internal logic."
Dev:"I don't know, it confused me. Once I figured out the JSON problem and wrote unit tests, I really started to make progress. I used a tuple for this ... functional parameters for that...added a custom event for ... Took me a few weeks, but it's all covered by unit tests."
Me: "Wow. The way you explained the project was; get data from splunk and populate data in SQLServer. With the code I sent you, sounded like a 15 minute project."
Dev: "Oooh nooo...its waaay more complicated than that. I have this very complex splunk query, which I don't understand, and then I have to perform all this parsing, update a database...which I have no idea how it works. Its really...really complicated."
Me: "The splunk query returns what..4 fields...and DBA-Joe provided the upsert stored procedure..sounds like a 15 minute project."
Dev: "Maybe for you...we're all not super geniuses that crank out code. I hope to be at your level some day."
<frack you ... condescending a-hole ...you've got the same seniority here as I do>
Me: "No seriously, the code I sent would have got you 90% done. Write your deserializer for those 4 fields, execute the stored procedure, and call it a day. I don't think the effort justifies the outcome. Isn't the data for a report they'll only run every few months?"
Dev: "Yea, but Mgr-Nick wanted unit tests and I have to follow orders. I tried to explain the situation, but you know how he is."
<fracking liar..Nick doesn't know the difference between a unit test and breathalyzer test. I know exactly what you told Nick>
Dev: "Thanks again for your help. Gotta get back to it. I put a due date of April for this project and time's running out."
APRIL?!! Good Lord he's going to drag this intern-level project for another month!
After he left, I dug around and found the splunk query, the upsert stored proc, and yep, in about 15 minutes I was done.1
This week I started a new job.
My role changed from "Full-stack-web-developer-sysadmin-DBA-helpdesk-strange-person-fixing-stuff-around" to "Back-end Developer".
Moreover, it's a full remote position (so difficult to find in Italy!), so:
1. I can wake up 1.5 hours later;
2. I don't have to waste anymore 2 hours every fucking day driving in traffic to reach the workplace;
3. I can use my fucking bathroom;
4. I can drink hot tea in August without being criticized. 😀
I'm fucking happy!13
Almost a decade ago (damn I'm getting old) I was working on a virtually dead project - i.e. it was the third, and final, iteration of a very failed project in a huge company with literally millions of Euros poured into it.
I came in as a junior SysAdmin/CM to substitute someone who was happily fucking off to another, better job. And alongside me a senior started as well to deal with the management/interaction between vendors and internal "teams" (6 peeps to deal with infrastructure requests from vendors - DBA, SysAdmin, Networking, Hardware commissioning, that sort of stuff).
Now, nothing against this senior - really cool dude - but he was objectively doing a poor management job. Partially due to some personal issues and a little bit of incompetence/disinterest.
So me, and the rest of these peeps that the vendors interacted with, were getting massacred with impossible requests/deadlines and blamed for almost everything wrong with this project.
Two weeks after we started, SeniorMan had to take a week off due to some personal issues so I was asked by higher management to take over his tasks as well until he's back.
The next day, I took all the peeps to get coffee and told them that from now on, regardless of what anyone tells them, unless it's from our direct manager (who pretty much didn't give a shit) every information/decision/deadlines negotiations goes through me and me alone. And they're not allowed to even consider anyone asking them for favours without talking to me first.
Now, this also came with the understanding that I would become the sole person to blame in case of fuck ups, and I gladly accepted because I would have quit anyway if things didn't change.
So everyone agreed.
And oh boy did we put the heat on those fucking vendors, because those two weeks that SeniorMan was busy drinking coffee with the upper management, I studied the entire project - I was familiar with every single network switch, VM, application and most importantly, all the agreements between vendors on which applications they would deliver and when.
They guys from my team also followed through with ignoring vendor requests that didn't come through me. They would literally hang up on them. CCed me on every single email. It was great.
A week later, we learned that SeniorMan would be gone another week, so upper management gave me a company phone because things were going well. SeniorMan would later quit because he pretty much wasn't needed by the time he came back.
With the phone and the blessings of the management I was pretty much the only entry point for vendors to deliver their applications. And I was strict as fuck. My negotiation policy was to multiply everything the peeps from my team told me by 3. If they didn't like it, well tough luck, give us more resources and wait 3 weeks to get them up to speed.
And within a month we became the highest performing team on that project. Outperforming the second best by as much as 6 to 1. All this with much less stress for everyone in the team.
Thanks to these changes, vendors actually had to do their job and deliver the promised services without blaming us. So we magically passed the first milestone, and suddenly went from failed project to the most promising project in the company.
Ah good times, good times...
I participated in a meeting, before hitting the second milestone, where the project lead from one of the vendors was trying to pin their incompetence on us being slow to deliver infrastructure requirements.
So I pulled out the Visio file that everyone in the room agreed to a few months prior, and one by one showed that not only everything was delivered on time but even earlier on some items. And that due to them being slow on testing their shitty software on our infrastructure everything is going to shit.
The CTO of the company was present in this meeting, and what followed can only be described as someone who had no idea what was going on, being hit in the face with some else's balls of steel. This CTO proceeded to yell out how much all the vendor contracts cost and that if these fucks didn't get their shit together he'll release the lawyers from the dungeon of debt.20
Just resigned from my company.
I’m “sadpy”, or “had”, half happy half sad.
They gave me a lot, I gave them a lot.
I feel sorry because now they won’t have a jolly anymore.
I used to do many kind of activities, from DBA to HW installation, from penetration tests to security analysis, from scripting to programming.
A full-stack all-do.
The new company wants me for my skills in electronics as well, I’ll be their all-do for IoT 😍
I’m quite happy for my new role, wish me luck 😊9
Our web department was deploying a fairly large sales campaign (equivalent to a ‘Black Friday’ for us), and the day before, at 4:00PM, one of the devs emails us and asks “Hey, just a heads up, the main sales page takes almost 30 seconds to load. Any chance you could find out why? Thanks!”
We click the URL they sent, and sure enough, 30 seconds on the dot.
Our department manager almost fell out of his chair (a few ‘F’ bombs were thrown).
DBAs sit next door, so he shouts…
Mgr: ”Hey, did you know the new sales page is taking 30 seconds to open!?”
DBA: “Yea, but it’s not the database. Are you just now hearing about this? They have had performance problems for over week now. Our traces show it’s something on their end.”
Mgr: “-bleep- no!”
Mgr tries to get a hold of anyone …no one is answering the phone..so he leaves to find someone…anyone with authority.
4:15 he comes back..
Mgr: “-beep- All the web managers were in a meeting. I had to interrupt and ask if they knew about the performance problem.”
Me: “Oh crap. I assume they didn’t know or they wouldn’t be in a meeting.”
Mgr: “-bleep- no! No one knew. Apparently the only ones who knew were the 3 developers and the DBA!”
Me: “Uh…what exactly do they want us to do?”
Mgr: “The –bleep- if I know!”
Me: “Are there any load tests we could use for the staging servers? Maybe it’s only the developer servers.”
DBA: “No, just those 3 developers testing. They could reproduce the slowness on staging, so no need for the load tests.”
Mgr: “Oh my –bleep-ing God!”
4:30 ..one of the vice presidents comes into our area…
VP: “So, do we know what the problem is? John tells me you guys are fixing the problem.”
Mgr: “No, we just heard about the problem half hour ago. DBAs said the database side is fine and the traces look like the bottleneck is on web side of things.”
VP: “Hmm, no, John said the problem is the caching. Aren’t you responsible for that?”
Mgr: “Uh…um…yea, but I don’t think anyone knows what the problem is yet.”
VP: “Well, get the caching problem fixed as soon as possible. Our sales numbers this year hinge on the deployment tomorrow.”
- VP leaves -
Mgr: “I was thinking the same thing. I’ll have networking run a trace.”
4:45 Networking run their trace, and sure enough, there was some relative path of ‘something’ pointing to a local resource not on development, it was waiting/timing out after 30 seconds. Fixed the path and page loaded instantaneously. Network admin walks over..
NetworkAdmin: “We had no idea they were having problems. If they told us last week, we could have identified the issue. Did anyone else think 30 second load time was a bit suspicious?”
4:50 VP walks in (“John” is the web team manager)..
VP: “John said the caching issue is fixed. Great job everyone.”
VP: “But the caching is fixed? Right? John said it was caching. Anyway, great job everyone. We’re going to have a great day tomorrow!”
NetworkAdmin: “Ouch…you feel that?”
Me: “Feel what?”
NetworkAdmin: “That bus John just threw us under.”
Mgr: “Yea, but I think John just saved 3 jobs. Remember that.”4
Most satisfying bug I've fixed?
Fixed a n+1 issue with a web service retrieving price information. I initially wrote the service, but it was taken over by a couple of 'world class' monday-morning-quarterbacks.
The "Worst code I've ever seen" ... "I can't believe this crap compiles" types that never met anyone else's code that was any good.
After a few months (yes months) and heavy refactoring, the service still returned price information for a product. Pass the service a list of product numbers, service returns the price, availability, etc, that was it.
After a very proud and boisterous deployment, over the next couple of days the service seemed to get slower and slower. DBAs started to complain that the service was causing unusually high wait times, locks, and CPU spikes causing problems for other applications. The usual finger pointing began which ended up with "If PaperTrail had written the service 'correctly' the first time, we wouldn't be in this mess."
Only mattered that I initially wrote the service and no one seemed to care about the two geniuses that took months changing the code.
The dev manager was able to justify a complete re-write of the service using 'proper development methodologies' including budgeting devs, DBAs, server resources, etc..etc. with a projected year+ completion date.
My 'BS Meter' goes off, so I open up the code, maybe 5 minutes...tada...found it. The corresponding stored procedure accepts a list of product numbers and a price type (1=Retail, 2=Dealer, and so on). If you pass 0, the stored procedure returns all the prices.
Code basically looked like this..
public List<Prices> GetPrices(List<Product> products, int priceTypeId)
foreach (var item in products)
List<int> productIdsParameter = new List<int>();
List<Price> prices = dataProvider.GetPrices(productIdsParameter, 0);
foreach (var price in prices)
if (price.PriceTypeID == priceTypeId)
prices = dataProvider.GetPrices(productIdsParameter, price.PriceTypeID);
* Omitting the other 'WTF?' code to handle the zero price type
I removed the double stored procedure call, updated the method signature to only accept the list of product numbers (which it was before the 'major refactor'), deployed the service to dev (the issue was reproducible in our dev environment) and had the DBA monitor.
The two devs and the manager are grumbling and mocking the changes (they never looked, they assumed I wrote some threading monstrosity) then the DBA walks up..
DBA: "We're good. You hit the database pretty hard and the CPU never moved. Execution plans, locks, all good to go."
<dba starts to walk away>
DevMgr: "No fucking way! Putting that code in a thread wouldn't have fix it"
Me: "Um, I didn't use threads"
Dev1: "You had to. There was no way you made that code run faster without threads"
Dev2: "It runs fine in dev, but there is no way that level of threading will work in production with thousands of requests. I've got unit tests that prove our design is perfect."
Me: "I looked at what the code was doing and removed what it shouldn't be doing. That's it."
DBA: "If the database is happy with the changes, I'm happy. Good job. Get that service deployed tomorrow and lets move on"
Me: "You'll remove the recommendation for a complete re-write of the service?"
DevMgr: "Hell no! The re-write moves forward. This, whatever you did, changes nothing."
DBA: "Hell yes it does!! I've got too much on my plate already to play babysitter with you assholes. I'm done and no one on my team will waste any more time on this. Am I clear?"
Seeing the dev manager face turn red and the other two devs look completely dumbfounded was the most satisfying bug I've fixed.5
The solution for this one isn't nearly as amusing as the journey.
I was working for one of the largest retailers in NA as an architect. Said retailer had over a thousand big box stores, IT maintenance budget of $200M/year. The kind of place that just reeks of waste and mismanagement at every level.
They had installed a system to distribute training and instructional videos to every store, as well as recorded daily broadcasts to all store employees as a way of reducing management time spend with employees in the morning. This system had cost a cool 400M USD, not including labor and upgrades for round 1. Round 2 was another 100M to add a storage buffer to each store because they'd failed to account for the fact that their internet connections at the store and the outbound pipe from the DC wasn't capable of running the public facing e-commerce and streaming all the video data to every store in realtime. Typical massive enterprise clusterfuck.
Then security gets involved. Each device at stores had a different address on a private megawan. The stores didn't generally phone home, home phoned them as an access control measure; stores calling the DC was verboten. This presented an obvious problem for the video system because it needed to pull updates.
The brilliant Infosys resources had a bright idea to solve this problem:
- Treat each device IP as an access key for that device (avg 15 per store per store).
- Verify the request ip, then issue a redirect with ANOTHER ip unique to that device that the firewall would ingress only to the video subnet
- Do it all with the F5
A few months later, the networking team comes back and announces that after months of work and 10s of people years they can't implement the solution because iRules have a size limit and they would need more than 60,000 lines or 15,000 rules to implement it. Sad trombones all around.
Then, a wild DBA appears, steps up to the plate and says he can solve the problem with the power of ORACLE! Few months later he comes back with some absolutely batshit solution that stored the individual octets of an IPV4, multiple nested queries to the same table to emulate subnet masking through some temp table spanning voodoo. Time to complete: 2-4 minutes per request. He too eventually gives up the fight, sort of, in that backhanded way DBAs tend to do everything. I wish I would have paid more attention to that abortion because the rationale and its mechanics were just staggeringly rube goldberg and should have been documented for posterity.
So I catch wind of this sitting in a CAB meeting. I hear them talking about how there's "no way to solve this problem, it's too complex, we're going to need a lot more databases to handle this." I tune in and gather all it really needs to do, since the ingress firewall is handling the origin IP checks, is convert the request IP to video ingress IP, 302 and call it a day.
While they're all grandstanding and pontificating, I fire up visual studio and:
- write a method that encodes the incoming request IP into a single uint32
- write an http module that keeps an in-memory dictionary of uint32,string for the request, response, converts the request ip and 302s the call with blackhole support
- convert all the mappings in the spreadsheet attached to the meetings into a csv, dump to disk
- write a wpf application to allow for easily managing the IP database in the short term
- deploy the solution one of our stage boxes
- add a TODO to eventually move this to a database
All this took about 5 minutes. I interrupt their conversation to ask them to retarget their test to the port I exposed on the stage box. Then watch them stare in stunned silence as the crow grows cold.
According to a friend who still works there, that code is still running in production on a single node to this day. And still running on the same static file database.
Unaware that this had been occurring for while, DBA manager walks into our cube area:
DBAMgr-Scott: "DBA-Kelly told me you still having problems connecting to the new staging servers?"
Dev-Carl: "Yea, still getting access denied. Same problem we've been having for a couple of weeks"
DBAMgr-Scott: "Damn it, I hate you. I got to have Kelly working with data warehouse project. I guess I've got to start working on fixing this problem."
Dev-Carl: "Ha ha..sorry. I've checked everything. Its definitely something on the sql server side."
DBAMgr-Scott: "I guess my day is shot. I've got to talk to the network admin, when I get back, lets put our heads together and figure this out."
Me: "A permissions issue on staging? All my stuff is working fine and been working fine for a long while."
Dev-Carl: "Yea, there is nothing different about any of the other environments."
Me: "That doesn't sound right. What's the error?"
Me: "No, the actual exception, never mind, I'll look it up in Splunk."
<in about 30 seconds, I find the actual exception, Win32Exception: Access is denied in OpenSqlFileStream, a little google-fu and .. >
Me: "Is the service using Windows authentication or SQL authentication?"
Dev-Carl: "SQL authentication."
Me: "Switch it to windows authentication"
<Dev-Carl changes authentication...service works like a charm>
Dev-Carl: "OMG, it worked! We've been working on this problem for almost two weeks and it only took you 30 seconds."
Me: "Now that it works, and the service had been working, what changed?"
Dev-Carl: "Oh..look at that, Dev-Jake changed the connection string two weeks ago. Weird. Thanks for your help."
<My brain is screaming "YOU NEVER THOUGHT TO LOOK FOR WHAT CHANGED!!!"
Me: "I'm happy I could help."4
Every year my team runs an award ceremony during which people win “awards” for mistakes throughout the year. This years was quite good.
The integration partner award- one of our sysAdmins was talking with a partner from another company over Skype and was having some issues with azure. He intended to send me a small rant but instead sent “fucking azure can go fuck itself, won’t let me update to managed disks from a vhd built on unmanaged” to our jv partner.
Sysadmin wannabe award (mine)- ran “Sudo chmod -R 700 /“ on one of our dev systems then had to spend the next day trying to fix it 😓
The ain’t no sanity clause award - someone ran a massive update query on a prod database without a where clause
The dba wannabe award - one of our support guys was clearing out a prod dB server to make some disk space and accidentally deleted one of the databases devices bringing it down.
The open source community award - one of the devs had been messing about with an apache proxy on a prod web server and it ended up as part of a botnet
There were others but I can’t remember them all5
The website for our biggest client went down and the server went haywire. Though for this client we don’t provide any infrastructure, so we called their it partner to start figuring this out.
They started blaming us, asking is if we had upgraded the website or changed any PHP settings, which all were a firm no from us. So they told us they had competent people working on the matter.
TL;DR their people isn’t competent and I ended up fixing the issue.
Hours go by, nothing happens, client calls us and we call the it partner, nothing, they don’t understand anything. Told us they can’t find any logs etc.
So we setup a conference call with our CXO, me, another dev and a few people from the it partner.
At this point I’m just asking them if they’ve looked at this and this, no good answer, I fetch a long ethernet cable from my desk, pull it to the CXO’s office and hook up my laptop to start looking into things myself.
IT partner still can’t find anything wrong. I tail the httpd error log and see thousands upon thousands of warning messages about mysql being loaded twice, but that’s not the issue here.
Check top and see there’s 257 instances of httpd, whereas 256 is spawned by httpd, mysql is using 600% cpu and whenever I try to connect to mysql through cli it throws me a too many connections error.
I heard the IT partner talking about a ddos attack, so I asked them to pull it off the public network and only give us access through our vpn. They do that, reboot server, same problems.
Finally we get the it partner to rollback the vm to earlier last night. Everything works great, 30 min later, it crashes again. At this point I’m getting tired and frustrated, this isn’t my job, I thought they had competent people working on this.
I noticed that the db had a few corrupted tables, and ask the it partner to get a dba to look at it. No prevail.
5’o’clock is here, we decide to give the vm rollback another try, but first we go home, get some dinner and resume at 6pm. I had told them I wanted to be in on this call, and said let me try this time.
They spend ages doing the rollback, and then for some reason they have to reconfigure the network and shit. Once it booted, I told their tech to stop mysqld and httpd immediately and prevent it from start at boot.
I can now look at the logs that is leading to this issue. I noticed our debug flag was on and had generated a 30gb log file. Tail it and see it’s what I’d expect, warmings and warnings, And all other logs for mysql and apache is huge, so the drive is full. Just gotta delete it.
I quietly start apache and mysql, see the website is working fine, shut it down and just take a copy of the var/lib/mysql directory and etc directory just go have backups.
Starting to connect a few dots, but I wasn’t exactly sure if it was right. Had the full drive caused mysql to corrupt itself? Only one way to find out. Start apache and mysql back up, and just wait and see. Meanwhile I fixed that mysql being loaded twice. Some genius had put load mysql.so at the top and bottom of php ini.
While waiting on the server to crash again, I’m talking to the it support guy, who told me they haven’t updated anything on the server except security patches now and then, and they didn’t have anyone familiar with this setup. No shit, it’s running php 5.3 -.-
Website up and running 1.5 later, mission accomplished.6
I was asleep and woke up and just had the thought: fuck Oracle and fuck Oracle DB. Going back to bed now 😴😴😴6
*me trying to add one more field to a SQL request*
*asks coworker* Hey, do you know how I'd get this field? You have to go through 4 tables just to find it because of the way the tables are related.
*coworker looks for a bit* Oh. That's crazy. We should ask the DBA.
*we ask our 60 year old DBA for some help.*
He walks us through a minefield of inner and outer joins and selects and we finally arrive at a solution. Kudos to our DBA, who in the eyes of us young devs is a wizard.3
Every year my team have an award ceremony for stupid things we do throughout the year. There is only really one worth highlighting this year
Alzheimer’s award- given to one of our DBAs for this svn commit message: “updated the comment block but forgot to make the code change”2
Yesterday I spent some time on the meta site for dba.stackexchange.com and found this one guy with 1 rep raging about how his questions aren't getting answered and how is answers are the best etc...
"I have 17 years of experience as a dba, blah, blah, blah, my answers are correct, blah, blah"
He got pretty destroyed by the mods and other users about how shit his answers were and how they weren't factually correct etc...
This just continues to show that no matter how much experience you have you won't always be right.
Same goes for my senior at work, he has 10 years more experience than me (I have 2) and he still asks for my point of view and help without being a dick about it.
I hope we'll all keep being nice people unlike that Stackexchange guy...2
TLDR - you shouldn't expect common sense from idiots who have access to databases.
I joined a startup recently. I know startups are not known for their stable architecture, but this was next level stuff.
There is one prod mongodb server.
The db has 300 collections.
200 of those 300 collections are backups/test collections.
25 collections are used to store LOGS!! They decided to store millions of logs in a nosql db because setting up a mysql server requires effort, why do that when you've already set up mongodb. Lol 😂
Each field is indexed separately in the log.
1 collection is of 2 tb and has more than 1 billion records.
Out of the 1 billion records, 1 million records are required, the rest are obsolete. Each field has an index. Apparently the asshole DBA never knew there's something called capped collection or partial indexes.
Trying to get approval to clean up the db since 3 months, but fucking bureaucracy. Extremely high server costs plus every week the db goes down since some idiot runs a query on this mammoth collection. There's one single set of credentials for everything. Everyone from applications to interns use the same creds.
And the asshole DBA left, making me in charge of handling this shit now. I am trying to fix this but am stuck to get approval from business management. Devs like these make me feel sad that they have zero respect for their work and inability to listen to people trying to improve the system.
Going to leave this place really soon. No point in working somewhere where you are expected to show up for 8 hours, irrespective of whether you even switch on your laptop.
Wish me luck folks.4
So here is my week 72 as a reviewer. But first, let me ask y'all. Am I weird to think that you should finish coding the thing and testing the thing before kicking it out to review? Cuz that's how I do it. And that is the process at my work place.
So anyway, I was doing this review. And it was very wrong. Like really, really wrong. We give a thorough intro to our product (perhaps too thorough) so this guy should have known every test case he had to cover. Or at least, if he was unsure, asked. It was all documented.
Anyway, he kicks out this peer review. First thing I notice, it is not following the standard. Fair enough, we didn't give him the coding standard. BUT HE DIDN'T EVEN MAINTAIN THE FORMAT OF THE ORIGINAL FILE. HE JUST DID HIS OWN THING!!! So I email him the coding standard and make a comment in the review. He denies the finding. No reason. Just turns it down. Strike 1.
Then, I'm going through and he didn't even cover all of the core cases. I found several core cases that he missed. And every edge case. Make a not of it. He fixes only the couple of examples I gave him. Strike 2.
Guy decided to redesign a major chunk of our interfaces. Our interfaces are not just used by us (hence interfaces). We designed them the way they were for a reason. It was not a fun design process. Myself, the architect, one of our customers, and the guy that did the implementation all told him to roll back his change. Especially since it wasn't in the scope of what he was doing. He wouldn't. Strike 3.
I go to the lead and bring him in. He has a talk with him. All of the sudden he is putting out multiple builds per finding. Like most times I will put out like 2 to 4 for the whole peer review. No he kicks out minimum one per finding and chokes the review queue. Strike 4.
Strike 5: he tells me, a former DBA, that I didn't know what I was talking about when I told him to move something into a new table, even after I told him that "while in database terms it doesn't make sense, this is for product robustness".
Strike 6: he was just a condescending asshole. Bragging about how he did this job and that job over his career. His longest position held was about 18 months. Bragged about working at my company and being some hotshot at the company: only worked here for 8 months and that was 5 years ago.
You know. I have never really wanted to fight someone after about undergrad. But he came close.7
Got in trouble today during a Data integrity meeting, everyone kept talking about data massaging. "Massage this data", "massage that data table", "insert massaged data".
Finally I just blurted out, "yeah massage it all you want but how do I get a data happy ending?"
I thought it was hilarious. The other DBA and backend devs thought it was jokes, my manager... Not so much.
Apparently, I need to keep "thoughts and comments about data happy endings to myself moving forward".
know what pisses me the fuck off? when the manager of another department jumps over me and goes straight to the head of my department for a request that they want from MY department.
Currently, there are 2 stupid bitches that insist on doing this fuckery. One of them keeps getting owned by our DBA since for whatever reason she sends her requests to me, just for the DBA to remind her that I ain't giving her access to shit and bla bla
The other is the head of the human resources department. It goes like this: sends wrong data, task gets delayed cuz we have to sort her shit, gets impatient, bitches at head of department and his boss about us taking long(bitch 3 hours ain't long and your shit ain't critical) just for me to reply back with images and LOOK FUCKTARD YOU MESS THIS UP red arrows showing how what she did was wrong and I had to fix it for her.
Sends a reply back only to me saying thanks, ah no pendeja, I will forward aaaaaaall of that shit to everyone else, tried throwing me under the bus? well now ima do it to you.
And fuck those 3 applications you requested, have fun adding shit manually through spreadsheets and then go eat shit and die.5
I think the worst work culture you can experience is nepotism and corruption in hierarchy. What do I mean? Well, this happened (and I think is still happening) in my last job. It was a huge logistic/delivery company. I was an intern, working as assistant developer of the only developer of the site. There was also a guy that was the technician, his assistant, a DBA and that's it.
Well, my partner and I were working on a system that managed almost all the operations of the company in this city.
Well, I supplied the dev two weeks when he was on vacation. I knew almost all the system. what happened? the manager from other city came with another Dev, and I'm not saying that I was an expert or something like that, but that dev from the other city was an incompetent. He couldn't even make a small GUI change without messing it all...
Guess what? The company paid him weekly round tickets to come and go from his city to ours (two hours of flight).
I was too disappointed I started searching another job. A week after getting my degree, I left my job and started in the one I am now. Before leaving, I asked my boss if there was a realistic chance to grow up. He answered no. To be honest, that didn't surprised me :/
The thing that makes me angry about this is that a lot of companies give chances to people that come from other cities, even if they don't know anything >:v
Oh, I almost forgot it: The last five months I was working there, they quit our office and send us to trailer-offices :/1
Me doing front-end work and get a stupid feature request from a customer...."I'll have to talk to our back-end developer and DBA to see if those changes are possible".
Guess what...Im the back-end developer and DBA and the answer is "hell no".2
Prod was down for 15 minutes due to failed DB connection attempts. Contacted DBA after the fact for an explanation. Response: We don't know, but it was probably temporary.
Dev: I think I just deleted data I shouldn't have. Can you do a rollback?
Me: No, I'm a DBA, I don't touch data. Did you commit?
Dev: Yes, I committed.
Me: Your data is gone.
Dev: But don't you guys do backups?
Me: We backup the Prod servers. We only back up Dev on request. Did you request a backup in Dev?
Dev: The Testers are going to be pissed.2
Never mess with a motivated developer. I will make your life difficult in return.
Me: we need server logs and stats daily for analysis
DBA: to get those, you need to open a ticket
Me: can't you just give me SFTP access and permissions to query the stats from the DB?
*Writes an Excel Template file that I basically just need to copy and paste from to create a ticket*
This process should not take me more than 2mins 👍😁😋🙂😙😙😙😙😙😙😙😙
For them.... 😈😈😈😈😈😈😈😈😈😈😈10
This is getting annoying.
For the past >half a year I've been chasing windmills. This is what my BAU day looks like:
- We login to client's network
- We start running some Sanity tests before the actual runs (actual runs are hell of an expensive (financially and time-wise) thing to launch) to make sure environment is OK.
- Sanity tests fail. wtf? Nothing's been changed since y-day!
- Spend ~3-4 hours digging logs, code, more logs,... Apparently some genius decided to change a single parameter.
- Spend another 1-2 hours trying to work around that parameter (since apparently that genius did have a task to do that, so we'll most likely have to find a way to live with it)
- Restart the whole env (~30min).
- Launch a Smoke, Sanity tests to verify env state.
- Launch the actual test
- Go home.
- We login to client's network
- We start running some Sanity tests before the actual runs to make sure environment is OK.
- Sanity tests pass.
- Run the actual test
- Concurrency on RDS database is sky-rocketing! WTF did that come from??? Nothing's been changed since y-day!!
- Spend ~1-2 hours looking for anything changed, dig some logs for anything unusual. Nothing.
- Escalate to DBA. 2 hours later DBA says "fix the app". thanks for nothing mate....
- Spend remaining 2 hours analysing AWR. Give up, restart the whole RDS instance. Another hour wasted.
- Time to go home. Out of curiosity run Sanity test -- all good. Run the actual test -- all good. wtf??
- Go home
- We login to client's network
- We start running some Sanity tests before the actual runs to make sure environment is OK.
- Sanity tests fail. wtf? Nothing's been changed since y-day!
- Spend ~3-4 hours digging logs, code, more logs,... Apparently some genius decided to change a single parameter.
- Spend another 1-2 hours trying to work around that parameter
- ..... I think you know where this is going.
And this keeps going on and on, day by day. Spending the better half of the day chasing windmills and doing our actual work on the last hour of the working day or even after that.
We have plenty of interesting tasks in our Jira but we're squirels spinning in the wheel and never being able to touch them.
It feels like I'm wasting my time. I could do so much more with my time!
[just needed to vent ]1
I find a poor tester copy/pasting data from the test environment to the live one, as he accidentally broke it. I ask the DBA, " why isn't syncing SQL records part of the deployment pipeline?"
"You're front end. This is my job. Go do your job."
"... but it's an easy query, and you're exposing us to human error."
"You need to go sit down."2
Nooooooooo! My wife told one of her friends that I'm a DBA. The response. "Oh. That's good to know. I've been having this problem with my cell phone." No! Why? Aughhh!
I just had the most surreal email discussion I think I've ever had...
I spent over two hours going back-and-forth over email with an enterprise DBA, trying to convince them I needed a primary key for a table. They created the table without a primary key (or any unique constraints... or indexes... but that's another discussion). I asked them to add one. Then had to justify why.
If you ever find yourself justifying why you need a primary key on a table in an RDBMS, that's the day you find yourself asking "is this real life?"
I want the last two hours of my life back. And a handful of Advil.1
On chat today.
Dude: can you run a script for me? We don't have permission.
Me: what kind of script? Who wrote it?
Dude: posts screenshot of DML select/update statement he tried to run.
Me: I'm a DBA. We don't run DML for people.
Dude: Oh. Can you give me the password?
Me: examine script and notice he tried to run it on QA DB.
Me: No. We don't memorize passwords, and this is QA; you need to check the password out of the safe. You also need a change ticket to DevOps, and they will run it for you.
At that point I ended the discussion, because running anything in QA or Prod without a change ticket gets you fired. And I like my job. Really annoyed.3
Top 10 database engines on db-engine... I never knew the top 3 were so far ahead of the game, if that site is anything to go by.15
In order to reduce support costs, manager instructed his team to remove all logging/reporting of errors in the company’s CRM application.
Team’s support tickets went down 80%, manager received an award for his efforts, but mysteriously, DBA/support workload increased, bad/missing data,
increased support tickets in other areas of the business (shipping, etc. that relied on correct data from the CRM) and other side-affectual behavior.
Even after pointing this out this correlation, showing before/after code, no one believed the two were related and I was accused of not being a ‘team player’.
“You and the other teams need to learn from his example!”. As ‘punishment’ was I was moved to the team managing the CRM application.1
DBA bitches about one of our developers SQL... implement changes... deploy... now it takes 12 minutes... up from 40 seconds... good job!7
At work, my closest relation is with the DBA. Dude is a genius when it comes to proper database management as well as having a very high level of understanding concerning server administration, how he got that good at that I have no clue, he just says that he likes to fuck around with servers, Linux in particular although he also knows a lot about Windows servers.
Thing is, the dude used to work as a dev way back when VB pre VB.NET was all the rage and has been generating different small tools for his team of analysts(I used to be a part of his team) to use with only him maintaining them. He mentioned how he did not like how Microsoft just said fk u to VB6 developers, but that he was happy as long as he could use VB. He relearned how to do most of the GUI stuff he was used to do with VB6 into VB.NEt and all was good with the world. I have seen his code, proper OOP practices and architectural decisions, etc etc. Nothing to complain about his code, seems easy enough to extend, properly documented as well.
Then he got with me in order to figure out how to breach the gap between building GUI applications into web form, so that we could just host those apps in one of our servers and his users go from there, boy was he not prepared to see the amount of fuckery that we do in the web development world. Last time my dude touched web development there was still Classic ASP with JScript and VBScript(we actually had the same employer at one point in the past in which I had to deal with said technology, not bad, but definitely not something I recommend for the current state of web development) and decided that the closest thing to what he was used was either PHP(which he did not enjoy, no problem with that really, he just didn't click with the language) and WebForms using VB.NET, which he also did not like on account of them basically being on support mode since Microsoft is really pushing for people to adopt dotnet core.
After came ASP.NET with MVC, now, he did like it, but still had that lil bug in his head that told him that sticking to core was probably a better idea since he was just starting, why not start with the newest and greatest? Then in hit(both of us actually) that to this day Microsoft still not has command line templates for building web applications in .net core using VB.NET. I thought it was weird, so I decided to look into. Turns out, that without using Razor, you can actually build Web APIs with VB.NET just fine if you just convert a C# template into VB.NET, the process was...err....tricky, and not something we would want to do for other projects, with that in we decided to look into Microsoft's reasons to not have VB.NET. We discovered how Microsoft is not keeping the same language features between both languages, having crown C# as the language of choice for everything Microsoft, to this point, it seems that Microsoft was much more focused in developing features for the excellent F# way more than it ever had for VB.NET at this point and that it was not a major strategy for them to adapt most of the .net core functionality inside of VB, we found articles when the very same Microsoft team stated of how they will be slowly adding the required support for VB and that on version 5 we would definitely have proper support for VB.NET ALTHOUGH they will not be adding any new development into the language.
Past experience with Microsoft seems to point at them getting more and more ready to completely drop the language, it does not matter how many people use it, they would still kill it :P I personally would rather keep it, or open source the language's features so that people can keep adding support to it(if they can of course) because of its historical significance rather than them just completely dropping the language. I prefer using C#, and most of my .net core applications use C#, its very similar to Java on a lot of things(although very much different in others) and I am fine with it being the main language. I just think that it sucks to leave such a large developer pool in the shadows with their preferred tool of choice and force them to use something else just like that.
My boy is currently looking at how I developed a sample api with validation, user management, mediatR and a custom project structure as well as a client side application using React and typescript swappable with another one built using Angular(i wanted to test the differences to see which one I prefer, React with Typescript is beautiful, would not want to use it without it) and he is hating every minute of it on account of how complex frontend development has become :V
Just wanted to vent a little about a non bothersome situation.8
Still dealing with the web department and their finger pointing after several thousand errors logged.
SeniorWebDev: “Looks like there were 250 database timeout errors at 11:02AM. DBAs might want to take a look.”
I look at the actual exceptions being logged (bulk of the over 1,600 logged errors)..
“Object reference not set to an instance of an object.”
Then I looked the email timestamp…11:00AM. We received the email notification *before* the database timeout errors occurred.
I gather some facts…when the exceptions started, when they ended, and used the stack trace to find the code not checking for null (maybe 10 minutes of junior dev detective work). Send the data to the ‘powers that be’ and carried on with my daily tasks.
I attached what I found (not the actual code, it was changed to protect the innocent)
Couple of hours later another WebDev replied…
WebDev: “These errors look like a database connectivity issue between the web site and the saleitem data service. Appears the logging framework doesn’t allow us to log any information about the database connection.”
FRACK!!...that Fracking lying piece of frack! Our team is responsible for the logging framework. I was typing up my response (having to calm down) then about a minute later the head DBA replies …
DBA: “Do you have any evidence of this? Our logs show no connectivity issues. The logging framework does have the ability to log an extensive amount of data regarding the database transaction. Database name, server, login, command text, and parameter values. Everything we need to troubleshoot. This is the link to the documentation …. If you implement the one line of code to gather the data, it will go a long way in helping us debug performance and connectivity issue. Thank you.”
DBA sends me a skype message “You’re welcome :)”
Ahh..nice to see someone else fed up with their lying bull...stuff.
SQL Rule 1. Always assume there are external processes that might affect your data. (for instance, triggers).
SQL Rule 2. In Denormalised data, never execute logic on dependant table values, always copy from the parent.
SQL Rule 3. When Denormalised data schemas are created the DBA knows what they are doing.
SQL Rule 3.1. If DBA knows what they is doing then according to Rule 1 there is no problem with adding in some triggers to maintain data clones as they are created.
SQL Rule 4. If you don't like or agree with triggers, deal with it. They are a first class tool in a first class RDBMS. In a multi-app or service environment there may be many other external processes massaging your data
SQL Rule 5. If all previous rules are not broken and the system has been running efficiently for many years DO NOT complain that there are triggers in the database that are doing and have been doing the same process that you just butchered (by violating Rule 1 and 2) in your makeshift "hello world, look what I can do from my phone" angular BS when the rest of the users are still relying on the existing runtime app.
SQL Rule 6. If you turn my triggers off, you sure as hell better turn them back on!1
The team headed to Arby's for lunch today. Our DBA walks up to the counter.
arby's guy: "Hey, what can I get you."
dba: "Hello. Yes, I'd like the largest french dip you are legally allowed to serve me."
Absolutely slayed everyone in-line and at the counter.2
Today was a manic-depressive kind of day. Spent the morning helping some developers with getting their code to run a stored procedure to drop old partitions, but it wasn't working on their end. It was a fairly simple proc. But working with partitions is a little like working with an array. I figured out that they were passing the wrong timestamp, and needed to add +1 to delete the right partition. Got that sorted out, and things were good. Lunch time.
After lunch I did some busy work, and then the PO comes up at about 2PM and says he's assigned some requests to me. The first was just attaching some scripts. Easy. The second, the user wants a couple of schemas exported ... at 6PM. I've been in the office since 6:45AM.
While I'm setting up some commands to run for the data export, a BA walks up and asks if I'm filling in for another DBA who is out for a few weeks. Yep. There's a change request that hasn't been assigned, and he normally does the work. I ask when it's due. Well, the pre-implementation was supposed to be done in the morning, but it wasn't, and we're in the implementation window ... half way through. I bring up the change task, and look at. Create new schema and users. That's all it says. The BA laughs. I tell I need more to go on. 10 minutes later he sends an email with the information. There's only two hours left in the window, and I can only use half of it, because the production guys have to their stuff, and we're in their window. Now I'm irritated, because I'm new to Oracle, and it's an unforgiving mistress. Fortunately, another DBA says he'll do it, so that we can get it done in time. But can't work it either, because Dev DBAs don't have access to QA, and the process required access for this task. Gets shelved until the access issue is resolved. It's now after 4:15PM. I'm going to in traffic with that 6PM deadline.
I manage to get home and to the computer by 5:45PM. Log in. Start VPN. Box pops on screen. Java needs to update. I chose skip update. Box pops up again. It won't let me log in until Java is current. Passed.
I finally get logged in, and it's 6:10PM. I'm late getting the job started. I pull up Putty and log into the first box, and paste my pre-prepared command in the command line and hit error. Command not found. I'm tired, so it's a moment to sink in. I don't have time for this.
I log into DBArtisan and pull up the first data base, use the wizard to set the job, and off it goes. Yay. Bring up the second database, and have enter the connect info. Host not found. Wut? Examine host name. Yep, it's correct. Try a different method. Host not found. Go back to Putty. Log in. Past string. Launch. Command not found. Now my brain is quitting on me. Why now? It's after 6:30PM. Fiddle with some settings, reset $Oracle home. Try again. Yay. It works. I'm done. It's after 7PM.
There is nothing like technology to snatch the euphoria of a success away from you. It's a love-hate thing, but I wouldn't trade it for anything else. I'm done. Good night.3
Hey, DBA guy! Security blocked this one port for the database. Can you change the database to a different port?
Me: No, I actually like working here.2
I JUST HAD ONE OF THOSE DAYS THAT MAKES ONE WANT TO BANG TWO BRICKS ON HEAD SND END THE PAIN THE STORY STARTS YESTETDAY WITH ISSUES AFTER A MIGRSTION AND THEY ASK ME TO HELP TROUBLESHOOT EVEN THOUGH I'M A DEV DBA AND THE ISSUE IS IN QA/SAT AND I HELP ANYWAY AND THEY CAN'T FIND A VIEW AND SO I LOOK EVERYWHERE AND CAN'T DOING IT EITHER AND IT DIDN'T EXIST IN PROD OR DEV SO I TELL THEM IT'S NOT THERE, AND THEY ARE LIKE, CAN YOU RETRIEVE IT FOR US AND I'M LIKE FROM WHERE? I DON'T KEEP VIEWS IN MY BUTT AND YOU GUYS ARE SMOKING CRACK AND THE GIVE ME THEIR QUERY WHICH CONTAIN THE VIEE ANYWAY AND THEY SAY CAN YOU RUN IT AND IT RUNS AND WORKS AND THEY CAN'T MAKE IT WORK AND IT WORKS BECAUSE IT DOESN'T CALL THE VIEW THEY HAVE ME SO NO PROBLEM THERE SO I FINALLY ASK THEM ARE YOU POINTING TO THE CORRECT DATABASE AND THEY'RE LIKE OH MAN WE TOLD YOU THE WRONG DATABASE AND SO I LOOK AT THE RIGHT DATABASE AND FIND THAT THE GRANTS ARE MISSING AND YEAH THANK YOU FOR TAKING EIGHT HOURS OF MY LIFE BECAUSE WE WERE IN THE WRONG DB YOU GAVE ME AND I HOPE THE FLAG OF A THOUSAND CAMELS INVEST YOUR ARMPITS AND THE CHIGGERS OF A THOUSAND SOUTHERN LAWNS INGEST YOUR SOCKS AND UNDERWEAR. YAAAAAA!!!!9
Alright lads here is the thing, have not been posting anything other than replies to things cuz I have been busy being miserable at school and dealing with work stuff.
Our manager left us back in February. Because she was leaving I decided that I wanted to try a different path and went on to become a programmer analyst for my institution, if anything I knew that it was going to be pretty boring work, but it came with nice monetary compensation and a foot in the door for other data science related jobs in the future. Thing is, the department head asked me to stay in the web technologies department because we had a lack of people there and hiring is hard as shit, we do not do remote jobs since our work usually requires a level of discretion and security. Thus I have been working in the web tech department since she left albeit with a different title since I aced the interview for the analyst position and the team there were more than happy to have me. I have done very few things for them, some reports here and there and mostly working directly with the DBA in some projects. One migration project would have costed my institution a total of 58k and we managed to save the cost by building the migration software ourselves.....honestly it was a fucking cake walk, if you had any doubts about the shaddyness of enterprise level applications regarding selling overpriced shit with different levels of complexity, keep them, enterprise is shaddy af indeed. But I digress.
I wrote the specification for the manager position along the previous manager, we had decided that the next candidate needed to be strong with development knowledge as well as other things as to properly understand and manage a software team, we made the academic requirement(fuck you, yes we did ask for academic requirements) to be either in the Computer Science/software engineering area or at least on the Business Administration side. We were willing to consider BA holders in exchange for having knowledge of the development process of different products and a complete understanding of what developers go through. NOT ONE SINGLE motherfucker was able to satisfy this, some of them were idiots that I knew from before that had ABSOLUTELY no business even considering applying to the position, the courage it took for some of these assholes to apply would have hurt their mothers, their God if they had one, and their country, they were just that fucking bad in their jobs as well as being overall shit people.
Then we had 1 candidate actually fall through the cracks enough to get an interview. My dude here was lying out of his ass through the interview process. According to him he had "lots of Laravel experience and experience managing Laravel projects" and mentioned repeatedly how it would be a technology that we should consider for our products. I was to interview him alongside the vice president of our institution due to the head of my department and the rest of the managers for I.T being on vacation leave all at the same bloody time.
Backstory before the interview:
Whilst I was going over the interview questions with the vice president literally offered me the job instead. I replied with honesty, reflecting how I did not originally wanted him but feeling that our institution was ready to settle on any candidate due to the lack of potentials. He was happy to do it since apparently both him and the HOD were expecting me to step up sooner or later. I was floored.
Regardless, out of kindness he wanted to go through the interview.
So, going back to the interview. As soon as the person in question referenced the framework I started to ask him about it, just simple questions, the first was "what are your thoughts on the Eloquent ORM? I am not too fond of it and want to know what you as a full time laravel dev think of it"
his reply: "I am sorry I am not too familiar with it, I don't know what that is" <--- I appreciated his honesty in this but thought it funny that someone would say that he was a Laravel developer whilst not knowing what an ORM was since you can't really get away from using it on the initial stages of learning about Laravel, maybe if one wanted to go through the hurdle of switching to something like doctrine...but even then, it was....odd.
So I met with the hod when he came back, he was stoked at the prospect of having me become the manager and I happily accepted the position. It will be hell, but I don't even need to hit the ground running since I have been the face of the department since ages. My team were ecstatic about it since we are all close friends and they have been following my directions without complaints(but the ocational eat a dick puto) for some time, we work well together and we are happy to finally have someone to stop the constant barrage that comes from people taking advantage of a missing manager.
Its gonna get good, its gonna get fun, and i am getting to see how shit goes.7
This was not exactly the worst work culture because the employees, it was because the upper level of the organization chart on the IT department.
I'm not quite sure how to translate the exact positions of that chart, but lets say that there is a General Manager, a couple of Area Managers (Infrastructure, Development), some Area Supervisors (2 or 3, by each area), and the grunts (that were us). Anyway, anything on the "Manager" was the source of all the toxicity on the department.
First and foremost, there was a lack of training for almost any employee. We were expected to know everything since day-1. Yes, the new employees had a (very) brief explanation about the technologies/languages were used, but they were expected to perform as a senior employee almost since the moment they cross the door. And forget about having some KT (Knowledge Transfer) sessions, they were none existent and if they existed, were only to solve a very immediate issue (now imagine what happened when someone quit*).
The general culture that they have to always say "yes" to the client/customer to almost anything without consulting to the development teams if that what was being asked to do was doable, or even feasible. And forget about doing a proper documentation about that change/development, as "that was needed yesterday and it needs to be done to be implemented tomorrow" (you know what I mean). This contributes to the previous point, as we didn't have enough time to train someone new because we had this absurd deadlines.
And because they cannot/wanted to say "NO", there were days when they came with an amount of new requirements that needed to be done and it didn't matter that we had other things to do. And the worst was that, until a couple of years (more or less), there was almost impossible to gather the correct requirements from the client/user, as they (managers) "had already" that requirement, and as they "know better" what the user wants, it was their vision what was being described on the requirements, not the users'...
And all that caused that, in a common basis, didn't have enough time to do all this stuff (mainly because the User Support) causing that we needed to do overtime, which almost always went unpaid (because a very ambiguous clause of the contract, and that we were "non-union workers"**). And this is my favorite point of this list, because, almost any overtime went unpaid, so basically we were expected to be working for free after the end of the work day (lets say, after the 17:00). Leaving "early" was almost a sin for the managers, as they always expected that we give more time to work that the indicated on the contract, and if not, they could raise a report to HR because the ambiguous clause allowed them to do it (among other childish things that they do).
Finally, the jewel of the crown, is that they never, but never acknowledge that they made a mistake. Never. That was impossible! If something failed on the things/systems/applications that they had assigned*** it was always our fault.
- "A report for the Finance Department is giving wrong information? It's the DBA's fault**** because although he manages that report, he couldn't imagine that I have an undocumented service (that runs before the creation the report) crashed because I modified a hidden and undocumented temporal table and forgot to update that service."
But, well, at least that's on the past. And although those aren't all the things that made that workplace so toxic, for me those were the most prominent ones.
* Well, here we I live it's very common to don't say anything about leaving the company until the very last day. Yes, I know that there are people that leave their "2-days notice", but it's not common (IMHO, of course). And yes, there are some of us that give a 1 or 2-weeks notice, but still it's not a common practice.
** I don't know how to translate this... We have a concept called "trusted employee", which is mainly used to describe any administrative employee, and that commonly is expected to give the 110% of what the contract says (unpaid overtimes, extra stuff to do, etc) and sadly it's an accepted condition (for whatever reasons). I chose "non-union workers" because in comparison with an union worker, we have less protections (besides the legal ways) regarding what I've described before. Curiously, there are also "operative workers", that doesn't belong to an union, but they have (sometimes) better protections that the administrative ones.
*** Yes, they were in charge of several systems, because they didn't trust us to handle/maintain them. And I'm sure that they still don't trust in their developers.
**** One of the managers, and the DBA are the only ones that handle some stuff (specially the one that involves "money"). The thing that allows to use the DBA as scapegoat is that such manager have more privileges and permissions than the DBA, as he was the previous DBA2
10 year anniversary 'celebration' for a couple of employees (one dev, one a DBA) and the VP of the department was saying kind words about them, talking about the 'good old days'.
VP to the DBA: "I apologize, when you started, you walked into my database architecture. I didn't know that much back then and never thought about the architecture much beyond a few years. Its amazing my design has lasted over 20 years and triple digit business growth..blah blah blah"
Inner voice: "Mother F-er!...My database was designed IN SPITE of your meddling and demanding to create 1,500 field tables. Shut the F up you egotistical bastard!"
I can't even count how many times I had to stop him from, for example, adding a 'ProductID' field to a Customer table.
Me: "Why did you add a product id field?"
VP: "How else will we know what product the customer wants to buy?"
Me: "You mean like a wish list? What if the customer wants more than one product?"
VP: "Oh, that’s easy, we'll create more fields when that happens. ProductID2. Microsoft made it really easy to add fields."
Me: "We already have a wish list table schema. Customer can have as many wish lists and as many products as they want."
VP: "I don't understand. All I want is a field for me to store the product I'm buying. I don't know why you make this so hard, its just one more field."
Now the VP is bragging all the success was due to his expertise?! Gaaaaahhhh!
I quelled my rage with ample quantities of donuts, juice, and chocolate milk.
Ever had a day that felt like you're shoveling snow from the driveway? In a blizzard? With thunderstorms & falling unicorns? Like you shovel away one m² & turn around and no footprints visible anymore? And snow built up to your neck?
Today my work day was like that.. xcept shit..shit instead of pretty & puffy snow!!
Working on things a & b, trying to not mess either one up, then comes shit x, coworker was updating production.. ofc something went wrong.. again not testing after the update..then me 'to da rescue'.. :/ hardly patch things up, so it works..in a way.. feature c still missing due to needed workarounds.. going back to a and b.. got disrupted by the same coworker who is nver listening, but always asking too much..
And when I think I finally have the b thing figured out a f-ing blocker from one of our biggest clients.. The whole system is unresponsive.. Needles to say, same guy in support for two companies (their end), so they filed the jira blocker with the wrong customer that doesn't have a SLA so no urgent emails..and then the phone calls.. and then the hell broke loose.. checking what is happening.. After frantic calls from our dba to anyone who even knows that our customer exists if they were doing sth on the db.. noup, not a single one was fucking with the prod db.. The hell! Materialised view created 10 mins ago that blocked everything..set to recreate every 10 minutes..with a query that I am guessing couldn't even select all that data in under 15.. dafaaaq?! Then we kill it..and again it is there.. We found out that customers dbas were testing something on live environment, oblivious that they mamaged to block the entire db..
FML, I'm going pokemon hunting.. :/ codename for ingress n beer..3
Most awkward video conference call?
Our department is in a 'virtual' book club, reading The Unicorn Project, and I asked..
Me: "So what similarities have you seen with the Phoenix project and projects we work on here?"
Dale: "Ha ha..sooo many. The biggest is the disconnect of managers with no clue of what goes on."
<Vice president of our department also in the book club>
VP: "Really? Dale, I'd like to know more about this."
<awkward silence with blank stares all around>
DBA: "Come on Dale...spill the beans. Got the VP right there."
Dale: "Um...nope...not going there...nope"
<Dale's screen goes black>
VP: "OK, so when Maxine asks ..."
I actually never felt the need to scream at a co-worker so let's talk about that time a co-worker screamed at me instead.
tl;dr : some asshole boss screamed and threatened me because someone else's project was shit and didn't work.
Context: I was in my third year of school internship (graded) and my experience is C, C++, C#, Python all in systems programming, no web.
I was working as an intern for a shit company that was selling a shit software to hospitals (though not medically critical, thank God) the only tech guy on site was the DBA (cool guy) the product was maintained by a single dev in VB from his house, the dude never showed up to work (you'll understand why) and an other intern who couldn't dev shit.
I was working with the DBA on an software making statistical analysis from DB exports, worked nice, no problems here if we forget the lack of specs or boundaries (except must work in ieShit).
The other intern was working on something else (don't ask me what it is) I just remember it was in GWT before the community revived it. His webapp was requesting the company http server for a file instead of having one of it's java servlet to fetch it (both apps ran on sane server) which caused a lot of shit especially CORS error. That guy left (end of contract) and leaves his shit as is, boss asked me to deploy the app, I fiddle with it to see if it works and when I find out it doesn't then that asshole starts screaming at me in front of every other employee present, starts threatening to burn me in the tech world and have me thrown out of my school for no goddamn reason than the other dude's project doesn't work.
After the screaming I leave and warn my school immediately.
I guess that's why the other dev never came to work.
I had three weeks of internship left, that I did from home and worked probably less than 2 hours a day so suck it asshole.
Still had a good grade because I was reviewed by the DBA and he was happy with the work I did.
It was only later that I realized that what he did was categorizing as harassment (at least in France) and decided that never again this would happen without a response from my lawyer.1
I decided to quit my job. I was supposedly hired as an Android developer, but during the past few months I found myself doing everything except Android development: SQL scritps, frontend web development, backend web development, RESTful API's, DBA, release engineering... There's nothig wrong about being versatile, it's actually fun, but I wasn't doing what I really wanted to do and, most importantly, the manager didn't appreciate the fact that I was doing things that I'm not supposed to do, and not only that, I was doing it just as good as some full time web devs who do nothing but web development in the company. I'm pissed off. They probably believe the next Android dev they hire will do all the shit I was doing, accepting the same pay. Fuck them.2
I dunno about coolest, but I did sort of cement my reputation as the "database guy" in my first job because of this.
My first job was with a group maintaining a series of websites. Because of the nature of the websites, every morning we had to pull the records from one database on one network, sneaker net the data to a database on another network, and import the data via custom data import function.
However, the live site would crash after 100 or so records were imported. The dba at the live site had to script out a custom data partitioning script to do his daily duties, but it definitely messed up his productivity.
Turns out, the custom mass import function had recycled the standard import function, which was only used to import 1 record at a time, and it never closed its database connections, because it never needed to. A one line fix to production code was delivered 6 months later (because that was our release cycle) and I came up with the temporary work around, which was basically removing the connection limit. It would still crash with the work around, but only with multiple days worth of data. So basically only on Monday. Also developed the test set for the import (15k+ records).
The more I work with performance, the less I like generated queries (incl. ORM-driven generators).
Like this other team came to me complaining that some query takes >3minutes to execute (an OLTP qry) and the HTTP timeout is 60 seconds, so.... there's a problem.
Sure, a simple explain analyze suggests that some UIDPK index is queried repeatedly for ~1M times (the qry plan was generated for 300k expected invocations), each Index Scan lasts for 0.15ms. So there you go.. Ofc I'd really like to see more decimal zeroes, rather than just 0.15, but still..
Rewriting the query with a CTE cut down the execution time to pathetic 0.04sec (40ms) w/o any loops in the plan.
I suggest that change to the team and I am responded a big fat NO - they cannot make any query changes since they don't have any control on their queries
but down to 0.04sec from 3+ minutes....
alright, let's try to VACUUM ANALYZE, although I doubt this will be of any help. IDK what I'll do if that doesn't change the execution plan :/ Prolly suggest finding a DBA (which they won't, as the client has no € for a DBA).
All this because developers, the very people sho should have COMPLETE control over the product's code, have no control over the SQLs.
me to dba: do you have any recommendations of sql or query improvements? dba: no, just let us know once you're done.
after sending them explain plans, new queries and asking for reviews with no response from them, i applied the changes in dev.
after applying changes.
dba: you should involve us in any development. we need to collaborate.
me: please check your emails over the past 3 weeks.
Here's to @Wisecrack:
Some time ago I pitched an idea to my boss about a platform we implement to optimize some fucked-up processes and in fact a whole project and I boasted some 20-30% increase in productivity. Yeah, I know ... what a fucking big mouth.
Truth be told they (almost all project members) went all for it so we started working on that software.
A small step for me, a GIANT LEAP IN A FUCKING CESSPOOL.
And of course it's just the two of us - me and my colleague - as always.
And we don't have requirements - as always.
And now there are deadlines too!
And people be like: IS IT READY YET?
So between playing a consultant, a product owner, systems architect, product manager, designer, front-end/back-end developer, DBA, DevOps engineer, YOU-NAME-IT-ROLE, and dealing with my everyday work-related bullshit (because yes, I do that too) I lost all appetite for it.
I actually loved this idea and what it can be born out of it, now I'm frustrated. It's still relevant and it will still benefit them, but I am already FUCKING SICK AND TIRED OF IT.
So my "oh, how I'd love to help them" personality is fighting my "let them sink in their own shit" personality and I'll see which will come on top. :)
Truth is if I had the "5-years-ago me" energy a good chunk of that project would be done by now. 😁
Also yesterday my daughter had shouted at old people and had thrown stuff at them while at kindergarten. I sure hope they deserved it LOL.
I went for an interview as a MS SQL DBA. They gave me a technical knowledge test which I passed then started asking about me, All was going well until they started telling me what the company did.
They harvested people's data from websites online questionnaires and collated it into saleable data.
This was too much for me and I'm not certain that it isn't a grey area legally. They were told that they were scum of the earth and they could go fuck themselves.4
How many of you have started a job and ended up doing something totally different, and how did deal with it?
I was hired to be an SQL Developer, writing reports, views, stored procedures etc, but knew I would be asked to do some work on some internal c# apps.
Roll on almost 2 years and I'm pretty much a DBA in all but name, and the C# app I was supposed to be doing a little work on is now mine, so as well as being the sole programmer, I'm also the product owner, 1st, 2nd, and 3rd line support.
All of this and I've not even had a change in title, let alone a pay rise that reflect the new roles and responsibilities I've taken on.9
Let me explain:
Tl;dr : someone fucked up, I took shit, it was a gwt project. In a sense I don't hate GWT because of the framework itself but because how I was introduced and forced to "work" with it.
Was working as a paid intern at a small company there were 3 devs 2 interns and one senior employee that only worked from home handling the shit ton of legacy VB6 code he wrote over several year and a boss with no technical knowledge. (Other unimportant people as well)
I was working with their DBA (cool dude) because I was writing statistic and report generating software.
The other intern was tasked of doing a gwt app that was supposed to use a input file.
Rather than asking the user to upload it with a file picker (I guess they exist in gwt I didn't got to dig in the framework) he was trying to load the file with a http request directed at the same host the app was running on.
It did not work.
Then his contract was other and the app was left in an unfinished state.
The boss then tried to have the app deployed, the remaining dev dodged the bullet invoking some bullshit because he was clearly incapable of doing it.
So it fell on me, couldn't deploy the app because it was not even close to working.
Tried to fix things and make it work.
Turns out he thought it would take me 3h to deploy when I clearly explained that the other guy didn't finish the app.
Boss got mad, threatened to ruin my studies and my future career.
Couldn't because my uni had my back.
Didn't want to see me anymore.
Couldn't break my contract.
Told me to work from home for the end of my internship.
I got 3 weeks early vacation and got paid, fuck him, fuck GWT, fuck his company.
Still got well marked for the internship as my supervisor was the DBA who was happy with my work.
Don't let your intern unsupervised, don't let your main dev work from home when you don't know shit, don't piss me off and send me work from home.
Management: "You are the DBA and BI head right?"
Management: "Awesome, so that means you can develop a quick software for me to use? It's really easy!"
Me: "Awesome, here are two options since its easy, you go make it yourself or..."
Management: "Is that sarcasm?"
Me: "...or you can ask the accountant to make it"
IT Director: "you deserved that bob."
Deployment to production last night, and the whole thing is halted because the DBA didn't find it necessary to show up. 2 hours of calling everyone possible and nothing. Thanks DBA, for your prompt and timely handling of the deployment... 😑
Sysadmin's nemesis: a DBA. Especially an oracle DBA. There's no other kind of tech worker I've seen who's more opposed to best practices.
How about for devs?4
So.. Yeah.. A huge corp decided they no longer need a dba. For at least 2 months, starting DEC.
Ora, enterprise features - all the way in. Meaning only dbas have the knowledge
I've got some corn to pop. Anyone fancy to join?4
On this episode of “My Work Life” we will once again find that the code is fine, the DBAs just did something brainless again.3
Two words that instill fear in any database developer: Cartesian Join.
I didn’t realize what was happening until the DBA paid me a personal visit asking what I was running that had cached over a billion rows.3
Completely fucked up replication of MySQL servers.
Remote: 2 different database Servers
--> made sense.
Except the misconfiguration. Or better: No configuration at all.
So how to solve the massiv delays and make everything even more crazy?
2 remote servers - 2 readonly slaves for reading data remote (master - slave)
2 local (internal) servers.
Remote - Local Master Master.
Unfucking this cluster fuck was a real nightmare.
It had to be done at night, cause everything needed to be ripped apart.
And the servers were the backend of a warehouse with supply chain and multiple selling channels (Amazon, eBay etcetera).
So. It had to run the next day at 05.00 clock so the incoming orders could be packaged / prepared for shipping.
That was fun. Not.
And the clusterfuck died spectaculously on my first work day - the old DBA was gone (fired....)
So they say they're missing some data in the database and that the architect thinks it is the code not sending the msg to log these into the database.
Now instead of giving me what wasn't logged or even OPENING A TICKET DIRECTED AT ME, or at least somewhere I can see it or an email they want to have a code review with 5 people (1 of which is q&a and another a DBA).
They're also putting me on standby for this issue. Even though the architect says he will not disclose info until he knows what the issue is in ~an hour.
Is this normal..? I'm finding it this very odd. Up until now they'd make a ticket and I'd research the issue and fix it or explain why it doesn't work. I don't think we've had issues doing it that way. Feels so... Undermining.2
I am a backend dev in a mostly web based enterprise, i haven't touched c# for a month and a half so....
i am thinking in becoming a dba1
Project requirements include a database. I don't have permissions to create a new database on the server, so I go to the person that spins up new servers and deals with group policy. They rustle some papers around, looking aggravated, throws up hands and says, "I guess I'm the DBA now..." Well, what the fuck am I supposed to do shithead? Ask the web team to do it? We don't have a DBA. My boss has been gone all week and, really, this isn't a hard task. You check a fucking box.
Whatever, I'll remember this when they need a favor from me.6
When you cache index a faster query but your co-worker from other part of the world clears it.... It's been six times now dude2
I really don't understand this particular Government Department's IT Unit. They have a system and network to maintain except:
- They don't have a DBA
- They don't have a dedicated Network Engineer or Security Staff
- Zero documentation on all of the systems that they are taking care of (its all in each assigned particular staff's brain they said)
- Unsure and untested way of restoring a backup into a system
- Server passwords are too simple and only one person was holding this whole time and its to an Administrator account. No individual user account.
- System was developed by an in-house developer who is now retired and left very little documentation on its usage but nothing on how its setup.
But, the system has been up and operational for the past 20 years and no major issues whatsoever with the users using it. I mean its a super simple system setup from the looks of it.
1 App Server connected to 1 DB Server, to serve 20-30 users. But it contains millions of records (2GB worth of data dump). I'm trying to swing to them to get me on a part time work to fix these gaps.
God save them for another 20 years.3
The feeling when you and the DBA completely fix an issue that has been fucking up your users and that the third party vendors themselves couldn't fix on your own teamwork is so..... fucking... addicting.
Wrote an email to the hod to let us off a bit late tomorrow morning, least I can do for this fucking server admin, sql class A mastermind, Oracle fucking super pro.
I really pray for all of you mfkers to get the same type of coworker. this dude has taught me a lot and I really jump at the first opportunity I get to work with him. His accomplishments for the institution are many really, its just one of those happy bromances man.
I raise my beer mug, to the best fucking DBA i have ever worked with.
For my next trick, I am going to make sure the dude gets the position for the manager of his department as soon as the current dude retires (should be soon) a great man himself, but short on giving his dba the praise he deserves.
The previous manager of my departament told me "pay attention to <DBA NAME> he is your secret weapon and you will be his" and by heavens sweet momma was right.
Was a dba for a while. Mostly because I was the only one who knew SQL. Was working with an experienced dev doing front end work with no experience with front end work.
One day he calls over the cube wall "hey the database is broken" so I trudge over there, and see he messed up the call to the BAL from his code behind page. Later, he calls over the cube wall again. Same thing. 3 to 4 times a day. For a week. Finally my default became "no it isn't" and I continued working.
Then when it finally was a database problem, he had this smug look of satisfaction. Yes I'm the idiot.
How can a shitty student information system that already costs $20k/yr have an optional shitty SOAP API module, that only allows read access to records, that has an initial setup fee of $5k plus $5k/yr?!5
Me and the dba are slowly migrating parts of our JVM stack into .NET AND even tho I love and will always love Java and its ecosystem....I am glad.
IIS as a server is something that I actually look forward to since deploying shit to it is always a breeze
Installing ssl certs is a breeze
Everything is a fucking breeze
Before any of you cocksuckers say anything: this is my opinion only5
When I say I'm in "DevOps" what I really mean is that I'm a full-stack engineer, DBA, system administrator, security engineer, auditor, cloud custodian, cost optimization expert, and everything else that doesn't get its own dedicated staff member here. Pretty much the catch all between developers and customer.
When a senior engineer who was essentially a DBA on his last project is made tech lead on an iOS team. He had no iOS background before this and now he thinks hes a friggin architect.1
I hate ot when your client has access to new relic and they panic and start creating critical issues when the linux db server is is 80% memory..
After a 2h conference call the client said he was going to contact a 3rd party dba because he didn't believe it was normal.
DBA tells us 2 weeks in advance about database upgrade, but we developers forgot as manager insists in managing tasks via emails.
Monday morning brings the news that our web app isn't functioning properly. DBA responded saying he informed us in advance.
Manager blames it on the DBA that he informed us too much in advance.
I can take my share of blame but why the poor DBA for doing his job.1
"Just try it on production"
- a coworker DBA shortly before he desperately had to figure out how to restore to the latest backup.
I once saw a DBA using two for loops in PL/SQL to join two tables. I wonder if he knew about the "alternative" way to do that...2
Created a hidden role in the Postgres database, with some tricky constraints, which only a DBA can find and remove, to give myself, disguised as the postgres account, a role which no one else in the system can have.
Called it "pharaoh". Thug life ;2
I'm now not only a full stack developer in the charge of my own Linux servers, Devops work, programming,and the MySQL DBA, but have been asked to take on the "small" responsibilities of our only Linux administrator (retiring). No mention of title change (which is lesser than all my work), nor salary increase. A person can only do so much. Don't think I'm accepting this lightly or quietly, but to be assumed to take on more responsibility without benefit is beyond me. Mind you, this came down from my director; my boss made me privy.5
I was told during my midyear that I need to start working on my "AWS certification." Hmmmm. OK. Since there isn't one for a DBA, I guess I'm going to become an SA.1
Just had a recruiter that actually got my attention. Said they need an Oracle DBA that is interested in being trained for DevOps/AWS/Docker, and wanted to know if I was interested. That was a hard one, but I already have great pay, benefits, and perks that I don't want to give up, and they can't guarantee to replace. Now, if I was tired of what I'm doing ...5
When your IT VP starts speaking blasphemy:
We all know what’s going on with the API. Next week we may see 6x order volumes.
We need to do everything possible to minimize the load on our prod database server.
Here are some guidelines we’re implementing immediately:
· I’m revoking most direct production SQL access. (even read only). You should be running analysis queries and data pulls out of the replication server anyway.
· No User Management activities are allowed between 9AM and 9PM EST. If you’re going to run a large amount of updates, please coordinate with a DBA to have someone monitoring.
· No checklist setup/maintenance activities are allowed at all. If this causes business impact please let me know.
· If you see are doing anything in [App Name] that’s running long, kill it and get a DBA involved.
Please keep the communication level high and stay vigilant in protecting our prod environment!"
RIP most of what I do at work.3
Ha! Our Ops Support DBA Manager just asked (tongue in cheek) "if we are now supporting MS Access, too?" To which of course, the answer is no. Business user who install Access on their desktop and use it for business, get to provide their own support. As their Dev DBA, I'll be more than happy to help them migrate their data to SQL Server, Oracle, or Teradata, depending on the Use Case for the data. But, no, we don't support Access. Ever.
When your team can't change from the oracle database because the dba team will be out jobs and database links will be lost (people havent heard of apis)... :feelsoracleman:
dev = fighter
ops = cleris
network = wizard
design = bard
dba = warlock
infosec = rogue
it is not mine but i think its great analogy. and because i always choose the warlock1
I wanted to show our DBA an example of a web api using .net core 3 in regards of how easy it is to create such things. The reason? he has been wanting to get back into programming after many years of just sticking to dba related stuff. The dude has talent and brains, he had worked years ago as a delphi dev and a vb6 dev and we had the same employer at one point, none of this man's apps have been faced out on account of how complete they are and easy to maintain for other devs was after he left. Regardless of the ancient tech stacl, the man shows ample promise and well.
Thing is, the apps I make on the Microsoft stack usually tend to C#, and my frontends are using TS, so I am more on the curlt bracket side of things and he said he was to convert my app(very basic crud example, but with auth, authorization and everything in between to plug into the frontend) to VB.NET. I thought it wouldn't be that much of a problem but apparently microsoft does not hold templates for webapi for vb.net
I thought it was shitty. VB gave Microsoft a lot of developer market back in the VB6 days, and even though I really love c# I see no reason why they would just say fuck you like that to vb.net. Shit still polls pretty high in terms of dev popularity and you can apply the same design ideas to VB without much effort.
I just think this is very shitty from Microsoft's part. Much like how Apple is forcing people to adapt to Swift when there is a huge amount of obj c out there.
I dislike when companies shift focus on tech stacks like that.2
>how long do you think this migration will take?
>i don't know the database structures are entirely different
>so how long?
Before my vacation I’d been chatting with one of our dbas about an etl tool we needed for a customer we’d already signed all the contracts with saying we would provide one for a historical database of old data. They had been looking at one from SAP but in typical fashion a license was worth more than the actual contract.
Anyway long story short on the weekend before I went back to work I rattled together a little python proof of concept using a couple of azure databases and when I went back demo’d it to the pm and dba they loved it and we built on the poc to have a working loader which saved us about £30k by not buying the SAP product and just wrote our own.
Developers don’t have access to the test database so I had to put in a request for a dba to add two columns and it took over two hours for them to complete it.5
Idea for future week question, maybe not a rant 'per se', but interesting endeed: 'Describe the moment when you knew you wanted to be a programmer/developer/DBA..."
Yes, it was a single problem. And in the component, I've pointed at (and stuck to the whole triage time) 2 days ago.
2 days wasted.
At least I can finally have some closure for the weekend. It would be a shitty weekend if this bug wouldn't have been caught by today's EOD.
// Just needed to vent.
Our approval process for creating new database tables is ridiculous. After your table has been created it goes through two other men, both of whom have been woking with SQL for centuries and could actually just create the damn thing themselves and leave me the fuck out of it! Our DBA isn’t really much of a DBA...7
Do you have a db specialist in teams? I often see frontend & backend ranting but no db Devs. Isn't that role so crucial as one lost index can cause massive performance drops?5
Been told ill be worming with Oracle DB'S today, as someone who's never touched on Oracle stuff before. Any advice?4
Universal DBA wanted.
- working with multiple flavours of Sql dbs and operating systems,,
- chance to design, implement and manage secure ha solutions of the above solutions,
- unprecedented time pressure for all above
- no personal life whatsoever,
- your own wfh equipment,
- 24/7 availability
Today the DBA-team (needed to keep our Oracle mess running) decided to change datatypes in tables that has been in production for 20 years. Without a headsup.
Just took us 5 hours to debug good ol' visual basic code......
Why would you do such a thing!?
That moment when you explain to your manager what the term "full-stack" means and next thing you know you've gone from being a "backend c#" to azure migration expert/dba/designer/api architect/network engineer/php wordpress developer... let's go back to when you'd never heard of full-stack
So at the end of our sprint retrospective we (me and one other dev) apologize to the project owner for the fact that QA had to happen on dev for a few days because the database changes were not properly applied by the DBA. It was a one time thing, a little weird but not a big deal.
A few hours later we get a call from a new DBA saying that the project owner said that we wanted the old DBA off our project. So now we sound like assholes ruining a guy's self confidence and made working with him awkward as hell.
Thanks, project owner.
When the DBA restores the live db to staging and all your connections fail on passwords. Every fucking day!1
My team has a Database Admin 2 position open on the Arvest Career site. We are looking for someone with Data Warehousing/Data Integration background with SQL Server, ETL, SSIS, or equivalent. Also looking for a physical DBA with background in SQL Server, performance tuning, partitioning, DR/HA, Database migrations, dB refresh, dB restore, building out clusters.
Proposed future career path in order of sequence: Data Analyst, Data Scientist, DBA, Business Analyst, Machine Learning Scientist, Data Engineer.
Just made the leap from a company with and I.T. Department of 8 people to a company with an I.T. Department of 5000 people. At the old company, I was a SysAdmin, DBA, Helpdesk, Programmer, etc. 7/8 were programmers. We all had our hands in pretty much every aspect of the companies technology. At the new company I am just a programmer. I only write code (well, I will soon. There is a LOT I have to learn about how this place operates). I worked at the old companies for almost 3 years (2 as an Intern, 10 months as Salaried Full-Time) so it's weird having most of those aspects of my old job stripped away, yet at the same time it's somewhat freeing in the sense that I am now only responsible for 1 slice of the pie, not the whole damn thing. Anyway, hope everyone is having a great monday!1
My uncle is a dba so gets it, my dad sort of gets it because he also works in the transport sector, my mom doesn't Really understand but knows it's computers. Everyone else thinks it's it support
How to do you guys take a 'shift' in your profile when working as a developer for some company?
Like if you are a web dev guy and want to get into native android dev or vice versa or start building apps for ios or getting into a complete different role of manager, dba, sysadmin or hr, or whatever there are profiles in a company?
Would your company allow you to take some 3-4 months course or would you do that i your spare time?7
Work email conversation with a COTS app support tech:
Tech: Could you provide me the following from Sql Server? (instructions followed)
Me: I am not the DBA, but I've sent them the request.
Tech: Could you send me the following from Sql Server while we wait? (Instructions followed)
Me: As I said, I don't have permissions to access what you want.
Tech: Oh, I see now, you most likely don't have proper access to be able to retrieve the information.
Me: ..yeah. Thanks.
Today, my boss comes up to me to tell me that the CR he had me set up was not created by the release manager because there wasn't a QA test ticket or DBA review ticket. Look, if you're gonna shoot shit down without any bullets loaded, at least have the safety off because you had 3 hours on Friday to tell me that before leaving it to Monday morning when my head is hotter than the roast I'm sippin' on.
Is it wrong to think that having 9 sets of clustered application servers is an overkill. And what is up with mssql dba requesting 100+ ips for multiple instances of mssql standard clustered databases (all ips to be used on 2 virtual servers). No, not talking about docker instances here... these are no microservices. Monitoring is on the nice to have list. No space left for backups. I am properly scared.1
Anyone one here played around with CouchDB before or use it for personal projects or work?
At face value, it seems like a pretty good DB, Just wanted to get some idea from people that have used it before if it is actually pretty good.
I'm not a dba and don't know the caveats of DB tuning or management. So I'm in need of a DB with simplicity and easy management in mind Hahaha.
I'm mostly working with data that is either in JSON or hashtable/dictionary format so it felt like a NoSQL DB would be easy-ish to save my data into, plus I don't think (I hope my guess is right here) that I need regular SQL type relations with the data I'm working with.
Please help me with my noob-iness!Thank you! 😄
Spent the day helping someone troubleshoot why they can't print from their Samsung tablet, because somehow as a DBA I "know how to do that stuff." Never figured out why it wasn't working.
My previous job was Engineer ( Ops part of DevOps), supporting the devs with VMs, configurations, dev and test environments, CI maintenance, DBA, DB-dev and such, it was sexy.
In my current company, I have no technical role, but today's task: build a small webpage in sharepoint in HTML.
And the perv part is: it's still bettet than having no Technical task at all...2
Things I say at work:
"...and the words 'ghetto' and 'database architecture' don't ever belong in the same sentence."
No matter how many times I educate them, my colleagues still end up confused between writing conditions in join statement and in where statement.
They think they're both same :'(
The application I work on starts throwing timeout errors for about every third user. Lead developer cannot figure out what happened. DBA is out of town and cannot be reached. I do a quick Google search and run the stored procedure sp_updatestats. Timeouts stop and there is a big performance boost on the application. Everyone congratulates me on fixing the problem, and now I'm reading up on MS SQL Server Statistics and wondering about what other magical tools everyone else knows about that me and my team are clueless on...