Join devRant
Do all the things like
++ or -- rants, post your own rants, comment on others' rants and build your customized dev avatar
Sign Up
Pipeless API
From the creators of devRant, Pipeless lets you power real-time personalized recommendations and activity feeds using a simple API
Learn More
Search - "server room on fire"
-
I once set a customers server on fire.
Litterally.
I put my laptop ontop of the server rack and the additional heat generated caused blue smoke to fill the room.
The whole building had to be evacuated.
Firetrucks came.
The customer eyeballed me quite a bit after he talked to the firemarshal regarding where the fire had originated.10 -
Hello everyone, found this place recently, decided to bore you with one (or many) Navy story... tech Navy story. I'll start from the end.
Little backstory: I've deployed a simple domain setup on the ship I served, nothing fancy, a server, a switch, 10 computers, all Windows (details on that at another rant). I enter the ship Monday morning, and the XO tells me that he can't access his online folders.
OK, I say, I'll get to it. I fire up my laptop, try to RDP to the server (I know, I know, burn me at the stake later) no connection. WTF? Is the service down? I try pinging. No luck. I tried pinging the switch. OK. Looking at the switch admin panel, I see the server's port is dead. "OK, probably the cable." (we have old ethernet cables)
So, I drag my ass over to the server (same room with ship comms) with the cable tester to confirm that. What do I see?
The IMBECILES had pulled the plug from the server so that they could charge their mobile phones. I literally slammed my head against the door (calming exercise in case of spontaneous murder impulses - the things you learn at the Academy). My CO was nearby, and lucky for the guys, he heard me yell at them, while throwing mobiles and chargers around.
"But we thought it was OK, we just wanted to charge our-"
I kid you not, I reached for the firefighter's axe.
My CO grabbed me by the collar and dragged me to his room. I explained to him (between two cigarettes) that we MUST get a UPS and a server cabinet (budget constraints in the military are something that will give you people nightmares, trust me). I carefully explained to him that unless we got those, nothing would prevent the next moron from destroying confidential data and me from murdering him.
I plugged in and booted the server, after installing a multi socket extension. Two days after, surprise surprise, the server was off again. That was the first time I opened the door to the CO's room with a low kick. I must have looked like a psycho on drugs, he gave approval for the purchase in twenty seconds flat.
After that, I installed the UPS and the cabinet. Everything went inside, from the UPS to the very plugs. Just a locked box with cables coming out.
One of the guys came to my room, and asked if I could unlock the cabinet so that they could plug a "device" they needed.
I actually reached for my folding knife.
Disclaimer: The story above is TRUE. Even the almost violent parts.23 -
“Yeah but you’re not a *real* developer”
Fuck. you.
I wrote 80% of this code base. I do 80% of the tickets/storyboard points. I do all of the QA. My nose is to the grindstone every fucking day honing this craft and sweating my balls off like a blacksmith staring into the red hot kiln while the sores of previous mistakes scream bloody murder from the unrelenting exposure to heat. I saw this amazing industry of opportunity, freedom and self examination and wanted in no matter what it took. I glued myself to every pithy resource I could possibly get my hands on and crawled through the muck and filth of it all until I could keep myself warm with the smallest spark of my own making. I stoked that spark until it became a fire and stoked that fire until I could set entire forests ablaze. I listened to the ungrateful people keeping warm by my combustion saying it “wasn’t hot enough” or “would have been a nicer colour if they did it” or “could have warmed up just fine jogging on the spot”. I made painstaking alterations to my ignition and watched my undeserving benefactors gradually be silenced and begin to sit quietly by the heat. I jumped into that inferno daily, was reduced to ash daily and emerged reborn daily. But you are right! I didn’t get scammed out of $40k+ studying technology in an archaic institution from instructors who don’t give a shit and answering “D all of the above” for 4+ years straight therefor my opinion doesn’t mean shit. Push your bullshit to prod and watch the server come burning out of the cloud as the apocalyptic swarm of angry tickets come flooding in why don’t you? Bet they didn’t teach you that in school. You’ve never poked around inside an open source codebase in your life. They are just a mystery boxes of magic that unless someone holds your hands with finely crafted instructions containing a 50/50 picture to word ratio you throw a hissy fit. Every problem that comes up instead of working to solve it you reflexively point to the first person in the room while thinking with your pea brain how you can possibly scapegoat them into taking the fall for whatever it is that’s come up today you couldn’t possibly understand.
Not a real developer?
Fuck. You.28 -
*goes to the local town hall to get my new ID*
A week ago:
Clerk: Sorry sir, our systems don't work anymore, we can't process your request!
Me: Epic. Is there any sysadmin in here that can fix this pronto?
C: No it's a centrally managed system. It's managed by the people in ${another town}.
M (thinking): Well how about you fucking call them then, fucking user. Screaming blood and fire when nothing is wrong server-side but doing nothing when there is. Fucking amazing, useless piece of shit.
One week later, i.e. today:
M: Hey, I'd like to renew my ID card. I've got this announcement document here and my current ID card.
C: Oh no I don't need the announcement document. I need your PIN and PUK code letter.
M (thinking): What the fuck do you need that for.. isn't that shit supposed to be my private information..?
*gives PIN and PUK part of the letter*
C: Alright, to register your new ID card, please enter your PUK and then your PIN in this card reader here twice.
M: Sure, but I'd like to change both afterwards. After all they're written on this piece of paper and I'm not sure that just destroying that will be enough.
C: Sure sure you can change them. Please authenticate with the codes written on the paper.
*Authenticates*
C: So you'd like to change your codes, right?
M: Yeah but I'd like to change it at home. You know, because I can't know for sure that this PC here is secure, the card reader has a wired connection to your PC (making it vulnerable to keyloggers) and so on.
C: Impossible. You can't change your PIN at home. (What about the PUK?!)
M: But I've done that several times with my Digipass for my previous passport.. it is possible and I've done it myself.
C: Tut tut, impossible. I know it's impossible and therefore it is.
M (thinking): Thanks for confirming that I really shouldn't enter my personal PIN on your fucking PC, incompetent bitch.
M: Alright, I'll just keep this PIN, try at home and if it's really impossible because the system changed to remove this functionality (which I highly doubt, that'd be really retarded), I'll come back later.
(Just to get rid of this old stupid woman's ignorance essentially.)
C: Sure sure...
Me: I'd also like to register as an organ donor. Where can I do that?
C: That'd be over there. *points to the other room in the town hall*
FUCKING THANK YOU LORDS OF THE WICKED RAVEN AND THE LIBERATED TUX, TO GET ME AWAY FROM THAT STUPID FUCKING BITCH!!!
.. anyway. I've got my new ID and I'm an official organ donor now 🙂6 -
Worst dev team failure I've experienced?
One of several.
Around 2012, a team of devs were tasked to convert a ASPX service to WCF that had one responsibility, returning product data (description, price, availability, etc...simple stuff)
No complex searching, just pass the ID, you get the response.
I was the original developer of the ASPX service, which API was an XML request and returned an XML response. The 'powers-that-be' decided anything XML was evil and had to be purged from the planet. If this thought bubble popped up over your head "Wait a sec...doesn't WCF transmit everything via SOAP, which is XML?", yes, but in their minds SOAP wasn't XML. That's not the worst WTF of this story.
The team, 3 developers, 2 DBAs, network administrators, several web developers, worked on the conversion for about 9 months using the Waterfall method (3~5 months was mostly in meetings and very basic prototyping) and using a test-first approach (their own flavor of TDD). The 'go live' day was to occur at 3:00AM and mandatory that nearly the entire department be on-sight (including the department VP) and available to help troubleshoot any system issues.
3:00AM - Teams start their deployments
3:05AM - Thousands and thousands of errors from all kinds of sources (web exceptions, database exceptions, server exceptions, etc), site goes down, teams roll everything back.
3:30AM - The primary developer remembered he made a last minute change to a stored procedure parameter that hadn't been pushed to production, which caused a side-affect across several layers of their stack.
4:00AM - The developer found his bug, but the manager decided it would be better if everyone went home and get a fresh look at the problem at 8:00AM (yes, he expected everyone to be back in the office at 8:00AM).
About a month later, the team scheduled another 3:00AM deployment (VP was present again), confident that introducing mocking into their testing pipeline would fix any database related errors.
3:00AM - Team starts their deployments.
3:30AM - No major errors, things seem to be going well. High fives, cheers..manager tells everyone to head home.
3:35AM - Site crashes, like white page, no response from the servers kind of crash. Resetting IIS on the servers works, but only for around 10 minutes or so.
4:00AM - Team rolls back, manager is clearly pissed at this point, "Nobody is going fucking home until we figure this out!!"
6:00AM - Diagnostics found the WCF client was causing the server to run out of resources, with a mix of clogging up server bandwidth, and a sprinkle of N+1 scaling problem. Manager lets everyone go home, but be back in the office at 8:00AM to develop a plan so this *never* happens again.
About 2 months later, a 'real' development+integration environment (previously, any+all integration tests were on the developer's machine) and the team scheduled a 6:00AM deployment, but at a much, much smaller scale with just the 3 development team members.
Why? Because the manager 'froze' changes to the ASPX service, the web team still needed various enhancements, so they bypassed the service (not using the ASPX service at all) and wrote their own SQL scripts that hit the database directly and utilized AppFabric/Velocity caching to allow the site to scale. There were only a couple client application using the ASPX service that needed to be converted, so deploying at 6:00AM gave everyone a couple of hours before users got into the office. Service deployed, worked like a champ.
A week later the VP schedules a celebration for the successful migration to WCF. Pizza, cake, the works. The 3 team members received awards (and a envelope, which probably equaled some $$$) and the entire team received a custom Benchmade pocket knife to remember this project's success. Myself and several others just stared at each other, not knowing what to say.
Later, my manager pulls several of us into a conference room
Me: "What the hell? This is one of the biggest failures I've been apart of. We got rewarded for thousands and thousands of dollars of wasted time."
<others expressed the same and expletive sediments>
Mgr: "I know..I know...but that's the story we have to stick with. If the company realizes what a fucking mess this is, we could all be fired."
Me: "What?!! All of us?!"
Mgr: "Well, shit rolls downhill. Dept-Mgr-John is ready to fire anyone he felt could make him look bad, which is why I pulled you guys in here. The other sheep out there will go along with anything he says and more than happy to throw you under the bus. Keep your head down until this blows over. Say nothing."11 -
Worst fight I've had with a co-worker?
Had my share of 'disagreements', but one that seemed like it could have gone to blows was a developer, 'T', that tried to man-splain me how ADO.Net worked with SQLServer.
<T walks into our work area>
T: "Your solution is going to cause a lot of problems in SQLServer"
Me: "No, its not, your solution is worse. For performance, its better to use ADO.Net connection pooling."
T: "NO! Every single transaction is atomic! SQLServer will prioritize the operation thread, making the whole transaction faster than what you're trying to do."
<T goes on and on about threads, made up nonsense about priority queues, on and on>
Me: "No it won't, unless you change something in the connection string, ADO.Net will utilize connection pooling and use the same SPID, even if you explicitly call Close() on the connection. You are just wasting code thinking that works."
T walks over, stands over me (he's about 6.5", 300+ pounds), maybe 6 inches away
T: "I've been doing .net development for over 10 years. I know what I'm doing!"
I turn my chair to face him, look up, cross my arms.
Me: "I know I'm kinda new to this, but let me show you something ..."
<I threw together a C# console app, simple connect, get some data, close the connection>
Me: "I'll fire up SQLProfiler and we can see the actual connection SPID and when sql server closes the SPID....see....the connection to SQLServer is still has an active SPID after I called Close. When I exit the application, SQLServer will drop the SPD....tada...see?"
T: "Wha...what is that...SQLProfiler? Is that some kind of hacking tool? DBAs should know about that!"
Me: "It's part of the SQLServer client tools, its on everyone's machine, including yours."
T: "Doesn't prove a damn thing! I'm going to do my own experiment and prove my solution works."
Me: "Look forward to seeing what you come up with ... and you haven't been doing .net for 10 years. I was part of the team that reviewed your resume when you were hired. You're going to have to try that on someone else."
About 10 seconds later I hear him from across the room slam his keyboard on his desk.
100% sure he would have kicked my ass, but that day I let him know his bully tactics worked on some, but wouldn't work on me.7 -
I took this contract and made the suggestion that we backup to the cloud and create a private repo on GIT. Client said no, local should be fine, they don't want someone stealing their code. I said okay fine.
AC just went out in the server room and they apparently had a leak from the AC to the power supply which they happened to put on top of the rack servers and switches. I'm surprised that place didn't catch fire, might be to early to call it.
All this on a Friday and we were 2 weeks away from launch party.
Not my fault, I clearly said we backup to cloud and use GIT on private repo.3 -
If there's something wrong in the server room
Who you gonna blame? (the intern)
If the hard drive's grilled
And it don't look good
Who you gonna blame? (the intern)
If you're seeing keys in your'r github's HEAD
Who you gonna blame? (the intern)
Your PC's on fire
and the website's dead
Who you gonna blame? (the intern)
Disclaimer: My internship was a pretty cool time actually... :)