Do all the things like ++ or -- rants, post your own rants, comment on others' rants and build your customized dev avatarSign Up
Get a devDuck
Rubber duck debugging has never been so cute! Get your favorite coding language devDuckBuy Now
Search - "voodoo"
Worst thing you've seen another dev do? Long one, but has a happy ending.
Classic 'Dev deploys to production at 5:00PM on a Friday, and goes home.' story.
The web department was managed under the the Marketing department, so they were not required to adhere to any type of coding standards and for months we fought with them on logging. Pre-Splunk, we rolled our own logging/alerting solution and they hated being the #1 reason for phone calls/texts/emails every night.
Wanting to "get it done", 'Tony' decided to bypass the default logging and send himself an email if an exception occurred in his code.
At 5:00PM on a Friday, deploys, goes home.
Around 11:00AM on Sunday (a lot folks are still in church at this time), the VP of IS gets a call from the CEO (who does not go to church) about unable to log into his email. VP has to leave church..drive home and find out he cannot remote access the exchange server. He starts making other phone calls..forcing the entire networking department to drive in and get email back up (you can imagine not a group of happy people)
After some network-admin voodoo, by 12:00, they discover/fix the issue (know it was Tony's email that was the problem)
We find out Monday that not only did Tony deploy at 5:00 on a Friday, the deployment wasn't approved, had features no one asked for, wasn't checked into version control, and the exception during checkout cost the company over $50,000 in lost sales.
Was Tony fired? Noooo. The web is our cash cow and Tony was considered a top web developer (and he knew that), Tony decided to blame logging. While in the discovery meeting, Tony told the bosses that it wasn't his fault logging was so buggy and caused so many phone calls/texts/emails every night, if he had been trained properly, this problem could have been avoided.
Well, since I was responsible for logging, I was next in the hot seat.
For almost 30 minutes I listened to every terrible thing I had done to Tony ever since he started. I was a terrible mentor, I was mean, I was degrading, etc..etc.
Me: "Where is this coming from? I barely know Tony. We're not even in the same building. I met him once when he started, maybe saw him a couple of times in meetings."
Andrew: "Aren't you responsible for this logging fiasco?"
Me: "Good Lord no, why am I here?"
Andrew: "I'll rephrase so you'll understand, aren't you are responsible for the proper training of how developers log errors in their code? This disaster is clearly a consequence of your failure. What do you have to say for yourself?"
Me: "Nothing. Developers are responsible for their own choices. Tony made the choice to bypass our logging and send errors to himself, causing Exchange to lockup and losing sales."
Andrew: "A choice he made because he was not properly informed of the consequences? Again, that is a failure in the proper use of logging, and why you are here."
Me: "I'm done with this. Does John know I'm in here? How about you get John and you talk to him like that."
'John' was the department head at the time.
Andrew:"John, have you spoken to Tony?"
John: "Yes, and I'm very sorry and very disappointed. This won't happen again."
John: "You know what. Did you even fucking talk to Tony? You just sit in your ivory tower and think your actions don't matter?"
Me: "Whoa!! What are you talking about!? My responsibility for logging stops with the work instructions. After that if Tony decides to do something else, that is on him."
John: "That is not how Tony tells it. He said he's been struggling with your logging system everyday since he's started and you've done nothing to help. This behavior ends today. We're a fucking team. Get off your damn high horse and help the little guy every once in a while."
Me: "I don't know what Tony has been telling you, but I barely know the guy. If he has been having trouble with the one line of code to log, this is the first I've heard of it."
John: "Like I said, this ends today. You are going to come up with a proper training class and learn to get out and talk to other people."
Over the next couple of weeks I become a powerpoint wizard and 'train' anyone/everyone on the proper use of logging. The one line of code to log. One line of code.
A friend 'Scott' sits close to Tony (I mean I do get out and know people) told me that Tony poured out the crocodile tears. Like cried and cried, apologizing, calling me everything but a kitchen sink,...etc. It was so bad, his manager 'Sally' was crying, her boss 'Andrew', was red in the face, when 'John' heard 'Sally' was crying, you can imagine the high levels of alpha-male 'gotta look like I'm protecting the females' hormones flowing.
Took almost another year, Tony released a change on a Friday, went home, web site crashed (losses were in the thousands of $ per minute this time), and Tony was not let back into the building on Monday (one of the best days of my life).9
Sometimes I feel I'm the live version of stack overflow for my family and friends...
Mom: The printer is acting up, how do I fix it?
Brother: I can't login to Netflix, help!
Grandparents: could you fix the radio/TV?
Gf: Apple can't validate my .uk mail
All to which I replied with the answer to life, the universe and everything...
FUCKING GOOGLE IT! Have you googled it?!
And the inevitable reply: no.
The only ones who get a free pass are my grandparents they were born in the 40's and think YouTube works on voodoo.9
A story for you all to enjoy. It is of my entry into a serious sysadmin position. They say when you're remembering something, you're remembering the last time you remembered it. I think it helps your memory to purposefully remember things now and then. Hence one reason I'm writing this.
I've worked as a sysadmin for a small ISP for 3ish years in the thick of the Canadian Rockies. I ended up here because I wanted to get the hell out of any big cities, which seem to breed depression in me. I started at the company as literally the NOC (network ops center) cleaning boy, although my boss knew I had linux experience (but not my 2 years+ of self taught Java/PHP/Python experience). There were 2 other employees at the time, plus our boss. I was 18 when I started, so naturally the two employees (installers) presumed I was some young punk who knew shit all, and THEY were gonna teach me a thing or two about working at an ISP. Ha..
Soon I was being given access to systems which had been neglected for years prior to my starting work there. They had 3 old servers with 2GB ram each (this was in 2013/2014) and dual core Xeon CPUs. One was their dedicated router, acting as nothing but a gateway for their public subnets, no firewall. The other had an install of airControl 2 (they had just began an upgrade from 10 year old Trango radio infrastructure to Ubiquiti), and the other was supposed to run "Powercode" (i cringe every time I see or hear the name) which was to alleviate the boss of countless hours of manual traffic accounting labour each month.
Unfortunately they had been running without any form of backups or network monitoring for about 10 years prior to my joining. I saw myself as being able to help, having a fresh mind (before joining I had been on a 1+ year computer hiatus), and luckily was given the freedom to do pretty much what I wanted with the hardware they had in a short amount of time. My observation at the time was that no one at this company had a quarter of the hands-on, get-shit-done, just-write-a-freakin-script IT experience I did, which has ended up being true. I now know that I knew practically nothing about Linux when I started here, even though I had been managing a dedicated debian (and for a while CentOS) box for a couple of years; my (still existing) Minecraft gaming community and server (ask for the domain!)
Without going through all the gory details, I did help their IT infrastructure - a fucking LOT. Not to stoke myself but by not acknowledging that, I feel fucked up. (imposter syndrome) And sadly given the nature of this profession, sometimes (most of the time) the only one that can give you recognition is yourself.
ok some gory details. they had a completely flat network, meanimg every device was accessible to every other device over layer 2, a shitload of broadcast traffic everywhere all the time, constant danger of someone plugging in their router backwards and becoming the new dhcp server for the network, easy MAC spoofing, ability to fuck everything by creating a network loop, etc. I am honestly amazed they lasted 10 years like that. All running on comsumer grade netgear unmanaged switches. VLANs? voodoo magic. Routing? how's that work?
To my sheer joy, they had a Mikrotik router on the shelf that someone tried but never succeeded at setting up. I knew nothimg about them, but dove in, and fell in love RouterOS. It was the key to fixing this network. Well, it and some new servers that supported virtualization.
It turns out that actual hardware virtualization wasn't necessary; I had at first set up the new servers utilizing it however, running XenServer with HA lizard for two-node high availability. That worked for about a year before hitting a seriously strange segfault issue that I found to be impossible to debug. our neighbouring ISP's (with whom we arent in direct competition) sysadmin and I got to talking, and he helped me get started with Puppet earlier this year. Out went Xenserver, in came LXD; containerization instead of virtualization. Waaayyy higher application density on the server as you no longer give each virtual machine a fixed RAM allotment. Puppet has been super fun to work with, am sitting at about 80% automation of everything with it, having spent the last 5 months working to puppetize every single thing that runs on our servers. Now I'm working mainly on achieving high availability across every service.
Shit I'm nearly at the character limit. The network is no longer flat - it has a routed backbone with MPLS/VPLS overlaid to keep every customer in the same L2 domain (to preserve public IPv4 address), but with complete Layer 2 isolation between customers to prevent the issues I mentioned earlier. We run every service essential to an ISP (netflow collection, complete network monitoring (Zabbix), CRM software) as well as others, and it's been the learning experience of a life time for me.
The only crappy thing has been the imposter syndrome & lonliness.5
Being mentored by a CSS ninja atm, It's starting to make more sense. It's less voodoo magic and more endless mind games with the browser that really wants to hurt you2
I am so excited, I just got a response from Voodoo.io on publishing my mobile game. Hopefully it makes a decent amount of money!4
just found this
// ...A shed load of voodoo is then executed...
// .. removing the need for this would have involved messing around with the existing code which is such a rat's nest that I would rather avoid it
started by writing a perl script to crawl a website. The problem with that, is the fact ot affected the way I program. And not in a g/ood/i way.
My client wants me to "help increase sales" by applying seo voodoo. For over two years they didn't even have Analytics. Now, after 12 days they're asking me what do I make of the results. Like that's a meaningful amount of data for analysis.3
Every couple of hours a certain request from our web app gets a CORS error from our server. Refresh the page and everything works perfectly. WTF...1
Alright members of the underground, voodoo community called ''computer science''.
I am done with my environment's subjective, limited POV and i'm here to ask the holy council of DevRant a simple question.
I'd hate to call it a thread but i'm secretly thumbs upping for it.
Alright, genies of the lamp, i ask of you the answer to "React-native vs Kotlin vs Xamarin" with a follow-up question of "which to consider continuing woth if I tried all of them, don't have any preferences, and don't know what kind of projects i'm going to work on?"3
Just seen in a old script that misbehaves
### Let the DB2 voodoo begin! #####
Seriously fuck you
If there's one thing that gets my goat it's "voodoo debuggers."
There's no actual need to dig into the root cause of a problem if you can blame the new thing you don't understand. Especially when later, after someone competent actually looks into it, the bug turns out to be a change in the old stuff that did it.
If there's two things that get my goat, it's people who fix something caused by human error or negligence and then don't write an automated test to catch it the next time it happens.