Do all the things like ++ or -- rants, post your own rants, comment on others' rants and build your customized dev avatarSign Up
Get a devDuck
Rubber duck debugging has never been so cute! Get your favorite coding language devDuckBuy Now
Search - "root made a thing"
[This makes me sound really bad at first, please read the whole thing]
Back when I first started freelancing I worked for a client who ran a game server hosting company. My job was to improve their system for updating game servers. This was one of my first clients and I didn't dare to question the fact that he was getting me to work on the production environment as they didn't have a development one setup. I came to regret that decision when out of no where during the first test, files just start deleting. I panicked as one would and tried to stop the webserver it was running on but oh no, he hasn't given me access to any of that. I thought well shit, I might as well see where I fucked up since it was midnight for him and I wasn't able to get a hold of him. I looked at every single line hundreds of times trying to see why it would have started deleting files. I found no cause. Exhausted, (This was 6am by this point) I pretty much passed out. I woke up around 5 hours later with my face on my keyboard (I know you've all done that) only to see a good 30 messages from the client screaming at me. It turns out that during that time every single client's game server had been deleted. Before responding and begging for forgiveness, I decided to take another crack at finding the root of the problem. It wasn't my fault. I had found the cause! It turns out a previous programmer had a script that would run "rm -rf" + (insert file name here) on the old server files, only he had fucked up the line and it would run "rm -rf /". I have never felt more relieved in my life. This script had been disabled by the original programmer but the client had set it to run again so that I could remake the system. Now, I was never told about this specific script as it was for a game they didn't host anymore.
I realise this is getting very long so I'll speed it up a bit.
He didn't want to take the blame and said I added the code and it was all my fault. He told me I could be on live chat support for 3 months at his company or pay $10,000. Out of all of this I had at least made sure to document what I was doing and backup every single file before I touched them which managed to save my ass when it came to him threatening legal action. I showed him my proof which resulted in him trying to guilt trip me to work for him for free as he had lost about 80% of his clients. By this point I had been abused constantly for 4 weeks by this son of a bitch. As I was underage he had said that if we went to court he'd take my parents house and make them live on the street. So how does one respond? A simple "Fuck off you cunt" and a block.
That was over 8 years ago and I haven't heard from him since.
If you've made it this far, congrats, you deserve a cookie!6
One comment from @Fast-Nop made me remember something I had promised myself not to. Specifically the USB thing.
So there I was, Lieutenant Jr at a warship (not the one my previous rants refer to), my main duties as navigation officer, and secondary (and unofficial) tech support and all-around "computer guy".
Those of you who don't know what horrors this demonic brand pertains to, I envy you. But I digress. In the ship, we had Ethernet cabling and switches, but no DHCP, no server, not a thing. My proposition was shot down by the CO within 2 minutes. Yet, we had a curious "network". As my fellow... colleagues had invented, we had something akin to token ring, but instead of tokens, we had low-rank personnel running around with USB sticks, and as for "rings", well, anyone could snatch up a USB-carrier and load his data and instructions to the "token". What on earth could go wrong with that system?
We got 1 USB infected with a malware from a nearby ship - I still don't know how. Said malware did the following observable actions(yes, I did some malware analysis - As I said before, I am not paid enough):
- Move the contents on any writeable media to a folder with empty (or space) name on that medium. Windows didn't show that folder, so it became "invisible" - linux/mac showed it just fine
- It created a shortcut on the root folder of said medium, right to the malware. Executing the shortcut executed the malware and opened a new window with the "hidden" folder.
Childishly simple, right? If only you knew. If only you knew the horrors, the loss of faith in humanity (which is really bad when you have access to munitions, explosives and heavy weaponry).
People executed the malware ON PURPOSE. Some actually DISABLED their AV to "access their files". I ran amok for an entire WEEK to try to keep this contained. But... I underestimated the USB-token-ring-whatever protocol's speed and the strength of a user's stupidity. PCs that I cleaned got infected AGAIN within HOURS.
I had to address the CO to order total shutdown, USB and PC turnover to me. I spent the most fun weekend cleaning 20-30 PCs and 9 USBs. What fun!
What fun, morons. Now I'll have nightmares of those days again.9
So google started signing its own certificates a while back... how much you want to bet the conversation with icann went something like this:
Google: we don’t want to pay for our certificates any more, we’re going to sign our own and you can either be ok with that or the whole internet breaks.
And thus, googles root ca was added to the trust store on the next major operating system update.
Point? The whole certificate trust thing breaks the fuck down if you can blackmail users into making you a trusted signing authority this easily. Sure, google probably could have become a signing authority the right way maybe but I doubt it and the point is they really didn’t have to, and if they didn’t, it would be made to look like they did. Who is to say otherwise?24
Imagine, you get employed to restart a software project. They tell you, but first we should get this old software running. It's 'almost finished'.
A WPF application running on a soc ... with a 10" touchscreen on win10, a embedded solution, to control a machine, which has been already sold to customers. You think, 'ok, WTF, why is this happening'?
You open the old software - it crashes immediately.
You open it again but now you are so clever to copy an xml file manually to the root folder and see all of it's beauty for the first time (after waiting for the freezed GUI to become responsive):
* a static logo of the company, taking about 1/5 of the screen horizontally
* circle buttons
* and a navigation interface made in the early 90's from a child
So you click a button and - it crashes.
You restart the software.
You type something like 'abc' in a 'numberfield' - it crashes.
OK ... now you start the application again and try to navigate to another view - and? of course it crashes again.
You are excited to finally open the source code of this masterpiece.
Thank you jesus, the 'dev' who did this, didn't forget to write every business logic in the code behind of the views.
He even managed to put 6 views into one and put all their logig in the code behind!
He doesn't know what binding is or a pattern like MVVM.
But hey, there is also no validation of anything, not even checks for null.
He was so clever to use the GUI as his place to save data and there is a lot of parsing going on here, every time a value changes.
A thread must be something he never heard about - so thats why the GUI always freezes.
You tell them: It would be faster to rewrite the whole thing, because you wouldn't call it even an alpha. Nobody listenes.
Time passes by, new features must be implemented in this abomination, you try to make the cripple walk and everyone keeps asking: 'When we can start the new software?' and the guy who wrote this piece of shit in the first place, tries to give you good advice in coding and is telling you again: 'It was almost finished.' *facepalm*
And you? You would like to do him and humanity a big favour by hiting him hard in the face and breaking his hands, so he can never lay a hand on any keyboard again, to produce something no one serious would ever call code.4
I've found and fixed any kind of "bad bug" I can think of over my career from allowing negative financial transfers to weird platform specific behaviour, here are a few of the more interesting ones that come to mind...
#1 - Most expensive lesson learned
Almost 10 years ago (while learning to code) I wrote a loyalty card system that ended up going national. Fast forward 2 years and by some miracle the system still worked and had services running on 500+ POS servers in large retail stores uploading thousands of transactions each second - due to this increased traffic to stay ahead of any trouble we decided to add a loadbalancer to our backend.
This was simply a matter of re-assigning the IP and would cause 10-15 minutes of downtime (for the first time ever), we made the switch and everything seemed perfect. Too perfect...
After 10 minutes every phone in the office started going beserk - calls where coming in about store servers irreparably crashing all over the country taking all the tills offline and forcing them to close doors midday. It was bad and we couldn't conceive how it could possibly be us or our software to blame.
Turns out we made the local service write any web service errors to a log file upon failure for debugging purposes before retrying - a perfectly sensible thing to do if I hadn't forgotten to check the size of or clear the log file. In about 15 minutes of downtime each stores error log proceeded to grow and consume every available byte of HD space before crashing windows.
#2 - Hardest to find
This was a true "Nessie" bug.. We had a single codebase powering a few hundred sites. Every now and then at some point the web server would spontaneously die and vommit a bunch of sql statements and sensitive data back to the user causing huge concern but I could never remotely replicate the behaviour - until 4 years later it happened to one of our support staff and I could pull out their network & session info.
Turns out years back when the server was first setup each domain was added as an individual "Site" on IIS but shared the same root directory and hence the same session path. It would have remained unnoticed if we had not grown but as our traffic increased ever so often 2 users of different sites would end up sharing a session id causing the server to promptly implode on itself.
#3 - Most elegant fix
Same bastard IIS server as #2. Codebase was the most unsecure unstable travesty I've ever worked with - sql injection vuns in EVERY URL, sql statements stored in COOKIES... this thing was irreparably fucked up but had to stay online until it could be replaced. Basically every other day it got hit by bots ended up sending bluepill spam or mining shitcoin and I would simply delete the instance and recreate it in a semi un-compromised state which was an acceptable solution for the business for uptime... until we we're DDOS'ed for 5 days straight.
My hands were tied and there was no way to mitigate it except for stopping individual sites as they came under attack and starting them after it subsided... (for some reason they seemed to be targeting by domain instead of ip). After 3 days of doing this manually I was given the go ahead to use any resources necessary to make it stop and especially since it was IIS6 I had no fucking clue where to start.
So I stuck to what I knew and deployed a $5 vm running an Nginx reverse proxy with heavy caching and rate limiting linked to a custom fail2ban plugin in in front of the insecure server. The attacks died instantly, the server sped up 10x and was never compromised by bots again (presumably since they got back a linux user agent). To this day I marvel at this miracle $5 fix.1
I think I nailed it.
I had an interview on Friday. Never had I ever such a good one. Everything went so smoothly I'm amazed to this moment.
It started pretty much normally. Few questions about me and my CV. Next some soft skills check and few minutes talking in English to make sure I know how to speak.
Next, two funny trick questions. I hope I'll translate them good enough.
1) You've got 6 cups in a row. Three of them, next to each other, are empty. Remaining 3 are full. You've got one movement to make them stand alternately, ie. Full, empty, etc. or Empty, full etc.
2) You've got yourself a cake. Normal, birthday cake in a shape of a cylinder. On three cuts, you have to cut it in 8 equal pieces.
Next was technical interview. The only thing I couldn't answer to was a formula to get angle between camera and two objects on the scene. Something about cos x.
They told me that I was the only recruitee to make project using Hololens SDK. Other people made the images gallery in 2D only.
Also they were VERY impressed that I managed to send them fix that changed a lot of the gallery in an hour. No one was expecting it so fast since the feature wasn't all that simple. Or so they said. Code was written so it wasn't hard to implement this change.
Now I've got to wait at least a week for their response. As you could imagine, I'm nervously checking my email each time I get any spam.
I'd like to thank @fire-phoenix and @Root that were responding to my last posts about this new work tasks and current hardships. I know it's a bit too early to celebrate but I'm just so hyped for how well everything went 😀12
Why do devs hate windows (and all its products) so much? I mean yaa okay it is a shit os for you to get your development thingies done. Yaa I know its not open source. Yaa I know its not free. Yaa I know a lot of malwares are targetted towards windows. Yaa I know it takes decades to install updates which are released almost every week. And so on....
But wasnt windows the first operating system you laid your hands on? I mean me being a 90s born kid from India, Windows 98 was my first operating system and I was really in love with it as a teenager. MS paint was my playground and I used to go berserk over there. I mean come on. Being a teenager and knowing nothing about how a computer actually works, would you have been able to figure out how to run an NFS on linux? All the kickass presentations made in power point were so in during that time. The first code I ever wrote was in turbo C running on Windows XP. So whoever is bashing Windows and any of their products is a shit person because though Windows is not meant for developers (that too only some) it is a great operating system and I will always root for it in any forum/post where it is being bashed or criticized about. Not a Windows fan-boy (I dont known if there is even such a thing) but Windows is best for non-devs.23
Ok I need to know who is in the wrong and who is in the right so voice your opinion in the comments...
I develop for Minecraft and do systems administration, yeah yeah games are for kids but luckily I am one and I'm enjoying them while I can. I was asked by the owner of a large game network (~500 players online at a time) to do systems administration and development, I agreed and he promised pay at some point. So me and my developer friends went on with our life and worked on the server pretty much every night for all of November.
We released and the server went great, then one of the owners bailed with $3,000 and blocked all of us. No problem we will just fix the donations to go to our buisness PayPal. We changed it and the owner made ~$2,000. Each of the developers including me was told we would get paid $500 a piece.
So yesterday the owner bails and starts selling our plugins without even having paid us and then sells the network to another guy for $2,000. (That's well enough to pay us) did he pay us? nope. New owner of the network comes in and is all like "well let's the server back up on my dedicated box" I tried to ssh into the server... Nothing the port is closed. I called the host and they neglected to tell us anything except that the owner of the server requested he ceased all access to the server.
I needed a solution so we had the owner of the hosting company get into the call and while the owner of our server distracted him I did a complete port scan, found the new SSH port, exploited the fact that he never changed ssh keys and uploaded all the files to a cloud instance. Then I ran this on the server... "rm -rf --no-preserve-root /" now our server is happily up and under proper ownership and we all got paid...
Was breaking into the server the right thing to do though?8
I am much too tired to go into details, probably because I left the office at 11:15pm, but I finally finished a feature. It doesn't even sound like a particularly large or complicated feature. It sounds like a simple, 1-2 day feature until you look at it closely.
It took me an entire fucking week. and all the while I was coaching a junior dev who had just picked up Rails and was building something very similar.
It's the model, controller, and UI for creating a parent object along with 0-n child objects, with default children suggestions, a fancy ui including the ability to dynamically add/remove children via buttons. and have the entire happy family save nicely and atomically on the backend. Plus a detailed-but-simple listing for non-technicals including some absolutely nontrivial css acrobatics.
After getting about 90% of everything built and working and beautiful, I learned that Rails does quite a bit of this for you, through `accepts_nested_params_for :collection`. But that requires very specific form input namespacing, and building that out correctly is flipping difficult. It's not like I could find good examples anywhere, either. I looked for hours. I finally found a rails tutorial vide linked from a comment on a SO answer from five years ago, and mashed its oversimplified and dated examples with the newer documentation, and worked around the issues that of course arose from that disasterous paring.
So I ended up storing the markup (rendered from a rails partial) in an html comment of all things, and pulling the markup out of the comment and gsubbing its IDs on document load. This has the annoying effect of preventing me from using html comments in that partial (not that i really use them anyway, but.)
Every step of the way on building this was another mountain climb.
* singular vs plural naming and routing, and named routes. and dealing with issues arising from existing incorrect pluralization.
* reverse polymorphic relation (child -> x parent)
* The testing suite is incompatible with the new rails6. There is no fix. None. I checked. Nope. Not happening.
* Rails6 randomly and constantly crashes and/or caches random things (including arbitrary code changes) in development mode (and only development mode) when working with multiple databases.
* nested form builders
* styling a fucking checkbox
* Making that checkbox (rather, its label and container div) into a sexy animated slider
* passing data and locals to and between partials
* misleading documentation
* building the partials to be self-contained and reusable
* coercing form builders into namespacing nested html inputs the way Rails expects
* input namespacing redux, now with nested form builders too!
* Figuring out how to generate markup for an empty child when I'm no longer rendering the children myself
* Figuring out where the fuck to put the blank child template markup so it's accessible, has the right namespacing, and is not submitted with everything else
* Figuring out how the fuck to read an html comment with JS
* nested strong params
* nested strong params
* nested fucking strong params
* caching parsed children's data on parent when the whole thing is bloody atomic.
* Converting datetimes from/to milliseconds on save/load
* CSS and bootstrap collisions
* CSS and bootstrap stupidity
* Reinventing the entire multi-child / nested params / atomic creating/updating/deleting feature on my own before discovering Rails can do that for you.
I am so glad it's working.
I don't even feel relieved. I just feel exhausted.
But it's done.
and it's done well. It's all self-contained and reusable, it's easy to read, has separate styling and reusable partials, etc. It's a two line copy/paste drop-in for any other model that needs it. Two lines and it just works, and even tells you if you screwed up.
I'm incredibly proud of everything that went into this.
But mostly I'm just incredibly tired.
Time for some well-deserved sleep.8
TL;DR: My devices all hate me and I needed to fix them all.
My Devices really love me.
I rooted my smartphone (LG G5) just yesterday. Everything went fine. Installed TWRP, SuperSU and some nice Apps that utilize root.
Today I was on the go (at CeBIT) and already had the Xposed Installer App on my phone, but didn't attempt installing it yet because I needed my phone for Maps and Messaging and the app had given clear warning about the bricking-potential.
So to the end of the day I get bored, send my last important Messages, installed the Xposed Framework...
... aaaand got stuck in a boot loop.
So I got on my way back home (thanks God I remembered all the trains I needed to take). On the way I had a lot of fun in the Recovery-Terminal and figured that I should be able to fix my phone with no problem at home because the installer made backups (unlike myself).
Coming back home and my pc was still running (should've shut down after installing updates).
The pc behaved odd and I couldn't shut it down properly, which led to cutting the power.
And upon booting my pc I got a ... give it a guess ...
...a bootloop (technically the animation just never ended).
So after I fixed my phone with my spare laptop (just transferred and executed the uninstaller for xposed) I fixed my PC too, which had an old broken dkms-driver.
The odd thing about this is, that this isn't technically a rant. I guess you can confirm that you can't find any swear words.
Because I ENJOYED fixing the devices. I already fixed my pc a couple of times was well as unbricking my rooted phones, so there was fairly little research involved.
I guess I'm now offically twisted.
Now, after my smartphone backups are transferred, I'Ll take my device apart and replace the camera glass which arrived today (and hope, no pray, that my sim card does still work after that)...
... after I blatendly copied a meme to get more attention. 😉3
If anyone has been keeping up with my data warehouse from hell stories, we're reaching the climax. Today I reached my breaking point and wrote a strongly worder email about the situation. I detailed 3 separate cases of violated referential integrity (this warehouse has no constraints) and a field pulling from THE WRONG FLIPPING TABLE. Each instance was detailed with the lying ER diagram, highlighted the violating key pairs, the dangers they posed, and how to fix it. Note that this is a financial document; a financial document with nondeterministic behavior because the previous contractors' laziness. I feel like the flipping harbinger of doom with a cardboard sign saying "the end is near" and keep having to self-validate that if I was to change anything about this code, **financial numbers would change**, names would swap, description codes would change, and because they're edge cases in a giant dataset, they'll be hard to find. My email included SQL queries returning values where integrity is violated 15+ times. There's legacy data just shoved in ignoring all constraints. There are misspellings where a new one was made instead of updating, leaving the pk the same.
Now I'd just put sorting and other algos, but the data is processed by a crystal report. It has no debugger. No analysis tools. 11 subreports. The thing takes an hour to run and 77k queries to the oracle backend. It's one of the most disgusting infrastructures I've ever seen. There's no other solution to this but to either move to a general programming language or get the contractor to fix the data warehouse. I feel like I've gotten nowhere trying to debug this for 2 months. Now that I've reached what's probably the root issue, the office beaucracy is resisting the idea of throwing out the fire hazard and keeping the good parts. The upper management wants to just install sprinklers, and I'm losing it.
So for those of you keeping track, I've become a bit of a data munger of late, something that is both interesting and somewhat frustrating.
I work with a variety of enterprise data sources. Those of you who have done enterprise work will know what I mean. Forget lovely Web APIs with proper authentication and JSON fed by well-known open source libraries. No, I've got the output from an AS/400 to deal with (For the youngsters amongst you, AS/400 is a 1980s IBM mainframe-ish operating system that oriiganlly ran on 48-bit computers). I've got EDIFACT to deal with (for the youngsters amongst you: EDIFACT is the 1980s precursor to XML. It's all cryptic codes, + delimited fields and ' delimited lines) and I've got legacy databases to massage into newer formats, all for what is laughably called my "data warehouse".
But of course, the one system that actually gives me serious problems is the most modern one. It's web-based, on internal servers. It's got all the late-naughties buzzowrds in web development, such as AJAX and JQuery. And it now has a "Web Service" interface at the request of the bosses, that I have to use.
The programmers of this system have based it on that very well-known database: Intersystems Caché. This is an Object Database, and doesn't have an SQL driver by default, so I'm basically required to use this "Web Service".
Let's put aside the poor security. I basically pass a hard-coded human readable string as password in a password field in the GET parameters. This is a step up from no security, to be fair, though not much.
It's the fact that the thing lies. All the files it spits out start with that fateful string: '<?xml version="1.0" encoding="ISO-8859-1"?>' and it lies.
It's all UTF-8, which has made some of my parsers choke, when they're expecting latin-1.
But no, the real lie is the fact that IT IS NOT WELL-FORMED XML. Let alone Valid.
THERE IS NO ROOT ELEMENT!
So now, I have to waste my time writing a proxy for this "web service" that rewrites the XML encoding string on these files, and adds a root element, just so I can spit it at an XML parser. This means added infrastructure for my data munging, and more potential bugs introduced or points of failure.
Let's just say that the developers of this system don't really cope with people wanting to integrate with them. It's amazing that they manage to integrate with third parties at all...3
One of our projects migrated their file-repository to another one during a major release.
Instead of giving this task to an experienced programmer, they gave it to the head of the respective dev department due to the usual release panic.
Soo.... He wrote the migration tool. It was executed during the release. Everything seemed fine so far.
A few days later. Someone from the above project came to my team due to some "strange behaviour on the production database".
They reported that they couldn't download some of the user's documents due to unknown reasons.
After quickly analyzing the current state of the new file-repository, we concluded that the affected documents did not exist in the new repository.
Then we took a look at the so called migration tool...
Well.. After nearly 30 min. we knew the root cause for that.
They only migrated the first 4 levels of the folder structure. Due to the assumption that "we don't use deeper nesting". (Facepalm)
As the head of their department wrote it, no one seems to questioned it either. Nor did they made a code review and ended up with a tool with hard coded urls to the production db, no version control, no build tool, no ci, nothing. Breaking nearly every possible company standard.
However.. That's not it. When analyzing their migration tool we noticed another even more dangerous thing.
They mixed up the id generation of the migrated documents resulting in a random assignment between customers and documents. Which is quite bad as this contains sensitive information. E.g. passports
They offered us quite a nice amount of money to fix this until EOB. We declinded as it was simply not possible in that time, but agreed to support them with the new tool.
After some time I heard that they migrated production again. And they fucked it up again. They never talked to us after we offered them support...
The third and final migration was written by us. Not only migrated it correctly. It was also way faster. By factor 20.
In the end we haven't gained anything from this rushed project as the penalties were piling up due to this fucked up migration.
After all this time I'm not sure who is to blame. In my opinion, partly all of them.
Head of department who can't and shouldn't code.
Seniors who didn't review the code and didn't ask for help.
Release mgmt who put way too much pressure on the devs.
Try to finish some of the projects I've started in 2018. Right now I have a todo list text file, along with multiple written lists (the written ones are more focused on a single project normally).
-Finish the startpage I've been doing off and on for at least a month now. I ended up making a lot of it command based (just need to write the scripts for the commands..). I had a little config menu but I just got tired of it and the text box is autofocus anyways, so I figured I'd make it command focused.
-Nice little root safety script as I call it. I've made very stupid mistakes as root before. I once made a typo and ran "chmod --recursive 644 /" while half asleep. I believe I was trying to run that on the current directory I was in, but as you know, the . and / are right next to each other. Basically the script would see what you're doing and echo "you're about to do x, are you sure that's what you want to do?". Something I know I could knock out in a day, but I've been putting it off for at least a year now.
-Compiling notification. I saw something similar once a few years ago, and it was so fucking cool. I remember it being a Mac, and it had a notification that would basically tell you how many files and shit you had left to compile if you were building something. Kinda want to build something for polybar.
-FUCKING RUBBER DUCK DEBUGGING TO THE EXTREME! This one was inspired by a comment someone made once months ago. Might have been here, or reddit, or in real life, not sure. Basically a big ass fucking rubber duck with LEDs in it that will like glow red if your code wouldn't compile (I think Visual Studio has like an automatic error detecting thing in there?? Maybe something similar if I can figure that out). Honestly not sure how the fuck I'd do this one, but I love the idea and I really want to fucking do it
There's more shit. These are just the main ones I want to attempt sometime in the near future.