Do all the things like ++ or -- rants, post your own rants, comment on others' rants and build your customized dev avatarSign Up
Get a devDuck
Rubber duck debugging has never been so cute! Get your favorite coding language devDuckBuy Now
Search - "4 hours of life lost"
[This makes me sound really bad at first, please read the whole thing]
Back when I first started freelancing I worked for a client who ran a game server hosting company. My job was to improve their system for updating game servers. This was one of my first clients and I didn't dare to question the fact that he was getting me to work on the production environment as they didn't have a development one setup. I came to regret that decision when out of no where during the first test, files just start deleting. I panicked as one would and tried to stop the webserver it was running on but oh no, he hasn't given me access to any of that. I thought well shit, I might as well see where I fucked up since it was midnight for him and I wasn't able to get a hold of him. I looked at every single line hundreds of times trying to see why it would have started deleting files. I found no cause. Exhausted, (This was 6am by this point) I pretty much passed out. I woke up around 5 hours later with my face on my keyboard (I know you've all done that) only to see a good 30 messages from the client screaming at me. It turns out that during that time every single client's game server had been deleted. Before responding and begging for forgiveness, I decided to take another crack at finding the root of the problem. It wasn't my fault. I had found the cause! It turns out a previous programmer had a script that would run "rm -rf" + (insert file name here) on the old server files, only he had fucked up the line and it would run "rm -rf /". I have never felt more relieved in my life. This script had been disabled by the original programmer but the client had set it to run again so that I could remake the system. Now, I was never told about this specific script as it was for a game they didn't host anymore.
I realise this is getting very long so I'll speed it up a bit.
He didn't want to take the blame and said I added the code and it was all my fault. He told me I could be on live chat support for 3 months at his company or pay $10,000. Out of all of this I had at least made sure to document what I was doing and backup every single file before I touched them which managed to save my ass when it came to him threatening legal action. I showed him my proof which resulted in him trying to guilt trip me to work for him for free as he had lost about 80% of his clients. By this point I had been abused constantly for 4 weeks by this son of a bitch. As I was underage he had said that if we went to court he'd take my parents house and make them live on the street. So how does one respond? A simple "Fuck off you cunt" and a block.
That was over 8 years ago and I haven't heard from him since.
If you've made it this far, congrats, you deserve a cookie!6
In today's episode of kidding on SystemD, we have a surprise guest star appearance - Apache Foundation HTTPD server, or as we in the Debian ecosystem call it, the Apache webserver!
So, imagine a situation like this - Its friday afternoon, you have just migrated a bunch of web domains under a new, up to date, system. Everything works just fine, until... You try to generate SSL certificates from Lets Encrypt.
Such a mundane task, done more than a thousand times already... Yet... No matter what you do, nothing works. Apache just returns a HTTP status code 403 - Forbidden.
Of course, what many folk would think of first when it came to a 403 error is - Ooooh, a permission issue somewhere in the directory structure!
So you check it... And re-check it to make sure... And even switch over to the user the webserver runs under, yet... You can access the challenge just fine, what the hell!
So you go deeper... And enable the most verbose level of logging apache is capable of - Trace8. That tells you... Not a whole lot more... Apparently, the webserver was unable to find file specified? But... Its right there, you can see it!
So you go another step deeper and start tracing the process' system calls to see exactly where it calls stat/lstat on the file, and you see that it... Calls lstat and... It... Returns -1? What the hell#2!
So, you compile a custom binary that calls lstat on the first argument given and prints out everything it returns... And... It works fine!
Until now, I chose to omit one important detail that might have given away the issue to the more knowledgeable right away. Our webservers have the URL /.well-known/acme-challenge/, used for ACME challenges, aliased somewhere else on the filesystem - To /tmp/challenges.
See the issue already?
Some *bleep* over at the Debian Package Maintainer group decided that Apache could save very sensitive data into /tmp, so, it would be for the best if they changed something that worked for decades, and enabled a SystemD service unit option "PrivateTmp" for the webserver, by default.
What it does is that, anytime a process started with this option enabled writes to /tmp/*, the call gets hijacked or something, and actually makes the write to a private /tmp/something/tmp/ directory, where something... Appeared as a completely random name, with the "apache2.service" glued at the end.
That was also the only reason why I managed fix this issue - On the umpteenth time of checking the directory structure, I noticed a "systemd-private-foobarbas-apache2.service-cookie42" directory there... That contained nothing but a "tmp" directory with 777 as its permission, owned by the process' user and group.
Overriding that unit file option finally fixed the issue completely.
I have just one question - Why? Why change something that worked for decades? I understand that, in case you save something into /tmp, it may be read by 3rd parties or programs, but I am of the opinion that, if you did that, its only and only your fault if you wrote sensitive data into the temporary directory.
And as far as I am aware, by default, Apache does not actually write anything even remotely sensitive into /tmp, so...
I wasted 4 hours of my life debugging this! Only to find out its just another SystemD-enabled "feature" now!
And as much as I love kidding on SystemD, this time, I see it more as a fault of the package maintainers, because... I found no default apache2/httpd service file in the apache repo mirror... So...9
Day 1 with Chromium OS: Inclusion of packages and stuff
Day 2 with Chromium OS: Setting up CI, and realize Azure is fucking gay because their own agents disconnects after 4 hours.
Day 3 with Chromium OS: resolve their shitty problem, now their own agents have no disk space. I blame Google.
Day 4 with Chromium OS: Fix CI in at 10 commits, give up and cry.
Day 5 with Chromium OS: Realized Travis might stood a chance, build time limit reached, now I'm shook.
Day 6 with Chromium OS: Buried myself with endless tabs of Gentoo documentation. Lost count on when's the last time I came out of my room.
Today with Chromium OS: I blame Google for making my life suffer more than the last time I had depression.
Conclusion: Chromium OS is Gentoo with extra steps and I hate it6
I don't know if someone has noticed but I haven't been on DevRant lately. It's not that the community is awesome. In the last month or two, I've had a blast of an experience here. I've just been avoiding screens, specifically texts in screens. I think something snapped on my head last week. Here's why:
As I've said in other rants/comments, I study history, and at the moment, I haven't found any career that has to read more than this one. Sometimes I've had to read about 1200 pages in less than three days. Last week I had to read 6 books which accounted for about 3500 pages. I was actively reading more than 600 pages a day. Now, this was for an investigation, and each of these reads had to be properly summarised with their respective arguments, thesis, etc. So I intensely read everything before Thursday, the day in which I had to present my work, in which I referenced about 10 books.
Apart from that, daily, I spent 4 hours coding. That's been the minimum I've done daily since I started learning.
I wasn't too tired. I'm used to read a lot, and coding is always fun. But the problem came in Friday when I woke up with a strange headache that spanned from my eyes to the back of my ears. Hurting especially on the sides of my forehead.
It eventually dissipated, but whenever I read something, the ache slowly came back. Loud noises and bright lights also brought it back. So you could imagine, everytime I tried to read a Rant, comment, etc, the headache came back. The same for coding and reading. For fucks sake I feel like I'm fucking crippled.
And no, the pain isn't the worst. Pain is pain and you can't do anything about it. The worst is that I'm developing some anxiety here. In all this time I have been learning daily nonstop. Coding was something I craved for everyday. Now I'm fucking wasting entire days in non-productive activities. I'm losing my fucking time here guys!
I'm afraid I have some anxiety problem with time. I've already fucking wasted entire years, now I don't want to continue wasting them and push my goals further away, I want to get to my goals as soon as I can because time and life can't be stopped and once time is lost, you can't fucking get it back. And, considering I'm still 21, I do notice this feeling is somehow irrational, but for fucks sake, I'm wasting fucking LIFE :(