Worst legacy experience...

Called in by a client who had had a pen test on their website and it showed up many, many security holes. I was tasked with coming in and implementing the required fixes.

Site turned out to be Classic ASP built on an MS Access database. Due to the nature of the client, everything had to be done on their premises (kind of ironic but there you go). So I'm on-site trying to get access to code and server. My contact was *never* at her desk to approve anything. IT staff "worked" 11am to 3pm on a long day. The code itself was shite beyond belief.

The site was full of forms with no input validation, origin validation and no SQL injection checks. Sensitive data stored in plain text in cookies. Technical errors displayed on certain pages revealing site structure and even DB table names. Server configured to allow directory listing in file stores so that the public could see/access whatever they liked without any permission or authentication checks. I swear this was written by the child of some staff member. No company would have had the balls to charge for this.

Took me about 8 weeks to make and deploy the changes to client's satisfaction. Could have done it in 2 with some support from the actual people I was suppose to be helping!! But it was their money (well, my money as they were government funded!).

TLDR; I just screwed a production server and rendered it useless!!!

Long story:

I went to install a product that we built at the customer's site, and was given a Linux running server, to deploy our app.

I work in windows, and barely know the basic Linux commands.

So I look at the files in the home directory, and see that the are a lot of files, so I ask the customer if it is ok that I move all the files to a separate directory.

He agrees, and me thinking that I am smart, proceed to enter the following commands in the terminal:

mkdir old
mv /* old

Of course I got an error that I don't have permission so my next command was:

sudo mv /* old

And that was the end of that computer.

The amazing part of the story is that as soon as it happened, I understood so much about Linux.

The file structure, sudo, the power of the terminal, aliases and so much more...

Real Linux fuck up coming up.

Be me.
Working on a project.

Accidentally used sudo with git pull, every file now is only accessible for root users.

Thinking to myself.. okay I'll just do chmod 777 to the current directory

Forgot how to use chmod
At first something like
Sudo chmod ./ 777

Not working
Maybe
Sudo chmod / 777

Not working
Remembering that it's the other way around

sudo chmod 777 /

Now... I fucked up. I forgot the dot, and for a sec I forgot that '/' means root directory and not current like './' does.

Few moments later the permission system of Ubuntu is utterly fucked. Everything is not working.

Need sudo in order to fix everything but sudo isn't working.

Few hours of crying later,I solved it thanks to some nice ppl online helping stupid people like me who used that command...😂

Back when I migrated my file server to a VM, I used robocopy to copy everything over (and caused quite a few fuckups before it'd behave halfway decent.. or so I thought). Now I tried to add my desktop's SSH key to the authorized_keys on said server but it wouldn't accept my key after that. After a bit of digging I've found that my entire home directory (where the file server hosts its mirror of my D: drive) had everything set to 777, and that's why the key isn't accepted. Great permission mode, isn't it? Much secure, very wow! Thank you so much robocopy!!!

So this just happened. I was working on a project and I just found a weird directory named '~' in there. I am on Linux so I simply did an "rm -rf ~" :/

It was too late when I realized it deleted all my files in my home directory. All my projects and configuration files. The sad part was it did not delete that shitty random directory because permission denied. Thank God I got into the habit of making weekly backups of my system and Thank God I use git.

Managed to make myself look like a fucking moron again today...

Can't mount NFS share, get "permission denied". Huh, that's weird... It's correctly exported.

Well it's correctly exported and rpcinfo -p $HOST times out... Must be firewall rule.

Firewall rule is changed but still no joy "permission denied"... Fuck sake networks, can't you do anything right first time?!!!

Firewall rule is correct I am reliably informed... Go about proving that it's not fucking correct and provide "evidence" to show this, I was a little bit more blunt than was strictly required.

Networks say they will take another look.

I turn NFS logging to verbose for my own interest and notice the line "path/to/directory is not a valid directory".

I, as a moron, had missed a "/" at the start of the path. That's why I still couldn't mount after the firewall change.

Go over and apologise in person and explain how I'm a total idiot.

$md5 = md5_file($file_uri);
=> Returns false

Mmh...

die(is_readable($file_uri));
=> Returns false

[Mmh intensifies]

if(!is_readable($file_uri)) {
chmod($file_uri, 0777);
}

=> Chmod() returns false

GAAAAH FUCK THIS I CALL IT A DAY

In today's episode of kidding on SystemD, we have a surprise guest star appearance - Apache Foundation HTTPD server, or as we in the Debian ecosystem call it, the Apache webserver!

So, imagine a situation like this - Its friday afternoon, you have just migrated a bunch of web domains under a new, up to date, system. Everything works just fine, until... You try to generate SSL certificates from Lets Encrypt.

Such a mundane task, done more than a thousand times already... Yet... No matter what you do, nothing works. Apache just returns a HTTP status code 403 - Forbidden.

Of course, what many folk would think of first when it came to a 403 error is - Ooooh, a permission issue somewhere in the directory structure!

So you check it... And re-check it to make sure... And even switch over to the user the webserver runs under, yet... You can access the challenge just fine, what the hell!

So you go deeper... And enable the most verbose level of logging apache is capable of - Trace8. That tells you... Not a whole lot more... Apparently, the webserver was unable to find file specified? But... Its right there, you can see it!

So you go another step deeper and start tracing the process' system calls to see exactly where it calls stat/lstat on the file, and you see that it... Calls lstat and... It... Returns -1? What the hell#2!

So, you compile a custom binary that calls lstat on the first argument given and prints out everything it returns... And... It works fine!

Until now, I chose to omit one important detail that might have given away the issue to the more knowledgeable right away. Our webservers have the URL /.well-known/acme-challenge/, used for ACME challenges, aliased somewhere else on the filesystem - To /tmp/challenges.

See the issue already?

Some *bleep* over at the Debian Package Maintainer group decided that Apache could save very sensitive data into /tmp, so, it would be for the best if they changed something that worked for decades, and enabled a SystemD service unit option "PrivateTmp" for the webserver, by default.

What it does is that, anytime a process started with this option enabled writes to /tmp/*, the call gets hijacked or something, and actually makes the write to a private /tmp/something/tmp/ directory, where something... Appeared as a completely random name, with the "apache2.service" glued at the end.

That was also the only reason why I managed fix this issue - On the umpteenth time of checking the directory structure, I noticed a "systemd-private-foobarbas-apache2.service-cookie42" directory there... That contained nothing but a "tmp" directory with 777 as its permission, owned by the process' user and group.

Overriding that unit file option finally fixed the issue completely.

I have just one question - Why? Why change something that worked for decades? I understand that, in case you save something into /tmp, it may be read by 3rd parties or programs, but I am of the opinion that, if you did that, its only and only your fault if you wrote sensitive data into the temporary directory.

And as far as I am aware, by default, Apache does not actually write anything even remotely sensitive into /tmp, so...

Why. WHY!
I wasted 4 hours of my life debugging this! Only to find out its just another SystemD-enabled "feature" now!

And as much as I love kidding on SystemD, this time, I see it more as a fault of the package maintainers, because... I found no default apache2/httpd service file in the apache repo mirror... So...

So I was just about to post a whole long rant about something breaking with an update. But I literally just found out, the whole thing was my fault because I changed something. I feel so fucking stupid. I went on a rant in a Discord chat with a couple friends, blamed fucking everything I could possibly think of.

Then I remembered when I tweaked a config file just a few days ago. "Maybe that has something to do with it....?" YUP, I'm a bit stupid.

Basically I changed an environment variable, and the variable I was referencing in it isn't being set (which is an issue itself, but I can figure that out), so instead of looking in that folder, it was looking in the root directory, and I was getting some permission denied errors cause..I was running the program as my normal user. Of course I shouldn't be able to write to root as a normal user.

I guess I'm a bit stupid sometimes when I'm sick.

Went to scan a file with the built in Win 10 scanner. It came up and said the file was excluded from scan. So it gave me a link to the exclusion directories of the virus scanner. I looked and it had the entire C drive as the last exclusion. I did not do this. I have specific directories on exclusion, but not the entire drive.

I got rid of this exclusion and scanned the entire drive. I didn't get any virus or worm hits.

Is this a sign of an infection?

It would have had to be changed with something that asked for permission to make administrative changes like an install. I honestly don't know what I would have installed that I gave permission to do this lately.

Is there a way to accidentally right click somehow to add a directory to the exclusion list?

This happened to me sometime back.

I want to try out a WordPress plugin in my local machine before installing on a production server. It is an Ubuntu machine. Downloaded and installed Xampp, then setup WordPress with MySQL. Now tried uploading the plugin zip file, it throws some permission error, asking to fix permissions or use FTP. I thought of just chmod 777 recursively for the WordPress directory to fix this easily.

Ran the command, looks like it is hung. Terminated using Ctrl+C and then ran the same command. Again it is taking much time. It should not take so much time to recursively change the permission of just a WordPress directory. Thought something was wrong. Before I realized the damage is already done.

Looks like I ran the command

sudo chmod -R 777 /

instead of

sudo chmod -R 777 ./

Fuck, I missed a dot in the command and it is changing permissions of everything in my machine. Saw the System monitor, CPU usage spiked to 100%. I can't close or open any program. Force shutdown the machine using the power key. It didn't boot again. Recovery mode didn't help. Looks like there is no easy way to restore back from this damage. Most of the files I need are backed up in the cloud, still, need a few more personal files so that I can format and reinstall Ubuntu. Realised I have Windows in dual booting. Boot into Windows and used some ext4 reader to recover the files, formatted and reinstalled the OS. Took a few hours to get back to my previous setup.

Lesson Learned: Don't use sudo unnecessarily.
Double check the command while executing.
Running a wrong command with root permission can fuckup your entire machine.

Quick Plesk config question...

Been getting open_basedir() notices in the WordPress logs, and frankly it's flooding the log right now. Sample below:

[24-Feb-2019 07:05:19 UTC] PHP Warning: file_exists(): open_basedir restriction in effect. File(/var/www/vhosts/webspacedomain.com/SiteInstallDirectory/wp-content/db.php) is not within the allowed path(s): (/var/www/vhosts/webspacedomain.com/:/tmp/) in /var/www/vhosts/webspacedomain.com/SiteInstallDirectory/wp-includes/load.php on line 397

Checking the settings for open_basedir in the domain's PHP settings, it's currently set to the following default value:

{WEBSPACEROOT}{/}{:}{TMP}{/}

By my read, that **should** be granting permission to the directory. I just checked it against the setting on the dev server (which doesn't report this error), and it's configured in the same manner. Only difference between Dev environment and this one is that the one in Dev is in vhosts/webspacedomain.net/DEV instead of just vhosts/webspacedomain.net

Is there something I'm missing here?