So my actual job is being a nurse at the local hospital, with coding being just a hobby. However, the way some IT–Related things are treated here are just mind-blowing. Here are some examples:

Issue: Printer is not recognized by network anymore due to not being properly plugged in

Solution: Someone has to tell the house technician, if the house technician is currently not available, ask his assistant who only works part time and like twice a week. House technician took the printer (God knows why), came back 2 days later and plugged it back in.

Issue: Printer 1 of 2 on ICU has run out of ink and since all computers default to printer 1, nobody can print.

Solution: Call the house technician, blah blah, house technician comes, takes ink cartridge of printer 2 and puts it into printer 1.

Issue: Public WiFi is broken, can be connected to but internet access is missing. Probably config issue as a result of a recent blackout.

Solution: Buy a new router, spend 5 days configuring it and complain about how hard networking is.

Issue: Computer is broken, needs to be exchanged with a new one, but how do we transfer the data?

Solution: Instead of just keeping the old hard drive, make a 182GB backup, upload it to the main file server and then download it again on the new computer.

Issue: Nurse returns from vacation, forgot the password to her network account.

Solution: Call the technician who then proceeds to open a new account, copies all the files from the old one and tells her to pick an easier password this time. She chooses "121213".

Today my manager asked me about my research into using RabbitMQ as a backup in case Azure Service Bus ever goes down.

Me: "Good. The way we designed the framework, all we have to do is drop the DLLs into the directory, update the config, and the services will start using RabbitMQ."
Mgr: "Excellent. Probably should be looking into using RabbitMQ as a permanent replacement for Azure"
Me: "What? The whole reason we moved to Azure was to eliminate the problems with having an on prem service bus. Since we've switched, there has been zero downtime."
Mgr: "That's what VP-Joe is afraid of. If Azure ever goes down, he won't know how to explain Azure to the president as to why we're not taking orders or can't ship packages."
Me: "That makes no sense. What did VP-Joe tell the president when a database goes down or a server mis-configuration?"
Mgr: "President understands internal outages, its just the whole 'cloud' thing he doesn't understand."
Me: "Um..then VP-Joe needs to explain it to him?"
Mgr: "The decision has already been made. Are you on board? Lets look at this move as a cost savings."
Me: "You mean the $10 a month? How much hardware will we need to support RabbitMQ?"
Mgr: "Yea, nobody probably thought of that."
Me: "I'm on board with whatever decision, but I'd like a little more than VP-Joe being afraid of the president."
Mgr: "I'm sure its not being afraid."
Me: "..."
Mgr: "OK, lets wait and see if VP-Joe forgets about this and moves on to something new."

Around 27 hours at new customer location.

They had a server failure due to incompetence.

They had fired their own IT guy and called us 6 months later because the server stopped responding.

First diagnostic. 2 drives are dead in a raid 5 with one hot spare. Raid controller then proved to be broken once the disks was replaced.

Waiting for new raid controller and installing.

Backup non existing, no one changed dat tape during the 6 months without IT. The tape was just a transparent plastic band, no media left.

Raid config is stored in static ram on controller, no backup!

Several hours in tech support to find out how to rebuild raid config from existing disks.

Proves to be impossible to rebuild raid set due to some checksum failures.

More hours with support to enable some diagnostic read only mode to mirror low level content to external drive.

Then many more hours to copy parts of the tree until it gets an error, restart after that and go on.

In the end we got around 70% back.

During this time I manage to be in contact with the raid manufacturers all support centers, one in europe, one in the us and one in Taiwan, switching each time one if them closed for the night.

The customer later declined a steady support contract due to us being to expensive ;)

Some just don’t want to learn.

My Sunday Morning until afternoon. FML. So I was experiencing nightly reboots of my home server for three days now. Always at 3:12am strange thing. Sunday morning (10am ca) I thought I'd investigate because the reboots affected my backups as well. All the logs and the security mails said was that some processes received signal 11. Strange. Checked the periodics tasks and executed every task manually. Nothing special. Strange. Checked smart status for all disks. Two disks where having CRC errors. Not many but a couple. Oh well. Changing sata cables again 🙄. But those CRC errors cannot be the reason for the reboots at precisely the same time each night. I noticed that all my zpools got scrubbed except my root-pool which hasn't been scrubbed since the error first occured. Well, let's do it by hand: zpool scrub zroot....Freeze. dafuq. Walked over to the server and resetted. Waited 10 minutes. System not up yet. Fuuu...that was when I first guessed that Sunday won't be that sunny after all. Connected monitor. Reset. Black screen?!?! Disconnected all disks aso. Reset. Black screen. Oh c'moooon! CMOS reset. Black screen. Sigh. CMOS reset with a 5 minute battery removal. And new sata cable just in cable. Yes, boots again. Mood lightened... Now the system segfaults when importing zroot. Good damnit. Pulled out the FreeBSD bootstick. zpool import -R /tmp zroot...segfault. reboot. Read-only zroot import. Manually triggering checksum test with the zdb command. "Invalid blckptr type". Deep breath now. Destroyed pool, recreated it. Zfs send/recv from backup. Some more config. Reboot. Boots yeah ... Doesn't find files??? Reboot. Other error? Undefined symbols???? Now I need another coffee. Maybe I did something wrong during recovery? Not very likely but let's do it again...recover-recover. different but same horrible errors. What in the name...? Pulled out a really old disk. Put it in, boots fine. So it must be the disks. Walked around the house and searched for some new disks for a new 2 disk zfs root mirror to replace the obviously broken disks. Found some new ones even. Recovery boot, minimal FreeBSD Install for bootloader aso. Deleted and recreated zroot, zfs send/recv from backup. Set bootfs attribute, reboot........
It works again. Fuckit, now it is 6pm, I still haven't showered. Put both disks through extensive tests and checked every single block. These disks aren't faulty. But for some reason they froze my system in a way so that I had to reset my BIOS and they had really low level data errors....? I Wonder if those disks have a firmware problem? So that was most of my Sunday. Nice, isn't it? But hey: calm sea won't make a good sailor, right?

I just found a vulnerability in my companies software.
Anyone who can edit a specific config file could implant some SQL there, which would later be executed by another (unknowing) user from within the software.

The software in question is B2B and has a server-client model, but with the client directly connecting to the database for most operations - but what you can do should be regulated by the software. With this cute little exploit I managed to drop a table from my test environment - or worse: I could manipulate data, so when you realize it it's too late to simply restore a DB backup because there might have been small changes for who knows how long. If someone was to use this maliciously the damages could be easily several million Euros for some of our customers (think about a few hundred thousand orders per day being deleted/changed).

It could also potentially be used for data exfiltration by changing protection flags, though if we're talking industry espionage they would probably find other ways and exploit the OS or DB directly, given that this attack requires specific knowledge of the software. Also we don't promise to safely store your crabby patty recipe (or other super secret secrets).

The good thing is that an attack would only possible for someone with both write access to that file and insider knowledge (though that can be gained by user of the software fairly easily with some knowledge of SQL).

Well, so much for logging off early on Friday.

This Christmas I transitioned into a new job. At the old job I was the only kubernetes-guy, so since they no longer have any developers who are confident working with that, they decided to go with LAMP-stack.

The data from dev-kubernetes-server was backed up by some guy and moved to an offsite-server, or so they told me. Turns out, he had backed up the kubectl-config-file, and not the databases. Now everything is wiped. Sure glad we still have that config-file!

Of course, since that was only our dev-server, there was nothing too important there, except for all the documentation. The only other backup? On my laptop, which I turned in to them, and is now wiped and used by one of the sales-guys.

Now I’m being called in at least twice a day, since I was their goto-guy for almost anything backend-related. Feels great, after they spent a couple of months attempting to rewrite everything in pure PHP (with a strict no-dependency policy for some reason). Fml.

Well, if your backups don't include /etc/nginx, make them include that directory. Mine's gone and I'll have to configure that again Friday evening. Woops.

<<prev. #wk235 advices>>

~ Study the Error log deeply, Google each line if needed. Don't give up.
~ Learn by doing. Don't just read/watch.
~ Practice breaking down the problem statement first in different components and hierarchies. Don't jump into coding right away.
~ Write some, review some. Don't put off review for later.
~ Even if you don't exactly follow the best security practices - always ensure that your program is safe for use. Especially for user-inputs, etc, pay attention.
~ Never distribute code with passwords/keys written in it.
~ Don't hard code stuff, use Config file, environment variables, etc.
~ Try to automate repetitive stuff like build and deploy etc
~ Save and backup you code.
~ No one knows everything, also, today's knowledge gets outdated tomorrow. Continuous learning is synonymous with this field.

<<next #wk235 advices>>

Today was a painful day when I realized that I need to backup my nginx configs like I backup my actual data. 20 minutes of downtime turned into an afternoon when I accidentally deleted the nginx config backups on my server. It's been... let's say fun.

The only way I can edit Puppet config files is by git. And the only way I can git pull/push/commit/etc is generating a ssh key with a private key and give my public key to my supervisor to the git server (wherever that may be).

Because I'm on Windows 10 and screwed up my installers, I completely forgot to backup my ssh keys before resetting it. FML

i need an adult. I know noone who would understand my worries, so you guys need to be it.

i have a nextcloud running on my raspberry pi. performance is horrible, dont ask, but it works.

i mostly use it to backup the photos of my phone sd card every night when my phone charges. Internally this works good. If i am elseplace it wont for obvious reasons.

In my youthful joy of doom i opened port 443 and forward it to my raspi. I get internet via cable and my ip is pretty much static (it was the same for 10 months). So external access is provided.

Now i thought, its stupid that i cannot sign an ssl certificate cause i dont have a domain. Lets buy domain. But before i do that i did some try runs with duckdns to test the principle.

Some back and forth, it works now. Pretty god, i could even make a cron job on the raspbi to renew (that should work right?). Only problem. randoname.duckdns.org doesnt work internally. Or should not at least.

So i googled a bit and it turns out that my router (a cable fritz!box i bought myself) can be a local network dns. Or cannot. Regardless what i try, it doesnt accept the changed config file.

Now the problem.

It works anyway. randoname.duckdns.org points to my external "static" ip and resolves to that from my internal network..so it works on my phone or laptop. if i traceroute the thing it goes via two hops out and finishes in less than 1ms.

Now to the problem:
I have no fokkin clue why. The expected behaviour would be that it shouldnt work. If i do what i intended todo on pc in the hosts file tracert works correctly, directly pointing to the internal ip.

What i cannot figure out, is it the fritz!box being smart? Is it my ISP being smart?

Reason to rant: i have absolutly NOONE to ask, i know not a single person who would even understand what troubles me. I want to learn, i want to know WHY not just some mindless russian patchwork of "if it works its good enough".

thats depressing.

Not best practice whatsoever because the box was most likely owned, but...

SSH kept defaulting back to port 22 when it wasn't supposed to. So, wrote a cronjob that checked diff between SSH config and backed up SSH config. If different, reload backup. Didn't get locked out again.

Box has of course been replaced.

!rant
There are so many options to pimp my linux system. Is there any point in time, where I will be finished customizing/prettifying my linux? I highly doubt.
BTW.. what is the best way to backup the whole config? (color schemes, themes, gnome extensions)

lol

found an old config file on my external drive for all my torrent files. awyisss. my SSD died out of nowhere last year and I thought I lost all my torrent configs! I had hundreds of TV series and stuff and I kept track which ones I watched and didn't in the client. so when the SSD died I lost all my knowledge of my progress. but I found this config file just now and imported it. omg booyah. I think I got one show since this backup. godsend

decided to export settings again and it said I can set it on a schedule. go to the scheduler tab in the settings and I have no clue what's going on. nothing about exporting settings, it just has schedule configurations that seem to conflict. then I realized. the main client maybe has a schedule tab icon. bam am right. so in settings I turn on scheduler and then the main app gives me access to the scheduler tab and if I go there I can click "add" and then I can schedule regular config backups

bruh this UI is so jank. but it actually is impressive. because. while I have experience in designing websites, when I played around with making a GUI in rust, which would be native, I have absolutely no clue how to make an app on this tech. now I'm looking at this complex torrent client with its bazillion features in absolute awe.

*takes notes*
I can only aspire to be so genius as to allow you to turn on and remove tabs in the settings menu. now it makes sense why all the windows always had awkwardly sized panels. this genius man.

however did he come up with that?! ALL THESE NEW STANDARDS

honestly somehow it never occurred to me that native apps and web apps would have totally different ergonomics. I feel like I've found some kind of lost art from the ancient world. aaaaaaa