My Friend: Dude our Linux Server is not working anymore!

Me: What? What did you do?

My friend: Nothing I swear!

Me: But you were last on it?

My friend: Yes. I just wanted to run a bash file and needed to give it permissions.

Me : WHAT DID YOU ENTER???!

My Friend: Chill man, just this command I found on the internet
chmod -R 600 /
chown -R root:root /

Me: WHY ARE YOU EVEN IN ROOT AND GOD DAMMIT WHY ARE YOU EVEN USING SOME RANDOM COMMAND FROM THE INTERNET. YOU KNOW YOU SHOULD NOT DO THIS OR JUST ASK!

My friend: Ok I did something wrong, how can I fix it?

Me: Did you make a backup or rsync of the server?

My friend: No. I just wanted to run this file.

Me: You holocausted the server. FUCK MY LIFE

Things I've learned throughout my 5 - 6 years as a programmer.
- StackOverflow is full of assholes.
- CMS's are for weaklings.
- The best feeling about waking up in the morning is figuring out how to solve that error in your code.
- You no longer think about normal people things. Your mind is full of code.
- You're practically a computer.
- ALWAYS backup and save your stuff or you WILL regret it. Enable autosave if possible.
- RIP your social life (if your friends don't know squat about programming)
- Darkness is better.
- Being a programmer is amazing.

Mac: Suddenly turns off
Me: Fuck my code..F***
Mac: No response at all
Me: reset SMC etc etc
Mac: I am dead (no battery detection, dies after 10 min on power adaptor)
Me: Skips a heart beat..(Git, oh yea git)

**Takes Mac to store, After diagnosis**

Apple Freaking Genius (AG): Your Mac has a mother board problem it needs to be replaced.
Me: Hmm what is the problem exactly??
AG: Issues in logic board and some other components.
Me: How much?
AG: Out of warranty so $$$ (60% of original amount)
Me: (wtf?) Really
AG: It's entire motherboard replacement .. bla bla

**Bring it home > open > everything seems ok on multimeter as per circuit diagram > finally finds a voltage drop that is not consistent > minute short circuit > remove > check further > nothing else > reassemble > hit power button > starts fine > freaking battery detected > works fine**

0 $ repair

Fixes two more devices @ 0 $ in friend circle

Builds a raspberry pi backup laptop with 3d printed body..Ubuntu.. you know can't live without a computer

#ThugLife #Engineer

CEO: You need to use git.
Designer: No

CEO: You can backup stuff.
Designer: No

CEO: You don't have to save copies!
Designer: No

CEO: You will use a really cool command line!
Designer: No

CEO: It has unlimited storage
Designer: So when we start?

So my Girlfriend bought a new iPhone at Verizon today. Cool story, I know, but here's where it's gone from there.

Firstly her debit wouldn't run as credit, so we used mine but that's the least of it (but began it).

So she has 16,000 photos... Alot, sure, but not the issue. Obviously with that amount of data she wasn't about to reasonably use iCloud to back it up (understandable only by me) so she was confronted both by me and the Verizon employee about this issue to where we both (the Verizon employee and I) agreed that an iTunes backup/restore was the only way to preserve her data. She was confused. No worry, told her I had it handled and the Verizon employee agreed. Great. Yet we get home and begin the process. My girlfriend was not on the latest iOS (understandable given the battery scenario and she was on an iPhone 6) and this was ridiculous to her because she had to update in order to do the iTunes back up. Whatever, I brushed it off. Her phone was updated, and backed up... Which took a while but we are talking 30gb (of which she had no understanding of how much that was). After the back up we discovered her new phone wasn't working due to a bad sim, great, no problem we have the old one... But oh no. "I don't want that shitty old sim" she said. Uhmm what... I say, and say let me get an earring (to switch the Sims) and she gave one to me and as soon as I went to pop the tray, she had a fucking heart attack as if I was demolishing her phone. I talk her down, get it switched, get the phone to restore (slow process as she's complaining... 30gb mind you) and it works. She goes to bed. Comes back, texts aren't working. I say imessages or texts (now she has no idea) I troubleshoot, seems nothing's working, and that's okay Verizon must of reinstated the new sim and deactivated the old (fine). I switch them and it works. She proceeds to berate me about the SIM cards because she didn't want the 'old shitty one' (the one that got us to the place of a functioning phone).

Now everything works and she claims a Genius bar employee would of done this in minutes.

I (obviously) lose my shit, now I'm sleeping on the couch.

Im an IT professional / programmer..... this shit really ticked me off.

A friend of mine: Look at this nice app I made 😎
*shows me a normal looking app with tabs and different views"
Me: Hmmm… looks good.
Friend: Do you know how I made the navigation between views? 😏
Me: I'm afraid to ask…
Friend: Since I didn't know how to use the piece of code I found on internet, I made multiple views one above another with visibility set to false, and during the navigation I just change the visibility. 😁
Me: WTF? Man… this isn't something you should be proud of… it shouldn't be done.
Friend: Do you also want to know how I backup the code?
Me: No…
Friend: I send it to myself via email.
Me: FUCK!
Friend: I also never use while loops, I prefer to use for instead with a break inside. 😊
*blocked*

Facebook publicly announced that it won't build a backdoor into its services for the intelligence agencies as for the latest requests to weaken/remove the encryption.

I can only imagine the intelligence agencies going like this now:

NSA director: Alright, as expected they said no so they won't have more damage to their public image, lets go for plan A 2.0!
NSA employee: Aaaand that is?
NSA director: Serve them a FISA court order requiring them to do this shit anyways but also serve a gag order so they can't tell legally.
NSA employee: Ahh, fair enough, I'll get that rolling. By the way, how did we do this with WhatsApp's encryption again?
NSA director: Oh that one was simple. There's a backup function which nearly everyone uses on either Android/iOS which does plaintext backups to Google Drive/iCloud.
NSA employee: Oh, okay. How do we access that data again?
NSA director: PRISM/XKeyScore!
NSA employee: Right, but then still the issue of how we even collect the encrypted messages from Facebo...
NSA director: PRISM/XKeyScore as well, don't worry about that.
NSA employee: But, how'd we justify this....?
NSA director: We probably never have to as these programs operate outside of the public view but otherwise just call terrorism/pedophelia... BAM, done.
NSA employee: Gotya, let's put this into motion!

And here comes the last part of my story so far.
After deploying the domain, configuring PCs, configuring the server, configuring the switch, installing software, checking that the correct settings have been applied, configuring MS Outlook (don't ask) and giving each and every user a d e t a i l e d tutorial on using the PC like a modern human and not as a Homo Erectus, I had to lock my door, put down my phone and disconnect the ship's announcement system's speaker in my room. The reasons?
- No one could use USB storage media, or any storage media. As per security policy I emailed and told them about.
- No one could use the ship's computers to connect to the internet. Again, as per policy.
- No one had any games on their Windows 10 Pro machines. As per policy.
- Everyone had to use a 10-character password, valid for 3 months, with certain restrictions. As per policy.
For reasons mentioned above, I had to (almost) blackmail the CO to draft an order enforcing those policies in writing (I know it's standard procedure for you, but for the military where I am it was a truly alien experience). Also, because I never trusted the users to actually backup their data locally, I had UrBackup clone their entire home folder, and a scheduled task execute a script storing them to the old online drive. Soon it became apparent why: (for every sysadmin this is routine, but this was my first experience)
- People kept deleting their files, whining to me to restore them
- People kept getting locked out because they kept entering their password WRONG for FIVE times IN a ROW because THEY had FORGOTTEN the CAPS lock KEY on. Had to enter three or four times during weekend for that.
- People kept whining about the no-USB policy, despite offering e-mail and shared folders.
The final straw was the updates. The CO insisted that I set the updates to manual because some PCs must not restart on their own. The problem is, some users barely ever checked. One particular user, when I asked him to check and do the updates, claimed he did that yesterday. Meanwhile, on the WSUS console: PC inactive for over 90 days.
I blocked the ship's phone when I got reassigned.
Phiew, finally I got all those off my chest! Thanks, guys. All of the rants so far remind me of one quote from Dave Barry:

That moment when your boss is introducing the new guy and you are silently making a list of all the git repos you need to back up before he makes it to his desk.

1. I join a company.

2. I get deeply involved in "how to run the company", and get nice compliments from both coworkers & management about my skills in conveying startup/scaleup advice & necessities to upper management.

3. With my ego inflated through all the sweet talk, I think "ah, what the hell, let's do this again", and I accept a Lead/CTO promotion. I have to join board meetings, write reports on quarterly plans and progress.

4. I get unhappy/stressed/burned-out because I really just want to be a developer, not a manager/executive.

5. Upper management understands, I give up my lead position, lock myself back into my coding cave.

6. I get annoyed because the requirements I receive become more and more disconnected from reality, half of the teams seem to have decided to stop using agile/scrum, the testing pipeline breaks all the time, I get an updated labor contract from HR by mail which smells like charred flesh, etc

7. The annoyances become too much to do ANY work. I yell at the other devs outside of the entrance of my cave. There is no answer, only a few painful moans and sighs.

8. I emerge from my cave. The city has turned into a desolate wasteland. The office is a burning ruin, the air sharp and heavy with black soot. Disemboweled corpses of developers litter the poisoned soil.

Product Managers dressed in stained ripped suits scream at each other while they try to reinforce concrete barricades with scotch tape and post-its. *THUMP* Something enormous is trying to break through. "Thank God, bittersweet, you're still alive! The stakeholders! They have mutated! We couldn't meet the promised deadlines! We've lost the whole mobile app department, and that kid there is the last of the backenders and he's only an intern! You're here to save us, right? RIGHT?".

In the corner, between the overflowing coffee machine and a withered cactus, a young boy has collapsed onto the floor. His face is covered in moldy coffee grounds, clasping on to his closed macbook for dear life, wide-open eyes staring into the void, mumbling: "didn't backup the database, and It's all gone" over and over.

A severely dented black Tesla with a dragging loose bumper breaks through the dried up vertical herb garden and the smoothiebar, and comes to a halt against the beanbags in a big cloud of styrofoam balls.

The CEO limps out, leaking blood all over the upholstery. He yells to the COO: "The datacenter is completely flooded with sewage! I saved the backup tapes though", holding a large nest of tangled black magnetic tape mixed with clumps of mud above his head.

9. I collect my outstanding salary and sell any rewarded options/shares for a low dumping price, take a 5 month holiday, and ask a recruiter about opportunities in a different city.

The website i made has been hacked today.

Stored in their server.

They didnt give me an access for it.

The user account in the cms i used for updating content while building the website was revoked when the website is completed.

Now they ask me for the latest backup.

I have no backup because how the hell i do a backup when i got no access to the cpanel.

The only backup is the zip file for initial uploading into their server and the contents were added after the website is on their server.

That goddamn IT guy who wont give me any access for “securty sake” is calling me furiously asking for the backup and how to set up the stuffs from the beginning.

I thought he was the one who know his shit but i was wrong.

Fuck me?

No.

Fuck you.

But i still responding to him telling him step by step how to do shit with some swearing and sarcasm.

ALWAYS BACKUP YOUR SHITS, MATE

PROBLEM: A tickets' company came to us last autumn. They said they have severe performance problems and asked us to help.
SOLUTION: covid and quarantines. All events have been ceased, noone's buying any tickets any more. Performance problems are no more. FIXED.

PROBLEM: Another company came to us recently. They said they have severe performance problems with their huge databases and asked us to help.
SOLUTION: a few days of heavy rain and their datacenter was flooded. along with the backup servers. No more data, no more performance problems with large databases. FIXED

Solving problems genie style!

Who's next?

I think I've shown in my past rants and comments that I'm pretty experienced. Looking back though, I was really fucking stupid. Since I haven't posted a rant yet on the weekly topics, I figure I would share this humbling little gem.

Way back in the ancient era known as 2009, I was working my first desk job as a "web designer". Apparently the owner of this company didn't know the difference between "designer", which I'm not, and "developer", which I am, nor the responsibilities of each role.

It was a shitty job paying $12/hour. It was such a nightmare to work at. I guess the silver lining is that this company now no longer exists as it was because of my mistake, but it was definitely a learning experience I hold in high regard even today. Okay, enough filler...

I was told to wipe the Dev server in order to start fresh and set up an entirely new distro of Linux. I was to swap out the drives with whatever was available from the non-production machines, set up the RAID 5 array and route it through the router and firewall, as we needed to bring this Dev server online to allow clients to monitor the work. I had no idea what any of this meant, but I was expected to learn it that day because the next day I would be commencing with the task.

Astonishingly, I managed to set up the server and everything worked great! I got a pat on the back and the boss offered me a 4 day weekend with pay to get some R&R. I decided to take the time to go camping. I let him know I would be out of town and possibly unreachable because of cell service, to which he said no problem.

Tuesday afternoon I walked into work and noticed two of the field techs messing with the Dev server I built. One was holding a drive while the other was holding a clipboard. I was immediately called into the boss's office.

He told me the drives on the production server failed during the weekend, resulting in the loss of the data. He then asked me where I got the drives from for the Dev server upgrade. I told him that they came from one of the inactive systems on the shelf. What he told me next through the deafening screams rendered me speechless.

I had gutted the drives from our backup server that was just set up the week prior. Every Friday at midnight, it would turn on through a remote power switch on a schedule, then the system would boot and proceed to copy over the production server's files into an archive for that night and shutdown when it completed. Well, that last Friday night/Saturday morning, the machine kicked on, but guess what didn't happen? The files weren't copied. Not only were they not copied, but the existing files that got backed up previously we're gone. Why? Because I wiped those drives when I put them into the Dev server.

I would up quitting because the conversation was very hostile and I couldn't deal with it. The next week, I was served with a suit for damages to this company. Long story short, the employer was found in the wrong from emails I saved of him giving me the task and not once stating that machine was excluded in the inactive machines I could salvage drives from. The company sued me because they were being sued by a client, whose entire company presence was hosted by us and we lost the data. In total just shy of 1TB of data was lost, all because of my mistake. The company filed for bankruptcy as a result of the lawsuit against them and someone bought the company name and location, putting my boss and its employees out of a job.

If there's one lesson I have learned that I take with the utmost respect to even this day, it's this: Know your infrastructure front to back before you change it, especially when it comes to data.

Storytime!

This customer comes in and practically throws a computer on the counter.

Customer: This computer isn't working. I've ran the diagnostics and it says it's software. *places a dvd case with a 32 bit Windows 7 disk in it on the counter* It had Windows 10 on it, but I want Windows 7 on it.

Me: Well, you may have issues with the drivers if you put Windows 7 on it--

Customer: I don't care, I just want Windows 7.

Me: You SHOULD care. That means no wifi, no display, no mouse... Windows 7 doesn't like Windows 10 hardware.

Customer: Then... check to see Windows 7 compatibility!

Me: Alright.... *makes notes to check for Windows 7 compatibility*

Me: So has this Windows 7 been used before?

Customer: Yes, it has.

Me: On how many computers?

Customer: I've installed it on two computers and it works just fine.

Me: That's weird because Windows license keys are for one computer only. Are both of them connected to the internet?

Customer: Yes.

Me: Well, okay then... *finishes up ticket*

Customer: I work in this field and I just don't understand why they don't come with the disks anymore. How much is a Windows 10 disk?

Me: *gives price*

Customer: And do you have any?

Me: Let me check *I go to where they are, find some and come back out*

Me: Unfortunately we're out at the moment and would have to special order some back in.

Customer: OK. So then how much to fix this computer?

Me: *price of installing Windows and backing up data*

Customer: That's halfway to the price of a new one of these!

Me: Well yes, an HP at Walmart... But you do have that option if you want to take it.

Customer: Well, why does it cost that much?

Me: Well, it's $labor1 to install Windows, $labor2 to do some basic setup and drivers, and $labor3 to backup and restore data.

Customer: Oh, well I don't want data.

Me: Okay, well then it would be $total - $labor3

Customer: ...Okay, fine

Me: *updates the ticket*

When she finally left I put it on the bench and the first message said "SMART ERROR." I then did 4 different tests that said "lol, the hard drive is failing."

If you "worked in this field," you would know that a SMART error is hard drive related.

If you worked in this field, you would know that Windows is only a 1PC license, so why are you lying about installing it with no issues on other computers?

If you worked in this field, you would know you would want a 64bit Windows on your computer.

If you worked in this field, you would know how to find a Windows 10 installation media online.

If you worked in this field, you would know that HPs are not good computers to get.

IF YOU FUCKING WORKED IN THIS FIELD YOU WOULDN'T BE SUCH A FUCKING CUNT.

There's one kind of ticket I fucking love. Wait, YOU LOVE TICKETS?! No, except for one.

Tickets where people ask us to restore backups.

Why do I love them?

Because the only fucking thing I need to do is reply with a link to a helpdesk item on our website because backups aren't something within our support range because it's easy for customers to do.

So whenever a ticket about restoring a backup comes in, it's as easy as opening the ticket, pasting the link with "I'd like to refer you to the following resource: " and pressing fucking "reply"

😄

Me: I'm unable to connect through VPN

IT guy at my office: I'll format your system and reinstall OS

Me: heck no!! There should be other effective solution.

IT guy: Yes, I can take backup of your files and then reinstall OS.

Me: Just get out!! F*ckin ididot!!

At my previous job, the person in charge of the Phabricator server didn't have a backup system in place. I yelled at him until he implemented one.

He had the server perform backups to the same drive. I yelled at him again, to no avail.

Well, after awhile the hard drive started failing, and it would only boot intermittently. After a lot of effort, he was able to salvage part of the backup data, but no more, meaning we lost a lot of bug reports and feedback, and developer tickets. We were able recover all of the older lost tickets from a previous server, so overall the loss was pretty small.

But I think he learned his lesson.
He definitely learned to listen.

[This makes me sound really bad at first, please read the whole thing]

Back when I first started freelancing I worked for a client who ran a game server hosting company. My job was to improve their system for updating game servers. This was one of my first clients and I didn't dare to question the fact that he was getting me to work on the production environment as they didn't have a development one setup. I came to regret that decision when out of no where during the first test, files just start deleting. I panicked as one would and tried to stop the webserver it was running on but oh no, he hasn't given me access to any of that. I thought well shit, I might as well see where I fucked up since it was midnight for him and I wasn't able to get a hold of him. I looked at every single line hundreds of times trying to see why it would have started deleting files. I found no cause. Exhausted, (This was 6am by this point) I pretty much passed out. I woke up around 5 hours later with my face on my keyboard (I know you've all done that) only to see a good 30 messages from the client screaming at me. It turns out that during that time every single client's game server had been deleted. Before responding and begging for forgiveness, I decided to take another crack at finding the root of the problem. It wasn't my fault. I had found the cause! It turns out a previous programmer had a script that would run "rm -rf" + (insert file name here) on the old server files, only he had fucked up the line and it would run "rm -rf /". I have never felt more relieved in my life. This script had been disabled by the original programmer but the client had set it to run again so that I could remake the system. Now, I was never told about this specific script as it was for a game they didn't host anymore.
I realise this is getting very long so I'll speed it up a bit.
He didn't want to take the blame and said I added the code and it was all my fault. He told me I could be on live chat support for 3 months at his company or pay $10,000. Out of all of this I had at least made sure to document what I was doing and backup every single file before I touched them which managed to save my ass when it came to him threatening legal action. I showed him my proof which resulted in him trying to guilt trip me to work for him for free as he had lost about 80% of his clients. By this point I had been abused constantly for 4 weeks by this son of a bitch. As I was underage he had said that if we went to court he'd take my parents house and make them live on the street. So how does one respond? A simple "Fuck off you cunt" and a block.

That was over 8 years ago and I haven't heard from him since.

If you've made it this far, congrats, you deserve a cookie!

Once I had to clone a repo and it was taking too long...

Went to gitlab to find out in the most wtf way that from 1mb it was 600mb+ now...

One of our new juniors pushed 600mb+ of a database backup to gitlab...

I came to her with a smile and asked in a jokingly manner (after cursing her for about half an hour in my head):

"lol, did you really not notice it took a fucking long time to upload it?"

The fucker was ashamed but just said : no, I think I pushed it and went home.

I constantly reminded her of it for the couple months...

Never done it again :)

What an awful day :(

The server where I host my 4 clients websites crashed.

Unable to reboot from the console.

I contact the support. 15 minutes later: "we'll look at this"

No news for 1 week despite my messages.

Then... 1st ticket escalation... 2nd ticket escalation... 3rd ticket escalation...

Answer: "Sorry, your server is down and cannot be repaired."

Fuck.

I ask "is there any way to get my data back?". Answer: "No, because we would shutdown the whole bay and all our clients would be impacted".

Fuck.

I subscribe to another server, at another provider.

I look at my backups... shit, the last one is 4 month ago!!

I restore the first website: OK

I restore the second website: OK

I restore the third website: My new server is "too recent" and not compatible. with this old Wordpress. Fuck! I'll look at this later...

I restore the fourth website: database is empty!! What??? I look at the SQL backup for this site... it failed...

I lost ALL my 4th client data!!!

I'm sooooo piece of crap!

Fuck the memes.

Fuck the framework battles.

Fuck the language battles.

Fuck the titles.

Anybody who has been in this field long enough knows that it doesn't matter if your linus fucking torvalds, there is no human who has lived or ever will live that simultaneously understands, knows, and remembers how to implement, in multiple languages, the following:

- jest mocks for complex React components (partial mocks, full mocks, no mocks at all!)
- token cancellation for asynchronous Tasks in C#
- fullstack CRUD, REST, and websocket communication (throw in gRPC for bonus points)
- database query optimization, seeding, and design
- nginx routing, https redirection
- build automation with full test coverage and environment consideration
- docker container versioning, restoration, and cleanup
- internationalization on both the front AND backends
- secret storage, security audits
- package management, maintenence, and deprecation reviews
- integrating with dozens of APIs
- fucking how to center a div

and that's a _comically_ incomplete list; barely scratches the surface of the full range of what a dev can encounter in a given day of writing software

have many of us probably done one or even all of these at different times? surely.

but does that mean we are supposed to draw that up at a moment's notice some cookie-cutter solution like a fucking robot and spit out an answer on a fax sheet?

recruiters, if you read this site (perhaps only the good ones do anyway so its wasted oxygen), just know that whoever you hire its literally the luck of the draw of how well they perform during the interview. sure, perhaps some perform better, but you can never know how good someone is until they literally start working at your org, so... have fun with that.

Oh and I almost forgot, again for you recruiters, on top of that list which you probably won't ever understand for the entirety of your lives, you can also add writing documentation, backup scripts, and orchestrating / administrating fucking JIRA or actually any somewhat technical dashboard like a CMS or website, because once again, the devs are the only truly competent ones - and i don't even mean in a technical sense, i mean in a HUMAN sense of GETTING SHIT DONE IN GENERAL.

There's literally 2 types of people in the world: those who sit around drawing flow charts and talking on the phone all day, and those WHO LITERALLY FUCKING BUILD THE WORLD

why don't i just run the whole fucking company at this point? you guys are "celebrating" that you made literally $5 dollars from a single customer and i'm just sitting here coding 12 hours a day like all is fine and well

i'm so ANGRY its always the same no matter where i go, non-technical people have just no clue, even when you implore them how long things take, they just nod and smile and say "we'll do it the MVP way". sure, fine, you can do that like 2 or 3 times, but not for 6 fucking months until you have a stack of "MVPs" that come toppling down like the garbage they are.

How do expect to keep the "momentum" of your customers and sales (I hope you can hear the hatred of each of these market words as I type them) if the entire system is glued together with ducktape because YOU wanted to expedite the feature by doing it the EASY way instead of the RIGHT way. god, just forget it, nobody is going to listen anyway, its like the 5th time a row in my life

we NEED tests!
we NEED to know our code coverage!
we NEED to design our system to handle large amounts of traffic!
we NEED detailed logging!
we NEED to start building an exception database!

BILBO BAGGINS! I'm not trying to hurt you! I'm trying to help you!

Don't really know what this rant was, I'm just raging and all over the place at the universe. I'm going to bed.

So, basically i am getting desperate, and i'm also angry... and want to cry, and i feel a failure.
My biggest error in this story is "believing"

First of all, I'm a starting entrepreneur as freelancer, started of 6 months ago, back then it all seemed bright, i had my first customer, they believed me, got a second assignment for that same customer, fulfilled the task in brilliance, and was ready to move on to a bigger customer base. Here's where shit goes wrong.

Working with another office that outsources people to various (goverment) contractors, Had a meeting with them, we would cooperate very soon. This was january.

PM: "We've got a gov Dept as a customer that wants to do project XYZ and starting in february, requirements are yadiyadiyadi, you up for it?
Me:"Sure, send me the specs, and timeframe and i'll apply with my offer)"
Project is about a bit backup system migration, lots of fancy shmancy tech used, 2 datacenters setup... a big project that can take up several months...
- sends offer- received an acceptance on the offer

PM " Great, we'll start end of february"
Me: "Ok, end of february is a go then, looking forward to cooperate"

FF to near end of february, receive msg from PM: "Project XYZ delayed until half if march"
Me: "Okay... what is the delay?"
PM:"Govt bureaucracy"
Me " Ok, let's keep in touch"

Near Half of March
PM " Project delayed again".
Me: " Okay... what's the reason?"
PM: "they have decided to take a different approach, and want 3 datacenters now"
Me: "That will change the offer i made for you, can you send me the specs?"
PM: "No, because they havent decided on the techs used, expect end of march"
Me: "Okay...but once you got the specs,s end them to me"

End of March
Pm: " Hy NeatNerdPrime, we still havent got any specs yet... they still forgot to make the shopping list"
Me: "(-_-) ... I thought they already had that covered"
PM:" No, new Govt budget cuts and lots of changes, basically, they still don't know what they really want. But we're just -delayed- , not -cancelled-"
Me: "Okay... i hope we get started soon"
PM: "expect launch beginning of april"
I was not really satisfied with this explanation.... but ok

Beginning of april, at april's folls day
PM "Project delayed again"
Me: " This is a joke right?"
PM: "Sadly, no, they delayed the project since they don't really know what they actually want, we are trying to give them the proposed solution... but they still need approval, and still need to set up shopping list"
Me:"when do we start then?"
PM: " I was told 17th of april'
Me:" I really hope it gets through, i had to decline some proposals for work for months due to this, this ain't funny"
PM: "I know, i'll make up for it"

15th of april...
PM " Project delayed again"
Me right now almost losing my shit
"why this time??? I thought you said 17th of april real launch!"
PM: "they fired their PM, need to hire a new one, delayed until May/June"
Me: "I've set aside some other assignments just because you said you were going to launch at those dates... This is really pushing my limits, can you give me SOME assurances?"
PM: "5th of june would be official go" Me: " Okay, i'm noting 5th of june in my agenda, let's get this through!"

And now i get a message...stating that the date of 5th of june , is for another project, at the same goct dept, but totally different project

WHAT THE FUCKING SHIT I'VE SPENT MONTHS WAITING FOR A PROJECT THAT WAS PRACTICALLY ALL SET AND GO AND EVERY SINGLE FUCKING TIME IT GETS DELAYED, EVERY FUCKING TIME I THINK "you know, i cannot accept these offers since that Govt project will start soon, i cannot do 2 assignments at once especially when they require me to work at govt office at the capital..." AND EVERY FUCKING TIME IT GETS DELAYED!!!

I feel at a loss now..... i've done i think the most horrible thing you can do as a independent and that's not accepting another assignment just because there was a concrete promise for a govt contract...

Almost dried up, nothing much left, had to do some spending because of a move to another apartment... I'm feeling really down, and angry...and down, but mostly angry, for not accepting those offers in the meantime...

Never in my life I was scared as today.
I recently left a big company to work for a small one as the first internal developer.
Had a small issue in the production server. The fix was easy, just remove a single table entry. And... *drum roll*... I forgot to add a where clause. All orders were lost.
No idea if we had backups or anything, I quickly called the one other IT dude in the company.
He had no clue where are the backups and how to find them.
Having some experience with Nmap, I quickly scanned our network and found a Nas device.
There was a backup, whole VHD backup. 300GB of it, the download speed is around 512kb/s. No way I can fix it before management finds out, but then an idea came to mind. Old glorious 7zip. Managed to extract only the database files, sent them to the server and quickly swapped them. Everything was fine... The manager connected 5 minutes later. Scariest 45 minutes of my life...

Today my manager asked me about my research into using RabbitMQ as a backup in case Azure Service Bus ever goes down.

Me: "Good. The way we designed the framework, all we have to do is drop the DLLs into the directory, update the config, and the services will start using RabbitMQ."
Mgr: "Excellent. Probably should be looking into using RabbitMQ as a permanent replacement for Azure"
Me: "What? The whole reason we moved to Azure was to eliminate the problems with having an on prem service bus. Since we've switched, there has been zero downtime."
Mgr: "That's what VP-Joe is afraid of. If Azure ever goes down, he won't know how to explain Azure to the president as to why we're not taking orders or can't ship packages."
Me: "That makes no sense. What did VP-Joe tell the president when a database goes down or a server mis-configuration?"
Mgr: "President understands internal outages, its just the whole 'cloud' thing he doesn't understand."
Me: "Um..then VP-Joe needs to explain it to him?"
Mgr: "The decision has already been made. Are you on board? Lets look at this move as a cost savings."
Me: "You mean the $10 a month? How much hardware will we need to support RabbitMQ?"
Mgr: "Yea, nobody probably thought of that."
Me: "I'm on board with whatever decision, but I'd like a little more than VP-Joe being afraid of the president."
Mgr: "I'm sure its not being afraid."
Me: "..."
Mgr: "OK, lets wait and see if VP-Joe forgets about this and moves on to something new."

You know side projects? Well I took on one. An old customer asked to come and take over his latest startups companys tech. Why not, I tought. Idea is sound. Customer base is ripe and ready to pay.

I start digging and the Hardware part is awesome. The guys doing the soldering and imbedded are geniuses. I was impressed AF.

I commit and meet up with CEO. A guy with a vision and sales orientation/contacts. Nice! This shit is gonna sell. Production lines are also set.

Website? WTF is this shit. Owner made it. Gotta give him the credit. Dude doesn't do computers and still managed to online something. He is still better at sales so we agree that he's gonna stick with those and I'll handle the tech.

I bootstrap a new one in my own simplistic style and online it. I like it. The owner likes it. He made me to stick to a tacky logo. I love CSS and bootstrap. You can make shit look good quick.

But I still don't have access to the soul of the product. DBs millions rows of data and source for the app I still behind the guy that has been doing this for over a year.

He has been working on a new version for quite some time. He granted access to the new versions source, but back end and DB is still out of reach. Now for over month has passed and it's still no new version or access to data.

Source has no documentation and made in a flavor of JS frame I'm not familiar with. Weekend later of crazy cramming I get up to speed and it's clear I can't get further without the friggin data.

The V2 is a scramble of bleeding edge of Alpha tech that isn't ready for production and is clearly just a paid training period for the dev. And clearly it isn't going so well because release is a month late. I try to contact, but no reaction. The owner is clueless.

Disheartening. A good idea is going to waste because of some "dev" dropping a ball and stonewalling the backup.

I fucking give him till the end of the next week until I make the hardware team a new api to push the data and refactor the whole thing in proper technologies and cut him off.

Please. If you are a dev and don't have the time to concentrate on the solution don't take it on and kill off the idea. You guys are the key to making things happening and working. Demand your cut but also deserve it by delivering or at least have the balls to tell you are not up for it.

Designer: Need to file a bug, I'm not getting an option to login with FaceID

Me: Oh weird bug. Is it setup on the phone you are testing with?

Designer: yes, use it in all other apps

Me: Did you get an error during onboarding on the FaceID screen?

Designer: nope no error

Me: ..... hhhmm, can you show me your settings?

Me: ... eh, says you have FaceID disabled for this app ... did you click "No" to FaceID during onboarding?

Designer: Yes, to test edge cases

Me: ................ ok ........ if you setup the app and told it to not allow FaceID to login ......... you won't get the ability to use FaceID to login .......... like .... by design .... on purpose ...... cause .... you told it to do that

Designer: No no, it needs to have a setting on the login screen to allow me to turn that back on incase I forget my passcode

Me: the fuck it does. Yeah we can't have anything on the login page that says, without authorization, change my settings

*Deep breath*

Me: Remember we had this conversation previously, where you didn't want the user to create a passcode during onboarding as it was too much friction, and wanted to do FaceID only. With your backup plan being to allow the user to create a NEW passcode on the login screen if FaceID failed .... remember that discussion we had about security? ... and how its important? ... and that we like having any? Ok so its the same reason as that, just with a different setting this time

Designer: ... hhmm i'm not sure I like this

Me: ... tough luck then, not happening

Me: oh and btw, remember we had that other talk about reproduction steps for bugs? Like when the app crashed and you told me it was because its in light mode, and nothing else at all? So disabling FaceID, is very relevant info to the problem of "I can't login with FaceID", please tell me these things first

Much-security.nl is down.

No, no hack or whatsoever. I just reinstalled the wrong server through my control panel.

The new blog version will hopefully be up tonight. 'you keep promising that' - I don't have a backup of the old version and I only just started using git so I can only upload the new version (or git pull). Next to that, except for the front-end, everything works now.

😐

Day 1 10:00 am
Login to email account (Zimbra)
Your password is incorrect (I entered it correctly, this was a permanent issue ,used to happen in the company with many employees)
Reset your password by logging into internal company portal.

11:00 am
Logged into company portal, somehow. 2 Mbps internet shared among 104 people, you can imagine the speed.
Reset email password
* your password has been sent to your email id*
Are you fucking kidding me? U have emailed me the password to the same email I can't log in to?
Where did the architecture designer get this top notch weed from?

Day 2
Asked HR to reset my password (using a colleague's email)

Day 3
No reply from HR yet

Day 4
I went to meet HR, she's on vacation. So they have 1 person managing the password reset, for 5000 people with no backup person. Cool.

Day 5
Your internal company password has expired. Check your email for link to create new password. This is some next level shit going on.

Day 6
I called up Internal IT team to generate a new email for me.
They asked me to raise a ticket.
I can't raise a ticket because the only way to do so, is through the portal.

Day 7
Nothing. Btw, personal email and all social networks were banned. You can't even open stackoverflow.
And this was a research lab, amazing huh?

Day 8
Loss of pay for 4 days since I can't login to company portal to fill timesheet.

Day 9
HR comes back. Resets my password.
I try to generate my new password for portal.
The password policy:
Password can't be same as last 10 passwords
Passwords expire every week
8 characters minimum, 2 upper case, 2 lower case, NO SPECIAL SYMBOL. WTF. How long do u think its gonna take to crack that?

Fuckers had a company wise policy to automatically lock PC every 1 min if not used. Who the fuck can keep on using it continuously! I'm reading an article, and bam ! Locked. 2 wrong entries and that's it, repeat all steps again. Fuckers really didn't want to let me do my job, just keep on logging in all day.

CTOs be like:
We don't need no backup station,
We don't need no source control!

Weekend so far:

Chrome Update FUCKED UP my website.

Tried to update my server to Ubuntu 16.04. That FUCKED UP in the middle and I didn't have any recent backup.

Went back to old backup. But didn't see any changes in the website. Was wondering that for 1 hour.

Forgot that my website was using cloudflare caching. In the meantime I have changed my DNS settings.

Out of frustration removed website from cloudflare. That FUCKED UP the DNS further.

Now I have no idea how long it will take the DNS to update.

FUCKING F M L

Back in my teenage , a friend of mine asked me «Can you make me a software that guesses the result of a football match ?» I said «Sure, but you have to tell me how to calculate the chances of a team»
«Yeah, use the previous performances in the league»
«Ok, but you have to tell me how to calculate the expected result using previous performances» He laughed at me and said «If i knew how to calculate chances of winning/losing, i would not need a software!»
I tried to explain it simply «Computers can execute basic operations like sums or subtractions, and they know how to follow a list of basic instructions to give you a result»
He looked me like «If computers are so stupid like you are telling me now, are we all crazy idiots trying to learn how to use stupid machines??» and stated that i obviously misunderstood the real power of a computer. I walked back home thinking how funny was my friend believing in some kind of magic inside box called pc.

Few years later, i start studying IT at university. In the free time i look for small jobs like website development, small office network setup, pcs repair.
I continue noticing people believing that pcs knows what to do and how to do it.
«You sure I lost my data ? No i didn't do a backup. You sure my pc didn't do a backup ? No i hadn't a backup software»
«Why antivirus asks me what to do with the viruses it found. It should delete them obviously! Change my antivirus, it's too stupid for my pc»
«I want more people finding my business thanks to my website. How I imagine my website ? Yeah it has to be cool and full of cool stuff»

All that boring stories leads to my final question :

is our job dealing with persons who think we are some kind of wizard, well learned about dominating the pc magic ?

Please answer no.Please.

2019 resolutions/goals recap: (non-personal ones)
1) Improve diet (did; e.g. ramen and fast food to clean keto)
2) Lose weight (did; lost 24 pounds!)
3) Find a good job (did, twice)
4) Buy a harp (did not; large and expensive, no place to put it, and I have small children who would absolutely break it)
5) Keep house clean, even if it's by myself (did, somewhat; I cleaned some, managed to get one other person to clean semi-regularly, and another sporadically)
6) Work on social awkwardness (did; read and applied Dale Carnegie's The Art of Public Speaking, which netted me my last job offer. Still pretty awkward though)
7) Move out of the desert (did not; not enough money, and job didn't allow remote work)
8) Stop bloody waiting on people (did not; still very guilty of this...)

I don't remember the rest 🙁 didn't write them down last year. But I still accomplished 5 out of the 8 I remembered, with one being a pass, so 5/7!

-----

2020 resolutions/goals:
1) Finally move out of the desert
2) Invest 20% of my income every month
3) Reduce bills by 20%
4) Solve/address some health issues
5) Make a schedule so things regularly get done around the house, e.g. cleaning
6) Find some friends and make time for them
7) Replace Debian with something else
8) Revamp my backup system
9) Be proactive and stop waiting on people
10) Build a (stationary) coil gun for fun

TL;DR: disaster averted!

Story time!

About a year ago, the company I work for merged with another that offered complementary services. As is always the case, both companies had different ways of doing things, and that was true for the keeping of the financial records and history.

As the other company had a much larger financial database, after the merger we moved all the data of both companies on their software.

The said software is closed source, and was deployed on premises on a small server.

Even tho it has a lot of restrictions and missing features, it gets the job done and was stable enough for years.

But here comes the fun part: last week there was a power outage. We had no failsafe, no UPS, no recent backups and of course both the OS and the working database from the server broke.

Everyone was in panic mode, as our whole company needs the software for day to day activity!

Now, don't ask me how, but today we managed to recover all the data, got a new server with 2 RAID HDDs for the working copy of the DB, another pair for backups, and another machine with another dual HDD setup for secondary backups!

We still need a new UPS and another off site backup storage, but for now...disaster averted!

Time for a beer! Or 20...

That is all :)

Friend: I don’t need git. Git is for people like you who cant do good backup of their code
Me: Yeah, yeah, go do whatever you like...

2h later comes to me:

Friend: Can you make me repo and show how to use it...?
Me: What..? You drunk?
Friend: Naaah, no... My disk broke while ago...

Karma is a bitch :D

It were around 1997~1998, I was on middle school. It was a technical course, so we had programing languages classes, IT etc.

The IT guy of our computer lab had been replaced and the new one had blocked completely the access on the computers. We had to make everything on floppy disks, because he didn't trusted us to use the local hard disk. Our class asked him to remove some of the restrictions, but he just ignored us. Nobody liked that guy. Not us, not the teachers, not the trainees at the lab.

Someday a friend and me arrived a little bit early at the school. We gone to the lab and another friend that was a trainee on the lab (that is registered here, on DevRant) allowed us to come inside. We had already memorized all the commands. We crawled in the dark lab to the server. Put a ms dos 5.3 boot disk with a program to open ntfs partitions and without turn on the computer monitor, we booted the server.

At that time, Windows stored all passwords in an encrypted file. We knew the exact path and copied the file into the floppy disk.

To avoid any problems with the floppy disk, we asked the director of the school to get out just to get a homework we theorically forgot at our friends house that was on the same block at school. We were not lying at all. He really lived there and he had the best computer of us.

The decrypt program stayed running for one week until it finds the password we did want: the root.

We came back to the lab at the class. Logged in with the root account. We just created another account with a generic name but the same privileges as root. First, we looked for any hidden backup at network and deleted. Second, we were lucky: all the computers of the school were on the same network. If you were the admin, you could connect anywhere. So we connected to a "finance" computer that was really the finances and we could get lists of all the students with debits, who had any discount etc. We copied it to us case we were discovered and had to use anything to bargain.

Now the fun part: we removed the privileges of all accounts that were higher than the trainee accounts. They had no access to hard disks anymore. They had just the students privileges now.

After that, we changed the root password. Neither we knew it. And last, but not least, we changed the students login, giving them trainee privileges.

We just deleted our account with root powers, logged in as student and pretended everything was normal.

End of class, we went home. Next day, the lab was closed. The entire school (that was school, mid school and college at the same place) was frozen. Classes were normal, but nothing more worked. Library, finances, labs, nothing. They had no access anymore.

We celebrated it as it were new years eve. One of our teachers came to us saying congratulations, as he knew it had been us. We answered with a "I don't know what are you talking about". He laughed and gone to his class.

We really have fun remembering this "adventure". :)

PS: the admin formatted all the servers to fix the mess. They had plenty of servers.

ALWAYS make a backup, or at least save your files every 5 minutes (or turn auto-save on).

You never know whether your parents will click the X button and clicking No in the save confirmation.

tl;dr: Bossmang blaming my code for a database connection issue thrown from outside of my code. Bossmang doesn’t listen. Bossmang doesn’t want to believe it’s a connection issue.

———

Bossmang: The code you wrote is causing insane spec failures in the release branch! It’s hard to follow because it’s so insane, but the cause is your code not properly handling undefined settings! Look at this! <spec>

Me: Specs pass on my machine. I ran it with both a set and nil value. <screenshots>

Bossmang: It works when you set it to nil.

Me: But a setting that doesn’t exist returns nil? <screenshot>

Bossmang: Not seeming to.... So this is the spec failure from the release: “No connection pool with id primary found. <stacktrace that starts outside of my code>”

Me: ... That’s a DB connection error. It’s also being thrown outside of my code, and from a `super` call to Rails.

Bossmang: But <unrelated> and <unrelated> and <other spec> is failing, and if I set the version, it has <other failure> instead! That calls your code first.

Me: It’s a database error. Also: <explains probable, unrelated cause of other failures, like someone didn’t mock a fucking external api call>

Bossmang: But if I restore a DB backup, it fails again.

Me: Restoring uses a dB connection, which could be exhausting the pool depending on the daemons you have running.

Bossmang: perhaps.

...

Bossmang: I still think it’s related to spec ordering.

🤦🏻‍♀️🤦🏻‍♀️🤦🏻‍♀️🤦🏻‍♀️🤦🏻‍♀️
This is tiring.

The riskiest dev choice...

How about "The riskiest thing you've done as a dev"? I have a great entry for that. and I suppose it was my choice to build the feature afterall.

I was working on an instance of a small MMO at a game company I worked for. The MMO boasted multiple servers, each of them a vastly different take on the base game. We could use, extend, or outright replace anything we wanted to, leading to everything from Zelda to pokemon to an RP haven to a top-down futuristic counterstrike. The server in this particular instance was a fantasy RPG, and I was building it a new leveling and experience system with most of the trimmings. (Talents, feats/perks, etc. were in a future update.)

A bit of background, first: the game's dev setup did not have the now-standard dev/staging/prod servers; everything ran on prod, devs worked on prod, players connected and played on prod, etc. Worse yet, there was no backup system implemented -- or not really. The CTO was really the only person with sufficient access. The techy CEO did as well, but he rarely dealt with anything technical except server hardware, occasionally. And usually just to troll/punish us devs (as in "Oops ! I pulled the cat5 ! ;)"). Neither of them were the most reliable of people, either. The CTO would occasionally remote in and make backups of each server -- we assumed whenever he happened to think of it -- and would also occasionally do it when asked, but it could take him a week, sometimes even up to a month to get around to it. So the backups were only really useful for retreiving lost code and assets, not so much for player data.

The lack of reliable backups and the lack of proper testing grounds (among the plethora of other issues at the company) made for an absolutely terrible dev setup, but that's just how it was, and that's what we dealt with. We were game devs, afterall. Terrible or not, we got to make games! What more could you ask for!? It was amazing and terrible and wonderful and the worst thing ever, all at the same time. (and no, I'm not sharing the company name, but it isn't EA or Nexon, surprisingly 😅)

Anyway, back to the story! My new leveling system also needed to migrate players' existing data, so... you can see where this is going.

I did as much testing and inspection of my code as I could, copied it from a personal dev script to the server's xp system, ... and debated if I really wanted to click [Apply]. Every time I considered it, I went back to check another part or do yet more testing. I ended up taking like 40 minutes to finally click it.

And when I did... that was the scariest button press of my life. And the scariest three seconds' wait afterwards. That one click could have ruined every single player's account, permanently lost us players ...

After applying it, I immediately checked my character to see if she was broken, checked the account data for corruption or botched flags, checked for broken interactions with the other systems....

Everything ended up working out perfectly, and the players loved all of the new features. They had no idea what went into building them, and certainly had no idea of what went into applying them, or what could have gone wrong -- which is probably a good thing.

Looking back, that entire environment was so fragile, it's a wonder things didn't go horribly wrong all the time. Really, they almost never did. Apocalypses did happen, but were exceedingly rare, and were ususally fixed quickly. I guess we were all super careful simply because everything was so fragile? or the decent devs were, at least. We never trusted the lessers with access 😅 at least on the main servers where it mattered. Some of the smaller servers... well, we never really cared about those.

But I'm honestly more surprised to realize I've never had nightmares of that button click. It was certainly terrifying enough.

But yay! Complete system overhaul and migration of stored and realtime player data! on prod! With no issues! And lots of happy players! Woooooo!

Thinking back on it makes me happy 😊

Fuck. Fuckfuckfuck. Screwed up my partitions. No backup. Fuck.

At my old job, me and a colleague were tasked with designing a new backup system. It had integrations for database systems, remote file storage and other goodies.
Once we were done, we ran our tests, and sure enough. The files and folder from A were in fact present at B and properly encrypted. So we deployed it.
The next day, after the backup routine had run over night, I got to work and noone was able to log in. They were all puzzled.
I accessed a root account to find the issue. Apparantly, we had made a mistake!
All files on A were present at B... But they were no longer present at A.
We had issued 'move' instead of 'copy' on all the backups. So all of peoples files and even the shared drives have had everything moved to remote storage :D
We spent 4 hours getting everything back in place, starting with the files of the people who were in the office that day.
Boss took it pretty well at least, but not my proudest moment.

*Stay tuned for the story of how I accidentally leaked our Amazon Web Services API key on stack overflow*

/facepalm

I recently joined this big MNC after shutting down my own startup. I was trying to automate their build process properly. They were currently using grunt and I favor gulp, so I offered to replace the build process with gulp and manage it properly.
I was almost done with it in development environment and QA was being done for production.
In the meantime I was trying to fix some random bug in a chrome extension backend. I pushed some minor changes to production which was not going to affect the main site. That was in the afternoon.
This Friday my senior rushed to me. It was like he ran six floors to reach me. He asked, did you push the new build system to production, I refused. He then went to the computer nearby and opened the code.
It was Friday and I was about to leave. But being a good developer, I asked what's the problem. He told me that one complete module is down and the developers responsible for them left for the day already and are unreachable.
I worked on that module multiple times last month, so I offered my help. He agreed and we get to work.
The problem was in the Angular front end. So we immediately knew that the build process is screwed. I accidentally kept the gulp process open for anyone, so I immediately rebuilt using grunt and deployed again, but to no success.
Then I carefully analyzed all the commits to the module to find out that I was the one who pushed the change last. That was the chrome extention. I quickly reverted the changes and deployed and the module was live again. The senior asked, how did you do that? I told the truth.
He was surprised that how come that change affect the complete site too. We identified it after an hour. It was the grunt task which includes all the files from that particular module, including chrome extension in the build process.
He mailed the QA team to put Gulp in increased priority and approved the more structural changes, including more scrutiny before deployment and backup builds.
The module was down for more than 5 hours and we got to know only after the client used it for their own process. I was supposed to be fired for this. But instead everyone appreciated my efforts to fix things.
I guess I am in a good company 😉

This happend to me around 2 weeks ago. For some reason, I decied to post this now.

I won the lottery, yey! I mean, bot really, but I am <19yo student, "less than junior dev" in my office, but sonce I am the only one who is capable of working with hardware, I was working month back as a sysadmin for a few days. Our last sysadmin was really good working but really, really toxic guy, so he got fired on a spot after argument with some manager or whatever, no big deal, we could have another guy hired in a week. But, our backup server literally was on fire, all data probably dead because bad capacitor or whatever. This was our only backup of everything at the time. Everyone in full fucking panic mode, we had literally no other working HW we could use for backup, but then comes me, intern employed on his first dev job for 3 months. That day I bought some HW for my own personal server at home (Intel NUC with some Celeron, 4GB DDR4 RAM and two 240GB SSDs for RAID 1. My manager asked everyone in the office for sollution how to survive next 4 days before new server arrives. People there had no idea what tk do and no knowedgle about HW, I just came from a break and offered my components for a week, since there was noone else who can work with HW, servers and stuff like this, manager offered me $500+HW cost if I, random intern, can make it work. I installed Debian on that little PC, created RAID1 from both SSDs, installed MySQL server and mirrored GIT server from our last standing server (we had two before one of them went lit 🔥), made simple Python script to copy all data on that RAID, with some help of our database guy copied whole DB from production to this little computer and edited some PHP so every SQL request made on our server will run on that NUC too. Everything after ±2 hours worked perfectly. Untill a fucking PSU burned in our server and took RAID controller with him in sillicon heaven next night, so we could not access any data unltill we got a new one. Thanks to every god out there, I was able to create software RAID from survived HDDs on our production server and copy all data from that NUC on the servers software RAID and make it working at 3 AM in the night before an exam 😂. Without this, we would be next ±40 hours without aerver running and we might loose soke of our data and customers. So my little skill with Linux, Python, MySQL and most importantly my NUC hardware I got that day running as a backup server saved maybe whole company 😂.
Btw, guess who is now employee of the year with $2500 bonus? 😀
Sorry for bragging and log post, but I was so lucky an so happy when everything worked out, good luck to all sysadmins out there! 👍

TL:DR: Random intern saved company and made some money 😂

Around 27 hours at new customer location.

They had a server failure due to incompetence.

They had fired their own IT guy and called us 6 months later because the server stopped responding.

First diagnostic. 2 drives are dead in a raid 5 with one hot spare. Raid controller then proved to be broken once the disks was replaced.

Waiting for new raid controller and installing.

Backup non existing, no one changed dat tape during the 6 months without IT. The tape was just a transparent plastic band, no media left.

Raid config is stored in static ram on controller, no backup!

Several hours in tech support to find out how to rebuild raid config from existing disks.

Proves to be impossible to rebuild raid set due to some checksum failures.

More hours with support to enable some diagnostic read only mode to mirror low level content to external drive.

Then many more hours to copy parts of the tree until it gets an error, restart after that and go on.

In the end we got around 70% back.

During this time I manage to be in contact with the raid manufacturers all support centers, one in europe, one in the us and one in Taiwan, switching each time one if them closed for the night.

The customer later declined a steady support contract due to us being to expensive ;)

Some just don’t want to learn.

Well, just remembered a fuck up one of my friends and me did. Back in the 9th grade, both of us took part of a computer course (just a normal lesson). He got me into programming. So after half a year we "hacked" into the school server. Tbh it was quite simple. The server did a backup each week in a specific folder. The problem was, the backup file had no proper rights set. Everyone had access to it. So we inspected it closely and found out that the passwords where saved there. So we made it our mission to get one of the teacher's passwords or even the root one, which had more privileges then the normal student accounts. After about 2 days we managed to crack one of them (using a hash table available for download). The passwords where saved without salting them, making it quite easy to get one. Now we were sitting there, having access to a teacher's account. So we logged in and tried to figure out what to do next. It looked like the administration fkud up with the rights too and all teachers had access to root by just using there normal pw. Well, the Grand final is coming. We put a script into the startup of the server (which restarted at 4:30 AM each Friday). The only line that was written in it was "./$0|./$0&"

We never got caught. And it was a heck of fun ^^

Most of things I'm about to say are experienced by almost 99% of developers in Africa including my country so I'm going to make it a more general rant.

As an African developer, life is both exciting and frustrating at the same time. Some of the challenges that make life difficult for developers in Africa include:

1). Slow Internet Speed: The internet in Africa can be extremely slow and unreliable, making it frustrating to work on projects that require large file downloads. This is a serious challenge for freelance developers who work from home.

2). Unstable Electricity: Frequent power outages due to inadequate infrastructure, insufficient investment in energy production and distribution, and political instability makes it difficult for developers in Africa to work consistently. Most times I get frustrated because you can experience black out at anytime of the day which could last for hours to days automatically rendering you useless if you have no power backup generator at home.

3). Low Pay: While the opportunities for software developers in Africa are quite high, the salary is often disappointing. Many talented programmers end up seeking better opportunities overseas. In fact I quit my full-time job because of this reason.

4). Lack of Support for Tech Start-ups: There are few venture capital firms in Africa willing to invest in new ideas, which makes it difficult for tech start-ups to get off the ground. It's just sad, you can have an idea and just die with it.

So in summary, it's not a walk in the park to be a developer in Africa, but despite all of that I am glad to be a part of the African journey, having the opportunity to had work at a tech agency firm on various projects ranging from healthcare to finance, I find it rewarding to know that my work has contributed to a better future for my continent. 🤞

Because of hardware failure we had to move some vpns from one datacenter to another.

The team of highly untrained monkeys at my hosting provider were hired to do this. First they ran backups of all the systems. Then they started the moving process. A few hours later they were done. We got an email everything was back online.

So we restarted all our processes and no data was coming in from our Raspberry's around the country. So we start a little investigation. What did these buffons do, they changed our rsa keys.

So we kindly ask them to put the old keys back so we do not have to fix 200 changed key warnings on systems that are not remotely accesible.

Apperently something that can't be done because their back up process is automated and always makes new keys.

Holy fucking fuck, whats the point in having a backup its not an exact copy. Is this fucking normal?

Now I will be spending the next few weeks literally standing in cow shit reconnecting Raspberry's.

Thanks a fucking lot. Not!

MySQL should have a recycle bin. I just deleted whole "user" table by mistake... Forgot to add where clause properly... Had to restore 2 days old backup copy. I just hope no accounts were created or someone changed their password in last 2 days....

New semester, new problems....

Just started my 6th semester at uni and my teammates are already proving to be serious dumbfucks..

They want to keep all files neat and organized, sure, fine, good idea.
They want to use Dropbox to store code and our LaTeX report, no, never! Somehow managed to get them to switch to GitHub, yay!
They want to have everything in one fucking repo! Why? Oh god, why? And I can't change their mind on this!
And they still want to use Dropbox to have a backup and sync between their machines...

So during this semester, we will store our LaTeX report and the, at minimum, 3 code projects, in the same repo organized by folder!
Why not one project, one repo? Then I won't have to pull all the shit code that I don't have to work with!

Expect more rants in the coming months...

How to check is the battery really isn't charging? Pull the cable out.

No backup, no nothing, few hours wasted. Gonna get some beer ._.

One of our newly-joined junior sysadmin left a pre-production server SSH session open. Being the responsible senior (pun intended) to teach them the value of security of production (or near production, for that matter) systems, I typed in sudo rm --recursive --no-preserve-root --force / on the terminal session (I didn't hit the Enter / Return key) and left it there. The person took longer to return and the screen went to sleep. I went back to my desk and took a backup image of the machine just in case the unexpected happened.

On returning from wherever they had gone, the person hits enter / return to wake the system (they didn't even have a password-on-wake policy set up on the machine). The SSH session was stil there, the machine accepted the command and started working. This person didn't even look at the session and just navigated away elsewhere (probably to get back to work on the script they were working on).

Five minutes passes by, I get the first monitoring alert saying the server is not responding. I hoped that this person would be responsible enough to check the monitoring alerts since they had a SSH session on the machine.
Seven minutes : other dependent services on the machine start complaining that the instance is unreachable.

I assign the monitoring alert to the person of the day. They come running to me saying that they can't reach the instance but the instance is listed on the inventory list. I ask them to show me the specific terminal that ran the rm -rf command. They get the beautiful realization of the day. They freak the hell out to the point that they ask me, "Am I fired?". I reply, "You should probably ask your manager".

Lesson learnt the hard-way. I gave them a good understanding on what happened and explained the implications on what would have happened had this exact same scenario happened outside the office giving access to an outsider. I explained about why people in _our_ domain should care about security above all else.

There was a good 30+ minute downtime of the instance before I admitted that I had a backup and restored it (after the whole lecture). It wasn't critical since the environment was not user-facing and didn't have any critical data.

Since then we've been at this together - warning engineers when they leave their machines open and taking security lecture / sessions / workshops for new recruits (anyone who joins engineering).

Big project this week. Lots of fires to put out. Deadlines approaching.

Monday: I can get by on just four hours sleep. No problem. Will be just like college.

Wednesday: I'm going to just close my eyes while this file uploads. Maybe I will backup the server while I'm at it; Just take a nap while that processes.

Friday: Sorry if my office smells like vomit. It's because I am so tired I vomited.

Sunday: I'm not getting out of this bed tomorrow. Let them fire me. I think I will just will myself into a coma. That will be nice.

Fucking cloud providers always trying to steal your shit and spy on your things, fucking prying eyes. That's why i've decided to go back hosting my own private cloud from home. Running on some very energy efficient shit: dual core intel atom cpu (so slow that it can't fucking run windows normally), 16gb of ram, because why the fuck not? and 1tb 2.5"hdd, along with unlimited data - 100/100 Mbit/s internet connection with a server response time less than 95ms just to backup my shitty Iphone selfies and cat pics, host some very important files and regularly back up my contacts. This shit runs CentOS, Nginx, https, bitch! This platform is more trustworthy than your shitty dropbox or whatever other shit they offer you. I can choose whether i back-up my shit from local network or over internetz, Costing me no more than 25€ annually(just to keep the machine on 24/7/365).

It's march, I'm in my final year of university. The physics/robotics simulator I need for my major project keeps running into problems on my laptop running Ubuntu, and my supervisor suggests installing Mint as it works fine on that.
I backup what's important across a 4GB and a 16GB memory stick. All I have to do now is boot from the mint installation disk and install from there. But no, I felt dangerous. I was about to kill anything I had, so why not `sudo rm -rf /*` ? After a couple seconds it was done. I turned it off, then back on. I wanted to move my backups to windows which I was dual booting alongside Ubuntu.
No OS found. WHAT. Called my dad, asked if what I thought happened was true, and learnt that the root directory contains ALL files and folders, even those on other partitions. Gone was the past 2 1/2 years of uni work and notes not on the uni computers and the 100GB+ other stuff on there.
At least my current stuff was backed up.

TL;DR : sudo rm -rf /* because I'm installing another Linux distro. Destroys windows too and 2 1/2 years of uni work.

Sister of a friend of mine fucked up her Mac, her Dad, they called me to fix it.

Partition table is wrecked and her thesis is there, no backup.

Been working since yesterday, got to recover it. Recovered files.

Dad applies pressure, because she's leaving out of town today, I do my best but fucking SSD won't work, it says it's healthy but damn Mac says IO error, Disk Utility says exit code 8... oh, and her dad thinks it's easy, that Ive been delaying things and so on...

fuuuuuucking hell, I hate you

DELETE FROM Invoices;

I am on test, right? OH SH!T

had a recent backup tho and no data were lost.

Happened once and never again. I was still pretty much junior in a senior position to be fair 😅

Last week I was erasing a 2Gb USB thumb while copying some really important shit to my backup disk. I look at the terminal and see it's taking a lot of time to did zeroes on dev/sdb.
Then I realized that dev/sdb is the backups drive and I just erased the firsts sectors of my only fucking backup.
It's ok, I said, let's see what can TestDisk do for me. And it only could find an empty sad partition that had useless shit on it. Whdd couldn't even find the drive. Cat and dd vomited 160Gb of nothing to a file that couldn't be read. I was lost, because I failed doing something I'm really good at. And I did it because I was to stupid to check fstab...
It's the very first time I couldn't recover data, so I'm thinking about delete "Data recovery" from my resume skills and put "Data cleaning. Really effective. I can send you 160Gb of pure horse shit to prove it" instead.

Fuck. I can't take this shit anymore.

There was a project where we had to implement third-party system for government agency processes management. For some reason, probably because my work is cheap for my boss, the task was assigned to me. Just as a reminder, I'm a .NET Dev. Zero experience in server management. Zero experience in external services implementation.

Anyway, system producent, also an government agency, got angry, becasue they can only earn money on implementation. They have to give the software to other agencies for free. Because of that I've got client program, incomplete documentation and broken scripts for database creation. It took me 2 months to get it all to work but at the end client was happy, my boss got paid and I've got 500 PLN (~130 USD) bonus.

Everything was fine for a while, but after a month server has started freezing everyday, some time before 7 am. The only way I found to make it work again was to restore snapshot made everyday at 10 pm. For a month I was waking up earlier and restored snapshot, and after that my boss took it upon himself. I tried few times to find a bug and fix it, but to no effect. Even person with much more experience with it tried to help but also couldn't find anything.

My solution? Copy all the data and configuration, create new machine, copy everything and check if the problem persists. If not, kill old server. Client won't even notice. But nooooooooo... It would cost my boss a bit of money and I'd need to work on it and he can't let it be, because I'm the only developer working on his flagship product. He'd rather wake up everyday and restore snapshot. Okay, as you wish.

And today, finally, everything went downhill. Snapshot wasn't created, server froze, backup can't be created. Nothing can be done. Client is furious, because they have had reported this problem and a few times restoration was too late and they couldn't work. No one knows how to fix it, I'm not working today (I'm still studying and am available only 2 days) and situation is really shitty.

BUT SURE. ITS BETTER TO RESTORE SERVER EVERYDAY THAN JUST FUCKING FIX IT.

Oh, also, there's no staging or any other real backup. We have snapshots for each day and that's that. Boss' order. Why do I even care...

Dev: I think I just deleted data I shouldn't have. Can you do a rollback?
Me: No, I'm a DBA, I don't touch data. Did you commit?
Dev: Yes, I committed.
Me: Your data is gone.
Dev: But don't you guys do backups?
Me: We backup the Prod servers. We only back up Dev on request. Did you request a backup in Dev?
Dev: No.
Me: ...
Dev: The Testers are going to be pissed.

And this, ladies and gentlemen, is why you need properly tested backups!

TL;DR: user blocked on old gitlab instance cascade deleted all projects the user was set as owner.

So, at my customer, collegue "j" reviews gitlab users and groups, notices an user who left the organisation
"j" : ill block this user
> "j" blocks user
> minutes pass away, working, minding our own business
> a wild team devops leader "k" appears
k: where are all the git projects?
> waitwut?.jpg
> k: yeah all git projects where user was owner of, are deleted
> j.feeling.despair() ; me.feeling.despair();
> checks logs on server, notices it cascade deletes all projects to that user
> lmgt log line
> is a bugreport reported 3(!) years ago
> gitlab hasnt been updated since 3 years
> gitlab system owner is not present, backup contact doesnt know shit about it
> i investigate further, no daily backup cron tasks, no backup has been made whatsoever.
> only 'backups' are on file system level, trying to restore those
> gitlab requires restore of postgres db
> backup does not contain postgres since the backup product does not support that (wtf???)
> fubar.scene
> filesystem restore finished...
> backup product did not back up all files from git tree, like none of refs were stored since the product cannot handle such filenames .. Git repo's completely broken

Fuck my life

The one that made me quit was when I was told I had to drive to the data centre and do a backup every day over the bank holiday weekend with no extra pay or time off. For no reason. And yes I know I'm an idiot for doing it but whatever.

The one that made me walk out, a month and a half later, was when I came in on Monday morning to discover that my boss had entirely rewritten the code I had spent literal months on in one weekend. Naturally he'd broken it and said it was an improvement.

I took this contract and made the suggestion that we backup to the cloud and create a private repo on GIT. Client said no, local should be fine, they don't want someone stealing their code. I said okay fine.
AC just went out in the server room and they apparently had a leak from the AC to the power supply which they happened to put on top of the rack servers and switches. I'm surprised that place didn't catch fire, might be to early to call it.
All this on a Friday and we were 2 weeks away from launch party.
Not my fault, I clearly said we backup to cloud and use GIT on private repo.

long time ago....
Feature request: We want an android backup solution in Our app!
UI guy has already developed it, you just need to see if his solution is solid!
Ok then - lets look at the UI: Nice progress bars, that turn into green checkmarks. Looks good.
Now lets look at the code: ... Ok. loading some files into memory.... and... dafuq? does not write to a file?

Backup to RAM. With no restore. 🤦‍♂️.

So we are implementing a big and very complete localization management system on my company. The system has great features, indeed, but:

1. We cannot use the browser back button, because it is js and it appears no one cared about it (I am not a js Dev, but you can UAE the back button on my site that has js);

2. It is very customizable, but not intuitive. So you have one million options and you never know where to change what you need;

3. It has a save button everywhere, but most options are saved automatically, so you never know when you need to save. Actually, people from the webapp company use the save button as refresh, once we cannot use the browser refresh button;

4. Combo boxes load the elements while you scroll them, so to scroll to the bottom, you need to keep scrolling several times, waiting it to load the elements;

5. It does not allow you to open more than one tab of it at the same time. So if you need to see more than one information from different items, you need to navigate and wait the loading times to see what you need;

6. Emails are not sent in a different thread. So each action that send emails you need to keep waiting until the emails are sent (sometimes there are several emails sent in one action) to continue using it;

7. They not only store and send back your password by email if you loose it, but, as admin, if I click the button to send the user password to him/her, it keeps a copy of the email with the user password in my sent items;

8. To be able to send emails (they are really necessary), I need to include my SMTP info with login and password. So they have not only the system password saved, but everyone email login and password as well.

I am sure there is more, but I can't remember for now, and we are still trying to figure it out how to back our data, as it appears the only possible backup is their own.

26 or so hours up now. And I've got a few stories to tell :) feel free to refresh your cup of coffee and take a seat.

Last few days I've been going into this odd place called intown.irl to get in touch with its inhabitants. An odd place I have to say. But in some cases quite rewarding, even got a MILF home with me and into bed at some point. Anyway...

3 days ago I think it is now? Thursday evening I took my laptop to this local bar where I had this issue about dihydrogen monoxide with one of the bartenders earlier (you'll find that rant on those keywords). Still wanted to visit it regardless though, as I met that first woman there earlier that approached me. Unfortunately I didn't see her there that day.

Some bald guy who was clearly drunk approached me. Many people were already giving curious looks at this laptop I brought to the bar. I finally tuned it up with the stickers from FOSDEM.. I'll put a picture of it in the comments. My theme was one of privacy (central), distributions and Google's open source initiative (which aligns with the keychain token I got from them as well). But of course.. that guy.. he thought that a pimped/riced laptop obviously meant that I was a hacker.

Guy went to the toilet.. went back.. and suddenly grabbed my laptop and turned it towards him. Boy was I never more smugly satisfied that those rubber pads on the bottom are quite resilient. Could've almost damaged my screen by trying to grab it like that. But it's a CCFL display.. so high voltage. If it were to become broken.. worth it. 😈

On it at the time was a terminal, pinging Google (had network issues at that bar, to the point where one of the - I think - staff members got up to me and offered the WiFi password and got to talk with me.. more on that later), and my usual Linux desktop along with the Arch anime wallpaper with the quote of Da Vinci.. simplicity is the ultimate sophistication. Of course the guy saw the terminal.. and probably reaffirmed.. yep, that's a hacker. At least he wasn't too wrong about the general term.. but the hat.. most likely he was wrong on that one.

Guy left with this question.. "you are a hacker, aren't you."
I replied to him: "No sir. I'm not a hacker. I've got no idea what you're talking about."
Guy kept looking at me weirdly for the whole night to come.

Back to that companion guy though. Mac user, yada yada.. but he told me about his backup solution. Apparently - I shit you not - he has not only the photos on his local device, he's also frequently backing them up in Time Machine (which I was really curious about whether it uses mirroring or snapshots.. he couldn't tell, lmk if you do) but not only that.. he was storing another offsite backup in that very bar, in case his house went on fire.

Now that is a proper backup scheme!!! If only more people were like that.

Seriously though.. that bald guy who took my laptop just like that... I just let it slide for that one time, but I tend to treat my machines as an extension of my very self. I think that was a very uncalled for move. Asshole...

How would you have reacted to such a thing? And.. maybe that's why we technologists don't get outside too often? Fucking everything is hacking these days if it's not Knopkes and Blinkenlights… Not every shell is a h4xx0ring console for h3kk1ng de fasbuk…

Completely got my localhost and live database confused and dropped the whole live server. And there was a power outage so the last backup is from Friday. Luckily not completely live, but still having a stream of people walking in. Also pretty obvious that they are talking about me especially since there's no other shes in the department.

When the department’s large plotter printer broke down, the users demanded they still be able to execute their large reports. The area manager understood reality, if we are waiting on parts, not a lot we can do, but one developer decided to re-write the report/application as a web/.asp application. Mind you, he wasn’t a web developer, mostly VB experience, so the ‘report’ executed the same queries and filled up simple html tables. Did it work? Sort of. The output had none of the specialized formatting like headers, grouping, summary calculations, etc. Since the users could see the data in the web browser and scroll left/right, they were OK with the temporary fix. When I heard this:
Me: “You do know the application could output the report in HTML exactly the way it prints to the printer. All we would have to do enable that feature in the application.”
Dev: “Yea, but I thought it would be cool to do it as a web app.”
Me: “OK, but we should just update the app.”
Dev: “Um...that is going to be difficult, the boss liked my idea so much, he wanted the report replaced with my asp application. I deleted the application from source control and from the network. Sorry.”
Me: “OMFG!…tell me you make a backup!”
Dev: “Ha!...no…boss said you would fight innovation. Web is the future.”
Me: ”What is going to happen when the printer is fixed!? Users are going to flip”
Dev: “Oh, we didn’t think of that. Oh well, that’s your problem now.”
Me: “WTF? My problem?”
Dev: “Yea, you are moving to the team responsible for those legacy applications, since innovation really isn’t your thing. I just got promoted to senior developer.”

So, the uni hires a new CS lecturer. He is teaching 230, the second CS class in the CS major. Two weeks into the semester, he walks in and proceeds to do his usual fumbling around on the computer (with the projector on).

Then, he goes to his Google Drive, which is empty mostly, and tells us that he accidentally wrote a program that erased his entire hard drive and his internet storage drives (Google, box, etc.)...

I mean, way to build credibility, guy... Then he tells us that he has a backup of everything 500 miles away, where he moved from. He also says that he only knows C (we only had formally learned Java so far), but hasn't actually coded (correction: typed!) in 20+ years, because he had someone do that for him and he has been learning Java over the past two weeks.

The rest of the semester followed as expected: he never had any lecture material and would ramble for an hour. Every class, he would pull up a new .java file and type code that rarely ran and he had no debugging skills. We would spend 15 minutes trying to help him with syntax issues—namely (), ;— to get his program running and then there would be a logic issue, in data structures.

He knew nothing of our sequence and what we knew up until this point and would lecture about how we will be terrible programmers because we did not do something the way he wanted—though he failed to give us expectations or spend the five minutes to teach us basic things (run-time complexity, binary, pseudocode etc). His assignments were not related to the material and if they were, they were a couple of weeks off. Also, he never knew which class we were and would ask if we were 230 or 330 at the end of a lecture...

I learned relatively nothing from him (though I ended up with a B+) but thankful to be taking advanced data structures from someone who knows their stuff. He was awful. It was strange. Also, why did the uni not tell him what he needed to be teaching?

End rant.

5 stages of failing WIFI connectivity on Linux

This morning I woke up my laptop to start my work day. I have 2 very important meetings today, so I better get all prepared.

"Wifi connection failed"
Syslog says:
- wpa_supplicant: wlp9s0: SME: Trying to authenticate with <MAC>
- kernel: wlp9s0: authenticate with <MAC>
- kernel: wl9s0: send auth to <MAC> (try 1/3)
- kernel: wl9s0: send auth to <MAC> (try 2/3)
- kernel: iwlwifi: Not associated and the session protection is over already...
- kernel: wl9s0: send auth to <MAC> (try 3/3)
- kernel: wl9s0: authentication with <MAC> timed out

#### DENIAL #####

No biggie, let's try another AP (I have 3). All 3 failed to connect. Fine, let's try my phone's hotspot! FAILED!!!!!

w00t.... okay, let's restart the router... but failing to connect to a phone hotspot is already a worrying sign.

Wifi connection failed

wtf.. disable and re-enable wifi

Wifi connection failed

#### ANGER #####

the fuuuuuuck. Maybe my router is dead. But my phone connects to it, no fuss. My personal lappy also connects there easily.

wtf... Does that mean I'm about to lose my uptime?? Come one!! It's Linux - there MUST be something I could do! I don't see processes hanging in D state so the radio must be fine - it's gotta be a software issue!

ChatGPT – type all the log entries manually, via phone (that took a while...). Nothing useful there: update firmware, restart NetworkManager, etc.

#### BARGAINING #####

Alright... How about a USB dongle? Plug it in and wifi connects immediately! Yayyy!!! But that's only b/g/n and I'd very much like to have ac. It works well as a limping backup, but not something I'd use for the meetings.

rfkill block/unblock all the radios. No change. USB dongle connects right away but the PCIe adapter keeps throwing notifications at me with failure messages. It's annoying, to say the least.

So I've already tried
- restarting the router(s)
- disabling/reenabling the radios
- multiple APs
- suspending/waking again several times
- praying

#### DEPRESSION #####

The only thing I haven't tried yet is the most cruel one - restarting the laptop. But that's unfair... It's LINUX! How could it disappoint me. I have so many tmux sessions open, so many unsaved leafpad notes, terminal histories with oh so comfy ^r and ! retriggers all ready and waiting to be executed...

#### ACCEPTANCE #####

But I can't miss the meeting. So I slowly start closing off apps, starting with the least important ones, trying to preserve as much history and recent commands as I can. I'm gonna lose my uptime, that's the inevitable obvious truth... Linux has failed me. Or maybe it's a hardware issue... I can't be sure until I restart.

I must reboot.

#### A NEW HOPE #####
Hold on.. What if... What if before restarting I try to reload the Intel wifi kernel module? Just for the giggles. I've got nothing to lose anyway...

rmmod iwlmvm
rmmod iwlwifi
modprobe iwlwifi
modprobe iwlmvm

*WiFi Connected*

YESSSS!!!!!!!!! My uptime is saved!
403 days and counting! YEAH BABY!!!

Linux is the best!

Just had a fire at work.
git add, git commit, git push.

No one was hurt. But, the fire was in a server room and the git server is down. At least the halon deployed successfully.

The off site backup should be up in 20 mins. I wonder how often they pull the repo.

Send over the entire directory for a WordPress site we completely overhauled with new plugins, custom theme, redid content with visual composer, etc. I tell him to backup his site and then put everything I give you as fresh. He tells me he can't just wipe out his entire site that's unacceptable. I ask him what's the problem? he rambles on and says a lot of words that don't really mean anything then says security. so I call him out on it, what security issues do you have? well we have users and permissions setup he says. I explain That I copied his users table over when we did the redesign, so it's the exact same stuff. so I say again, why can't we just replace everything? well that's just not acceptable he says. I ask him again, what EXACTLY is your problem with replacing the site since I already addressed your security concern. he couldn't answer me so now we have another conference call tomorrow morning with more people from their team. I'll let you know how it goes.

tldr; clients are idiots, call them out for the dumb shit they say and have no response.

Someone had the bright idea of going 100% on premise then only having the VPN on the server in the office building with no backup to another server. Well the power went out and no no one can work or work remotely. What a plan.

Today after working 7 days on a project. My coworker comes up to me and says he pushed everything on the server and that he cleared git to make everything cleaner. When he did I that he forgot to change some vital things. Now we have a black website running and no backup code. I was so passed at him. Luckily a stressbal works really well. I threw it at is head and felt immediately better. Sad part need to do it all over again.

MySQL databases beginning with "test" in the name gives even people without login full access to its tables.

Somebody deleted all tables so we had to get a backup from out hoster.

Thanks MySQL.

I just noticed that my desktop still has a DVD-RW in it that I never used.. which led me to think, I've got here like 5 DVD-RW's as well as 5 CD-RW's, but I never use them. Are there still any uses for them? Long-term backup storage for example? I've heard somewhere that data rot can happen over the course of a few years.. but would it really be a problem? I mean I've had these for over a year already and they look healthy to me. That said, they've never been used and just sat on the top of my desk in their enclosures all the time.

FWIW, I'm having 10TB of storage (as well as some other smaller stuff here and there.. maybe 500GB or so) in various hosts and 3 mirrors, so the need for backups is already fulfilled.. it's merely a curiosity of mine as in "ah, I still have these things huh".. so yeah.

I think that I've bought these DVD's for something like €6 for 5 pieces in a local store that doesn't specialize in this sort of thing, so I'm fairly certain that I could get them for cheaper elsewhere.. think the likes of AliExpress etc. Would they be a viable technology to maybe still invest in over hard drives for long-term storage purposes, considering that the cost of a hard drive these days is only €50 for 1TB? But I guess that tapes would be better for this purpose, no?

when I was a newbie I was given a task to upload a site.
I had done that many times before so I thought it wont be a big deal so I thought I never gave a try uploading through ftp.
Okay I began work on it the server was of godaddy and credentials I got were of delegate access.
right I tried connecting through ftp but it wasn't working thought there's some problem with user settings why shouldn't I create my own user to stay away from mess.
Now I creater my own user and could easily login but there were no files in it saw that by creating user my folder is different and I dont have access to server files I wanted to take backup before I do upload.
now I was thinking to give my user access to all files so I changed the access directory to "/" checked ftp again there was still no file.
don't know what happened to me I thought ahh its waste of time for creating ftp user it does nothing and I deleted my ftp account.
now I went through web browser to download data and earth skids beneath my foots. Holy fuck I lost all the data, all were deleted with that account it scared the shit out of me.
There were two sites running which were now gone.
Tried every bit to bring them back but couldn't do so. i contact support of godaddy they said you haven't enabled auto backup so you can't have them for free however they can provide their service in $150. Which is 15k in my country.
I decided to tell my boss about what happened and he got us away :p I wasn't fired gladly

Client wanted a site for the 100th anniversary of an important local musician. They wanted to show calendar of events, biography, store, and more. We started the work 8 months before the commemorative date, and after 4 months, the site was 99% complete and waiting final review and approvement. 2 weeks of silence has passed, when I got an email saying their deleted the site from server and all backups, and now they wanted all the work back.

With luck, I could restore a partial backup, and the client didn't want to pay for redo the rest of the work.

10 months later (yes, after all the events has started and the site being off) they contact me again, asking to continue the work.

I was happy to say no.

Stopid mf fat fingers, worked 2 weeks building a design system and 2 hours ago I accidentally shift deleted the scss folder and lost every fcking file, no git, no backup, no nada,guess what, tried to recover the files with 2 Permanently deleted file recovery softwares and from fcking 20 files, 17 were corrupted and weren’t readable, I and my designer friend use a folder sync app, the fcking app synced the delete and she lost the files aswell, fooockiinggg shieeeet, to my don’t know how where luck I managed to recover the copiled javascript chunck from my vue app that had the css styles embeded in the file, you know where I found the js file? iN ThE fcking cache of google chrome. Today I almost broke down to tears, but nonetheless it was a reeee moment for me.

It’s still to easy.
I hope one day software will get so complicated no one will be able to fix it.

Somewhere in future :
- government established law that new AI system is only one that can accept new law
- every financial operation is monitored by government supervision AI
- we developed robots that are taking care of us
- everyone is happy cause work for money, shelter and food is now optional
- education is fully digital and managed by AI
- whole knowledge is based on asking questions, we don’t need to write and read anymore
- we use one common language and our knowledge specialization increased

A little more time passed by in this utopia.
- after power loss most of data got corrupted
- last man who knew how to restore backup died last night ( R.I.P. admin we will not forget you )
- people trying to save knowledge base to rebuild part of this civilization but no one knows how to make a paper because it haven’t been used for ages
- we decided to put what is left from knowledge on stone but we forgot how to write since everything is audio or video and most of time we were spending in VR
- someone decided that we draw some pictures
- all of use are now drawing animal heads like we remember ourselves from VR, let people know our tech is good
- some people love cats so they try to make cats from stones
- volcano eruptions destroyed most of stones that we made

Starving waiting for another respawn of my DNA sequence. I hope we manage to survive this time.

Once upon a time, an IT major named adamyeti thought it was a good idea to work on projects directly off of a flash drive with no backup. Halfway through a large ASP.NET project, the drive failed. I fumbled through a free drive recovery tool, but all of the data was scrambled and corrupted.

I ended up having to start from scratch on the project, but I learned my lesson for sure.

!dev

Why do I hate my extended family coming over for lunch and dinner you ask?

> kids, who will ruin the remote by pressing the keys so hard, I'll have to get a new remote.

> NO PEACE. I'll have to move from my comfort spot to another spot where, again kids, will come and ask if I have "GAMES" in my mobile or laptop

> and this happened after lunch while watching a movie which I never watched before, my imbecile cousin decides to spoil the entire movie just like that, like, FUCK YO, LIKE REALLY, I KNOW YOU'RE MY RELATIVE IN SOME WAY, BUT FUUUUUUCK YOUUUUUU, spoilers is one of the things I cannot stand.

> I really do not like to be annoyed again and again and again and again, so please stop asking me if I want to have lunch or dinner with everyone because I really HATE the talk during that time.

> I leave my laptop for one microsecond and they surround it like scavengers, I MEAN FOR FUCKS SAKE, GIVE ME MY PRIVACY, (I have my own room, but it's under renovation).

The best I could do was to put on headsets and pretend like I'm working while browsing LinkedIn.

> "Oh I see you have HD TV, but the picture is blurred" NO SHIT, SHERLOCK, It is due to I chose not to buy HD Pack because I live stream HD Channels and cable is a backup

Updated the site Joomla from 3.6 to 3.7. Then got a call saying some pages weren't working. "No problem," I said, "we have backups." Restored a backup, but then found out it was corrupt... The entire site went down 😭

This day I have received the most glorious news in e-pistolary form. For some years, I was suffering in support of a client who was, well, insufferable. My presence there paralleled the divine comedy in both essence and fact.

I opened the missive, expecting another plea to bail them out of whatever clusterfuck they found themselves in. Instead, what I found was something truly magical.

"Hey Human,

I hope this finds you well. I'm not sure if you remember a few years back, we were trying to decide between IBM Cloud and AWS. Well, after years of battling FF*, we're finally moving ahead with AWS. He failed one too many times to deliver anything visibly. After you left, there was no one left he could use to steal credit, ideas, and work.

FF is still pushing to have them use IBM cloud as a "warm backup" in the event "AWS fails." We will see where that goes.

I figured you'd like to know; you were the void in the wilderness for a long time. I don't want to think about how much time we could have saved if we had just listened.

PeeEm**"

This event represents a personal victory, albeit belated, over a few peoples' absurd amount of privilege. Towards the end, I was vicious about my contestation to the insanity of adopting a desperate hedge attempt-as-cloud offering from a failing company. Some examples:

// cloud 'strategy meeting'
Moi: What cloud platform are we looking at using?
FF: We're looking at IBM cloud and AWS as a second.
Moi: Why is that? I understand you're obligated to rep your offering first, but that decision doesn't seem to have the customer's best interest at heart.
FF: IBM cloud is a market leader; AWS isn't as good.
Moi: I see. I mean, that's the tech equivalent of the company's fleet management considering monkeys on tricycles as a strong competitor to service trucks, but I get what you mean.

// steering meeting
Director: Who can we look to as an example? Who is currently using the IBM cloud?
Moi: No one; they account for a single-digit portion of the actual cloud market. Their long game to sell you a "Hybrid Cloud," which means put some front end payload in a CDN, and buy n-frame units of IBM z servers for the DC with IBM gateway appliances acting as connective tissue. So it's not the cloud at all, really.
Director: How does it compare in cost?
Moi: It's generally 40% more expensive than other clouds, and it only goes higher as you option their software.
Director: What about Watson? I hear Watson is good?
Moi: It's a brand name. Most of the "Watson" product is just a facade on top of FOSS products like Spark, Hadoop, Elasticsearch, etc.
Director: Those were words. They sounded good. FF say it's good tho so we'll believe him because we're from the same city.
Moi: *deletes Director from LinkedIn*

Moral of the story: Never trust a vendor that only recommends their products.

*FF = FatFuck - an embarrassingly rotund individual whose girth is roughly equivalent to his height. He shit his way into an IBM architect position in his mid-20s purely due to winning the visa lottery. He had fake hair glued to his head for his wedding to hide his male pattern baldness; his arrange-married wife undoubtedly cries herself to sleep after sex.

**PeeEm - the then project manager, now portfolio manager of some satellite projects. An overall decent human being, capable.

So, a few months back my mother had some issues with her windows 10 box not being able to do proper backups to a backup partition. At the same time I was pulling insane hours at work and writing on a eBook on commission for a guy, besides having small kids with on and off flu and shit.

Needless to say, I didn't have time to look at the backup issues. Well, even though my mom is one of those dogs you can't teach new tricks, she has always been resourceful enough to get help with things.

This time she picked up the phone and called Microsoft Support, got some guys to remote in and take a look. They messed around a bit and said they were done.

She phoned me up later that day to tell me how proud she was of herself for doing that. Of course, she skipped telling me the important bit about she actually calling them, rather describing it as "Microsoft was just on my computer and fixed it".

You can imagine my immediate reaction, cold sweat running down my back, adrenalin rushing in as I dug through the details of what had happened.

A few days later she calls me up again and tells me the problem is back, and we agree that even though the MS dudes was not able to fix it at first, she should try again, as she had a ticket to reference.

The next attempt by MS actually fried her partitions, and apparently they had f-ed up trying to delete and recreate the backup partition.

That's not the worst of it though. Since they fried her disk, her computer crashed and naturally the remoting won't work. In our country, they have no people on the ground to do hands on help, and they didn't have a partner near by. Her not having a win 10 usb stick, nor a spare computer to make one, she was in a surreal predicament.

She was also quite pissed, and pissed off mums are not to be messed with. She managed to get Microsoft to agree to cover the costs of a non-partner to visit her to fix the problems, and using her as the middle man, they made an agreement with the 3rd-party tech support company.

After the box was fixed though, some more issues arose... regarding billing. The 3rd-party tech support was unable to get in contact with the person at MS that was going to sign off on the bill, and again using my mom as the middle man, it was agreed that my mom, as the customer, was to be reimbursed for the bill to the 3rd-party.

Guess what... 3 months went by, with weekly follow-ups and nagging from my mom, and still no money...

At this time, I had time to help her, and after some digging and borderline stalking, we managed to get the phone numbers of some of the higher ups in my country, and she started calling them directly.

After talking to a couple who refused to help, she reached the Vice President of the country branch, and was finally able to talk to someone who gave a shit.

Still took over a month more to actually get the money, but now she had someone who actually gave her statuses, receipts and ETAs.

FUCK!

Boss wanted me to make changes in company's website which was based on wordpres s.

I knew it could be done by tweaking some JS code, but I have very less experience with wordpress

But wordpress is easy man(Internet told me).

Give me 5 minutes, you will see the changes in production.
Being lazy af I directly logged in to ftp, checked out some files, updated some code, I was good to go.

Before pushing it, I opened the website and it was GONE ٩(๑´0`๑)۶

Now there was no public_html in the root.
I was fucked. I have accidentally deleted the website that had no backup.

And the best part I was on leave from
next day.

I was looking everywhere for backups, looked into google cache to get the contents. I have to recreate the complete site now.

Just when I was asking questions on choice of my profession and simultaneously looking here and there in FTP for backups,
I found the jewel "public_html".

It happens out that I have accidentally moved the folder to some other directory.

Phewww.

Moved it back to root. Site was up and running.

Reassured myself that I deserve to be a dev.

Backed up complete site, made the changes.

Uploaded it.

And the best part, amount of wordpress I learned in those three hours was way more than I could have learnt in many weeks.

Lessons Learnt :

A) ALWAYS keep backups.
B) You SHOULD NOT make changes on prod directly
C) You become superhuman when your brain know you are going to be fucked 😂

Running a wild UPDATE statement against an inventory database, it was chaotic and didn't have backups.... yea I know 😅 the backup service died god knows when and no one noticed. I was only new at the time so #notMyFault

UPDATE stock_on_hand = 0; WHERE id IN(1,2,3,4,5,88,972,7388);
# rows affected 1,234,567,890

> I think I almost died inside.

Oh the fun that mess was to clean up.
The positive outcome of this was, we had backups working again not long after and the inventory counts where accurate after that stock take.

Today’s DevOps public service announcement... don’t test your server provisioning scripts locally. Especially when this gem is in there:

rm -f ~/.ssh/id_rsa
echo ‘vault secret/ssh_key’ > ~/.ssh/id_rsa

Well, I no longer have my key, but the script works! I’m sitting with a very locked down server key

Returning from vacation tmr... But apparently we just had a "someone at github deleted all the repos and no backup"-like prod issue....

Well guess I say least can say "¯\_(ツ)_/¯ not my area, can't help u"

I accidentally deleted my boot loader and now my PC won't start. I tried boot repair but no hope. I don't have a win backup either

I really felt like a badass one time when I managed to recover all projects on our dev server after a full meltdown of the HDD.

We had no recent backups, because our backup server was down for a few months, and our (at the time small) company was in a tight spot on finances, and couldn't get a replacement.

The problem was that the HDD on the backup server failed, but we were storing all projects also on the dev server, along with our local git repos (no GitHub at the time for us), but then the dev server HDD also broke, and I used every piece of data recovery software I found trying to recover the data, until one actually managed to read the raw data from the HDD and store it as a virtual drive, that I then used to try and build another partition index and it actually worked!

Lost about 10% of the data, but that was enough, as i managed to recover all the git repos and databases...

I don't even remember the tools that got the job done in the end, but that was one hell of a week, and at the end I felt like a true IT God!

True story!

PS: 2 weeks later we had a new backup server, another offsite backup solution and a GitHub account for the company. Was delayed on salary in order to manage it (me and the CEO both agreed to give our pay for one month to get them), but worth it!

NewLifeNewHope update No.2 / Day 5

My Server Is finally opened and i installed 2x8 TB WDC Harddrives and SATA 128 SSD. I know this server is freaking garbage, but i got this PC for free, and have somewhat good-ish upgrade path, so heres the spec :
-Gigabyte H110m-S2 LGA1151
-Intel i3-6300
-8GB DDR4 single channel RAM
-128 GB SSD
-2x8TB Harddrives
-TP-Link 1000mbps NIC

so the plan is to make this server as the Main Repository -- yes no offsite backup plan for now -- and also i want to make this server as an Email Server and for hosting my company's website. I've already asked for static IP from my ISP and will take effect tommorow.

I need help for choosing the Operating System (i like centOS) for my server. and to setting this Server to work like what i planned but i don't know where to start, Any help and/or References will be great !

Can someone help me understand?

I subscribed to a nifty IT-releated magazine, and on its back, there's an ad for "Dedicated root server hosting", nothing unusual at a first glance, but after I read the issue, I decided to humor them and see what it is that they offered, and... It just... Doesn't make sense to me!

An ad for "Dedicated Root Server" - What is a dedicated root server first of all? Root servers of any infrastructure sound pretty important.

But, the ad also boasts "High speed performance with the new Intel Core i9-9900K octa-core processor", that's the first weird thing.

Why would anyone responsible enough want to put an i9 into a highly-reliable root server, when the thing doesn't even support ECC? Also, come on, octa-core isn't much, I deal with servers that have anywhere between 2 and 24 cores. 8 isn't exactly a win, even if it has a higher per-core clock.

Oh, also, further down the ad has a list of, seeming, advantages/specs of the servers, they proclaim that the CPU "incl. Hyper-Threading-Technology"... Isn't that... Standard when it comes to servers? I have never seen a server without hyperthreading so far at my job.

"64 GBs of DDR4 RAM" - Fair enough, 64 gigs is a good amount, but... Again, its not ECC, something I would never put into a server.

"2 x 8 TB SATA Enterprise Hard Drive 7200 rpm" - Heh, "enterprise hard drive", another cheap marketing word, would impress me more if they mentioned an actual brand/model, but I'll bite, and say that at least the 7200 rpm is better than I expected.

"100 GBs of Backup Space" - That's... Really, really little. I've dealt with clients who's single database backup is larger than that. Especially with 2x8 TB HDD (Even accounting for software raids on top)

This one cracks me up - "Traffic unlimited"
Whaaaat?! You are not gonna give me a limit to the total transferred traffic to the internet for my server in your data center? Oh, how generous of you, only, the other case would make the server just an expensive paperweight! I thought this ad was for semi-professionals at least, so why mention traffic, and not bandwidth, the thing that matters much more when it comes to servers? How big of a bandwidth do I get? Don't tell me you use dialup for your "Dedicated Root Server"s!

"Location Germany or Finland" - Fair enough, geolocation can matter when it comes to latency.

"No minimum contract" - Oooh, how kiiiind of you, again, you are not gonna charge me extra for using the server only as long as I pay? How nice!

"Setup Fee £60" - I guess, fair enough, the server is not gonna set itself up, only...

The whole ad is for "monthly from £55.50", that's quite the large fee for setup.

Oh, and a cherry on top, the tiny print on the bottom mentions: "All prices exclude VAT and are a subject to..." blah blah blah.
Really? I thought that this sort of almost customer deceipt is present only in the common people's sphere!

I must say, there's being unimpressed, and then... There's this. Why, just... Why? Anyone understands this? Because I don't...

I think I made someone angry, then sad, then depressed.

I usually shrink a VM before archiving them, to have a backup snapshot as a template. So Workflow: prepare, test, shrink,