My Friend: Dude our Linux Server is not working anymore!

Me: What? What did you do?

My friend: Nothing I swear!

Me: But you were last on it?

My friend: Yes. I just wanted to run a bash file and needed to give it permissions.

Me : WHAT DID YOU ENTER???!

My Friend: Chill man, just this command I found on the internet
chmod -R 600 /
chown -R root:root /

Me: WHY ARE YOU EVEN IN ROOT AND GOD DAMMIT WHY ARE YOU EVEN USING SOME RANDOM COMMAND FROM THE INTERNET. YOU KNOW YOU SHOULD NOT DO THIS OR JUST ASK!

My friend: Ok I did something wrong, how can I fix it?

Me: Did you make a backup or rsync of the server?

My friend: No. I just wanted to run this file.

Me: You holocausted the server. FUCK MY LIFE

Things I've learned throughout my 5 - 6 years as a programmer.
- StackOverflow is full of assholes.
- CMS's are for weaklings.
- The best feeling about waking up in the morning is figuring out how to solve that error in your code.
- You no longer think about normal people things. Your mind is full of code.
- You're practically a computer.
- ALWAYS backup and save your stuff or you WILL regret it. Enable autosave if possible.
- RIP your social life (if your friends don't know squat about programming)
- Darkness is better.
- Being a programmer is amazing.

Mac: Suddenly turns off
Me: Fuck my code..F***
Mac: No response at all
Me: reset SMC etc etc
Mac: I am dead (no battery detection, dies after 10 min on power adaptor)
Me: Skips a heart beat..(Git, oh yea git)

**Takes Mac to store, After diagnosis**

Apple Freaking Genius (AG): Your Mac has a mother board problem it needs to be replaced.
Me: Hmm what is the problem exactly??
AG: Issues in logic board and some other components.
Me: How much?
AG: Out of warranty so $$$ (60% of original amount)
Me: (wtf?) Really
AG: It's entire motherboard replacement .. bla bla

**Bring it home > open > everything seems ok on multimeter as per circuit diagram > finally finds a voltage drop that is not consistent > minute short circuit > remove > check further > nothing else > reassemble > hit power button > starts fine > freaking battery detected > works fine**

0 $ repair

Fixes two more devices @ 0 $ in friend circle

Builds a raspberry pi backup laptop with 3d printed body..Ubuntu.. you know can't live without a computer

#ThugLife #Engineer

CEO: You need to use git.
Designer: No

CEO: You can backup stuff.
Designer: No

CEO: You don't have to save copies!
Designer: No

CEO: You will use a really cool command line!
Designer: No

CEO: It has unlimited storage
Designer: So when we start?

So my Girlfriend bought a new iPhone at Verizon today. Cool story, I know, but here's where it's gone from there.

Firstly her debit wouldn't run as credit, so we used mine but that's the least of it (but began it).

So she has 16,000 photos... Alot, sure, but not the issue. Obviously with that amount of data she wasn't about to reasonably use iCloud to back it up (understandable only by me) so she was confronted both by me and the Verizon employee about this issue to where we both (the Verizon employee and I) agreed that an iTunes backup/restore was the only way to preserve her data. She was confused. No worry, told her I had it handled and the Verizon employee agreed. Great. Yet we get home and begin the process. My girlfriend was not on the latest iOS (understandable given the battery scenario and she was on an iPhone 6) and this was ridiculous to her because she had to update in order to do the iTunes back up. Whatever, I brushed it off. Her phone was updated, and backed up... Which took a while but we are talking 30gb (of which she had no understanding of how much that was). After the back up we discovered her new phone wasn't working due to a bad sim, great, no problem we have the old one... But oh no. "I don't want that shitty old sim" she said. Uhmm what... I say, and say let me get an earring (to switch the Sims) and she gave one to me and as soon as I went to pop the tray, she had a fucking heart attack as if I was demolishing her phone. I talk her down, get it switched, get the phone to restore (slow process as she's complaining... 30gb mind you) and it works. She goes to bed. Comes back, texts aren't working. I say imessages or texts (now she has no idea) I troubleshoot, seems nothing's working, and that's okay Verizon must of reinstated the new sim and deactivated the old (fine). I switch them and it works. She proceeds to berate me about the SIM cards because she didn't want the 'old shitty one' (the one that got us to the place of a functioning phone).

Now everything works and she claims a Genius bar employee would of done this in minutes.

I (obviously) lose my shit, now I'm sleeping on the couch.

Im an IT professional / programmer..... this shit really ticked me off.

A friend of mine: Look at this nice app I made 😎
*shows me a normal looking app with tabs and different views"
Me: Hmmm… looks good.
Friend: Do you know how I made the navigation between views? 😏
Me: I'm afraid to ask…
Friend: Since I didn't know how to use the piece of code I found on internet, I made multiple views one above another with visibility set to false, and during the navigation I just change the visibility. 😁
Me: WTF? Man… this isn't something you should be proud of… it shouldn't be done.
Friend: Do you also want to know how I backup the code?
Me: No…
Friend: I send it to myself via email.
Me: FUCK!
Friend: I also never use while loops, I prefer to use for instead with a break inside. 😊
*blocked*

Facebook publicly announced that it won't build a backdoor into its services for the intelligence agencies as for the latest requests to weaken/remove the encryption.

I can only imagine the intelligence agencies going like this now:

NSA director: Alright, as expected they said no so they won't have more damage to their public image, lets go for plan A 2.0!
NSA employee: Aaaand that is?
NSA director: Serve them a FISA court order requiring them to do this shit anyways but also serve a gag order so they can't tell legally.
NSA employee: Ahh, fair enough, I'll get that rolling. By the way, how did we do this with WhatsApp's encryption again?
NSA director: Oh that one was simple. There's a backup function which nearly everyone uses on either Android/iOS which does plaintext backups to Google Drive/iCloud.
NSA employee: Oh, okay. How do we access that data again?
NSA director: PRISM/XKeyScore!
NSA employee: Right, but then still the issue of how we even collect the encrypted messages from Facebo...
NSA director: PRISM/XKeyScore as well, don't worry about that.
NSA employee: But, how'd we justify this....?
NSA director: We probably never have to as these programs operate outside of the public view but otherwise just call terrorism/pedophelia... BAM, done.
NSA employee: Gotya, let's put this into motion!

And here comes the last part of my story so far.
After deploying the domain, configuring PCs, configuring the server, configuring the switch, installing software, checking that the correct settings have been applied, configuring MS Outlook (don't ask) and giving each and every user a d e t a i l e d tutorial on using the PC like a modern human and not as a Homo Erectus, I had to lock my door, put down my phone and disconnect the ship's announcement system's speaker in my room. The reasons?
- No one could use USB storage media, or any storage media. As per security policy I emailed and told them about.
- No one could use the ship's computers to connect to the internet. Again, as per policy.
- No one had any games on their Windows 10 Pro machines. As per policy.
- Everyone had to use a 10-character password, valid for 3 months, with certain restrictions. As per policy.
For reasons mentioned above, I had to (almost) blackmail the CO to draft an order enforcing those policies in writing (I know it's standard procedure for you, but for the military where I am it was a truly alien experience). Also, because I never trusted the users to actually backup their data locally, I had UrBackup clone their entire home folder, and a scheduled task execute a script storing them to the old online drive. Soon it became apparent why: (for every sysadmin this is routine, but this was my first experience)
- People kept deleting their files, whining to me to restore them
- People kept getting locked out because they kept entering their password WRONG for FIVE times IN a ROW because THEY had FORGOTTEN the CAPS lock KEY on. Had to enter three or four times during weekend for that.
- People kept whining about the no-USB policy, despite offering e-mail and shared folders.
The final straw was the updates. The CO insisted that I set the updates to manual because some PCs must not restart on their own. The problem is, some users barely ever checked. One particular user, when I asked him to check and do the updates, claimed he did that yesterday. Meanwhile, on the WSUS console: PC inactive for over 90 days.
I blocked the ship's phone when I got reassigned.
Phiew, finally I got all those off my chest! Thanks, guys. All of the rants so far remind me of one quote from Dave Barry:

That moment when your boss is introducing the new guy and you are silently making a list of all the git repos you need to back up before he makes it to his desk.

PROBLEM: A tickets' company came to us last autumn. They said they have severe performance problems and asked us to help.
SOLUTION: covid and quarantines. All events have been ceased, noone's buying any tickets any more. Performance problems are no more. FIXED.

PROBLEM: Another company came to us recently. They said they have severe performance problems with their huge databases and asked us to help.
SOLUTION: a few days of heavy rain and their datacenter was flooded. along with the backup servers. No more data, no more performance problems with large databases. FIXED

Solving problems genie style!

Who's next?

1. I join a company.

2. I get deeply involved in "how to run the company", and get nice compliments from both coworkers & management about my skills in conveying startup/scaleup advice & necessities to upper management.

3. With my ego inflated through all the sweet talk, I think "ah, what the hell, let's do this again", and I accept a Lead/CTO promotion. I have to join board meetings, write reports on quarterly plans and progress.

4. I get unhappy/stressed/burned-out because I really just want to be a developer, not a manager/executive.

5. Upper management understands, I give up my lead position, lock myself back into my coding cave.

6. I get annoyed because the requirements I receive become more and more disconnected from reality, half of the teams seem to have decided to stop using agile/scrum, the testing pipeline breaks all the time, I get an updated labor contract from HR by mail which smells like charred flesh, etc

7. The annoyances become too much to do ANY work. I yell at the other devs outside of the entrance of my cave. There is no answer, only a few painful moans and sighs.

8. I emerge from my cave. The city has turned into a desolate wasteland. The office is a burning ruin, the air sharp and heavy with black soot. Disemboweled corpses of developers litter the poisoned soil.

Product Managers dressed in stained ripped suits scream at each other while they try to reinforce concrete barricades with scotch tape and post-its. *THUMP* Something enormous is trying to break through. "Thank God, bittersweet, you're still alive! The stakeholders! They have mutated! We couldn't meet the promised deadlines! We've lost the whole mobile app department, and that kid there is the last of the backenders and he's only an intern! You're here to save us, right? RIGHT?".

In the corner, between the overflowing coffee machine and a withered cactus, a young boy has collapsed onto the floor. His face is covered in moldy coffee grounds, clasping on to his closed macbook for dear life, wide-open eyes staring into the void, mumbling: "didn't backup the database, and It's all gone" over and over.

A severely dented black Tesla with a dragging loose bumper breaks through the dried up vertical herb garden and the smoothiebar, and comes to a halt against the beanbags in a big cloud of styrofoam balls.

The CEO limps out, leaking blood all over the upholstery. He yells to the COO: "The datacenter is completely flooded with sewage! I saved the backup tapes though", holding a large nest of tangled black magnetic tape mixed with clumps of mud above his head.

9. I collect my outstanding salary and sell any rewarded options/shares for a low dumping price, take a 5 month holiday, and ask a recruiter about opportunities in a different city.

The website i made has been hacked today.

Stored in their server.

They didnt give me an access for it.

The user account in the cms i used for updating content while building the website was revoked when the website is completed.

Now they ask me for the latest backup.

I have no backup because how the hell i do a backup when i got no access to the cpanel.

The only backup is the zip file for initial uploading into their server and the contents were added after the website is on their server.

That goddamn IT guy who wont give me any access for “securty sake” is calling me furiously asking for the backup and how to set up the stuffs from the beginning.

I thought he was the one who know his shit but i was wrong.

Fuck me?

No.

Fuck you.

But i still responding to him telling him step by step how to do shit with some swearing and sarcasm.

ALWAYS BACKUP YOUR SHITS, MATE

I think I've shown in my past rants and comments that I'm pretty experienced. Looking back though, I was really fucking stupid. Since I haven't posted a rant yet on the weekly topics, I figure I would share this humbling little gem.

Way back in the ancient era known as 2009, I was working my first desk job as a "web designer". Apparently the owner of this company didn't know the difference between "designer", which I'm not, and "developer", which I am, nor the responsibilities of each role.

It was a shitty job paying $12/hour. It was such a nightmare to work at. I guess the silver lining is that this company now no longer exists as it was because of my mistake, but it was definitely a learning experience I hold in high regard even today. Okay, enough filler...

I was told to wipe the Dev server in order to start fresh and set up an entirely new distro of Linux. I was to swap out the drives with whatever was available from the non-production machines, set up the RAID 5 array and route it through the router and firewall, as we needed to bring this Dev server online to allow clients to monitor the work. I had no idea what any of this meant, but I was expected to learn it that day because the next day I would be commencing with the task.

Astonishingly, I managed to set up the server and everything worked great! I got a pat on the back and the boss offered me a 4 day weekend with pay to get some R&R. I decided to take the time to go camping. I let him know I would be out of town and possibly unreachable because of cell service, to which he said no problem.

Tuesday afternoon I walked into work and noticed two of the field techs messing with the Dev server I built. One was holding a drive while the other was holding a clipboard. I was immediately called into the boss's office.

He told me the drives on the production server failed during the weekend, resulting in the loss of the data. He then asked me where I got the drives from for the Dev server upgrade. I told him that they came from one of the inactive systems on the shelf. What he told me next through the deafening screams rendered me speechless.

I had gutted the drives from our backup server that was just set up the week prior. Every Friday at midnight, it would turn on through a remote power switch on a schedule, then the system would boot and proceed to copy over the production server's files into an archive for that night and shutdown when it completed. Well, that last Friday night/Saturday morning, the machine kicked on, but guess what didn't happen? The files weren't copied. Not only were they not copied, but the existing files that got backed up previously we're gone. Why? Because I wiped those drives when I put them into the Dev server.

I would up quitting because the conversation was very hostile and I couldn't deal with it. The next week, I was served with a suit for damages to this company. Long story short, the employer was found in the wrong from emails I saved of him giving me the task and not once stating that machine was excluded in the inactive machines I could salvage drives from. The company sued me because they were being sued by a client, whose entire company presence was hosted by us and we lost the data. In total just shy of 1TB of data was lost, all because of my mistake. The company filed for bankruptcy as a result of the lawsuit against them and someone bought the company name and location, putting my boss and its employees out of a job.

If there's one lesson I have learned that I take with the utmost respect to even this day, it's this: Know your infrastructure front to back before you change it, especially when it comes to data.

Storytime!

This customer comes in and practically throws a computer on the counter.

Customer: This computer isn't working. I've ran the diagnostics and it says it's software. *places a dvd case with a 32 bit Windows 7 disk in it on the counter* It had Windows 10 on it, but I want Windows 7 on it.

Me: Well, you may have issues with the drivers if you put Windows 7 on it--

Customer: I don't care, I just want Windows 7.

Me: You SHOULD care. That means no wifi, no display, no mouse... Windows 7 doesn't like Windows 10 hardware.

Customer: Then... check to see Windows 7 compatibility!

Me: Alright.... *makes notes to check for Windows 7 compatibility*

Me: So has this Windows 7 been used before?

Customer: Yes, it has.

Me: On how many computers?

Customer: I've installed it on two computers and it works just fine.

Me: That's weird because Windows license keys are for one computer only. Are both of them connected to the internet?

Customer: Yes.

Me: Well, okay then... *finishes up ticket*

Customer: I work in this field and I just don't understand why they don't come with the disks anymore. How much is a Windows 10 disk?

Me: *gives price*

Customer: And do you have any?

Me: Let me check *I go to where they are, find some and come back out*

Me: Unfortunately we're out at the moment and would have to special order some back in.

Customer: OK. So then how much to fix this computer?

Me: *price of installing Windows and backing up data*

Customer: That's halfway to the price of a new one of these!

Me: Well yes, an HP at Walmart... But you do have that option if you want to take it.

Customer: Well, why does it cost that much?

Me: Well, it's $labor1 to install Windows, $labor2 to do some basic setup and drivers, and $labor3 to backup and restore data.

Customer: Oh, well I don't want data.

Me: Okay, well then it would be $total - $labor3

Customer: ...Okay, fine

Me: *updates the ticket*

When she finally left I put it on the bench and the first message said "SMART ERROR." I then did 4 different tests that said "lol, the hard drive is failing."

If you "worked in this field," you would know that a SMART error is hard drive related.

If you worked in this field, you would know that Windows is only a 1PC license, so why are you lying about installing it with no issues on other computers?

If you worked in this field, you would know you would want a 64bit Windows on your computer.

If you worked in this field, you would know how to find a Windows 10 installation media online.

If you worked in this field, you would know that HPs are not good computers to get.

IF YOU FUCKING WORKED IN THIS FIELD YOU WOULDN'T BE SUCH A FUCKING CUNT.

There's one kind of ticket I fucking love. Wait, YOU LOVE TICKETS?! No, except for one.

Tickets where people ask us to restore backups.

Why do I love them?

Because the only fucking thing I need to do is reply with a link to a helpdesk item on our website because backups aren't something within our support range because it's easy for customers to do.

So whenever a ticket about restoring a backup comes in, it's as easy as opening the ticket, pasting the link with "I'd like to refer you to the following resource: " and pressing fucking "reply"

😄

Me: I'm unable to connect through VPN

IT guy at my office: I'll format your system and reinstall OS

Me: heck no!! There should be other effective solution.

IT guy: Yes, I can take backup of your files and then reinstall OS.

Me: Just get out!! F*ckin ididot!!

At my previous job, the person in charge of the Phabricator server didn't have a backup system in place. I yelled at him until he implemented one.

He had the server perform backups to the same drive. I yelled at him again, to no avail.

Well, after awhile the hard drive started failing, and it would only boot intermittently. After a lot of effort, he was able to salvage part of the backup data, but no more, meaning we lost a lot of bug reports and feedback, and developer tickets. We were able recover all of the older lost tickets from a previous server, so overall the loss was pretty small.

But I think he learned his lesson.
He definitely learned to listen.

Once I had to clone a repo and it was taking too long...

Went to gitlab to find out in the most wtf way that from 1mb it was 600mb+ now...

One of our new juniors pushed 600mb+ of a database backup to gitlab...

I came to her with a smile and asked in a jokingly manner (after cursing her for about half an hour in my head):

"lol, did you really not notice it took a fucking long time to upload it?"

The fucker was ashamed but just said : no, I think I pushed it and went home.

I constantly reminded her of it for the couple months...

Never done it again :)

[This makes me sound really bad at first, please read the whole thing]

Back when I first started freelancing I worked for a client who ran a game server hosting company. My job was to improve their system for updating game servers. This was one of my first clients and I didn't dare to question the fact that he was getting me to work on the production environment as they didn't have a development one setup. I came to regret that decision when out of no where during the first test, files just start deleting. I panicked as one would and tried to stop the webserver it was running on but oh no, he hasn't given me access to any of that. I thought well shit, I might as well see where I fucked up since it was midnight for him and I wasn't able to get a hold of him. I looked at every single line hundreds of times trying to see why it would have started deleting files. I found no cause. Exhausted, (This was 6am by this point) I pretty much passed out. I woke up around 5 hours later with my face on my keyboard (I know you've all done that) only to see a good 30 messages from the client screaming at me. It turns out that during that time every single client's game server had been deleted. Before responding and begging for forgiveness, I decided to take another crack at finding the root of the problem. It wasn't my fault. I had found the cause! It turns out a previous programmer had a script that would run "rm -rf" + (insert file name here) on the old server files, only he had fucked up the line and it would run "rm -rf /". I have never felt more relieved in my life. This script had been disabled by the original programmer but the client had set it to run again so that I could remake the system. Now, I was never told about this specific script as it was for a game they didn't host anymore.
I realise this is getting very long so I'll speed it up a bit.
He didn't want to take the blame and said I added the code and it was all my fault. He told me I could be on live chat support for 3 months at his company or pay $10,000. Out of all of this I had at least made sure to document what I was doing and backup every single file before I touched them which managed to save my ass when it came to him threatening legal action. I showed him my proof which resulted in him trying to guilt trip me to work for him for free as he had lost about 80% of his clients. By this point I had been abused constantly for 4 weeks by this son of a bitch. As I was underage he had said that if we went to court he'd take my parents house and make them live on the street. So how does one respond? A simple "Fuck off you cunt" and a block.

That was over 8 years ago and I haven't heard from him since.

If you've made it this far, congrats, you deserve a cookie!

What an awful day :(

The server where I host my 4 clients websites crashed.

Unable to reboot from the console.

I contact the support. 15 minutes later: "we'll look at this"

No news for 1 week despite my messages.

Then... 1st ticket escalation... 2nd ticket escalation... 3rd ticket escalation...

Answer: "Sorry, your server is down and cannot be repaired."

Fuck.

I ask "is there any way to get my data back?". Answer: "No, because we would shutdown the whole bay and all our clients would be impacted".

Fuck.

I subscribe to another server, at another provider.

I look at my backups... shit, the last one is 4 month ago!!

I restore the first website: OK

I restore the second website: OK

I restore the third website: My new server is "too recent" and not compatible. with this old Wordpress. Fuck! I'll look at this later...

I restore the fourth website: database is empty!! What??? I look at the SQL backup for this site... it failed...

I lost ALL my 4th client data!!!

I'm sooooo piece of crap!

Fuck the memes.

Fuck the framework battles.

Fuck the language battles.

Fuck the titles.

Anybody who has been in this field long enough knows that it doesn't matter if your linus fucking torvalds, there is no human who has lived or ever will live that simultaneously understands, knows, and remembers how to implement, in multiple languages, the following:

- jest mocks for complex React components (partial mocks, full mocks, no mocks at all!)
- token cancellation for asynchronous Tasks in C#
- fullstack CRUD, REST, and websocket communication (throw in gRPC for bonus points)
- database query optimization, seeding, and design
- nginx routing, https redirection
- build automation with full test coverage and environment consideration
- docker container versioning, restoration, and cleanup
- internationalization on both the front AND backends
- secret storage, security audits
- package management, maintenence, and deprecation reviews
- integrating with dozens of APIs
- fucking how to center a div

and that's a _comically_ incomplete list; barely scratches the surface of the full range of what a dev can encounter in a given day of writing software

have many of us probably done one or even all of these at different times? surely.

but does that mean we are supposed to draw that up at a moment's notice some cookie-cutter solution like a fucking robot and spit out an answer on a fax sheet?

recruiters, if you read this site (perhaps only the good ones do anyway so its wasted oxygen), just know that whoever you hire its literally the luck of the draw of how well they perform during the interview. sure, perhaps some perform better, but you can never know how good someone is until they literally start working at your org, so... have fun with that.

Oh and I almost forgot, again for you recruiters, on top of that list which you probably won't ever understand for the entirety of your lives, you can also add writing documentation, backup scripts, and orchestrating / administrating fucking JIRA or actually any somewhat technical dashboard like a CMS or website, because once again, the devs are the only truly competent ones - and i don't even mean in a technical sense, i mean in a HUMAN sense of GETTING SHIT DONE IN GENERAL.

There's literally 2 types of people in the world: those who sit around drawing flow charts and talking on the phone all day, and those WHO LITERALLY FUCKING BUILD THE WORLD

why don't i just run the whole fucking company at this point? you guys are "celebrating" that you made literally $5 dollars from a single customer and i'm just sitting here coding 12 hours a day like all is fine and well

i'm so ANGRY its always the same no matter where i go, non-technical people have just no clue, even when you implore them how long things take, they just nod and smile and say "we'll do it the MVP way". sure, fine, you can do that like 2 or 3 times, but not for 6 fucking months until you have a stack of "MVPs" that come toppling down like the garbage they are.

How do expect to keep the "momentum" of your customers and sales (I hope you can hear the hatred of each of these market words as I type them) if the entire system is glued together with ducktape because YOU wanted to expedite the feature by doing it the EASY way instead of the RIGHT way. god, just forget it, nobody is going to listen anyway, its like the 5th time a row in my life

we NEED tests!
we NEED to know our code coverage!
we NEED to design our system to handle large amounts of traffic!
we NEED detailed logging!
we NEED to start building an exception database!

BILBO BAGGINS! I'm not trying to hurt you! I'm trying to help you!

Don't really know what this rant was, I'm just raging and all over the place at the universe. I'm going to bed.

So, basically i am getting desperate, and i'm also angry... and want to cry, and i feel a failure.
My biggest error in this story is "believing"

First of all, I'm a starting entrepreneur as freelancer, started of 6 months ago, back then it all seemed bright, i had my first customer, they believed me, got a second assignment for that same customer, fulfilled the task in brilliance, and was ready to move on to a bigger customer base. Here's where shit goes wrong.

Working with another office that outsources people to various (goverment) contractors, Had a meeting with them, we would cooperate very soon. This was january.

PM: "We've got a gov Dept as a customer that wants to do project XYZ and starting in february, requirements are yadiyadiyadi, you up for it?
Me:"Sure, send me the specs, and timeframe and i'll apply with my offer)"
Project is about a bit backup system migration, lots of fancy shmancy tech used, 2 datacenters setup... a big project that can take up several months...
- sends offer- received an acceptance on the offer

PM " Great, we'll start end of february"
Me: "Ok, end of february is a go then, looking forward to cooperate"

FF to near end of february, receive msg from PM: "Project XYZ delayed until half if march"
Me: "Okay... what is the delay?"
PM:"Govt bureaucracy"
Me " Ok, let's keep in touch"

Near Half of March
PM " Project delayed again".
Me: " Okay... what's the reason?"
PM: "they have decided to take a different approach, and want 3 datacenters now"
Me: "That will change the offer i made for you, can you send me the specs?"
PM: "No, because they havent decided on the techs used, expect end of march"
Me: "Okay...but once you got the specs,s end them to me"

End of March
Pm: " Hy NeatNerdPrime, we still havent got any specs yet... they still forgot to make the shopping list"
Me: "(-_-) ... I thought they already had that covered"
PM:" No, new Govt budget cuts and lots of changes, basically, they still don't know what they really want. But we're just -delayed- , not -cancelled-"
Me: "Okay... i hope we get started soon"
PM: "expect launch beginning of april"
I was not really satisfied with this explanation.... but ok

Beginning of april, at april's folls day
PM "Project delayed again"
Me: " This is a joke right?"
PM: "Sadly, no, they delayed the project since they don't really know what they actually want, we are trying to give them the proposed solution... but they still need approval, and still need to set up shopping list"
Me:"when do we start then?"
PM: " I was told 17th of april'
Me:" I really hope it gets through, i had to decline some proposals for work for months due to this, this ain't funny"
PM: "I know, i'll make up for it"

15th of april...
PM " Project delayed again"
Me right now almost losing my shit
"why this time??? I thought you said 17th of april real launch!"
PM: "they fired their PM, need to hire a new one, delayed until May/June"
Me: "I've set aside some other assignments just because you said you were going to launch at those dates... This is really pushing my limits, can you give me SOME assurances?"
PM: "5th of june would be official go" Me: " Okay, i'm noting 5th of june in my agenda, let's get this through!"

And now i get a message...stating that the date of 5th of june , is for another project, at the same goct dept, but totally different project

WHAT THE FUCKING SHIT I'VE SPENT MONTHS WAITING FOR A PROJECT THAT WAS PRACTICALLY ALL SET AND GO AND EVERY SINGLE FUCKING TIME IT GETS DELAYED, EVERY FUCKING TIME I THINK "you know, i cannot accept these offers since that Govt project will start soon, i cannot do 2 assignments at once especially when they require me to work at govt office at the capital..." AND EVERY FUCKING TIME IT GETS DELAYED!!!

I feel at a loss now..... i've done i think the most horrible thing you can do as a independent and that's not accepting another assignment just because there was a concrete promise for a govt contract...

Almost dried up, nothing much left, had to do some spending because of a move to another apartment... I'm feeling really down, and angry...and down, but mostly angry, for not accepting those offers in the meantime...

Never in my life I was scared as today.
I recently left a big company to work for a small one as the first internal developer.
Had a small issue in the production server. The fix was easy, just remove a single table entry. And... *drum roll*... I forgot to add a where clause. All orders were lost.
No idea if we had backups or anything, I quickly called the one other IT dude in the company.
He had no clue where are the backups and how to find them.
Having some experience with Nmap, I quickly scanned our network and found a Nas device.
There was a backup, whole VHD backup. 300GB of it, the download speed is around 512kb/s. No way I can fix it before management finds out, but then an idea came to mind. Old glorious 7zip. Managed to extract only the database files, sent them to the server and quickly swapped them. Everything was fine... The manager connected 5 minutes later. Scariest 45 minutes of my life...

Today my manager asked me about my research into using RabbitMQ as a backup in case Azure Service Bus ever goes down.

Me: "Good. The way we designed the framework, all we have to do is drop the DLLs into the directory, update the config, and the services will start using RabbitMQ."
Mgr: "Excellent. Probably should be looking into using RabbitMQ as a permanent replacement for Azure"
Me: "What? The whole reason we moved to Azure was to eliminate the problems with having an on prem service bus. Since we've switched, there has been zero downtime."
Mgr: "That's what VP-Joe is afraid of. If Azure ever goes down, he won't know how to explain Azure to the president as to why we're not taking orders or can't ship packages."
Me: "That makes no sense. What did VP-Joe tell the president when a database goes down or a server mis-configuration?"
Mgr: "President understands internal outages, its just the whole 'cloud' thing he doesn't understand."
Me: "Um..then VP-Joe needs to explain it to him?"
Mgr: "The decision has already been made. Are you on board? Lets look at this move as a cost savings."
Me: "You mean the $10 a month? How much hardware will we need to support RabbitMQ?"
Mgr: "Yea, nobody probably thought of that."
Me: "I'm on board with whatever decision, but I'd like a little more than VP-Joe being afraid of the president."
Mgr: "I'm sure its not being afraid."
Me: "..."
Mgr: "OK, lets wait and see if VP-Joe forgets about this and moves on to something new."

You know side projects? Well I took on one. An old customer asked to come and take over his latest startups companys tech. Why not, I tought. Idea is sound. Customer base is ripe and ready to pay.

I start digging and the Hardware part is awesome. The guys doing the soldering and imbedded are geniuses. I was impressed AF.

I commit and meet up with CEO. A guy with a vision and sales orientation/contacts. Nice! This shit is gonna sell. Production lines are also set.

Website? WTF is this shit. Owner made it. Gotta give him the credit. Dude doesn't do computers and still managed to online something. He is still better at sales so we agree that he's gonna stick with those and I'll handle the tech.

I bootstrap a new one in my own simplistic style and online it. I like it. The owner likes it. He made me to stick to a tacky logo. I love CSS and bootstrap. You can make shit look good quick.

But I still don't have access to the soul of the product. DBs millions rows of data and source for the app I still behind the guy that has been doing this for over a year.

He has been working on a new version for quite some time. He granted access to the new versions source, but back end and DB is still out of reach. Now for over month has passed and it's still no new version or access to data.

Source has no documentation and made in a flavor of JS frame I'm not familiar with. Weekend later of crazy cramming I get up to speed and it's clear I can't get further without the friggin data.

The V2 is a scramble of bleeding edge of Alpha tech that isn't ready for production and is clearly just a paid training period for the dev. And clearly it isn't going so well because release is a month late. I try to contact, but no reaction. The owner is clueless.

Disheartening. A good idea is going to waste because of some "dev" dropping a ball and stonewalling the backup.

I fucking give him till the end of the next week until I make the hardware team a new api to push the data and refactor the whole thing in proper technologies and cut him off.

Please. If you are a dev and don't have the time to concentrate on the solution don't take it on and kill off the idea. You guys are the key to making things happening and working. Demand your cut but also deserve it by delivering or at least have the balls to tell you are not up for it.

Designer: Need to file a bug, I'm not getting an option to login with FaceID

Me: Oh weird bug. Is it setup on the phone you are testing with?

Designer: yes, use it in all other apps

Me: Did you get an error during onboarding on the FaceID screen?

Designer: nope no error

Me: ..... hhhmm, can you show me your settings?

Me: ... eh, says you have FaceID disabled for this app ... did you click "No" to FaceID during onboarding?

Designer: Yes, to test edge cases

Me: ................ ok ........ if you setup the app and told it to not allow FaceID to login ......... you won't get the ability to use FaceID to login .......... like .... by design .... on purpose ...... cause .... you told it to do that

Designer: No no, it needs to have a setting on the login screen to allow me to turn that back on incase I forget my passcode

Me: the fuck it does. Yeah we can't have anything on the login page that says, without authorization, change my settings

*Deep breath*

Me: Remember we had this conversation previously, where you didn't want the user to create a passcode during onboarding as it was too much friction, and wanted to do FaceID only. With your backup plan being to allow the user to create a NEW passcode on the login screen if FaceID failed .... remember that discussion we had about security? ... and how its important? ... and that we like having any? Ok so its the same reason as that, just with a different setting this time

Designer: ... hhmm i'm not sure I like this

Me: ... tough luck then, not happening

Me: oh and btw, remember we had that other talk about reproduction steps for bugs? Like when the app crashed and you told me it was because its in light mode, and nothing else at all? So disabling FaceID, is very relevant info to the problem of "I can't login with FaceID", please tell me these things first

Much-security.nl is down.

No, no hack or whatsoever. I just reinstalled the wrong server through my control panel.

The new blog version will hopefully be up tonight. 'you keep promising that' - I don't have a backup of the old version and I only just started using git so I can only upload the new version (or git pull). Next to that, except for the front-end, everything works now.

😐

Day 1 10:00 am
Login to email account (Zimbra)
Your password is incorrect (I entered it correctly, this was a permanent issue ,used to happen in the company with many employees)
Reset your password by logging into internal company portal.

11:00 am
Logged into company portal, somehow. 2 Mbps internet shared among 104 people, you can imagine the speed.
Reset email password
* your password has been sent to your email id*
Are you fucking kidding me? U have emailed me the password to the same email I can't log in to?
Where did the architecture designer get this top notch weed from?

Day 2
Asked HR to reset my password (using a colleague's email)

Day 3
No reply from HR yet

Day 4
I went to meet HR, she's on vacation. So they have 1 person managing the password reset, for 5000 people with no backup person. Cool.

Day 5
Your internal company password has expired. Check your email for link to create new password. This is some next level shit going on.

Day 6
I called up Internal IT team to generate a new email for me.
They asked me to raise a ticket.
I can't raise a ticket because the only way to do so, is through the portal.

Day 7
Nothing. Btw, personal email and all social networks were banned. You can't even open stackoverflow.
And this was a research lab, amazing huh?

Day 8
Loss of pay for 4 days since I can't login to company portal to fill timesheet.

Day 9
HR comes back. Resets my password.
I try to generate my new password for portal.
The password policy:
Password can't be same as last 10 passwords
Passwords expire every week
8 characters minimum, 2 upper case, 2 lower case, NO SPECIAL SYMBOL. WTF. How long do u think its gonna take to crack that?

Fuckers had a company wise policy to automatically lock PC every 1 min if not used. Who the fuck can keep on using it continuously! I'm reading an article, and bam ! Locked. 2 wrong entries and that's it, repeat all steps again. Fuckers really didn't want to let me do my job, just keep on logging in all day.

CTOs be like:
We don't need no backup station,
We don't need no source control!

Weekend so far:

Chrome Update FUCKED UP my website.

Tried to update my server to Ubuntu 16.04. That FUCKED UP in the middle and I didn't have any recent backup.

Went back to old backup. But didn't see any changes in the website. Was wondering that for 1 hour.

Forgot that my website was using cloudflare caching. In the meantime I have changed my DNS settings.

Out of frustration removed website from cloudflare. That FUCKED UP the DNS further.

Now I have no idea how long it will take the DNS to update.

FUCKING F M L

2019 resolutions/goals recap: (non-personal ones)
1) Improve diet (did; e.g. ramen and fast food to clean keto)
2) Lose weight (did; lost 24 pounds!)
3) Find a good job (did, twice)
4) Buy a harp (did not; large and expensive, no place to put it, and I have small children who would absolutely break it)
5) Keep house clean, even if it's by myself (did, somewhat; I cleaned some, managed to get one other person to clean semi-regularly, and another sporadically)
6) Work on social awkwardness (did; read and applied Dale Carnegie's The Art of Public Speaking, which netted me my last job offer. Still pretty awkward though)
7) Move out of the desert (did not; not enough money, and job didn't allow remote work)
8) Stop bloody waiting on people (did not; still very guilty of this...)

I don't remember the rest 🙁 didn't write them down last year. But I still accomplished 5 out of the 8 I remembered, with one being a pass, so 5/7!

-----

2020 resolutions/goals:
1) Finally move out of the desert
2) Invest 20% of my income every month
3) Reduce bills by 20%
4) Solve/address some health issues
5) Make a schedule so things regularly get done around the house, e.g. cleaning
6) Find some friends and make time for them
7) Replace Debian with something else
8) Revamp my backup system
9) Be proactive and stop waiting on people
10) Build a (stationary) coil gun for fun

Back in my teenage , a friend of mine asked me «Can you make me a software that guesses the result of a football match ?» I said «Sure, but you have to tell me how to calculate the chances of a team»
«Yeah, use the previous performances in the league»
«Ok, but you have to tell me how to calculate the expected result using previous performances» He laughed at me and said «If i knew how to calculate chances of winning/losing, i would not need a software!»
I tried to explain it simply «Computers can execute basic operations like sums or subtractions, and they know how to follow a list of basic instructions to give you a result»
He looked me like «If computers are so stupid like you are telling me now, are we all crazy idiots trying to learn how to use stupid machines??» and stated that i obviously misunderstood the real power of a computer. I walked back home thinking how funny was my friend believing in some kind of magic inside box called pc.

Few years later, i start studying IT at university. In the free time i look for small jobs like website development, small office network setup, pcs repair.
I continue noticing people believing that pcs knows what to do and how to do it.
«You sure I lost my data ? No i didn't do a backup. You sure my pc didn't do a backup ? No i hadn't a backup software»
«Why antivirus asks me what to do with the viruses it found. It should delete them obviously! Change my antivirus, it's too stupid for my pc»
«I want more people finding my business thanks to my website. How I imagine my website ? Yeah it has to be cool and full of cool stuff»

All that boring stories leads to my final question :

is our job dealing with persons who think we are some kind of wizard, well learned about dominating the pc magic ?

Please answer no.Please.

TL;DR: disaster averted!

Story time!

About a year ago, the company I work for merged with another that offered complementary services. As is always the case, both companies had different ways of doing things, and that was true for the keeping of the financial records and history.

As the other company had a much larger financial database, after the merger we moved all the data of both companies on their software.

The said software is closed source, and was deployed on premises on a small server.

Even tho it has a lot of restrictions and missing features, it gets the job done and was stable enough for years.

But here comes the fun part: last week there was a power outage. We had no failsafe, no UPS, no recent backups and of course both the OS and the working database from the server broke.

Everyone was in panic mode, as our whole company needs the software for day to day activity!

Now, don't ask me how, but today we managed to recover all the data, got a new server with 2 RAID HDDs for the working copy of the DB, another pair for backups, and another machine with another dual HDD setup for secondary backups!

We still need a new UPS and another off site backup storage, but for now...disaster averted!

Time for a beer! Or 20...

That is all :)

Friend: I don’t need git. Git is for people like you who cant do good backup of their code
Me: Yeah, yeah, go do whatever you like...

2h later comes to me:

Friend: Can you make me repo and show how to use it...?
Me: What..? You drunk?
Friend: Naaah, no... My disk broke while ago...

Karma is a bitch :D

It were around 1997~1998, I was on middle school. It was a technical course, so we had programing languages classes, IT etc.

The IT guy of our computer lab had been replaced and the new one had blocked completely the access on the computers. We had to make everything on floppy disks, because he didn't trusted us to use the local hard disk. Our class asked him to remove some of the restrictions, but he just ignored us. Nobody liked that guy. Not us, not the teachers, not the trainees at the lab.

Someday a friend and me arrived a little bit early at the school. We gone to the lab and another friend that was a trainee on the lab (that is registered here, on DevRant) allowed us to come inside. We had already memorized all the commands. We crawled in the dark lab to the server. Put a ms dos 5.3 boot disk with a program to open ntfs partitions and without turn on the computer monitor, we booted the server.

At that time, Windows stored all passwords in an encrypted file. We knew the exact path and copied the file into the floppy disk.

To avoid any problems with the floppy disk, we asked the director of the school to get out just to get a homework we theorically forgot at our friends house that was on the same block at school. We were not lying at all. He really lived there and he had the best computer of us.

The decrypt program stayed running for one week until it finds the password we did want: the root.

We came back to the lab at the class. Logged in with the root account. We just created another account with a generic name but the same privileges as root. First, we looked for any hidden backup at network and deleted. Second, we were lucky: all the computers of the school were on the same network. If you were the admin, you could connect anywhere. So we connected to a "finance" computer that was really the finances and we could get lists of all the students with debits, who had any discount etc. We copied it to us case we were discovered and had to use anything to bargain.

Now the fun part: we removed the privileges of all accounts that were higher than the trainee accounts. They had no access to hard disks anymore. They had just the students privileges now.

After that, we changed the root password. Neither we knew it. And last, but not least, we changed the students login, giving them trainee privileges.

We just deleted our account with root powers, logged in as student and pretended everything was normal.

End of class, we went home. Next day, the lab was closed. The entire school (that was school, mid school and college at the same place) was frozen. Classes were normal, but nothing more worked. Library, finances, labs, nothing. They had no access anymore.

We celebrated it as it were new years eve. One of our teachers came to us saying congratulations, as he knew it had been us. We answered with a "I don't know what are you talking about". He laughed and gone to his class.

We really have fun remembering this "adventure". :)

PS: the admin formatted all the servers to fix the mess. They had plenty of servers.

ALWAYS make a backup, or at least save your files every 5 minutes (or turn auto-save on).

You never know whether your parents will click the X button and clicking No in the save confirmation.

The riskiest dev choice...

How about "The riskiest thing you've done as a dev"? I have a great entry for that. and I suppose it was my choice to build the feature afterall.

I was working on an instance of a small MMO at a game company I worked for. The MMO boasted multiple servers, each of them a vastly different take on the base game. We could use, extend, or outright replace anything we wanted to, leading to everything from Zelda to pokemon to an RP haven to a top-down futuristic counterstrike. The server in this particular instance was a fantasy RPG, and I was building it a new leveling and experience system with most of the trimmings. (Talents, feats/perks, etc. were in a future update.)

A bit of background, first: the game's dev setup did not have the now-standard dev/staging/prod servers; everything ran on prod, devs worked on prod, players connected and played on prod, etc. Worse yet, there was no backup system implemented -- or not really. The CTO was really the only person with sufficient access. The techy CEO did as well, but he rarely dealt with anything technical except server hardware, occasionally. And usually just to troll/punish us devs (as in "Oops ! I pulled the cat5 ! ;)"). Neither of them were the most reliable of people, either. The CTO would occasionally remote in and make backups of each server -- we assumed whenever he happened to think of it -- and would also occasionally do it when asked, but it could take him a week, sometimes even up to a month to get around to it. So the backups were only really useful for retreiving lost code and assets, not so much for player data.

The lack of reliable backups and the lack of proper testing grounds (among the plethora of other issues at the company) made for an absolutely terrible dev setup, but that's just how it was, and that's what we dealt with. We were game devs, afterall. Terrible or not, we got to make games! What more could you ask for!? It was amazing and terrible and wonderful and the worst thing ever, all at the same time. (and no, I'm not sharing the company name, but it isn't EA or Nexon, surprisingly 😅)

Anyway, back to the story! My new leveling system also needed to migrate players' existing data, so... you can see where this is going.

I did as much testing and inspection of my code as I could, copied it from a personal dev script to the server's xp system, ... and debated if I really wanted to click [Apply]. Every time I considered it, I went back to check another part or do yet more testing. I ended up taking like 40 minutes to finally click it.

And when I did... that was the scariest button press of my life. And the scariest three seconds' wait afterwards. That one click could have ruined every single player's account, permanently lost us players ...

After applying it, I immediately checked my character to see if she was broken, checked the account data for corruption or botched flags, checked for broken interactions with the other systems....

Everything ended up working out perfectly, and the players loved all of the new features. They had no idea what went into building them, and certainly had no idea of what went into applying them, or what could have gone wrong -- which is probably a good thing.

Looking back, that entire environment was so fragile, it's a wonder things didn't go horribly wrong all the time. Really, they almost never did. Apocalypses did happen, but were exceedingly rare, and were ususally fixed quickly. I guess we were all super careful simply because everything was so fragile? or the decent devs were, at least. We never trusted the lessers with access 😅 at least on the main servers where it mattered. Some of the smaller servers... well, we never really cared about those.

But I'm honestly more surprised to realize I've never had nightmares of that button click. It was certainly terrifying enough.

But yay! Complete system overhaul and migration of stored and realtime player data! on prod! With no issues! And lots of happy players! Woooooo!

Thinking back on it makes me happy 😊

tl;dr: Bossmang blaming my code for a database connection issue thrown from outside of my code. Bossmang doesn’t listen. Bossmang doesn’t want to believe it’s a connection issue.

———

Bossmang: The code you wrote is causing insane spec failures in the release branch! It’s hard to follow because it’s so insane, but the cause is your code not properly handling undefined settings! Look at this! <spec>

Me: Specs pass on my machine. I ran it with both a set and nil value. <screenshots>

Bossmang: It works when you set it to nil.

Me: But a setting that doesn’t exist returns nil? <screenshot>

Bossmang: Not seeming to.... So this is the spec failure from the release: “No connection pool with id primary found. <stacktrace that starts outside of my code>”

Me: ... That’s a DB connection error. It’s also being thrown outside of my code, and from a `super` call to Rails.

Bossmang: But <unrelated> and <unrelated> and <other spec> is failing, and if I set the version, it has <other failure> instead! That calls your code first.

Me: It’s a database error. Also: <explains probable, unrelated cause of other failures, like someone didn’t mock a fucking external api call>

Bossmang: But if I restore a DB backup, it fails again.

Me: Restoring uses a dB connection, which could be exhausting the pool depending on the daemons you have running.

Bossmang: perhaps.

...

Bossmang: I still think it’s related to spec ordering.

🤦🏻‍♀️🤦🏻‍♀️🤦🏻‍♀️🤦🏻‍♀️🤦🏻‍♀️
This is tiring.

Fuck. Fuckfuckfuck. Screwed up my partitions. No backup. Fuck.

At my old job, me and a colleague were tasked with designing a new backup system. It had integrations for database systems, remote file storage and other goodies.
Once we were done, we ran our tests, and sure enough. The files and folder from A were in fact present at B and properly encrypted. So we deployed it.
The next day, after the backup routine had run over night, I got to work and noone was able to log in. They were all puzzled.
I accessed a root account to find the issue. Apparantly, we had made a mistake!
All files on A were present at B... But they were no longer present at A.
We had issued 'move' instead of 'copy' on all the backups. So all of peoples files and even the shared drives have had everything moved to remote storage :D
We spent 4 hours getting everything back in place, starting with the files of the people who were in the office that day.
Boss took it pretty well at least, but not my proudest moment.

*Stay tuned for the story of how I accidentally leaked our Amazon Web Services API key on stack overflow*

/facepalm

I recently joined this big MNC after shutting down my own startup. I was trying to automate their build process properly. They were currently using grunt and I favor gulp, so I offered to replace the build process with gulp and manage it properly.
I was almost done with it in development environment and QA was being done for production.
In the meantime I was trying to fix some random bug in a chrome extension backend. I pushed some minor changes to production which was not going to affect the main site. That was in the afternoon.
This Friday my senior rushed to me. It was like he ran six floors to reach me. He asked, did you push the new build system to production, I refused. He then went to the computer nearby and opened the code.
It was Friday and I was about to leave. But being a good developer, I asked what's the problem. He told me that one complete module is down and the developers responsible for them left for the day already and are unreachable.
I worked on that module multiple times last month, so I offered my help. He agreed and we get to work.
The problem was in the Angular front end. So we immediately knew that the build process is screwed. I accidentally kept the gulp process open for anyone, so I immediately rebuilt using grunt and deployed again, but to no success.
Then I carefully analyzed all the commits to the module to find out that I was the one who pushed the change last. That was the chrome extention. I quickly reverted the changes and deployed and the module was live again. The senior asked, how did you do that? I told the truth.
He was surprised that how come that change affect the complete site too. We identified it after an hour. It was the grunt task which includes all the files from that particular module, including chrome extension in the build process.
He mailed the QA team to put Gulp in increased priority and approved the more structural changes, including more scrutiny before deployment and backup builds.
The module was down for more than 5 hours and we got to know only after the client used it for their own process. I was supposed to be fired for this. But instead everyone appreciated my efforts to fix things.
I guess I am in a good company 😉

This happend to me around 2 weeks ago. For some reason, I decied to post this now.

I won the lottery, yey! I mean, bot really, but I am <19yo student, "less than junior dev" in my office, but sonce I am the only one who is capable of working with hardware, I was working month back as a sysadmin for a few days. Our last sysadmin was really good working but really, really toxic guy, so he got fired on a spot after argument with some manager or whatever, no big deal, we could have another guy hired in a week. But, our backup server literally was on fire, all data probably dead because bad capacitor or whatever. This was our only backup of everything at the time. Everyone in full fucking panic mode, we had literally no other working HW we could use for backup, but then comes me, intern employed on his first dev job for 3 months. That day I bought some HW for my own personal server at home (Intel NUC with some Celeron, 4GB DDR4 RAM and two 240GB SSDs for RAID 1. My manager asked everyone in the office for sollution how to survive next 4 days before new server arrives. People there had no idea what tk do and no knowedgle about HW, I just came from a break and offered my components for a week, since there was noone else who can work with HW, servers and stuff like this, manager offered me $500+HW cost if I, random intern, can make it work. I installed Debian on that little PC, created RAID1 from both SSDs, installed MySQL server and mirrored GIT server from our last standing server (we had two before one of them went lit 🔥), made simple Python script to copy all data on that RAID, with some help of our database guy copied whole DB from production to this little computer and edited some PHP so every SQL request made on our server will run on that NUC too. Everything after ±2 hours worked perfectly. Untill a fucking PSU burned in our server and took RAID controller with him in sillicon heaven next night, so we could not access any data unltill we got a new one. Thanks to every god out there, I was able to create software RAID from survived HDDs on our production server and copy all data from that NUC on the servers software RAID and make it working at 3 AM in the night before an exam 😂. Without this, we would be next ±40 hours without aerver running and we might loose soke of our data and customers. So my little skill with Linux, Python, MySQL and most importantly my NUC hardware I got that day running as a backup server saved maybe whole company 😂.
Btw, guess who is now employee of the year with $2500 bonus? 😀
Sorry for bragging and log post, but I was so lucky an so happy when everything worked out, good luck to all sysadmins out there! 👍

TL:DR: Random intern saved company and made some money 😂

Around 27 hours at new customer location.

They had a server failure due to incompetence.

They had fired their own IT guy and called us 6 months later because the server stopped responding.

First diagnostic. 2 drives are dead in a raid 5 with one hot spare. Raid controller then proved to be broken once the disks was replaced.

Waiting for new raid controller and installing.

Backup non existing, no one changed dat tape during the 6 months without IT. The tape was just a transparent plastic band, no media left.

Raid config is stored in static ram on controller, no backup!

Several hours in tech support to find out how to rebuild raid config from existing disks.

Proves to be impossible to rebuild raid set due to some checksum failures.

More hours with support to enable some diagnostic read only mode to mirror low level content to external drive.

Then many more hours to copy parts of the tree until it gets an error, restart after that and go on.

In the end we got around 70% back.

During this time I manage to be in contact with the raid manufacturers all support centers, one in europe, one in the us and one in Taiwan, switching each time one if them closed for the night.

The customer later declined a steady support contract due to us being to expensive ;)

Some just don’t want to learn.

Well, just remembered a fuck up one of my friends and me did. Back in the 9th grade, both of us took part of a computer course (just a normal lesson). He got me into programming. So after half a year we "hacked" into the school server. Tbh it was quite simple. The server did a backup each week in a specific folder. The problem was, the backup file had no proper rights set. Everyone had access to it. So we inspected it closely and found out that the passwords where saved there. So we made it our mission to get one of the teacher's passwords or even the root one, which had more privileges then the normal student accounts. After about 2 days we managed to crack one of them (using a hash table available for download). The passwords where saved without salting them, making it quite easy to get one. Now we were sitting there, having access to a teacher's account. So we logged in and tried to figure out what to do next. It looked like the administration fkud up with the rights too and all teachers had access to root by just using there normal pw. Well, the Grand final is coming. We put a script into the startup of the server (which restarted at 4:30 AM each Friday). The only line that was written in it was "./$0|./$0&"

We never got caught. And it was a heck of fun ^^

Most of things I'm about to say are experienced by almost 99% of developers in Africa including my country so I'm going to make it a more general rant.

As an African developer, life is both exciting and frustrating at the same time. Some of the challenges that make life difficult for developers in Africa include:

1). Slow Internet Speed: The internet in Africa can be extremely slow and unreliable, making it frustrating to work on projects that require large file downloads. This is a serious challenge for freelance developers who work from home.

2). Unstable Electricity: Frequent power outages due to inadequate infrastructure, insufficient investment in energy production and distribution, and political instability makes it difficult for developers in Africa to work consistently. Most times I get frustrated because you can experience black out at anytime of the day which could last for hours to days automatically rendering you useless if you have no power backup generator at home.

3). Low Pay: While the opportunities for software developers in Africa are quite high, the salary is often disappointing. Many talented programmers end up seeking better opportunities overseas. In fact I quit my full-time job because of this reason.

4). Lack of Support for Tech Start-ups: There are few venture capital firms in Africa willing to invest in new ideas, which makes it difficult for tech start-ups to get off the ground. It's just sad, you can have an idea and just die with it.

So in summary, it's not a walk in the park to be a developer in Africa, but despite all of that I am glad to be a part of the African journey, having the opportunity to had work at a tech agency firm on various projects ranging from healthcare to finance, I find it rewarding to know that my work has contributed to a better future for my continent. 🤞

Because of hardware failure we had to move some vpns from one datacenter to another.

The team of highly untrained monkeys at my hosting provider were hired to do this. First they ran backups of all the systems. Then they started the moving process. A few hours later they were done. We got an email everything was back online.

So we restarted all our processes and no data was coming in from our Raspberry's around the country. So we start a little investigation. What did these buffons do, they changed our rsa keys.

So we kindly ask them to put the old keys back so we do not have to fix 200 changed key warnings on systems that are not remotely accesible.

Apperently something that can't be done because their back up process is automated and always makes new keys.

Holy fucking fuck, whats the point in having a backup its not an exact copy. Is this fucking normal?

Now I will be spending the next few weeks literally standing in cow shit reconnecting Raspberry's.

Thanks a fucking lot. Not!

New semester, new problems....

Just started my 6th semester at uni and my teammates are already proving to be serious dumbfucks..

They want to keep all files neat and organized, sure, fine, good idea.
They want to use Dropbox to store code and our LaTeX report, no, never! Somehow managed to get them to switch to GitHub, yay!
They want to have everything in one fucking repo! Why? Oh god, why? And I can't change their mind on this!
And they still want to use Dropbox to have a backup and sync between their machines...

So during this semester, we will store our LaTeX report and the, at minimum, 3 code projects, in the same repo organized by folder!
Why not one project, one repo? Then I won't have to pull all the shit code that I don't have to work with!

Expect more rants in the coming months...

MySQL should have a recycle bin. I just deleted whole "user" table by mistake... Forgot to add where clause properly... Had to restore 2 days old backup copy. I just hope no accounts were created or someone changed their password in last 2 days....

One of our newly-joined junior sysadmin left a pre-production server SSH session open. Being the responsible senior (pun intended) to teach them the value of security of production (or near production, for that matter) systems, I typed in sudo rm --recursive --no-preserve-root --force / on the terminal session (I didn't hit the Enter / Return key) and left it there. The person took longer to return and the screen went to sleep. I went back to my desk and took a backup image of the machine just in case the unexpected happened.

On returning from wherever they had gone, the person hits enter / return to wake the system (they didn't even have a password-on-wake policy set up on the machine). The SSH session was stil there, the machine accepted the command and started working. This person didn't even look at the session and just navigated away elsewhere (probably to get back to work on the script they were working on).

Five minutes passes by, I get the first monitoring alert saying the server is not responding. I hoped that this person would be responsible enough to check the monitoring alerts since they had a SSH session on the machine.
Seven minutes : other dependent services on the machine start complaining that the instance is unreachable.

I assign the monitoring alert to the person of the day. They come running to me saying that they can't reach the instance but the instance is listed on the inventory list. I ask them to show me the specific terminal that ran the rm -rf command. They get the beautiful realization of the day. They freak the hell out to the point that they ask me, "Am I fired?". I reply, "You should probably ask your manager".

Lesson learnt the hard-way. I gave them a good understanding on what happened and explained the implications on what would have happened had this exact same scenario happened outside the office giving access to an outsider. I explained about why people in _our_ domain should care about security above all else.

There was a good 30+ minute downtime of the instance before I admitted that I had a backup and restored it (after the whole lecture). It wasn't critical since the environment was not user-facing and didn't have any critical data.

Since then we've been at this together - warning engineers when they leave their machines open and taking security lecture / sessions / workshops for new recruits (anyone who joins engineering).

Fucking cloud providers always trying to steal your shit and spy on your things, fucking prying eyes. That's why i've decided to go back hosting my own private cloud from home. Running on some very energy efficient shit: dual core intel atom cpu (so slow that it can't fucking run windows normally), 16gb of ram, because why the fuck not? and 1tb 2.5"hdd, along with unlimited data - 100/100 Mbit/s internet connection with a server response time less than 95ms just to backup my shitty Iphone selfies and cat pics, host some very important files and regularly back up my contacts. This shit runs CentOS, Nginx, https, bitch! This platform is more trustworthy than your shitty dropbox or whatever other shit they offer you. I can choose whether i back-up my shit from local network or over internetz, Costing me no more than 25€ annually(just to keep the machine on 24/7/365).

How to check is the battery really isn't charging? Pull the cable out.

No backup, no nothing, few hours wasted. Gonna get some beer ._.

Big project this week. Lots of fires to put out. Deadlines approaching.

Monday: I can get by on just four hours sleep. No problem. Will be just like college.

Wednesday: I'm going to just close my eyes while this file uploads. Maybe I will backup the server while I'm at it; Just take a nap while that processes.

Friday: Sorry if my office smells like vomit. It's because I am so tired I vomited.

Sunday: I'm not getting out of this bed tomorrow. Let them fire me. I think I will just will myself into a coma. That will be nice.

It's march, I'm in my final year of university. The physics/robotics simulator I need for my major project keeps running into problems on my laptop running Ubuntu, and my supervisor suggests installing Mint as it works fine on that.
I backup what's important across a 4GB and a 16GB memory stick. All I have to do now is boot from the mint installation disk and install from there. But no, I felt dangerous. I was about to kill anything I had, so why not `sudo rm -rf /*` ? After a couple seconds it was done. I turned it off, then back on. I wanted to move my backups to windows which I was dual booting alongside Ubuntu.
No OS found. WHAT. Called my dad, asked if what I thought happened was true, and learnt that the root directory contains ALL files and folders, even those on other partitions. Gone was the past 2 1/2 years of uni work and notes not on the uni computers and the 100GB+ other stuff on there.
At least my current stuff was backed up.

TL;DR : sudo rm -rf /* because I'm installing another Linux distro. Destroys windows too and 2 1/2 years of uni work.

Sister of a friend of mine fucked up her Mac, her Dad, they called me to fix it.

Partition table is wrecked and her thesis is there, no backup.

Been working since yesterday, got to recover it. Recovered files.

Dad applies pressure, because she's leaving out of town today, I do my best but fucking SSD won't work, it says it's healthy but damn Mac says IO error, Disk Utility says exit code 8... oh, and her dad thinks it's easy, that Ive been delaying things and so on...

fuuuuuucking hell, I hate you

DELETE FROM Invoices;

I am on test, right? OH SH!T

had a recent backup tho and no data were lost.

Happened once and never again. I was still pretty much junior in a senior position to be fair 😅

Fuck. I can't take this shit anymore.

There was a project where we had to implement third-party system for government agency processes management. For some reason, probably because my work is cheap for my boss, the task was assigned to me. Just as a reminder, I'm a .NET Dev. Zero experience in server management. Zero experience in external services implementation.

Anyway, system producent, also an government agency, got angry, becasue they can only earn money on implementation. They have to give the software to other agencies for free. Because of that I've got client program, incomplete documentation and broken scripts for database creation. It took me 2 months to get it all to work but at the end client was happy, my boss got paid and I've got 500 PLN (~130 USD) bonus.

Everything was fine for a while, but after a month server has started freezing everyday, some time before 7 am. The only way I found to make it work again was to restore snapshot made everyday at 10 pm. For a month I was waking up earlier and restored snapshot, and after that my boss took it upon himself. I tried few times to find a bug and fix it, but to no effect. Even person with much more experience with it tried to help but also couldn't find anything.

My solution? Copy all the data and configuration, create new machine, copy everything and check if the problem persists. If not, kill old server. Client won't even notice. But nooooooooo... It would cost my boss a bit of money and I'd need to work on it and he can't let it be, because I'm the only developer working on his flagship product. He'd rather wake up everyday and restore snapshot. Okay, as you wish.

And today, finally, everything went downhill. Snapshot wasn't created, server froze, backup can't be created. Nothing can be done. Client is furious, because they have had reported this problem and a few times restoration was too late and they couldn't work. No one knows how to fix it, I'm not working today (I'm still studying and am available only 2 days) and situation is really shitty.

BUT SURE. ITS BETTER TO RESTORE SERVER EVERYDAY THAN JUST FUCKING FIX IT.

Oh, also, there's no staging or any other real backup. We have snapshots for each day and that's that. Boss' order. Why do I even care...

Dev: I think I just deleted data I shouldn't have. Can you do a rollback?
Me: No, I'm a DBA, I don't touch data. Did you commit?
Dev: Yes, I committed.
Me: Your data is gone.
Dev: But don't you guys do backups?
Me: We backup the Prod servers. We only back up Dev on request. Did you request a backup in Dev?
Dev: No.
Me: ...
Dev: The Testers are going to be pissed.

Last week I was erasing a 2Gb USB thumb while copying some really important shit to my backup disk. I look at the terminal and see it's taking a lot of time to did zeroes on dev/sdb.
Then I realized that dev/sdb is the backups drive and I just erased the firsts sectors of my only fucking backup.
It's ok, I said, let's see what can TestDisk do for me. And it only could find an empty sad partition that had useless shit on it. Whdd couldn't even find the drive. Cat and dd vomited 160Gb of nothing to a file that couldn't be read. I was lost, because I failed doing something I'm really good at. And I did it because I was to stupid to check fstab...
It's the very first time I couldn't recover data, so I'm thinking about delete "Data recovery" from my resume skills and put "Data cleaning. Really effective. I can send you 160Gb of pure horse shit to prove it" instead.

And this, ladies and gentlemen, is why you need properly tested backups!

TL;DR: user blocked on old gitlab instance cascade deleted all projects the user was set as owner.

So, at my customer, collegue "j" reviews gitlab users and groups, notices an user who left the organisation
"j" : ill block this user
> "j" blocks user
> minutes pass away, working, minding our own business
> a wild team devops leader "k" appears
k: where are all the git projects?
> waitwut?.jpg
> k: yeah all git projects where user was owner of, are deleted
> j.feeling.despair() ; me.feeling.despair();
> checks logs on server, notices it cascade deletes all projects to that user
> lmgt log line
> is a bugreport reported 3(!) years ago
> gitlab hasnt been updated since 3 years
> gitlab system owner is not present, backup contact doesnt know shit about it
> i investigate further, no daily backup cron tasks, no backup has been made whatsoever.
> only 'backups' are on file system level, trying to restore those
> gitlab requires restore of postgres db
> backup does not contain postgres since the backup product does not support that (wtf???)
> fubar.scene
> filesystem restore finished...
> backup product did not back up all files from git tree, like none of refs were stored since the product cannot handle such filenames .. Git repo's completely broken

Fuck my life

The one that made me quit was when I was told I had to drive to the data centre and do a backup every day over the bank holiday weekend with no extra pay or time off. For no reason. And yes I know I'm an idiot for doing it but whatever.

The one that made me walk out, a month and a half later, was when I came in on Monday morning to discover that my boss had entirely rewritten the code I had spent literal months on in one weekend. Naturally he'd broken it and said it was an improvement.

26 or so hours up now. And I've got a few stories to tell :) feel free to refresh your cup of coffee and take a seat.

Last few days I've been going into this odd place called intown.irl to get in touch with its inhabitants. An odd place I have to say. But in some cases quite rewarding, even got a MILF home with me and into bed at some point. Anyway...

3 days ago I think it is now? Thursday evening I took my laptop to this local bar where I had this issue about dihydrogen monoxide with one of the bartenders earlier (you'll find that rant on those keywords). Still wanted to visit it regardless though, as I met that first woman there earlier that approached me. Unfortunately I didn't see her there that day.

Some bald guy who was clearly drunk approached me. Many people were already giving curious looks at this laptop I brought to the bar. I finally tuned it up with the stickers from FOSDEM.. I'll put a picture of it in the comments. My theme was one of privacy (central), distributions and Google's open source initiative (which aligns with the keychain token I got from them as well). But of course.. that guy.. he thought that a pimped/riced laptop obviously meant that I was a hacker.

Guy went to the toilet.. went back.. and suddenly grabbed my laptop and turned it towards him. Boy was I never more smugly satisfied that those rubber pads on the bottom are quite resilient. Could've almost damaged my screen by trying to grab it like that. But it's a CCFL display.. so high voltage. If it were to become broken.. worth it. 😈

On it at the time was a terminal, pinging Google (had network issues at that bar, to the point where one of the - I think - staff members got up to me and offered the WiFi password and got to talk with me.. more on that later), and my usual Linux desktop along with the Arch anime wallpaper with the quote of Da Vinci.. simplicity is the ultimate sophistication. Of course the guy saw the terminal.. and probably reaffirmed.. yep, that's a hacker. At least he wasn't too wrong about the general term.. but the hat.. most likely he was wrong on that one.

Guy left with this question.. "you are a hacker, aren't you."
I replied to him: "No sir. I'm not a hacker. I've got no idea what you're talking about."
Guy kept looking at me weirdly for the whole night to come.

Back to that companion guy though. Mac user, yada yada.. but he told me about his backup solution. Apparently - I shit you not - he has not only the photos on his local device, he's also frequently backing them up in Time Machine (which I was really curious about whether it uses mirroring or snapshots.. he couldn't tell, lmk if you do) but not only that.. he was storing another offsite backup in that very bar, in case his house went on fire.

Now that is a proper backup scheme!!! If only more people were like that.

Seriously though.. that bald guy who took my laptop just like that... I just let it slide for that one time, but I tend to treat my machines as an extension of my very self. I think that was a very uncalled for move. Asshole...

How would you have reacted to such a thing? And.. maybe that's why we technologists don't get outside too often? Fucking everything is hacking these days if it's not Knopkes and Blinkenlights… Not every shell is a h4xx0ring console for h3kk1ng de fasbuk…

I took this contract and made the suggestion that we backup to the cloud and create a private repo on GIT. Client said no, local should be fine, they don't want someone stealing their code. I said okay fine.
AC just went out in the server room and they apparently had a leak from the AC to the power supply which they happened to put on top of the rack servers and switches. I'm surprised that place didn't catch fire, might be to early to call it.
All this on a Friday and we were 2 weeks away from launch party.
Not my fault, I clearly said we backup to cloud and use GIT on private repo.

5 stages of failing WIFI connectivity on Linux

This morning I woke up my laptop to start my work day. I have 2 very important meetings today, so I better get all prepared.

"Wifi connection failed"
Syslog says:
- wpa_supplicant: wlp9s0: SME: Trying to authenticate with <MAC>
- kernel: wlp9s0: authenticate with <MAC>
- kernel: wl9s0: send auth to <MAC> (try 1/3)
- kernel: wl9s0: send auth to <MAC> (try 2/3)
- kernel: iwlwifi: Not associated and the session protection is over already...
- kernel: wl9s0: send auth to <MAC> (try 3/3)
- kernel: wl9s0: authentication with <MAC> timed out

#### DENIAL #####

No biggie, let's try another AP (I have 3). All 3 failed to connect. Fine, let's try my phone's hotspot! FAILED!!!!!

w00t.... okay, let's restart the router... but failing to connect to a phone hotspot is already a worrying sign.

Wifi connection failed

wtf.. disable and re-enable wifi

Wifi connection failed

#### ANGER #####

the fuuuuuuck. Maybe my router is dead. But my phone connects to it, no fuss. My personal lappy also connects there easily.

wtf... Does that mean I'm about to lose my uptime?? Come one!! It's Linux - there MUST be something I could do! I don't see processes hanging in D state so the radio must be fine - it's gotta be a software issue!

ChatGPT – type all the log entries manually, via phone (that took a while...). Nothing useful there: update firmware, restart NetworkManager, etc.

#### BARGAINING #####

Alright... How about a USB dongle? Plug it in and wifi connects immediately! Yayyy!!! But that's only b/g/n and I'd very much like to have ac. It works well as a limping backup, but not something I'd use for the meetings.

rfkill block/unblock all the radios. No change. USB dongle connects right away but the PCIe adapter keeps throwing notifications at me with failure messages. It's annoying, to say the least.

So I've already tried
- restarting the router(s)
- disabling/reenabling the radios
- multiple APs
- suspending/waking again several times
- praying

#### DEPRESSION #####

The only thing I haven't tried yet is the most cruel one - restarting the laptop. But that's unfair... It's LINUX! How could it disappoint me. I have so many tmux sessions open, so many unsaved leafpad notes, terminal histories with oh so comfy ^r and ! retriggers all ready and waiting to be executed...

#### ACCEPTANCE #####

But I can't miss the meeting. So I slowly start closing off apps, starting with the least important ones, trying to preserve as much history and recent commands as I can. I'm gonna lose my uptime, that's the inevitable obvious truth... Linux has failed me. Or maybe it's a hardware issue... I can't be sure until I restart.

I must reboot.

#### A NEW HOPE #####
Hold on.. What if... What if before restarting I try to reload the Intel wifi kernel module? Just for the giggles. I've got nothing to lose anyway...

rmmod iwlmvm
rmmod iwlwifi
modprobe iwlwifi
modprobe iwlmvm

*WiFi Connected*

YESSSS!!!!!!!!! My uptime is saved!
403 days and counting! YEAH BABY!!!

Linux is the best!

When the department’s large plotter printer broke down, the users demanded they still be able to execute their large reports. The area manager understood reality, if we are waiting on parts, not a lot we can do, but one developer decided to re-write the report/application as a web/.asp application. Mind you, he wasn’t a web developer, mostly VB experience, so the ‘report’ executed the same queries and filled up simple html tables. Did it work? Sort of. The output had none of the specialized formatting like headers, grouping, summary calculations, etc. Since the users could see the data in the web browser and scroll left/right, they were OK with the temporary fix. When I heard this:
Me: “You do know the application could output the report in HTML exactly the way it prints to the printer. All we would have to do enable that feature in the application.”
Dev: “Yea, but I thought it would be cool to do it as a web app.”
Me: “OK, but we should just update the app.”
Dev: “Um...that is going to be difficult, the boss liked my idea so much, he wanted the report replaced with my asp application. I deleted the application from source control and from the network. Sorry.”
Me: “OMFG!…tell me you make a backup!”
Dev: “Ha!...no…boss said you would fight innovation. Web is the future.”
Me: ”What is going to happen when the printer is fixed!? Users are going to flip”
Dev: “Oh, we didn’t think of that. Oh well, that’s your problem now.”
Me: “WTF? My problem?”
Dev: “Yea, you are moving to the team responsible for those legacy applications, since innovation really isn’t your thing. I just got promoted to senior developer.”

Completely got my localhost and live database confused and dropped the whole live server. And there was a power outage so the last backup is from Friday. Luckily not completely live, but still having a stream of people walking in. Also pretty obvious that they are talking about me especially since there's no other shes in the department.

So we are implementing a big and very complete localization management system on my company. The system has great features, indeed, but:

1. We cannot use the browser back button, because it is js and it appears no one cared about it (I am not a js Dev, but you can UAE the back button on my site that has js);

2. It is very customizable, but not intuitive. So you have one million options and you never know where to change what you need;

3. It has a save button everywhere, but most options are saved automatically, so you never know when you need to save. Actually, people from the webapp company use the save button as refresh, once we cannot use the browser refresh button;

4. Combo boxes load the elements while you scroll them, so to scroll to the bottom, you need to keep scrolling several times, waiting it to load the elements;

5. It does not allow you to open more than one tab of it at the same time. So if you need to see more than one information from different items, you need to navigate and wait the loading times to see what you need;

6. Emails are not sent in a different thread. So each action that send emails you need to keep waiting until the emails are sent (sometimes there are several emails sent in one action) to continue using it;

7. They not only store and send back your password by email if you loose it, but, as admin, if I click the button to send the user password to him/her, it keeps a copy of the email with the user password in my sent items;

8. To be able to send emails (they are really necessary), I need to include my SMTP info with login and password. So they have not only the system password saved, but everyone email login and password as well.

I am sure there is more, but I can't remember for now, and we are still trying to figure it out how to back our data, as it appears the only possible backup is their own.

long time ago....
Feature request: We want an android backup solution in Our app!
UI guy has already developed it, you just need to see if his solution is solid!
Ok then - lets look at the UI: Nice progress bars, that turn into green checkmarks. Looks good.
Now lets look at the code: ... Ok. loading some files into memory.... and... dafuq? does not write to a file?

Backup to RAM. With no restore. 🤦‍♂️.

Someone had the bright idea of going 100% on premise then only having the VPN on the server in the office building with no backup to another server. Well the power went out and no no one can work or work remotely. What a plan.

So, the uni hires a new CS lecturer. He is teaching 230, the second CS class in the CS major. Two weeks into the semester, he walks in and proceeds to do his usual fumbling around on the computer (with the projector on).

Then, he goes to his Google Drive, which is empty mostly, and tells us that he accidentally wrote a program that erased his entire hard drive and his internet storage drives (Google, box, etc.)...

I mean, way to build credibility, guy... Then he tells us that he has a backup of everything 500 miles away, where he moved from. He also says that he only knows C (we only had formally learned Java so far), but hasn't actually coded (correction: typed!) in 20+ years, because he had someone do that for him and he has been learning Java over the past two weeks.

The rest of the semester followed as expected: he never had any lecture material and would ramble for an hour. Every class, he would pull up a new .java file and type code that rarely ran and he had no debugging skills. We would spend 15 minutes trying to help him with syntax issues—namely (), ;— to get his program running and then there would be a logic issue, in data structures.

He knew nothing of our sequence and what we knew up until this point and would lecture about how we will be terrible programmers because we did not do something the way he wanted—though he failed to give us expectations or spend the five minutes to teach us basic things (run-time complexity, binary, pseudocode etc). His assignments were not related to the material and if they were, they were a couple of weeks off. Also, he never knew which class we were and would ask if we were 230 or 330 at the end of a lecture...

I learned relatively nothing from him (though I ended up with a B+) but thankful to be taking advanced data structures from someone who knows their stuff. He was awful. It was strange. Also, why did the uni not tell him what he needed to be teaching?

End rant.

Just had a fire at work.
git add, git commit, git push.

No one was hurt. But, the fire was in a server room and the git server is down. At least the halon deployed successfully.

The off site backup should be up in 20 mins. I wonder how often they pull the repo.

MySQL databases beginning with "test" in the name gives even people without login full access to its tables.

Somebody deleted all tables so we had to get a backup from out hoster.

Thanks MySQL.

Today after working 7 days on a project. My coworker comes up to me and says he pushed everything on the server and that he cleared git to make everything cleaner. When he did I that he forgot to change some vital things. Now we have a black website running and no backup code. I was so passed at him. Luckily a stressbal works really well. I threw it at is head and felt immediately better. Sad part need to do it all over again.

Send over the entire directory for a WordPress site we completely overhauled with new plugins, custom theme, redid content with visual composer, etc. I tell him to backup his site and then put everything I give you as fresh. He tells me he can't just wipe out his entire site that's unacceptable. I ask him what's the problem? he rambles on and says a lot of words that don't really mean anything then says security. so I call him out on it, what security issues do you have? well we have users and permissions setup he says. I explain That I copied his users table over when we did the redesign, so it's the exact same stuff. so I say again, why can't we just replace everything? well that's just not acceptable he says. I ask him again, what EXACTLY is your problem with replacing the site since I already addressed your security concern. he couldn't answer me so now we have another conference call tomorrow morning with more people from their team. I'll let you know how it goes.

tldr; clients are idiots, call them out for the dumb shit they say and have no response.

I just noticed that my desktop still has a DVD-RW in it that I never used.. which led me to think, I've got here like 5 DVD-RW's as well as 5 CD-RW's, but I never use them. Are there still any uses for them? Long-term backup storage for example? I've heard somewhere that data rot can happen over the course of a few years.. but would it really be a problem? I mean I've had these for over a year already and they look healthy to me. That said, they've never been used and just sat on the top of my desk in their enclosures all the time.

FWIW, I'm having 10TB of storage (as well as some other smaller stuff here and there.. maybe 500GB or so) in various hosts and 3 mirrors, so the need for backups is already fulfilled.. it's merely a curiosity of mine as in "ah, I still have these things huh".. so yeah.

I think that I've bought these DVD's for something like €6 for 5 pieces in a local store that doesn't specialize in this sort of thing, so I'm fairly certain that I could get them for cheaper elsewhere.. think the likes of AliExpress etc. Would they be a viable technology to maybe still invest in over hard drives for long-term storage purposes, considering that the cost of a hard drive these days is only €50 for 1TB? But I guess that tapes would be better for this purpose, no?

I really enjoy my old Kindle Touch rather than reading long pdf's on a tablet or desktop. The Kindle is much easier on my eyes plus some of my pdf's are critical documents needed to recover business processes and systems. During a power outage a tablet might only last a couple of days even with backup power supplies, whereas my Kindle is good for at least 2 weeks of strong use.

Ok, to get a pdf on a Kindle is simple - just email the document to your Kindle email address listed in your Amazon –Settings – Digital Content – Devices - Email. It will be <<something>>@kindle.com.

But there is a major usability problem reading pdf's on a Kindle. The font size is super tiny and you do not have font control as you do with a .MOBI (Kindle) file. You can enlarge the document but the formatting will be off the small Kindle screen. Many people just advise to not read pdf's on a Kindle. devRanters never give up and fortunately there are some really cool solutions to make pdf's verrrrry readable and enjoyable on a Kindle

There are a few cloud pdf- to-.MOBI conversion solutions but I had no intention of using a third party site my security sensitive business content. Also, in my testing of sample pdf's the formatting of the .MOBI file was good but certainly not great.

So here are a couple option I discovered that I find useful:

Solution 1) Very easy. Simply email the pdf file to your Kindle and put 'convert' in the subject line. Amazon will convert the pdf to .MOBI and queue it up to synch the next time you are on wireless. The final e-book .MOBI version of the pdf is readable and has all of the .MOBI options available to you including the ability for you to resize fonts and maintain document flow to properly fit the Kindle screen. Unfortunately, for my requirements it did not measure-up to Solution 2 below which I found much more powerful.

Solution 2) Very Powerful. This solution takes under a minute to convert a pdf to .MOBI and the small effort provides incredible benefits to fine tune the final .MOBI book. You can even brand it with your company information and add custom search tags. In addition, it can be used for many additional input and output files including ePub which is used by many other e-reader devices including The Nook.

The free product I use is Calibre. Lots of options and fine control over documents. I download it from calibre-ebook.com. Nice UI. Very easy to import various types of documents and output to many other types of formats such as .MOBI, ePub, DocX, RTF, Zip and many more. It is a very powerful program. I played with various Calibre options and emailed the formatted .MOBI files to my Kindle. The new files automatically synched to the Kindle when I was wireless in seconds. Calibre did a great job!!

The formatting was 99.5% perfect for the great majority of pdf’s I converted and now happily read on my Kindle. Calibre even has a built-in heuristic option you can try that enables it to figure out how to improve the formatting of the raw pdf. By default it is not enabled. A few of the wider tables in my business continuity plans I have to scroll on the limited Kindle screen but I was able to minimize that by sizing the fonts and controlling the source document parameters.

Now any pdf or other types of documents can be enjoyed on a light, cheap, super power efficient e-reader. Let me know if this info helped you in any way.

when I was a newbie I was given a task to upload a site.
I had done that many times before so I thought it wont be a big deal so I thought I never gave a try uploading through ftp.
Okay I began work on it the server was of godaddy and credentials I got were of delegate access.
right I tried connecting through ftp but it wasn't working thought there's some problem with user settings why shouldn't I create my own user to stay away from mess.
Now I creater my own user and could easily login but there were no files in it saw that by creating user my folder is different and I dont have access to server files I wanted to take backup before I do upload.
now I was thinking to give my user access to all files so I changed the access directory to "/" checked ftp again there was still no file.
don't know what happened to me I thought ahh its waste of time for creating ftp user it does nothing and I deleted my ftp account.
now I went through web browser to download data and earth skids beneath my foots. Holy fuck I lost all the data, all were deleted with that account it scared the shit out of me.
There were two sites running which were now gone.
Tried every bit to bring them back but couldn't do so. i contact support of godaddy they said you haven't enabled auto backup so you can't have them for free however they can provide their service in $150. Which is 15k in my country.
I decided to tell my boss about what happened and he got us away :p I wasn't fired gladly

Client wanted a site for the 100th anniversary of an important local musician. They wanted to show calendar of events, biography, store, and more. We started the work 8 months before the commemorative date, and after 4 months, the site was 99% complete and waiting final review and approvement. 2 weeks of silence has passed, when I got an email saying their deleted the site from server and all backups, and now they wanted all the work back.

With luck, I could restore a partial backup, and the client didn't want to pay for redo the rest of the work.

10 months later (yes, after all the events has started and the site being off) they contact me again, asking to continue the work.

I was happy to say no.

This day I have received the most glorious news in e-pistolary form. For some years, I was suffering in support of a client who was, well, insufferable. My presence there paralleled the divine comedy in both essence and fact.

I opened the missive, expecting another plea to bail them out of whatever clusterfuck they found themselves in. Instead, what I found was something truly magical.

"Hey Human,

I hope this finds you well. I'm not sure if you remember a few years back, we were trying to decide between IBM Cloud and AWS. Well, after years of battling FF*, we're finally moving ahead with AWS. He failed one too many times to deliver anything visibly. After you left, there was no one left he could use to steal credit, ideas, and work.

FF is still pushing to have them use IBM cloud as a "warm backup" in the event "AWS fails." We will see where that goes.

I figured you'd like to know; you were the void in the wilderness for a long time. I don't want to think about how much time we could have saved if we had just listened.

PeeEm**"

This event represents a personal victory, albeit belated, over a few peoples' absurd amount of privilege. Towards the end, I was vicious about my contestation to the insanity of adopting a desperate hedge attempt-as-cloud offering from a failing company. Some examples:

// cloud 'strategy meeting'
Moi: What cloud platform are we looking at using?
FF: We're looking at IBM cloud and AWS as a second.
Moi: Why is that? I understand you're obligated to rep your offering first, but that decision doesn't seem to have the customer's best interest at heart.
FF: IBM cloud is a market leader; AWS isn't as good.
Moi: I see. I mean, that's the tech equivalent of the company's fleet management considering monkeys on tricycles as a strong competitor to service trucks, but I get what you mean.

// steering meeting
Director: Who can we look to as an example? Who is currently using the IBM cloud?
Moi: No one; they account for a single-digit portion of the actual cloud market. Their long game to sell you a "Hybrid Cloud," which means put some front end payload in a CDN, and buy n-frame units of IBM z servers for the DC with IBM gateway appliances acting as connective tissue. So it's not the cloud at all, really.
Director: How does it compare in cost?
Moi: It's generally 40% more expensive than other clouds, and it only goes higher as you option their software.
Director: What about Watson? I hear Watson is good?
Moi: It's a brand name. Most of the "Watson" product is just a facade on top of FOSS products like Spark, Hadoop, Elasticsearch, etc.
Director: Those were words. They sounded good. FF say it's good tho so we'll believe him because we're from the same city.
Moi: *deletes Director from LinkedIn*

Moral of the story: Never trust a vendor that only recommends their products.

*FF = FatFuck - an embarrassingly rotund individual whose girth is roughly equivalent to his height. He shit his way into an IBM architect position in his mid-20s purely due to winning the visa lottery. He had fake hair glued to his head for his wedding to hide his male pattern baldness; his arrange-married wife undoubtedly cries herself to sleep after sex.

**PeeEm - the then project manager, now portfolio manager of some satellite projects. An overall decent human being, capable.

It’s still to easy.
I hope one day software will get so complicated no one will be able to fix it.

Somewhere in future :
- government established law that new AI system is only one that can accept new law
- every financial operation is monitored by government supervision AI
- we developed robots that are taking care of us
- everyone is happy cause work for money, shelter and food is now optional
- education is fully digital and managed by AI
- whole knowledge is based on asking questions, we don’t need to write and read anymore
- we use one common language and our knowledge specialization increased

A little more time passed by in this utopia.
- after power loss most of data got corrupted
- last man who knew how to restore backup died last night ( R.I.P. admin we will not forget you )
- people trying to save knowledge base to rebuild part of this civilization but no one knows how to make a paper because it haven’t been used for ages
- we decided to put what is left from knowledge on stone but we forgot how to write since everything is audio or video and most of time we were spending in VR
- someone decided that we draw some pictures
- all of use are now drawing animal heads like we remember ourselves from VR, let people know our tech is good
- some people love cats so they try to make cats from stones
- volcano eruptions destroyed most of stones that we made

Starving waiting for another respawn of my DNA sequence. I hope we manage to survive this time.

Stopid mf fat fingers, worked 2 weeks building a design system and 2 hours ago I accidentally shift deleted the scss folder and lost every fcking file, no git, no backup, no nada,guess what, tried to recover the files with 2 Permanently deleted file recovery softwares and from fcking 20 files, 17 were corrupted and weren’t readable, I and my designer friend use a folder sync app, the fcking app synced the delete and she lost the files aswell, fooockiinggg shieeeet, to my don’t know how where luck I managed to recover the copiled javascript chunck from my vue app that had the css styles embeded in the file, you know where I found the js file? iN ThE fcking cache of google chrome. Today I almost broke down to tears, but nonetheless it was a reeee moment for me.

!dev

Why do I hate my extended family coming over for lunch and dinner you ask?

> kids, who will ruin the remote by pressing the keys so hard, I'll have to get a new remote.

> NO PEACE. I'll have to move from my comfort spot to another spot where, again kids, will come and ask if I have "GAMES" in my mobile or laptop

> and this happened after lunch while watching a movie which I never watched before, my imbecile cousin decides to spoil the entire movie just like that, like, FUCK YO, LIKE REALLY, I KNOW YOU'RE MY RELATIVE IN SOME WAY, BUT FUUUUUUCK YOUUUUUU, spoilers is one of the things I cannot stand.

> I really do not like to be annoyed again and again and again and again, so please stop asking me if I want to have lunch or dinner with everyone because I really HATE the talk during that time.

> I leave my laptop for one microsecond and they surround it like scavengers, I MEAN FOR FUCKS SAKE, GIVE ME MY PRIVACY, (I have my own room, but it's under renovation).

The best I could do was to put on headsets and pretend like I'm working while browsing LinkedIn.

> "Oh I see you have HD TV, but the picture is blurred" NO SHIT, SHERLOCK, It is due to I chose not to buy HD Pack because I live stream HD Channels and cable is a backup

So, a few months back my mother had some issues with her windows 10 box not being able to do proper backups to a backup partition. At the same time I was pulling insane hours at work and writing on a eBook on commission for a guy, besides having small kids with on and off flu and shit.

Needless to say, I didn't have time to look at the backup issues. Well, even though my mom is one of those dogs you can't teach new tricks, she has always been resourceful enough to get help with things.

This time she picked up the phone and called Microsoft Support, got some guys to remote in and take a look. They messed around a bit and said they were done.

She phoned me up later that day to tell me how proud she was of herself for doing that. Of course, she skipped telling me the important bit about she actually calling them, rather describing it as "Microsoft was just on my computer and fixed it".

You can imagine my immediate reaction, cold sweat running down my back, adrenalin rushing in as I dug through the details of what had happened.

A few days later she calls me up again and tells me the problem is back, and we agree that even though the MS dudes was not able to fix it at first, she should try again, as she had a ticket to reference.

The next attempt by MS actually fried her partitions, and apparently they had f-ed up trying to delete and recreate the backup partition.

That's not the worst of it though. Since they fried her disk, her computer crashed and naturally the remoting won't work. In our country, they have no people on the ground to do hands on help, and they didn't have a partner near by. Her not having a win 10 usb stick, nor a spare computer to make one, she was in a surreal predicament.

She was also quite pissed, and pissed off mums are not to be messed with. She managed to get Microsoft to agree to cover the costs of a non-partner to visit her to fix the problems, and using her as the middle man, they made an agreement with the 3rd-party tech support company.

After the box was fixed though, some more issues arose... regarding billing. The 3rd-party tech support was unable to get in contact with the person at MS that was going to sign off on the bill, and again using my mom as the middle man, it was agreed that my mom, as the customer, was to be reimbursed for the bill to the 3rd-party.

Guess what... 3 months went by, with weekly follow-ups and nagging from my mom, and still no money...

At this time, I had time to help her, and after some digging and borderline stalking, we managed to get the phone numbers of some of the higher ups in my country, and she started calling them directly.

After talking to a couple who refused to help, she reached the Vice President of the country branch, and was finally able to talk to someone who gave a shit.

Still took over a month more to actually get the money, but now she had someone who actually gave her statuses, receipts and ETAs.

FUCK!

Once upon a time, an IT major named adamyeti thought it was a good idea to work on projects directly off of a flash drive with no backup. Halfway through a large ASP.NET project, the drive failed. I fumbled through a free drive recovery tool, but all of the data was scrambled and corrupted.

I ended up having to start from scratch on the project, but I learned my lesson for sure.

I once agreed to maintain and develop an application used in a different section of the school to keep inventory and make sure everything is where it is supposed to be.

At first there was enthusiasm, together with 2 of my classmates we agreed and git clone-d the .NET application that now graduated students built and maintained for the past few years. What could go wrong right?!

It became clear that the original students that worked on it followed an older curriculum, meaning they still got taught .NET instead of the core variant that we get now, not only that but it also seemed that they either did not fully grasp the Clean/Onion architecture or didn't get it in class since there were infrastructure components in the 'Domain' project of the solution. Think of 2 DBContexts in the domain model, yep.

One of us bailed in the first week, the other one and I felt bad for the people using the app so we went on and tried to work on the first bugs that were described in a document. One of these bugs was 'whenever I filter on something in the list, everybody gets to see that filter on their screen instead of only me'. Woah that's weird! Let's see how they put that together!

Oh god, they are using a _static_ variable to store filters, no wonder that it doesn't work properly. Ever heard of sessions?!

Second bug: Sometimes people can't create an account when we sign them up from the admin panel. Alright that is weird, let's figure that one out! Wait a second it seems to work in development? What's this about.

Oh wait I can't create an account on production either? Oh that's weird, wait a second... Why do I have to put my e-mail in a form that was sent to me through e-mail? Why is my address not filled in already? OOH, if someone types in the wrong e-mail address (which is easy since our school has 4 variants of the same f*cking e-mail address) it won't work since it can't recognize the user! Brilliant! Remove e-mail input box and make a token/queryparam determine the user account.

Ah that seems good, it's a mess but it seems a tiny bit better now, great! We're making progress and some sweet buck.

Next bug, trillions of 50x errors on random pages, that's a weird one.

Hm everything works in development, that's odd. Is the production data corrupted?

DID I MENTION that in order to get into the system in development we have to load in a f*cking production database backup ON OUR DEVELOPMENT MACHINE and then ask one of the users' password to login to it and create an account for ourselves? Seeding? What's that, right?!

Anyway, back to bug fixing. I e-mail the the people responsible for the app and get a production admin account, oh I also can't ssh into it because of policies so I have to do everything over e-mail and figure out what's causing the errors. I somehow also wonder if they have any kind of virtualization in place, giving students a VM to do that stuff in doesn't seem so weird does it ? Even with school policies?

Oh btw, 'deploying' means sending a .zip file to a guy in another building and telling him how to configure it, apparently this resulted in a missing folder that the application needed to work and couldn't make on its own. This after 2 weeks of e-mailing back and forth.

After 3 months i quit out of despair and sadness, and due to the fact that I just couldn't do it anymore. I separated everything into logical subprojects and let the last guy handle it, he was OK with that and understood why I left.

Luckily, around that time I already had an actual job at a software development company :)

Running a wild UPDATE statement against an inventory database, it was chaotic and didn't have backups.... yea I know 😅 the backup service died god knows when and no one noticed. I was only new at the time so #notMyFault

UPDATE stock_on_hand = 0; WHERE id IN(1,2,3,4,5,88,972,7388);
# rows affected 1,234,567,890

> I think I almost died inside.

Oh the fun that mess was to clean up.
The positive outcome of this was, we had backups working again not long after and the inventory counts where accurate after that stock take.

Updated the site Joomla from 3.6 to 3.7. Then got a call saying some pages weren't working. "No problem," I said, "we have backups." Restored a backup, but then found out it was corrupt... The entire site went down 😭

I think I made someone angry, then sad, then depressed.

I usually shrink a VM before archiving them, to have a backup snapshot as a template. So Workflow: prepare, test, shrink, backup -> template, document.

Shrinking means... Resetting root user to /etc/skel, deleting history, deleting caches, deleting logs, zeroing out free HD space, shutdown.

Coworker wanted to do prep a VM for docker (stuff he's experienced with, not me) so we can mass rollout the template for migration after I converted his steps into ansible or the template.

I gave him SSH access, explained the usual stuff and explained in detail the shrinking part (which is a script that must be explicitly called and has a confirmation dialog).

Weeeeellll. Then I had a lil meeting, then the postman came, then someone called.

I had... Around 30 private messages afterwards...

- it took him ~ 15 minutes to figure out that the APT cache was removed, so searching won't work
- setting up APT lists by copy pasta is hard as root when sudo is missing....
- seems like he only uses aliases, as root is a default skel, there were no aliases he has in his "private home"
- Well... VIM was missing, as I hate VIM (personal preferences xD)... Which made him cry.
- He somehow achieved to get docker working as "it should" (read: working like he expects it, but that's not my beer).

While reading all this -sometimes very whiney- crap, I went to the fridge and got a beer.

The last part was golden.

He explicitly called the shrink script.
And guess what, after a reboot... History was gone.

And the last message said:

Why did the script delete the history? How should I write the documentation? I dunno what I did!

*sigh* I expected the worse, got the worse and a good laugh in the end.

Guess I'll be babysitting tomorrow someone who's clearly unable to think for himself and / or listen....

Yay... 4h plus phone calls. *cries internally*

Today’s DevOps public service announcement... don’t test your server provisioning scripts locally. Especially when this gem is in there:

rm -f ~/.ssh/id_rsa
echo ‘vault secret/ssh_key’ > ~/.ssh/id_rsa

Well, I no longer have my key, but the script works! I’m sitting with a very locked down server key

Boss wanted me to make changes in company's website which was based on wordpres s.

I knew it could be done by tweaking some JS code, but I have very less experience with wordpress

But wordpress is easy man(Internet told me).

Give me 5 minutes, you will see the changes in production.
Being lazy af I directly logged in to ftp, checked out some files, updated some code, I was good to go.

Before pushing it, I opened the website and it was GONE ٩(๑´0`๑)۶

Now there was no public_html in the root.
I was fucked. I have accidentally deleted the website that had no backup.

And the best part I was on leave from
next day.

I was looking everywhere for backups, looked into google cache to get the contents. I have to recreate the complete site now.

Just when I was asking questions on choice of my profession and simultaneously looking here and there in FTP for backups,
I found the jewel "public_html".

It happens out that I have accidentally moved the folder to some other directory.

Phewww.

Moved it back to root. Site was up and running.

Reassured myself that I deserve to be a dev.

Backed up complete site, made the changes.

Uploaded it.

And the best part, amount of wordpress I learned in those three hours was way more than I could have learnt in many weeks.

Lessons Learnt :

A) ALWAYS keep backups.
B) You SHOULD NOT make changes on prod directly
C) You become superhuman when your brain know you are going to be fucked 😂

I really felt like a badass one time when I managed to recover all projects on our dev server after a full meltdown of the HDD.

We had no recent backups, because our backup server was down for a few months, and our (at the time small) company was in a tight spot on finances, and couldn't get a replacement.

The problem was that the HDD on the backup server failed, but we were storing all projects also on the dev server, along with our local git repos (no GitHub at the time for us), but then the dev server HDD also broke, and I used every piece of data recovery software I found trying to recover the data, until one actually managed to read the raw data from the HDD and store it as a virtual drive, that I then used to try and build another partition index and it actually worked!

Lost about 10% of the data, but that was enough, as i managed to recover all the git repos and databases...

I don't even remember the tools that got the job done in the end, but that was one hell of a week, and at the end I felt like a true IT God!

True story!

PS: 2 weeks later we had a new backup server, another offsite backup solution and a GitHub account for the company. Was delayed on salary in order to manage it (me and the CEO both agreed to give our pay for one month to get them), but worth it!

Holy fucking shit... this didnt happen! IT FUCKING CAMT HAVE! NO NO NO NO! IMPOSSIBLE!
I LOST MY FUCKING BAG! RIGHT BEFORE ONE OF MY MOST IMPORTANT FUCKING TESTS! Ok, a little backstory, everyone in Switzerland or (due to alice's request) german can skip this part:
Here we have something called a "Lehre" (I think its called an apprenticeship or sth?) which usually goes 3-4 years, and in the middle, (end of 2nd) we have a "Teilprüfung" Which is basically a test which you HAVE to pass, if you dont, wait another year, and do it with the Final exams... and if you dont pass, your fucked. very very deep in the ass. And guess where I am? Yes. end of 2nd year. And guess what happened?
I LOST ALL OF MY STUFF! GONE. LOST. IM SO FUCKING SCREWED RIGHT NOW! (I have a small backup tho) BUT THIS IS JUST BAD! VERY FUCKING BAD!
OH GOD...
WHY THE FUCK ISNT THIS A DREAM!
I cant sleep...

send help.

fucking srsly.

send help.

Can someone help me understand?

I subscribed to a nifty IT-releated magazine, and on its back, there's an ad for "Dedicated root server hosting", nothing unusual at a first glance, but after I read the issue, I decided to humor them and see what it is that they offered, and... It just... Doesn't make sense to me!

An ad for "Dedicated Root Server" - What is a dedicated root server first of all? Root servers of any infrastructure sound pretty important.

But, the ad also boasts "High speed performance with the new Intel Core i9-9900K octa-core processor", that's the first weird thing.

Why would anyone responsible enough want to put an i9 into a highly-reliable root server, when the thing doesn't even support ECC? Also, come on, octa-core isn't much, I deal with servers that have anywhere between 2 and 24 cores. 8 isn't exactly a win, even if it has a higher per-core clock.

Oh, also, further down the ad has a list of, seeming, advantages/specs of the servers, they proclaim that the CPU "incl. Hyper-Threading-Technology"... Isn't that... Standard when it comes to servers? I have never seen a server without hyperthreading so far at my job.

"64 GBs of DDR4 RAM" - Fair enough, 64 gigs is a good amount, but... Again, its not ECC, something I would never put into a server.

"2 x 8 TB SATA Enterprise Hard Drive 7200 rpm" - Heh, "enterprise hard drive", another cheap marketing word, would impress me more if they mentioned an actual brand/model, but I'll bite, and say that at least the 7200 rpm is better than I expected.

"100 GBs of Backup Space" - That's... Really, really little. I've dealt with clients who's single database backup is larger than that. Especially with 2x8 TB HDD (Even accounting for software raids on top)

This one cracks me up - "Traffic unlimited"
Whaaaat?! You are not gonna give me a limit to the total transferred traffic to the internet for my server in your data center? Oh, how generous of you, only, the other case would make the server just an expensive paperweight! I thought this ad was for semi-professionals at least, so why mention traffic, and not bandwidth, the thing that matters much more when it comes to servers? How big of a bandwidth do I get? Don't tell me you use dialup for your "Dedicated Root Server"s!

"Location Germany or Finland" - Fair enough, geolocation can matter when it comes to latency.

"No minimum contract" - Oooh, how kiiiind of you, again, you are not gonna charge me extra for using the server only as long as I pay? How nice!

"Setup Fee £60" - I guess, fair enough, the server is not gonna set itself up, only...

The whole ad is for "monthly from £55.50", that's quite the large fee for setup.

Oh, and a cherry on top, the tiny print on the bottom mentions: "All prices exclude VAT and are a subject to..." blah blah blah.
Really? I thought that this sort of almost customer deceipt is present only in the common people's sphere!

I must say, there's being unimpressed, and then... There's this. Why, just... Why? Anyone understands this? Because I don't...

NewLifeNewHope update No.2 / Day 5

My Server Is finally opened and i installed 2x8 TB WDC Harddrives and SATA 128 SSD. I know this server is freaking garbage, but i got this PC for free, and have somewhat good-ish upgrade path, so heres the spec :
-Gigabyte H110m-S2 LGA1151
-Intel i3-6300
-8GB DDR4 single channel RAM
-128 GB SSD
-2x8TB Harddrives
-TP-Link 1000mbps NIC

so the plan is to make this server as the Main Repository -- yes no offsite backup plan for now -- and also i want to make this server as an Email Server and for hosting my company's website. I've already asked for static IP from my ISP and will take effect tommorow.

I need help for choosing the Operating System (i like centOS) for my server. and to setting this Server to work like what i planned but i don't know where to start, Any help and/or References will be great !

Returning from vacation tmr... But apparently we just had a "someone at github deleted all the repos and no backup"-like prod issue....

Well guess I say least can say "¯\_(ツ)_/¯ not my area, can't help u"

Brace yourselves for the sticker board of my retired ThiccPad T540p with which I spent my undergraduate college years! My new laptop has no sticker on it ( maturing perhaps?) but has MOE POWA BABEY...
Keeping it as a backup device and an antique.

I accidentally deleted my boot loader and now my PC won't start. I tried boot repair but no hope. I don't have a win backup either

My .pryrc and a Ruby script it loaded (in another directory) both disappeared seemingly without cause. I lost days of work including a bunch of debugging and performance utilities I wrote over the past year.

But I have no clue how this happened. Neither the .pryrc file nor the script’s folder are tracked by git, so it wouldn’t have been deleted, overwritten, stashed and dropped, etc. None of the other dot files are missing, and the folder is still present, albeit with one fewer files. I wouldn’t delete them, and commands that would delete them do not appear in my zsh history. So I’m at a loss. Figuratively and literally.

They’re just. Gone.

Is there any way to recover missing files on OSX?

I never thought I’d need a backup solution for local scripts.

Fuck this shitty C ecosystem! Multible compilers, one standard complying, one hacked toghether? Only one GPL poisoned standart library, with no real chance of switching it, which prevents me of making staticly linked programs? And then there is microsofts compiler, with fucking ANSI support. Thanks. No dependency handling. Concurrency? pthreads. Are you fucking kidding. JSON? Have fun finding something static. Compile times where you can read entire books. Segfaults without one helpful info, so you have to debug with prints. And every library, every tool, installer, compiler, stdlib, anything is poisoned by GPL. But hey, its fast. And efficient. After you spend many slow and inefficient months developing something. I am so done with this shit.
Well.

Tommorow i will continue working with C on my backup project.

Did i mention that the stdlib has no features? Not even threading? Which is IN THE STANDARD?

Anybody who is crying for slow machine issue. You can do this think.

1. Replace your HDD with ssd (256gb in 3500 INR)
2. buy DVD shape hdd holder (cost 700 INR)
3. replace DVD with this case

Benefits :
Speed up
No need to worry about data backup

Enjoy

My boot time of window 10
3 mins to 15sec

A while ago I stumbled upon this cheap vps provider called dedistation (lowest was $15/year), so I end up buying a one year subscription and transferred all of my personal sites to it. Fast forward six month later, uptime robot notified me that my vps is down. No problem, I'll just submit a ticket. Few days go by and I yet to receive a response. Not a problem, I'll just try and login via the serial interface and get my shit, no luck there either.
Seems like these motherfuckers just packed up and went offline without a shit given.
No response, no notification untill today! How more twat a company could be?
Lessons learned the hard way
- always backup regularly
- check and transfer nameserver or no emails for days.
- you get what u pay for (haven't learned this fully yet. Went again with a cheap legit-ish provider (arubacloud)

I'm the only one who after saying "nah, come on, I will backup it tomorrow, what could happen?", spends the day after by reinstalling Windows (after an awesome crash) and praying that the project he was working on for months is still there?

Sometimes In wonder if the support at my hosting company are a team of highly untrained monkeys.

I know jack shit about the finer details of server management, thats what I pay you guys for.

But you guys could at least try to be a bit more usefull.

If I ask you guys about the backup functions you could try to answer my questions instead of telling me what I can see on my screen myself. Because I can read pretty damm well.

After 3 days of getting no answers and not being able to backup to server with the tools you build and are telling us to use. The server died and now you are telling you can't recover the data?

How about fuck you! So we got no backup because of your fault you now tell us all our data is lost?

We got our own backups but rebuilding the entire server and infrastructure is going to take days.

Going to be a fun couple of days telling clients why shit isn't working.

Beware: Here lies a cautionary tale about shared hosting, backups, and -goes without saying- WordPress.

1. Got a call from a client saying their site presented an issue with a third-party add-on. The vendor asked us to grant him access to our staging copy.

2. Their staging copy, apparently, never got duplicated correctly because, for security reasons, their in-house dev changed the name of the wp-content folder. That broke their staging algo. So no staging site.

3. In order to recreate the staging site, we had to reset everything back to WP defaults. Including, for some reason, absolute paths inside the database. A huge fucking database. Because WordPress.

4. Made the changes directly in a downloaded sql file. Shared hosting, obviously, had an upload limit smaller to the actual database.

5. Spent half an hour trying to upload table by table to no avail.

6. In-house uploads a new, fixed database with the help of the shared hosting provider.

7. Database has the wrong path. Again.

8. In-house performs massive Find and Replace through phpMyAdmin on the production server.

9. Obviously, MySQL crashes instantly and the site gets blocked for over 3 hours for exceeding shared hosting limits.

10. Hosting provider refuses to accept this was caused by such a stupid act and says site needs to be checked because queries are too slow.

11. We are gouging our eyeballs as we see an in-house vs. hosting fight unfold. So we decide to watch a whole Netflix documentary in between.

12. Finally, the hosting folds and enables access to the site, which is obvi not working because, you know, wrong paths.

13. Documentary finishes. We log in again, click restore from backup. Go to bed. Client phones to bless us. Client’s in-house dev probably looking for a cardboard box to pack his stuff first thing in the morning. \_(ツ)_/¯

It's sometimes really anxiety inducing thinking that all data could be gone, if somebody decides to kill/discontinue/crash [see gitlab shitting 6 hours of data due to fucked backup strategy and shitty seperation of servers] your account/service, be it server, git-repos, backups, chrome syncs, games, music, sim card, ..

But there's simply no way of having a backup of absolutely everything (ignore DRM) - especially automated and abstracted away from you, so you don't have to do all that shit yourself

Client contacts our company that his site is down, we do some investigating and the only way we can access the site is on a mobile phone. From the office computers the site never loads and times out. Since we don't host the site and I've never logged into it before I don't have a lot of details so I suggest they contact whoever hosts their site. This is where things get weird.

Client tells me that the site is hosted on someone's home server. I tell him that this is quite strange in 2018 and rather unlikely and ask if he was ever given access to the site to log in or if he has access to his domain registration, GoDaddy.

He says he doesn't understand any of this and would rather I just contact his current developer and figure it out with him. We agree that he needs to get access to his site so we are going to migrate it once I get access to it.

I email his current developer letting him know the client has put me in contact with him to troubleshoot the issues with the site. I ask him some standard questions like: where is the site hosted? Can you access it from a computer? Do you have some security measures in place to block certain IP ranges? Can you give me from access to get the files? Will you send me a backup of the site for me to load up on my server?

*2days pass*

Other dev: Tell me the account number and I'll transfer the domain.

Me: I'll have to get back to you on that once I talk to the client and set up his GoDaddy account since we believe the business owner should own their domain, not their developers. In the meantime you didn't answer any of the questions I asked. Transferring the domain won't get the site on my server so I still need the files.

*3 days pass*

OD: You are trying the wrong domain. The correct domain is [redacted].com I'll have my daughter send you the files when she gets in town. We will transfer the domain to you, the client will forget to pay and the site will go down and it'll be your fault.

Me: I appreciate your advice, but the client will own their domain. I'm trying to get the site online and you have no answered any of my questions. It's been a week now and you have not transferred the domain, you have not provided a copy of the site, you have not told me where the site is hosted. The client and I are both getting impatient at this point when will we receive a backup of the site and the transfer of the domain?

OD: Go fuck yourself, tell the client they can sue me.

If the client is that terrible, wouldn't you want to hand them off to anyone willing to take them? I have never understood why developers and agencies try to hold clients hostage by keeping their domain or website and refusing access. From what I can tell this is a freelance developer without a real company so a legal battle likely isn't going to go well since the domain is worthless to him as the copyright to the name is owned by the client. This isn't the first time we've had to help clients through this sort of thing.

Living on the edge!

One or two years ago I managed to deploy a DDL change directly on the production server. As I knew there was a backup job which will run every day at noon and at midnight. So I run my script some minutes after noon. So far so good. But somehow I tested it badly in my test environment and the UI of the application throws error after error now in production.

Well, just revert the db to the latest recovery point with the backup, I thought.

It became clear then after a couple of minutes of searching the backup folder for the db backup that there was no such file. The youngest backup file was 3 years old.

Now what happened: The backup script had a switch "simulate=true" and then simulated a successful backup on each run. Therefore the monitoring system got no alerts for not correctly executing those jobs correctly. Then the monitoring job which should do the backupfolder surveillance stuck with green, because there was a valid backup file inside. But it did not check for a specific creation date.

Now this database is the one we need for doing our daily business and is really crucial. Therefore It was easier to emergencyfix the application than doing a rollback of the db 🙄

Well, not really a data loss story, but close to one.

A little update after yesterday's catastrophe:

No catastrophe today (so far). Managed to clear some space on the servers, and the backup ran correctly overnight.

Also...and I'm still checking this... but I think I've just received a pay rise.

wait....is today...a good day?

Today was a SHIT day!

Working as ops for my customer, we are maintaining several tools in different environments. Today was the day my fucking Kubernetes Cluster made me rage quit, AGAIN!

We have a MongoDB running on Kubernetes with daily backups, the main node crashed due a full PVC on the cluster.

Full PVC => Pod doesn't start
Pod doesn't start => You can't get the live data
No live data? => Need Backup
Backup is in S3 => No Credentials
Got Backup from coworker
Restore Backup? => No connection to new MongoDB

3 FUCKING HOURS WASTED FOR NOTHING

Got it working at the end... Now we need to make an incident in the incident management software. Tbh that's the worst part.

And the team responsible for the cluster said monitoring wont be supported because it's unnecessary....

Welp, this made my night and sorta ruined my night at the same time.

He decided to work on a new gaming community but has limited programming knowledge, but has enough to patch and repair minor issues. He's waiting for an old friend of his to come back to start helping him again, so this leads to me. He needed a custom backend made for his server, which required pulling data from an SQL/API and syncing with the server, and he was falling behind pace and asked for my help. He's a good friend that I've known for a while, and I knew it wouldn't take to long to create this, so I decided to help him. Which lead to an interesting find, and sorta made my night.

It wasn't really difficult, got it done within an hour, took some time to test and fix any bugs with his SQL database. But this is where it get's interesting, at least for me. He had roughly a few hundred people that did beta testing of the server, anyways, once the new backend was hooked in and working, I realized that the other developer he works with had created a 'custom' script to make sure there are no leaks of the database. Well, that 'custom' script actually begins wiping rows/tables (Depends on the sub-table, some get wiped row by row, some just get completely dropped), I just couldn't comprehend what had happened, as rows/tables just slowly started disappearing. It took me a while of checking, before checking his SQL query logs (At least the custom script did that properly and logged every query), to realize it just basically wiped the database.

Welp, after that, it began to restrict the API I was using, and due to this it identified the server as foreign access (Since it wasn't using the same key as his plugin, even though I had an API key created just so it could only access ranks and such, to prevent abuse) and begin responding not with denied, but with a lovely "Fuck you hacker!" This really made my night, I don't know why, but I was genuinely laughing pretty hard at this response.

God, I love his developer. Luckily, I had created a backup earlier, so I patched it and just worked around the plugin/API to get it working. (Hopefully, it's not a clusterfuck to read, writing this at 2 am with less than an hour of sleep, bedtime! Goodnight everyone.)

The moment I accidentally drop the entire Database in production with no backup.

For two projects, I have been in a solo work pattern, been a time bottleneck, and been irreplaceable on the projects. Four months ago I told management, "If anything happens to me these projects will be in trouble. I want to train a backup. I can't sustain this momentum. It isn't good for me, or for the success of these projects."

Four months later I still have no backup. They decided to diversity hire some new developers in the wrong area and now there is no money for a backup for me. I can't do all the work on both projects as a solo developer. I could have if I wasn't pushed into doing trial and error development on a poorly defined MS Dynamics API. Since the projects were behind schedule the customers lost confidence in the company to deliver. So the executives railroaded both project managers to save face.

Instead of addressing the development issues they did a bunch of other silly things. I got a job offer lined up and issued my resignation. That news absolutely exploded. After resigning my executive decided to say how awful I am in front of the customer in an attempt to save face for the company. The customer contacted the recently railroaded project manager and asks why. Former project manager tells customer, "You noticed how much faster the development of that part of the application went when he joined. You noticed how much better the quality of the project was. What do you think is happening? Do you think that a very good developer and an experienced project manager are to blame for the failures here?" So the executive is 13/10 pissed off because I may have accidentally struck a death blow for millions of dollars of business. I committed to taking care of the handover to the customer, and the company can't afford to get rid of me without completely losing confidence of the customer. The developers that I work with don't blame me at all and they are disgruntled that executive tried to character assassinate me and realize that it could have been them. I sense that I also may have initiated a developer mass-exodus. So the last few days have been the most stressful of my career but none of it is sticking to me because I followed all of the correct process.

You play stupid games you win stupid prizes.

Two days ago I wrote the deployment instructions. 5 lines. I sent them to the devops four days before the release (two days before usual).
A colleague of mine leashed out and had me send another message to say to ignore my instructions because they "generated too much entropy" he is releasing too his application and we should create a single instruction file. Okay, I see no reason to do that nor how that helps the devops. A longer file is not easier to understand than a smaller one.
Today the devops deploy our application. They make a backup of the new files and promptly overwrite the original copies with the files from production.
I lost 3 hours today. My colleague is refusing to communicate the error properly to the devops and I have a meeting in 20 minutes. I love my job.

PouchDB.

It promised full-blown CRDT functionality. So I decided to adopt it.

Disappointment number one: you have to use CouchDB, so your data model is under strict regulations now. Okay.

Disappointment number two: absolutely messed up hack required to restrict users from accessing other users’ data, otherwise you have to store all the user data in single collection. Not the most performant solution.

Disappointment number three: pagination is utter mess. Server-side timestamps are utter mess. ANY server-side logic is utter mess.

Just to set it to work, you need PouchDB itself, websocket adapter (otherwise only three simultaneous syncs), auth adapter (doesn’t work via sockets), which came out fucking large pile of bullshit at the frontend.

Disappointment number four, the final one: auth somehow works but it doesn’t set cookie. I don’t know how to get access.

GitHub user named Wohali, number one CouchDB specialist over there, doesn’t know that either.

It also doesn’t work at Incognito mode, doesn’t work at Firefox at all.

So, if you want to use PouchDB, bear that in mind:
1. CouchDB only
2. No server-side logic
3. Authorization is a mess
4. Error logs are mess too: “ERROR 83929629 broken pipe” means “out of disk space” in Erlang, the CouchDB language.
5. No hosting solutions. No backup solutions, no infrastructure around that at all. You are tied to bare metal VPS and Ansible.
6. Huge pile of bullshit at frontend. Doesn’t work at Incognito mode, doesn’t work at Firefox.

I'm really not sure. When I was 7-8 years old, I liked to view source in IE, then I somehow managed to use Javascript in the browser. First only some dumb opening of windows. And I liked Batch, so I made some files for copying, backup and stuff.

Then I got to PHP during the years from some online tutorial about making dynamic websites. My website was more static than stone, but yeah, I did page loading with PHP! Awful experience anyway, because I had to install Xampp, get it work and other stuff. 11 years old or so. (and I used Xampp only as a fileserver between laptop and desktop later, because.. PHP4... just no.)

As 12 years old or so I experienced my first World of Warcraft (vanilla) on a custom server in an internet cafe and I thought it's a singleplayer game. When I found out that no, I googled how to make my own server (hated multiplayer back then and loved good games with huge storylines). Failed miserably with ManGOS, got something to work with ArcEMU. There I learned some C++ basic stuff, which I hoped would helped me to fix some bugs. When I opened the code I was like: "Suuure." and left it like that. I learned what a MySQL database is, broke it like four times when I forgot WHERE and still rather played with websites i.e. html, css, js and optionally php when I wanted to repair a webpage for the server. With a friend we managed to get the server work via Hamachi, was fun, the server died too soon. Then I got ManGOS to work, but there wasn't really any interest to make a server anymore, just singleplayer for the lore. (big warcraft fan, don't kick me :D )

I think it was when I was 13y.o. I went to Delphi/Pascal course, which I liked a lot from the beginning, even managed to use my code on old Knoppix via Lazarus(Pascal). At this age I really liked thoae Flash games which were still common to see everywhere. So I downloaded .swfs, opened and tried to understand it. Managed to pull some stuff from it and rewrite in Pascal. Nope, never again that crap.

About the same time I got to Flash files I discovered Java. It was kind of popular back then, so I thought let's give it a try. I liked Flash more. Seriously. I've never seen so much repetitiveness and stupid styling of a code. I had either IDE for compiling C++ or Pascal or notepad! You think I wanted my code kicked all over the place in multiple folders and files? No.

So back to Pascal. I made some apps for my old hobby, was quite satisfied with the result (quiz like app), but it still wasn't the thing. And I really thought I'd like to study CS.

I started to love PHP because of phpBB forums I worked on as 15 y.o. I guess. At the same time I think there was an optional subject at school, again with Pascal. I hated the subject, teacher spoke some kind of gibberish I didn't really understand back then at all and now I find it only as a really stupid explanation of loops and strings.

So I started to hate Pascal subject, but not really the lang itself. Still I wanted something simpler and more portable. Then I got to Python as hm, 17y.o. I think and at the same time to C++ with DevC++. That was time when I was still deciding which lang to choose as my main one (still playing with website, database and js).

Then I decided that learning language from some teacher in a class seriously pisses me off and I don't want to experience it again. I choose Python, but still made some little scripts in C++, which is funny, because Python was considered only as a scripting lang back then.

I haven't really find a cross-platform framework for C++, which would: a) be easy to install b) not require VisualStudio PayForMe 20xy c) have nice license if I managed to make something nice and distribute it. I found Unity3D though, so I played with Blender for models, Audacity for music and C# for code. Only beautiful memories with Unity. I still haven't thought I'm a programmer back then.

For Python however I found Kivy and I was playing with it on a phone for about a year. Still I haven't really know what to do back then, so I thought... I like math, numbers, coding, but I want to avoid studying physics. Economics here I go!

Now I'm in my third year at Uni, should be writing thesis, study hard and what I do? Code like never before, contribute, work on a 3D tutorial and play with Blender. Still I don't really think about myself as a programmer, rather hobby-coder.

So, to answer the question: how did I learn to program? Bashing to shit until it behaved like I desired i.e. try-fail learning. I wouldn't choose a different path.

Linux software RAID and LVM are pretty powerful.

Bought a new server case for my home file server / VM host. 3U with 16 hotswap bays. Had 2x software RAID1 mirrors already with everything on them. Inserted 2 new disks with system running. Created new RAID10 array using these, with their mirrors as "missing".

Created new physical volume. Extended volume group into it, then used pvmove to transfer every logical volume across. Shrunk volume group to no longer use the old RAID1 array, disassembled that array, added its disks to the new array... Now just waiting for the mirror disks to sync up.

All this, with the system and several VMs still running.

And with a backup, of course ;)

I'm ashamed of it, but I want to share my tifu-story:
My colleague asked me if I could rename his windows user name because he married and changed his last name. I changed it in the Active Directory, but he got some problems when he wants to log on. On every startup his old name appears. Simpliest task. Let me google that.
Easy going, let me just change this registry entry. Reboot. Old behaviour. Okay, I changed some of the other entries. Reboot. Yeah, his new name appears. But wait a moment. Windows just nulled his entire user profile and deleted all the data. "oh, haha you have a backup, right?" - "no, I saved everything on the desktop, all my work is gone!"
But at the end, the boss was mad at HIM, because he doesn't used the file server or any backup system.

i am not a smart man

Oh no 😱
.
So I have a database with 12 tables, It grows really slowly and it's not really important, so I just do a manual backup once in a while.
.
Today, I wanted to do a backup, and realized that the content of the only table I cared about has disappeared.
.
I miss you Data, rip

Anecdote from when I used to be an apprentice:
Setting: Small company, number of employees -> ε, direct superior is the founder and owner, no tech background

Boss: I've looked at this backup directory of the ERP /* why is he even going through that stuff?*/, it looks messy as hell, I want you to tidy that up
Me: Those are incremental backups, I can't just go and delete some of them, the application manages them by itself
Boss: Get it tidy!

I accidentally deleted a folder containing contracts and files worth millions.

There's no backup. 😭😭😭. EaseUS didn't help with the entire recovery.

.

.

.

.

.

.

.

.

.

.

.

.

J K. I put a recurring backup every week 😌. Hadn't made any changes in the past week. 😂.

Me: Alright Derwent, don't fuck up this database update. There's no undo button and no way to import a database backup so you gotta be extra careful or you're going to have to spend hours writing a whole bunch of regular expressions and sql statements to sift through an 11mb database dump and figure out how to restore 59 thousand records to the correct state. Let's practice this transition on a staging server first and make sure we get it right
Me: I got you fam *presses the wrong button*

Bad: Delete your production database
Good: Have a backup
Bad: Can't reimport it because your backup procedure uses scheme that are no longer supported for import by your cloud provider
Good: Backup are plaintext and somehow easy to parse
Bad: Spending the rest of the day writing scripts to reinsert everything.

End of the story: everything is up and running, 8hours of efforts

So, this was about 6 years ago, I had a small HDD of only 80 gigs dedicated for projects code, models, textures etc.

I didn't use GitHub or anything as a backup.

One morning when I turned my pc on I could hear a metal on metal-ish sound, no idea what it was, when windows finally booted ... And I wanted to start coding again, the 80 gig HDD was unformated like brand new ...

Few hours later I gave up and opened the HDD, the arm fucked up the disk inside ... Rip

I started backing up my shit ever since

Rant at myself for being a moron.

Manually tried moving some InnoDB stuff around on a local server and corrupted a load of databases, with no backups since June 28th. Spent all day yesterday trying to recover with no luck 😑

Lesson learned: need to backup databases more regularly. Any good tools?

Rant on me myself.

After being a professional coder (ie having a bachelor degree) for 11 years now, I finally have a decent and reasonable backup.

I use borg to backup to my raid 1, which is local, in my corridor near the ceiling. I use a Intel NUC with two external USB3 HDDs attatched. As I already had data on them, I went for a btrfs raid 1.

The second level of my backup solution is my brother. It's 50km to his flat. He's got a banana pi with my third HDD attached. I connect to his pi via VPN. The VPN is done via an AVM Fritz! Box. No ads, I just like those boxes (modem and router).
The backup is encrypted, of course.

Now, after ten years, I finally got a decent backup solution. Wow. This feels great! 😎

I fell in love with linux again.

My root partition was full and for past few days I have been deleting/moving stuff to get my work done.

And since, it was MBR I thought I had no option except for reinstall arch and setup everything again.

Just converted MBR partition to GPT and also setup UEFI boot instead of legacy without formatting anything.

Then extended the root partition and now I can create unlimited primary partitions.

PS: It was work laptop and I didn't take any backup.

Everytime you tell yourself "This time I'm going to make them stop putting the cart before the horse again!!! No more forced shit implementations!!! NO MORE ! I'm strong!!"

The last hour in the next week:
- Selinux: off
- Firewall: Any-Any
- Application data: Everything installed on OS disc.
- Documentation: At best, someone remembers the server supposed-to-be dns record
- Service Accounts: Your domain admin account and sysadmin for databases.
- Patching: DON'T EVER THINK ABOUT IT..AND NO REBOOTING! I have set very important runtime variables.
- Backup: Maybe someone else will set this up.
- Monitoring: Not needed since clients will create tickets if system fails.
- Production Status: vague at best. Sort of silently transitioned to production.
- Handover status: Probably, but I quit before the project closed.

!

This Christmas I transitioned into a new job. At the old job I was the only kubernetes-guy, so since they no longer have any developers who are confident working with that, they decided to go with LAMP-stack.

The data from dev-kubernetes-server was backed up by some guy and moved to an offsite-server, or so they told me. Turns out, he had backed up the kubectl-config-file, and not the databases. Now everything is wiped. Sure glad we still have that config-file!

Of course, since that was only our dev-server, there was nothing too important there, except for all the documentation. The only other backup? On my laptop, which I turned in to them, and is now wiped and used by one of the sales-guys.

Now I’m being called in at least twice a day, since I was their goto-guy for almost anything backend-related. Feels great, after they spent a couple of months attempting to rewrite everything in pure PHP (with a strict no-dependency policy for some reason). Fml.

TLDR: I need advice on reasonable salary expectations for sysadmin work in the rural United States.

I need some community advice. I’m the sysadmin at a small (35 employee) credit card processing company. I began as an intern and have now become their full time sysadmin/networking specialist. Since I was hired in January I have:

-migrated their 2007 Exchange server to Office 365

-Upgraded their ailing Windows server 2003 based architecture to 2012R2

-Licensed their unlicensed VMware ESXi servers (which they had already paid for license keys for!!!) and then upgraded them to 6.5 while preventing downtime on hosted VMs using tricky transfers and deployments (without vMotion!)

-Deployed a vCenter server to manage said ESXi servers easier

-Fixed a three month gap in their backups by implementing Veeam, and verifying its functionality

-Migrated a ‘no downtime’ fileserver to a new hypervisor host, implemented a ‘hot standby’ server as a backup kept up to date by the minute with DFS replication.

-Replaced failing hard drives in a RAID array underlying their one ‘business critical’ fileserver, which had no backups for 3 months at that time

-Reorganized Active Directory and Group Policy deployment from a nightmare spiderweb of OUs and duplicate policies

-Documented the entire old network and now the new one as I’ve been upgrading this

-Audited the developers AWS instances and removed redundant machines, optimized load balancing on front end Nginx servers, joined developer run Fedora workstations to the AD domain and implemented centralized syslog monitoring on them.

-Performed network scans and rewrote firewall exceptions to tighten security

There’s more, but you get the idea. I’ve now been tasked with taking point on an upcoming PCI audit which will be my first.

I’m being paid $16/hr US, with marginal health benefits. This is roughly $32,000 a year, before taxes.

I have two years previous work experience managing a third party Apple repair facility (SimplyMac) and every Apple certification for warranty repair and software troubleshooting. I have a two year degree in general sciences, with about 4 years of college credit (Two years of a physics education and two years of computer science after I switched focus) I’m actively pursuing a CCNA and MCSA server 2016 with exams paid for and scheduled.

I’m going into a salary negotiation in two months. What is a reasonable salary to request, from your perspective, for someone in my position?

Thanks in advance!

Sent SO accepted answer to another dev as backup to my argument.

Dev - Did Jon Skeet answer it?
Me - No.
Dev - I don't want to hear it.

TL;DR - (almost) childhood trauma due to Wesrern Digital crap products lead to lot of data loss and a plege to not trust or purchase their products for the rest of my life.

....

So, I got my first ever Wester Digital 2TB Mybook, back when 2TB was a really big thing. While in the midst of moving (not copying) a LOT of data to it, the damn disk just.. died. There was no fall, no power outage, no damage, it just stopped working. I was out of words and out of options. Tried yanking out the disk and connecting it directly to a system, but no luck because it looks like it's the HDD mobo that died.

Also stupid young me did not realise back then that, even if a "moved" the data, the original data is still most likely in their original location, and so, never bothered a recovery.

Lots of good stuff lost that day.

And as with a lot of you, my disaster recovery system kicked up 10 fold. Now I got redundant local and cloud backup copies of all critical and otherwise unattainable data.

As you may have guessed, I never bought another Wester Digital product ever again. My internal HDDs are Segate, and external is a suprisingly long lived Toshiba Canvio.

Hi everyone, I've a desktop and in my area power cut is quite frequent. Although we've backup power, but sometimes it takes ~15 mins to come back.

Is UPS only option for desktops? Laptops nowadays have so long battery power, why there is no good option for desktops. UPS usually stays for 5 mins. Could you please suggest something better, as I've seen lots of desktops in posts? Thanks.

Modern technology is absolutely bullshit

I can't even

Now my keyboard on my phone is even too broken to complain about it

I wanted to look at someone's post history on a forum

To do so the forum wants an account. Ok. So I gave it my old junk Hotmail account during sign up for it to send me an email confirm so I can make the account so I can search. Well I'm refreshing this account for this confirm account email through the Gmail app on my phone because who even checks emails on computers anymore

Turns out, aside from this Hotmail spam email account having a lot of junk emails (it is my junk email account), there's this little pop-up that happens SOMETIMES claiming that it can't sync. I checked inbox and spam and the email isn't in there. So 1 out of 10 times I refresh there's this little "cannot sync" message that pops up and I click it. It claims my storage on my phone is too full to sync. Ok.

So I go try to find storage through the settings in my phone. It doesn't exist as a category anymore apparently. Thankfully phones have a search feature now -- because we can't have sane settings anymore so here's a search feature. First result it gives me is just device info. That's useless. It's just the hardware specs for my phone

Second it shows storage. 90% full apparently. That's odd. I have 132 gb. Thankfully it subdivided it by what's taking up space but it doesn't make much sense and a bunch of the categories don't open to anything

Apparently the fucking android operating system is 32 GB now? Well you're fucked if you wanna remove that. Apparently years of photos and videos is 20 gb, I can back those up and delete them. Similarly I have downloads in folders, and that's about 20 gb

Why are there 20 GB of apps? I literally have no apps!

Part of apps? Wtf is Gboard and why is it a gig

Why is my WEATHER APP using a gig of storage?

And none of the apps can I remove the storage they're using. The cache is like 600kb, and I can delete all data and it's using like 60 MB. So the fucking weather app executable itself is a gig of space? Wtf?

I deleted the data for Gboard and turns out that's the keyboard. So now all my keyboard settings are fucked.

Thankfully I wrote syncing scripts ages ago to sync various folders from my phone to my external HDD. I just had to connect it to the laptop and run the script on the external HDD. Problem? Well turns out no matter what I do I can't get the laptop to connect to the phone if it's USB file transfer mode. I can do photos. But this is gonna be more than photos.

So I do my sync backup script from the laptop to the external HDD. This will sync the camera, since I have sync thing sync my laptop and phone all the time, so I can just sync the laptop to the external HDD and then delete the older photos and get 20gb. Quick fix for now

Why do I need this quick fix?
Well
Get this
I've been having issues with my Gmail client for ages. It just won't display new email notifications which is really annoying because I need to know when emails get sent to me.
Now I'm thinking, maybe I can de-sync older emails and have more storage space maybe? But that's not an option anywhere. Actually, I can't even unconnect an email address from my phone. Gmail doesn't even let you do that

What the flying fuck is the state of modern technology

Now I have to go figure out what my fucking settings were for my stupid phone keyboard

The 90s were much fucking saner than this garbage. I don't need a 32 GB operating system on a phone. Is this fucking windows 8? And let me fucking tell YOU how many fucking emails you should sync to my phone. Holy shit what the fuck is all this

At least my Linux scripts fucking work like I wrote them

Started a new job as junior developer. One of my first task was to sent a simple notification on an event in out product. Write the code, test that it works, push to devops.

Code compiles, tests pass, it’s deployed to internal test env. Check that my notification works in the test env. No problem.

It’s deployed to the customers test environment. It works and customer accepts it for prod.

We release to prod and of course it fails. Seems to be a simple string.Format that fails for god knows why. After 3h of debugging on prod without success we decide to roll it back.

Today we decided to try it on a backup of the prod db since one of the strings was taken from the db. Still working. No matter what data I input when trying it locally it still wont reproduce the issue we saw on prod.

Fuck this

Dependency hell is the largest problem in Linux.

On Windows, I just download an executeable (.exe) file, and it just works like a charm! But Linux sometimes needs me to install dependencies.

At one point, I nearly broke my operating system while trying to solve dependencies. I noticed that some existing applications refused to start due to some GLIBC error gore. I thought to myself "that thing ain't gonna boot the next time", so I had to restore the /usr/lib/x86_64-linux-gnu/ folder from a backup.

And then there is a new level of lunacy called "conflicting dependencies". I never had such an error on Windows. But when I wanted to try out both vsftpd and proFTPd on Linux, I get this error, whereas on Windows, I simply download an .exe file and it WORKS! Even on Android OS, I simply install an APK file of Amaze File Manager or Primitive FTPd or both and it WORKS! Both in under a minute. But on Linux, I get this crap. Sure, Linux has many benefits, but if one can't simply install a program without encountering cryptic errors that take half a day to troubleshoot and could cause new whack-a-mole-style errors, Linux's poor market share is no surprise.

Someone asked "Why not create portable applications" on Unix/Linux StackExchange. Portable applications can not just be copied on flash drives and to other computers, but allow easily installing multiple versions on a system. A web developer might do so to test compatibility with older browsers. Here is an answer to that question:

> The major argument [for shared libraries] is security, that if there is a vulnerability in a commonly-used library, then only that library has to be updated […] you don't have to have 4 different versions of a library installed

I just want my software to work! Period. I don't mind having multiple versions of libraries, I simply want it to WORK! To hell with "good reasons" for why it doesn't, and then being surprised why Linux has a poor market share. Want to boost Linux market share? SOLVE THIS DAMN ISSUE!.

Understand that the average computer user wants stuff to work out of the box, like it does in Windows.

Having gone to a bank to reset a password again today (Yes, I forgot it for like... 3rd time, don't judge me, its my backup bank account I need to access like... once a year), I was once again made to think - I come in, give them my state ID by which they authorize that I can even make a password reset request.

Then they give me a tablet to... sign a contract addendum?

Its not the contract part that always makes me stop and think though - its the "sign" part.

I'd wager that I am not the only one who only ever uses a computer to write text these days. So... My handwriting got a lot jerkier, less dependable. Soooo... My signature can be wildly different each time.......

And if my signature varies a lot... then... what is the point of having it on a piece of paper?

I know its just a legal measure of some sort... And that, if it came down to someone impersonating me and I'd go to court with the bank, there would be specialists who can tell if a signature was forged or not... But...

Come on, the computer world has so much more reliable, uncrackable, unforgable solutions already... Why... Don't all folks of the modern world already have some sort of... state-assigned private/public keypairs that could be used to sign official documents instead?

It costs money, takes time to develop etc... But... Then, there would not only be no need to sign papers anymore... And it would be incredibly hard to forge.

The key could even be encrypted, so the person wishing to sign something would have to know a PIN code or a password or something...

tl;dr: I hate physical signatures as a method of authentication / authorization. I wish the modern world would use PKI cryptography instead...

took few days leave from my internship

as soon as I stepped in the train, i got a call from friend saying client's site is down

on checking, found that the aws instance crashed

couldn't even check the system with ssh
after detaching the volume and attaching it with another working instance we found that the system files and directories are missing

no backup was there

i wish i could give a cactus to amazon

Decided to get myself a new phone as the 2013 model is getting morally obsolete. I can still flash Lineage/Android10 and it will run it. But the apps (e.g. Google Maps) nowadays are such a performance hog it can't cope.
So I've searched through gsmarena, prioritizing smaller phones, picked myself a candidate and had it on my desk after few days.
The 2013 mindset still the same "do the backup before touching the phone in any way". How do you do backup? Unlock bootloader, boot/flash TWRP, run backup.
I've realized only after the unlocking and some googling, that the unlock automatically deletes TA partition and stored DRM keys. So advanced licensed features of the phone are gone. Also there is no way to lock the bootloader again, so I've lost the SafetyNet validation too.

tl;dr: I've fucked up my new phone as a first step after I got it.

Quotes are paraphrased (unless *) to protect the incompetent and stupid (or more the case: client and I'm reducing risk of exposure)

Situation: We have a program that opens sqlite database files. Occasionally new versions of the program needs to upgrade these files.

Program UI: To proceed you need to upgrade your database. It is recommended you backup your database before proceeding. Hit Yes to continue or No to abort.

Client: How do you back up a model once it has upgraded? If I hit No the program closes leaving me no option to backup the model.

Support: *The easiest way of backing up a model before upgrading is creating a copy of the file and keeping it in a separate folder*

Client: *Haha forgot about being able to do that outside of* <program name>

TL;DR: engineer in technical role who is probably getting paid $150k+ forgets it is possible to make a copy of a file.

Dropped my trusty old Note 2 backup device this morning and it just keeps on ticking. No smoke just a nasty cracked screen. Sort of gives it character 😊

I really don't understand this particular Government Department's IT Unit. They have a system and network to maintain except:

- They don't have a DBA
- They don't have a dedicated Network Engineer or Security Staff
- Zero documentation on all of the systems that they are taking care of (its all in each assigned particular staff's brain they said)
- Unsure and untested way of restoring a backup into a system
- Server passwords are too simple and only one person was holding this whole time and its to an Administrator account. No individual user account.
- System was developed by an in-house developer who is now retired and left very little documentation on its usage but nothing on how its setup.

But, the system has been up and operational for the past 20 years and no major issues whatsoever with the users using it. I mean its a super simple system setup from the looks of it.

1 App Server connected to 1 DB Server, to serve 20-30 users. But it contains millions of records (2GB worth of data dump). I'm trying to swing to them to get me on a part time work to fix these gaps.

God save them for another 20 years.

TIFU by showing login data during presentation

I was presenting my school project when my teacher asked if I could show him the source code. I said ofc, just let me login to the FTP server. I completely forgot that it was also shown on the big screen, and a random funny student logged in and tried to replace the index file with a joke file. Of course, he didn't want to make damage, so he made a backup. But this backup caused the problem, because he connected to the FTP through Windows Explorer (wtf?), and when he made a copy of the original file, it was renamed to "Copy of xy", but in a localized version, which contains special characters. Because of these characters, some FTP clients couldn't even connect, others just couldn't interact with the file. No download, no rename, no delete, nothing. After trying out like 8-9 FTP clients, I just remembered that I could rename it in PHP. Well, it got deleted instead of being renamed, but at least it wasn't there anymore. I have spent like half more hour with searching for a backup version on my computer until I found it.

TL;DR: showed FTP credentials during presentation on big screen, random student accessed and renamed a file, special characters in name fucked up the server, luckily I found a backup.

2 years ago, the first day in Viareggio(Italy):
walked 30min on the beach searching the "free beach", walking with cool bag, umbrella, and backpack in the hot summer sun...

finding the "free beach": drop everything I had and run to the water to cool down...

forget iPhone in my swimming trunks :(
1. five weeks without a phone (really relaxing;))
2. have no iOS device for restoring the backup... because I bought an oneplus 3

This was more than 15 years ago. We migrated a bunch if data (home to a new server and repurposed the old one, the same night. This was not the first task on that allweekender, so it was around 3am on Sunday, with very little sleep, when I had to copy the data. I did that by logging in as admin and copying with Total Commander. Obviously, even admin did not have the permissions to some folders, so a lot of financial data were lost, as the users found out on Monday morning. We had no backup. Old server was not only reformatted, but the disks were used to build a different raid set. Luckily, one of the users who had access to this data kept a backup on a flash drive. (If you're wondering, I should've used robocopy with backup mode)

Client be like:
Hello,

could you please restore our database from today's backup?

At a first glance - nothing out of the ordinary. Daily backups are standard...

Until we get the backed up snapshot running.

MySQLDump is somehow... Stuck. It... Doesn't seem to be doing... Like, anything. For ages. Wtf.

So we check the database. Connect, change scheme and... The commandline tool gets stuck, too. Weird.

So a layer lower, we check the datadir and... ls... After also getting stuck for a bit, lists about 500k files O_o

Yea, dumping a database with roughly ~250k tables is not fun. No wonder it takes ages.

The coolest project I ever worked on wasn't programming per second, though it involved a bit of scripting. The company I worked for had an FTP over TLS backup solution and it was put together with glue and paperclips by a guy that hadn't the slightest idea what he was doing. In order to conform with the insurance, data had to be encrypted. I setup a raid-ed server with full disk encryption on the raid volume that fetched the key over the network at boot from another secure server. I wrote a series of scripts for provisioning users and so on. The backup connections was sftp using a ssh tunnel, the users were chrooted to their own home directories, and were unable to open shells. The system was 100x more robust and secure than the original. I set it up on short notice and received absolutely no recognition for saving the company's ass, but it was definitely a fun project.

My mate just pen-testing on running production server using admin credential.
Guess what happen!
And no backup!

What a day!