Most people.
"Dance like no one is looking"

Me.
"Sudo like you have backups"

Too much stupid today..

Client: "Our server is almost full, fix it"

Me: "Okay, you can upgrade the VM, or I can see if theres anything that can be cleaned up?"

C: "Clean it up"

Me: *Finds 5 extra MySQL DB's that amount to 50-60GB in total*

Me: "Well I can remove some of your extra databases, and some of the local backups, as they're also backed up offsite. They're currently unused and dated back to the beginning of 2017, everything is out of date anyways."

C: "No"

Me: "Okay, is there a reason?"

C: "No"

Me: "Okay, so we'll upgrade the VM?"

C: "No"

Me: "I'm sorry, but those are really your two options"

C: "Just fix it"
...................
...................
...................
What in the ducking fuck !@#$#%?
If you're so fuggin smart, then why do they pay me?
I'm clearly stupid and unable to read your mind.

A "support" guy my boss got in. I had told my boss numerous times, "Get rid of this guy, he's only wasting our time and money. And he's going to end up doing something where we will end up having to put out the fires."

Sure as a pair of nuts on a squirrel, this crazy bastard goes and DELETES a client's database. Yes folks, in fucking production. A live database. The heart of the business' transactions are... *poof*... GONE!!!

No backups for the day! No synchronisation beforehand! No nothing... just GONE!!! Fucking flat-lining!

Well, when I realised what he has done, I had to remove myself from the room before shit got outta hand!

I told the boss man that is the last straw and he needs to go...

The long and short of it...
- The client had luckily only lost about half a days data.
- I'm no longer at the company.
- This dumb fuck still is.

I had just started my new job and deleted 3 years of data that the client had spent over £450,000 collecting 😱

another developer used my PC to quickly access the clients database while I was out the room as I had sql management studio open. I went back to my PC thinking I was connected to my local database, did a few truncate tables to test my software and :0 minutes later I get a call asking why there was no data on the server!

Thank god for backups 😓

Facebook publicly announced that it won't build a backdoor into its services for the intelligence agencies as for the latest requests to weaken/remove the encryption.

I can only imagine the intelligence agencies going like this now:

NSA director: Alright, as expected they said no so they won't have more damage to their public image, lets go for plan A 2.0!
NSA employee: Aaaand that is?
NSA director: Serve them a FISA court order requiring them to do this shit anyways but also serve a gag order so they can't tell legally.
NSA employee: Ahh, fair enough, I'll get that rolling. By the way, how did we do this with WhatsApp's encryption again?
NSA director: Oh that one was simple. There's a backup function which nearly everyone uses on either Android/iOS which does plaintext backups to Google Drive/iCloud.
NSA employee: Oh, okay. How do we access that data again?
NSA director: PRISM/XKeyScore!
NSA employee: Right, but then still the issue of how we even collect the encrypted messages from Facebo...
NSA director: PRISM/XKeyScore as well, don't worry about that.
NSA employee: But, how'd we justify this....?
NSA director: We probably never have to as these programs operate outside of the public view but otherwise just call terrorism/pedophelia... BAM, done.
NSA employee: Gotya, let's put this into motion!

I’m kind of pissy, so let’s get into this.
My apologies though: it’s kind of scattered.

Family support?
For @Root? Fucking never.

Maybe if I wanted to be a business major my mother might have cared. Maybe the other one (whom I call Dick because fuck him, and because it’s accurate) would have cared if I suddenly wanted to become a mechanic. But in both cases, I really doubt it. I’d probably just have been berated for not being perfect, or better at their respective fields than they were at 3x my age.

Anyway.
Support being a dev?
Not even a little.

I had hand-me-down computers that were outmoded when they originally bought them: cutting-edge discount resale tech like Win95, 33/66mhz, 404mb hd. It wouldn’t even play an MP3 without stuttering.

(The only time I had a decent one is when I built one for myself while in high school. They couldn’t believe I spent so much money on what they saw as a silly toy.)

Using a computer for anything other than email or “real world” work was bad in their eyes. Whenever I was on the computer, they accused me of playing games, and constantly yelled at me for wasting my time, for rotting in my room, etc. We moved so often I never had any friends, and they were simply awful to be around, so what was my alternative? I also got into trouble for reading too much (seriously), and with computers I could at least make things.

If they got mad at me for any (real or imagined) reason (which happened almost every other day) they would steal my things, throw them out, or get mad and destroy them. Desk, books, decorations, posters, jewelry, perfume, containers, my chair, etc. Sometimes they would just steal my power cables or network cables. If they left the house, they would sometimes unplug the internet altogether, and claim they didn’t know why it was down. (Stealing/unplugging cables continued until I was 16.) If they found my game CDs, those would disappear, too. They would go through my room, my backpack and its notes/binders/folders/assignments, my closet, my drawers, my journals (of course my journals), and my computer, too. And if they found anything at all they didn’t like, they would confront me about it, and often would bring it up for months telling me how wrong/bad I was. Related: I got all A’s and a B one year in high school, and didn’t hear the end of it for the entire summer vacation.

It got to the point that I invented my own language with its own vocabulary, grammar, and alphabet just so I could have just a little bit of privacy. (I’m still fluent in it.) I would only store everything important from my computer on my only Zip disk so that I could take it to school with me every day and keep it out of their hands. I was terrified of losing all of my work, and carrying a Zip disk around in my backpack (with no backups) was safer than leaving it at home.

I continued to experiment and learn whatever I could about computers and programming, and also started taking CS classes when I reached high school. Amusingly, I didn’t even like computers despite all of this — they were simply an escape.

Around the same time (freshman in high school) I was a decent enough dev to actually write useful software, and made a little bit of money doing that. I also made some for my parents, both for personal use and for their businesses. They never trusted it, and continually trashtalked it. They would only begrudgingly use the business software because the alternatives were many thousands of dollars. And, despite never ever having a problem with any of it, they insisted I accompany them every time, and these were often at 3am. Instead of being thankful, they would be sarcastically amazed when nothing went wrong for the nth time. Two of the larger projects I made for them were: an inventory management system that interfaced with hand scanners (VB), and another inventory management system for government facility audits (Access). Several websites, too. I actually got paid for the Access application thanks to a contract!

To put this into perspective, I was selected to work on a government software project about a year later, while still in high school. That didn’t impress them, either.

They continued to see computers as a useless waste of time, and kept telling me that I would be unemployable, and end up alone.

When they learned I was dating someone long-distance, and that it was a she, they simply took my computer and didn’t let me use it again for six months. Really freaking hard to do senior projects without a computer. They begrudgingly allowed me to use theirs for schoolwork, but it had a fraction of the specs — and some projects required Flash, which the computer could barely run.

Between the constant insults, yelling, abuse (not mentioned here), total lack of privacy, and the theft, destruction, etc. I still managed to teach myself about computers and programming.

In short, I am a dev despite my parents’ best efforts to the contrary.

no backups we lose data like real men

What seems to be the problem? Oh, is your wordpress site hacked/infected with malware?

So I guess you decided to disable updates because it might break your shitty little site? And I guess you thought those warnings you got from me and multiple colleagues about what could happen when you didn't update your wordpress bullshit weren't that serious?

Hold on, you want *US* to restore *YOUR* hosting backups?

Hahahahahaha-no.

Go clean up your own fucking bullshit. But, before you click that restore button, please take a cactus, carve 'I am a stupid wordpress cunt' into it, dip it in a bathtub of with blood mixed-infected cum and shove it up your ass.

Oh yeah I'm aware that that won't help your situation but it might keep you from reproducing and at least it'll give me some satisfaction.

There's one kind of ticket I fucking love. Wait, YOU LOVE TICKETS?! No, except for one.

Tickets where people ask us to restore backups.

Why do I love them?

Because the only fucking thing I need to do is reply with a link to a helpdesk item on our website because backups aren't something within our support range because it's easy for customers to do.

So whenever a ticket about restoring a backup comes in, it's as easy as opening the ticket, pasting the link with "I'd like to refer you to the following resource: " and pressing fucking "reply"

😄

My brother did something so stupid, I'm even doubting my relationship to him.

So yesterday he goes of to a friend of his and takes his MBP with him.

Later that evening he messages me : dude i got trolled big time, but reaaaally big time.

I ask him what's up, and what he replied to me, i still cant comprehend.

He said that he had lag on his mb when playing a game so he went to the internet and he came across a post with a command ... 'sudo rm -rf /' and someone else replying 'thanks bud, solved the problem for me'. So he went ahead and removed his root partition lol . I was like : WHY WOULD YOU EVEN ENTER A COMMAND YOU DONT KNOW WITHOUT LOOKING IT UP.

It made me think of another post here 'sudo like you have no backups'
Lol

At my previous job, the person in charge of the Phabricator server didn't have a backup system in place. I yelled at him until he implemented one.

He had the server perform backups to the same drive. I yelled at him again, to no avail.

Well, after awhile the hard drive started failing, and it would only boot intermittently. After a lot of effort, he was able to salvage part of the backup data, but no more, meaning we lost a lot of bug reports and feedback, and developer tickets. We were able recover all of the older lost tickets from a previous server, so overall the loss was pretty small.

But I think he learned his lesson.
He definitely learned to listen.

What an awful day :(

The server where I host my 4 clients websites crashed.

Unable to reboot from the console.

I contact the support. 15 minutes later: "we'll look at this"

No news for 1 week despite my messages.

Then... 1st ticket escalation... 2nd ticket escalation... 3rd ticket escalation...

Answer: "Sorry, your server is down and cannot be repaired."

Fuck.

I ask "is there any way to get my data back?". Answer: "No, because we would shutdown the whole bay and all our clients would be impacted".

Fuck.

I subscribe to another server, at another provider.

I look at my backups... shit, the last one is 4 month ago!!

I restore the first website: OK

I restore the second website: OK

I restore the third website: My new server is "too recent" and not compatible. with this old Wordpress. Fuck! I'll look at this later...

I restore the fourth website: database is empty!! What??? I look at the SQL backup for this site... it failed...

I lost ALL my 4th client data!!!

I'm sooooo piece of crap!

Never in my life I was scared as today.
I recently left a big company to work for a small one as the first internal developer.
Had a small issue in the production server. The fix was easy, just remove a single table entry. And... *drum roll*... I forgot to add a where clause. All orders were lost.
No idea if we had backups or anything, I quickly called the one other IT dude in the company.
He had no clue where are the backups and how to find them.
Having some experience with Nmap, I quickly scanned our network and found a Nas device.
There was a backup, whole VHD backup. 300GB of it, the download speed is around 512kb/s. No way I can fix it before management finds out, but then an idea came to mind. Old glorious 7zip. Managed to extract only the database files, sent them to the server and quickly swapped them. Everything was fine... The manager connected 5 minutes later. Scariest 45 minutes of my life...

I fucked up again.

Someone needed a flashdrive for a presentation. Forgot that I keep personal code on on of them for backups (it's always nice when there is no reception to have an offline copy of my code, for instance when we go out into the bush or to remote areas).

I gave them my flashdrive. Forgot it had the code on.

Now someone at head office has taken my program, claimed it as their own - and has just earned themselves a decent amount of money (praised in the monthly company newsletter).

My program has been stolen (by my own stupidity) and butchered. It has been made into something that it was never intended to be.

Fuck that guy.

Yesterday I fucked up big time.
First time in my career (I’m 23).

I just started working this week at a new company startup that had no programmers before me. They have a bunch of websites under their control that were on all different hosting solutions, and we decided to move them all to AWS.
I moved a few and was managing the folder rights on the server.
What happened next made my heart skip a few beats.
Bear in mind I’m not an expert in Linux.
I wanted to chmod to the folder I was currently in, and typed ‘sudo chmod -R 770 /‘ thinking for a while that the ‘/‘ would do it on my current dir.

Fuck. As I saw what was happening I pressed ctrl + c as fast as I could. But the damage had been done.

Fast forward a couple hours I deleted the broken instance, and created a new one from scratch. Had to do everything again but managed to do it in just a couple hours, moving as fast as I could without making such stupid mistakes again.

I was honest about it from the first minute it happened, and told my boss right away that I fucked up and had to start over, with a couple of hours of downtime.
Luckily not much was lost and I took a snapshot right after I was finished and will look into auto backups next week.

TL;DR: disaster averted!

Story time!

About a year ago, the company I work for merged with another that offered complementary services. As is always the case, both companies had different ways of doing things, and that was true for the keeping of the financial records and history.

As the other company had a much larger financial database, after the merger we moved all the data of both companies on their software.

The said software is closed source, and was deployed on premises on a small server.

Even tho it has a lot of restrictions and missing features, it gets the job done and was stable enough for years.

But here comes the fun part: last week there was a power outage. We had no failsafe, no UPS, no recent backups and of course both the OS and the working database from the server broke.

Everyone was in panic mode, as our whole company needs the software for day to day activity!

Now, don't ask me how, but today we managed to recover all the data, got a new server with 2 RAID HDDs for the working copy of the DB, another pair for backups, and another machine with another dual HDD setup for secondary backups!

We still need a new UPS and another off site backup storage, but for now...disaster averted!

Time for a beer! Or 20...

That is all :)

We had issues with lack of disk space on our production SQL server. Another developer decided to delete the databases he thought weren't in use to clear some space.

Ever think about checking first?!

Production chaos!

I just can't understand what will lead an so called Software Company, that provides for my local government by the way, to use an cloud sever (AWS ec2 instance) like it were an bare metal machine.

They have it working, non-stop, for over 4 years or so. Just one instance. Running MySQL, PostgreSQL, Apache, PHP and an f* Tomcat server with no less than 10 HUGE apps deployed. I just can't believe this instance is still up.

By the way, they don't do backups, most of the data is on the ephemeral storage, they use just one private key for every dev, no CI, no testing. Deployment are nightmares using scp to upload the .war...

But still, they are running several several apps for things like registering citizen complaints that comes in by hot lines. The system is incredibly slow as they use just hibernate without query optimizations to lookup and search things (n+1 query problems).

They didn't even bother to get a proper domain. They use an IP address and expose the port for tomcat directly. No reverse proxy here! (No ssl too)

I've been out of this company for two years now, it was my first work as a developer, but they needed help for an app that I worked on during my time there. I was really surprised to see that everything still the same. Even the old private key that they emailed me (?!?!?!?!) back then still worked. All the passwords still the same too.

I have some good rants from the time I was there, and about the general level of the developers in my region. But I'll leave them for later!

Is it just me or this whole shit is crazy af?

left a company over 3 years ago because they wanted me to dumb my code down so that the other devs could understand it. they wouldn't allow me to use classes in my code lol. anyway, 3+ years later figured I would try to log in to some of the admin panels... passwords still the same. MySQL dbs... passwords the same... cpanel... passwords the same. smh. even if I still worked there the passwords should be changed every so often. top notch security right there. funniest part is they don't even do backups or use VCS for the code. sad sad company. glad I'm no longer there. my personal projects have more security, redundancy and fail over lol

The riskiest dev choice...

How about "The riskiest thing you've done as a dev"? I have a great entry for that. and I suppose it was my choice to build the feature afterall.

I was working on an instance of a small MMO at a game company I worked for. The MMO boasted multiple servers, each of them a vastly different take on the base game. We could use, extend, or outright replace anything we wanted to, leading to everything from Zelda to pokemon to an RP haven to a top-down futuristic counterstrike. The server in this particular instance was a fantasy RPG, and I was building it a new leveling and experience system with most of the trimmings. (Talents, feats/perks, etc. were in a future update.)

A bit of background, first: the game's dev setup did not have the now-standard dev/staging/prod servers; everything ran on prod, devs worked on prod, players connected and played on prod, etc. Worse yet, there was no backup system implemented -- or not really. The CTO was really the only person with sufficient access. The techy CEO did as well, but he rarely dealt with anything technical except server hardware, occasionally. And usually just to troll/punish us devs (as in "Oops ! I pulled the cat5 ! ;)"). Neither of them were the most reliable of people, either. The CTO would occasionally remote in and make backups of each server -- we assumed whenever he happened to think of it -- and would also occasionally do it when asked, but it could take him a week, sometimes even up to a month to get around to it. So the backups were only really useful for retreiving lost code and assets, not so much for player data.

The lack of reliable backups and the lack of proper testing grounds (among the plethora of other issues at the company) made for an absolutely terrible dev setup, but that's just how it was, and that's what we dealt with. We were game devs, afterall. Terrible or not, we got to make games! What more could you ask for!? It was amazing and terrible and wonderful and the worst thing ever, all at the same time. (and no, I'm not sharing the company name, but it isn't EA or Nexon, surprisingly 😅)

Anyway, back to the story! My new leveling system also needed to migrate players' existing data, so... you can see where this is going.

I did as much testing and inspection of my code as I could, copied it from a personal dev script to the server's xp system, ... and debated if I really wanted to click [Apply]. Every time I considered it, I went back to check another part or do yet more testing. I ended up taking like 40 minutes to finally click it.

And when I did... that was the scariest button press of my life. And the scariest three seconds' wait afterwards. That one click could have ruined every single player's account, permanently lost us players ...

After applying it, I immediately checked my character to see if she was broken, checked the account data for corruption or botched flags, checked for broken interactions with the other systems....

Everything ended up working out perfectly, and the players loved all of the new features. They had no idea what went into building them, and certainly had no idea of what went into applying them, or what could have gone wrong -- which is probably a good thing.

Looking back, that entire environment was so fragile, it's a wonder things didn't go horribly wrong all the time. Really, they almost never did. Apocalypses did happen, but were exceedingly rare, and were ususally fixed quickly. I guess we were all super careful simply because everything was so fragile? or the decent devs were, at least. We never trusted the lessers with access 😅 at least on the main servers where it mattered. Some of the smaller servers... well, we never really cared about those.

But I'm honestly more surprised to realize I've never had nightmares of that button click. It was certainly terrifying enough.

But yay! Complete system overhaul and migration of stored and realtime player data! on prod! With no issues! And lots of happy players! Woooooo!

Thinking back on it makes me happy 😊

At my old job, me and a colleague were tasked with designing a new backup system. It had integrations for database systems, remote file storage and other goodies.
Once we were done, we ran our tests, and sure enough. The files and folder from A were in fact present at B and properly encrypted. So we deployed it.
The next day, after the backup routine had run over night, I got to work and noone was able to log in. They were all puzzled.
I accessed a root account to find the issue. Apparantly, we had made a mistake!
All files on A were present at B... But they were no longer present at A.
We had issued 'move' instead of 'copy' on all the backups. So all of peoples files and even the shared drives have had everything moved to remote storage :D
We spent 4 hours getting everything back in place, starting with the files of the people who were in the office that day.
Boss took it pretty well at least, but not my proudest moment.

*Stay tuned for the story of how I accidentally leaked our Amazon Web Services API key on stack overflow*

/facepalm

Rather than singling out one person, I wanna present what I see as incompetent/stupid/ignorant:
- no will to learn
- failure to follow the very specific instructions & later asking for help when they FUBR sth & not even knowing what they did to fuck up in the first place
- asking how to solve stuff, then ignoring the suggestions & doing sth totally against recommendations
- failure to remember most basic stuff, especially if not writing it down to look at later when needed
- failure to check logs & 'google' stuff before asking why something isn't working the way they want it
- after two weeks, asking me how feature xy works, mind you they coded it, not me
- asking me why they did something in a specific way - WTF, am I a mind reader?! Who designed that crap?! Me or you?!!
- being passive/aggressive & snarky when told to do something or being asked why isn't it done already
- not testing their shit properly
- not making backups when upgrading (production) servers
- not checking the input value, no validation.. even after many many debacles on production with null ref exceptions
- failure to admit they fucked up
- not learning from (their) mistakes

The website for our biggest client went down and the server went haywire. Though for this client we don’t provide any infrastructure, so we called their it partner to start figuring this out.
They started blaming us, asking is if we had upgraded the website or changed any PHP settings, which all were a firm no from us. So they told us they had competent people working on the matter.

TL;DR their people isn’t competent and I ended up fixing the issue.

Hours go by, nothing happens, client calls us and we call the it partner, nothing, they don’t understand anything. Told us they can’t find any logs etc.

So we setup a conference call with our CXO, me, another dev and a few people from the it partner.
At this point I’m just asking them if they’ve looked at this and this, no good answer, I fetch a long ethernet cable from my desk, pull it to the CXO’s office and hook up my laptop to start looking into things myself.

IT partner still can’t find anything wrong. I tail the httpd error log and see thousands upon thousands of warning messages about mysql being loaded twice, but that’s not the issue here.
Check top and see there’s 257 instances of httpd, whereas 256 is spawned by httpd, mysql is using 600% cpu and whenever I try to connect to mysql through cli it throws me a too many connections error.

I heard the IT partner talking about a ddos attack, so I asked them to pull it off the public network and only give us access through our vpn. They do that, reboot server, same problems.

Finally we get the it partner to rollback the vm to earlier last night. Everything works great, 30 min later, it crashes again. At this point I’m getting tired and frustrated, this isn’t my job, I thought they had competent people working on this.

I noticed that the db had a few corrupted tables, and ask the it partner to get a dba to look at it. No prevail.

5’o’clock is here, we decide to give the vm rollback another try, but first we go home, get some dinner and resume at 6pm. I had told them I wanted to be in on this call, and said let me try this time.

They spend ages doing the rollback, and then for some reason they have to reconfigure the network and shit. Once it booted, I told their tech to stop mysqld and httpd immediately and prevent it from start at boot.

I can now look at the logs that is leading to this issue. I noticed our debug flag was on and had generated a 30gb log file. Tail it and see it’s what I’d expect, warmings and warnings, And all other logs for mysql and apache is huge, so the drive is full. Just gotta delete it.

I quietly start apache and mysql, see the website is working fine, shut it down and just take a copy of the var/lib/mysql directory and etc directory just go have backups.

Starting to connect a few dots, but I wasn’t exactly sure if it was right. Had the full drive caused mysql to corrupt itself? Only one way to find out. Start apache and mysql back up, and just wait and see. Meanwhile I fixed that mysql being loaded twice. Some genius had put load mysql.so at the top and bottom of php ini.

While waiting on the server to crash again, I’m talking to the it support guy, who told me they haven’t updated anything on the server except security patches now and then, and they didn’t have anyone familiar with this setup. No shit, it’s running php 5.3 -.-

Website up and running 1.5 later, mission accomplished.

# 3 weeks ago
customer informed us that the app will get quite some load since the beginning of July

# this monday
last spare hdd on database's SAN died.
I told everyone another hdd is to follow it very soon eithet whether nothing is done at all or if anyone attempts to touch the san array [cuz that's what redundant raids do...]. No fucks given by anyone, no attempts to have maintenance and planned outage within 24 hrs....

# this morning
another hdd has failed and now 1tb of data is lost. No way to restore backups cuz there is no database to restore them to....

# 20 minutes later I head out to get some popcorn. It's gonna be a fun week!

So... The planned heavy load [and revenue along with it] is not gonna happen... I guess they are gonna ha a week long outage.

That's what happens when you ignore warning shots fired at your face.

Because of hardware failure we had to move some vpns from one datacenter to another.

The team of highly untrained monkeys at my hosting provider were hired to do this. First they ran backups of all the systems. Then they started the moving process. A few hours later they were done. We got an email everything was back online.

So we restarted all our processes and no data was coming in from our Raspberry's around the country. So we start a little investigation. What did these buffons do, they changed our rsa keys.

So we kindly ask them to put the old keys back so we do not have to fix 200 changed key warnings on systems that are not remotely accesible.

Apperently something that can't be done because their back up process is automated and always makes new keys.

Holy fucking fuck, whats the point in having a backup its not an exact copy. Is this fucking normal?

Now I will be spending the next few weeks literally standing in cow shit reconnecting Raspberry's.

Thanks a fucking lot. Not!

Me to customer, several years ago:
Your server is over 10 years old, and the support for it will soon be gone together with the possibility to take backups. You should upgrade your website ASAP.

Customer said:
ok, we will look into it.

This day:
Customer:
Hi, our site is down and we need backups from yesterday. Can you please fix?

Me:
Sorry, no. *send copy of the email several years ago*

*Got a request for installing and configuring an online shop for a client*

Me: Do you have a web space already?
Client: No, I don't want to pay for it. (FYI: They only cost about 20€ a year)
Me: Okay, but free hosters are often slow and unprofessional. I really do not recommend using free hosting services.
Client: Doesn't matter, do it.
Me: *Working on the shop for several weeks, finally goes online*
One week later, client contacts me saying shop is offline. I realize the free hoster he used shut down their services (bankruptcy), resulting in the loss of about 90% of the work that I had done (no proper backups due to complexity)
Client: How can that even happen? You'll redo the shop, right?

♪~ ᕕ(ᐛ)ᕗ

“Don’t learn multiple languages at the same time”

Ignored that. Suddently I understood why he said that. Mixed both languages. In holiday rechecked it and it was ok.

Sometimes mistakes can lead to good things. After relearning I understood it much better.

“Don’t learn things by head” was another one. Because that’s useless. If you want to learn a language, try to understand it.
I fully agree with that. I started that way too learning what x did what y did, ... But after a few I found out this was inutile. Since then, I only have problems with Git

Another one. At release of Swift, my code was written in Obj-C. But I would like to adopt Swift. This was in my first year of iOS development, if I can even call it development. I used these things called “Converters”. But 3/4 was wrong and caused bugs. But the Issues in swift could handle that for me. After some time one told me “Stop doing that. Try to write it yourself.”

One of the last ones: “Try to contribute to open source software, instead of creating your own version of it. You won’t reinvent the wheel right? This could also be usefull for other users.”

Next: “If something doesn’t work the first time, don’t give up. Create Backups” As I did that multiple times and simply deleted the source files. By once I had a problem no iOS project worked. Didn’t found why. I was about to delete my Mac. Because of Apple’s WWDR certificate. Since then I started Git. Git is a new way of living.

Reaching the end: “We are developers. Not designers. We can’t do both. If a client asks for another design because they don’t like the current one tell them to hire one” - Remebers me one of my previous rants about the PDF “design”

Last one: “Clients suck. They will always complain. They need a new function. They don’t need that... And after that they wont bill ya for that. Because they think it’s no work.”

Sorry, forgot this one: “Always add backdoors. Many times clients wont pay and resell it or reuse it. With backdoors you can prohibit that.”

I think these are all things I loved they said to me. Probably forgot some.

TLDR; My 2TB HDD got wiped in one fell swoop by a 9-year old child.

You know... I've never been too great about keeping backups. Even to this day, I only keep one or two local backups and nothing on the "cloud".

So this was about 5 years ago. At the time, I was living together with my girlfriend - who would later become my wife. She had a son from a previous relationship, who at the time was 9 years old.

I had a small desk in the living room of our one-bedroom apartment, that I used for my computer, which has been a laptop for a long time now. One unfortunate thing about the layout of the apartment was that the wall plug near my desk was attached to a light switch.

I had a 2TB external hard drive - with its own power cable - plugged into my laptop. Then, things started to move in slow motion... The GF's son comes inside from playing, my GF asks him to turn off the light. He reaches over, and shuts off power to my laptop - and the external hard drive.

He must have hit that switch at JUST the right fucking time. The laptop ran on battery, no big deal. The hard drive, when I powered it back up - was wiped clean. I tried data recovery on it, but the HDD was encrypted, which makes things more complicated.

Needless to say, I was not happy. I never got that data back, but I did learn not to expose my hard drives to 9 year olds. Very dangerous little creatures.

You want to know the best part? He destroyed another hard drive of mine, a few years later. Should I tell that story?

!rant
Linux vs Microsoft

Well, this war is certainly one of the oldest. IMO,
Linux - great for automating stuff, free, and customisable.
Windows - user friendly, softwares much more easily available, much easier to use.

Frankly, I have tried using Linux a lot of times, but never liked it one bit. I am a GUI fan and hate to type commands for every little thing. Plus installing Ubuntu wiped out my disk once and I lost all my school memories ( this was in 2008, I didn't know much about backups, was quite young) ,so I am quite vary of it. I just don't feel it to be intuitive. Just to do a simple task, I loathe to learn difficult commands, and just read the syntax.

However, I have no bias against people who use Linux.

It is like religion, live and let live, follow whatever suits you.

On devrant, why's there so much hate for Windows? Because it is paid? Because it has updates? So what!
I never had a problem with it, I update once a month, takes 10 mins. If you set up your active hours correctly, it works great, you can disable updates also. Windows 10 is highly stable. It is paid, but in my country almost all laptops come with windows preinstalled. The OS-less laptops are about $10 cheaper, which is not that much to freak about.

Would love to hear your views and logical arguments.
Please be polite.

Tech support for Friend...
Again...

Ok...
A friend ran down the hallway to my Appartement, I opened the door for him and asked him "What The Fuck... Is wrong with you?!" He was just coughing like...
My... Laptop.... Is... Slow...
When he then eventually came to the point where he could Breath normally again I asked him to login into his by then super with shitty software bloated PC... It was a mess... A Desktop full of Shit... only Shit... I
then opened Chrome and Task Manager...
and holy shit... EVERYTHING! Literally ANYTHING was under 100% Load... Yet his shitty Fans werent spinning up... Shutdown the Laptop and opened it... not to my surprise it was full of Dust... I cleaned it and booted it again... still 100% Load... I couldnt do shit without being bombarded with porn and Hentai Ads... So... I asked him "Did you make any backups to this machine?" He was like.. "*Really long explenation* No *Really long Explenation*"
Great... i thought to myself... Then I said to him his PC was so trash that i only could Reset it to 0... He was like DO ANYTHING YOU CAN LITERALLY ANYTHING!
I then explained him, Resetting this PC Means literally 0 A Fresh start, nothing of your data will be there literally nothing...
He again Was like DO IT!
So I did... When it was done i showed him the result... He FREAKED OUT LIKE THE WORLD WAS GOING TO FUCKING END, HE SCREAMED AT ME WHAT THE HELL HAVE YOU DONE? REVERSE IT! REVERSE IT NOW!
I Then said to him, I reset it to 0, I said this would happen...
He then took his now perfectly running Laptop and Ran away...

tl;dr
Tech support for friend, Reset PC to 0, Freaked out AF...

!rant

So I am building a website and already finished it and ready to be deployed.
Got server access and found that there are many other websites files and folder.

Note : multiple copies of website that I am working on.

So I raise my concern with client about which folder is actually handling the website. She contact the old dev guy and got the webroot for that website. She told me to delete everything and clean it up and make tge current websites available.

I realize that there are many other websites may rely on those files and folder and inform about the consequences. And finally told her I will not touch any files except for the website I am working on.

So, I deployed the website and works good as expected. After 3 weeks clients comes back with an issue that one of her websites is notworking and trying to blame me for it.

Fyi : before my deploymemt she says that she will talk to hosting providers to clean their webspace. I am not sure on that yet.

So I tried to do auditing and found that after 2 days frm my deployment some has already wiped everything from those other websites.
I showed my audits and its someone did it.
You should contact your hosting provider or the other dev who she contacted for wiping.

From the begining I am telling that lady to take backups. She said no need.

The reason I didnt took it because I am not working on those website and obviously client isnt going to pay me for those back.

There is still more to go waiting for her resonses.

I hate GoDaddy, we all hate GoDaddy. We know this. I had VPS running a single site that was there for 8 years and only hanging on there for a few business reasons. It was to move this past May to another server, then June, then July ... continually waiting for the final phase to pass so I can move it. Backups running nightly to a separate GoDaddy FTP server ... for 8 years. All good for 8 years. Our company gets a new credit card ... I'm away and forget to update it with GoDaddy ... billing fails 3 times and they delete EVERTHING. No care about being a good customer for 8 years (14 total with other servers), no phone call, no nothing. Poof, gone. Now, I know I should have transferred backups locally or, at the very least, somewhere else. I just never expected this type of treatment. FUCK GODADDY!

"Wanted: Dangerous PHP Developer"

I edit on live during peak hours with no backups using basic Notepad over an unencrypted network. That dangerous enough?

It's march, I'm in my final year of university. The physics/robotics simulator I need for my major project keeps running into problems on my laptop running Ubuntu, and my supervisor suggests installing Mint as it works fine on that.
I backup what's important across a 4GB and a 16GB memory stick. All I have to do now is boot from the mint installation disk and install from there. But no, I felt dangerous. I was about to kill anything I had, so why not `sudo rm -rf /*` ? After a couple seconds it was done. I turned it off, then back on. I wanted to move my backups to windows which I was dual booting alongside Ubuntu.
No OS found. WHAT. Called my dad, asked if what I thought happened was true, and learnt that the root directory contains ALL files and folders, even those on other partitions. Gone was the past 2 1/2 years of uni work and notes not on the uni computers and the 100GB+ other stuff on there.
At least my current stuff was backed up.

TL;DR : sudo rm -rf /* because I'm installing another Linux distro. Destroys windows too and 2 1/2 years of uni work.

Had 2 days of vacation. Theoretically (plus weekend, plus 2 days) 6 days.

Worked today… At Saturday.

Some administrators forgot to properly check bandwidth limitations....

*rolls eyes*

We had a major version upgrade of some server software at Monday.

Guess why I got called...

Of course it MUST be the software upgrade.

It couldn't be the new hardware that was setup 2 weeks ago and on which a lot of "important" VMs were migrated.

*eyes roll inside till only white is visible*

The even more annoying thing is that it wasn't that hard to figure out.

Looking at monitoring, we had spikes on 20 Gbit/s (roughly 2.x Gigabyte/sec - Ethernet) connection of some server at roughly 1.9 plus Gigabyte/sec.

IO latency spikes that made the graph look like a heartbeat EKG with severe tachycardia...

*additionally to white eyes starts cursing in reverse latin*

Incompetent admin answer: Booboo that can only be your fault - the developers must investigate.

Me (just a tad more polite): Meep Meep mother fucker, get your shit together. If the software would eat that much, the network would be a niece chunk of charcoal. Plus the time (sending instead of links to monitoring pictures… guess the lazy fucktard who's brain is a vacuum didn't even bother to check it)...

NOTICE SOMETHING?!

Incompetent admin: It starts at the same time. Always.

After wasting roughly another hour of time discussing with him, I just hanged up the video call.

Called someone I knew from the admin department and turns out that - drumrolls please - the incompetent admin was someone who got recruited 3 months ago…

*turning into antichrist*

I then had a not so polite discussion about how the only competent people could take days off (all except incompetent admin were on vacation) and the seemingly incompetent fresh recruit - who by the way NEVER mentioned this - was the only one left of the admin department. Which would be bad alone, but no - he even got the 24/7 emergency support role for the whole weekend.

Sometimes this company and HR especially notoriously drive me insane...

Guess next week there will be some HR barbecue.

But yeah. After a lot of raging around we nailed it down to the traffic of backups and could fix it.

Roughly 4 hours of analysis, communication, raging and hatred.

Just one hour implementing shit.

*goozfraba*

I couldn't sleep. I was staring at the blinking cursor. A slow, comforting blinking. Like everyone else, I had become a slave to the JavaScript ecosystem. If I saw something like a new build system, or a new framework, I had to have it.

My client changed the requirements again. I'm in pain.

- "You want to see pain?" my colleague said. Go read Apple support forums. That's pain.

I became addicted. Every time I died and every time I was born again. Resurrected.
During the night, I was crying in the Apple forums for an official answer that would never come. During the day, I was surfing StackOverflow to fix my problems. You get "single-serving" friends there. They help you, you help them, and then you never see them again.

- "Then you install Stack and boom, you're done. It's that easy to go functional."

That's how I met him.

- "You know why they make so many javascript frameworks?"
- "No, why?"
- "So that they can distract you while they put backdoors in them. So that you don't have time to check all of their code".
- "You are by far the most interesting "single-serving" friend I've ever met"

Then, my hard disk died. Of course, I didn't have backups: nobody has enough space for all those node_modules folders. All my addictions, lost.

Then I wrote him. If you asked me now, I couldn't tell you why I wrote him. We chatted a lot.

- "It's late, I should really go search another hdd on ebay"
- "Ebay? You called me so you could have my old hard disk."
- "No, I..."
- "Come on."

He sent me his old hard disk. It was a 256MB hard disk, but it was fine for running Arch. Then he asked me to rant about my problems in front of him.

- "I want you to rant as hard as you can"
- "Are you serious?"

We ranted all night about our bosses and clients and their fucked up requests. We kept in touch, and after a while more people were ranting with us. Every week, he gave the rules that he and I decided.

- "The first rule of devRant is -- you don't talk about devRant. The second rule of devRant is -- you don't talk about devRant."

I like to think this is how devRant started. This might also be the reason why we never see @trogus, only @dfox. A lot of shit still needs to happen.

TLDR - you shouldn't expect common sense from idiots who have access to databases.

I joined a startup recently. I know startups are not known for their stable architecture, but this was next level stuff.
There is one prod mongodb server.
The db has 300 collections.
200 of those 300 collections are backups/test collections.
25 collections are used to store LOGS!! They decided to store millions of logs in a nosql db because setting up a mysql server requires effort, why do that when you've already set up mongodb. Lol 😂
Each field is indexed separately in the log.
1 collection is of 2 tb and has more than 1 billion records.
Out of the 1 billion records, 1 million records are required, the rest are obsolete. Each field has an index. Apparently the asshole DBA never knew there's something called capped collection or partial indexes.

Trying to get approval to clean up the db since 3 months, but fucking bureaucracy. Extremely high server costs plus every week the db goes down since some idiot runs a query on this mammoth collection. There's one single set of credentials for everything. Everyone from applications to interns use the same creds.
And the asshole DBA left, making me in charge of handling this shit now. I am trying to fix this but am stuck to get approval from business management. Devs like these make me feel sad that they have zero respect for their work and inability to listen to people trying to improve the system.

Going to leave this place really soon. No point in working somewhere where you are expected to show up for 8 hours, irrespective of whether you even switch on your laptop.
Wish me luck folks.

I had a dream that I was installing some sort of package and the PM warned that it was a known malicious package and I was like "This must be a mistake, no it isn't" so I installed it anyways and then an ASCII-Art Michael Jackson started dancing on my screen to loud music and I shut down the computer and woke up and panicked because I didn't have any backups

Does linux suck? Imho, Yes.
A lot of the people bash windows regarding automatic restarts, updates, bsod etc.
I may be unusually lucky, but the last bsod I saw was in 2014 because of a faulty synaptics driver.

I've really tried to use linux to see what the hype is all about. Quite frankly, it sucked. The first time it wiped out all my data, I realized the value of backups. Hence I do not have a single pic of my school now, thanks to complicated ubuntu mounting.
Next is driver support. When I plug in a device, I expect it to work. I don't want to spend a day googling for drivers.
Why the fuck would I want to use a black terminal which gives me a headache. Am I in 1980? Which sadistic asshole designed vim ?
I have seen linux developers who claim to be linux experts and love linux. They take so long to do simple shit. For god's sake don't tell me there are GUI versions of linux available. I'd rather work in windows 95.
Why in the world would anyone want to use ls to see the contents of a directory! It is seriously so fucking unproductive.
I can't just download a software, click next a couple of times, and be done. No no no. I've to do sudo apt get update. Then try to find the fucking package. And if all goes well, there's always the dependency issue which is going to bite me in the ass. If google and stackoverflow go down, most linux devs will die a cruel death.

Fuck you linux.
I'm not saying windows 10 is the best, but at least I don't have to crawl through shit to use it. If you don't like automatic updates, disable it you moron. It is easier than renaming a damn file in linux.

Can someone please explain to me WHY THE FUCK non devs feel like they know shit. I DON'T GIVE A FUCK ABOUT HOW YOU FOUND SOME UNTRUE SHIT ON GIZMODO. I'VE KNOW SO MUCH ABOUT THIS SHIT, AND YOU LOOKED UP THE FIRST EXAMPLE YOU COULD FIND THE SUPPORTS YOUR CASE. The most recent time this happened was OVER THE LAST FEW DAYS when this DUMBASS that my friends and I BULLY but HE STILL HANGS AROUND. (By bully, yes sometimes we are mean to him, but we're not out to get him. He comes to us and we don't wanna be with him). So after the SEVENTH groupchat (on two apps) he created that night, HE WANTED TO SWITCH BACK TO ANOTHER APP I SPENT A WEEK TRYING TO GET THEM TO SWITCH FROM PREVIOUSLY (It was whatsapp, i got them to switch to telegram). THEN HE TRIED to ARGUE with me about how TELEGRAM wasnt secure. HE SEARCHED "is telegram secure" on Google and chose the FIRST ARTICE from the previously mentioned, GIZMODO which says that TELEGRAM chats ARENT ENCRYPTED by DEFAULT. HOW THE FUCK DO DUMBASSES GET THIS KIND OF PUBLICITY. There's a difference between ENCRYPTION and END TO END DUMBASS. Then he told me whatsapp is more secure than telegram. NO ITS FUCKING NOT. In telegram, your encryption keys CHANGE every chat, or every 100 messages. To my best research, whatsapp only has ONE key per USER. I could go on forever about how chat backups in whatsapp are UNENCRYPTED or how FACEBOOK stores your data, but blocked you works to.

I am so sick of the stupidity and illogical reasoning of clients.

Client: Descriptions are no longer syncing. Can you please fix.
Me: Problem fixed and deployed.

Client: All the descriptions got overwritten by the sync descriptions. Can you please have manual uploads overwrite the descriptions that sync (but basically auto guess what the client wants). We may need a toggle.

Me: Toggle added.

Client: Can you go through the 100+ sites backups and restore all the product descriptions?

It's like are you serious right now!!??

Back to the cheeseburger concept here...

Client: Can I have a cheeseburger (comes with pickles, onions, tomatoes, lettuce), no pickles. A Coke? Oh, but I would like pickles on my cheeseburger.

Tender: Here is your order.

Client: Why did you put pickles on this!!?? I asked for NO pickles!

Tender: You added pickles towards the end, so we put the pickles in.

Client: No! I thought you would have known based off of my original statement that I asked for a cheeseburger with no pickles. That is the override!

Narrator: See how illogical things can get. We can't just assume/guess based off of illogical reasoning.

Dev: I think I just deleted data I shouldn't have. Can you do a rollback?
Me: No, I'm a DBA, I don't touch data. Did you commit?
Dev: Yes, I committed.
Me: Your data is gone.
Dev: But don't you guys do backups?
Me: We backup the Prod servers. We only back up Dev on request. Did you request a backup in Dev?
Dev: No.
Me: ...
Dev: The Testers are going to be pissed.

And this, ladies and gentlemen, is why you need properly tested backups!

TL;DR: user blocked on old gitlab instance cascade deleted all projects the user was set as owner.

So, at my customer, collegue "j" reviews gitlab users and groups, notices an user who left the organisation
"j" : ill block this user
> "j" blocks user
> minutes pass away, working, minding our own business
> a wild team devops leader "k" appears
k: where are all the git projects?
> waitwut?.jpg
> k: yeah all git projects where user was owner of, are deleted
> j.feeling.despair() ; me.feeling.despair();
> checks logs on server, notices it cascade deletes all projects to that user
> lmgt log line
> is a bugreport reported 3(!) years ago
> gitlab hasnt been updated since 3 years
> gitlab system owner is not present, backup contact doesnt know shit about it
> i investigate further, no daily backup cron tasks, no backup has been made whatsoever.
> only 'backups' are on file system level, trying to restore those
> gitlab requires restore of postgres db
> backup does not contain postgres since the backup product does not support that (wtf???)
> fubar.scene
> filesystem restore finished...
> backup product did not back up all files from git tree, like none of refs were stored since the product cannot handle such filenames .. Git repo's completely broken

Fuck my life

Son a fucking bitch!
I ordered a brand new shiney SSD for my Inspiron 7573 which came in today. I was all set up with system backups just in case and install media for Debian and Windows standing by. I pull out my tools and realize that the laptop isn't going to have a screw inside to hold the drive in since it didn't come with a SSD. No big deal, I pull out the magnetic bowl of extra screws from all the previous repairs I've done and start looking for the screw I need. I can't find the screw I need so I go online hoping maybe I'll find someplace in town that will have one single screw with no luck.

Now I get to wait 2 more days for the screw I need to arrive from Amazon.

A colleague of mine had to debug performance problems in a foreign, proprietary application that is ancient.

To be crystal clear: Only reason that thing exists is because some old geezers fear change.

Asked me for help cause it's an _ancient_ MS SQL server that is luckily running on hardware owned by us.

Finding the credentials was already a funny task.

We had to access the vault (not joking here, we have a physical vault for storing sensitive data and critical backups), grab a folder and find the necessary data cause no one ever dares to touch that thing.

The application is btw for a sort of ERP / inventory system that is used in some ancient shops not yet migrated...

Yeah. Story speaks for itself.

Anyway, after dusting off ourselves, we were able to connect.

Was a bit ... Interesting. Everything's in german. The worst kind of german.

After looking at the first tables, I started giggling.

My colleague knew immediately that this was a sign of danger (insert Simpson meme here), raised his eyebrows and asked "How bad is it....".

Me, still giggling, "lemme take a further look, this is gold".

*long sigh from the colleague*

Well... It ended with me putting my hands in front of my eyes, turning around and saying: "I cannot look at it anymore, it hurts too much...."

To summarize:
- German table names
- When a table exceeded 300 plus columns, they added another table with the same plus suffix "_ddd"… where ddd is an zero filled integer sequence like 001
- To join this mess, they created views... Named "generator" - Sequence Number ... Some had the beginning of table names appended, which doesn't make it less confusing.
- the process list was listing queries running longer than 5 mins.

Which isn't at all surprising when generating carrtesian products of N tables with left join.

I've seen shit.... I've seen a lot of shit.

But that shit scared me.

Forgot to secure my mongo db instance, found half the data gone, and a new db holding me at ransom , learn_how_to_recover_your_data , good thing offline backups are a thing.

Recovered in no time, never will I ever repeat this mistake

Gaming community of mine launched their slick new website with their new "ticket system" where people could put in tickets to get help by volunteers.

2 hours and an approval by one of the admins later I managed to inject forge http request into literally every form on that side. Modify permissions, delete users, edit tickets, put invalid values into every attribute of them... In other words break everything.
Turns out the whole thing was coded as a first time project by a person who has no clue about web development and noone is in charge of anything really. There are no requirements, no beta testing, no version control or backups, but at least they had a hard deadline. 🤣

Still not sure if I wanna fix their shit and do it properly or just enjoy seeing it crash and burn.

Due to hardware failure build server was wiped out.
So all configurations are gone.
- There is no code to recreate builds
- No backups :D

The bright side, we can finally do it right ;)

TL;DR: My devices all hate me and I needed to fix them all.

My Devices really love me.

I rooted my smartphone (LG G5) just yesterday. Everything went fine. Installed TWRP, SuperSU and some nice Apps that utilize root.

Today I was on the go (at CeBIT) and already had the Xposed Installer App on my phone, but didn't attempt installing it yet because I needed my phone for Maps and Messaging and the app had given clear warning about the bricking-potential.

So to the end of the day I get bored, send my last important Messages, installed the Xposed Framework...
... aaaand got stuck in a boot loop.

So I got on my way back home (thanks God I remembered all the trains I needed to take). On the way I had a lot of fun in the Recovery-Terminal and figured that I should be able to fix my phone with no problem at home because the installer made backups (unlike myself).

Coming back home and my pc was still running (should've shut down after installing updates).
The pc behaved odd and I couldn't shut it down properly, which led to cutting the power.
And upon booting my pc I got a ... give it a guess ...
...a bootloop (technically the animation just never ended).

So after I fixed my phone with my spare laptop (just transferred and executed the uninstaller for xposed) I fixed my PC too, which had an old broken dkms-driver.

The odd thing about this is, that this isn't technically a rant. I guess you can confirm that you can't find any swear words.
Because I ENJOYED fixing the devices. I already fixed my pc a couple of times was well as unbricking my rooted phones, so there was fairly little research involved.
I guess I'm now offically twisted.

Now, after my smartphone backups are transferred, I'Ll take my device apart and replace the camera glass which arrived today (and hope, no pray, that my sim card does still work after that)...
... after I blatendly copied a meme to get more attention. 😉

You know the worst thing about being a freelancer? You're expected to wear every fucking hat and you don't get normal hours.

Over the past few days I have been working with a client of a client attampting to fix his server. He's running CentOS on VMWare and somehow ended up breaking the system.

Upon inspection there was no way to fix his system remotely. It wouldn't even boot in recovery mode. So we've been attempting to recover his data so that we can reinstall CentOS and not have to start completely from scratch.

So for the past 3 days straight I have been remotely logging in to a Debian Live CD and manually sending folders to a FTP server of his. He has somewhere close to 30 sites on this server, and upwards of 1 million files in total.

Yesterday either the system freaked out or he did something, but the entire fucking system stopped responding which forced me to reboot it, reinsert the live CD, reinstall evertything, and re-mount his broken systems drives.

Here we are 3 days in, we're still not done, and I'm getting slightly pissy because if you don't know Linux well enough to fix this shit yourself, you shouldn't be acting as your own sysadmin for 30+ sites.

Also, backups are a thing right? VMWare also has snapshots. I know the extra storage isn't cheap, but it's a hell of a lot cheaper than paying soemone like me $35/hr to go and fix all of your shitty mistakes.

Have a couple I want to air today.

First was at my first gig as a dev, 4-5 months out of school. I was the only dev at a startup where the owner was a computer illiterate psycopath with serious temper tantrums. We're talking slamming doors, shouting at you while you are on the phone with customers, the works...

Anyways, what happened was that we needed to do an update in our database to correct some data on a few order lines regarding a specific product. Guess who forgot the fucking where-clause... Did I mention this boss was a cheap ass, dollar stupid, penny wise asshole that refused to have anything but the cheapest hosting? No backups, no test/dev/staging environment, no local copies... Yeah, live devving in prod, fucking all customers with a missing semi-colon (or where clause).

Amazingly, his sheer incompetance saved my ass, because even if I explained it, he didn't get it, and just wanted it fixed as best we could.

The second time was at a different company where we were delivering managed network services for a few municipalities. I was working netops at that time, mostly Cisco branded stuff, from Voice-over-IP and wifi to switches and some routing.

One day I was rolling out a new wireless network, and had to add the VLAN to the core switch on the correct port. VLAN's, for those who don't know, are virtual networks you can use to run several separated networks on the same cable.

To add a VLAN on a Cisco switch one uses the command:

switchport access vlan add XYZ

My mistake was omitting the 'add', which Cisco switches happily accept without warning. That command however can be quite disruptive as it replaces all of the excisting VLAN's with the new one.

Not a big deal on a distribution switch supplying an office floor or something, but on a fucking core switch in the datacenter this meant 20K user had no internet, no access to the applications in the DS, no access to Active Directory etc. Oh and my remote access to that switch also went down the drain...

Luckily a colleague of mine was on site with a console cable and access to config backups. Shit was over within 15 minutes. My boss at that time was thankfully a pragmatic guy who just responded "Well, at least you won't make that mistake again" when we debriefed him after the dust settled.

I was in second year of University when I joined the internship, I knew the business idea sucks and he wouldn't be able to carry out the operations either. Little did I know that I will work with the dumbest team ever, literally, the dumbest.

So, the major chunk of the software was outsourced to a consultancy. I was a tech intern, and we were developing an Android App that will save your parking location, let you reserve locations and all etc.

I knew I have stepped on a wrong turf, but again, I had nothing better to do that summer. So, for a very meager stipend, I said yes to a very stupid project. Let the stupidity flow...

~ The boss, had quit his job for this dumb idea with no funding, no team, nothing.

~ He was pursuing a certification course in Android Development from somewhere, where their final project will be a calculator!

~ He had little to no tech skills, hardly knew Java but was leading an Android App Dev project in Java. He had little to no managerial, marketing or sales skills either.

~ For a brief period, I had to work along with the consultancy guys to ramp up their work. They would take backups in a USB drive every evening, and share each others code using the same. VCS died a painful death that day.

~ They hardly wrote functions, rather, wrote very long code in the main (onCreate) function. Code style died of cancer.

~ They couldn't compress an image before sending it to a server. I had to do it for them.

~ Had no concept of creating utility classes.

And best of all,
~ Wrote 20 cases (switch case) with the same code! Instead of using a loop...

I just noticed that my desktop still has a DVD-RW in it that I never used.. which led me to think, I've got here like 5 DVD-RW's as well as 5 CD-RW's, but I never use them. Are there still any uses for them? Long-term backup storage for example? I've heard somewhere that data rot can happen over the course of a few years.. but would it really be a problem? I mean I've had these for over a year already and they look healthy to me. That said, they've never been used and just sat on the top of my desk in their enclosures all the time.

FWIW, I'm having 10TB of storage (as well as some other smaller stuff here and there.. maybe 500GB or so) in various hosts and 3 mirrors, so the need for backups is already fulfilled.. it's merely a curiosity of mine as in "ah, I still have these things huh".. so yeah.

I think that I've bought these DVD's for something like €6 for 5 pieces in a local store that doesn't specialize in this sort of thing, so I'm fairly certain that I could get them for cheaper elsewhere.. think the likes of AliExpress etc. Would they be a viable technology to maybe still invest in over hard drives for long-term storage purposes, considering that the cost of a hard drive these days is only €50 for 1TB? But I guess that tapes would be better for this purpose, no?

This was about 3 years ago. I’m on vacation and just getting off the plane, when my boss calls me on his cellphone. Apparently the crontab on our main file upload server had gotten nuked, and he was asking if there were any backups.

A word about this server. I work with video, so this thing is doing about a few gigabits of traffic incoming at any moment. The cron jobs are necessary to move and organize these massive files into a sane scheme for processing. Hundreds of drop folders receiving thousands of files resulting in terabytes of data every single day. Our storage vendor tells us we have the third largest deployment they know about.

No cron jobs mean all of this content is just sitting around piling up. I tell him sorry, try contacting $otherAdmin since he’s more familiar with that system.

A few days later, after the vacation, I come back in. $boss and $otherAdmin have reconstructed the crontab from scratch after an all nighter.

I ask how it got deleted.

$boss was training some people how to set up new customers on this file server, and he told the trainees to open the crontab in read-only mode. One of them ran:

crontab -r

Yes, we back up our crontabs now.

Client wanted a site for the 100th anniversary of an important local musician. They wanted to show calendar of events, biography, store, and more. We started the work 8 months before the commemorative date, and after 4 months, the site was 99% complete and waiting final review and approvement. 2 weeks of silence has passed, when I got an email saying their deleted the site from server and all backups, and now they wanted all the work back.

With luck, I could restore a partial backup, and the client didn't want to pay for redo the rest of the work.

10 months later (yes, after all the events has started and the site being off) they contact me again, asking to continue the work.

I was happy to say no.

Pretty fucking sure it’s Monday... Critical Server KO’ed during hours... Going to be a long day....

Sucks not having the funds to implement preventative maintenance and redundancy... Thank god for fucking backups.

To the Level C’s in our company.... take this as a wake up call you incompetent, undereducated, no dick-having-ass’, spitfucks!

Running a wild UPDATE statement against an inventory database, it was chaotic and didn't have backups.... yea I know 😅 the backup service died god knows when and no one noticed. I was only new at the time so #notMyFault

UPDATE stock_on_hand = 0; WHERE id IN(1,2,3,4,5,88,972,7388);
# rows affected 1,234,567,890

> I think I almost died inside.

Oh the fun that mess was to clean up.
The positive outcome of this was, we had backups working again not long after and the inventory counts where accurate after that stock take.

Lessons I've learnt so far on programming

-- Your best written code today can be your worst tomorrow (Focus more on optimisation than style).
-- Having zero knowledge of a language then watching video tutorials is like purchasing an arsenal before knowing what a gun is (Read the docs instead).
-- It's works on my machine! Yes, because you built on Lenovo G-force but never considered the testers running on Intel Pentium 0.001 (Always consider low end devices).
-- "Programming" is you telling a story and without adding "comments" you just wrote a whole novel having no punctuation marks (Always add comments, you will thank yourself later for it I promise).
-- In programming there is nothing like "done"! You only have "in progress" or "abandoned" (Deploy progressively).
-- If at this point you still don't know how to make an asynchronous call in your favourite language, then you are still a rookie! take that from me. (Asynchronous operation is a key feature in programming that every coder should know).
-- If it's more than two conditions use "Switch... case" else stick with "If... else" (Readability should never be under-rated).
-- Code editors can MAKE YOU and BREAK YOU. They have great impact on your coding style and delivery time (Choose editors wisely).
-- Always resist the temptation of writing the whole project from scratch unless needs be (Favor patching to re-creation).
-- Helper methods reduces code redundancy by a large chunk (Always have a class in your project with helper methods).
-- There is something called git (Always make backups).
-- If you don't feel the soothing joy that comes in fixing a bug then "programming" is a no-no (Coding is fun only when it works).
-- Get angry with the bugs not the testers they're only noble messengers (Bugs are your true enemy).
-- You would learn more than a lot reading the codes of others and I mean a lot! (Code review promotes optimisation and let's you know when you are writing macaroni).
-- If you can do it without a framework you have yourself a big fat plus (Frameworks make you entirely dependent).
-- Treat your code like your pet, stop taking care of it and it dies! (Codes are fragile and needs regular updates to stay relevant).

Programming is nothing but fun and I've learnt that a long time ago.

So, a few months back my mother had some issues with her windows 10 box not being able to do proper backups to a backup partition. At the same time I was pulling insane hours at work and writing on a eBook on commission for a guy, besides having small kids with on and off flu and shit.

Needless to say, I didn't have time to look at the backup issues. Well, even though my mom is one of those dogs you can't teach new tricks, she has always been resourceful enough to get help with things.

This time she picked up the phone and called Microsoft Support, got some guys to remote in and take a look. They messed around a bit and said they were done.

She phoned me up later that day to tell me how proud she was of herself for doing that. Of course, she skipped telling me the important bit about she actually calling them, rather describing it as "Microsoft was just on my computer and fixed it".

You can imagine my immediate reaction, cold sweat running down my back, adrenalin rushing in as I dug through the details of what had happened.

A few days later she calls me up again and tells me the problem is back, and we agree that even though the MS dudes was not able to fix it at first, she should try again, as she had a ticket to reference.

The next attempt by MS actually fried her partitions, and apparently they had f-ed up trying to delete and recreate the backup partition.

That's not the worst of it though. Since they fried her disk, her computer crashed and naturally the remoting won't work. In our country, they have no people on the ground to do hands on help, and they didn't have a partner near by. Her not having a win 10 usb stick, nor a spare computer to make one, she was in a surreal predicament.

She was also quite pissed, and pissed off mums are not to be messed with. She managed to get Microsoft to agree to cover the costs of a non-partner to visit her to fix the problems, and using her as the middle man, they made an agreement with the 3rd-party tech support company.

After the box was fixed though, some more issues arose... regarding billing. The 3rd-party tech support was unable to get in contact with the person at MS that was going to sign off on the bill, and again using my mom as the middle man, it was agreed that my mom, as the customer, was to be reimbursed for the bill to the 3rd-party.

Guess what... 3 months went by, with weekly follow-ups and nagging from my mom, and still no money...

At this time, I had time to help her, and after some digging and borderline stalking, we managed to get the phone numbers of some of the higher ups in my country, and she started calling them directly.

After talking to a couple who refused to help, she reached the Vice President of the country branch, and was finally able to talk to someone who gave a shit.

Still took over a month more to actually get the money, but now she had someone who actually gave her statuses, receipts and ETAs.

FUCK!

So, idiot me decided it would be a good idea to never get around to configuring my UPS to gracefully shutdown my server after a powercut lasting more than x duration...

Long story short, we had a powercut that lasted 4 minutes or so longer than the battery in the UPS could keep the server up for...

UPS died, server went pew, and after rebooting itself once the power came back on, my raid array wouldn’t mount anymore...

After Googling around, it seemed like running e2fsck would solve the problem.

Didn’t seem to do the trick... and tired me at 3am decided it would be a good idea to poke around.

Pretty sure I ran a command wrong, or two, because now I can’t even mount the fricken array in read only, and fsck complains with a shit ton errors...

Been researching for hours, and no dice...

Test Disk shows the ext4 partition, but fails to list any files...

I may have destroyed the tables or something... I’m a noob at this point.

I’m able to access files with the RStudio tool, however this doesn’t help with file names and directory structure 😭

Is it all over for my 5 years worth of photos and other bits and pieces that I don’t have any backups of ? 😂😭😭

If any of y’all are pros with data recovery and can help a fellow boi out, I’d be more than happy to pay for ya time !

So one of my clients got their wordpress site hacked and basically just redirects to scam links and well.. I looked at in the server file manager and their are like three directories with this wordpress site (not clones but the same?) one in the root, a version in a folder called old and another in temp.. with 3 separate wp databases.. DNS entries had malware redirects, the wp-content folder was writable to the public and contained a temp folder with tons of encoded malware and ip links to malicious sites.. there was encoded malware in index.php, has like 20+ plugins, oh and the theme uses a dynamic web builder so the code is basically unreadable in source and scattered.. and the redirects seem to happen randomly or at least on a new session or something. Oh.. and did I mention there are no backups? 😃

Updated the site Joomla from 3.6 to 3.7. Then got a call saying some pages weren't working. "No problem," I said, "we have backups." Restored a backup, but then found out it was corrupt... The entire site went down 😭

"let's use git for this game jam"

Wait! Don't go! I love git and use it on every project I work on! You'll have to hear me out here.

This was 4 years ago, at my first Global Game Jam. Every jam and game I'd worked on up to that point, I was the only Dev; no need for git, as backups were more than enough. I joined a group with high hopes for the game jam, with three coders and a proper art team.

The entire jam was "1 step forward 2 steps back", as git somehow constantly overwrote code as fast as we could write it.

By the end of the jam we barely had anything to show for our hard work. The takeaway isn't even about git. It's simply to never work with other people. Git is a great protocol but it can't stop people from accidentally fucking other people over. Every jam since, I've worked on my own and had a far better time of it.

I really felt like a badass one time when I managed to recover all projects on our dev server after a full meltdown of the HDD.

We had no recent backups, because our backup server was down for a few months, and our (at the time small) company was in a tight spot on finances, and couldn't get a replacement.

The problem was that the HDD on the backup server failed, but we were storing all projects also on the dev server, along with our local git repos (no GitHub at the time for us), but then the dev server HDD also broke, and I used every piece of data recovery software I found trying to recover the data, until one actually managed to read the raw data from the HDD and store it as a virtual drive, that I then used to try and build another partition index and it actually worked!

Lost about 10% of the data, but that was enough, as i managed to recover all the git repos and databases...

I don't even remember the tools that got the job done in the end, but that was one hell of a week, and at the end I felt like a true IT God!

True story!

PS: 2 weeks later we had a new backup server, another offsite backup solution and a GitHub account for the company. Was delayed on salary in order to manage it (me and the CEO both agreed to give our pay for one month to get them), but worth it!

Boss wanted me to make changes in company's website which was based on wordpres s.

I knew it could be done by tweaking some JS code, but I have very less experience with wordpress

But wordpress is easy man(Internet told me).

Give me 5 minutes, you will see the changes in production.
Being lazy af I directly logged in to ftp, checked out some files, updated some code, I was good to go.

Before pushing it, I opened the website and it was GONE ٩(๑´0`๑)۶

Now there was no public_html in the root.
I was fucked. I have accidentally deleted the website that had no backup.

And the best part I was on leave from
next day.

I was looking everywhere for backups, looked into google cache to get the contents. I have to recreate the complete site now.

Just when I was asking questions on choice of my profession and simultaneously looking here and there in FTP for backups,
I found the jewel "public_html".

It happens out that I have accidentally moved the folder to some other directory.

Phewww.

Moved it back to root. Site was up and running.

Reassured myself that I deserve to be a dev.

Backed up complete site, made the changes.

Uploaded it.

And the best part, amount of wordpress I learned in those three hours was way more than I could have learnt in many weeks.

Lessons Learnt :

A) ALWAYS keep backups.
B) You SHOULD NOT make changes on prod directly
C) You become superhuman when your brain know you are going to be fucked 😂

Back when SharePoint was still foreign to me, and I didn't know the pain of administrating it, I had the idea that files were copied to my local machine. I saw no need to preserve backups from before I started, especially since they already existed on the server, so I got rid of them.

Also hooked up to SharePoint was an email handler. Whenever a case was created or deleted, an email went out to the entire department. Guess what happened when I deleted 250,000 records?

Fortunately, SharePoint has a recycle bin. Unfortunately, restoring those files generated another 250,000 emails. To the whole department.

I bought many donuts to appease the crowd baying for my blood.

Well... I once accidentally deleted a classmates entire assignment. Basically we were working together on one and we had the code in Github, I had named the repo after the module code.
He was having some weird git issues and I thought it would be easier to just delete and re-clone on his machine. You can probably see where this is going.

Me: rm -rf <DIR NAME> Enter
Him: wait, which folder did you just delete

Turns out he had the repo cloned inside another directory with the EXACT SAME NAME, which also contained his previous assignment, the only copy of it in the entire universe (it was a group project and they did it all on his laptop with no source control, which i found hilarious).
It wasnt so bad since that assignment was already submitted and graded, but a bit of a fail on both our parts.

Resolved issues with iCloud and my moms massive photo library...

Months later. "What's Apple Music?"

It's just a streaming service mom.

"I've been paying $10 a month for it, why am I paying for it?"

"Do you like to listen to music a lot? Do you want it?"

"No I don't even know what it is, you must've done this with that apple cloud thing"

"Mom those are different things, I helped you with your cloud storage it's not the same"

"Cancel it for me!!😡"

All I did was clear my moms iCloud storage enough that she could do some photo backups from her phone, months ago. Never even moved her into the paid tier for the cloud storage, let alone an Apple Music subscription that she had for months and didn't want or use. Don't worry, I cancelled it for her.

PSA: DON'T I repeat DON'T use DD before noon.
You will end up like me and overwrite your primary data drive which you haven't backed up since over a year. This method has a 100% (1/1) success rate and will drastically decrease your will to live.
Also FUCK MONDAYS!

Sometimes In wonder if the support at my hosting company are a team of highly untrained monkeys.

I know jack shit about the finer details of server management, thats what I pay you guys for.

But you guys could at least try to be a bit more usefull.

If I ask you guys about the backup functions you could try to answer my questions instead of telling me what I can see on my screen myself. Because I can read pretty damm well.

After 3 days of getting no answers and not being able to backup to server with the tools you build and are telling us to use. The server died and now you are telling you can't recover the data?

How about fuck you! So we got no backup because of your fault you now tell us all our data is lost?

We got our own backups but rebuilding the entire server and infrastructure is going to take days.

Going to be a fun couple of days telling clients why shit isn't working.

!dev

Im working with morrons...

So someone had bright idea of having clone setup of our servers in other hosting company (in previous company we have rendundant setup). Whatever, maybe they want to be resistant to thermonuclear war or whatever, like the project wasn't underfunded already... Whatever, fuq it.

And with that, I have like really, really really fucking ABSOLUTELY BULLSHIT STUPID questions thrown at me.

So, this particular instance of bullshit started with trivial, literally "how much storage capacity we will need" I anwsered at least 4TB, preferably on redundant disk array, I've added small table what uses what and how much etc.

Than I got mail back...
"Thats not enough information:
1. What we need to say to company ABC
2. What we need to say to company XYZ
3. (this one actually had sense) Backups conception
4. **WILL WE PAY FOR SERVERS**
5. other important things (literally)"

So let's break it down.
Im backend guy. What the fuck do I know what you gonna say to XYZ or ABC. I dont give a shit, for me it's clicking setup new server and Im done for, you are overcomplicating as shit and require special care from hosting company that you will pay extra.
Next one, the killer one. What the fucking fuck. my anwser was literally "yes, we need to pay for servers, servers arent free."

Now tell me. How the fuck it is possible that someone can be such an idiot to ask questions like that. and I dont mean #3. maybe even #1 and 2 is like looking someone to throw responsibility on. But why the fuck I have to anwser mail that literally asked me if servers are free.

No, fuck off idiot, I have actual work. Take your bullshit and spread it somewhere else.

E:

and before anyone asks. No, Im not working in kindergarden but I often feel like I am indeed working within kindergarden full of 30+ mentally handicapped toddlers.

Beware: Here lies a cautionary tale about shared hosting, backups, and -goes without saying- WordPress.

1. Got a call from a client saying their site presented an issue with a third-party add-on. The vendor asked us to grant him access to our staging copy.

2. Their staging copy, apparently, never got duplicated correctly because, for security reasons, their in-house dev changed the name of the wp-content folder. That broke their staging algo. So no staging site.

3. In order to recreate the staging site, we had to reset everything back to WP defaults. Including, for some reason, absolute paths inside the database. A huge fucking database. Because WordPress.

4. Made the changes directly in a downloaded sql file. Shared hosting, obviously, had an upload limit smaller to the actual database.

5. Spent half an hour trying to upload table by table to no avail.

6. In-house uploads a new, fixed database with the help of the shared hosting provider.

7. Database has the wrong path. Again.

8. In-house performs massive Find and Replace through phpMyAdmin on the production server.

9. Obviously, MySQL crashes instantly and the site gets blocked for over 3 hours for exceeding shared hosting limits.

10. Hosting provider refuses to accept this was caused by such a stupid act and says site needs to be checked because queries are too slow.

11. We are gouging our eyeballs as we see an in-house vs. hosting fight unfold. So we decide to watch a whole Netflix documentary in between.

12. Finally, the hosting folds and enables access to the site, which is obvi not working because, you know, wrong paths.

13. Documentary finishes. We log in again, click restore from backup. Go to bed. Client phones to bless us. Client’s in-house dev probably looking for a cardboard box to pack his stuff first thing in the morning. \_(ツ)_/¯

As a Dev in college working for teams in college, I don't really have a need to use git, since most of 'self proclaimed prestigious programmer child prodigies' I work with have no idea what it is; but I use it anyways as good programming practice and ease of backups.

So I tried using a GUI client after months of the git bash, and even though I looked up a few tutorials (was embarrassed the whole fucking time). I ended up adding, committing and pushing via bash.

Can anyone explain me how is the GUI client helpful in large projects and stuff?

It's sometimes really anxiety inducing thinking that all data could be gone, if somebody decides to kill/discontinue/crash [see gitlab shitting 6 hours of data due to fucked backup strategy and shitty seperation of servers] your account/service, be it server, git-repos, backups, chrome syncs, games, music, sim card, ..

But there's simply no way of having a backup of absolutely everything (ignore DRM) - especially automated and abstracted away from you, so you don't have to do all that shit yourself

I am building a website inspired by devrant but have never built a server network before, and as im still a student I have no industry experience to base a design on, so was hoping for any advice on what is important/ what I have fucked up in my plan.

The attached image is my currently planned design. Blue is for the main site, and is a cluster of app servers to handle any incoming requests.

Green is a subdomain to handle images, as I figured it would help with performance to have image uploads/downloads separated from the main webpage content. It also means I can keep cache servers and app servers separated.

Pink is internal stuff for logging and backups and probably some monitoring stuff too.

Purple is databases. One is dedicated for images, that way I can easily back them up or load them to a cache server, and the other is for normal user data and posts etc.

The brown proxy in the middle is sorta an internal proxy which the servers need to authenticate with to connect to, that way I can just open the database to the internal proxy, and deny all other requests, and then I can have as many app servers as I want and as long as they authenticate with the proxy, they can access the database without me changing any firewall rules. The other 2 proxies just distribute requests between the available servers in the pool.

Any advice would be greatly appreciated! Thanks in advanced :D

Follow up on a previous rant:

I visited a customer to talk about the reporting discrepancy between two applications.

It turns out the applications were custom built by outsourced developers from Russia, that communicate with each other through a byzantine (and completely undocumented) series of web services, excel import/export tasks, and a customized SSRS environment.

These are spread across at least half a dozen servers, some on-premise and some cloud based, there are at least 3 SQL servers (2 running 2005, one running 2000), a 10 year old local install of TFS (which no one knows a username/password for), and who-knows-what-else.

They laid off their entire IT team years ago, and they have no backups.

I'm not certain anyone there even understands what the software is supposed to be doing beyond the most general terms.

No one knows if they even have source code.

Biggest case of "nope!" I've encountered in more than 20 years of IT experience.

Today was a SHIT day!

Working as ops for my customer, we are maintaining several tools in different environments. Today was the day my fucking Kubernetes Cluster made me rage quit, AGAIN!

We have a MongoDB running on Kubernetes with daily backups, the main node crashed due a full PVC on the cluster.

Full PVC => Pod doesn't start
Pod doesn't start => You can't get the live data
No live data? => Need Backup
Backup is in S3 => No Credentials
Got Backup from coworker
Restore Backup? => No connection to new MongoDB

3 FUCKING HOURS WASTED FOR NOTHING

Got it working at the end... Now we need to make an incident in the incident management software. Tbh that's the worst part.

And the team responsible for the cluster said monitoring wont be supported because it's unnecessary....

End of the world ?

Github/Bitbucket losing all its data while you are transitioning to a new system with no backups.

I'd rather face a nuclear war.

I run update without where on mysql console on production database Today.

CLASSIC

Just because I needed to fix database after bug fix on the backend of the application.
I thought I wrote good sql statement after executing it on my local machine and then everything got bad.

Luckily it was only one column with some cached statistics data and I checked that it was not important data before I actually started fixing stuff but still ...
Almost got hard attack afterwards.
Made a script to fix this column and it took me only 15 minutes but still...

Bug was caused in part I got no unit tests and application grow after 3 years of development from simple one for one customer and volumes of documents around 50k to over 40 customers and volumes over 2mil per month, don’t know how many pages each, just in one year after we completed all needed features.

I have daily backups and logs of every api operation but still.
I think this got to far for one backend developer.

I got scared that I will loose money cause I am contractor and the only backend developer working on it.
I am so tired of this right now I think I need a break from work.

Responsibility is killing me so hard right now.
It will take a week to get back to normal.

Created Linux instalation flashdrive on my notebook like thousand times before. Simple dd if=img of=/dev/sdb . Tried installing system from it but somehow doesn't work. And the it hit me. I have both magmetic drive and SSD in my laptop! So insted of flashdrive, I have bootable beging of my SSD where my encrypted lvm used to be :-( Luckilly I lost just EFI, boot, swap, rootfs, few git repositories and ccache.

So ... I'm just watching an anime (Musaigen no Phantom World) ... and this scene isn't related to the main plot. This is the intro of the third episode - the intros of the first two episodes were about the "phantoms", "illusions" ... :)

It starts with a blue screen and the little fairy (it's a phantom btw) said: "When the PC goes crazy, you panic quickly."

In the screenshot he says: "It is important to back up your data somewhere in case of an emergency."

A second screenshot is added in the comments.

i asked my dad for help with a GRUB issue (EFI file wasn't seen in my BIOS anymore, nor booted when pointed directly at, even after ALL THE CONFIGURATIONS POSSIBLE) and i walked away for a while, content he'd figure it out (there's still a few things he knows more than me about.) I come back 30 minutes later and he's zero-filled my main drive and is halfway through installing Win10. His reasoning? "I'm installing surveillance software since you won't give me your college passwords and I need access to your college's site and your account. I can't do that on Debian."

I didn't give him authorization for this, and I thought he had zeroed my backups drive too, but it turns out it was having I/O issues (my controller is finicky sometimes, a boot cycle with it removed fixed it, luckily I can't write to drives it doesn't like when it's being a shithead)

What do? I can't sue as he owns almost everything I use and the house I live in and would no doubt kick me out and take all "my" stuff, but I feel like this really can't go ignored. I can't just talk to him about it as he thinks anything he wants done has to be done as he sees himself as above all other people, so he just shouts me down...

Business Continuity / DR 101...
How could GitLab go down? A deleted directory? What!
A tired sysadmin should not be able to cause this much damage.
Did they have a TESTED dr plan? An untested plan is no plan. An untested plan does not count. An untested plan is an invitation to what occurred.
That the backups did not work does not cut it - sorry GitLab. Thorough testing is required before a disruptive event.
Did they do a thorough risk assessment?
We call this a 'lesson learned' in my BC/DR profession. Everyone please learn by it.
I hope GitLab is ok.

JUST DO GIT!

Anecdote from when I used to be an apprentice:
Setting: Small company, number of employees -> ε, direct superior is the founder and owner, no tech background

Boss: I've looked at this backup directory of the ERP /* why is he even going through that stuff?*/, it looks messy as hell, I want you to tidy that up
Me: Those are incremental backups, I can't just go and delete some of them, the application manages them by itself
Boss: Get it tidy!

TLDR: I need advice on reasonable salary expectations for sysadmin work in the rural United States.

I need some community advice. I’m the sysadmin at a small (35 employee) credit card processing company. I began as an intern and have now become their full time sysadmin/networking specialist. Since I was hired in January I have:

-migrated their 2007 Exchange server to Office 365

-Upgraded their ailing Windows server 2003 based architecture to 2012R2

-Licensed their unlicensed VMware ESXi servers (which they had already paid for license keys for!!!) and then upgraded them to 6.5 while preventing downtime on hosted VMs using tricky transfers and deployments (without vMotion!)

-Deployed a vCenter server to manage said ESXi servers easier

-Fixed a three month gap in their backups by implementing Veeam, and verifying its functionality

-Migrated a ‘no downtime’ fileserver to a new hypervisor host, implemented a ‘hot standby’ server as a backup kept up to date by the minute with DFS replication.

-Replaced failing hard drives in a RAID array underlying their one ‘business critical’ fileserver, which had no backups for 3 months at that time

-Reorganized Active Directory and Group Policy deployment from a nightmare spiderweb of OUs and duplicate policies

-Documented the entire old network and now the new one as I’ve been upgrading this

-Audited the developers AWS instances and removed redundant machines, optimized load balancing on front end Nginx servers, joined developer run Fedora workstations to the AD domain and implemented centralized syslog monitoring on them.

-Performed network scans and rewrote firewall exceptions to tighten security

There’s more, but you get the idea. I’ve now been tasked with taking point on an upcoming PCI audit which will be my first.

I’m being paid $16/hr US, with marginal health benefits. This is roughly $32,000 a year, before taxes.

I have two years previous work experience managing a third party Apple repair facility (SimplyMac) and every Apple certification for warranty repair and software troubleshooting. I have a two year degree in general sciences, with about 4 years of college credit (Two years of a physics education and two years of computer science after I switched focus) I’m actively pursuing a CCNA and MCSA server 2016 with exams paid for and scheduled.

I’m going into a salary negotiation in two months. What is a reasonable salary to request, from your perspective, for someone in my position?

Thanks in advance!

Allrighty, so we have a huge migration upcoming. The planning started early this spring. We've split the whole process into separate tasks and estimated each of them. Also marked all the tasks client should take care of itself so save funds and time. All-in-all the whole thing estimated like 4 months if we did it [single dev, tremendous amounts of communication with various parties, buy and prepare the infra, adapt app to the changes, testing, monitoring, etc.] and like a month if client did the tasks we shouldn't be doing. The funding for migration is time-bound and can only be used before December. Cool! We got notified that by the end of April we should be good to go! Plenty of time to do things right!

April comes. Silence. Mid-april we resch out to the client. Since there's plenty of time left migration is getting lower priority to other tasks. Well allright, sort of makes sense. We should migrate mid-July. Cool!

July comes. Client replies that everyone's on vacation now. Gotta wait for August - will do the quicker version of migration to make it on time. Well allright....

August comes. Everyone's vusy with whatever they've postponed during summer. Hopefully we'll start migration in September. Mhm...

September comes. We're invited to a meeting by project funders to explain tasks' breakdown, justify the time needed to make the migration. We're being blamed for surreal estimations and poor organization of tasks as nothing's happened yet... [they were the ones who always were postponing things....]. Moreover, they can only spare 20% of infra resources required for data alone anf they want us to make that enough for all environments, all components, all backups, all databases,... You get the pic.
The leader of the meeting semi silently mumbled to other participants 'Well then I'm afrsid we can't make a full migration in time.. Only partial. That's very unfortunate, very. That's why we should not have incopetent vendors [*glancing at us*]'

somehow we agreed we'll get the resources mid-November and we should be thankful for him bcz he'll have to pull some strings for... us..

I left the meeting with my fists squeezed so hard! But it's okay, we got smth useful: resources and start date. Although it leaves us with less than a month to do smth requiring a month for a sunny-day scenario. Nvm, still doable.

Last week we get an email that resources will be available at the beginning of December [after deadline] and we should start a full migration no sooner than Nov 12. Which leaves us with 50% of our estimated fucking optimistic scenario time and not enough resources to even move a single db.

Fuck I hate politics in dev... Is it wrong for me to want to tie them to a pole, set them on a veeery slow fire and take a piss on them while they're screaming their shitty lungs out? I'd enjoy the view and the scream. I know I would. And while enjoying I might be tempted to take a burning 20cm diameter wooden stick and shove it up their assholes. Repeatedly. Round-robin. Promissing them I'll take it out in 5 seconds and pulling it out after 2 minutes.

Can I?

Rant at myself for being a moron.

Manually tried moving some InnoDB stuff around on a local server and corrupted a load of databases, with no backups since June 28th. Spent all day yesterday trying to recover with no luck 😑

Lesson learned: need to backup databases more regularly. Any good tools?

Apple just changed their file system to apfs across the board, but no recent documentation on the release. It automaticalky changes from hfs+ when you install the macos 10.12.4 update. In early docs it says that time machine is not supported, but surely they cant kill everyones short term backups?

Maybe a stupid question but I can't seem to google the right words.

In Linux, how do I reference the same device (disk) through reboots? Like, there's no guarantee that /dev/sda1 will be the same device twice, right?

Like, if I have three or four disks and I want one for "backups", how do I choose that disk *every time*?

Client be like:
Hello,

could you please restore our database from today's backup?

At a first glance - nothing out of the ordinary. Daily backups are standard...

Until we get the backed up snapshot running.

MySQLDump is somehow... Stuck. It... Doesn't seem to be doing... Like, anything. For ages. Wtf.

So we check the database. Connect, change scheme and... The commandline tool gets stuck, too. Weird.

So a layer lower, we check the datadir and... ls... After also getting stuck for a bit, lists about 500k files O_o

Yea, dumping a database with roughly ~250k tables is not fun. No wonder it takes ages.

If any of you have been following my last few rants, you'll know I've been working on a project with a particularly difficult client, trying to meet wholly unrealistic deadlines with only one other developer.

The situation has reached the climax. The client had a call with our project manager and boss on Monday to discuss things. Despite them still not having paid a single bill since October, they've demanded the release date be moved to the 6th April. Apparently we'd agreed to release on this date, despite making no such promises, the (optimistic) deadline we were working towards has always been, since it was set about 2 weeks ago, the 16th April.

Apparently AWS migration won't take as long as we think it will, because the designers that do the CSS for this project say so, despite knowing nothing about the architecture of the requirements of the system once live (like if backups are required and what of).

The bottom line is that client is ending development with us the day after the project goes live to give it to their own in-house team. If they want us to work more after the date, they have to buy blocks of days.

To make things better, a large part of new functionality relies on an external API we can't even begin to do learning tests with, let alone integrate due to back-office errors on their end. They've had since Friday to give us our token, yet here we are.

Something tells me my holidays booked for for the first week of April are going by the wayside.

I am SOOO fucking sick of being asked if our website and gaming servers are going to be GDPR compliant. All these game owners in a panic changing everything they do just to conform to this law.

Fuck GDPR. In all reality COME AT ME BITCH. The EU wants to grow a pair of balls and act like the world internet police? Bring it the FUCK on. You can't even stop pirating in your own country, so how the FUCK are you going to regulate and enforce this law on HUNDREDS of THOUSANDS of servers, when your punk ass government can't even shutdown a single torrenting website.

Give me a fucking break, and shame on you pussies for allowing it. All you people running around scared acting like your private gaming servers are important. I give a shit less how much work you put into your server. I have put more work than most anyone else, but you don't see me trying to act self important as if my gaming server is some fortune 500 company.

Your server isn't important and neither are you. The government doesn't give a shit about your server so can we all just stop acting like this fucking matters. NO ONE FUCKING CARES ABOUT YOUR SERVER.

NO ONE is going to come and sue you for not complying. GDPR is for business, and anyone that wants to argue no look it says right here it applies to all is a fucking MORON. Do you idiots stop and think or do you just believe everything typed out on paper.

THEY CANT ENFORCE THIS ON EVERYONE. They don't have the resources. So use your fucking heads and stop being so fucking scared of a law that has no resources to stop you. THEY CAN"T DO ANYTHING. EU and whoever made their polices, I DARE them to try and touch my server, I WANT them to start something with me, just so I can show the rest of the world why the Internet is still the wild west and why they have no power over me.

You think pirate bay is the only one who knows how to hide their server? You think pirate bay is the only one who keeps backups of their server to be able to re release in an instant somewhere else in the world? Bitch get real this is the internet, a place where a 5 year old can buy hand grenades from the Red Silk Road, and you wanna talk to me about your privacy? Go fuck yourself.

It's not my problem some douche bag went onto a site that used his personal information in the wrong manner. So how about you do what everyone else does and browse ANONYMOUSLY. But no it would be to easy for governments to make their own citizens responsible. Instead they have to hold all of YOUR hands, because you people are to stupid to protect yourself.

Wake the fuck up world, and stop being a bunch of whining little brats who cry for the government to bubble wrap your world so you can live safer. Natural selection is long overdue for a lot of morons still breathing air.

People just can’t show up on time. I respect people’s time and I do not waste a minute of anyone s time. I do not like it. Time is a precious resource and we should all treat it as such. I expect the same from other people too.

There was a meeting supposed to happen a month or so back. Me and another guy. This meeting was basically giving him the resource utilization and some cost related details for the budget our team used up. They wanted a more granular report than the generic ones we submitted.

He scheduled it once and about ten minutes before he cancelled it. Fair enough.

A week later he sent an invite and the next day he cancelled it a few hours before the meeting. Fair enough I’m still not pissed.

A month or so since the first scheduled meeting, he scheduled a meeting and I turn up and he s not there and I wait for about half hour before I leave. Next day I get an email saying he s sorry. Now I’m pissed.

A couple weeks back he schedules a meeting and I turn up and I could see through the glass door and he s talking to someone in the room he s in. He signals to me to wait for five minutes. It takes about 20. Finally I go in and he s relieved so he can finally get the data from me. I tell him I waited for twenty minutes and I have to somewhere and asked him to reschedule the meeting. He asks when and I tell him a time where I know he won’t turn up. About 8 30 in the morning. He says yes.

Next day I show up he s not there and I wait for ten minutes and just he s walking in to the floor I exit the room and run to my place. He comes over and asks and I tell him I waited for sometime and I’m busy now and he mumbles and goes away.

By this time it is already a week or so after the deadline. The bigger boss from finance turns up and asks me why I haven’t given them the data I asked for. I tell him they schedule meetings and don’t turn up on time and my other work is getting affected productivity blah blah and he says okay make the report and give it to me. I tell him no I’m not wasting anymore time on this.

He goes to my boss creates a little ruckus my boss comes over and he goes wtf. I tell him what happened. He says it’s okay give them the report. I’m like I don’t know where I kept it must be somewhere I can’t waste anymore time on this. Guy from finance joins us at this point and angrily says he’ll find it himself and don’t need our help anymore.

The file is in my laptop I use for travel. Just a local copy. Zero fucking backups. And I just deleted it.

Fuck. You.

I've already ranted about this, the hdd randomly broke over night. I was (i shit you not) just about to set up backups for it this day.

Being relatively new to linux but confident with bash and cli and stuff.. reading "I/O Error" as output of nearly any command on a server rented somewhere 150 km away from me was like a punch in the face.:D

It wasn't directly bad, but it was kinda sad, I had a (now don't laugh - a man gotta chill from time to time) minecraft server running there with tons of mods and we were multiple 100s of hours into it already..
But not only that, my projects weren't on any git or anything anymore (local copies were gone, guess what gitlab i set up proudly i used..) and there was no recovering these little loved ones, together with my website.

It was a black day, my group i had to work with in university doubted me because for them i wasn't able to manage a git server properly and i hope it does not happen again..):

Have some downtime today, so since I lucked out and found some old backups (from before I used Git) of a project I was planning on revisiting, I decided to fire it up and see what I can do to get that going again.

And discovering just how much my coding style has changed since then...

[Code is in PHP, for reference]

* Virtually no documentation (whereas my current style is near-obsessive with PHPdoc blocks)
* Where comments exist, they only use // and are a full tab after the end of the line
* All assignment operators are dutifully aligned on tabs
* Have to update the entire codebase because it relies on deprecated `mysql_*` calls
* Have to flip all of the quotes throughout the codebase because I used double-quotes as my primary at the time instead of single quotes.
* Also relied on magic quotes for injecting variable content into strings
* Associative array practices varied; sometimes the names are encased in double quotes, but I just hit a block where it's all leaving it to the compiler to interpret unquoted string literals

And perhaps the most egregious so far...

* Any time we get database results back the process for cycling through them is to do `$count = mysql_num_rows($result);` (or $count2, etc.), then do a `for ($i = 0; $i < $count; $i++)` (again, or $j, $k, etc.), instead of just a simple `while ($data = $result->fetch_assoc())`

Anybody else sometimes do not like being a team lead?

I am the senior most person in my team (in fact took interviews of the rest of them). While I love the work and the team, I sometimes feel that I do not get to code that much. I am mostly assigning work, or helping others. Even if I get in the zone, there is always someone who needs some assistance or some meeting, which breaks the flow. Also the tension about non technical stuff like salary increments, giving feedback and assigning backups. I do love it mostly, but my ass is nearly always on fire. The team loves me though. I am still young, took this responsibility because the owner trusted me with this (he has no complaints). Thoughts ?

"RAID isn't backup"
"RAID is super not backup"
"Don't do RAID, do backups"

B-But I want to do backups... to a backup nas using RAID. No good?

I'm currently testing live and that includes trying sql injection, i have no backups. So if i forgot to escape string somewhere, I'm fucked. I like to live dangerous :D

Alos, i always test sql injection with "--; DROP ALL TABLES;" Casue... It's a bad idea..?

>stick holding important data goes bad ("Insert DVD" IT'S A FUCKING FLASH DRIVE)
>no backups as it's a flash drive so my PC never did an auto-backup despite being configured to do so
>might be recoverable
>"short pins 2 and 3 while running tool to reconfigure NAND and controller then recover data from quick format with TestDisk"
*looks at drive*

Well.
Fuck.

it's not even sharp enough to slit my wrists with

I have found a DVD with a Windows XP backup file with my late mother's photos on it. However, nothing seems to open the file. I, of course, did a lot of Google searches on how to do this and what it seems to come down to is a need to run the restore on an XP, Windows 7, or Server 2008 machine. There are several legacy ntbackup restore programs out there, and Microsoft seems to still have a page for downloading that, but it relies on the long-deprecated Removable Storage Service that no longer exists on Windows 10. So, a) Microsoft are a bunch of jerks for not supporting restores of legacy backups and b) does anyone have a successful method you've used to get a BKF file to restore? All the things I've tried say the BKF file is corrupted, but how to know for sure when Removable Storage Service isn't present?

Rant/Question
So I had this friend who has his own company, hosting is part of their business. He gave me a VM to host some websites, test stuff and whatnot. Free. I have quite some important stuff on there (personal, and a few demo sites for clients). No backups of my own, because they do nightly tape backups, and RAID accross physically seperated datacenters. Today, he just shut down my VM - for personal reasons (really personal, nothing to do with his company whatsoever, he hates me lately for something I can't influence). He doesn't say weather or when Ill get the data, and gave me no time to pull it. Can/should I go to the police? I just want my data back, but I don't know if I can do anything as it was free (no contract).
Switzerland, btw.

Me: "Ok, downloaded the (windows) iso, now I need to create a bootable usb"
* Pulls out 4Gb usb drive *
Father: "That iso won't fit on that one"
M: "Oh yeah right, do you have one for me"
F: "Sure"
* hands over 64Gb usb drive *
M: "thanks"
* Checks content *
* "Backups 15" directory and other old files *
"Guess I can use that"
* dds iso on usb drive *
dd: "done"
M: * Reboots *
Pc: "Nani the fuck is that supposed to be"
M: * searches online for solutions, tries out 2 different ones, one being provided by microsoft *
P: "lol no"
F: * comes in *
"Is it working?"
M: "Nah, windows a bitch"
F: "ok, can you copy me some files on the usb in the meantime?"
M: "Sure..."
{ How did he know that the drive is currently empty? }
* copies files onto stick *
F: * sees empty usb drive*
"Did you delete the contents on the usb?"
M: "Kinda, that happens when you create a bootable usb drive"
F: "..."
M: "Why do you ask, was the something important on the drive?"
F: "Idk, doesn't matter anyways now, since the contents are gone"

Btw, it still doesn't boot from the usb drive. Windows 10 iso is a bitch.

What's your favorite vps hoster?
I'm currently using scaleway and love it, but recently learned that they offer no protection against data loss.
So I'm looking for an alternative for a project in production that has automatic backups as well as unmetered bandwidth.

In the past, apps I've written have used a flat file backend. It's very fast, but obviously clunky to have a big structure of flat files for an app. It ran circles around framework-based RDBMS backends, as performance is concerned, but again, it was clunky. Managing backups and permissions on tens or hundreds of thousands of small files was no fun. Optimizing code for scaling was fun- generating indexes, making shortcuts -but something was still missing. Early in 2017 I discovered redis. A nosql backend that just stores variables and lives almost entirely in memory. Excellent modules and frameworks for every language. It was EXACTLY what I'd needed, even though I didn't know I did. I spent a good deal of time in 2017 converting apps from flat files to redis, and cackled with glee as they became the apps I wanted them to be. Earlier this week, I started building my first app that started with redis, instead of flat files, and I can't stop gushing to anyone who will listen. Redis for president!

Been asking the devops guy to schedule database backups for some of our side services for weeks! Just waiting for the day it all goes to hell and have no disaster recovery plan to fall back on.

I hate the elasticsearch backup api.

From beginning to end it's an painful experience.

I try to explain it, but I don't think I will be able to cover it all.

The core concept is:
- repository (storage for snapshots)
- snapshots (actual backup)

The first design flaw is that every backup in an repository is incremental. ES creates an incremental filesystem tree.

Some reasons why this is a bad idea:
- deletion of (older) backups is slow, as newer backups need to be checked for integrity
- you simply have to trust ES that it does the right thing (given the bugs it has... It seems like a very bad idea TM)
- you have no possibility of verification of snapshots

Workaround... Create many repositories as each new repository forces an full backup.........

The second thing: ES scales. Many nodes / es instances form a cluster.

Usually backup APIs incorporate these in their design. ES does not.

If an index spans 12 nodes and u use an network storage, yes: a maximum of 12 nodes will open an eg NFS connection and start backuping.

It might sound not so bad with 12 nodes and one index...

But it get's pretty bad with 100s of indexes and several dozen nodes...

And there is no real limiting in ES. You can plug a few holes, but all in all, when you don't plan carefully your backups, you'll get a pretty f*cked up network congestion.

So traffic shaping must be manually added. Yay...

The last thing is the API itself.

It's a... very fragile thing.

Especially in older ES releases, the documentation is like handing you a flex instead of toilet paper for a wipe.

Documentation != API != Reality.

Especially the fault handling left me more than once speechless...

Eg:
/_snapshot/storage/backup
gives you a state PARTIAL
/_snapshot/storage/backup/_status
gives you a state SUCCESS

Why? The first one is blocking and refers to the backup status itself. The second one shouldn't be blocking and refers to the backup operation.

And yes. The backup operation state is SUCCESS, while the backup state might be PARTIAL (hence no full backup was made, there were errors).

So we have now an additional API that we query that then wraps the API of elasticsearch. With all these shiny scary workarounds like polling, since some APIs are blocking which might lead to a gateway timeout...

Gateway timeout? Yes. Since some operations can run a LONG (multiple hours) time and you don't want to have a ton of open connections hogging resources... You let the loadbalancer kill it. Most operations simply run in ES in the background, while the connection was killed.

So much joy and fun, isn't it?

Now add the latest SMR scandal and a few faulty (as in SMR instead of CMD) hdds in a hundred terabyte ZFS pool and you'll get my frustration level.

PS: The cluster has several dozen terabyte and a lot od nodes. If you have good advice, you're welcome - but please think carefully about this fact.

I might have accidentially vaporized people sending me links with solutions that don't work on large scale TM.

Back in 2005, I had quite a few bits of music I was working on (just as a hobby). A lot of these had not been finished, but I'd sent excerpts in medium-quality MP3 format to a friend. I had an external backup drive - a regular hard drive in an USB enclosure. After a while, this drive started making unpleasant whining sounds so I sent it off for replacement.

During that time I made the foolish decision to try and plug a floppy drive in while the PC was powered on. Something touched the bottom of the hard drive and the power went off. I powered it back on again and heard a fizzing sound, there were some flashes from the hard drive and a burning smell. Yep, the disk was dead - and my backup drive was gone.

I'm still not entirely sure what happened, my best guess is that I had an exposed piece of wire from one of my hacky case mods (I had a thing for blue LEDs) which touched the circuitry of the hard drive. Almost every project, piece of software I'd created, every photo I'd taken, and most unfinished music I'd made up until that point - gone. I was pretty devastated about it. I only had a handful of things survived which I'd burned onto CD previously.

I managed to get some excerpts back from my friend, and re-created my favourite pieces of music based on those. I've moved on to other projects and write much better code now, so mostly I am no longer bothered. I do wish I could re-listen to some of the music I had made back then though.

Needless to say, I no longer fiddle around with the innards of my computers while they are on, store everything on mirrored drives and also ensure I always have a backup somewhere (and am working on remote backups and having several days of backups...)

I never want that to happen again

Yeah, sure.. I'll configure your DNS & mail server.. while I'm at it, I'll also optimise your db settings & configure backups..and don't forget LDAP & group policies!! Sure, no probs!! :/

Do you think you have to go to school for that? No no no. Some definition from a book won't help you, nor a person who haven't seen the real code for some time.

Sit and write. Anything! Still nothing? Printf("Hello World"); make conditions, think big, break the shit out of it and you'll learn along the way.

And do backups of git on remote. Two at least!

So until now the default env var User directory paths for Windows were all located under %userProfile%. To my surprise:

cd %userProfile%\Desktop

no longer works. Indeed, when you are even just signed in to OneDrive (which is the default, even if you disabled backups!), these retards decided it was a good idea to move all the standard library folders one level up to %userProfile%\OneDrive\ instead.

And the worst is that the locations continue to be swapped when you enable/ disable OneDrive.. WTF Microsoft

So.....
Cassandra vm had a crash yesterday...
2 nodes with rep factor 1. (FML)

One node wouldnt start... Eventually found out one of the commit replays had an exception (the one at the time of the crash).

Boss trying to push me towards a fix all this time which was:
"Let's delete the vm and have Cassandra running on one vm"

There are not we enough curses in the world.
🖕🖕🖕🖕🖕🖕🖕🖕🖕🖕🖕

P.S. there are no backups.

This one time last year a colleague found out that some data went missing and suggested to recover the data from a backup. When trying to create a new database instance in the Google Cloud Platform (if everything works it's amazing!) it failed.
Not knowing why this happened, I tried to revert that backup to the production database, after creating a backup using the GCP. Needless to say that failed as well, resulting in a corrupted database instance where I couldn't access the created backups anymore.
This all went at around 10pm and the only users of our product are currently in the same timezone and use it from around 7.30AM until 6PM so no one besides our team knew the server was down.
After a long night chatting Google's support team the database was successfully recovered and the only harm done was sleep depravation for me and a colleague.

Apparently there was a bug in the GCP. It was resolved in two hours and the last time a breaking bug was in that piece was more than seventy days earlier.

I did at least learn to create local backups as well, instead of relying on the tools of the same product...

Best: the moment I saw the corrupted database spin up again and not losing my job because of it.

That day when I destroyed the hard drive of my mother's PC when I was 12 years old. (The drive wasn't properly screwed in the case, and so after a few repositions of the case, it would one day start up sounding like a jet engine and after that scary sound: silence.

My mother took it relatively well, mostly because she saw how I was beating myself up over it. I was so mad at myself as I knew better! I knew how to create a backup strategy, I'm theory. I never really put in into practice though.

She had no backups of course. No way of regaining her data.

Now she does. And now she regularly calls me to initialize a backup of her current data on her external drive. (And every few months I sync her data over to my place on a data storage just in case she loses both her PC and external drive in a fire.)

Normal people : Save thier work regulary.
Me :

Oh no matter, notepadd++ saves opens files somewhere, i'm good with that.

There are important lohins, passwords, some temp stuf and work in progress for something, there are backups and information needed.

God, these people...

Little backstory. I'm making an training application and we have a MySQL database set up where some elements of the training are configured. This is so learning experts can easily change some aspects of the training without programmer's help.

Meanwhile, I'm also in the middle of a server migration, because our current server is running a lot of deprecated software and is in dire need of replacement.
This is going pretty slowly, though, because of other, high-priority, work that keeps being shoved my way.

Now, someone accidentally deletes a bunch of data from one of the schemas. No big deal in my book, the training is still in development and we have nightly backups of the database.
So I shoot a support ticket to the hosting provider and ask them to restore a specific schema, telling them to restore the image to some other machine and dump the tables in an MySQL file so I can restore it that way.
I also told them to get the backup of the OLD server, not the NEW one we're still migrating to.

About an hour later, I get a message that they dumped the schema's files in a Temp folder on the D drive. So I RDP to the server to check and... The files aren't there. Just before writing a response asking where the file is, I remembered the server I was migrating to and checked that server, and there were the files.

I had already migrated part of our databases and was testing compatibility before I moved to something else.
The hosting provider just dumped the files of the wrong server, despite me telling them exactly which server to use.

This is not the first time this hosting provider has let me down...
I'm really considering jumping to another if they keep doing this...

So I finally got a rant to tell, about me myself and I. Were working on my web host (personal gladly) and was trying to get ACME working without root access. I messed up and somehow got a folder named “~”, in a directory. I thought “well that folder is unnecessary” and I ran “rm -rf ~”. The moment I pressed “enter”, I realized what could’ve happened, and it did happen. My whole web-root gone. No backups. Just a big facepalm on my forehead ...
Take lesson, fix backups...

To have some Backups on your NAS in the Office is awesome... to have no Access from Remote Location if you need it...#$/$##!

!dev but a parable

I worked at a Walmart Photo Lab with a Fujifilm photo processor. I had a guy ask for his pictures but they weren’t printed, I could see his order but there was no “payload” ( think PO header with no PO lines). He said he ordered 600+ pictures off his SD card, then blew them away because they were ordered.

As I had no physical pictures, there was nothing I could do but say “sorry”. He was mad, but there was nothing I could do.

Moral of the story, verify backups before wiping the system.

When I was 7, I got my hands on an Amstrad CPC-464. This was my first exposure to code, copying examples out of the handbook. Shortly after that my school got their first IT suite, with thirty machines running Windows 98. I remember lunchtimes spent playing ZipZaps, a game that shamelessly capitalised on the first Fast and Furious films. I learned how to create macros in Office, and after getting a machine at home with Windows 3.1 I also learned some basic DOS. When I was 12 we got our first XP machine, which I spent hours on with MSN messenger and mucking around with scripts. That machine eventually succumbed to my brother repeatedly powering it on/off, something I still kind of hold against my mother to this day.

After going into care, I bought an old XP laptop from a friend, a machine that I used extensively. I mined my first bitcoin on that machine, bitcoin that could have made me a rich man today if I had only taken backups seriously.

My next machine came with Vista, which was upgraded to 7 shortly afterwards. This is when I got a bit more seriously into code, contributing to a game written in C++ (Armagetron Advanced, if you're interested). I also learned a great deal about automation using this machine, and when I got my second desktop machine at 18 (which at the time was still extremely out of date), I built my first working web server with IIS. I've been through four desktops since then, one of which just about survived a house fire.

Now I run a company of my own, doing development work at a lower cost for social enterprises, and developing a SaaS platform that will eventually make me a living all on its own. This year I hope to finally stop having to worry about debt, income, where I'm getting my next meal from and when I can finally be self sufficient, almost seven years since the care system spit me out after conveniently forgetting to tell me I could have stayed there until after Uni.

I am proud, though, of coming so far with no college or university degree. I'm by no means an expert, but I'd call myself proficient enough in a couple of languages to be capable of making a career of it.

So I was wondering... What's the worst burn you've seen from an unknown angry dev that you had to clean up?

A recent one of mine was a client in need with nuked hosting, nuked domain records, no backups, no access to any accounts and the business set to permanently closed on Google.

I thought that last one was a nice touch!

When you unknowingly/accidentally/inadvertently delete something someone was working on for a long time and they don’t tell you until several months later when all the backups are long rotated out. No way to fix it and nothing left to do but accept blame and sanctions and work to regain lost trust. Kill me now.

tldr: Deleted Win10, turned out for better
---------
**This happened a few months ago**
---------
So one day I decide I have had enough of Windows 10 and wanted to go back to Linux (A long time ago I had dual booted Ubuntu, but messed it up changing video drivers). So I create a Mint installation USB and get to work. I boot up the installation and delete the old partitions from the Ubuntu install. I install Mint and boot it up, everything works fine and dandy! I decide I want to get back on Windows to get back a few files that I wanted copies over. I turn off the computer and *try* to boot up Windows. I get an error message that I am UNABLE to boot. WHAT! After further checking, I realize that I had deleted the MBR partition of Windows. Pretty sure I could remake it if I tried hard enough, but I am starting to realize that it feels good to be totally MS independent! Now I am using all open-source software available to Linux and have no need for Windows. I do miss some of the games though...

PS: No files were lost due to backups. Save Lives, Do Backups!

Has anyone ever figured out why the fuck Android reboots seemingly out of nowhere at random times?

Is it kernel panics? Is it some shitty bug that has been plaguing this cursed OS since its inception? [sidenote: boy that was a mouthful of 's'es]
Or are we already at the mercy of the Big Brother, and someone at the controls likes to play random pranks on us when they're bored?

Thinking of switching to LineageOS more and more, but I'm kinda worried about resetting my phone since I've never done full backups/restores and I have no experience in the matter...

I work in a repair shop, I have 300+ sticks of RAM laying around, from DDR to DDR3 (DDR3 of 4+ GB and DDR4 are either good for replacement on new computers or are sold immediately so don't count them as laying).
Also I have 200+ hard drives, IDE, SATA, SAS, 2.5, 3.5, 20 GB to 1 TB. Lots of them.

What can I do with all this crap? Not enough matching CPU and MB to make a working system (unless I make a very obsolete one but there's no point in doing so). I also am using single hard drives for backups (only 500+ GB SATA-3 though) so I'm mostly concerned with all that old ram. What can I do with them??

lol

found an old config file on my external drive for all my torrent files. awyisss. my SSD died out of nowhere last year and I thought I lost all my torrent configs! I had hundreds of TV series and stuff and I kept track which ones I watched and didn't in the client. so when the SSD died I lost all my knowledge of my progress. but I found this config file just now and imported it. omg booyah. I think I got one show since this backup. godsend

decided to export settings again and it said I can set it on a schedule. go to the scheduler tab in the settings and I have no clue what's going on. nothing about exporting settings, it just has schedule configurations that seem to conflict. then I realized. the main client maybe has a schedule tab icon. bam am right. so in settings I turn on scheduler and then the main app gives me access to the scheduler tab and if I go there I can click "add" and then I can schedule regular config backups

bruh this UI is so jank. but it actually is impressive. because. while I have experience in designing websites, when I played around with making a GUI in rust, which would be native, I have absolutely no clue how to make an app on this tech. now I'm looking at this complex torrent client with its bazillion features in absolute awe.

*takes notes*
I can only aspire to be so genius as to allow you to turn on and remove tabs in the settings menu. now it makes sense why all the windows always had awkwardly sized panels. this genius man.

however did he come up with that?! ALL THESE NEW STANDARDS

honestly somehow it never occurred to me that native apps and web apps would have totally different ergonomics. I feel like I've found some kind of lost art from the ancient world. aaaaaaa