We build a backup infrastructure at work to make sure that clients can restore their files and databases themselves when something gets fucked up.

We also have step by step tutorial on how to do this.

Every fucking day we get requests to restore backups.

Mostly used reason is "I'm a technical so I won't understand it".
With all due respect, if you don't understand this and keep asking without even trying, please don't host with us.
Because, if you did as I asked and actually read through the entire article, you would.
In case you're wondering, anytime one of us asks what part they don't understand, that question is simply ignored and they pushing for us restoring it anyways continues.

Sometimes they get angry and want to talk to someone higher up or start complaining that they're paying loads of money already and that it would just take us a second anyways.

If you would read the fucking tutorial/manual instead of trying to eat out your mother's badly shaved pussy and hopefully choke on it while you're at it, you wouldn't come asking us for it.
If you genuinely don't understand this article, feel free to ask but also provide us with cocksucking feedback.

Why do you think you have the right anyways to ask us to do it for free? We maintain the backup infrastructure which definitely isn't cheap but we do it so that you, pubic sniffing weazel, can do this shit on your fucking own.

You're entitled to ask us for help but not for asking us to restore your bullshit for free every freaking time.

Tip: give your parents some condoms. Because that way they hopefully won't reproduce again, we don't need more of you in this universe.

One day I developed a simple website for a goldsmith who I already new for a year or so.

We discussed everything and agreed on a feature set, price and a deadline when it should be ready. Based on this we signed a contract and I started my work.

Unfortunately at the same time I lost most of my childhood friends. I moved to a new city and started to study computer science, which was awesome on the contrary.

This is where the horror began.

I was totally occupied by the studying, my partner, myself and by the shit of life.

It knocked on my door. The horror decided to pay me a visit.

"Had a look at your calendar recently? Just saying..."

Shit! The deadline came closer and closer everyday and the pile of work undone grew with it. At that point I had to do something. I don't know what it was or how I did it, but somehow I managed to finish the project just in time. I was totally not proud of it, but it featured what was required.

The day before I contacted my client, the horror knocked on my door again. He said:

"You really should have a look at your hard drive."
"Why? everything seems allright."
"Well, then look closer."
"Fuck."
"Right."

Well, there are backups at least, I thought to myself. I'll just recover the last state. That was an annoying thought, but nothing serious. That's just one or two days of w... - Wait, what? Where are my backups? What the actual fuck? Why is the zip file broken? Why doesn't the flash drive work anymore? FUUUCK!!

I was lost. It was a complete nightmare.

Each time my telephone rang the following days, my heart skipped a beat. Finally my client's name appeared on the display. I answered the call, my hands shaking.

"Hey there! I'm calling to discuss the website project with you."
"Well, about that..."
"Yeah, I know you put a huge amount of efford in it so I'm really sorry to say that I on the other hand can't effort the money. Actually I'd like to simply forget about this whole idea."

Seriously? What the fuck just happend? I suddenly noticed a sticky note infront of me reading:
"It was really fun to see you suffer, but I have to go! See ya
- The Horror"

"Hello, are you still there? Do you hear me?", yelled a voice through my phone.
"Uh, yeah. You know, that project was a lot of work and... but you know what? It was actually a pretty fun exercise and I'm doing well over here, so because it's you I'd agree."
I heared a reliefed sigh from the other end of the line.
"Really good! I owe you something! Bye!"

What. The. Fuck.

TL;DR: disaster averted!

Story time!

About a year ago, the company I work for merged with another that offered complementary services. As is always the case, both companies had different ways of doing things, and that was true for the keeping of the financial records and history.

As the other company had a much larger financial database, after the merger we moved all the data of both companies on their software.

The said software is closed source, and was deployed on premises on a small server.

Even tho it has a lot of restrictions and missing features, it gets the job done and was stable enough for years.

But here comes the fun part: last week there was a power outage. We had no failsafe, no UPS, no recent backups and of course both the OS and the working database from the server broke.

Everyone was in panic mode, as our whole company needs the software for day to day activity!

Now, don't ask me how, but today we managed to recover all the data, got a new server with 2 RAID HDDs for the working copy of the DB, another pair for backups, and another machine with another dual HDD setup for secondary backups!

We still need a new UPS and another off site backup storage, but for now...disaster averted!

Time for a beer! Or 20...

That is all :)

I just can't understand what will lead an so called Software Company, that provides for my local government by the way, to use an cloud sever (AWS ec2 instance) like it were an bare metal machine.

They have it working, non-stop, for over 4 years or so. Just one instance. Running MySQL, PostgreSQL, Apache, PHP and an f* Tomcat server with no less than 10 HUGE apps deployed. I just can't believe this instance is still up.

By the way, they don't do backups, most of the data is on the ephemeral storage, they use just one private key for every dev, no CI, no testing. Deployment are nightmares using scp to upload the .war...

But still, they are running several several apps for things like registering citizen complaints that comes in by hot lines. The system is incredibly slow as they use just hibernate without query optimizations to lookup and search things (n+1 query problems).

They didn't even bother to get a proper domain. They use an IP address and expose the port for tomcat directly. No reverse proxy here! (No ssl too)

I've been out of this company for two years now, it was my first work as a developer, but they needed help for an app that I worked on during my time there. I was really surprised to see that everything still the same. Even the old private key that they emailed me (?!?!?!?!) back then still worked. All the passwords still the same too.

I have some good rants from the time I was there, and about the general level of the developers in my region. But I'll leave them for later!

Is it just me or this whole shit is crazy af?

So ehm, tl;dr: KEEP DAILY BACKUPS. EVEN IF SOMEONE SAYS NOT TO.

7:48
Manager: Hey Tom, is the server down?
Me: Nah, should be ok, I just did some maintenance this sunday.
Manager: But I can't get [some work data from SQL server]
*Nervous giggle*

9:14: Some random off-site cunt they hired didn't read the notes that said "DO NOT REMOVE DATABASE [xyz]"
9:20-ish: Web don't even have the DB. And you said that we'll figure out what to do with backups later
*Suddenly manager starts to panic*
11:47: Found backup of the entire server on and old server that we had for spare parts, still running tho.
12:something: Everything back up and working.

Really glad I kept the old server running and doing daily backups. Saved our ass for the second time. And finally, new off-site backup is planned this week.

*Got a request for installing and configuring an online shop for a client*

Me: Do you have a web space already?
Client: No, I don't want to pay for it. (FYI: They only cost about 20€ a year)
Me: Okay, but free hosters are often slow and unprofessional. I really do not recommend using free hosting services.
Client: Doesn't matter, do it.
Me: *Working on the shop for several weeks, finally goes online*
One week later, client contacts me saying shop is offline. I realize the free hoster he used shut down their services (bankruptcy), resulting in the loss of about 90% of the work that I had done (no proper backups due to complexity)
Client: How can that even happen? You'll redo the shop, right?

♪~ ᕕ(ᐛ)ᕗ

Why does almost everyone act as if the world they live in is perfect, or is supposed to be perfect?

This is about approaching IT infrastructures, but goes way beyond IT, into daily lives.

Daniel Kahneman wrote about the "Econs" - a mythical creature that behaves according to rules and rational thoughts, that everybody is guided by, as opposed to Humans, who are irrational, intuitive and emotional.
My beef is with a wider perception, beyond economical analysis, profit, investment and so on.

Examples:
Organization A uses a 15 year old system that is crappy beyond description, but any recent attempt to replace it have failed. Josh thinks that this is a crappy organization, any problem lies within the replacement of that system, and all resources should be devoted to that. Josh lives in a perfect world - where shit can be replaced, where people don't have to live with crappy systems. Josh is stupid, unless he can replace that old system with something better. Don't be Josh. Adapt to the fucking reality, unless you have the power to change it.

Peter is a moron who downloads pirated software with cracks, at the office. He introduced a ransomware that encrypted the entire company NAS. Peter was fired obviously, but Sylvia, the systems administrator, got off easily because Peter the moron was the scapegoat. Sylvia truly believes that it's not her fault, that Peter happened to be a cosmic overgrown lobotomized amoeba. Sylvia is a fucking idiot, because she didn't do backups, restrict access, etc. Because she relied on all people being rational and smart, as people in her imaginary world would be.

Amit finished a project for his company, which is a nice modern website frontend. Tom, the manager says that the website doesn't work with Internet Explorer 8, and Amit is outraged that Tom would even ask this, quoting that IE8 is a dinosaur that should've been euthanized before even hatching. Amit doesn't give a shit about the fact that 20% of the revenue comes from customers that use IE8, what's more important to him is that in his perfect imaginary world everybody uses new hardware and software, and if someone doesn't - it's their fault and that's final. Amit is a fucking asshole. Don't be like Amit.

React to the REAL world, not what you WANT the world to be. Otherwise you're one of them.
The real world can be determined by looking at all the fuck ups and bad situations, admit that they happen, that they're real, that they will keep happening unless you do something that will make them impossible to happen or exist.

Acting as if these bad things don't exist, or that they won't exist because someone would or should change it, is retarded.

Does linux suck? Imho, Yes.
A lot of the people bash windows regarding automatic restarts, updates, bsod etc.
I may be unusually lucky, but the last bsod I saw was in 2014 because of a faulty synaptics driver.

I've really tried to use linux to see what the hype is all about. Quite frankly, it sucked. The first time it wiped out all my data, I realized the value of backups. Hence I do not have a single pic of my school now, thanks to complicated ubuntu mounting.
Next is driver support. When I plug in a device, I expect it to work. I don't want to spend a day googling for drivers.
Why the fuck would I want to use a black terminal which gives me a headache. Am I in 1980? Which sadistic asshole designed vim ?
I have seen linux developers who claim to be linux experts and love linux. They take so long to do simple shit. For god's sake don't tell me there are GUI versions of linux available. I'd rather work in windows 95.
Why in the world would anyone want to use ls to see the contents of a directory! It is seriously so fucking unproductive.
I can't just download a software, click next a couple of times, and be done. No no no. I've to do sudo apt get update. Then try to find the fucking package. And if all goes well, there's always the dependency issue which is going to bite me in the ass. If google and stackoverflow go down, most linux devs will die a cruel death.

Fuck you linux.
I'm not saying windows 10 is the best, but at least I don't have to crawl through shit to use it. If you don't like automatic updates, disable it you moron. It is easier than renaming a damn file in linux.

Joined a call during a potentially important life event. Work laptop password expired this morning, personal laptop didn't connect to vpn, desktop not connecting to vpn. No one knew what's wrong on the call (just that it's not working).

Not a good call.

I'll admit - I come from a WordPress background of almost 9 years in the making. I guess I can justify it because of all of the sites I created using it, it was the best that it could be on WP. Fast, efficient, custom - none of that off-the-shelf themeforest crap. I created everything custom. I actually knew what was going on behind the scenes of WP.

And then a buddy of mine and I had an idea for a new company/software project. I was smart enough to know that WP was not the foundation for this, so I did some NodeJS/Express tutorials. Started learning React, and really getting into the Javascript world.

And now I'm wondering WHY IN THE ABSOLUTE FUCK I ever bothered trying to become an expert in WP. It's the largest use of PHP in the fucking world and it doesn't even have native composer support. And by the time you actually get your project set up using composer you have to add a fucking mirror of the wordpress.org plugin repo to get anything to work. It's 2018 and you'd think that WP and composer would have all of this shit figured out by now.

And don't get me started on git - as soon as you have more than 1 person working on a WP site, I hope you have hourly backups of your DB because someones work will get overwritten. So you all either need to work on the same staging area of work around each other by pushing/pulling the DB and schedule your workflows.

I guess WP CLI and the REST API are a step in the right direction, but the foundation of everything is just so fucked up.

I don't feel like I've wasted my web dev career, but I definitely wish I had started down this path a lot earlier. I guess you don't know what you don't know. Thanks for reading!

"let's use git for this game jam"

Wait! Don't go! I love git and use it on every project I work on! You'll have to hear me out here.

This was 4 years ago, at my first Global Game Jam. Every jam and game I'd worked on up to that point, I was the only Dev; no need for git, as backups were more than enough. I joined a group with high hopes for the game jam, with three coders and a proper art team.

The entire jam was "1 step forward 2 steps back", as git somehow constantly overwrote code as fast as we could write it.

By the end of the jam we barely had anything to show for our hard work. The takeaway isn't even about git. It's simply to never work with other people. Git is a great protocol but it can't stop people from accidentally fucking other people over. Every jam since, I've worked on my own and had a far better time of it.

As a Dev in college working for teams in college, I don't really have a need to use git, since most of 'self proclaimed prestigious programmer child prodigies' I work with have no idea what it is; but I use it anyways as good programming practice and ease of backups.

So I tried using a GUI client after months of the git bash, and even though I looked up a few tutorials (was embarrassed the whole fucking time). I ended up adding, committing and pushing via bash.

Can anyone explain me how is the GUI client helpful in large projects and stuff?

A client hired someone to work on a new feature while we were working on something else. The new guy makes huge commits that we don't have the time to read, really.
I merged and deployed my work only to find that the whole database was wiped. Apparently, the new guy pushed some code that reset the database.
I Spent the rest of the day looking through backups trying to restore the database.

I am SOOO fucking sick of being asked if our website and gaming servers are going to be GDPR compliant. All these game owners in a panic changing everything they do just to conform to this law.

Fuck GDPR. In all reality COME AT ME BITCH. The EU wants to grow a pair of balls and act like the world internet police? Bring it the FUCK on. You can't even stop pirating in your own country, so how the FUCK are you going to regulate and enforce this law on HUNDREDS of THOUSANDS of servers, when your punk ass government can't even shutdown a single torrenting website.

Give me a fucking break, and shame on you pussies for allowing it. All you people running around scared acting like your private gaming servers are important. I give a shit less how much work you put into your server. I have put more work than most anyone else, but you don't see me trying to act self important as if my gaming server is some fortune 500 company.

Your server isn't important and neither are you. The government doesn't give a shit about your server so can we all just stop acting like this fucking matters. NO ONE FUCKING CARES ABOUT YOUR SERVER.

NO ONE is going to come and sue you for not complying. GDPR is for business, and anyone that wants to argue no look it says right here it applies to all is a fucking MORON. Do you idiots stop and think or do you just believe everything typed out on paper.

THEY CANT ENFORCE THIS ON EVERYONE. They don't have the resources. So use your fucking heads and stop being so fucking scared of a law that has no resources to stop you. THEY CAN"T DO ANYTHING. EU and whoever made their polices, I DARE them to try and touch my server, I WANT them to start something with me, just so I can show the rest of the world why the Internet is still the wild west and why they have no power over me.

You think pirate bay is the only one who knows how to hide their server? You think pirate bay is the only one who keeps backups of their server to be able to re release in an instant somewhere else in the world? Bitch get real this is the internet, a place where a 5 year old can buy hand grenades from the Red Silk Road, and you wanna talk to me about your privacy? Go fuck yourself.

It's not my problem some douche bag went onto a site that used his personal information in the wrong manner. So how about you do what everyone else does and browse ANONYMOUSLY. But no it would be to easy for governments to make their own citizens responsible. Instead they have to hold all of YOUR hands, because you people are to stupid to protect yourself.

Wake the fuck up world, and stop being a bunch of whining little brats who cry for the government to bubble wrap your world so you can live safer. Natural selection is long overdue for a lot of morons still breathing air.

I've already ranted about this, the hdd randomly broke over night. I was (i shit you not) just about to set up backups for it this day.

Being relatively new to linux but confident with bash and cli and stuff.. reading "I/O Error" as output of nearly any command on a server rented somewhere 150 km away from me was like a punch in the face.:D

It wasn't directly bad, but it was kinda sad, I had a (now don't laugh - a man gotta chill from time to time) minecraft server running there with tons of mods and we were multiple 100s of hours into it already..
But not only that, my projects weren't on any git or anything anymore (local copies were gone, guess what gitlab i set up proudly i used..) and there was no recovering these little loved ones, together with my website.

It was a black day, my group i had to work with in university doubted me because for them i wasn't able to manage a git server properly and i hope it does not happen again..):

Oh my God. It's my last night in Italy and I can't wait for it to end - I'm sleeping in Naples.

I only had to walk 80m from my car to my bnb, but I was so afraid for my dear life I think I have some poopoo in my pantalones.

If I don't survive this night, these of you who know where my backups are, please, continue and finish my work.

What a horrible way to end a vacation.

I hate the elasticsearch backup api.

From beginning to end it's an painful experience.

I try to explain it, but I don't think I will be able to cover it all.

The core concept is:
- repository (storage for snapshots)
- snapshots (actual backup)

The first design flaw is that every backup in an repository is incremental. ES creates an incremental filesystem tree.

Some reasons why this is a bad idea:
- deletion of (older) backups is slow, as newer backups need to be checked for integrity
- you simply have to trust ES that it does the right thing (given the bugs it has... It seems like a very bad idea TM)
- you have no possibility of verification of snapshots

Workaround... Create many repositories as each new repository forces an full backup.........

The second thing: ES scales. Many nodes / es instances form a cluster.

Usually backup APIs incorporate these in their design. ES does not.

If an index spans 12 nodes and u use an network storage, yes: a maximum of 12 nodes will open an eg NFS connection and start backuping.

It might sound not so bad with 12 nodes and one index...

But it get's pretty bad with 100s of indexes and several dozen nodes...

And there is no real limiting in ES. You can plug a few holes, but all in all, when you don't plan carefully your backups, you'll get a pretty f*cked up network congestion.

So traffic shaping must be manually added. Yay...

The last thing is the API itself.

It's a... very fragile thing.

Especially in older ES releases, the documentation is like handing you a flex instead of toilet paper for a wipe.

Documentation != API != Reality.

Especially the fault handling left me more than once speechless...

Eg:
/_snapshot/storage/backup
gives you a state PARTIAL
/_snapshot/storage/backup/_status
gives you a state SUCCESS

Why? The first one is blocking and refers to the backup status itself. The second one shouldn't be blocking and refers to the backup operation.

And yes. The backup operation state is SUCCESS, while the backup state might be PARTIAL (hence no full backup was made, there were errors).

So we have now an additional API that we query that then wraps the API of elasticsearch. With all these shiny scary workarounds like polling, since some APIs are blocking which might lead to a gateway timeout...

Gateway timeout? Yes. Since some operations can run a LONG (multiple hours) time and you don't want to have a ton of open connections hogging resources... You let the loadbalancer kill it. Most operations simply run in ES in the background, while the connection was killed.

So much joy and fun, isn't it?

Now add the latest SMR scandal and a few faulty (as in SMR instead of CMD) hdds in a hundred terabyte ZFS pool and you'll get my frustration level.

PS: The cluster has several dozen terabyte and a lot od nodes. If you have good advice, you're welcome - but please think carefully about this fact.

I might have accidentially vaporized people sending me links with solutions that don't work on large scale TM.

This is a continuation of my previous rant...
I did it! I overcame my anxiety to work on the dev database (don't worry, I made backups). At first I got really anxious and almost panicked at the thought of possibly messing up and procrastinated a few minutes. After the intial anxiety has passed I just did it and it worked. I didn't destroy anything nor did I made any mistakes. One step closer to becoming more confident when working with company assets

I work in a repair shop, I have 300+ sticks of RAM laying around, from DDR to DDR3 (DDR3 of 4+ GB and DDR4 are either good for replacement on new computers or are sold immediately so don't count them as laying).
Also I have 200+ hard drives, IDE, SATA, SAS, 2.5, 3.5, 20 GB to 1 TB. Lots of them.

What can I do with all this crap? Not enough matching CPU and MB to make a working system (unless I make a very obsolete one but there's no point in doing so). I also am using single hard drives for backups (only 500+ GB SATA-3 though) so I'm mostly concerned with all that old ram. What can I do with them??