Just got word that my old company terminated an ec2 instance and ended up deleting all the volumes attached to the instance.

They were using attached volumes as backups 不不不不

Who does that?

A lot of customers are very unhappy campers today.

They are all ecommerce sites, some of them with revenue in the millions.

No sites, no backups, no income.

Over a hundred companies don't have sites this am.

Always research who you entrust your own business to!

And always, always, have your own backup solution. Don't trust other companies they are doing the right thing. Because most of time they aren't.

I'm sure there will be lawsuits.

However, there response is even more comical.

Dear customer who entrusted there very own livelihood to us:

We have discovered a regression in our systems that failed to enable termination protection. We have fixed this regression and have found the issue in other instances that have also been fixed.

All backups have been lost. If you have your own backups please give them to us and we will rebuild your site. [However all recent orders are lost, in fact all orders, products, images, customizations, from the most probably dated backups, if any, are lost. Emphasis mine]

Nothing about we are improving our backup method to make it redundant. Just "we switched on termination protection" wtf? 20 dollars a month at rsync.net would have saved a company. Instead, they entrust backups to an attached volume? How is that even called a backup! It's attached!

  • 26
    If you are on AWS anyway, why not use AWS backup?

    Devops often surprises me in their homebrewed solutions — although at most companies the CEO ends up doing server management because he doesn't trust anyone else and he did a JS course online once.
  • 12
    Sweet mother of god, not even snapshot backup on something not attached like.. I dunno S3
  • 4

    You speak truth!
  • 2
    Oh Man!
  • 10

    The irony about this, is the tech that supposedly made the mistake was the CTO 不不不不
  • 7
    The company was a stepping stone of me getting back in the industry after failing at my first jr. dev position. They did take a chance on me, and I really exceeded there expectations. But the relationship degraded severely after I resigned.

    So it's bittersweet hearing this news, on the one hand I obviously don't want them to fail, they do have good people that work there, the other hand, karma is a bitch.
  • 5
    @hash-table CTO? That's what I said, the manager who spent a weekend on codecademy.
  • 7
    Wow. Just wow.
  • 4

    yeah, given ebs volume snapshots are extremely cheap and you can have as many as you want, why anyone wouldn't at least take a daily snapshot is beyond me. But in the end AWS Backup is definitely the solution they should have been using in first place. Given the nature of their business.

    I don't think it will destroy them entirely as they do have sufficient revenue to weather the storm, (unless the lawsuits, if any, sink them) but you never know.

    However, I can tell you Senior Management there is overly cocky, judgemental, and complete assholes, that care only about themselves and their personal image. (the owner has even in past threatened to sue customers over bad reviews, has bought off companies for silence, and fired employees for telling truth)

    It is all catching up to them.
  • 3
    That honestly is fucked up.
  • 5
    i'm the kinda guy who doesn't do backups, if something is important to me, I copy it to some online storage or to an external HDD

    but not doing backups of SOMETHING THAT MAKES YOUR LIVING... holy shit
  • 3
    Wtf, even I have backups in more than one continent... And we really would survive losing it
Your Job Suck?
Get a Better Job
Add Comment