More and more, I am getting frustrated/depressed from the attitude of our customers who complain, moan and get angry about issues in their infrastructure, while at the same time, refusing to pay more so the issues could be mitigated.

Like, a client's angry with us today for having one of their non-production-critical databases inaccessible for... Hmm... About 8 hours now (So a whole workday).

Like... I get it, some of your employees couldn't work with it offline, but like... What the hell do we do? You keep data from as far back as several years ago in there, without partitioning, without exports, in a mix of innodb and myisam, so when the DB crashes, and its replication has to be reset from zero, reimporting all the data takes hours upon hours, and importing .sql files just takes time.

Or another client who got angry when their app fell out of the internet, cuz one of their myisam-based log tables crashed, and had to be repaired, with data spanning several years back, meaning it took hours to fix...

The more I work with these "basic" and "simple" infrastructure designs that is *not* redundant, or HA, the more I wonder -- How do the big names out there do it? How do you design systems with fault tolerance so a single DB table crash doesn't lead to the whole app getting inaccessible?

We have... One, exactly one, client, who uses MariaDB with Gallera, and that cluster is *amazing*, it just keeps chugging along, without a care in the world. But it cost them quite a lot, as they had to buy 3 DB servers, instead of 1...

  • 2
    So you're telling me that I can't host my primary, secondary and dr databases on the same server?

    πŸ‡ΊπŸ‡Έ This is America sir! 😑

    If it's not obvious, I'm being sarcastic.
  • 2
    But but but but, everything except from a basic bitch monolith implemention on the same server would be over-engineering.........

    Welcome to this age, where thinking about a propper design before implementing or/and setting anything up is considered over-engineering.

    Honestly its sad we have came to the point where we dont really think about how it could scale and other problems before going full code monkies or installing random crap on the servers and lanuching it to the production.
  • 2
    @sariel Whats next, you are telling me its a dumb idea only taking the backup on the same databases and only save them on the server as well.
Add Comment