TL;DR: disaster averted!

Story time!

About a year ago, the company I work for merged with another that offered complementary services. As is always the case, both companies had different ways of doing things, and that was true for the keeping of the financial records and history.

As the other company had a much larger financial database, after the merger we moved all the data of both companies on their software.

The said software is closed source, and was deployed on premises on a small server.

Even tho it has a lot of restrictions and missing features, it gets the job done and was stable enough for years.

But here comes the fun part: last week there was a power outage. We had no failsafe, no UPS, no recent backups and of course both the OS and the working database from the server broke.

Everyone was in panic mode, as our whole company needs the software for day to day activity!

Now, don't ask me how, but today we managed to recover all the data, got a new server with 2 RAID HDDs for the working copy of the DB, another pair for backups, and another machine with another dual HDD setup for secondary backups!

We still need a new UPS and another off site backup storage, but for now...disaster averted!

Time for a beer! Or 20...

That is all :)

So I am running this crypto project that has dynamically generated private keys for a wallet stored in a Redis database. Nowhere else. The keys are generated on the fly.

At the moment of the happening the wallet had over 3.000 USD on it. I am testing new code locally, supposedly on a local Redis DB. Of a sudden, my code wipes the crypto keys and it turns out that I was connected to the live instance. 😱 Better don't ask me how.

Shock of my life. You know, when you turn pale and dark in your eyes, blood stops in your veins and you just want to die? Worst-case scenario that could have happened. All that money lost in crypto space.

Turns out, my good Redis hosting company kept backups for the past 7 days. Keys restored. Happiest moment of my life.

So ehm, tl;dr: KEEP DAILY BACKUPS. EVEN IF SOMEONE SAYS NOT TO.

7:48
Manager: Hey Tom, is the server down?
Me: Nah, should be ok, I just did some maintenance this sunday.
Manager: But I can't get [some work data from SQL server]
*Nervous giggle*

9:14: Some random off-site cunt they hired didn't read the notes that said "DO NOT REMOVE DATABASE [xyz]"
9:20-ish: Web don't even have the DB. And you said that we'll figure out what to do with backups later
*Suddenly manager starts to panic*
11:47: Found backup of the entire server on and old server that we had for spare parts, still running tho.
12:something: Everything back up and working.

Really glad I kept the old server running and doing daily backups. Saved our ass for the second time. And finally, new off-site backup is planned this week.

The website for our biggest client went down and the server went haywire. Though for this client we don’t provide any infrastructure, so we called their it partner to start figuring this out.
They started blaming us, asking is if we had upgraded the website or changed any PHP settings, which all were a firm no from us. So they told us they had competent people working on the matter.

TL;DR their people isn’t competent and I ended up fixing the issue.

Hours go by, nothing happens, client calls us and we call the it partner, nothing, they don’t understand anything. Told us they can’t find any logs etc.

So we setup a conference call with our CXO, me, another dev and a few people from the it partner.
At this point I’m just asking them if they’ve looked at this and this, no good answer, I fetch a long ethernet cable from my desk, pull it to the CXO’s office and hook up my laptop to start looking into things myself.

IT partner still can’t find anything wrong. I tail the httpd error log and see thousands upon thousands of warning messages about mysql being loaded twice, but that’s not the issue here.
Check top and see there’s 257 instances of httpd, whereas 256 is spawned by httpd, mysql is using 600% cpu and whenever I try to connect to mysql through cli it throws me a too many connections error.

I heard the IT partner talking about a ddos attack, so I asked them to pull it off the public network and only give us access through our vpn. They do that, reboot server, same problems.

Finally we get the it partner to rollback the vm to earlier last night. Everything works great, 30 min later, it crashes again. At this point I’m getting tired and frustrated, this isn’t my job, I thought they had competent people working on this.

I noticed that the db had a few corrupted tables, and ask the it partner to get a dba to look at it. No prevail.

5’o’clock is here, we decide to give the vm rollback another try, but first we go home, get some dinner and resume at 6pm. I had told them I wanted to be in on this call, and said let me try this time.

They spend ages doing the rollback, and then for some reason they have to reconfigure the network and shit. Once it booted, I told their tech to stop mysqld and httpd immediately and prevent it from start at boot.

I can now look at the logs that is leading to this issue. I noticed our debug flag was on and had generated a 30gb log file. Tail it and see it’s what I’d expect, warmings and warnings, And all other logs for mysql and apache is huge, so the drive is full. Just gotta delete it.

I quietly start apache and mysql, see the website is working fine, shut it down and just take a copy of the var/lib/mysql directory and etc directory just go have backups.

Starting to connect a few dots, but I wasn’t exactly sure if it was right. Had the full drive caused mysql to corrupt itself? Only one way to find out. Start apache and mysql back up, and just wait and see. Meanwhile I fixed that mysql being loaded twice. Some genius had put load mysql.so at the top and bottom of php ini.

While waiting on the server to crash again, I’m talking to the it support guy, who told me they haven’t updated anything on the server except security patches now and then, and they didn’t have anyone familiar with this setup. No shit, it’s running php 5.3 -.-

Website up and running 1.5 later, mission accomplished.

TLDR - you shouldn't expect common sense from idiots who have access to databases.

I joined a startup recently. I know startups are not known for their stable architecture, but this was next level stuff.
There is one prod mongodb server.
The db has 300 collections.
200 of those 300 collections are backups/test collections.
25 collections are used to store LOGS!! They decided to store millions of logs in a nosql db because setting up a mysql server requires effort, why do that when you've already set up mongodb. Lol 😂
Each field is indexed separately in the log.
1 collection is of 2 tb and has more than 1 billion records.
Out of the 1 billion records, 1 million records are required, the rest are obsolete. Each field has an index. Apparently the asshole DBA never knew there's something called capped collection or partial indexes.

Trying to get approval to clean up the db since 3 months, but fucking bureaucracy. Extremely high server costs plus every week the db goes down since some idiot runs a query on this mammoth collection. There's one single set of credentials for everything. Everyone from applications to interns use the same creds.
And the asshole DBA left, making me in charge of handling this shit now. I am trying to fix this but am stuck to get approval from business management. Devs like these make me feel sad that they have zero respect for their work and inability to listen to people trying to improve the system.

Going to leave this place really soon. No point in working somewhere where you are expected to show up for 8 hours, irrespective of whether you even switch on your laptop.
Wish me luck folks.

I woke up early and thaught I could finish some database work today.
Beeing tipsy I deleted two production tables and shit my pants.

Luckily I do backups since the last retarded db-deletion.

And this, ladies and gentlemen, is why you need properly tested backups!

TL;DR: user blocked on old gitlab instance cascade deleted all projects the user was set as owner.

So, at my customer, collegue "j" reviews gitlab users and groups, notices an user who left the organisation
"j" : ill block this user
> "j" blocks user
> minutes pass away, working, minding our own business
> a wild team devops leader "k" appears
k: where are all the git projects?
> waitwut?.jpg
> k: yeah all git projects where user was owner of, are deleted
> j.feeling.despair() ; me.feeling.despair();
> checks logs on server, notices it cascade deletes all projects to that user
> lmgt log line
> is a bugreport reported 3(!) years ago
> gitlab hasnt been updated since 3 years
> gitlab system owner is not present, backup contact doesnt know shit about it
> i investigate further, no daily backup cron tasks, no backup has been made whatsoever.
> only 'backups' are on file system level, trying to restore those
> gitlab requires restore of postgres db
> backup does not contain postgres since the backup product does not support that (wtf???)
> fubar.scene
> filesystem restore finished...
> backup product did not back up all files from git tree, like none of refs were stored since the product cannot handle such filenames .. Git repo's completely broken

Fuck my life

Forgot to secure my mongo db instance, found half the data gone, and a new db holding me at ransom , learn_how_to_recover_your_data , good thing offline backups are a thing.

Recovered in no time, never will I ever repeat this mistake

I'll admit - I come from a WordPress background of almost 9 years in the making. I guess I can justify it because of all of the sites I created using it, it was the best that it could be on WP. Fast, efficient, custom - none of that off-the-shelf themeforest crap. I created everything custom. I actually knew what was going on behind the scenes of WP.

And then a buddy of mine and I had an idea for a new company/software project. I was smart enough to know that WP was not the foundation for this, so I did some NodeJS/Express tutorials. Started learning React, and really getting into the Javascript world.

And now I'm wondering WHY IN THE ABSOLUTE FUCK I ever bothered trying to become an expert in WP. It's the largest use of PHP in the fucking world and it doesn't even have native composer support. And by the time you actually get your project set up using composer you have to add a fucking mirror of the wordpress.org plugin repo to get anything to work. It's 2018 and you'd think that WP and composer would have all of this shit figured out by now.

And don't get me started on git - as soon as you have more than 1 person working on a WP site, I hope you have hourly backups of your DB because someones work will get overwritten. So you all either need to work on the same staging area of work around each other by pushing/pulling the DB and schedule your workflows.

I guess WP CLI and the REST API are a step in the right direction, but the foundation of everything is just so fucked up.

I don't feel like I've wasted my web dev career, but I definitely wish I had started down this path a lot earlier. I guess you don't know what you don't know. Thanks for reading!

Several actually

- root DB access on production
- git commit -am "Stuff" (message)
- Not doing backups
- Using personal licensed software for company work
- 1 commit per day or weekend
- Taking work to home

Sorry to keep whining about my stupid fucking job, but y'all, I think I'm nearing my limit.

There's some good...I am pretty much free to resolve issues any way I want to, as the only other person in the company who "codes" only knows one old ass language that doesn't apply to 90% of the rest of the tech stack at all, and some SQL - all of that to say, we may disagree, but ultimately, these matters are always deferred to me at the end of the day, insofar as the actual implementation goes (which is to say I am not micromanaged). At least as far as non-visuals are concerned, because those of course, are the most important things. Button colors and shit, woo hoo**. That's what we should focus on as we're bringing in potentially millions of dollars per month - the god damn button color and collapsible accordions based on data type over the shit ass DB performance bottleneck, the lack of redundancy or backups (aside from the one I made soon after I started -- literally saved everyone today because of that. My thanks? None, and more bullshit tasks) or the 300GB+ spaghetti code nightmare that is the literal circulatory system of the FUCKING COMPANY. Hundreds of people depend on it for their livelihoods, and those of their families, but fuck me in the face, right? I'm just a god damn nerd who has worked for the federal government, a handful of fortune 500's, a couple of fortune 100's, some startups, etc. But the fuck do I know about the lifecycle of companies?

I could continue ranting, but what's the point? I've got a nice little adage that I've started to live by, and y'all might appreciate it: "If everything is a priority/is important, nothing is". These folks just don't fucking get it. I'm torn because, on the one hand, they waste my time and kinda underpay me, in addition to forcing me to be onsite for 50 hours a week. They don't listen to me, couldn't give a flying shit about my experientially based opinions. I'm just a fucking chimp with a typewriter, there to take commands like a fucking waiter. But there's a lot of job security, assuming I don't fucking snap one day, and the job market for devs (I'm sure I don't need to tell you) is hostile atm. I'm also drinking far more than usual, and I really need to do something about that. It's only wednesday - I think...not 100% on that truth be told, and I logged my fourth trip to the liquor store this week already.

**Dear backenders - don't ever learn front end, or if you do, just lie about it to avoid being designated full stack. It's not worth it.

A script to run overnight SQL backups, a colleague updated the DB blindly, corrupted most of the data, when checking the backups there was a typo in the script so they were all 0kb in size!

Luckily I had taken a manual one the day before...

Allrighty, so we have a huge migration upcoming. The planning started early this spring. We've split the whole process into separate tasks and estimated each of them. Also marked all the tasks client should take care of itself so save funds and time. All-in-all the whole thing estimated like 4 months if we did it [single dev, tremendous amounts of communication with various parties, buy and prepare the infra, adapt app to the changes, testing, monitoring, etc.] and like a month if client did the tasks we shouldn't be doing. The funding for migration is time-bound and can only be used before December. Cool! We got notified that by the end of April we should be good to go! Plenty of time to do things right!

April comes. Silence. Mid-april we resch out to the client. Since there's plenty of time left migration is getting lower priority to other tasks. Well allright, sort of makes sense. We should migrate mid-July. Cool!

July comes. Client replies that everyone's on vacation now. Gotta wait for August - will do the quicker version of migration to make it on time. Well allright....

August comes. Everyone's vusy with whatever they've postponed during summer. Hopefully we'll start migration in September. Mhm...

September comes. We're invited to a meeting by project funders to explain tasks' breakdown, justify the time needed to make the migration. We're being blamed for surreal estimations and poor organization of tasks as nothing's happened yet... [they were the ones who always were postponing things....]. Moreover, they can only spare 20% of infra resources required for data alone anf they want us to make that enough for all environments, all components, all backups, all databases,... You get the pic.
The leader of the meeting semi silently mumbled to other participants 'Well then I'm afrsid we can't make a full migration in time.. Only partial. That's very unfortunate, very. That's why we should not have incopetent vendors [*glancing at us*]'

somehow we agreed we'll get the resources mid-November and we should be thankful for him bcz he'll have to pull some strings for... us..

I left the meeting with my fists squeezed so hard! But it's okay, we got smth useful: resources and start date. Although it leaves us with less than a month to do smth requiring a month for a sunny-day scenario. Nvm, still doable.

Last week we get an email that resources will be available at the beginning of December [after deadline] and we should start a full migration no sooner than Nov 12. Which leaves us with 50% of our estimated fucking optimistic scenario time and not enough resources to even move a single db.

Fuck I hate politics in dev... Is it wrong for me to want to tie them to a pole, set them on a veeery slow fire and take a piss on them while they're screaming their shitty lungs out? I'd enjoy the view and the scream. I know I would. And while enjoying I might be tempted to take a burning 20cm diameter wooden stick and shove it up their assholes. Repeatedly. Round-robin. Promissing them I'll take it out in 5 seconds and pulling it out after 2 minutes.

Can I?

Mine: not listening to early warning sings that shit won't work. As a result we implemented the wrong technology and had to roll back after a month in production. Lost some customers data, though we had backups, so it was just a delta of a day or so. Still, not pretty, had a rough night writing scripts at 3am to check data validity after.

I will throw in another one from a colleague of mine. He was running some database migrations and ended up pointing the DB migration tool to production by accident. Oops. That one required a restore as well, if I remember correctly.

I was working for a project with one of the project managers. Despite several discussions, he was not ready to have provisioned for procurement of couple of extra drives for database backups. Also because it's always how they worked, developers were allowed to make changes to the production databases directly.

Since I knew it was going to be burning some day, despite his negligence, I ran a script to take full database backups every night, compress, and remove old backups all to do in the drives we had on server. Sat it automated using scheduler.

One day it happened that one of the junior developers deleted one major table taking whole production down. Next thing you know everyone went crazy. Since I felt bad for the managers and users, I was able to restore database using backup from last night.

You know who jumped in first before senior management to take credit of all this and got some nice kudos..that project manager. Also, you know who got burned..it would not be a rant if I did not got schooled for not following on the wisdom of project manager.

Anyways, we are still not taking database backups (as per project manager)

How do you restore partial data from a mysql backup? Don't worry, nothing is wrong, I'm just thinking about how would I restore something if shit hits the fan.

Our current strategy for database backups is to just run mysqldump during the night, using a cronjob (feel freue to suggest a better way ;))

1) Restoring the full db: just read that sql file into the mysql command.

2) Restoring just one (or some) tables: open the file in an IDE, just select the lines you're after, copy them to a new file, read that one (possible issues: let's say we have a table B to which entries of table A are related and we just want to restore table A. We can't nuke table B too, as also table C is refering to it, so we have to do some orphant removal in B afterwards)

3) Restoring selected entries in specific tables: setup a new db, read the full backup in there, dump these entries to a new file and read that into the real db

How do you so it? Any better aproaches/tools?

Saturday morning 9:30. Alright, let's crack this case. Automated PG backups of DB in a docker swarm attempt 5.

Fuck my life, my morning was very variable it went from "in 4 day's the DB server domain will expire and will be screwed" (was created by a collaborator before my time in the company and not is going dark) to "oh here is the server and there is even daily backups" so it ended well but it was a hell of a journey.

Yeah, sure.. I'll configure your DNS & mail server.. while I'm at it, I'll also optimise your db settings & configure backups..and don't forget LDAP & group policies!! Sure, no probs!! :/