My mom got cancer and passed away two years later. I had made several videos to document her life. All of that, plus photos and video of our family, lived on a backup drive that got dropped on the floor a few months after she passed away. Head crash that wasn’t discovered until it was turned on again and forever destroyed the platter. It was the largest set of photos and video I had of her. All forever gone.

Our team really needs some workflow arrangement, and this time it was me who screwed up.
So we have to push an update to the Play Store and the App Store the Friday, the app is well tested on test environment then production environment, we got the ok so I uploaded a build, the app management team then continued the process of publishing..

During the weekend the app was approved and live to almost 500k user that can receive the update.

I got a phone call from the Project Manager at almost midnight, the time was really suspicious so I answered.
- Me: Hello.
- PM: Hi, sorry to call you now but the app is live and we have a problem.
- Me: what kind of problem? Let me check.

So I updated the app on my phone and opened it while I am on call.. I almost had heart attack!! WE PUBLISHED A VERSION POINTING TO THE TEST ENVIRONMENT. Holly shit

- Me: shit call the app management team NOW.

Eventually we removed the app from sale (unpublished it) and we submitted a new version immediately, once it was approved the next day we made the app available again (so for those who didn’t update yet, there will be no update to a faulted version, and no new users landing to a version with test data), I received one or two calls from friends telling me why the app is not on the store (our app is used nationally, so it’s really important).

Thank God there was no big show on twitter or other social media.. but it’s really a good lesson to learn.

I understand this is totally my fault, thankfully I didn’t get fired 😅

3 weeks into a new job I learned that my predecessor (who resigned and was out the door two days after I started) didn’t know how to secure s3 buckets when all of our production image assets got replaced with elder porn.

Jury’s still out if it was actually him the whole time.

I lost 2TB of family photos and videos a few years ago by dropping a single hard drive. Nearly all of it was later found on people's computers who forgot to clean them up when we got the NAS. I'm not so eager to delete data once it's backed up ever since.

Murphy is alive and well - and working on your project.

After all the rants I've written on this topic, no, no, fuck no. I ain't answering jackshit. The trauma is very real. I'm trying to not have a panic attack just remembering few of the times I've lost work, personal data, side projects, accounts, you name it.
Oh dear God it's hard to breathe...

Whatsapp...if you're trying to switch over to a new phone or just reinstalling on the same phone for some reason.
It creates a "local" backup everyday at 2.30 am and a Google drive backup depending on the frequency you set in the settings (I usually keep it to "Weekly"). I've lost important messages this way more than I care to admit.

MongoDB..

200 searchable records (heavily searched and essential to the app) kept getting deleted from their index on our search engine.

we couldnt figure out why, but we wanted to resolve this quickly next time it happens so we got them back up, duplicated the index, and whenever we see they got deleted we resync them from the replica.

Eventually i found that there was a piece of code that updates the index by deleting everything from it and uploading everything back but it was buggy in doing the latter.

At old e-commerce job, some orders were coming through with most of the shipping info missing. The only info filled out was the State. When we looked at Heap, we could see the user was filling in those fields. There was both frontend and backend validation for required form data, so the user shouldn’t have been able to checkout without an address.

When I looked at the BE logic, I saw addresses were retrieved from our database by using a method called GetOrCreateDefaultAddress. When the website couldn’t find the address in the db, it created a new one where the only address field that was filled in was the state.

Unfortunately, this default address creation was happening after the submit button had been hit. There was no logic to validate the address this late in the checkout because the earlier form validation in the process should have caught this.

The orders did have email addresses, so customer service did have a way to contact the customer. I have no idea what happened to the user’s address. Was it never saved? Did it get caught up in a cron job to delete old users and addresses from the db??