229
cdrice
4y

Received "emergency update" code from internal enterprise security team. Wasn't given time to do code review; was assured code was reviewed and solid.

Pushed code to over 6k lower-level servers before finding this gem buried deep within:
...
cd /foo; rm -rf *; cd /
...

(This ran as root, and yes, the cwd was / from earlier in the code).

/foo, of course, did not exist on some servers.

Now, it is those servers which do not exist.

FML

Comments
  • 18
    Welcome to devrant. Lovely first rant. Have an upvote
  • 27
    This kind of fuck up requires an email with some stern language. Also, the security team probably lost privileges to ask for emergency updates.
  • 8
    @nickhh Thank you so much for the kind welcome! I can't believe I've never heard of this until today. Glad to (finally) be here!
  • 31
    I had a similar encounter with a very complex building script for an Android NDK/C++/Lua/ActionScript project made for another branch of the company I worked for. As we "ported" games it was very common to just clone the repo, read the README, change some stuff and run the building system. There was a environment variable that needed to be declared but it was undocumented. It SECURELY DELETED my whole drive. Nice.
  • 2
    Amazing...
  • 2
    ... ouch
  • 6
    When I was updating my dev tools for a project, I added a way to copy my freshly packaged app to my Steam directory to test its behavior in a Steam environment. This, of course, required removing the existing files. I tested the code in a local folder, and forgot to update the path in the code... so when I tested the copy-on-package feature, it deleted my entire project folder.

    I had to clone everything from github again ._.
    The git clone of shame...
  • 3
    Still not good, but using && over ; would have saved you here. Really though, automating rm -rf is just evil.
  • 2
    @iguana Agreed; but, as mentioned, not my code. Was buried deep in a pile of spaghetti code from another team. Using that logic had absolutely no place in production code - it was a mixed blessing this took out some non-production servers before it rolled out to prod. Seeing that code still makes me cringe. "Code was reviewed" my ass.
  • 1
    @cdrice wow, 6k servers? What tool do you use to manage them?
  • 4
    @azous For ad-hoc jobs like this, BMC BladeLogic. Our team supports just shy of 20k *nix targets altogether. Enterprise total is a much scarier number.
  • 9
    Are you the AWS Devop in question?
  • 2
    @Kimmax Hahaha, nope - different enterprise over here. I have been using phrases like "We should ask ourselves: What would Amazon do?" during conference calls - I guess I'll have to dial that back for a while...
  • 1
    Love your posts mate, always classic rants, as it's supposed to be.
  • 0
    @Noob 😁 🤓 I seem to have a lot to rant about where I work...
Add Comment