75
linuxxx
31d

Waaaay too many but let's go with this one for now.

At my previous job there was a web application which was generating about 1gb of log data a second. Server was full and the 'fullstack engineers' we called had zero clue about backend stuff and couldn't fix it.

Me and another engineer worked our asses off to figure this out but eventually the logging stopped and it went back to normal.

Great, right?

For that moment. I was the on-call server engineer and at like 3am I got called awake because this shit was happening again.
Sleep drunk with my phone I ssh'd into the server, not sure about what to do at first but then suddenly: let's chattr the goddamn log file...

$ chattr +i /var/log/logfile

Bam, worked, done, back to sleep.

(this comment + param marks the file in a way that it can only be read until the mark is removed, so you can't write to it or move it or remove it or whatever)

Comments
  • 17
    I'm writing that down. I literally never saw that before.
  • 17
    @PrivateGER Have to run it as root! Also, to unchattr, use the -i param.

    Note: you CANNOT see the chattr status of a file with the generic ls command, you have to do "lsattr" and if you see an 'i' before one of the files, it's chattred.
  • 14
    @PrivateGER Second note, sometimes when a program can't write log output it can crash so be careful with this 😅
  • 10
    no fucks given lmao
  • 9
    @neeno what would you have done at 3am when all you want is sleep? Extinguish the fire out and deal with it next Thursday.
  • 4
    > full stack engineers had zero clue
    *instand ++ because feeling with you*
  • 3
    If I were you, I'd chattr / and get this done, this way we make sure server never gets full again, and no more late night calls
  • 2
    Really nice one!
  • 1
    How come you didn’t wake up 1 hour later with the program doing the logging crashing because it overflowed in memory and couldn’t flush the logs to disk? 😦
  • 1
    @brogrammerology That wasn't our responsibility and we didn't monitor that as it... wasn't our responsibility
  • 1
    @gitpush Nah, the logging was there for a reason, they just needed to fix this issue.
  • 0
    @linuxxx '>/go/away/come/back/another/day.log'

    Is my solution when people throw the 'fix eeet - I need it now' at me.
  • 1
    @theKarlisK Yeah I'd do that haha but we didn't have control over the log file path 😬
  • 0
    My 2 cents
Add Comment