10
Aldar
12d

>Discovers a new low level profiling tool that could help us at work with stuck process debugging and gets all hyped
>Installs on test machine, tool doesn't work
>Wonders why. Oh. Needs a kernel module to work, compiled and loaded
>"Well, its my test machine... Guess that's no problem..." but... my hype died down a bit. Kernel module installation just for a new tool that aggregates all other commonly used tools? eh... Maybe it will blow me out of my shoes still

>Installs and loads the module
>Tool works. Turns out its just a htop-like tool, with shortcuts to launch specific other profiling tools like strace/ltrace/lsof/netstat/ss etc...
"Oh... That's boring. Maybe it has all those tools built in at least?"
>Tries to run ltrace - tool exits as ltrace is not installed

Lol
>Installs ltrace and launches tool again. Tries to ltrace a process and
>Nothing. Nothing happens. For seconds... Then kicks me off of SSH
WTF?
>Tries to ping machine... silence
Did... our net go down again? (Having issues due to a storm going over our area these few days)
>Pings google and... gets instant reply
More wtf
>Pings the hypervisor the machine was running on
Works like normal

Oh... Oh no. Please tell me it didn't!
>Logs into the hypervisor UI, checks machine state
Running OK
>Opens machine console aaaaand... Yep. Stacktrace as well as a lot of kernel mumbo-jumbo... It took the machine down to kernel panic.

I never went so quick from "We need this tool deployed everywhere" to "Omg I need to get rid of this crap as soon as possible" lol.

And just for those wondering, it was sysdig.

Comments
  • 0
    Sysdig, really? That's really odd. Should work like a charm. Unless you have a kernel running that deviates from the generic one given from your OS distro (Mind you, kernel images from distro vendor are digitally signed nowadays) so it should be no issue. Might help to raise an issue on their github project?
Add Comment