33

Let me tell you a story.

Our company has a homegrown monitoring solution. Keeps track of our deployments and alerts us when something is broken. Really nice for the most part, except a little issue where we get up to 25 alerts PER DAY that our PRODUCTION ENVIRONMENT IS DOWN. Including weekends.

With this many false positives, we quickly learn to ignore the alerts and miss real incidents.

So we approached this team, remember its our own tool, and told them about the problem. Turns out it is a known issue. And here's the kicker: they aren't planning on fixing it!

It gets better. Rather than fix this glaring issue, their solution is to make ANOTHER ALERT that lets us know the monitoring is misbehaving.

To recap, we can now expect to get up to 25 false positive alerts per day that our production is down, followed immediately by more alerts that the monitor is broken, which means we can ignore the previous alert.

As our PM said when he heard this: fuck that noise. We are escalating the shit out of this!

Comments
  • 7
    You should write an auto responder that writes chatty emails and sends text messages to the person responsible. Ask another 2 person to use your auto responder too. He's gonma fix it real quick for sure.
  • 4
    wow, this is beyond stupid
  • 0
    Right up until you said your PM was going to escalate that shitte, I thought I knew what company you were talking about... *shudder*
  • 0
    *** shakes head ***
  • 0
    OP, listen this man! @azzuwan
  • 0
    Oh my! But if they like ugly hacks why don't they wrap the whole thing in a extra layer that keeps the notification for a little bit and only send it if there is no following message about monitoring acting up?
  • 0
    There are a number of good free monitoring tools out there. I have used spiceworks on a small network and it performed amazingly well
Add Comment