We are researching enhancing our current alerting system (we use Splunk) to be 'smarter' about who is emailed/texted/whatever when there are problems in our applications.

Currently, if there are over 50 errors logged within a 15 minute period, a email/phone/text blast to nearly 100 individuals ranging from developers, network admins, DBAs, and vice presidents.

Our plan is to group errors by team and let each team manage their own applications. Alert on 1 error, 5, 500...we don't care, let the team work out the particulars.

The trick was interfacing with Splunk's API (that's a long rant by itself)

In about a day or so I was able to use Splunk's WebHook feature to notify a WebAPI service I threw together to send myself an email with details about the underlying data (simulating the kind of alert we would send to the team)

I thought ...cool... it worked. Show it off to the team, most thought it was a good start, except one:

Dev: "The errors are not grouped by team."
Me: "No, I threw the webapi service together to demonstrate how we can extract the splunk bits to get access to the teams"
Dev: "Well...this won't work at all."
Me: "Um..what?"
Dev: "The specification c l e a r l y states the email will be team based. This email was only sent to you and has all the teams and their applications"
Me: "Um...uh...the service can, if we want to go using a service route. Grouping by team name is easy using a LINQ query. I just through this service together yesterday."
Dev: "I don't know. Sounds like I need to schedule a meeting to discuss what you are proposing. I don't think emailing all that to everyone is a good idea."

WTF! Did you not listen to what I said?!!!

Oh well..the dev's proposal is to use splunk's email notification and custom Exchange rules with callbacks into splunk that resend...oh good lord ...a fracking rube goldberg of a config nightmare ...

I suspect we'll go the service route once I finish the service before the meeting.