A great post by Nick Craver on how they do logging at Stack Overflow.
They log from HAProxy into SQL Server. OH. MY. GOD.
Bosun is cool. It’s easy to get metrics into, and intelligent for people to use for alerting once they’re there. Testing on existing data is fantastic.
A key feature of Bosun I really love is the ability to test an alert against history while designing it. This helps seeing when it would have triggered. It’s an awesome sanity check. Let’s be honest, monitoring isn’t perfect, it was never perfect, and it won’t ever be perfect. A lot of monitoring comes from lessons learned, because the things that go wrong often include things you never even considered going wrong…and that means you didn’t have monitoring and/or alerts on them from day 1.