Bosun supports full customization over notification contents, and provides a templating engine to generate whatever content you need from an alert. Multiple components affect how your notifications look:
We support sending notifications via:
There are also various different types of notification events, each having different kinds of underlying data and semantics:
Alerts mostly define the rules that we alert on. We can control notifications with the template, critNotification, and warnNotification keys.
alert high_cpu {
crit = 1
template = high_cpu
critNotification = email,slack
}
defines an alert linked to the high_cpu template, and the email and slack notifications.
Templates define specific templates for rendering your content. For guidance on constructing templates, see the relevant documentation. A template is essentially a set of key/value pairs. It may define as many keys as it wishes with whatever keys it likes. There are a few special keys however:
body and subject are the only required template keys. These are what show up on the bosun dashboard, as well as the default for most other notifications.emailBody and emailSubject are not required, but if present, they will be used as the body and subject for email notifications.Any other template key will may be defined and will be used by any notifications that select it.
A template may inherit another template, which copies all of the key/value pairs into the child template. This is useful if you have some set of common formatting templates that may be shared among multiple templates. An example:
# base template for all slack notifications. Creates json message using alert subject.
template slack {
slackBody = `{
"text": "{{.Subject}} <{{.Incident}}|view in bosun>",
"username": "bosun",
"icon_url": "https://i.imgur.com/ogj0wkj.png",
}`
}
template high_cpu {
body = {{.Subject}}
subject = `High CPU on {{.Group.host}}`
# inherit slack template
inherit = slack
}
notification slack {
post = ${sys.SLACK_URL}
#select slack body template
bodyTemplate = slackBody
}
alert high_cpu {
crit = avg(series("host=server01", epoch(), 1))
template = high_cpu
critNotification = slack
}
There are a few situations where it matters if we use plain text templates, or html templates. Html templates perform some extra sanitization for when we expect to display the content, and they also perform css-inlining to be more compatible with email clients. The rules are simple:
body and emailBody are always rendered as html templates.HTML (like myCustomHTML) will be rendered as html.subject) will be rendered as plain-text.A notification’s job is to choose what content gets sent, and where to send it. It is common to make a unique notification for each unique email address or list that bosun sends to, and for each url/api it calls. For alerts, the notification choses which of the pre-rendered templates to send. For actions and unknowns, it will pick the right template, and render it on the fly. Rules for template selection are as follows:
For alert emails, the subject and body are chosen according to the following priorities:
emailSubjectTemplate, use that key from the template.emailSubject key defined, use that.subject template will be used. It will be rendered with the .IsEmail flag set, if you would like to customize part of it for email.bodyTemplate, use that key.emailBody key defined, use that.body template, rendering with .IsEmail set to true.Bosun can send http notifications using the following precedence rules:
postTemplate or getTemplate, those rendered templates will be used as the notification url.post or get values are used.bodyTemplate, use that rendered template as the post body.subject template.Action notifications are a little different than alert notifications. They are rendered as actions happen, and they use a different context than the alert templates, and has the following data available:
{{.States}} is a list of all incidents affected.{{.User}} is the user who performed the action.{{.Message}} is the message they entered.{{.ActionType}} is the type of action.If multiple actions are performed at once, they are grouped together by default. You can disable this, and send a notification for each individual alert key by setting groupActions = false in the notification. You can get the first incident from States with {{$first := index .States 0}} if this is the case.
You can choose whether a notification sends action notifications or not on a per-action basis using the runOnActions key. You may set it to all or none, or to any comma separated list of action types from Ack, Close, Forget, ForceClose, Purge, Note, DelayedClose, or CancelClose.
If you do not override anything in the notification, bosun will use its’ own built in action template for action notifications. You can otherwise specify a template to use for all actions, or to override only spcific actions. You may customize a number of fields individually as well. The general form for these keys is:
action{TemplateType}{ActionType?}
Where “templateType” is one of Body, Get, Post, or EmailSubject, and “ActionType” if present, is one of Ack, Close, Forget, ForceClose, Purge, Note, DelayedClose, or CancelClose. If Action Type is not specified, it will apply to all actions types, unless specifically overridden.
For example, setting actionBody = keyX, will use the keyX template for all action notification bodies for all action types, but actionBodyAck = keyY, will use the keyY template only for acknowledge actions.
Example Slack Action notification:
template slack {
#format action like "steve_brown Acknowledged incident 45 (High CPU on server01): "I was running a load test"
slackActionBody = `{{$first := index .States 0}}{
"text": "{{.User}} {{.ActionType}} <{{.IncidentLink $first.Id}}|incident {{$first.Id}}> ({{$first.Subject}}): \"{{.Message}}\"",
"username": "bosun",
"icon_url": "https://i.imgur.com/ogj0wkj.png",
}`
}
notification slack {
post = ${sys.SLACK_URL}
bodyTemplate = slackBody
runOnActions = Ack
groupActions = false
actionBody = slackActionBody
}
When an alert goes “unknown”, it will send a special notification through critNotification to let you know. Similar to actions, these notifications are rendered on-demand, with a special context. Bosun attempts to group these appropriately to reduce spam. The context has:
{{.Time}}, a timestamp of when the unknown event occurred.{{.Name}}, bosun’s description of the tags or alert name for the grouping.{{.Group}}, list of alert keys that are unknown.If you want to control the way bosun groups unknowns, you can set the unknownMinGroupSize value in a notification to override bosun’s global default value. This controls the number of alert keys that need to share a common tag in order for us to consider them “similar”. You can set this to 0 to never group unknowns together into groups, and send a notification for each alert key.
Bosun also has a threshold for how many unknown notifications it will send at all in a single batch of notifications. This can be overridden in a notification with the unknownThreshold value. Setting to 0 will again remove the limit. Once the limit is reached, the final batch will switch over to using the “multiple unknown groups” templates, which use a similar context, except {{.Groups}} is now a lookup of group names to a list of alert keys.
You can define which template keys a notification will use for unknown notifications by setting
unknownBody, unknownPost, unknownGet, and unknownEmailSubject
or
unknownMultiBody, unknownMultiPost, unknownMultiGet, and unknownMultiEmailSubject.
If a template is not set as needed by the notification type, a built-in default template will be used.