NAIS Alert reference¶
This document describes all possible configuration values in the Alert spec, commonly known as the alert.yaml file.
alerts¶
Type: array
Required: false
Example
spec:
alerts:
- action: kubectl describe pod {{ $labels.kubernetes_pod_name }} -n {{ $labels.kubernetes_namespace }}` for events, og `kubectl logs {{ $labels.kubernetes_pod_name }} -n {{ $labels.kubernetes_namespace }}` for logger
alert: applikasjon nede
description: App {{ $labels.app }} er nede i namespace {{ $labels.kubernetes_namespace }}
documentation: https://doc.nais.io/observability/alerts/
expr: kube_deployment_status_replicas_available{deployment="<appname>"} > 0
for: 2m
severity: danger
sla: Mellom 8 og 16
alerts[].action¶
What human actions are needed to resolve or investigate this alert.
Type: string
Required: true
Example
spec:
alerts:
- action: kubectl describe pod {{ $labels.kubernetes_pod_name }} -n {{ $labels.kubernetes_namespace }}` for events, og `kubectl logs {{ $labels.kubernetes_pod_name }} -n {{ $labels.kubernetes_namespace }}` for logger
alert: applikasjon nede
description: App {{ $labels.app }} er nede i namespace {{ $labels.kubernetes_namespace }}
documentation: https://doc.nais.io/observability/alerts/
expr: kube_deployment_status_replicas_available{deployment="<appname>"} > 0
for: 2m
severity: danger
sla: Mellom 8 og 16
alerts[].alert¶
The name of the alert.
Type: string
Required: true
Example
spec:
alerts:
- action: kubectl describe pod {{ $labels.kubernetes_pod_name }} -n {{ $labels.kubernetes_namespace }}` for events, og `kubectl logs {{ $labels.kubernetes_pod_name }} -n {{ $labels.kubernetes_namespace }}` for logger
alert: applikasjon nede
description: App {{ $labels.app }} er nede i namespace {{ $labels.kubernetes_namespace }}
documentation: https://doc.nais.io/observability/alerts/
expr: kube_deployment_status_replicas_available{deployment="<appname>"} > 0
for: 2m
severity: danger
sla: Mellom 8 og 16
alerts[].description¶
Simple description of the triggered alert.
Type: string
Required: false
Example
spec:
alerts:
- action: kubectl describe pod {{ $labels.kubernetes_pod_name }} -n {{ $labels.kubernetes_namespace }}` for events, og `kubectl logs {{ $labels.kubernetes_pod_name }} -n {{ $labels.kubernetes_namespace }}` for logger
alert: applikasjon nede
description: App {{ $labels.app }} er nede i namespace {{ $labels.kubernetes_namespace }}
documentation: https://doc.nais.io/observability/alerts/
expr: kube_deployment_status_replicas_available{deployment="<appname>"} > 0
for: 2m
severity: danger
sla: Mellom 8 og 16
alerts[].documentation¶
URL for documentation for this alert.
Type: string
Required: false
Example
spec:
alerts:
- action: kubectl describe pod {{ $labels.kubernetes_pod_name }} -n {{ $labels.kubernetes_namespace }}` for events, og `kubectl logs {{ $labels.kubernetes_pod_name }} -n {{ $labels.kubernetes_namespace }}` for logger
alert: applikasjon nede
description: App {{ $labels.app }} er nede i namespace {{ $labels.kubernetes_namespace }}
documentation: https://doc.nais.io/observability/alerts/
expr: kube_deployment_status_replicas_available{deployment="<appname>"} > 0
for: 2m
severity: danger
sla: Mellom 8 og 16
alerts[].expr¶
Prometheus expression that triggers an alert. Explore expressions in the Prometheus-interface
Type: string
Required: true
Example
spec:
alerts:
- action: kubectl describe pod {{ $labels.kubernetes_pod_name }} -n {{ $labels.kubernetes_namespace }}` for events, og `kubectl logs {{ $labels.kubernetes_pod_name }} -n {{ $labels.kubernetes_namespace }}` for logger
alert: applikasjon nede
description: App {{ $labels.app }} er nede i namespace {{ $labels.kubernetes_namespace }}
documentation: https://doc.nais.io/observability/alerts/
expr: kube_deployment_status_replicas_available{deployment="<appname>"} > 0
for: 2m
severity: danger
sla: Mellom 8 og 16
alerts[].for¶
Duration before the alert should trigger.
Type: string
Required: true
Pattern: ^\d+[smhdwy]$
Example
spec:
alerts:
- action: kubectl describe pod {{ $labels.kubernetes_pod_name }} -n {{ $labels.kubernetes_namespace }}` for events, og `kubectl logs {{ $labels.kubernetes_pod_name }} -n {{ $labels.kubernetes_namespace }}` for logger
alert: applikasjon nede
description: App {{ $labels.app }} er nede i namespace {{ $labels.kubernetes_namespace }}
documentation: https://doc.nais.io/observability/alerts/
expr: kube_deployment_status_replicas_available{deployment="<appname>"} > 0
for: 2m
severity: danger
sla: Mellom 8 og 16
alerts[].severity¶
Alert level for Slack messages.
Type: string
Required: false
Default value: danger
Pattern: ^$|good|warning|danger|#([A-Fa-f0-9]{6}|[A-Fa-f0-9]{3})
Example
spec:
alerts:
- action: kubectl describe pod {{ $labels.kubernetes_pod_name }} -n {{ $labels.kubernetes_namespace }}` for events, og `kubectl logs {{ $labels.kubernetes_pod_name }} -n {{ $labels.kubernetes_namespace }}` for logger
alert: applikasjon nede
description: App {{ $labels.app }} er nede i namespace {{ $labels.kubernetes_namespace }}
documentation: https://doc.nais.io/observability/alerts/
expr: kube_deployment_status_replicas_available{deployment="<appname>"} > 0
for: 2m
severity: danger
sla: Mellom 8 og 16
alerts[].sla¶
Time before a human should resolve the alert.
Type: string
Required: false
Example
spec:
alerts:
- action: kubectl describe pod {{ $labels.kubernetes_pod_name }} -n {{ $labels.kubernetes_namespace }}` for events, og `kubectl logs {{ $labels.kubernetes_pod_name }} -n {{ $labels.kubernetes_namespace }}` for logger
alert: applikasjon nede
description: App {{ $labels.app }} er nede i namespace {{ $labels.kubernetes_namespace }}
documentation: https://doc.nais.io/observability/alerts/
expr: kube_deployment_status_replicas_available{deployment="<appname>"} > 0
for: 2m
severity: danger
sla: Mellom 8 og 16
inhibitRules¶
A list of inhibit rules. An inhibition rule mutes an alert (target) matching a set of matchers when an alert (source) exists that matches another set of matchers. Both target and source alerts must have the same label values for the label names in the labels list.
Relevant information:
Type: array
Required: false
Example
spec:
inhibitRules:
- labels:
- label
- name
sources:
key: value
sourcesRegex:
key: value(.)?
targets:
key: value
targetsRegex:
key: value(.)+
inhibitRules[].labels¶
Labels that must have an equal value in the source and target alert for the inhibition to take effect.
Type: array
Required: false
Example
spec:
inhibitRules:
- labels:
- label
- name
sources:
key: value
sourcesRegex:
key: value(.)?
targets:
key: value
targetsRegex:
key: value(.)+
inhibitRules[].sources¶
Matchers for which one or more alerts have to exist for the inhibition to take effect. These are key/value pairs.
Type: object
Required: false
Example
spec:
inhibitRules:
- labels:
- label
- name
sources:
key: value
sourcesRegex:
key: value(.)?
targets:
key: value
targetsRegex:
key: value(.)+
inhibitRules[].sourcesRegex¶
Regex matchers for which one or more alerts have to exist for the inhibition to take effect. These are key/value pairs, where the value can be a regex.
Type: object
Required: false
Example
spec:
inhibitRules:
- labels:
- label
- name
sources:
key: value
sourcesRegex:
key: value(.)?
targets:
key: value
targetsRegex:
key: value(.)+
inhibitRules[].targets¶
Matchers that have to be fulfilled in the alerts to be muted. These are key/value pairs.
Type: object
Required: false
Example
spec:
inhibitRules:
- labels:
- label
- name
sources:
key: value
sourcesRegex:
key: value(.)?
targets:
key: value
targetsRegex:
key: value(.)+
inhibitRules[].targetsRegex¶
Regex matchers that have to be fulfilled in the alerts to be muted. These are key/value pairs, where the value can be a regex.
Type: object
Required: false
Example
spec:
inhibitRules:
- labels:
- label
- name
sources:
key: value
sourcesRegex:
key: value(.)?
targets:
key: value
targetsRegex:
key: value(.)+
receivers¶
A list of notification recievers. You can use one or more of: e-mail, slack, sms. There needs to be at least one receiver.
Type: object
Required: false
Example
spec:
receivers:
email:
to: myteam@nav.no
slack:
channel: '#alert-channel'
icon_emoji: ':chart_with_upwards_trend:'
icon_url: http://lorempixel.com/48/48
prependText: Oh noes!
send_resolved: true
username: Alertmanager
sms:
recipients: "12345678"
send_resolved: false
webhook:
http_config:
proxy_url: webproxy.nav
tls_config:
insecure_skip_verify: true
max_alerts: 0
send_resolved: true
url: https://the.feature.now
receivers.email¶
Alerts via e-mails
Type: object
Required: false
Example
spec:
receivers:
email:
to: myteam@nav.no
receivers.email.send_resolved¶
Whether or not to notify about resolved alerts.
Type: boolean
Required: false
Default value: false
receivers.email.to¶
Type: string
Required: true
Example
spec:
receivers:
email:
to: myteam@nav.no
receivers.slack¶
Slack notifications are sent via Slack webhooks.
Type: object
Required: false
Example
spec:
receivers:
slack:
channel: '#alert-channel'
icon_emoji: ':chart_with_upwards_trend:'
icon_url: http://lorempixel.com/48/48
prependText: Oh noes!
send_resolved: true
username: Alertmanager
receivers.slack.channel¶
The channel or user to send notifications to. Can be specified with and without #.
Type: string
Required: true
Example
spec:
receivers:
slack:
channel: '#alert-channel'
receivers.slack.icon_emoji¶
Emoji to use as the icon for this message
Type: string
Required: false
Example
spec:
receivers:
slack:
icon_emoji: ':chart_with_upwards_trend:'
receivers.slack.icon_url¶
URL to an image to use as the icon for this message
Type: string
Required: false
Example
spec:
receivers:
slack:
icon_url: http://lorempixel.com/48/48
receivers.slack.prependText¶
Text to prepend every Slack message with severity danger.
Type: string
Required: false
Example
spec:
receivers:
slack:
prependText: Oh noes!
receivers.slack.send_resolved¶
Whether or not to notify about resolved alerts.
Type: boolean
Required: false
Default value: true
Example
spec:
receivers:
slack:
send_resolved: true
receivers.slack.username¶
Set your bot's user name.
Type: string
Required: false
Example
spec:
receivers:
slack:
username: Alertmanager
receivers.sms¶
Alerts via SMS
Type: object
Required: false
Example
spec:
receivers:
sms:
recipients: "12345678"
send_resolved: false
receivers.sms.recipients¶
Type: string
Required: true
Example
spec:
receivers:
sms:
recipients: "12345678"
receivers.sms.send_resolved¶
Whether or not to notify about resolved alerts.
Type: boolean
Required: false
Default value: true
Example
spec:
receivers:
sms:
send_resolved: false
receivers.webhook¶
Alerts via custom web application
Type: object
Required: false
Example
spec:
receivers:
webhook:
http_config:
proxy_url: webproxy.nav
tls_config:
insecure_skip_verify: true
max_alerts: 0
send_resolved: true
url: https://the.feature.now
receivers.webhook.http_config¶
A http_config allows configuring the HTTP client that the receiver uses to communicate with HTTP-based API services.
Type: object
Required: false
Example
spec:
receivers:
webhook:
http_config:
proxy_url: webproxy.nav
tls_config:
insecure_skip_verify: true
receivers.webhook.http_config.proxy_url¶
Optional proxy URL.
Type: string
Required: false
Example
spec:
receivers:
webhook:
http_config:
proxy_url: webproxy.nav
receivers.webhook.http_config.tls_config¶
Configures the TLS settings.
Type: object
Required: false
Example
spec:
receivers:
webhook:
http_config:
tls_config:
insecure_skip_verify: true
receivers.webhook.http_config.tls_config.insecure_skip_verify¶
Disable validation of the server certificate.
Type: boolean
Required: false
Default value: false
Example
spec:
receivers:
webhook:
http_config:
tls_config:
insecure_skip_verify: true
receivers.webhook.max_alerts¶
The maximum number of alerts to include in a single webhook message. Alerts above this threshold are truncated. When leaving this at its default value of 0, all alerts are included.
Type: integer
Required: true
Default value: 0
Example
spec:
receivers:
webhook:
max_alerts: 0
receivers.webhook.send_resolved¶
Whether or not to notify about resolved alerts.
Type: boolean
Required: false
Default value: true
Example
spec:
receivers:
webhook:
send_resolved: true
receivers.webhook.url¶
The endpoint to send HTTP POST requests to.
Type: string
Required: true
Example
spec:
receivers:
webhook:
url: https://the.feature.now
route¶
Type: object
Required: false
Example
spec:
route:
group_by:
- label
- name
groupInterval: 5m
groupWait: 30s
repeatInterval: 3h
route.groupInterval¶
How long to wait before sending a notification about new alerts that are added to a group of alerts for which an initial notification has already been sent.
Type: string
Required: false
Default value: 5m
Pattern: ((([0-9]+)y)?(([0-9]+)w)?(([0-9]+)d)?(([0-9]+)h)?(([0-9]+)m)?(([0-9]+)s)?(([0-9]+)ms)?|0)
Example
spec:
route:
groupInterval: 5m
route.groupWait¶
How long to initially wait to send a notification for a group of alerts. Allows to wait for an inhibiting alert to arrive or collect more initial alerts for the same group.
Type: string
Required: false
Default value: 10s
Pattern: ((([0-9]+)y)?(([0-9]+)w)?(([0-9]+)d)?(([0-9]+)h)?(([0-9]+)m)?(([0-9]+)s)?(([0-9]+)ms)?|0)
Example
spec:
route:
groupWait: 30s
route.group_by¶
The labels by which incoming alerts are grouped together.
Type: array
Required: false
Example
spec:
route:
group_by:
- label
- name
route.repeatInterval¶
How long to wait before sending a notification again if it has already been sent successfully for an alert.
Type: string
Required: false
Default value: 1h
Pattern: ((([0-9]+)y)?(([0-9]+)w)?(([0-9]+)d)?(([0-9]+)h)?(([0-9]+)m)?(([0-9]+)s)?(([0-9]+)ms)?|0)
Example
spec:
route:
repeatInterval: 3h