默认的告警格式如下：
1 alert for alertname=ContainerUnhealthy namespace=default
Alerts Firing:
Labels:
- alertname = ContainerUnhealthy
- alerttype = event
- cluster = default
- container = nginx
- namespace = default
- pod = nginx-f744fc676-vl9sb
- severity = warning
Annotations:
- message = Readiness probe failed: Get "http://10.42.135.40:80/health": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
- summary = Container is unhealthy
- summaryCn = 容器状态不良需要修改成中文格式。
解决
kubesphere使用的是notification-manager，直接修改alertmanager添加alertmanager-additional-config secret这种方式是无效的。
这是默认的模板文件：notification-manager-template
{{ define "nm.default.subject" }}{{ .Alerts | len }} alert{{ if gt (len .Alerts) 1 }}s{{ end }} for {{ range .GroupLabels.SortedPairs }} {{ .Name }}={{ .Value }} {{ end }}
{{- end }}
{{ define "__nm_alert_list" }}{{ range . }}Labels:
{{ range .Labels.SortedPairs }}{{ if ne .Name "runbook_url" }}- {{ .Name }} = {{ .Value }}{{ end }}
{{ end }}Annotations:
{{ range .Annotations.SortedPairs }}{{ if ne .Name "runbook_url"}}- {{ .Name }} = {{ .Value }}{{ end }}
{{ end }}{{ end }}{{ end }}
{{ define "nm.default.text" }}{{ template "nm.default.subject" . }}
{{ if gt (len .Alerts.Firing) 0 -}}
Alerts Firing:
{{ template "__nm_alert_list" .Alerts.Firing }}
{{- end }}
{{ if gt (len .Alerts.Resolved) 0 -}}
Alerts Resolved:
{{ template "__nm_alert_list" .Alerts.Resolved }}
{{- end }}
{{- end }}
{{ define "__nm_alert_list_markdown" }}{{ range . }}### Labels:
{{ range .Labels.SortedPairs }}{{ if ne .Name "runbook_url" }}- {{ .Name }} = {{ .Value }}{{ end }}
{{ end }}### Annotations:
{{ range .Annotations.SortedPairs }}{{ if ne .Name "runbook_url"}}- {{ .Name }} = {{ .Value }}{{ end }}
{{ end }}{{ end }}{{ end }}
{{ define "nm.default.markdown" }}# {{ template "nm.default.subject" . }}
{{ if gt (len .Alerts.Firing) 0 -}}
## <font color=#CD0000>Alerts Firing:</font>
{{ template "__nm_alert_list_markdown" .Alerts.Firing }}
{{- end }}
{{ if gt (len .Alerts.Resolved) 0 -}}
## <font color=#00CD00>Alerts Resolved:</font>
{{ template "__nm_alert_list_markdown" .Alerts.Resolved }}
{{- end }}
{{- end }}
{{ define "nm.default.html" }}
<html xmlns="http://www.w3.org/1999/xhtml" xmlns="http://www.w3.org/1999/xhtml" style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;">
<head style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;">
<meta name="viewport" content="width=device-width" style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;" />
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;" />
<title style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;">{{ template "__subject" . }}</title>
</head>
<body itemscope="" itemtype="http://schema.org/EmailMessage" style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; -webkit-font-smoothing: antialiased; -webkit-text-size-adjust: none; height: 100%; line-height: 1.6em; width: 100% !important; background-color: #f6f6f6; margin: 0; padding: 0;" bgcolor="#f6f6f6">
<table style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; width: 100%; background-color: #f6f6f6; margin: 0;" bgcolor="#f6f6f6">
<tr style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;">
<td style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; vertical-align: top; margin: 0;" valign="top"></td>
<td width="600" style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; vertical-align: top; display: block !important; max-width: 600px !important; clear: both !important; width: 100% !important; margin: 0 auto; padding: 0;" valign="top">
<div style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; max-width: 600px; display: block; margin: 0 auto; padding: 0;">
<table width="100%" cellpadding="0" cellspacing="0" style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; border-radius: 3px; background-color: #fff; margin: 0; border: 1px solid #e9e9e9;" bgcolor="#fff">
<tr style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;">
<td style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 16px; vertical-align: top; color: #fff; font-weight: 500; text-align: center; border-radius: 3px 3px 0 0; background-color: #E6522C; margin: 0; padding: 20px;" align="center" bgcolor="#E6522C" valign="top">
{{ .Alerts | len }} alert{{ if gt (len .Alerts) 1 }}s{{ end }} for {{ range .GroupLabels.SortedPairs }}
{{ .Name }}={{ .Value }}
{{ end }}
</td>
</tr>
<tr style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;">
<td style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; vertical-align: top; margin: 0; padding: 10px;" valign="top">
<table width="100%" cellpadding="0" cellspacing="0" style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;">
{{ if gt (len .Alerts.Firing) 0 }}
<tr style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;">
<td style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; vertical-align: top; margin: 0; padding: 0 0 20px;" valign="top">
<strong style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;">[{{ .Alerts.Firing | len }}] Firing</strong>
</td>
</tr>
{{ end }}
{{ range .Alerts.Firing }}
<tr style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;">
<td style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; vertical-align: top; margin: 0; padding: 0 0 20px;" valign="top">
<strong style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;">Labels</strong><br style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;" />
{{ range .Labels.SortedPairs }}{{ if ne .Name "runbook_url" }}{{ .Name }} = {{ .Value }}{{ end }}<br style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;" />{{ end }}
{{ if gt (len .Annotations) 0 }}<strong style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;">Annotations</strong><br style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;" />{{ end }}
{{ range .Annotations.SortedPairs }}{{ if ne .Name "runbook_url" }}{{ .Name }} = {{ .Value }}{{ end }}<br style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;" />{{ end }}
</td>
</tr>
{{ end }}
{{ if gt (len .Alerts.Resolved) 0 }}
{{ if gt (len .Alerts.Firing) 0 }}
<tr style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;">
<td style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; vertical-align: top; margin: 0; padding: 0 0 20px;" valign="top">
<br style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;" />
<hr style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;" />
<br style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;" />
</td>
</tr>
{{ end }}
<tr style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;">
<td style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; vertical-align: top; margin: 0; padding: 0 0 20px;" valign="top">
<strong style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;">[{{ .Alerts.Resolved | len }}] Resolved</strong>
</td>
</tr>
{{ end }}
{{ range .Alerts.Resolved }}
<tr style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;">
<td style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; vertical-align: top; margin: 0; padding: 0 0 20px;" valign="top">
<strong style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;">Labels</strong><br style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;" />
{{ range .Labels.SortedPairs }}{{ if ne .Name "runbook_url" }}{{ .Name }} = {{ .Value }}{{ end }}<br style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;" />{{ end }}
{{ if gt (len .Annotations) 0 }}<strong style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;">Annotations</strong><br style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;" />{{ end }}
{{ range .Annotations.SortedPairs }}{{ if ne .Name "runbook_url" }}{{ .Name }} = {{ .Value }}{{ end }}<br style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;" />{{ end }}
</td>
</tr>
{{ end }}
</table>
</td>
</tr>
</table>
</div>
</td>
<td style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; vertical-align: top; margin: 0;" valign="top"></td>
</tr>
</table>
</body>
</html>
{{ end }}
notification-manager的告警数据结构参考这里：
https://github.com/kubesphere/notification-manager/blob/master/docs/template.md#Multilingual-support
创建configmap
这里必须使用nm.default.text或者nm.default.markdown，代码里写死了。
apiVersion: v1
data:
template: |2
{{ define "nm.default.text" }}
{{- if gt (len .Alerts.Firing) 0 -}}
{{- range $index, $alert := .Alerts -}}
========= 监控告警 =========
告警名称: {{ $alert.Labels.alertname }}
告警级别: {{ $alert.Labels.severity }} 级
故障主机: {{ $alert.Labels.instance }} {{ $alert.Labels.device }}
集群名称: {{ $alert.Labels.cluster }}
命名空间: {{ $alert.Labels.namespace }}
Pod名称: {{ $alert.Labels.pod }}
告警主题: {{ .Annotations.summary }} {{ .Annotations.summaryCn }}
告警详情: {{ $alert.Annotations.message }}{{ $alert.Annotations.description}}
主机标签: {{ range .Labels.SortedPairs }} [{{ .Name }}: {{ .Value | html }} ] {{- end }}
故障时间: {{ ($alert.StartsAt.Add 28800e9).Format "2023-01-29 12:00:00" }}
========= = end = =========
{{- end }}
{{- end }}
{{- if gt (len .Alerts.Resolved) 0 -}}
{{- range $index, $alert := .Alerts -}}
========= 告警恢复 =========
告警主题: {{ .Annotations.summary }} {{ .Annotations.summaryCn }}
告警主机: {{ .Labels.instance }}
告警类型: {{ .Labels.alertname }}
告警级别: {{ $alert.Labels.severity }} 级
集群名称: {{ $alert.Labels.cluster }}
命名空间: {{ $alert.Labels.namespace }}
Pod名称: {{ $alert.Labels.pod }}
告警主题: {{ .Annotations.summary }} {{ .Annotations.summaryCn }}
故障时间: {{ ($alert.StartsAt.Add 28800e9).Format "2023-01-29 12:00:00" }}
恢复时间: {{ ($alert.EndsAt.Add 28800e9).Format "2023-01-29 12:00:00" }}
========= = end = =========
{{- end }}
{{- end }}
{{- end }}
kind: ConfigMap
metadata:
annotations:
meta.helm.sh/release-name: notification-manager
meta.helm.sh/release-namespace: kubesphere-monitoring-system
labels:
app.kubernetes.io/managed-by: Helm
name: notification-templ
namespace: kubesphere-monitoring-system
修改notification-manager
notification-manager创建了3个crd：configs，notificationmanagers，receivers。
直接修改notificationmanagers的cm name即可。
k edit nm
notification-manager的pod会自动重启。
查看告警
========= 监控告警 =========
告警名称: ContainerUnhealthy
告警级别: warning 级
故障主机:
集群名称: default
命名空间: default
Pod名称: nginx-558877494d-hfrvp
告警主题: Container is unhealthy 容器状态不良
告警详情: Readiness probe failed: Get "http://10.42.3.115:80/health": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
主机标签: [alertname: ContainerUnhealthy ] [alerttype: event ] [cluster: default ] [container: nginx ] [namespace: default ] [pod: nginx-558877494d-hfrvp ] [severity: warning ]
故障时间: 2023-01-29 12:26:40
========= = end = =========
你好,alertmanager-main-0 是怎么配置企业微信应用告警的
有notification-manager服务后，平台管理–平台设置–通知配置–企业微信，填写你的企业id，应用 AgentId，应用 Secret，设置告警条件，指定接收人即可。
但是上述配置之后其他通知渠道都用不了了呀,,其它模板根据哪里可以进行修改