Wghdr's Blog!

k8s, rancher/kubeSphere

kubesphere修改告警模板

wghdr

默认的告警格式如下：

1 alert for  alertname=ContainerUnhealthy  namespace=default
Alerts Firing:
Labels:
- alertname = ContainerUnhealthy
- alerttype = event
- cluster = default
- container = nginx
- namespace = default
- pod = nginx-f744fc676-vl9sb
- severity = warning
Annotations:
- message = Readiness probe failed: Get "http://10.42.135.40:80/health": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
- summary = Container is unhealthy
- summaryCn = 容器状态不良

需要修改成中文格式。

解决

kubesphere使用的是notification-manager，直接修改alertmanager添加alertmanager-additional-config secret这种方式是无效的。

file

这是默认的模板文件：notification-manager-template

{{ define "nm.default.subject" }}{{ .Alerts | len }} alert{{ if gt (len .Alerts) 1 }}s{{ end }} for {{ range .GroupLabels.SortedPairs }} {{ .Name }}={{ .Value }} {{ end }}
    {{- end }}

    {{ define "__nm_alert_list" }}{{ range . }}Labels:
    {{ range .Labels.SortedPairs }}{{ if ne .Name "runbook_url" }}- {{ .Name }} = {{ .Value }}{{ end }}
    {{ end }}Annotations:
    {{ range .Annotations.SortedPairs }}{{ if ne .Name "runbook_url"}}- {{ .Name }} = {{ .Value }}{{ end }}
    {{ end }}{{ end }}{{ end }}

    {{ define "nm.default.text" }}{{ template "nm.default.subject" . }}
    {{ if gt (len .Alerts.Firing) 0 -}}
    Alerts Firing:
    {{ template "__nm_alert_list" .Alerts.Firing }}
    {{- end }}
    {{ if gt (len .Alerts.Resolved) 0 -}}
    Alerts Resolved:
    {{ template "__nm_alert_list" .Alerts.Resolved }}
    {{- end }}
    {{- end }}

    {{ define "__nm_alert_list_markdown" }}{{ range . }}### Labels:
    {{ range .Labels.SortedPairs }}{{ if ne .Name "runbook_url" }}- {{ .Name }} = {{ .Value }}{{ end }}
    {{ end }}### Annotations:
    {{ range .Annotations.SortedPairs }}{{ if ne .Name "runbook_url"}}- {{ .Name }} = {{ .Value }}{{ end }}
    {{ end }}{{ end }}{{ end }}

    {{ define "nm.default.markdown" }}# {{ template "nm.default.subject" . }}
    {{ if gt (len .Alerts.Firing) 0 -}}
    ## <font color=#CD0000>Alerts Firing:</font>
    {{ template "__nm_alert_list_markdown" .Alerts.Firing }}
    {{- end }}
    {{ if gt (len .Alerts.Resolved) 0 -}}
    ## <font color=#00CD00>Alerts Resolved:</font>
    {{ template "__nm_alert_list_markdown" .Alerts.Resolved }}
    {{- end }}
    {{- end }}

    {{ define "nm.default.html" }}
      <html xmlns="http://www.w3.org/1999/xhtml" xmlns="http://www.w3.org/1999/xhtml" style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;">
      <head style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;">
        <meta name="viewport" content="width=device-width" style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;" />
        <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;" />
        <title style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;">{{ template "__subject" . }}</title>
      </head>
      <body itemscope="" itemtype="http://schema.org/EmailMessage" style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; -webkit-font-smoothing: antialiased; -webkit-text-size-adjust: none; height: 100%; line-height: 1.6em; width: 100% !important; background-color: #f6f6f6; margin: 0; padding: 0;" bgcolor="#f6f6f6">
      <table style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; width: 100%; background-color: #f6f6f6; margin: 0;" bgcolor="#f6f6f6">
        <tr style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;">
          <td style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; vertical-align: top; margin: 0;" valign="top"></td>
          <td width="600" style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; vertical-align: top; display: block !important; max-width: 600px !important; clear: both !important; width: 100% !important; margin: 0 auto; padding: 0;" valign="top">
            <div style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; max-width: 600px; display: block; margin: 0 auto; padding: 0;">
              <table width="100%" cellpadding="0" cellspacing="0" style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; border-radius: 3px; background-color: #fff; margin: 0; border: 1px solid #e9e9e9;" bgcolor="#fff">
                <tr style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;">
                  <td style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 16px; vertical-align: top; color: #fff; font-weight: 500; text-align: center; border-radius: 3px 3px 0 0; background-color: #E6522C; margin: 0; padding: 20px;" align="center" bgcolor="#E6522C" valign="top">
                    {{ .Alerts | len }} alert{{ if gt (len .Alerts) 1 }}s{{ end }} for {{ range .GroupLabels.SortedPairs }}
                      {{ .Name }}={{ .Value }}
                    {{ end }}
                  </td>
                </tr>
                <tr style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;">
                  <td style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; vertical-align: top; margin: 0; padding: 10px;" valign="top">
                    <table width="100%" cellpadding="0" cellspacing="0" style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;">
                      {{ if gt (len .Alerts.Firing) 0 }}
                        <tr style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;">
                          <td style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; vertical-align: top; margin: 0; padding: 0 0 20px;" valign="top">
                            <strong style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;">[{{ .Alerts.Firing | len }}] Firing</strong>
                         </td>
                        </tr>
                      {{ end }}
                      {{ range .Alerts.Firing }}
                        <tr style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;">
                          <td style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; vertical-align: top; margin: 0; padding: 0 0 20px;" valign="top">
                            <strong style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;">Labels</strong><br style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;" />
                            {{ range .Labels.SortedPairs }}{{ if ne .Name "runbook_url" }}{{ .Name }} = {{ .Value }}{{ end }}<br style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;" />{{ end }}
                            {{ if gt (len .Annotations) 0 }}<strong style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;">Annotations</strong><br style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;" />{{ end }}
                            {{ range .Annotations.SortedPairs }}{{ if ne .Name "runbook_url" }}{{ .Name }} = {{ .Value }}{{ end }}<br style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;" />{{ end }}
                          </td>
                        </tr>
                      {{ end }}
                      {{ if gt (len .Alerts.Resolved) 0 }}
                        {{ if gt (len .Alerts.Firing) 0 }}
                          <tr style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;">
                            <td style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; vertical-align: top; margin: 0; padding: 0 0 20px;" valign="top">
                              <br style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;" />
                              <hr style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;" />
                              <br style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;" />
                            </td>
                          </tr>
                        {{ end }}
                        <tr style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;">
                          <td style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; vertical-align: top; margin: 0; padding: 0 0 20px;" valign="top">
                            <strong style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;">[{{ .Alerts.Resolved | len }}] Resolved</strong>
                          </td>
                        </tr>
                      {{ end }}
                      {{ range .Alerts.Resolved }}
                        <tr style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;">
                          <td style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; vertical-align: top; margin: 0; padding: 0 0 20px;" valign="top">
                            <strong style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;">Labels</strong><br style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;" />
                            {{ range .Labels.SortedPairs }}{{ if ne .Name "runbook_url" }}{{ .Name }} = {{ .Value }}{{ end }}<br style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;" />{{ end }}
                            {{ if gt (len .Annotations) 0 }}<strong style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;">Annotations</strong><br style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;" />{{ end }}
                            {{ range .Annotations.SortedPairs }}{{ if ne .Name "runbook_url" }}{{ .Name }} = {{ .Value }}{{ end }}<br style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;" />{{ end }}
                           </td>
                         </tr>
                       {{ end }}
                     </table>
                   </td>
                 </tr>
               </table>
            </div>
          </td>
          <td style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; vertical-align: top; margin: 0;" valign="top"></td>
        </tr>
      </table>
      </body>
      </html>
    {{ end }}

notification-manager的告警数据结构参考这里：
https://github.com/kubesphere/notification-manager/blob/master/docs/template.md#Multilingual-support

file

创建configmap

这里必须使用nm.default.text或者nm.default.markdown，代码里写死了。

apiVersion: v1
data:
  template: |2   
    {{ define "nm.default.text" }}
    {{- if gt (len .Alerts.Firing) 0 -}}
    {{- range $index, $alert := .Alerts -}}

    =========  监控告警 =========
    告警名称: {{ $alert.Labels.alertname }}
    告警级别: {{ $alert.Labels.severity }} 级
    故障主机: {{ $alert.Labels.instance }} {{ $alert.Labels.device }}
    集群名称: {{ $alert.Labels.cluster }}
    命名空间: {{ $alert.Labels.namespace }}
    Pod名称: {{ $alert.Labels.pod }}
    告警主题: {{ .Annotations.summary }} {{ .Annotations.summaryCn }}
    告警详情: {{ $alert.Annotations.message }}{{ $alert.Annotations.description}}
    主机标签: {{ range .Labels.SortedPairs  }}  [{{ .Name }}: {{ .Value  | html }} ] {{- end }} 
    故障时间: {{ ($alert.StartsAt.Add 28800e9).Format "2023-01-29 12:00:00" }}
    ========= = end =  =========
    {{- end }}
    {{- end }}

    {{- if gt (len .Alerts.Resolved) 0 -}}
    {{- range $index, $alert := .Alerts -}}

    ========= 告警恢复 =========
    告警主题: {{ .Annotations.summary }} {{ .Annotations.summaryCn }}
    告警主机: {{ .Labels.instance }}
    告警类型: {{ .Labels.alertname }}
    告警级别: {{ $alert.Labels.severity }} 级
    集群名称: {{ $alert.Labels.cluster }}
    命名空间: {{ $alert.Labels.namespace }}
    Pod名称: {{ $alert.Labels.pod }}
    告警主题: {{ .Annotations.summary }} {{ .Annotations.summaryCn }}
    故障时间: {{ ($alert.StartsAt.Add 28800e9).Format "2023-01-29 12:00:00" }}
    恢复时间: {{ ($alert.EndsAt.Add 28800e9).Format "2023-01-29 12:00:00" }}
    ========= = end =  =========
    {{- end }}
    {{- end }}
    {{- end }}

kind: ConfigMap
metadata:
  annotations:
    meta.helm.sh/release-name: notification-manager
    meta.helm.sh/release-namespace: kubesphere-monitoring-system
  labels:
    app.kubernetes.io/managed-by: Helm
  name: notification-templ
  namespace: kubesphere-monitoring-system

file

修改notification-manager

notification-manager创建了3个crd：configs，notificationmanagers，receivers。

file

直接修改notificationmanagers的cm name即可。

k edit nm

file

notification-manager的pod会自动重启。

查看告警

=========  监控告警 =========
告警名称: ContainerUnhealthy
告警级别: warning 级
故障主机:  
集群名称: default
命名空间: default
Pod名称: nginx-558877494d-hfrvp
告警主题: Container is unhealthy 容器状态不良
告警详情: Readiness probe failed: Get "http://10.42.3.115:80/health": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
主机标签:   [alertname: ContainerUnhealthy ]  [alerttype: event ]  [cluster: default ]  [container: nginx ]  [namespace: default ]  [pod: nginx-558877494d-hfrvp ]  [severity: warning ] 
故障时间: 2023-01-29 12:26:40
========= = end =  =========
5 1 投票
文章评分
订阅评论
提醒
guest

3 评论
最旧
最新 最多投票
内联反馈
查看所有评论
沉踩踩踩
沉踩踩踩
1 年 前

你好,alertmanager-main-0 是怎么配置企业微信应用告警的

0
回复
wghdr
作者
wghdr
1 年 前
回复给  沉踩踩踩

有notification-manager服务后，平台管理–平台设置–通知配置–企业微信，填写你的企业id，应用 AgentId，应用 Secret，设置告警条件，指定接收人即可。

0
回复
沉踩踩踩
沉踩踩踩
1 年 前

但是上述配置之后其他通知渠道都用不了了呀,,其它模板根据哪里可以进行修改

0
回复

相关文章

开始在上面输入您的搜索词，然后按回车进行搜索。按ESC取消。

返回顶部
1
0
希望看到您的想法，请您发表评论x