使用prometheus监控etcd

测试etcd metrics

我的环境：master ip是192.168.5.100-102，根据自己的etcd节点ip进行修改。

# 查看etcd配置文件中的证书文件位置
egrep "key-file|cert-file" /etc/kubernetes/manifests/etcd.yaml
for ((i=100;i<103;i++));do curl -s --cert /etc/kubernetes/pki/etcd/server.crt --key /etc/kubernetes/pki/etcd/server.key  https://192.168.5.$i:2379/metrics -k | tail -1;done

创建etcd svc

cat etcd-svc.yaml

apiVersion: v1
kind: Service
metadata:
  name: prometheus-kube-etcd
  labels:
    app: prometheus-kube-etcd
    jobLabel: kube-etcd
    release: prometheus
spec:
  type: ClusterIP
  ports:
  - name: https-metrics
    port: 2379
    protocol: TCP
    targetPort: 2379
---
apiVersion: v1
kind: Endpoints
metadata:
  name: prometheus-kube-etcd
  labels:
    app: prometheus-kube-etcd
    jobLabel: kube-etcd
    release: prometheus
subsets:
- addresses:
  - ip: 192.168.5.100
  - ip: 192.168.5.101
  - ip: 192.168.5.102
  ports:
  - name: https-metrics
    port: 2379
    protocol: TCP

kubectl apply -f etcd-svc.yaml -n kube-system
kubectl get svc,ep -n kube-system | grep etcd

测试使用svc访问

for ((i=100;i<103;i++));do curl -s --cert /etc/kubernetes/pki/etcd/server.crt --key /etc/kubernetes/pki/etcd/server.key  https://10.42.10.4:2379/metrics -k | tail -1;done

创建etcd secret

kubectl -n cattle-monitoring-system create secret generic etcd-certs --from-file=/etc/kubernetes/pki/etcd/server.key  --from-file=/etc/kubernetes/pki/etcd/server.crt --from-file=/etc/kubernetes/pki/etcd/ca.crt
kubectl get secret -n cattle-monitoring-system

挂载secret到prometheus

我的prometheus是部署在cattle-monitoring-system中的

# 添加secrets字段
kubectl edit prometheus -n cattle-monitoring-system
kubectl get prometheus -n cattle-monitoring-system -o yaml

保存后prometheus的pod会自动重启

查看是否挂载成功

kubectl exec -it prometheus-rancher-monitoring-prometheus-0 -n cattle-monitoring-system -- ls /etc/prometheus/secrets/etcd-certs

创建etcd ServiceMonitor

cat servicemonitor.yaml

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    app: prometheus-operator-kube-etcd
    release: rancher-monitoring
  name: prometheus-operator-kube-etcd
  namespace: cattle-monitoring-system
spec:
  endpoints:
  - port: https-metrics
    scheme: https
    tlsConfig:
      caFile: /etc/prometheus/secrets/etcd-certs/ca.crt
      certFile: /etc/prometheus/secrets/etcd-certs/server.crt
      insecureSkipVerify: true
      keyFile: /etc/prometheus/secrets/etcd-certs/server.key
  namespaceSelector:
    matchNames:
    - kube-system
  selector:
  # 这里要匹配etcd的label
    matchLabels:
      app: prometheus-kube-etcd