上次只记录了ca证书更换后的一些问题解决，这次又有一套k8s 1.14环境ca证书过期了，这里记录下更换ca证书的步骤。

生成证书

生成ca证书

首先查看原ca证书的具体信息，如CN，组织，加密方式。需要根据实际环境进行修改。

cfssl certinfo -cert ca.pem

根据原证书创建ca-config.json和ca-csr.json。

cat ca-csr.json
{
    "CN": "Kubernetes",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "Hangzhou",
            "ST": "Zhejiang",
            "O": "CA",
            "OU": "Wowjoy"
        }
    ]
}

cat ca-config.json
{
  "signing": {
    "default": {
      "expiry": "87600h"
    },
    "profiles": {
      "kubernetes": {
         "expiry": "87600h",
         "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ]
      }
    }
  }
}

生成ca证书。

cfssl gencert -initca ca-csr.json | cfssljson -bare ca -

生成etcd证书

和ca一样的步骤，查看信息，创建json文件，生成证书。

cat etcd-csr.json
{
    "CN": "etcd",
    "hosts": [
    "10.36.191.151",
    "10.36.191.152",
    "10.36.191.153",
    "127.0.0.1"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "Hangzhou",
            "ST": "Zhejiang",
            "O": "CA",
            "OU": "Wowjoy"
        }
    ]
}

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes etcd-csr.json | cfssljson -bare etcd

生成kube-apiserver证书

cat kubernetes-csr.json
{
    "CN": "kubernetes",
    "hosts": [
      "kubernetes",
      "kubernetes.default",
      "kubernetes.default.svc",
      "kubernetes.default.svc.cluster",
      "kubernetes.default.svc.cluster.local",
      "127.0.0.1",
      "10.36.191.151",
      "10.36.191.152",
      "10.36.191.153",
      "172.168.0.1"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "Hangzhou",
            "ST": "Zhejiang",
            "O": "Kubernetes",
            "OU": "Wowjoy"
        }
    ]
}

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kubernetes-csr.json | cfssljson -bare kubernetes

生成kube-controller-manger证书

cat kube-controller-manager-csr.json
{
  "CN": "system:kube-controller-manager",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "L": "Hangzhou",
      "ST": "Zhejiang",
      "O": "system:kube-controller-manager",
      "OU": "Wowjoy"
    }
  ]
}

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-controller-manager-csr.json | cfssljson -bare kube-controller-manager

生成kube-scheduler证书

cat kube-scheduler-csr.json
{
  "CN": "system:kube-scheduler",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "L": "Hangzhou",
      "ST": "Zhejiang",
      "O": "system:kube-scheduler",
      "OU": "Wowjoy"
    }
  ]
}

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-scheduler-csr.json  | cfssljson -bare kube-scheduler

生成kube-proxy证书

cat kube-proxy-csr.json
{
  "CN": "system:kube-proxy",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "L": "Hangzhou",
      "ST": "Zhejiang",
      "O": "system:node-proxier",
      "OU": "Wowjoy"
    }
  ]
}

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy

生成service-account证书

cat service-account-csr.json
{
  "CN": "service-accounts",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "L": "Hangzhou",
      "ST": "Zhejiang",
      "O": "Kubernetes",
      "OU": "Wowjoy"
    }
  ]
}

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes service-account-csr.json | cfssljson -bare service-account

生成admin证书

cat admin-csr.json
{
  "CN": "admin",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "L": "Hangzhou",
      "ST": "Zhejiang",
      "O": "system:masters",
      "OU": "Wowjoy"
    }
  ]
}

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes admin-csr.json | cfssljson -bare admin

生成的证书如下。

生成kubeconfig文件

首先查看原kubeconfig配置文件中的user字段，脚本中需要做对应修改。

admin

cat admin.sh
#! /bin/bash
kubectl config set-cluster kubernetes \
    --certificate-authority=ca.pem \
    --embed-certs=true \
    --server=https://127.0.0.1:6443 \
    --kubeconfig=admin.kubeconfig

  kubectl config set-credentials admin \
    --client-certificate=admin.pem \
    --client-key=admin-key.pem \
    --embed-certs=true \
    --kubeconfig=admin.kubeconfig

  kubectl config set-context default \
    --cluster=kubernetes \
    --user=admin \
    --kubeconfig=admin.kubeconfig

  kubectl config use-context default --kubeconfig=admin.kubeconfig

kube-controller-manager

cat kube-controller-manager.sh
#! /bin/bash
kubectl config set-cluster kubernetes \
    --certificate-authority=ca.pem \
    --embed-certs=true \
    --server=https://127.0.0.1:6443 \
    --kubeconfig=kube-controller-manager.kubeconfig

  kubectl config set-credentials system:kube-controller-manager \
    --client-certificate=kube-controller-manager.pem \
    --client-key=kube-controller-manager-key.pem \
    --embed-certs=true \
    --kubeconfig=kube-controller-manager.kubeconfig

  kubectl config set-context default \
    --cluster=kubernetes \
    --user=system:kube-controller-manager \
    --kubeconfig=kube-controller-manager.kubeconfig

  kubectl config use-context default --kubeconfig=kube-controller-manager.kubeconfig

kube-scheduler

cat kube-scheduler.sh
#! /bin/bash
 kubectl config set-cluster kubernetes \
    --certificate-authority=ca.pem \
    --embed-certs=true \
    --server=https://127.0.0.1:6443 \
    --kubeconfig=kube-scheduler.kubeconfig

  kubectl config set-credentials system:kube-scheduler \
    --client-certificate=kube-scheduler.pem \
    --client-key=kube-scheduler-key.pem \
    --embed-certs=true \
    --kubeconfig=kube-scheduler.kubeconfig

  kubectl config set-context default \
    --cluster=kubernetes \
    --user=system:kube-scheduler \
    --kubeconfig=kube-scheduler.kubeconfig

  kubectl config use-context default --kubeconfig=kube-scheduler.kubeconfig

kube-proxy

#! /bin/bash
kubectl config set-cluster kubernetes \
--certificate-authority=/srv/newcert/ca.pem \
--embed-certs=true \
--server=${KUBE_APISERVER} \
--kubeconfig=/srv/newcert/kube-proxy.kubeconfig
kubectl config set-credentials kube-proxy \
--client-certificate=kube-proxy.pem \
--client-key=kube-proxy-key.pem \
--embed-certs=true \
--kubeconfig=/srv/newcert/kube-proxy.kubeconfig
kubectl config set-context default \
--cluster=kubernetes \
--user=kube-proxy \
--kubeconfig=/srv/newcert/kube-proxy.kubeconfig
kubectl config use-context default --kubeconfig=/srv/newcert/kube-proxy.kubeconfig

bootstrap

#!/bin/bash
KUBE_APISERVER=https://127.0.0.1:8443
BOOTSTRAP_TOKEN="121b223e3229419162e00c9dc29d936a"

## 生成bootstrap kubeconfig
kubectl config set-cluster kubernetes \
--certificate-authority=/srv/newcert/ca.pem \
--embed-certs=true \
--server=${KUBE_APISERVER} \
--kubeconfig=/srv/newcert/bootstrap.kubeconfig
kubectl config set-credentials kubelet-bootstrap \
--token=${BOOTSTRAP_TOKEN} \
--kubeconfig=/srv/newcert/bootstrap.kubeconfig
kubectl config set-context default \
--cluster=kubernetes \
--user=kubelet-bootstrap \
--kubeconfig=/srv/newcert/bootstrap.kubeconfig
kubectl config use-context default --kubeconfig=/srv/newcert/bootstrap.kubeconfig

10.conf

拷贝原来的10.conf，然后修改其中的certificate-authority-data字段，可以从bootstrap.kubeconfig中复制。

替换证书

master

先备份原证书目录。然后将新证书和kubeconfig文件拷贝到原证书目录。

cp -a /srv/kubernetes /root
cp -a admin-key.pem admin.pem ca-key.pem ca.pem etcd-key.pem etcd.pem kube-controller-manager-key.pem kube-controller-manager.kubeconfig kube-controller-manager.pem kubernetes-key.pem kubernetes.pem kube-scheduler-key.pem kube-scheduler.kubeconfig kube-scheduler.pem service-account-key.pem service-account.pem /srv/kubernetes/
其他master
scp -r admin-key.pem admin.pem ca-key.pem ca.pem etcd-key.pem etcd.pem kube-controller-manager-key.pem kube-controller-manager.kubeconfig kube-controller-manager.pem kubernetes-key.pem kubernetes.pem kube-scheduler-key.pem kube-scheduler.kubeconfig kube-scheduler.pem service-account-key.pem service-account.pem master2:/srv/kubernetes/
scp -r admin-key.pem admin.pem ca-key.pem ca.pem etcd-key.pem etcd.pem kube-controller-manager-key.pem kube-controller-manager.kubeconfig kube-controller-manager.pem kubernetes-key.pem kubernetes.pem kube-scheduler-key.pem kube-scheduler.kubeconfig kube-scheduler.pem service-account-key.pem service-account.pem master3:/srv/kubernetes/

重启etcd,kube-apiserver,kube-controller-manager,kube-scheduler
ansible master -m shell -a "systemctl daemon-reload"
ansible master -m shell -a "systemctl restart etcd"
ansible master -m shell -a "systemctl restart kube-apiserver"
ansible master -m shell -a "systemctl restart kube-controller-manager"
ansible master -m shell -a "systemctl restart kube-scheduler"

node

在其中一台node中备份证书，其他节点证书都一样的。然后将新证书和kubeconfig文件拷贝到原证书目录。

cp -a /srv/kubernetes /root
for i in `cat node.txt`; do scp -r ca.pem kube-proxy-key.pem  kube-proxy.pem bootstrap.kubeconfig ca-key.pem kube-proxy.kubeconfig 10.conf    $i:/srv/kubernetes/;done
ansible node -m shell -a "systemctl daemon-reload && systemctl restart kubelet && systemctl restart kube-proxy"

后续步骤就是node节点重新生成kubelet-client-current.pem和kubelet-server-current.pem证书，批准节点的csr，再删除coredns，calico等sa对应的secret重启pod，恢复正常。