背景

本地环境使用的vsphere，当k8s集群资源不足，pod无法创建，处于Pending状态时，要么清理一部分资源，要么扩容节点。扩容又分为横向和纵向扩容。横向增加节点数，纵向增加机器配置。不管是横向还是纵向都得手动去配置虚拟机，然后再把虚拟机加入到集群中。有没有一种方法可以实现自动扩容并添加到集群中呢？

答案是有的，但只能使用增加节点的方法，因为如果虚拟机没有开启cpu/内存热插拔的功能，扩容资源是需要重启虚拟机的，有些系统也不支持热拔插，而且开了还会占用虚拟化资源，这个在线上是不现实的。

实现

1.部署测试nginx服务，构造pending状态pod

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:stable-alpine
        resources:
          requests:
            memory: 16G
---
apiVersion: v1
kind: Service
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  type: CLusterIP
  ports:
  - port: 80
    protocol: TCP
    name: http
  selector:
    app: nginx

2.判断集群中是否有Pending状态且时间大于30m的pod

这是用30m来判断集群是否真的没有资源了。下面是shell脚本示例：

cat autoscale.sh

#!/bin/bash
pods=$(kubectl get pods --all-namespaces --field-selector=status.phase=Pending -o json | jq '.items[] | select(.status.conditions[] | select(.type=="PodScheduled" and .status=="False")) | select(((now - (.metadata.creationTimestamp | fromdate)) / 60) > 30) | .metadata.namespace + "/" + .metadata.name' | cut -d "\"" -f 2)

if [ -z "$pods" ]; then
    echo "集群中没有处于Pending状态的pod,无需扩容"
    exit 0
else
    need_scaling=false
    for pod in $pods; do
        namespace=$(echo "$pod" | cut -d "/" -f 1)
        pod_name=$(echo "$pod" | cut -d "/" -f 2)
        message=$(kubectl get pod -n "$namespace" "$pod_name" -o jsonpath='{.status.conditions[].message}')

        if [[ -n $message ]]; then
            # echo "$message"
            if [[ $message == *"Insufficient cpu"* ]] || [[ $message == *"Insufficient memory"* ]]; then
                need_scaling=true
                break
            fi
        fi
    done

    if $need_scaling; then
        echo "集群CPU或内存不足,需要扩容"
        echo "下面执行扩容脚本addnode.sh"
        bash /root/addnode.sh
        if [ $? -eq 0 ]; then
            echo "扩容成功"
        else
            echo "扩容失败"
        fi
    fi
fi

3.扫描并输出网段中一个可用的ip

扫描并输出172.16.255.200-172.16.255.250网段中第一个可用的ip。下面是shell脚本示例：

cat genip.sh

#!/bin/bash
IPADDR=false
IPV4="172.16.255"

while [[ $IPADDR = false ]]; do
    IPADDR=false
    # read -p "Please enter an ipv4 address segment, like [192.168.10]: " IPV4
    if [[ $(echo "$IPV4" | awk -F. '{ print NF }') -eq 3 ]]; then
        for i in $(echo "$IPV4" | awk -F. '{print $1,$2,$3}'); do
            if [[ $i =~ ^[1-9][0-9]{0,2}$ && $i -ge 1 && $i -le 255 ]]; then
                IPADDR=true
            else
                echo -e "Input ipv4 entered is wrong, please enter the correct ipv4 segment!"
                IPADDR=false
                break
            fi
        done
    else
        echo -e "Input ipv4 entered is wrong, please enter the correct ipv4 segment!"
    fi
done

FIRST=false
SECOND=false
START=200
END=250

while [[ $FIRST = false || $SECOND = false || $START -gt $END ]]; do
    # read -p "Please enter the starting IP address to scan (for example: 1): " START
    if [[ $START -ge 1 && $START -le 254 && $START =~ ^[1-9][0-9]{0,2}$ ]]; then
        FIRST=true
    else
        echo -e "Input $START entered is wrong, please enter the correct starting IP address!"
        FIRST=false
        continue
    fi

    # read -p "Please enter the ending IP address to scan (for example: 255): " END
    if [[ $END -ge 1 && $END -le 254 && $END =~ ^[1-9][0-9]{0,2}$ ]]; then
        SECOND=true
    else
        echo -e "Input $END entered is wrong, please enter the correct ending IP address!"
        SECOND=false
        continue
    fi
done

# echo "Scanning..."

for ((i = START; i <= END; i++)); do
    (
        ping "$IPV4".$i -c 1 -w 1 &>/dev/null
        if [ $? -eq 0 ]; then
            echo "$IPV4".$i >>used_ip.text
        else
            echo "$IPV4".$i >>unused_ip.text
        fi
    ) &
done
wait

if [ -f unused_ip.text ]; then
    ip=$(sort -t'.' -k 4n unused_ip.text | head -n 1)
    echo "$ip"
    rm -f used_ip.text
    rm -f unused_ip.text
else
    echo ""
fi

4.执行terraform添加虚拟机

上篇文章中介绍了如何通过vsphere provider来添加虚拟机。其中terraform.tfvars文件的vsphere_ipv4_address需要配置为172.16.255.200，方便脚本修改。

下面是terraform函数的示例：

node_ip=$(bash /root/genip.sh)
echo "虚拟机的ip地址是$node_ip"
tfvar="terraform.tfvars"
tfvar_path="/root/terraform/vsphere/$tfvar"

function terraform() {
    sed -i "s/172.16.255.200/$node_ip/g" $tfvar_path
    cd /root/terraform/vsphere/ || exit
    /usr/bin/terraform init
    if [ $? -eq 0 ]; then
        echo "terraform init success"
    else
        echo "terraform init failed"
        exit 1
    fi

    /usr/bin/terraform apply -auto-approve
    if [ $? -eq 0 ]; then
        echo "terraform apply success"
    else
        echo "terraform apply failed"
        exit 1
    fi
}