报错
线上收到告警
查看eureka控制台有如下信息:EMERGENCY! EUREKA MAY BE INCORRECTLY CLAIMING INSTANCES ARE UP WHEN THEY'RE NOT. RENEWALS ARE LESSER THAN THRESHOLD AND HENCE THE INSTANCES ARE NOT BEING EXPIRED JUST TO BE SAFE.
Renews threshold是425,Renews (last min)是404。
查看服务状况是正常的。报错的pod是历史pod。
综上需要清理eureka脏数据。
解决
使用下面的清理脚本
cat delete.sh
#!/bin/bash
for i in `cat $PWD/svc`
do
Name=`echo $i | awk -F"[-]" '{print $1,$2}' | sed -e 's# #\-#g'`
Names=`echo $Name | tr 'a-z' 'A-Z'`
echo $Names
curl -X DELETE http://127.0.0.1:7001/eureka/apps/${Names}/${i}
done
touch svc
k cp delete.sh eureka-0:/retail-eureka
k cp svc eureka-0:/retail-eureka
k exec -it eureka-0 bash
# 把状态为DOWN的服务名粘贴到svc文件中。
vi svc
hcloud-cardmanager-f87d685c9-5d69b:hcloud-cardManager:7773
hcloud-cardmanager-f87d68Sc9-bd2qw:hcloud-cardManager:7773
sh delete.sh
返回eureka ui刷新,down的节点以及删除掉了。
注意:如果提取到的Name节点名字和Application不一致,执行脚本会失败,这时需要手动执行删除,
curl -X DELETE http://127.0.0.1:7001/eureka/apps/${Names}/${i}