使用GitOps的思想来进行CICD。
安装tekton
官方文档:https://tekton.dev/docs/installation/, 注意k8s版本需要在1.28以上。
kubectl apply --filename https://storage.googleapis.com/tekton-releases/pipeline/latest/release.yaml
kubectl apply --filename \
https://storage.googleapis.com/tekton-releases/triggers/latest/release.yaml
kubectl apply --filename \
https://storage.googleapis.com/tekton-releases/triggers/latest/interceptors.yaml
apply之后会拉不下来镜像,这里使用Docker Playground来拉取。
首先创建实例,然后点击EDITOR
弹出来个页面,在里面粘贴镜像id,再点击SAVE。
当前目录下会生成一个名为root的文件。
回到命令行,登录dockerhub,拉取镜像并推送。
docker logout
docker login -u user
docker pull $(cat root)
docker images
docker tag && docker push
如果用latest版本可以直接使用我的镜像:
namespace: tekton-pipelines-resolvers
tekton-pipelines-remote-resolvers: wgh9626/tekton-pipeline-resolvers:latest
namespace: tekton-pipelines
tekton-events-controller: wgh9626/tekton-pipeline-events:latest
tekton-pipelines-controller: wgh9626/tekton-pipeline-controller:latest
tekton-pipelines-webhook: wgh9626/tekton-pipeline-webhook:latest
tekton-triggers-controller: wgh9626/tekton-triggers-controller:latest
tekton-triggers-webhook: wgh9626/tekton-triggers-webhook:latest
tekton-triggers-core-interceptors: wgh9626/tekton-triggers-core-interceptors:latest
安装dashboard
kubectl apply --filename https://storage.googleapis.com/tekton-releases/dashboard/latest/release.yaml
# 修改为NodePort
k edit svc -n tekton-pipelines tekton-dashboard
安装Argo-cd
安装步骤参考:https://wghdr.top/?p=4402
开启TemplateNoDefaults
默认安装的OpenKruise会进行pod/pvc template的默认值注入,这个行为会跟argo-cd的sync判断逻辑冲突,所以在安装OpenKruise需要打开Gates TemplateNoDefaults
helm upgrade kruise openkruise/kruise --set featureGates="TemplateNoDefaults=true"
# or
helm upgrade kruise openkruise/kruise --set featureGates="AllAlpha=true"
配置Argo-cd CloneSet Health Check
Argo-cd根据该配置能够实现CloneSet自定义资源的检查,如CloneSet是否发布完成,以及Pod是否ready等。官方文档:https://argo-cd.readthedocs.io/en/stable/operator-manual/health/#custom-health-checks
具体做法就是创建一个ConfigMap,数据内容为对CloneSet的自定义健康检查lua脚本。
apiVersion: v1
kind: ConfigMap
metadata:
labels:
app.kubernetes.io/name: argocd-cm
app.kubernetes.io/part-of: argocd
name: argocd-cm
namespace: argocd
data:
resource.customizations.health.apps.kruise.io_CloneSet: |
hs = {}
-- if paused
if obj.spec.updateStrategy.paused then
hs.status = "Suspended"
hs.message = "CloneSet is Suspended"
return hs
end
-- check cloneSet status
if obj.status ~= nil then
if obj.status.observedGeneration < obj.metadata.generation then
hs.status = "Progressing"
hs.message = "Waiting for rollout to finish: observed cloneSet generation less then desired generation"
return hs
end
if obj.status.updatedReplicas < obj.spec.replicas then
hs.status = "Progressing"
hs.message = "Waiting for rollout to finish: replicas hasn't finished updating..."
return hs
end
if obj.status.updatedReadyReplicas < obj.status.updatedReplicas then
hs.status = "Progressing"
hs.message = "Waiting for rollout to finish: replicas hasn't finished updating..."
return hs
end
hs.status = "Healthy"
return hs
end
-- if status == nil
hs.status = "Progressing"
hs.message = "Waiting for cloneSet"
return hs
脚本解释
- 首先初始化,hs 是一个表,用来存储健康状态信息。hs.status: 表示健康状态,hs.message: 表示健康状态信息。
- 如果 CloneSet 的更新策略被暂停(paused = true),返回Suspended状态和message信息。
- 检查 CloneSet 的状态,如果status不为空,作进一步检查。
- 判断 observedGeneration 是否落后于期望值,如果小于表示 CloneSet 尚未完成最新配置的应用。返回Progressing状态和message信息。
- 如果更新的副本数(updatedReplicas)少于期望的副本数(replicas),表示更新尚未完成。返回Progressing状态和message信息。
- 如果已更新的就绪副本数(updatedReadyReplicas)少于更新的副本数(updatedReplicas),说明一些更新的副本尚未完全就绪。返回Progressing状态和message信息。
- 如果以上检查都通过,则认为 CloneSet 处于健康状态。返回Healthy状态。
- 如果状态为空,表示 CloneSet 还未完全初始化。返回Progressing状态和message信息。
k apply -f health-check.yaml
创建流水线
创建docker registry密钥
用于docker push image。
k create secret docker-registry dockersecret --docker-server=docker.io --docker-username=user --docker-password=password --docker-email=123@qq.com
k get secrets dockersecret -o yaml
创建推送镜像task
使用官方的helloworld demo,包含了单元测试,Dockerfile和源码,helloworld的chart包。https://github.com/zmberg/samples/tree/hello_world/helloworld
apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
labels:
app: helloworld
name: helloworld-build-push
spec:
stepTemplate:
workingDir: /workspace
params:
- name: gitrepositoryurl
type: string
- name: branch
type: string
- name: docker_repo
type: string
- name: app_name
type: string
steps:
# git clone
- name: git-clone-and-checkout
image: bitnami/git:latest
command: ["sh", "-ce"]
args:
- >
set -e
echo $(params.gitrepositoryurl)
git clone $(params.gitrepositoryurl) ./ && git checkout $(params.branch)
# unit test
- name: auto-test
image: golang:1.16
command: [ "sh", "-ce" ]
args:
- >
set -e
cp -R /workspace/$(params.app_name) /go/src/ && cd /go/src/$(params.app_name) && pwd;
go test
# docker build & push registry
- name: push-to-registry
image: wgh9626/kaniko-executor:latest
args:
- --dockerfile=Dockerfile
- --destination=$(params.docker_repo):$(params.branch)
- --context=./$(params.app_name)
- --cache=true
- --cache-dir=/cache
- --use-new-run
volumeMounts:
- name: kaniko-secret
mountPath: "/kaniko/.docker"
volumes:
# docker push secret
- name: kaniko-secret
secret:
secretName: dockersecret
items:
- key: .dockerconfigjson
path: config.json
创建argocd密钥
# 注意server不能加http://
k create secret generic argosecret --from-literal=username=admin --from-literal=password=sobVnyv8KLrlyBAa --from-literal=server=1.2.3.4:32519
创建argocd部署task
apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
labels:
app: helloworld
name: helloworld-argocd
spec:
params:
- name: gitrepositoryurl
type: string
- name: branch
type: string
- name: docker_repo
type: string
- name: app_name
type: string
- name: app_ns
type: string
- name: k8s_server
type: string
steps:
- name: argocd-deploy
image: wgh9626/argocd:latest
command:
- sh
args:
- '-ce'
- >
set -e
echo "upgrade app $(params.app_name)"; username=`cat /var/secret/username`; password=`cat /var/secret/password`; server=`cat /var/secret/server`;
argocd login ${server} --insecure --username ${username} --password ${password}
argocd app create $(params.app_name) --upsert --repo $(params.gitrepositoryurl) --path $(params.app_name)/charts --dest-namespace $(params.app_ns) --dest-server $(params.k8s_server) --revision $(params.branch) --helm-set image.repository=$(params.docker_repo) --helm-set image.tag=$(params.branch) --helm-set installation.namespace=$(params.app_ns)
argocd app list; argocd app sync $(params.app_name)
argocd app wait $(params.app_name) --health
volumeMounts:
- name: argocd-secret
mountPath: "/var/secret"
volumes:
- name: argocd-secret
secret:
secretName: argosecret
创建Pipeline
定义tekton pipeline,总共由上面两个task任务组成。并且执行完 构建-测试-推送镜像Task之后,再执行argocd部署。
apiVersion: tekton.dev/v1beta1
kind: Pipeline
metadata:
name: helloworld-pipeline
spec:
params:
- name: gitrepositoryurl
type: string
- name: branch
type: string
- name: docker_repo
type: string
- name: app_name
type: string
- name: app_ns
type: string
- name: k8s_server
type: string
tasks:
- name: helloworld-build-push
taskRef:
name: helloworld-build-push
params:
- name: gitrepositoryurl
value: $(params.gitrepositoryurl)
- name: branch
value: $(params.branch)
- name: docker_repo
value: $(params.docker_repo)
- name: app_name
value: $(params.app_name)
- name: helloworld-argocd
taskRef:
name: helloworld-argocd
params:
- name: gitrepositoryurl
value: $(params.gitrepositoryurl)
- name: branch
value: $(params.branch)
- name: docker_repo
value: $(params.docker_repo)
- name: app_name
value: $(params.app_name)
- name: app_ns
value: $(params.app_ns)
- name: k8s_server
value: $(params.k8s_server)
runAfter:
- helloworld-build-push
执行Tekton Pipeline
我这里从gitHub转到gitee了,防止无法访问github。需要对应修改Git地址和分支。
apiVersion: tekton.dev/v1beta1
kind: PipelineRun
metadata:
name: helloworld-pipeline-run-1
spec:
pipelineRef:
name: helloworld-pipeline
params:
- name: gitrepositoryurl
value: https://gitee.com/wghdrr/samples.git
- name: branch
value: master
- name: docker_repo
value: wgh9626/kruise
- name: app_name
value: helloworld
- name: app_ns
value: helloworld
- name: k8s_server
value: https://kubernetes.default.svc
执行失败了,因为连不上docker.io...
手动下载镜像,然后修改Task的镜像拉取策略,可以看到默认是Always,修改为IfNotPresent或Never。https://tekton.dev/docs/pipelines/pipeline-api/
再次执行pipiline。还是失败的,仍然连接了docker.io,把镜像换成阿里云。
registry.cn-hangzhou.aliyuncs.com/wgh9626/git:latest
registry.cn-hangzhou.aliyuncs.com/wgh9626/golang:1.16
registry.cn-hangzhou.aliyuncs.com/wgh9626/kaniko-executor:latest
# 修改Dockerfile中的镜像
registry.cn-hangzhou.aliyuncs.com/wgh9626/static:nonroot
再次执行,这次没有连接docker.io了,pod创建成功,两个task都执行成功了。
查看镜像:
查看部署日志:
查看部署情况:
我这里机器资源不足,所以注释了values.yaml
中的resource。
部署中会出现探针检测失败的情况,在cloneset.yaml
会有三个。中注释掉探针检测。所以
ControllerRevision
在argocd中点击sync即可。