Wghdr's Blog!

k8s, 中间件

k8s skywalking 9.3.0部署

wghdr

环境

环境中已经部署了EFK，namespace是logging，这里直接使用EFK中的ES，就没有再单独部署。

部署

可以使用yaml来部署，也可以添加helm仓库，使用helm来部署。这里使用的是yaml方式。

创建namespace

kubectl create namespace skywalking

创建rbac

apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    app: skywalking-oap
  name: skywalking-oap
  namespace: skywalking
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  namespace: skywalking
  name: skywalking-oap
  labels:
    app: skywalking-oap
rules:
  - apiGroups: [""]
    resources: ["pods","configmaps"]
    verbs: ["get", "watch", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  namespace: skywalking
  name: skywalking-oap
  labels:
    app: skywalking-oap
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: skywalking-oap
subjects:
  - kind: ServiceAccount
    name: skywalking-oap
    namespace: skywalking

创建es-svc

apiVersion: v1
kind: Service
metadata:
  name: "elasticsearch"
  namespace: skywalking
  labels:
    app: elasticsearch
spec:
  type: ExternalName
  externalName: "elasticsearch.logging.svc.cluster.local"

# 如果报错无法访问es可以改成
apiVersion: v1
kind: Service
metadata:
  labels:
    app: elasticsearch
  name: elasticsearch
  namespace: skywalking
spec:
  ports:
  - name: elasticsearch
    port: 9200
    protocol: TCP
    targetPort: 9200
  type: ClusterIP
---
apiVersion: v1
kind: Endpoints
metadata:
  name: elasticsearch
  namespace: skywalking
subsets:
- addresses:
  - ip: es的ip地址
  ports:
  - port: 9200
    protocol: TCP

部署oap-es-init-job

apiVersion: batch/v1
kind: Job
metadata:
  namespace: skywalking
  name: "skywalking-es-init"
  labels:
    app: skywalking-job
spec:
  template:
    metadata:
      name: "skywalking-es-init"
      labels:
        app: skywalking-job
    spec:
      restartPolicy: Never
      initContainers:
      - name: wait-for-elasticsearch
        image: busybox:latest
        imagePullPolicy: IfNotPresent
        # elasticsearch 地址
        command: ['sh', '-c', 'for i in $(seq 1 60); do nc -z -w3 elasticsearch或者es-ip 9200 && exit 0 || sleep 5; done; exit 1']
      containers:
      - name: oap
        image: apache/skywalking-oap-server:9.3.0
        imagePullPolicy: IfNotPresent
        env:
        - name: JAVA_OPTS
          value: "-Xmx2g -Xms2g -Dmode=init"
        - name: SW_STORAGE
          value: "elasticsearch"
        - name: SW_STORAGE_ES_CLUSTER_NODES
          value: "elasticsearch或者es-ip:9200"
        - name: SW_ES_USER
          value: "elastic"
        - name: SW_ES_PASSWORD
          value: "xxxxx"

es的地址可以写svc名，如果报错svc无法解析，也可以写ip地址。

部署oap-deployment

等待job执行完成后，部署deployment。

file

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: skywalking
    component: "oap"
  name: skywalking-oap
  namespace: skywalking
spec:
  replicas: 2
  selector:
    matchLabels:
      app: skywalking
      component: "oap"
  template:
    metadata:
      labels:
        app: skywalking
        component: "oap"
    spec:
      serviceAccountName: skywalking-oap
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 1
            podAffinityTerm:
              topologyKey: kubernetes.io/hostname
              labelSelector:
                matchLabels:
                  app: "skywalking"
                  component: "oap"
      initContainers:
      - name: wait-for-elasticsearch
        image: busybox:latest
        imagePullPolicy: IfNotPresent
        # elasticsearch 地址修改
        command: ['sh', '-c', 'for i in $(seq 1 60); do nc -z -w3 elasticsearch或者es-ip 9200 && exit 0 || sleep 5; done; exit 1']
      containers:
      - name: oap
        image: apache/skywalking-oap-server:9.3.0
        imagePullPolicy: IfNotPresent
        livenessProbe:
          tcpSocket:
            port: 12800
          initialDelaySeconds: 15
          periodSeconds: 20
        readinessProbe:
          tcpSocket:
            port: 12800
          initialDelaySeconds: 15
          periodSeconds: 20
        ports:
        - containerPort: 11800
          name: grpc
        - containerPort: 12800
          name: rest
        env:
        - name: JAVA_OPTS
          value: "-Dmode=no-init -Xmx2g -Xms2g"
        - name: SW_CLUSTER
          value: kubernetes
        - name: SW_CLUSTER_K8S_NAMESPACE
          value: "skywalking"
        - name: SW_CLUSTER_K8S_LABEL
          value: "app=skywalking,component=oap"
        - name: SKYWALKING_COLLECTOR_UID
          valueFrom:
            fieldRef:
              fieldPath: metadata.uid
        - name: SW_STORAGE
          value: "elasticsearch"
        - name: TZ
          value: Asia/Shanghai
        - name: SW_STORAGE_ES_CLUSTER_NODES
           # elasticsearch 地址
          value: "elasticsearch或者es-ip:9200"
        - name: SW_ES_USER
          value: "elastic"
        - name: SW_ES_PASSWORD
          value: "xxxxx"
---
apiVersion: v1
kind: Service
metadata:
  namespace: skywalking
  name: skywalking-oap
  labels:
    app: skywalking
    component: "oap"
spec:
  type: ClusterIP
  ports:
  - port: 12800
    name: rest
  - port: 11800
    name: grpc
  selector:
    app: skywalking
    component: "oap"

file

file

部署ui

apiVersion: v1
kind: Service
metadata:
  labels:
    app: skywalking
    component: "ui"
  name: skywalking-ui
  namespace: skywalking
spec:
  type: ClusterIP
  ports:
    - port: 80
      targetPort: 8080
      protocol: TCP
  selector:
    app: skywalking
    component: "ui"
---
apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: skywalking
  name: skywalking-ui
  labels:
    app: skywalking
    component: "ui"
spec:
  replicas: 1
  selector:
    matchLabels:
      app: skywalking
      component: "ui"
  template:
    metadata:
      labels:
        app: skywalking
        component: "ui"
    spec:
      containers:
      - name: ui
        image: apache/skywalking-ui:9.3.0
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 8080
          name: page
        env:
        - name: SW_OAP_ADDRESS
          value: "http://skywalking-oap:12800"
        - name: TZ
          value: Asia/Shanghai

file

添加域名

可选，也可以修改svc为nodePort。

apiVersion: v1
items:
- apiVersion: networking.k8s.io/v1
  kind: Ingress
  metadata:
    name: skywalking
    namespace: skywalking
  spec:
    rules:
    - host: skywalking.qwer.com
      http:
        paths:
        - backend:
            service:
              name: skywalking-ui
              port:
                number: 80
          path: /
          pathType: ImplementationSpecific
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

访问。

file

相关报错

  • init容器wait-for-elasticsearch error，没有日志输出，超时后状态变为Init:Error。

file

原因

EFK中的ES有设置密码，skywalking没有添加相关配置，导致无法连接ES。

解决

在deployment的yaml中添加如下的配置：（上面的配置已经加过了）

- name: SW_ES_USER
  value: "elastic"
- name: SW_ES_PASSWORD
  value: "xxxxx"
  • table：alarm_record does not exist, OAP is running in "no-init" mode, wating… retry 3s later.

file

原因

没有执行init，直接部署了deployment，并且skywalking的启动模式是-Dmode=no-init。

解决

先执行init的job，再执行deployment。或者去掉-Dmode=no-init。

  • no provider found for module storage，if you’re sure it’s not required module and want to remove it, set the selector to –

file

原因

yaml中SW_STORAGE配置为了elasticsearch7。

解决

修改为SW_STORAGE=elasticsearch，让skywalking自动识别。

  • skywalking Illegal character in scheme name at index

file

原因

ui的yaml中指定oap-server的地址没有加http

env:
- name: SW_OAP_ADDRESS
  value: "skywalking-oap:12800"

解决

添加http即可。

env:
- name: SW_OAP_ADDRESS
  value: "http://skywalking-oap:12800"

agent配置

java微服务接入skywalking可以使用SkyWalking Java Agent来上报监控数据，需要java服务在启动参数中通过 -javaagent:<skywalking-agent-path> 指定skywalking agent探针包，通常有以下几种方式：

  • 使用官方提供的基础镜像skywalking-base；
  • 将agent包构建到已存在的镜像中；
  • 通过pvc或者hostpath来挂载agent目录到容器；
  • 通过sidecar来挂载agent目录；

前两种都需要修改原来的基础镜像，如果服务很少还可以考虑。服务很多的话修改起来很麻烦，所以不推荐。第三种需要修改服务的yaml，增加agent目录的挂载。第四种sidecar也需要修改服务的yaml，通过共享的volume将agent目录挂载到容器中。

这里选择的是sidecar模式。

构建skywalking agent镜像

版本需要对应，下载9.3.0的agent，

wget https://archive.apache.org/dist/skywalking/9.3.0/apache-skywalking-apm-9.3.0.tar.gz
tar xvf apache-skywalking-apm-9.3.0.tar.gz

创建Dockerfile：

FROM busybox:latest
COPY apache-skywalking-apm-bin/agent/ /usr/skywalking/agent/

构建镜像：

docker build -t skywalking-agent-sidecar:9.3.0 .

创建测试服务

apiVersion: apps/v1
kind: Deployment
metadata:
  name: skywalking-demo
  namespace: skywalking
  labels:
    app: skywalking-demo
spec:
  replicas: 1
  selector:
    matchLabels:
      app: skywalking-demo
  template:
    metadata:
      labels:
        app: skywalking-demo
    spec:
      initContainers:
        - image: skywalking-agent-sidecar:9.3.0
          name: skywalking-agent-sidecar
          imagePullPolicy: IfNotPresent
          command: [ "sh" ]
          args:
            [
                "-c",
                "cp -R /usr/skywalking/agent/* /skywalking/agent",
            ]
          volumeMounts:
            - mountPath: /skywalking/agent
              name: skywalking-agent
      containers:
        - name: skywalking-demo
          image: skywalking-demo:xxx # 你的服务镜像
          imagePullPolicy: IfNotPresent
          env:
            - name: TZ
              value: Asia/Shanghai
            - name: BUILD_TAG
              value: ${BUILD_TAG}
            - name: NAMESPACE
              value: skywalking
            #这里通过JAVA_TOOL_OPTIONS，而不是JAVA_OPTS可以实现不通过将agent命令加入到java应用jvm参数而实现agent的集成
            - name: JAVA_TOOL_OPTIONS
              value: -javaagent:/usr/skywalking/agent/skywalking-agent.jar
            - name: SW_AGENT_NAME
              value: skywalking-demo
            - name: SW_AGENT_COLLECTOR_BACKEND_SERVICES
              value: skywalking-oap:11800
          ports:
            - containerPort: 8080
          resources:
            requests:
              cpu: 200m
              memory: 500Mi
          volumeMounts:
            - mountPath: /usr/skywalking/agent
              name: skywalking-agent
      volumes:
        - name: skywalking-agent
          emptyDir: { }
---
apiVersion: v1
kind: Service
metadata:
  name: skywalking-demo
  namespace: default
  labels:
    app: skywalking-demo
spec:
  ports:
    - name: port
      port: 80
      protocol: TCP
      targetPort: 8080
  selector:
    app: skywalking-demo
  type: ClusterIP

使用init容器，通过sh -c cp -R /usr/skywalking/agent/* /skywalking/agent命令将skywalking agent目录复制到emptyDir共享目录中。

java启动的命令也可以这么写：

- command:
    - java
    - -javaagent:/data/agent/skywalking-agent.jar # 这里的路径可以是相对路径
    - -jar
    - demo.jar
0 0 投票数
文章评分
订阅评论
提醒
guest

0 评论
最旧
最新 最多投票
内联反馈
查看所有评论

相关文章

开始在上面输入您的搜索词，然后按回车进行搜索。按ESC取消。

返回顶部
0
希望看到您的想法，请您发表评论x