环境准备 下载离线安装文件 准备数据库 PostgreSQL 安装部署 k3s 准备服务器 导入镜像包 拷贝主程序 k3s 安装 安装 Helm 部署工具 部署 Cert Manager 部署 Rancher 集群管理工具 导入镜像 部署 Rancher 部署网络存储 使用 Ceph（推荐） 使用 NFS 网络文件系统 部署 Harbor 私有化仓库 部署 pv 和 pvc 安装 NFS 依赖 部署 Harbor 部署 ChartMuseum 管理工具 下载 ChartMuseum 镜像 部署 ChartMuseum Helm Chart 上传 Charts 到 ChartMuseum 部署 frps 反向代理 编译 frps 开发 frps Helm Chart 部署到 k3s 管理集群 验证结果 管理新的集群（业务集群） 环境准备 下载离线安装文件 本例子用 v1.30.4+k3s1 安装脚本 install.sh wget https://github.com/k3s-io/k3s/raw/refs/tags/v1.30.4+k3s1/install.sh 主程序 k3s wget https://github.com/k3s-io/k3s/releases/download/v1.30.4%2Bk3s1/k3s 镜像压缩包 wget https://github.com/k3s-io/k3s/releases/download/v1.30.4%2Bk3s1/k3s-airgap-images-amd64.tar 准备数据库 PostgreSQL 数据库 name ip notes database 192.168.100.100 PostgreSQL 创建 compose.yml services: postgres: image: postgres:alpine container_name: postgres restart: unless-stopped environment: POSTGRES_USER: k3s POSTGRES_PASSWORD: talent POSTGRES_DB: k3s ports: - "5432:5432" volumes: - ./data:/var/lib/postgresql/data - ./init-db.sh:/docker-entrypoint-initdb.d/init-db.sh 创建脚本 init-db.sh #!/bin/bash set -e psql -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" --dbname=postgres <<-EOSQL CREATE DATABASE harbor; EOSQL 启动 PostgreSQL 数据库容器 docker compose up -d 验证数据库 docker exec -it postgres psql -U k3s -c '\l' List of databases Name | Owner | Encoding | Locale Provider | Collate | Ctype | Locale | ICU Rules | Access privileges -----------+-------+----------+-----------------+------------+------------+--------+-----------+------------------- harbor | k3s | UTF8 | libc | en_US.utf8 | en_US.utf8 | | | k3s | k3s | UTF8 | libc | en_US.utf8 | en_US.utf8 | | | ... (5 rows) 安装部署 k3s 准备服务器 准备三台机器，都作为 master 运行，本节内容需要在三台都执行。 name ip notes master01 192.168.100.101 必须 master02 192.168.100.102 可选 master03 192.168.100.103 可选 cat <<EOF >> /etc/hosts 192.168.100.100 database 192.168.100.101 master01 192.168.100.102 master02 192.168.100.103 master03 EOF 导入镜像包 离线安装重要步骤 mkdir -p /var/lib/rancher/k3s/agent/images/ cp k3s-airgap-images-amd64.tar /var/lib/rancher/k3s/agent/images/ 拷贝主程序 k3s 拷贝进执行目录，不需要下载主程序 chmod a+x k3s && cp k3s /usr/local/bin/ 安装 执行安装脚本，并连接数据库 INSTALL_K3S_SKIP_DOWNLOAD=true \ sh install.sh -s - server \ --datastore-endpoint="postgres://k3s:talent@database:5432/k3s?sslmode=disable" \ --token K10c71232f051944de58ab058871ac7a85b42cbdee5c6df94deb6bb493c79b15a92 查看镜像状态 ctr i ls -q docker.io/rancher/klipper-helm:v0.8.4-build20240523 docker.io/rancher/klipper-lb:v0.4.9 docker.io/rancher/local-path-provisioner:v0.0.28 docker.io/rancher/mirrored-coredns-coredns:1.10.1 docker.io/rancher/mirrored-library-busybox:1.36.1 docker.io/rancher/mirrored-library-traefik:2.10.7 docker.io/rancher/mirrored-metrics-server:v0.7.0 docker.io/rancher/mirrored-pause:3.6 三台机器都执行完成，查看集群 node 状态 kubectl get nodes NAME STATUS ROLES AGE VERSION master01 Ready control-plane,master 4m49s v1.30.4+k3s1 master02 Ready control-plane,master 51s v1.30.4+k3s1 master03 Ready control-plane,master 45s v1.30.4+k3s1 查看 pod 状态 kubectl get pods -A NAMESPACE NAME READY STATUS RESTARTS AGE kube-system coredns-576bfc4dc7-f6bsb 1/1 Running 0 27m kube-system helm-install-traefik-crd-5b2x4 0/1 Completed 0 27m kube-system helm-install-traefik-pl6k7 0/1 Completed 1 27m kube-system local-path-provisioner-5f9d8-lhntz 1/1 Running 0 27m kube-system metrics-server-557ff575fb-77xzw 1/1 Running 0 27m kube-system svclb-traefik-151f91a9-8tkdq 2/2 Running 0 23m kube-system svclb-traefik-151f91a9-b7pwz 2/2 Running 0 23m kube-system svclb-traefik-151f91a9-kwwp8 2/2 Running 0 25m kube-system traefik-5fb479b77-r84mv 1/1 Running 0 25m 安装 Helm 部署工具 下载 Helm 二进制，安装 wget https://get.helm.sh/helm-v3.16.2-linux-amd64.tar.gz tar -zxvf helm-v3.16.2-linux-amd64.tar.gz chmod a+x linux-amd64/helm && cp helm /usr/local/bin/ 设置环境变量 加入到 ～/.bashrc 自启动 echo "export KUBECONFIG=/etc/rancher/k3s/k3s.yaml" >> ~/.bashrc && source ~/.bashrc 部署 Cert Manager 在线下载 Cert Manager 镜像，并导出，复制到三台服务器 docker images --format "{{.Repository}}:{{.Tag}}" |grep cert-manager > cert-manager.txt docker save $(awk '{printf "%s ", $0}' cert-manager.txt) | gzip > cert-manager.tar.gz 导入 Cert Manager 镜像 gzip -d -c cert-manager-images.tar.gz | ctr i import - 下载所需文件，复制到某一台服务器后 wget https://github.com/cert-manager/cert-manager/releases/download/v1.15.3/cert-manager.crds.yaml wget https://charts.jetstack.io/charts/cert-manager-v1.15.3.tgz 部署 cert-manager 配置 kubectl apply -f cert-manager.crds.yaml customresourcedefinition.apiextensions.k8s.io/certificaterequests.cert-manager.io created customresourcedefinition.apiextensions.k8s.io/certificates.cert-manager.io created customresourcedefinition.apiextensions.k8s.io/challenges.acme.cert-manager.io created customresourcedefinition.apiextensions.k8s.io/clusterissuers.cert-manager.io created customresourcedefinition.apiextensions.k8s.io/issuers.cert-manager.io created customresourcedefinition.apiextensions.k8s.io/orders.acme.cert-manager.io created 使用 Helm 部署 cert-manager 离线包 helm install cert-manager ./cert-manager-v1.15.3.tgz --namespace cert-manager --create-namespace or helm install cert-manager ./cert-manager-v1.15.3.tgz --namespace cert-manager --create-namespace --set crds.enabled=true kubectl get pods -n cert-manager NAME READY STATUS RESTARTS AGE cert-manager-9647b459d-dqd8b 1/1 Running 0 23m cert-manager-cainjector-5d8798687c-l2kkf 1/1 Running 0 23m cert-manager-startupapicheck-6gglh 0/1 Completed 1 23m cert-manager-webhook-c77744d75-wd2kx 1/1 Running 0 23m 部署 Rancher 集群管理工具 导入镜像 在线下载 Rancher 镜像，并导出，复制到三台服务器 docker images --format "{{.Repository}}:{{.Tag}}" |grep rancher > rancher.txt docker save $(awk '{printf "%s ", $0}' rancher.txt) | gzip > rancher-images.tar.gz 导入 Rancher 镜像 gzip -d -c rancher-images.tar.gz | ctr i import - 部署 Rancher 下载 Rancher Helm 包 wget https://releases.rancher.com/server-charts/latest/rancher-2.9.2.tgz 使用 Helm 部署 Rancher 离线包 helm install rancher ./rancher-2.9.2.tgz \ --namespace cattle-system \ --set hostname=manager.yiqisoft.cn \ --create-namespace 查询 Rancher 安装结果 kubectl get pods -n cattle-system NAME READY STATUS RESTARTS AGE rancher-57c9747d96-p498t 1/1 Running 0 52m rancher-57c9747d96-pdrtn 1/1 Running 1 (52m ago) 52m rancher-57c9747d96-xm9sl 1/1 Running 0 52m rancher-webhook-77b49b9ff9-fqdxf 1/1 Running 0 51m 配置 dns 指向，访问 Rancher 页面 dns 指向到三台服务器，如果有条件，要设置 HAProxy 和 KeepAlived 做负载均衡。 address=/manager.yiqisoft.cn/192.168.100.101 address=/manager.yiqisoft.cn/192.168.100.102 address=/manager.yiqisoft.cn/192.168.100.103 部署网络存储 使用 Ceph（推荐） 准备一个 Ceph 集群用于存储（这里省略，请自行补脑）。 使用 NFS 网络文件系统 简单实用，但是效率不高，安全性一般。 在一个文件服务器上部署 NFS，这里用 database 主机 apt update apt install nfs-kernel-server -y mkdir -p /opt/nfs_share chown nobody:nogroup /opt/nfs_share chmod 777 /opt/nfs_share echo "/opt/nfs_share *(rw,sync,no_subtree_check)" >> /etc/exports exportfs -ra systemctl restart nfs-kernel-server 部署 Harbor 私有化仓库 部署 pv 和 pvc 编辑 pv-pvc.yaml apiVersion: v1 kind: PersistentVolume metadata: name: nfs-pv spec: capacity: storage: 30Gi accessModes: - ReadWriteMany nfs: path: /opt/nfs_share server: database --- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: nfs-pvc spec: accessModes: - ReadWriteMany resources: requests: storage: 10Gi volumeName: nfs-pv storageClassName: "" 应用到 namespace harbor kubectl create namespace harbor kubectl apply -f pv-pvc.yaml -n harbor 结果 persistentvolume/nfs-pv created persistentvolumeclaim/nfs-pvc created 查看结果，nfs-pvc 必须是 Bound kubectl get pvc -n harbor 结果 NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE nfs-pvc Bound nfs-pv 30Gi RWX <unset> 92s 查看 nfs-pv 必须是 Bound kubectl get pv -n harbor 结果 NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS VOLUMEATTRIBUTESCLASS REASON AGE nfs-pv 30Gi RWX Retain Bound harbor/nfs-pvc <unset> 97s 安装 NFS 依赖 所有的 Master 上需要安装 nfs-common，并验证（可选），可以访问 /tmp/nfs 目录 apt install nfs-common -y mkdir /tmp/nfs mount -t nfs database:/opt/nfs_share /tmp/nfs ll /tmp/nfs 部署 Harbor 在线下载 Harbor Helm 文件，并复制到某一台服务器 wget https://helm.goharbor.io/harbor-1.15.1.tgz 在线方式，导出默认 values，修改 helm show values harbor/harbor > harbor-values.yml # 内容修改如下 expose: ingress: hosts: core: hub.yiqisoft.cn externalURL: https://hub.yiqisoft.cn persistentVolumeClaim: registry:/exter existingClaim: "nfs-pvc" storageClass: "-" database: type: external external: host: "192.168.100.100" port: "5432" username: "k3s" password: "talent" coreDatabase: "harbor" harborAdminPassword: "Harbor12345" 在线下载 Harbor 镜像，导出 docker images --format "{{.Repository}}:{{.Tag}}" |grep goharbor > harbor.txt docker save $(awk '{printf "%s ", $0}' harbor.txt) | gzip > harbor-images.tar.gz 导入 Harbor 镜像 gzip -d -c harbor-images.tar.gz | ctr i import - 安装 Harbor，需要通过互联网下载所有的镜像，时间根据网络带宽决定 helm install harbor ./harbor-1.15.1.tgz -f harbor-values.yml -n harbor 确认 Harbor 是否安装成功 kubectl get pods -n harbor 都显示 READY 即可 NAME READY STATUS RESTARTS AGE harbor-core-666cf84bc6-vq8mp 1/1 Running 0 85s harbor-jobservice-c68bc669-6twbp 1/1 Running 0 85s harbor-portal-d6c99f896-cq9fx 1/1 Running 0 85s harbor-redis-0 1/1 Running 0 84s harbor-registry-5984c768c8-4xnrg 2/2 Running 0 84s harbor-trivy-0 1/1 Running 0 84s 配置 dns 指向，访问 Harbor 页面 dns 指向到三台服务器，如果有条件，要设置 HAProxy 和 KeepAlived 做负载均衡。 address=/hub.yiqisoft.cn/192.168.100.101 address=/hub.yiqisoft.cn/192.168.100.102 address=/hub.yiqisoft.cn/192.168.100.103 验证 Harbor 工作 curl -k https://hub.yiqisoft.cn/v2/ {"errors":[{"code":"UNAUTHORIZED","message":"unauthorized: unauthorized"}]} 部署 ChartMuseum 管理工具 下载 ChartMuseum 镜像 在线下载 ChartMuseum 镜像，并导出，复制到三台服务器 docker save hub.yiqisoft.cn/yiqisoft/chartmuseum:v0.16.2 | gzip > chartmuseum-images.tar.gz 上传到私有仓库 docker push hub.yiqisoft.cn/yiqisoft/chartmuseum:v0.16.2 或导入 ChartMuseum 镜像 gzip -d -c chartmuseum-images.tar.gz | ctr i import - 部署 ChartMuseum Helm Chart 下载 ChartMuseum chart wget https://github.com/chartmuseum/charts/releases/download/chartmuseum-3.10.3/chartmuseum-3.10.3.tgz 部署 Chart helm install chartmuseum ./chartmuseum-3.10.3.tgz \ --namespace chartmuseum \ --create-namespace \ --set env.open.DISABLE_API=false \ --set service.type=LoadBalancer \ --set persistence.enabled=true \ --set persistence.existingClaim=pvc-chart \ --set env.secret.BASIC_AUTH_USER=yiqisoft \ --set env.secret.BASIC_AUTH_PASS=talent 部署一个 Ingress apiVersion: networking.k8s.io/v1 kind: Ingress name: chartmuseum namespace: chartmuseum spec: ingressClassName: traefik rules: - host: chart.yiqisoft.cn http: paths: - backend: service: name: chartmuseum port: number: 8080 path: / pathType: Prefix 或在 Rancher UI 中增加一个类似配置。 上传 Charts 到 ChartMuseum 使用类似这样的命令即可上传，ChartMuseum 会自动索引文件到 index.yaml curl --data-binary "@yiedge-base-3.1.0.tgz" http://chart.yiqisoft.cn/api/charts 检查上传结果 curl chart.yiqisoft.cn/index.yaml apiVersion: v1 entries: yiedge-base: - apiVersion: v1 appVersion: Pnicom created: "2024-11-11T10:11:55.473460704Z" description: A Helm Chart for YiEDGE base digest: 14b9a971a53a2482a6d4a930daefe2b0fee1cb1bcef909b023cc34959b4c142a home: https://www.yiqisoft.cn icon: https://www.yiqisoft.cn/images/yiqilogo.png keywords: - IoT - Middleware - YiEDGE name: yiedge-base sources: - https://github.com/yiqisoft urls: - charts/yiedge-base-3.1.0.tgz version: 3.1.0 generated: "2024-11-11T10:11:56Z" serverInfo: {} 部署 frps 反向代理 编译 frps clone 仓库，并编译 docker image git clone https://github.com/fatedier/frp.git git checkout v0.61.0 docker build -f dockerfiles/Dockerfile-for-frps . -t hub.yiqisoft.cn/yiqisoft/frps:v0.61.0 push 到私有仓库 docker push hub.yiqisoft.cn/yiqisoft/frps:v0.61.0 开发 frps Helm Chart 给出范例，根据自己的需求完成。 Helm Chart 目录结构 frp-server/ ├── charts/ # 可选，依赖的子 Chart ├── templates/ # 存放模板文件 │ ├── deployment.yaml # 部署模板 │ ├── service.yaml # 服务模板 │ ├── configmap.yaml # 配置文件模板 │ ├── ingress.yaml # 可选的 Ingress 配置 ├── values.yaml # 默认配置文件 ├── Chart.yaml # Chart 描述文件 Chart.yaml apiVersion: v1 name: frps version: "0.0.1" appVersion: frp Server description: A Helm Chart for frp server keywords: - frp - reverse proxy home: https://www.yiqisoft.cn icon: https://www.yiqisoft.cn/images/yiqilogo.png sources: - https://github.com/yiqisoft values.yaml 根据需要修改 domain 和 subDomainHost，还有一些对外开放的端口 service.nodePorts.http/frp/dashboard rreplicaCount: 1 fullnameOverride: "frp-server" domain: "*.h.yiqisoft.cn" # dynamic domain image: repository: hub.yiqisoft.cn/yiqisoft/frps tag: v0.61.0 pullPolicy: IfNotPresent service: type: NodePort nodePorts: http: 30080 # vhost http nodePort frp: 30000 # frp service nodePort dashboard: 30500 # dashboard nodePort frpConfig: bindPort: 7000 vhostHTTPPort: 8888 subDomainHost: "h.yiqisoft.cn" webServer: addr: "0.0.0.0" port: 7500 user: "yiqisoft" password: "talent123!" pprofEnable: true log: to: "/tmp/frps.log" level: "info" maxDays: 7 auth: token: "yiqisoft" deployment.yaml apiVersion: apps/v1 kind: Deployment metadata: name: {{ .Values.fullnameOverride }} labels: app: {{ .Values.fullnameOverride }} spec: replicas: {{ .Values.replicaCount }} selector: matchLabels: app: {{ .Values.fullnameOverride }} template: metadata: labels: app: {{ .Values.fullnameOverride }} spec: containers: - name: frp-server image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}" imagePullPolicy: {{ .Values.image.pullPolicy }} args: - "-c" - "/frps.toml" ports: - name: frp-vhost containerPort: {{ .Values.frpConfig.vhostHTTPPort }} - name: frp-port containerPort: {{ .Values.frpConfig.bindPort }} - name: frp-dashboard containerPort: {{ .Values.frpConfig.webServer.port }} volumeMounts: - name: frp-config mountPath: /frps.toml subPath: frps.toml volumes: - name: frp-config configMap: name: {{ .Values.fullnameOverride }}-config service.yaml apiVersion: v1 kind: Service metadata: name: {{ .Values.fullnameOverride }}-service labels: app: {{ .Values.fullnameOverride }} spec: type: NodePort ports: - name: frp-vhost port: {{ .Values.frpConfig.vhostHTTPPort }} # 配置的 HTTP 端口 targetPort: frp-vhost protocol: TCP nodePort: {{ .Values.service.nodePorts.http }} # 从 values.yaml 中读取对外暴露的端口 - name: frp-port port: {{ .Values.frpConfig.bindPort }} # 配置的 bindPort targetPort: frp-port protocol: TCP nodePort: {{ .Values.service.nodePorts.frp }} # 从 values.yaml 中读取对外暴露的端口 - name: frp-dashboard port: {{ .Values.frpConfig.webServer.port }} # 配置的 dashboard 端口 targetPort: frp-dashboard protocol: TCP nodePort: {{ .Values.service.nodePorts.dashboard }} # 从 values.yaml 中读取对外暴露的端口 selector: app: {{ .Values.fullnameOverride }} ingress.yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: {{ .Values.fullnameOverride }}-ingress labels: app: {{ .Values.fullnameOverride }} annotations: nginx.ingress.kubernetes.io/rewrite-target: / spec: rules: - host: "{{ .Values.domain }}" http: paths: - path: / pathType: Prefix backend: service: name: {{ .Values.fullnameOverride }}-service port: number: {{ .Values.frpConfig.vhostHTTPPort }} configmap.yaml apiVersion: v1 kind: ConfigMap metadata: name: {{ .Values.fullnameOverride }}-config data: frps.toml: | bindPort = {{ .Values.frpConfig.bindPort }} vhostHTTPPort = {{ .Values.frpConfig.vhostHTTPPort }} subDomainHost = "{{ .Values.frpConfig.subDomainHost }}" webServer.addr = "{{ .Values.frpConfig.webServer.addr }}" webServer.port = {{ .Values.frpConfig.webServer.port }} webServer.user = "{{ .Values.frpConfig.webServer.user }}" webServer.password = "{{ .Values.frpConfig.webServer.password }}" webServer.pprofEnable = {{ .Values.frpConfig.webServer.pprofEnable }} log.to = "{{ .Values.frpConfig.log.to }}" log.level = "{{ .Values.frpConfig.log.level }}" log.maxDays = {{ .Values.frpConfig.log.maxDays }} auth.token = "{{ .Values.frpConfig.auth.token }}" 打包 Chart helm package frps-server Successfully packaged chart and saved it to: /opt/charts/frps-0.0.1.tgz 上传到私有 ChartMuseum curl --data-binary "@frps-0.0.1.tgz" http://chart.yiqisoft.cn/api/charts 部署到 k3s 管理集群 通过手工方式将 frps-0.0.1.tgz 部署到集群 helm install frps-server ./frp-0.0.1.tgz -n frps --createnamespace 通过 Rancher UI 部署更为简单一些。 验证结果 需要在私有 dns 解析地址，例如 dnsmasq address=/h.yiqisoft.cn/192.168.100.101 address=/*.h.yiqisoft.cn/192.168.100.101 访问 http://h.yiqisoft.cn:30500 可以管理 frps frpc 连接服务器 tcp://h.yiqisoft.cn:30000，即可访问 client-id.h.yiqisoft.cn 管理新的集群（业务集群） 某些情况下，可能需要清理 CoreDNS 缓存 kubectl -n kube-system rollout restart deployment coredns
准备工作 硬件 软件 基础配置 数据库配置 Docker 数据库 PostgreSQL MySQL 部署 K3S 管理集群 主机名 K3S 核心安装 自动安装(推荐) 手动二进制文件安装(不推荐) Master / Worker 手动加入(如有必要) 安装 Rancher 集群管理工具和 Harbor Registry 仓库 准备工作 安装 Kubernetes 包管理工具 Helm 安装证书管理器 cert-manager 安装 Rancher 安装 Harbor 使用 Ceph（推荐） 使用 NFS 部署 pv 和 pvc 部署 Harbor Helm 批量上传所需 image 到 Harbor 部署 K3S 业务集群 所有的 Master 和 Worker 都加入业务集群 Rancher 中创建一个新的集群 Master 加入集群(初始化业务集群) Worker 加入业务集群 (部署/更新) Harbor hub CA 证书到所有的 Master 和 Worker (手动操作)导出 Harbor hub HTTPS 证书 (自动部署)Harbor Hub CA 证书 新建一个 Secret 新建一个 DaemonSet 基于 K3S 部署业务应用(边缘计算框架) EdgeX Foundry Base 部署辅助应用：流媒体，推理服务器，NVR 部署 OpenVINO AI 应用 总结 准备工作 硬件 主 K3S Master 节点：3台, master101, master102, master103 业务 K3S Master 节点：3台，master151, master152, master153 Worker 节点：N台，随意, worker01, worker02, ... 软件 基础配置 操作系统：Ubuntu Server 22.04 最小化安装系统，包含：openssh-server, iputils-ping, net-tools, vim PRETTY_NAME="Ubuntu 22.04.5 LTS" NAME="Ubuntu" VERSION_ID="22.04" VERSION="22.04.5 LTS (Jammy Jellyfish)" 管理集群 主机名 IP 其他 database 192.168.198.100 数据库+NFS存储，如有需要做 HA master101 192.168.198.101 管理 K3S master master102 192.168.198.102 管理 K3S master master103 192.168.198.103 管理 K3S master 业务集群 主机名 IP 其他 master151 192.168.123.151 业务 K3S master + worker master152 192.168.123.152 业务 K3S master + worker master153 192.168.123.153 业务 K3S master + worker worker161 192.168.123.161 业务 worker worker162 192.168.123.162 业务 worker 数据库配置 Docker 准备 docker 环境 curl -fsSL https://get.docker.com | sh - 验证 docker docker images REPOSITORY TAG IMAGE ID CREATED SIZE 设置一个 Registry 镜像 cat /etc/docker/daemon.json { "insecure-registries" : [ "0.0.0.0/0" ], "registry-mirrors": [ "https://xxx.mirror.swr.myhuaweicloud.com" ] } 重启 docker systemctl restart docker 数据库 PostgreSQL 创建 compose.yml services: postgres: image: postgres:alpine container_name: postgres restart: unless-stopped environment: POSTGRES_USER: k3s POSTGRES_PASSWORD: talent POSTGRES_DB: k3s ports: - "5432:5432" volumes: - ./data:/var/lib/postgresql/data - ./init-db.sh:/docker-entrypoint-initdb.d/init-db.sh 创建脚本 init-db.sh #!/bin/bash set -e psql -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" --dbname=postgres <<-EOSQL CREATE DATABASE harbor; EOSQL 启动 PostgreSQL 数据库容器 docker compose up -d 验证数据库 docker exec -it postgres psql -U k3s -c '\l' List of databases Name | Owner | Encoding | Locale Provider | Collate | Ctype | Locale | ICU Rules | Access privileges -----------+-------+----------+-----------------+------------+------------+--------+-----------+------------------- harbor | k3s | UTF8 | libc | en_US.utf8 | en_US.utf8 | | | k3s | k3s | UTF8 | libc | en_US.utf8 | en_US.utf8 | | | ... (5 rows) MySQL 因为 Rancher 只支持 MySQL 不支持 PostgreSQL 创建 compose.yml services: mysql: image: mysql:latest container_name: mysql restart: unless-stopped environment: MYSQL_ROOT_PASSWORD: talent MYSQL_DATABASE: rancher MYSQL_USER: rancher MYSQL_PASSWORD: talent ports: - "3306:3306" volumes: - ./data:/var/lib/mysql - ./init:/docker-entrypoint-initdb.d 创建脚本 init/permissions.sql GRANT ALL PRIVILEGES ON *.* TO 'rancher'@'%' IDENTIFIED BY 'talent'; FLUSH PRIVILEGES; 启动 MySQL 数据库容器 docker compose up -d 验证数据库 docker exec -it mysql mysql -u rancher -p'talent' -e "SHOW DATABASES;" +--------------------+ | Database | +--------------------+ | rancher | +--------------------+ 部署 K3S 管理集群 本章节内容在管理 K3S 主机 上执行 主机名 所有的 Master 配置相同的 /etc/hosts127.0.0.1 localhost 127.0.1.1 master101 # CHAMGEME 192.168.198.100 database 192.168.198.101 master101 192.168.198.102 master102 192.168.198.103 master103 K3S 核心安装 自动安装(推荐) 可以连接互联网的情况下 默认安装最新版本，这里是: v1.30.5+k3s1，注意：k3s:talent 是你自己的数据库用户名密码，使用一样的 token curl -sfL https://get.k3s.io | sh -s - server \ --datastore-endpoint="postgres://k3s:talent@database:5432/k3s?sslmode=disable" --token K10c71232f051944de58ab058871ac7a85b42cbdee5c6df94deb6bb493c79b15a92::server:3d6c2d90cec2e63db66bde0e6f24013d 验证安装 kubectl get nodes NAME STATUS ROLES AGE VERSION master101 Ready control-plane,master 43s v1.30.5+k3s1 其他 Master 安装方式一样。在任何一台 Master 上确认安装完成 kubectl get nodes NAME STATUS ROLES AGE VERSION master101 Ready control-plane,master 117s v1.30.5+k3s1 master102 Ready control-plane,master 118s v1.30.5+k3s1 master103 Ready control-plane,master 29s v1.30.5+k3s1 手动二进制文件安装(不推荐) 下载二进制文件 wget https://github.com/k3s-io/k3s/releases/download/vX.X.X/k3s -O /usr/local/bin/k3s chmod +x /usr/local/bin/k3s 初始化 k3s server --cluster-init 设置自启动脚本 所有 Master 机器上一样的配置 cat <<EOF | sudo tee /etc/systemd/system/k3s.service [Unit] Description=Lightweight Kubernetes Documentation=https://k3s.io After=network.target [Service] ExecStart=/usr/local/bin/k3s server --datastore-endpoint="postgres://k3s:talent@database:5432/k3s?sslmode=disable" --token K10c71232f051944de58ab058871ac7a85b42cbdee5c6df94deb6bb493c79b15a92::server:3d6c2d90cec2e63db66bde0e6f24013d Restart=always RestartSec=5s [Install] WantedBy=multi-user.target EOF 更新自启动设置 systemctl daemon-reload systemctl enable k3s systemctl start k3s 查看集群状态 任何一台 Master 都可以 /usr/local/bin/k3s kubectl get nodes 结果 NAME STATUS ROLES AGE VERSION master101 Ready control-plane,master 10m v1.30.4+k3s1 master102 Ready control-plane,master 10m v1.30.4+k3s1 master103 Ready control-plane,master 81s v1.30.4+k3s1 Master / Worker 手动加入(如有必要) 获取集群 token cat /var/lib/rancher/k3s/server/node-token Master 直接加入集群（不建议） k3s server --server https://master101:6443 --token <token> 或修改 /etc/systemd/system/k3s.service Worker加入（可选） 替换<node-token> 为你的token cat <<EOF | sudo tee /etc/systemd/system/k3s-agent.service [Unit] Description=Lightweight Kubernetes Documentation=https://k3s.io After=network.target [Service] ExecStart=/usr/local/bin/k3s agent --server https://master101:6443 --token <node-token> Restart=always RestartSec=5s [Install] WantedBy=multi-user.target EOF 更新自启动设置 systemctl daemon-reload systemctl enable k3s-agent systemctl start k3s-agent 不管怎么操作，基础三个主机的 K3S 集群要跑起来，保证基础工作完成。 安装 Rancher 集群管理工具和 Harbor Registry 仓库 准备工作 安装 Kubernetes 包管理工具 Helm 选择一个 Master 或全部 Master，安装 Helm curl -fsSL https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash - 设置环境变量 加入到 /etc/profile 自启动 echo "export KUBECONFIG=/etc/rancher/k3s/k3s.yaml" >> /etc/profile && source /etc/profile 安装证书管理器 cert-manager helm 脚本安装 kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.15.3/cert-manager.crds.yaml helm repo add jetstack https://charts.jetstack.io helm repo update helm install cert-manager jetstack/cert-manager --namespace cert-manager \ --create-namespace --version v1.15.3 验证安装成功 kubectl get pods --namespace cert-manager NAME READY STATUS RESTARTS AGE cert-manager-9647b459d-9jwpj 1/1 Running 0 2m47s cert-manager-cainjector-5d8798687c-66r95 1/1 Running 0 2m47s cert-manager-webhook-c77744d75-k22h5 1/1 Running 0 2m47s 设置自签名 Issuer 创建 issuer-cert.yaml, 这里创建两个： rancher.yiqisoft.com hub.yiqisoft.com, 后面访问集群业务就是通过域名来通信 apiVersion: cert-manager.io/v1 kind: ClusterIssuer metadata: name: selfsigned-issuer spec: selfSigned: {} --- apiVersion: cert-manager.io/v1 kind: Certificate metadata: name: rancher-cert namespace: cattle-system spec: secretName: rancher-cert-secret issuerRef: name: selfsigned-issuer kind: ClusterIssuer commonName: rancher.yiqisoft.com duration: 315360000s renewBefore: 240h dnsNames: - rancher.yiqisoft.com --- apiVersion: cert-manager.io/v1 kind: Certificate metadata: name: hub-cert namespace: harbor spec: secretName: hub-cert-secret issuerRef: name: selfsigned-issuer kind: ClusterIssuer commonName: hub.yiqisoft.com duration: 315360000s renewBefore: 240h dnsNames: - hub.yiqisoft.com 安装自定义 Certificate 先创建两个 namespace，再安装脚本 kubectl create namespace harbor kubectl create namespace cattle-system kubectl apply -f issuer-cert.yaml 结果输出 clusterissuer.cert-manager.io/selfsigned-issuer created certificate.cert-manager.io/rancher-cert created certificate.cert-manager.io/hub-cert created 安装 Rancher 增加 Rancher Helm 仓库 helm repo add rancher-latest https://releases.rancher.com/server-charts/latest helm repo update 安装 Rancher 这个过程比较漫长，需要从互联网下载 helm install rancher rancher-latest/rancher \ --namespace cattle-system \ --set hostname=rancher.yiqisoft.com \ --set replicas=3 \ --set extraEnv[0].name=CATTLE_DATABASE_ENDPOINT \ --set extraEnv[0].value=database:3306 \ --set extraEnv[1].name=CATTLE_DATABASE_USER \ --set extraEnv[1].value=rancher \ --set extraEnv[2].name=CATTLE_DATABASE_PASSWORD \ --set extraEnv[2].value=talent \ --set extraEnv[3].name=CATTLE_DATABASE_NAME \ --set extraEnv[3].value=rancher 验证安装成功 kubectl get pods --namespace cattle-system 结果 NAME READY STATUS RESTARTS AGE rancher-567cc5c6ff-bzx42 1/1 Running 0 65m rancher-567cc5c6ff-mgvns 1/1 Running 0 86m rancher-567cc5c6ff-rbgjn 1/1 Running 0 86m rancher-webhook-77b49b9ff9-mp2dm 1/1 Running 0 142m 安装 Harbor Harbor 作为内部 docker hub 使用 使用 Ceph（推荐） 准备一个 Ceph 集群用于存储（这里省略）。 使用 NFS 简单实用，但是效率不高，安全性一般。 在一个文件服务器上部署 NFS，这里用 database 主机apt update apt install nfs-kernel-server -y mkdir -p /opt/nfs_share chown nobody:nogroup /opt/nfs_share chmod 777 /opt/nfs_share echo "/opt/nfs_share *(rw,sync,no_subtree_check)" >> /etc/exports exportfs -ra systemctl restart nfs-kernel-server 部署 pv 和 pvc 编辑 pv-pvc.yaml apiVersion: v1 kind: PersistentVolume metadata: name: nfs-pv spec: capacity: storage: 30Gi accessModes: - ReadWriteMany nfs: path: /opt/nfs_share server: database --- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: nfs-pvc spec: accessModes: - ReadWriteMany resources: requests: storage: 10Gi volumeName: nfs-pv storageClassName: "" 应用到 namespace harbor kubectl apply -f pv-pvc.yaml -n harbor 结果 persistentvolume/nfs-pv created persistentvolumeclaim/nfs-pvc created 查看结果，nfs-pvc 必须是 Bound kubectl get pvc -n harbor 结果 NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE nfs-pvc Bound nfs-pv 30Gi RWX <unset> 92s 查看 nfs-pv 必须是 Bound kubectl get pv -n harbor 结果 NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS VOLUMEATTRIBUTESCLASS REASON AGE nfs-pv 30Gi RWX Retain Bound harbor/nfs-pvc <unset> 97s 部署 Harbor Helm 安装 NFS 依赖 所有的 Master 上需要安装 nfs-common，并验证（可选），可以访问 /tmp/nfs 目录 apt install nfs-common -y mkdir /tmp/nfs mount -t nfs database:/opt/nfs_share /tmp/nfs ll /tmp/nfs 增加 Harbor Helm 仓库 helm repo add harbor https://helm.goharbor.io helm repo update 导出默认 values，修改 helm show values harbor/harbor > harbor-values.yml # 内容修改如下 expose: ingress: hosts: core: hub.yiqisoft.cn externalURL: https://hub.yiqisoft.cn persistentVolumeClaim: registry:/exter existingClaim: "nfs-pvc" storageClass: "-" database: type: external external: host: "192.168.198.100" port: "5432" username: "k3s" password: "talent" coreDatabase: "harbor" harborAdminPassword: "Harbor12345" 安装 Harbor，需要通过互联网下载所有的镜像，时间根据网络带宽决定 helm install harbor harbor/harbor -f harbor-values.yml -n harbor 确认 Harbor 是否安装成功 kubectl get pods -n harbor 都显示 READY 即可 NAME READY STATUS RESTARTS AGE harbor-core-666cf84bc6-vq8mp 1/1 Running 0 85s harbor-jobservice-c68bc669-6twbp 1/1 Running 0 85s harbor-portal-d6c99f896-cq9fx 1/1 Running 0 85s harbor-redis-0 1/1 Running 0 84s harbor-registry-5984c768c8-4xnrg 2/2 Running 0 84s harbor-trivy-0 1/1 Running 0 84s 验证 Harbor 工作状态 curl -k https://hub.yiqisoft.com/v2/ -u admin:Harbor12345 提示 {} 证明成功 {} 以下在 database 主机上运行 登录 Harbor docker login hub.yiqisoft.com -u admin -p Harbor12345 登录成功 Login Succeeded 上传已经存在的 PostgreSQL image 测试 docker tag postgres:alpine hub.yiqisoft.com/library/postgres:alpine docker push hub.yiqisoft.com/library/postgres:alpine 结果显示 The push refers to repository [hub.yiqisoft.com/library/postgres] 284f9fec9bcb: Pushed 07188499987f: Pushed ... 63ca1fbb43ae: Pushed alpine: digest: sha256:f0407919d966a86e9***c81641801e43c29b size: 2402 批量上传所需 image 到 Harbor 在准备好 docker image 的机器上运行 导出需要的 image docker images --format "{{.Repository}}:{{.Tag}}" |grep yiqisoft.com > images.txt 新建 /etc/docker/daemon.json，跳过安全证书 { "insecure-registries" : [ "0.0.0.0/0" ] } 重启 docker systemctl restart docker 新建一个脚本 push-images.sh #!/bin/bash # Check if the correct number of arguments is provided if [ $# -ne 1 ]; then echo "Usage: $0 <path_to_images_file>" exit 1 fi # Assign the first argument to the file variable file=$1 # Check if the file exists if [ ! -f "$file" ]; then echo "File not found: $file" exit 1 fi while read image; do echo "Pushing image: $image" docker push $image if [ $? -eq 0 ]; then echo "Successfully pushed $image" else echo "Failed to push $image" fi done < "$file" 登录 Harbor 并上传: push image docker login hub.yiqisoft.com -u admin -p Harbor12345 bash push-images.sh images.txt 查看 Harbor Registry 仓库结果 curl -k https://hub.yiqisoft.com/v2/_catalog -u admin:Harbor12345 |json_pp 结果显示 { "repositories" : [ "edgexfoundry/app-service-gateway", "edgexfoundry/consul", "edgexfoundry/core-command", "edgexfoundry/core-common-config-bootstrapper", "edgexfoundry/core-data", "edgexfoundry/core-metadata", "edgexfoundry/device-modbus", "edgexfoundry/device-onvif-camera", "edgexfoundry/device-openvino-face-recognition", "edgexfoundry/device-openvino-object-detection", "edgexfoundry/edgex-ui", "edgexfoundry/redis", "edgexfoundry/support-notifications", "edgexfoundry/support-scheduler", "library/hello-world", "library/postgres", "rancher/system-agent", "yiqisoft/media_server", "yiqisoft/model_server", "yiqisoft/nginx", "yiqisoft/nvr" ] } 部署 K3S 业务集群 以下操作注意：全部在新建的业务集群中操作 所有的 Master 和 Worker 都加入业务集群 Rancher 中创建一个新的集群 Rancher UI 操作（省略）, Cluster Name: yiqisoft, Kubernetes Version: 1.30.4+k3s1 Master 加入集群(初始化业务集群) 通过新集群的 Registration 加入，可以自定义一个 namespace。需要几个 Master 根据自己业务需要来定。 导入 hub CA echo | openssl s_client -showcerts -connect hub.yiqisoft.com:443 2>/dev/null | \ openssl x509 -outform PEM > /etc/ssl/certs/hub.yiqisoft.com.crt 分别在 master151-master153 主机上执行，主机名不能重复 curl --insecure -fL https://rancher.yiqisoft.com/system-agent-install.sh | \ sudo sh -s - --server https://rancher.yiqisoft.com --label 'cattle.io/os=linux' \ --token xft4fsrjgkn4rzqxrgrb4mztkcxtz4nhpwm58t84q54lvpsscdgd72 \ --ca-checksum 30d78ad0ece6355c40a4b3764ba02dfc96388a9c20f6b4e2bffb131cf7402c1f \ --etcd --controlplane --worker 查看业务集群 node kubectl get nodes 确认 Master 都在线 NAME STATUS ROLES AGE VERSION master151 Ready control-plane,etcd,master,worker 32m v1.30.4+k3s1 master152 Ready control-plane,etcd,master,worker 3h39m v1.30.4+k3s1 master153 Ready control-plane,etcd,master,worker 23m v1.30.4+k3s1 Worker 加入业务集群 首先通过 curl 直接输出 hub 证书到主机证书目录 containerd 拉取 image 时需要信任此证书 CA echo | openssl s_client -showcerts -connect hub.yiqisoft.com:443 2>/dev/null | \ openssl x509 -outform PEM > /etc/ssl/certs/hub.yiqisoft.com.crt 需要等 Master 加入成功后（至少一个成功注册），Worker 主机再次通过 Registration 加入，只需要选择 Worker 即可，需要定义一个 namespace label， 比如： namespace=worker01，以后可单独部署到独立应用 到 namespace 作为隔离使用。 在所有业务 Worker 上执行，同样注意主机名不能重复 curl --insecure -fL https://rancher.yiqisoft.com/system-agent-install.sh | \ sudo sh -s - \ --server https://rancher.yiqisoft.com \ --label 'cattle.io/os=linux' \ --token xft4fsrjgkn4rzqxrgrb4mztkcxtz4nhpwm58t84q54lvpsscdgd72 \ --ca-checksum 30d78ad0ece6355c40a4b3764ba02dfc96388a9c20f6b4e2bffb131cf7402c1f \ --worker \ --label namespace=worker161 验证 Worker 节点 kubectl get nodes -A 结果正常 NAME STATUS ROLES AGE VERSION master151 Ready control-plane,etcd,master,worker 74m v1.30.4+k3s1 master152 Ready control-plane,etcd,master,worker 4h21m v1.30.4+k3s1 master153 Ready control-plane,etcd,master,worker 65m v1.30.4+k3s1 worker161 Ready worker 25m v1.30.4+k3s1 (部署/更新) Harbor hub CA 证书到所有的 Master 和 Worker (手动操作)导出 Harbor hub HTTPS 证书 增加自签名证书到 ca-certificates 所有的 Master 和 Worker 上执行（不太可行）echo | openssl s_client -showcerts -connect hub.yiqisoft.com:443 2>/dev/null | \ openssl x509 -outform PEM > /etc/ssl/certs/hub.yiqisoft.com.crt (自动部署)Harbor Hub CA 证书 新建一个 Secret 在业务集群中，把 Harbor 证书导入集群 kube-system kubectl create secret generic hub-ca --from-file=ca.crt=/hub.yiqisoft.com.crt -n kube-system 查看 kubectl get secret -A |grep hub-ca 结果 NAMESPACE NAME TYPE DATA AGE kube-system hub-ca kubernetes.io/tls 2 101m 新建一个 DaemonSet 新建一个部署文件: ca-distributor.yaml，用于分发 hub 的证书，非常重要，否则无法从私有 Registry 拉取镜像。 apiVersion: apps/v1 kind: DaemonSet metadata: name: ca-distributor namespace: kube-system spec: selector: matchLabels: name: ca-distributor template: metadata: labels: name: ca-distributor spec: containers: - name: ca-distributor image: busybox:latest command: - /bin/sh - -c - "cp /etc/ca-certificates/tls.crt /etc/ssl/certs/hub.yiqisoft.cn.crt && echo 'Certificate Updated' && sleep 3600" volumeMounts: - name: ca-cert mountPath: /etc/ca-certificates - name: ssl-certs mountPath: /etc/ssl/certs volumes: - name: ca-cert secret: secretName: hub-ca - name: ssl-certs hostPath: path: /etc/ssl/certs type: Directory restartPolicy: Always 部署 yaml kubectl apply -f ca-distributor.yaml 验证结果,确保所有的节点都执行成功。 kubectl get pods -A |grep ca-distributor kube-system ca-distributor-ckhgh 1/1 Running 0 100s kube-system ca-distributor-hqzwt 1/1 Running 0 101s kube-system ca-distributor-lj72n. 1/1 Running 0 101s kube-system ca-distributor-mpp7m 1/1 Running 0 101s 在 Worker 上拉取镜像 image k3s ctr i pull hub.yiqisoft.com/library/hello-world:latest 验证结果 hub.yiqisoft.cn/library/hello-world:latest: resolved |++++++++++++++++++++++++++++++++++++++| manifest-sha256:d37ada95d47ad12224c205a938129df7a3e52345828b4fa27b03a98825d1e2e7: done |++++++++++++++++++++++++++++++++++++++| config-sha256:d2c94e258dcb3c5ac2798d32e1249e42ef01cba4841c2234249495f87264ac5a: done |++++++++++++++++++++++++++++++++++++++| layer-sha256:c1ec31eb59444d78df06a974d155e597c894ab4cda84f08294145e845394988e: done |++++++++++++++++++++++++++++++++++++++| elapsed: 0.6 s total: 524.0 (871.0 B/s) unpacking linux/amd64 sha256:d37ada95d47ad12224c205a938129df7a3e52345828b4fa27b03a98825d1e2e7... done: 54.873311ms 基于 K3S 部署业务应用(边缘计算框架) 以下所有部署都是通过 Helm Chart 工具进行，需要自行编写。 EdgeX Foundry Base 部署基本框架结果kubectl get pods -n worker161 NAME READY STATUS RESTARTS AGE edgex-core-command-tspw7 1/1 Running 0 5m41s edgex-core-common-config-bootstrapper-92q25 1/1 Running 0 5m25s edgex-core-consul-8lqxz 1/1 Running 0 6m4s edgex-core-data-nv26d 1/1 Running 0 6m23s edgex-core-metadata-l6c49 1/1 Running 0 5m3s edgex-device-onvif-camera-5tt4w 1/1 Running 0 3m26s edgex-mqtt-broker-j2jdr 1/1 Running 0 81s edgex-redis-rf4v7 1/1 Running 0 4m50s edgex-support-notifications-ws6l5 1/1 Running 0 4m28s edgex-support-scheduler-drx2p 1/1 Running 0 3m56s edgex-ui-v55vm 1/1 Running 0 3m45s 部署辅助应用：流媒体，推理服务器，NVR 部署辅助应用 业务 pod 之间有依赖关系，需等待其他 pod 起来才能工作，会不断的重试kubectl get pods -n worker161 |grep -v edgex 结果 NAME READY STATUS RESTARTS AGE media-server-wtjxk 1/1 Running 1 (5m29s ago) 13m model-server-9fpb2 1/1 Running 0 89s nvr-gqlst 1/1 Running 2 (5m29s ago) 11m 部署 OpenVINO AI 应用 部署基于 OpenVINO 推理框架的 AI 应用 podkubectl get pods -n worker161 |grep openvino 结果 NAME READY STATUS RESTARTS AGE edgex-device-openvino-face-recognition-bgch4 1/1 Running 0 5m43s edgex-device-openvino-object-detection-tbvwk 1/1 Running 0 7m21s 总结 通过本部署方案，可以管理基于 EdgeX 的边缘计算方案，EdgeX 具备边缘自治，即使脱离 K3S 集群，也照样能正常工作。 同时，此方案可以管理成千上万个边缘设备，之间互不干扰，适合大型项目部署。 云端配置相对较为复杂，但是边缘端就轻松很多，只需要手动注册到 K3S 集群，其他都可以通过 K8S API 进行部署和管理。 对比 K8S 而言，K3S 非常适合在边缘端工作，占用资源少，管理较为方便。 其实，最终结果就是：K3S 适合管理边缘容器分发。
如何在 Kubernetes 中部署 ThingsBoard？步骤很简单，过程很艰辛！
目录 关注我们 关注 ThingsBoard 微信公众号 访问 ThingsBoard 中文社区网站 众所周知，Kubernetes（k8s）太火了，太牛了，咱们把部署在 ThingsBoard 上更是理所当然。一句话总结：步骤很简单，过程很艰辛！ 准备 Docker 和 Kubernetes 环境 安装 Docker Desktop CE 版本 安装 Kubenetes 本地版本 kubectl version --client !/bin/bash get images show images !/bin/bash get images (ui) show images kubectl version kubectl apply -f k8s-dashboard.yaml kubectl proxy kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v0.41.2/deploy/static/provider/cloud/deploy.yaml kubectl apply -f ingress-nginx.yaml docker pull k8s.gcr.io/ingress-nginx/controller:v0.41.2@sha256:1f4f402b9c14f3ae92b11ada1dfe9893a88f0faeb0b2f4b903e2c67a0c3bf0de kubectl get pods -n ingress-nginx -l app.kubernetes.io/name=ingress-nginx --watch 关注我们 关注 ThingsBoard 微信公众号 手动增加微信公众号：thingsboard_cn 访问 ThingsBoard 中文社区网站 访问网址：http://www.thingsboard.club 众所周知，Kubernetes（k8s）太火了，太牛了，咱们把部署在 ThingsBoard 上更是理所当然。一句话总结：步骤很简单，过程很艰辛！ 准备 Docker 和 Kubernetes 环境 安装 Docker Desktop CE 版本 笔者使用的老版本 MacBook Pro 13，下载安装 Docker Desktop CE 版本，这个比较简单，官方下载，安装即可完成，启动。 ![docker.png][1] 启用 Docker Desktop 中 Kubernetes ![kubernetes-in-docker.png][2] 哇塞，是不是太简单了？当然不会那么容易搞定，你会一直出现 “Starting” 的状态，别傻等，要去 Google 一下。关键字：如何成功启动 Docker 自带的 Kubernetes？ 总之，你一定要找到办法解决你的 Kubernetes 成功启动。 安装 Kubenetes 本地版本 安装 Kubernetes，就是简单，本地安装搞定 # brew install kubernetes-cli kubectl version --client Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.4", GitCommit:"d360454c9bcd1634cf4cc52d1867af5491dc9c5f", GitTreeState:"clean", BuildDate:"2020-11-12T01:09:16Z", GoVersion:"go1.15.4", Compiler:"gc", Platform:"darwin/amd64"} - 为了让 Docker Desktop 启动 k8s，需下载相关的 docker images，需要通过一个脚本 docker-images-k8s.sh，然后运行，从 aliyun 下载 docker image 比较快。 !/bin/bash set -e KUBE_VERSION=v1.19.3 KUBE_PAUSE_VERSION=3.2 ETCD_VERSION=3.4.13-0 COREDNS_VERSION=1.7.0 GCR_URL=k8s.gcr.io ALIYUN_URL=registry.cn-hangzhou.aliyuncs.com/google_containers get images images=(kube-proxy:${KUBE_VERSION} kube-scheduler:${KUBE_VERSION} kube-controller-manager:${KUBE_VERSION} kube-apiserver:${KUBE_VERSION} pause:${KUBE_PAUSE_VERSION} etcd:${ETCD_VERSION} coredns:${COREDNS_VERSION}) for imageName in ${images[@]} ; do docker pull $ALIYUN_URL/$imageName docker tag $ALIYUN_URL/$imageName $GCR_URL/$imageName docker rmi $ALIYUN_URL/$imageName done show images docker images - 为了查看 k8s 运行状态，web 管理工具 kubernetes dashboard 自然少不了，新建 docker-images-dashboard.sh，运行 !/bin/bash set -e KUBE_DASHBOARD_VERSION=v2.0.3 METRICS_SCRAPER=v1.0.4 KUBENETESUI_URL=kubernetesui ALIYUN_KUBENETESUI_URL=registry.cn-hangzhou.aliyuncs.com/kubernetes_ns get images (ui) imagesui=(dashboard:${KUBE_DASHBOARD_VERSION} metrics-scraper:${METRICS_SCRAPER}) for imageName in ${imagesui[@]} ; do docker pull $ALIYUN_KUBENETESUI_URL/$imageName docker tag $ALIYUN_KUBENETESUI_URL/$imageName $KUBENETESUI_URL/$imageName docker rmi $ALIYUN_KUBENETESUI_URL/$imageName done show images docker images - 再去看 Docker Desktop 中已经启动 k8s了，耐心一点，这个时间比较长，验证一下，发现 Server Version 出现了证明服务器安装成功，且客户端可以连接上 kubectl version Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.4", GitCommit:"d360454c9bcd1634cf4cc52d1867af5491dc9c5f", GitTreeState:"clean", BuildDate:"2020-11-12T01:09:16Z", GoVersion:"go1.15.4", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.3", GitCommit:"1e11e4a2108024935ecfcb2912226cedeafd99df", GitTreeState:"clean", BuildDate:"2020-10-14T12:41:49Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"linux/amd64"} - 启用 k8s dashboard，需要调用一个 yaml [k8s-dashboard.yaml][3] kubectl apply -f k8s-dashboard.yaml namespace/kubernetes-dashboard created serviceaccount/kubernetes-dashboard created service/kubernetes-dashboard created secret/kubernetes-dashboard-certs created secret/kubernetes-dashboard-csrf created secret/kubernetes-dashboard-key-holder created configmap/kubernetes-dashboard-settings created role.rbac.authorization.k8s.io/kubernetes-dashboard created clusterrole.rbac.authorization.k8s.io/kubernetes-dashboard created rolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created clusterrolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created deployment.apps/kubernetes-dashboard created service/dashboard-metrics-scraper created deployment.apps/dashboard-metrics-scraper created kubectl proxy Starting to serve on 127.0.0.1:8001 - 启动成功后，打开浏览器，就可以登陆 Dashboard 要么证书登陆，要么 token 登陆，先执行： TOKEN=$(kubectl -n kube-system describe secret default| awk '$1=="token:"{print $2}') kubectl config set-credentials docker-desktop --token="${TOKEN}" echo $TOKEN 显示的内容就是 token 1. 证书登陆时，选择 ～/.kube/config 2. token 登陆就是上面那串 ![k8s-dashbaord.png][4] 登陆成功后 ![k8s-dashbaord-home.png][5] # 安装 ThingsBoard 到 k8s ## 安装 ThingsBoard > 根据官方文档，照着执行就好：https://github.com/thingsboard/thingsboard/tree/master/k8s - 保证网络畅通，因为需要从 docker hub 下载 images，建议用国内的镜像地址 - 过程比较麻烦，不好监控，只能不停的刷 docker images 命令查看是不是把镜像拉下来了 REPOSITORY TAG IMAGE ID CREATED SIZE zookeeper 3.5 ce6733a90146 3 days ago 245MB postgres 12 386fd8c60839 3 days ago 314MB k8s.gcr.io/ingress-nginx/controller 81d7cdfa4169 9 days ago 280MB jettech/kube-webhook-certgen v1.5.0 344297e197b6 3 weeks ago 44.7MB docker/desktop-kubernetes kubernetes-v1.19.3-cni-v0.8.5-critools-v1.17.0 7f85afe431d8 5 weeks ago 285MB k8s.gcr.io/kube-proxy v1.19.3 cdef7632a242 5 weeks ago 118MB k8s.gcr.io/kube-apiserver v1.19.3 a301be0cd44b 5 weeks ago 119MB k8s.gcr.io/kube-scheduler v1.19.3 aaefbfa906bd 5 weeks ago 45.7MB k8s.gcr.io/kube-controller-manager v1.19.3 9b60aca1d818 5 weeks ago 111MB thingsboard/tb-coap-transport latest 5635dd5c4fe6 2 months ago 865MB thingsboard/tb-http-transport latest 7f60b6a3bd83 2 months ago 864MB thingsboard/tb-mqtt-transport latest 8dfe9a5bc20d 2 months ago 864MB thingsboard/tb-node latest a66803235c46 2 months ago 1.09GB thingsboard/tb-web-ui latest e215f4aa841a 2 months ago 426MB thingsboard/tb-js-executor latest 7c4fdfa00e27 2 months ago 494MB k8s.gcr.io/etcd 3.4.13-0 0369cf4303ff 2 months ago 253MB kubernetesui/dashboard v2.0.3 503bc4b7440b 5 months ago 225MB k8s.gcr.io/coredns 1.7.0 bfe3a36ebd25 5 months ago 45.2MB redis 4.0 191c4017dcdd 7 months ago 89.3MB kubernetesui/metrics-scraper v1.0.4 86262685d9ab 8 months ago 36.9MB docker/desktop-storage-provisioner v1.1 e704287ce753 8 months ago 41.8MB docker/desktop-vpnkit-controller v1.0 79da37e5a3aa 8 months ago 36.6MB k8s.gcr.io/pause 3.2 80d28bedfe5d 9 months ago 683kB wurstmeister/kafka 2.12-2.2.1 86a9c7e5fd97 11 months ago 422MB - 过了很长一段时间，有可能几个小时后，你查看 docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES a9d8b9a28efc e704287ce753 "/storage-provisione…" 6 minutes ago Up 6 minutes k8s_storage-provisioner_storage-provisioner_kube-system_50005a87-62d2-45a0-a39e-c34af43d24c9_3 83e1a62bb612 aaefbfa906bd "kube-scheduler --au…" 6 minutes ago Up 6 minutes k8s_kube-scheduler_kube-scheduler-docker-desktop_kube-system_ee4c94eb845abf1878fb3c4c489b1365_7 b4e4d52ae0bc thingsboard/tb-node "start-tb-node.sh" 7 hours ago Up 7 hours k8s_server_tb-node-5b47bb8bd6-x8g8b_thingsboard_532519fd-7441-4a10-a0b3-3458bb9784a3_5 462b9a657c4f thingsboard/tb-node "start-tb-node.sh" 7 hours ago Up 7 hours k8s_server_tb-node-5b47bb8bd6-vqrvb_thingsboard_3f25d857-1624-4d27-90a5-2c54c35747ee_5 4f3a74233acd thingsboard/tb-js-executor "start-js-executor.sh" 7 hours ago Up 7 hours k8s_server_tb-js-executor-56bbb8f446-vvb9v_thingsboard_46063547-8f06-4e81-85b4-00cb4b2a5f42_7 211329243cdd thingsboard/tb-js-executor "start-js-executor.sh" 7 hours ago Up 7 hours k8s_server_tb-js-executor-56bbb8f446-l5gmw_thingsboard_29f294c1-5fd0-44a5-98a0-3880b6fe7126_7 3a6f946e86be thingsboard/tb-js-executor "start-js-executor.sh" 7 hours ago Up 7 hours k8s_server_tb-js-executor-56bbb8f446-h8zqk_thingsboard_82bc9e2b-5db7-409a-9be5-4e7c21f481b8_6 28085da4bbae thingsboard/tb-js-executor "start-js-executor.sh" 7 hours ago Up 7 hours k8s_server_tb-js-executor-56bbb8f446-ftwf8_thingsboard_0c3f60a2-fd08-4951-9d98-80d08bfa439c_6 ed33cf23175c thingsboard/tb-js-executor "start-js-executor.sh" 7 hours ago Up 7 hours k8s_server_tb-js-executor-56bbb8f446-h2vkl_thingsboard_990be16d-12d8-4f7d-8ce4-1ef4a83ac31b_7 6b7fc0bd2fbb thingsboard/tb-js-executor "start-js-executor.sh" 7 hours ago Up 7 hours k8s_server_tb-js-executor-56bbb8f446-pwc4z_thingsboard_28c0106c-9587-466f-90b3-b3af6eaed7ea_7 520ed4711677 thingsboard/tb-js-executor "start-js-executor.sh" 7 hours ago Up 7 hours k8s_server_tb-js-executor-56bbb8f446-gvx2g_thingsboard_afeb8b5c-ab53-48ba-927f-2d20b5fc6c3b_6 5baeda96c4a3 thingsboard/tb-js-executor "start-js-executor.sh" 7 hours ago Up 7 hours k8s_server_tb-js-executor-56bbb8f446-2z89t_thingsboard_d7ec851c-8648-465a-a855-0a23575e3c3b_6 81f692bc2768 thingsboard/tb-js-executor "start-js-executor.sh" 7 hours ago Up 7 hours k8s_server_tb-js-executor-56bbb8f446-7rl82_thingsboard_08864e79-2af0-42b5-b592-4c1a2db1d5cd_6 cb8bf860467a thingsboard/tb-js-executor "start-js-executor.sh" 7 hours ago Up 7 hours k8s_server_tb-js-executor-56bbb8f446-ljwz8_thingsboard_92de7aab-fca3-4e0c-ac3e-02213603d359_7 917ba1ee6732 thingsboard/tb-js-executor "start-js-executor.sh" 7 hours ago Up 7 hours k8s_server_tb-js-executor-56bbb8f446-c27qd_thingsboard_0c79ed1e-1d65-4d46-a7b1-08de77402549_6 0ada7d6b48b9 thingsboard/tb-js-executor "start-js-executor.sh" 7 hours ago Up 7 hours k8s_server_tb-js-executor-56bbb8f446-k4ncf_thingsboard_cdd283fc-d0a2-45b6-b533-00dd4d38e782_7 ffc91b31b24c thingsboard/tb-js-executor "start-js-executor.sh" 7 hours ago Up 7 hours k8s_server_tb-js-executor-56bbb8f446-5p9tg_thingsboard_87e80c57-0f84-43eb-bf24-13cc309e4cec_7 13b6d7dfb387 thingsboard/tb-js-executor "start-js-executor.sh" 7 hours ago Up 7 hours k8s_server_tb-js-executor-56bbb8f446-zvgrp_thingsboard_1fad5bb8-5793-48e7-8dcc-7cbaf0475192_6 0f63ffec3cbb thingsboard/tb-js-executor "start-js-executor.sh" 7 hours ago Up 7 hours k8s_server_tb-js-executor-56bbb8f446-fvvrx_thingsboard_88c65221-c5d3-419d-a155-21cfee3b7157_7 29c432f70609 thingsboard/tb-js-executor "start-js-executor.sh" 7 hours ago Up 7 hours k8s_server_tb-js-executor-56bbb8f446-zvd2t_thingsboard_e0315452-8b8f-4c0a-9488-d56e296ec962_7 58a2f29e2f6a thingsboard/tb-js-executor "start-js-executor.sh" 7 hours ago Up 7 hours k8s_server_tb-js-executor-56bbb8f446-gpndh_thingsboard_99c29bd4-cb0f-4fb8-8061-ebdda21359de_7 637a32ae8d4d thingsboard/tb-js-executor "start-js-executor.sh" 7 hours ago Up 7 hours k8s_server_tb-js-executor-56bbb8f446-r2r5b_thingsboard_6ba2cc34-d6bb-4c14-96c8-0c11e20fd051_7 0d49592908fd thingsboard/tb-js-executor "start-js-executor.sh" 7 hours ago Up 7 hours k8s_server_tb-js-executor-56bbb8f446-j756s_thingsboard_0be1997a-4d1b-4d01-a3ab-55d6e468f39d_7 5e342a30e5c6 thingsboard/tb-js-executor "start-js-executor.sh" 7 hours ago Up 7 hours k8s_server_tb-js-executor-56bbb8f446-9mmf2_thingsboard_97fc3fa7-ad20-4ffc-a34a-cda7aba93305_7 044277041f7a thingsboard/tb-mqtt-transport "start-tb-mqtt-trans…" 7 hours ago Up 7 hours k8s_server_tb-mqtt-transport-fb5f5d5cd-xkwcn_thingsboard_d91bc9c9-de41-4866-8fe1-03831af3a58c_3 e7b427121ff2 thingsboard/tb-http-transport "start-tb-http-trans…" 7 hours ago Up 7 hours k8s_server_tb-http-transport-64f6f66bc7-zzvqb_thingsboard_520ed11c-239e-4083-919b-fec38277f233_3 a2a4da430f0d thingsboard/tb-http-transport "start-tb-http-trans…" 7 hours ago Up 7 hours k8s_server_tb-http-transport-64f6f66bc7-ztmng_thingsboard_0e628481-0398-4ddd-ad97-31c5987796f9_3 2749cf276c80 wurstmeister/kafka "start-kafka.sh" 7 hours ago Up 7 hours k8s_server_tb-kafka-78957b6557-fc8js_thingsboard_2a3d3e60-0b40-45ba-91da-70f29f270a04_5 f296ce350482 thingsboard/tb-mqtt-transport "start-tb-mqtt-trans…" 7 hours ago Up 7 hours k8s_server_tb-mqtt-transport-fb5f5d5cd-bs4tw_thingsboard_ef25b3b1-658e-4e4c-bd1c-a0ca2ab20ce4_3 d7ebb30d2fc7 postgres "docker-entrypoint.s…" 7 hours ago Up 7 hours k8s_postgres_postgres-5cf775766f-clf89_thingsboard_599ecf9f-068c-416e-ab73-95760f1099db_1 2ad0a6662ca9 thingsboard/tb-coap-transport "start-tb-coap-trans…" 7 hours ago Up 7 hours k8s_server_tb-coap-transport-59cc8bd7bc-82zfk_thingsboard_73404a55-e170-4441-b733-5417db6461ad_1 33c4da97f3d7 thingsboard/tb-coap-transport "start-tb-coap-trans…" 7 hours ago Up 7 hours k8s_server_tb-coap-transport-59cc8bd7bc-g8jdw_thingsboard_a48518d7-c929-4a1f-bcc8-c16fdf903fbf_1 3f8d5b8e9d9f thingsboard/tb-web-ui "start-web-ui.sh" 7 hours ago Up 7 hours k8s_server_tb-web-ui-5958f67cc-lg974_thingsboard_5de819ae-f832-4ab6-bdce-0f29daf037ff_1 beb462957a30 redis "docker-entrypoint.s…" 7 hours ago Up 7 hours k8s_server_tb-redis-7fb674db9-pzndj_thingsboard_6ff6ba21-933d-4366-89eb-3d815f5282c9_1 e24d2ea2e00a thingsboard/tb-web-ui "start-web-ui.sh" 7 hours ago Up 7 hours k8s_server_tb-web-ui-5958f67cc-4k4cj_thingsboard_6f3795e9-ded9-46b8-8067-b4afd63f2571_1 c2161ce93009 zookeeper "/docker-entrypoint.…" 7 hours ago Up 7 hours k8s_server_zookeeper-547c6ffb99-bq42m_thingsboard_11a1e06d-bbf1-4b55-bab5-7d4e4b876f20_1 4e02d9bde8ab 79da37e5a3aa "/kube-vpnkit-forwar…" 7 hours ago Up 7 hours k8s_vpnkit-controller_vpnkit-controller_kube-system_6fa3177c-f779-414d-8acb-1342dd87849b_1 30f7af7ffa00 kubernetesui/dashboard "/dashboard --insecu…" 7 hours ago Up 7 hours k8s_kubernetes-dashboard_kubernetes-dashboard-5dbf55bd9d-bcc2v_kubernetes-dashboard_1e414605-d8c1-4918-8e9b-18bc2c01efe6_1 517bc1b7ac3e 81d7cdfa4169 "/usr/bin/dumb-init …" 7 hours ago Up 7 hours k8s_controller_ingress-nginx-controller-c4f944d4d-wcqxv_ingress-nginx_329aa6a7-2ee5-4747-b77d-82182d9a872f_1 d674f87354ca 86262685d9ab "/metrics-sidecar" 7 hours ago Up 7 hours k8s_dashboard-metrics-scraper_dashboard-metrics-scraper-7b59f7d4df-5dkwl_kubernetes-dashboard_3235eff1-3822-4138-8d0e-d45de95d3cbf_1 faf05611eaa3 k8s.gcr.io/pause:3.2 "/pause" 7 hours ago Up 7 hours k8s_POD_tb-js-executor-56bbb8f446-gvx2g_thingsboard_afeb8b5c-ab53-48ba-927f-2d20b5fc6c3b_1 1a1956a65635 k8s.gcr.io/pause:3.2 "/pause" 7 hours ago Up 7 hours k8s_POD_tb-coap-transport-59cc8bd7bc-82zfk_thingsboard_73404a55-e170-4441-b733-5417db6461ad_1 d5a382ca986e k8s.gcr.io/pause:3.2 "/pause" 7 hours ago Up 7 hours k8s_POD_tb-js-executor-56bbb8f446-pwc4z_thingsboard_28c0106c-9587-466f-90b3-b3af6eaed7ea_1 c80049396050 k8s.gcr.io/pause:3.2 "/pause" 7 hours ago Up 7 hours k8s_POD_tb-kafka-78957b6557-fc8js_thingsboard_2a3d3e60-0b40-45ba-91da-70f29f270a04_1 7b192ede32e5 k8s.gcr.io/pause:3.2 "/pause" 7 hours ago Up 7 hours k8s_POD_tb-coap-transport-59cc8bd7bc-g8jdw_thingsboard_a48518d7-c929-4a1f-bcc8-c16fdf903fbf_1 c33e9576e5a9 k8s.gcr.io/pause:3.2 "/pause" 7 hours ago Up 7 hours k8s_POD_tb-http-transport-64f6f66bc7-zzvqb_thingsboard_520ed11c-239e-4083-919b-fec38277f233_1 6e90c5e851b7 k8s.gcr.io/pause:3.2 "/pause" 7 hours ago Up 7 hours k8s_POD_tb-js-executor-56bbb8f446-h2vkl_thingsboard_990be16d-12d8-4f7d-8ce4-1ef4a83ac31b_1 e74011796607 k8s.gcr.io/pause:3.2 "/pause" 7 hours ago Up 7 hours k8s_POD_tb-js-executor-56bbb8f446-ftwf8_thingsboard_0c3f60a2-fd08-4951-9d98-80d08bfa439c_1 8e7b08eaf0e4 k8s.gcr.io/pause:3.2 "/pause" 7 hours ago Up 7 hours k8s_POD_tb-js-executor-56bbb8f446-7rl82_thingsboard_08864e79-2af0-42b5-b592-4c1a2db1d5cd_1 fd3c552a04ee k8s.gcr.io/pause:3.2 "/pause" 7 hours ago Up 7 hours k8s_POD_tb-js-executor-56bbb8f446-vvb9v_thingsboard_46063547-8f06-4e81-85b4-00cb4b2a5f42_1 d3dc754a1f71 k8s.gcr.io/pause:3.2 "/pause" 7 hours ago Up 7 hours k8s_POD_tb-js-executor-56bbb8f446-l5gmw_thingsboard_29f294c1-5fd0-44a5-98a0-3880b6fe7126_1 372fd6514108 k8s.gcr.io/pause:3.2 "/pause" 7 hours ago Up 7 hours k8s_POD_tb-js-executor-56bbb8f446-gpndh_thingsboard_99c29bd4-cb0f-4fb8-8061-ebdda21359de_1 caf096a5bf42 k8s.gcr.io/pause:3.2 "/pause" 7 hours ago Up 7 hours k8s_POD_tb-node-5b47bb8bd6-x8g8b_thingsboard_532519fd-7441-4a10-a0b3-3458bb9784a3_1 a80bbd2267ec bfe3a36ebd25 "/coredns -conf /etc…" 7 hours ago Up 7 hours k8s_coredns_coredns-f9fd979d6-ff45j_kube-system_df1c5b8b-26f5-48cc-a853-c050090e69ef_1 1f1b01587d81 k8s.gcr.io/pause:3.2 "/pause" 7 hours ago Up 7 hours k8s_POD_tb-redis-7fb674db9-pzndj_thingsboard_6ff6ba21-933d-4366-89eb-3d815f5282c9_1 992b5afa6aa1 k8s.gcr.io/pause:3.2 "/pause" 7 hours ago Up 7 hours k8s_POD_tb-web-ui-5958f67cc-lg974_thingsboard_5de819ae-f832-4ab6-bdce-0f29daf037ff_1 271ad2fced2b k8s.gcr.io/pause:3.2 "/pause" 7 hours ago Up 7 hours k8s_POD_tb-js-executor-56bbb8f446-fvvrx_thingsboard_88c65221-c5d3-419d-a155-21cfee3b7157_1 344ddec06f1e k8s.gcr.io/pause:3.2 "/pause" 7 hours ago Up 7 hours k8s_POD_tb-js-executor-56bbb8f446-h8zqk_thingsboard_82bc9e2b-5db7-409a-9be5-4e7c21f481b8_1 6ad3af03d452 k8s.gcr.io/pause:3.2 "/pause" 7 hours ago Up 7 hours k8s_POD_vpnkit-controller_kube-system_6fa3177c-f779-414d-8acb-1342dd87849b_1 413d3e3fef5f k8s.gcr.io/pause:3.2 "/pause" 7 hours ago Up 7 hours k8s_POD_tb-http-transport-64f6f66bc7-ztmng_thingsboard_0e628481-0398-4ddd-ad97-31c5987796f9_1 b42279032251 k8s.gcr.io/pause:3.2 "/pause" 7 hours ago Up 7 hours k8s_POD_tb-js-executor-56bbb8f446-zvd2t_thingsboard_e0315452-8b8f-4c0a-9488-d56e296ec962_1 472cd017176e bfe3a36ebd25 "/coredns -conf /etc…" 7 hours ago Up 7 hours k8s_coredns_coredns-f9fd979d6-k54r2_kube-system_9355c0e5-b8fc-4c3c-9644-89749adeee97_1 3e0fe843bcbe k8s.gcr.io/pause:3.2 "/pause" 7 hours ago Up 7 hours k8s_POD_postgres-5cf775766f-clf89_thingsboard_599ecf9f-068c-416e-ab73-95760f1099db_1 e199dac37dec k8s.gcr.io/pause:3.2 "/pause" 7 hours ago Up 7 hours k8s_POD_tb-mqtt-transport-fb5f5d5cd-xkwcn_thingsboard_d91bc9c9-de41-4866-8fe1-03831af3a58c_1 e72505d9d064 k8s.gcr.io/pause:3.2 "/pause" 7 hours ago Up 7 hours k8s_POD_tb-js-executor-56bbb8f446-ljwz8_thingsboard_92de7aab-fca3-4e0c-ac3e-02213603d359_1 43362e2031d8 k8s.gcr.io/pause:3.2 "/pause" 7 hours ago Up 7 hours k8s_POD_tb-js-executor-56bbb8f446-2z89t_thingsboard_d7ec851c-8648-465a-a855-0a23575e3c3b_1 9c876bbc23be k8s.gcr.io/pause:3.2 "/pause" 7 hours ago Up 7 hours k8s_POD_tb-js-executor-56bbb8f446-c27qd_thingsboard_0c79ed1e-1d65-4d46-a7b1-08de77402549_1 11b89fccd02d k8s.gcr.io/pause:3.2 "/pause" 7 hours ago Up 7 hours k8s_POD_tb-node-5b47bb8bd6-vqrvb_thingsboard_3f25d857-1624-4d27-90a5-2c54c35747ee_1 9a047136e01b k8s.gcr.io/pause:3.2 "/pause" 7 hours ago Up 7 hours k8s_POD_tb-js-executor-56bbb8f446-k4ncf_thingsboard_cdd283fc-d0a2-45b6-b533-00dd4d38e782_1 c13f8513edf1 k8s.gcr.io/pause:3.2 "/pause" 7 hours ago Up 7 hours k8s_POD_tb-js-executor-56bbb8f446-zvgrp_thingsboard_1fad5bb8-5793-48e7-8dcc-7cbaf0475192_1 0fada7a20fe0 k8s.gcr.io/pause:3.2 "/pause" 7 hours ago Up 7 hours k8s_POD_tb-js-executor-56bbb8f446-j756s_thingsboard_0be1997a-4d1b-4d01-a3ab-55d6e468f39d_1 3e8bd573933b k8s.gcr.io/pause:3.2 "/pause" 7 hours ago Up 7 hours k8s_POD_tb-js-executor-56bbb8f446-5p9tg_thingsboard_87e80c57-0f84-43eb-bf24-13cc309e4cec_1 936695bf38e1 k8s.gcr.io/pause:3.2 "/pause" 7 hours ago Up 7 hours k8s_POD_tb-web-ui-5958f67cc-4k4cj_thingsboard_6f3795e9-ded9-46b8-8067-b4afd63f2571_1 15bc2206a101 k8s.gcr.io/pause:3.2 "/pause" 7 hours ago Up 7 hours k8s_POD_ingress-nginx-controller-c4f944d4d-wcqxv_ingress-nginx_329aa6a7-2ee5-4747-b77d-82182d9a872f_1 9fb9bcc3f42e k8s.gcr.io/pause:3.2 "/pause" 7 hours ago Up 7 hours k8s_POD_tb-mqtt-transport-fb5f5d5cd-bs4tw_thingsboard_ef25b3b1-658e-4e4c-bd1c-a0ca2ab20ce4_1 e5ff72d6497d cdef7632a242 "/usr/local/bin/kube…" 7 hours ago Up 7 hours k8s_kube-proxy_kube-proxy-4fx4m_kube-system_61c86591-198c-4bcc-ac9a-756ba949c71c_1 e8b88981ec37 k8s.gcr.io/pause:3.2 "/pause" 7 hours ago Up 7 hours k8s_POD_zookeeper-547c6ffb99-bq42m_thingsboard_11a1e06d-bbf1-4b55-bab5-7d4e4b876f20_1 2366f0c673fc k8s.gcr.io/pause:3.2 "/pause" 7 hours ago Up 7 hours k8s_POD_tb-js-executor-56bbb8f446-r2r5b_thingsboard_6ba2cc34-d6bb-4c14-96c8-0c11e20fd051_1 dc3c77cd895b k8s.gcr.io/pause:3.2 "/pause" 7 hours ago Up 7 hours k8s_POD_tb-js-executor-56bbb8f446-9mmf2_thingsboard_97fc3fa7-ad20-4ffc-a34a-cda7aba93305_1 7cef6c1b6fd8 k8s.gcr.io/pause:3.2 "/pause" 7 hours ago Up 7 hours k8s_POD_coredns-f9fd979d6-ff45j_kube-system_df1c5b8b-26f5-48cc-a853-c050090e69ef_1 7f3bf1c89eef k8s.gcr.io/pause:3.2 "/pause" 7 hours ago Up 7 hours k8s_POD_dashboard-metrics-scraper-7b59f7d4df-5dkwl_kubernetes-dashboard_3235eff1-3822-4138-8d0e-d45de95d3cbf_1 cedd269cda38 k8s.gcr.io/pause:3.2 "/pause" 7 hours ago Up 7 hours k8s_POD_coredns-f9fd979d6-k54r2_kube-system_9355c0e5-b8fc-4c3c-9644-89749adeee97_1 1ff5b8974607 k8s.gcr.io/pause:3.2 "/pause" 7 hours ago Up 7 hours k8s_POD_kubernetes-dashboard-5dbf55bd9d-bcc2v_kubernetes-dashboard_1e414605-d8c1-4918-8e9b-18bc2c01efe6_1 38c2265588a8 k8s.gcr.io/pause:3.2 "/pause" 7 hours ago Up 7 hours k8s_POD_storage-provisioner_kube-system_50005a87-62d2-45a0-a39e-c34af43d24c9_1 47dfcf754624 k8s.gcr.io/pause:3.2 "/pause" 7 hours ago Up 7 hours k8s_POD_kube-proxy-4fx4m_kube-system_61c86591-198c-4bcc-ac9a-756ba949c71c_1 2410f48b0dcb 0369cf4303ff "etcd --advertise-cl…" 7 hours ago Up 7 hours k8s_etcd_etcd-docker-desktop_kube-system_c7cc6a3c3118f127f5fd469ef69477e0_1 8282e8feb792 a301be0cd44b "kube-apiserver --ad…" 7 hours ago Up 7 hours k8s_kube-apiserver_kube-apiserver-docker-desktop_kube-system_2085f5be68201a12aeffb940f7ba4272_1 a8d6bd3cc82a 9b60aca1d818 "kube-controller-man…" 7 hours ago Up 7 hours k8s_kube-controller-manager_kube-controller-manager-docker-desktop_kube-system_07d42b7bf3823b30f57219558fcb55e2_1 acebbb6ddd18 k8s.gcr.io/pause:3.2 "/pause" 7 hours ago Up 7 hours k8s_POD_kube-scheduler-docker-desktop_kube-system_ee4c94eb845abf1878fb3c4c489b1365_1 d395721ce499 k8s.gcr.io/pause:3.2 "/pause" 7 hours ago Up 7 hours k8s_POD_kube-controller-manager-docker-desktop_kube-system_07d42b7bf3823b30f57219558fcb55e2_1 ff1cfd6ae1b2 k8s.gcr.io/pause:3.2 "/pause" 7 hours ago Up 7 hours k8s_POD_kube-apiserver-docker-desktop_kube-system_2085f5be68201a12aeffb940f7ba4272_1 2060020e4bc4 k8s.gcr.io/pause:3.2 "/pause" 7 hours ago Up 7 hours k8s_POD_etcd-docker-desktop_kube-system_c7cc6a3c3118f127f5fd469ef69477e0_1 好多个 container 在跑，好壮观 > 这时，你以为可以成功访问 ThingsBoard 主页吗？不行，真的不行，因为还差一个步骤，非常重要，那就是你的机器还没有 ingress ，建议你去好好读一读这个知识点。 ## 安装 Ingress Nginx - 直接去官方网站看文档，照着执行，但是你需要科学上网：https://github.com/kubernetes/ingress-nginx/blob/master/docs/deploy/index.md kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v0.41.2/deploy/static/provider/cloud/deploy.yaml 想办法把这个文件搞下来，因为这是部署 Ingress Nginx 的配置文件 我提供一个备份： [ingress-nginx.yaml][6]，启动： kubectl apply -f ingress-nginx.yaml namespace/ingress-nginx unchanged serviceaccount/ingress-nginx unchanged configmap/ingress-nginx-controller configured clusterrole.rbac.authorization.k8s.io/ingress-nginx unchanged clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx unchanged role.rbac.authorization.k8s.io/ingress-nginx unchanged rolebinding.rbac.authorization.k8s.io/ingress-nginx unchanged service/ingress-nginx-controller-admission unchanged service/ingress-nginx-controller unchanged deployment.apps/ingress-nginx-controller configured validatingwebhookconfiguration.admissionregistration.k8s.io/ingress-nginx-admission configured serviceaccount/ingress-nginx-admission unchanged clusterrole.rbac.authorization.k8s.io/ingress-nginx-admission unchanged clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx-admission unchanged role.rbac.authorization.k8s.io/ingress-nginx-admission unchanged rolebinding.rbac.authorization.k8s.io/ingress-nginx-admission unchanged job.batch/ingress-nginx-admission-create unchanged job.batch/ingress-nginx-admission-patch unchanged - 下载 ingress-nginx 镜像 docker pull k8s.gcr.io/ingress-nginx/controller:v0.41.2@sha256:1f4f402b9c14f3ae92b11ada1dfe9893a88f0faeb0b2f4b903e2c67a0c3bf0de 因为系统提示，这个镜像无法下载，或这很慢，你需要手动执行。看看有没有成功： kubectl get pods -n ingress-nginx -l app.kubernetes.io/name=ingress-nginx --watch NAME READY STATUS RESTARTS AGE ingress-nginx-admission-create-sd5zj 0/1 Completed 0 16m ingress-nginx-admission-patch-pfjhp 0/1 Completed 2 16m ingress-nginx-controller-c4f944d4d-wcqxv 0/1 Running 0 16m ingress-nginx-controller-c4f944d4d-wcqxv 1/1 Running 0 16m - 查看本地是否已经成功监听 80 和 443 端口，如果存在，那就是跑通了 netstat -na |grep LIST tcp46 0 0 .1883 . LISTEN tcp46 0 0 .443 . LISTEN tcp46 0 0 .80 .* LISTEN - 查看 k8s dashboard 状态 ![tb-in-k8s.png][7] - ThingsBoard 登陆界面，出现太重要了 ![tg-login.png][8] - 成功登陆，且新建租户正常 ![tb-tenant.png][9] ## 最后，不建议在本地跑 k8s，速度太慢了，不如跑 jar 来的方便，调试起来也不方便。开发环境用 mvn spring-boot:run，测试环境直接部署在 CentOS 或 Ubuntu 环境单机跑足够用了，大量数据采集，访问用户数量大，才可以试试 k8s 环境，欢迎大家留言讨论！ ## [1]: https://yiqisoft.cn/blogs/usr/uploads/2020/11/3847183869.png [2]: https://yiqisoft.cn/blogs/usr/uploads/2020/11/1301930728.png [3]: https://yiqisoft.cn/blogs/usr/uploads/2020/11/1712863204.yaml [4]: https://yiqisoft.cn/blogs/usr/uploads/2020/11/15528435.png [5]: https://yiqisoft.cn/blogs/usr/uploads/2020/11/746719120.png [6]: https://yiqisoft.cn/blogs/usr/uploads/2020/11/2160153585.yaml [7]: https://yiqisoft.cn/blogs/usr/uploads/2020/11/1052366375.png [8]: https://yiqisoft.cn/blogs/usr/uploads/2020/11/310291162.png [9]: https://yiqisoft.cn/blogs/usr/uploads/2020/11/2850801996.png