linuxea:habor2.5的helm仓库和镜像仓库使用(5) 在早先，容器仓库有Portus和harbor最为瞩目，harbor是vmware的产品，而前者则是由suse团队维护，但在2019portus提供了最后一个版本。但随着时间的推移，harbor提供了更多与时俱进的功能服务，这也导致harbor愈发的收到关注和使用。并且harbor是由vmware中国区成员参与，众所周知，一款提供中文界面且有优秀的产品，往往都是更受欢迎的。harbor不仅可以存放容器镜像，还可以提供helm的仓库，简单列出他具备的功能云原生注册表：支持容器镜像和Helm图表，Harbor 充当云原生环境（如容器运行时和编排平台）的注册表。基于角色的访问控制：用户通过“项目”访问不同的存储库，并且用户可以对项目下的图像或 Helm 图表具有不同的权限。基于策略的复制：可以使用过滤器（存储库、标签和标签）基于策略在多个注册表实例之间复制（同步）图像和图表。如果遇到任何错误，Harbor 会自动重试复制。这可用于辅助负载平衡、实现高可用性以及促进混合和多云场景中的多数据中心部署。漏洞扫描：Harbor 定期扫描映像以查找漏洞，并进行策略检查以防止部署易受攻击的映像。LDAP/AD 支持：Harbor 与现有的企业 LDAP/AD 集成以进行用户身份验证和管理，并支持将 LDAP 组导入 Harbor，然后可以授予特定项目的权限。OIDC 支持：Harbor 利用 OpenID Connect (OIDC) 来验证由外部授权服务器或身份提供者认证的用户的身份。可以启用单点登录以登录到 Harbor 门户。图像删除和垃圾收集：系统管理员可以运行垃圾收集作业，以便可以删除图像（悬空清单和未引用的 blob）并定期释放它们的空间。Notary：支持使用 Docker Content Trust（利用 Notary）对容器镜像进行签名，以保证真实性和出处。此外，还可以激活防止部署未签名映像的策略。图形用户门户：用户可以轻松浏览、搜索存储库和管理项目。审计：通过日志跟踪对存储库的所有操作。RESTful API：提供 RESTful API 是为了方便管理操作，并且易于用于与外部系统集成。嵌入式 Swagger UI 可用于探索和测试 API。易于部署：Harbor 可以通过 Docker compose 以及 Helm Chart 进行部署，最近还添加了一个 Harbor Operator。以上内容特征从github获取，阅读本章，你将了解如何利用harbor配置基本的docker容器仓库和helm仓库的使用harbor如果你是nginx，则可以直接使用如下命令创建。openssl req -x509 -nodes -days 36500 -newkey rsa:2048 -keyout linuxea.key -out linuxea.crt -subj /C=CH/ST=ShangHai/L=Xian/O=Devops/CN=linuxea.test.com1.准备证书我们按照官方文档进行创建ssl证书与之不同的是我们尽可能的将证书配置的旧一些。因为无论在什么环境，证书最大的问题就是会过期，后期替换的时候会波及到正常使用。域名：harbor.local.com创建的证书目录：/data/cert-date +%F复制下面的命令在sh脚本中修改${YOU_DOMAIN}和${CERT_PATH}后执行即可CERT_PATH=/data/cert-`date +%F`/ YOU_DOMAIN=harbor.local.com mkdir -p ${CERT_PATH} openssl genrsa -out ca.key 4096 openssl req -x509 -new -nodes -sha512 -days 365000 \ -subj "/C=CN/ST=Beijing/L=Beijing/O=example/OU=Personal/CN=${YOU_DOMAIN}" \ -key ca.key \ -out ca.crt openssl genrsa -out ${YOU_DOMAIN}.key 4096 openssl req -sha512 -new \ -subj "/C=CN/ST=Beijing/L=Beijing/O=example/OU=Personal/CN=${YOU_DOMAIN}" \ -key ${YOU_DOMAIN}.key \ -out ${YOU_DOMAIN}.csr cat > v3.ext <<-EOF authorityKeyIdentifier=keyid,issuer basicConstraints=CA:FALSE keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment extendedKeyUsage = serverAuth subjectAltName = @alt_names [alt_names] DNS.1=${YOU_DOMAIN} DNS.2=yourdomain DNS.3=hostname EOF openssl x509 -req -sha512 -days 365000 \ -extfile v3.ext \ -CA ca.crt -CAkey ca.key -CAcreateserial \ -in ${YOU_DOMAIN}.csr \ -out ${YOU_DOMAIN}.crt cp ${YOU_DOMAIN}.crt ${YOU_DOMAIN}.key ${CERT_PATH} 2.harbor和docker接着将证书复制到harbor和docker的目录，使其生效1.复制crt和key 到harbor的证书目录cp ${YOU_DOMAIN}.crt ${YOU_DOMAIN}.key ${CERT_PATH} 2.转换yourdomain.com.crt为yourdomain.com.cert, 供 Docker 使用openssl x509 -inform PEM -in harbor.local.com.crt -out harbor.local.com.cert本机mkdir -p /etc/docker/certs.d/harbor.local.com/ cp harbor.local.com.cert /etc/docker/certs.d/harbor.local.com/ cp harbor.local.com.key /etc/docker/certs.d/harbor.local.com/ cp ca.crt /etc/docker/certs.d/harbor.local.com/ systemctl reload docker其他节点scp harbor.local.com.cert harbor.local.com.key ca.crt 172.16.15.136:/etc/docker/certs.d/harbor.local.com/ systemctl reload docker如果不是80端口请创建文件夹/etc/docker/certs.d/yourdomain.com:port或/etc/docker/certs.d/harbor_IP:port.3.harbor.yml经过过滤得到如下配置[root@b.linuxea.com harbor]# egrep -v "^$|^#|^ #|^ #" harbor.yml hostname: reg.mydomain.com http: port: 80 https: port: 443 certificate: /your/certificate/path private_key: /your/private/key/path harbor_admin_password: Harbor12345 database: password: root123 max_idle_conns: 100 max_open_conns: 900 data_volume: /data trivy: ignore_unfixed: false skip_update: false offline_scan: false insecure: false jobservice: max_job_workers: 10 notification: webhook_job_max_retry: 10 chart: absolute_url: disabled log: level: info local: rotate_count: 50 rotate_size: 200M location: /var/log/harbor _version: 2.5.0 proxy: http_proxy: https_proxy: no_proxy: components: - core - jobservice - trivy upload_purging: enabled: true age: 168h interval: 24h dryrun: false3.1.将服务器证书和密钥复制到 Harbor 主机上的 certficates 文件夹中。创建一个存储目录：mkdir /data/harbor/data配置文件。进行sed替换即可，主要修改如下：hostname：域名certificate： cert目录地址private_key： key目录地址harbor_admin_password： 登录密码data_volume： docker-compose的所有挂载目录sed -i 's@hostname: reg.mydomain.com@hostname: harbor.local.com@g' harbor.yml sed -i 's@certificate: /your/certificate/path@certificate: /data/cert-2022-06-28/harbor.local.com.crt@g' harbor.yml sed -i 's@private_key: /your/private/key/path@private_key: /data/cert-2022-06-28/harbor.local.com.key@g' harbor.yml sed -i 's@harbor_admin_password: Harbor12345@harbor_admin_password: admin@g' harbor.yml sed -i 's@data_volume: /data@data_volume: /data/harbor/data@g' harbor.yml4.安装harbor一切准备妥当，执行./install.sh脚本自动安装[root@b.linuxea.com harbor]# ./install.sh [Step 0]: checking if docker is installed ... Note: docker version: 19.03.6 [Step 1]: checking docker-compose is installed ... ....... [Step 5]: starting Harbor ... Creating network "harbor_harbor" with the default driver Creating harbor-log ... done Creating harbor-db ... done Creating registry ... done Creating redis ... done Creating harbor-portal ... done Creating registryctl ... done Creating harbor-core ... done Creating harbor-jobservice ... done Creating nginx ... done ✔ ----Harbor has been installed and started successfully.----5.测试容器仓库回到其他节点测试docker的应有配置目录结构如下[root@a.linuxea.com ~]# tree /etc/docker/ /etc/docker/ ├── certs.d │   └── harbor.local.com │   ├── ca.crt │   ├── harbor.local.com.cert │   └── harbor.local.com.key ├── daemon.json └── key.json 2 directories, 5 files (base) [root@a.linuxea.com ~]# cat /etc/docker/daemon.json {"insecure-registries":["172.16.100.150:8443",harbor.local.com]}登录测试[root@a.linuxea.com ~]# docker login harbor.local.com Username: admin Password: WARNING! Your password will be stored unencrypted in /root/.docker/config.json. Configure a credential helper to remove this warning. See https://docs.docker.com/engine/reference/commandline/login/#credentials-store Login Succeeded上传镜像测试创建一个仓库harbor提供的命令如下docker push harbor.local.com/base/REPOSITORY[:TAG]测试[root@a.linuxea.com ~]# docker tag mysql:8.0.16 harbor.local.com/base/mysql:8.0.16 [root@a.linuxea.com ~]# docker push harbor.local.com/base/mysql:8.0.16 The push refers to repository [harbor.local.com/base/mysql] 605d208195c7: Pushed 9d87c3455758: Pushed 80f1020054a4: Pushed b0425df45fae: Pushed 680666c6bf72: Pushed 7e7fffcdabb3: Pushed 77737de99484: Pushed 2f1b41b24201: Pushed 007a7f930352: Pushed c6926fcee191: Pushed b78ec9586b34: Pushed d56055da3352: Pushed 8.0.16: digest: sha256:036b8908469edac85afba3b672eb7cbc58d6d6b90c70df0bb3fe2ab4fd939b22 size: 2828docker没有问题后配置helm仓库helm31.helm3安装在官网下载一个helm，解压后并将可执行文件放置sbin下wget https://get.helm.sh/helm-v3.8.2-linux-amd64.tar.gz tar xf helm-v3.8.2-linux-amd64.tar.gz cp linux-amd64/helm /usr/local/sbin安装完成[root@a.linuxea.com ~]# helm version version.BuildInfo{Version:"v3.8.2", GitCommit:"5cb9af4b1b271d11d7a97a71df3ac337dd94ad37", GitTreeState:"clean", GoVersion:"go1.17.5"}2.添加helm源添加一个azure源，并将其更新helm repo add stable http://mirror.azure.cn/kubernetes/charts/ helm repo list helm repo update helm search repo stable3.登录helm[root@a.linuxea.com ~]# helm registry login harbor.local.com Username: admin Password: Login Succeeded我们还需要让系统信任这个ca，于是我们将 /etc/docker/certs.d/harbor.local.com/ca.crt的内容追加到如/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem文件中，并复制ca.crt到/etc/pki/ca-trust/source/anchors/cp /etc/docker/certs.d/harbor.local.com/ca.crt /etc/pki/ca-trust/source/anchors/3.中转charts我们一个共有的mirros的包中转到私有的仓库上。在中转之前，需要添加一个源来提供现有的charts下载redishelm fetch stable/redis[root@a.linuxea.com ~]# ls redis-10.5.7.tgz redis-10.5.7.tgz推送到harbor.local.com推送 chart 到当前项目helm push redis-10.5.7.tgz oci://harbor.local.com/redis登录harbor查看点进这个Artifacts内，就能看到更多的信息4.上传本地charts现在在本地创建一个charts用作测试，上传到harbor中[root@a.linuxea.com data]# helm create test Creating test [root@a.linuxea.com data]# ls test/ charts Chart.yaml templates values.yaml打包推送[root@a.linuxea.com data]# helm package test Successfully packaged chart and saved it to: /data/test-0.1.0.tgz推送到helm服务器[root@a.linuxea.com data]# helm push test-0.1.0.tgz oci://harbor.local.com/redis Pushed: harbor.local.com/redis/test:0.1.0 Digest: sha256:1a86bc2ae87a8760398099a9c0966ce41141eacc7270673d03dfc4005bc349db5.使用私有charts回到仓库里面，鼠标放在拉取按钮上将会显示拉取的命令如下helm pull oci://harbor.local.com/redis/redis --version 10.5.7拉到本地[root@a.linuxea.com opt]# helm pull oci://harbor.local.com/redis/redis --version 10.5.7 Pulled: harbor.local.com/redis/redis:10.5.7 Digest: sha256:41643fa64d23797d0a874a2b264c9fc1f7323b08b9a02fb3010d72805b54bc3a [root@a.linuxea.com opt]# ls redis-10.5.7.tgz解压后使用template可以看到模板的配置清单信息[root@a.linuxea.com opt]# tar xf redis-10.5.7.tgz [root@a.linuxea.com redis]# helm template test ./安装测试helm upgrade --install -f values.yaml test-redis stable/redis --namespace redis --create-namespace--create-namespace： 如果名称空间不存在就创建upgrade： 如果存在就更新，不存在就创建[root@a.linuxea.com redis]# helm install -f values.yaml test-redis stable/redis --namespace redis --create-namespace WARNING: This chart is deprecated NAME: test-redis LAST DEPLOYED: Tue Jun 28 14:50:56 2022 NAMESPACE: redis STATUS: deployed REVISION: 1 TEST SUITE: None NOTES: This Helm chart is deprecated Given the `stable` deprecation timeline (https://github.com/helm/charts#deprecation-timeline), the Bitnami maintained Redis Helm chart is now located at bitnami/charts (https://github.com/bitnami/charts/). The Bitnami repository is already included in the Hubs and we will continue providing the same cadence of updates, support, etc that we've been keepinghere these years. Installation instructions are very similar, just adding the _bitnami_ repo and using it during the installation (`bitnami/<chart>` instead of `stable/<chart>`) $ helm repo add bitnami https://charts.bitnami.com/bitnami$ helm install my-release bitnami/<chart> # Helm 3$ helm install --name my-release bitnami/<chart> # Helm 2 To update an exisiting _stable_ deployment with a chart hosted in the bitnami repository you can executerepo add bitnami https://charts.bitnami.com/bitnami $ helm upgrade my-release bitnami/<chart> Issues and PRs related to the chart itself will be redirected to `bitnami/charts` GitHub repository. In the same way, we'll be happy to answer questions related to this migration process in this issue (https://github.com/helm/charts/issues/20969) created as a common place for discussion. ** Please be patient while the chart is being deployed ** Redis can be accessed via port 6379 on the following DNS name from within your cluster: test-redis-master.redis.svc.cluster.local To get your password run: export REDIS_PASSWORD=$(kubectl get secret --namespace redis test-redis -o jsonpath="{.data.redis-password}" | base64 --decode) To connect to your Redis server: 1. Run a Redis pod that you can use as a client: kubectl run --namespace redis test-redis-client --rm --tty -i --restart='Never' \ --env REDIS_PASSWORD=$REDIS_PASSWORD \ --image docker.io/bitnami/redis:5.0.7-debian-10-r32 -- bash 2. Connect using the Redis CLI: redis-cli -h test-redis-master -a $REDIS_PASSWORD To connect to your database from outside the cluster execute the following commands: kubectl port-forward --namespace redis svc/test-redis-master 6379:6379 & redis-cli -h 127.0.0.1 -p 6379 -a $REDIS_PASSWORD查看密码[root@a.linuxea.com redis]# kubectl get secret --namespace redis test-redis -o jsonpath="{.data.redis-password}" | base64 --decode VeiervwDUG查看运行状态由于一些配置没有准备，此时redis是pending的，但是helm安装是成功的。我们的目的达到了[root@k8s-02 ~]# kubectl -n redis get all NAME READY STATUS RESTARTS AGE pod/test-redis-master-0 0/1 Pending 0 2m53s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/test-redis-headless ClusterIP None <none> 6379/TCP 2m53s service/test-redis-master ClusterIP 10.101.161.177 <none> 6379/TCP 2m53s NAME READY AGE statefulset.apps/test-redis-master 0/1 2m53s已经测试完成，现在卸载掉即可([root@a.linuxea.com redis]# helm -n redis ls NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION test-redis redis 1 2022-06-28 14:50:56.56116374 +0800 CST deployed redis-10.5.7 5.0.7 [root@a.linuxea.com redis]# helm -n redis uninstall test-redis release "test-redis" uninstalled [root@a.linuxea.com redis]# helm -n redis ls NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION基于ip的helm在很多场景中，我们需要把一个http的改成https并且还要他支持https，并且还是ip，这在harbor官网已经有说明，老话重提的。1.准备证书与此前不同的是，我们需要将subjectAltName = @alt_names的值也改成ip地址subjectAltName = IP:IPADDRESS如下CERT_PATH=/data/cert-`date +%F`/ YOU_DOMAIN=harbor.local.com mkdir -p ${CERT_PATH} openssl genrsa -out ca.key 4096 openssl req -x509 -new -nodes -sha512 -days 365000 \ -subj "/C=CN/ST=Beijing/L=Beijing/O=example/OU=Personal/CN=${YOU_DOMAIN}" \ -key ca.key \ -out ca.crt openssl genrsa -out ${YOU_DOMAIN}.key 4096 openssl req -sha512 -new \ -subj "/C=CN/ST=Beijing/L=Beijing/O=example/OU=Personal/CN=${YOU_DOMAIN}" \ -key ${YOU_DOMAIN}.key \ -out ${YOU_DOMAIN}.csr cat > v3.ext <<-EOF authorityKeyIdentifier=keyid,issuer basicConstraints=CA:FALSE keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment extendedKeyUsage = serverAuth subjectAltName = @alt_names [alt_names] DNS.1=${YOU_DOMAIN} DNS.2=yourdomain DNS.3=hostname EOF openssl x509 -req -sha512 -days 365000 \ -extfile v3.ext \ -CA ca.crt -CAkey ca.key -CAcreateserial \ -in ${YOU_DOMAIN}.csr \ -out ${YOU_DOMAIN}.crt cp ${YOU_DOMAIN}.crt ${YOU_DOMAIN}.key ${CERT_PATH} 以172.16.100.150为例CERT_PATH=/data/cert-`date +%F`/ YOU_DOMAIN=172.16.100.150:8443 mkdir -p ${CERT_PATH} openssl genrsa -out ca.key 4096 openssl req -x509 -new -nodes -sha512 -days 365000 \ -subj "/C=CN/ST=Beijing/L=Beijing/O=example/OU=Personal/CN=${YOU_DOMAIN}" \ -key ca.key \ -out ca.crt openssl genrsa -out ${YOU_DOMAIN}.key 4096 openssl req -sha512 -new \ -subj "/C=CN/ST=Beijing/L=Beijing/O=example/OU=Personal/CN=${YOU_DOMAIN}" \ -key ${YOU_DOMAIN}.key \ -out ${YOU_DOMAIN}.csr cat > v3.ext <<-EOF authorityKeyIdentifier=keyid,issuer basicConstraints=CA:FALSE keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment extendedKeyUsage = serverAuth subjectAltName = IP:172.16.100.150 [alt_names] DNS.1=${YOU_DOMAIN} DNS.2=yourdomain DNS.3=hostname EOF openssl x509 -req -sha512 -days 365000 \ -extfile v3.ext \ -CA ca.crt -CAkey ca.key -CAcreateserial \ -in ${YOU_DOMAIN}.csr \ -out ${YOU_DOMAIN}.crt cp ${YOU_DOMAIN}.crt ${YOU_DOMAIN}.key ${CERT_PATH}2.harbor.yaml配置harbor.yaml部分配置hostname: 172.16.100.150 http: port: 8080 https: port: 8443 certificate: /etc/ssl/certs/172.16.100.150:8443.crt private_key: /etc/ssl/certs/172.16.100.150:8443.key配置完成后需要执行./prepare并且重启3.docker和ca仍然进行证书拷贝首先拷贝当前节点的 cp 172.16.100.150\:8443.* /etc/docker/certs.d/172.16.100.150\:8443/在将当前的证书打包拷贝到其他需要使用helm上传下载的节点 tar -zcf 8443.tar 172.16.100.150\:8443 scp 8443.tar 172.16.15.136:/etc/docker/certs.d/目录结构如下[root@docker-156 certs.d]# tree /etc/docker/certs.d/ /etc/docker/certs.d/ ├── 172.16.100.150:8443 │   ├── 172.16.100.150:8443.cert │   ├── 172.16.100.150:8443.crt │   ├── 172.16.100.150:8443.csr │   ├── 172.16.100.150:8443.key │   └── ca.crt仍然需要让系统信任这个ca，于是我们将 ca.crt的内容追加到如/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem文件中，并复制ca.crt到/etc/pki/ca-trust/source/anchors/cp /etc/docker/certs.d/harbor.local.com/ca.crt /etc/pki/ca-trust/source/anchors/而后就可以正常推送和下载了

linuxea: kustomize变量传入 kustomize一直是备受欢迎的yaml配置管理之一，在过去kustomize一直在解决 “提供一种操作配置数据的方法，而不会使原始配置无法被 Kubernetes 使用。”，我们不用去和其他工具做比较，因为这就是kustomize的魅力所在。但是kustomize也有自己的缺点，它无法像helm那样灵活多变，比如，在配置多个一个Ingress的域名的时候，这在helm中将会非常简单，但是在kustomize中，几乎无法通过kustomize本身来解决，但是官方提供了var，而后var被诟病，于是出现了valueAdd，但很可惜，valueAdd并不是为了解决这个问题。valueAdd是从vars演变而来，但是valueAdd并不是最好的方式这么多的方式，均在解决一个核心的问题，环境变量env和字符自定义删除vars是计划的一部分，很显然，目前并没有更好的方式来解决更多的问题为了应对这个情况，使用最原始的envsubst成了一个选项。如果你并不希望你的配置清单是原始的，而是一些环境变量，大量的模板语法，你可以尝试helm。但用helm来管理大量零散的清单配置，在我看来是条不归路。因为事情在演变的过程中会不断的超过预期，变得复杂。而一旦复杂只会增加额外的成本。像往常一样去配置一个kustomize的目录[root@linuxea.com ~/kustomize]# tree ./ ./ └── env-path ├── base │   ├── deployment.yaml │   └── kustomization.yaml ├── kustomize.exe ├── overlays │   ├── dev │   │   ├── env.file │   │   ├── kustomization.yaml │   │   └── replacement.yaml │   ├── pre-prod │   │   ├── kustomization.yaml │   │   └── patch-shared-env.yaml │   └── prod │   ├── kustomization.yaml │   ├── patch-env-from.yaml │   ├── prod-1-env.yaml │   └── prod-2-env.yaml └── transformers └── setProject ├── kustomization.yaml └── setProject.yaml 8 directories, 14 filesdeployment.yamlapiVersion: apps/v1 kind: Deployment metadata: name: test spec: replicas: 3 selector: matchLabels: template: metadata: labels: spec: containers: - name: test image: alpine:3.12.12 imagePullPolicy: Always command: [ "/bin/sh"] args: ["-c","echo $(ALT_GREETING) $(ENABLE_RISKY) $SW_AGENT_TRACE_IGNORE_PATH; sleep 36000" ] ports: - containerPort: 8080 env: - name: ALT_GREETING valueFrom: configMapKeyRef: name: envinpod key: ALT_GREETING - name: ENABLE_RISKY valueFrom: configMapKeyRef: name: envinpod key: ENABLE_RISKY - name: SW_AGENT_TRACE_IGNORE_PATH valueFrom: configMapKeyRef: name: envinpod key: SW_AGENT_TRACE_IGNORE_PATH 我们分别传入了三个环境变量$(ALT_GREETING) $(ENABLE_RISKY) $SW_AGENT_TRACE_IGNORE_PATH首先通过env.valueFrom.configMapKeyRef来进行传递。因为这是k8s configmapkeyref的常用方式 env: - name: ALT_GREETING valueFrom: configMapKeyRef: name: envinpod key: ALT_GREETING - name: ENABLE_RISKY valueFrom: configMapKeyRef: name: envinpod key: ENABLE_RISKY - name: SW_AGENT_TRACE_IGNORE_PATH valueFrom: configMapKeyRef: name: envinpod key: SW_AGENT_TRACE_IGNORE_PATH kustomization导入[root@linuxea.com ~/kustomize/env-path]# cat base/kustomization.yaml resources: - deployment.yaml而在overlays下的dev中，引用了这些环境变量[root@linuxea.com ~/kustomize/env-path/overlays]# cat dev/env.file ALT_GREETING=Hiya ENABLE_RISKY=false TEST-NAME=marksugar SW_AGENT_TRACE_IGNORE_PATH=GET:/health,GET:/aggreg/health,/eureka/**,xxl-job/**在kustomization中的配置如下apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization # 名称空间 # namespace: test1 # 前缀 namePrefix: fat- bases: - ../../base # configmap变量 configMapGenerator: - name: envinpod env: env.file # 副本数 replicas: - name: test count: 5 # 标签 commonLabels: app.kubernetes.io/name: nginx app: mark # 镜像 images: - name: alpine newTag: 3.12.12其中关键的在于configMapGenerator: - name: envinpod env: env.file我们渲染下看[root@linuxea.com ~/kustomize/env-path]# kustomize build overlays/dev/ apiVersion: v1 data: ALT_GREETING: Hiya ENABLE_RISKY: "false" SW_AGENT_TRACE_IGNORE_PATH: GET:/health,GET:/aggreg/health,/eureka/**,xxl-job/** TEST-NAME: marksugar kind: ConfigMap metadata: labels: app: mark app.kubernetes.io/name: nginx name: fat-envinpod-8hbm9d86m9 --- apiVersion: apps/v1 kind: Deployment metadata: labels: app: mark app.kubernetes.io/name: nginx name: fat-test spec: replicas: 5 selector: matchLabels: app: mark app.kubernetes.io/name: nginx template: metadata: labels: app: mark app.kubernetes.io/name: nginx spec: containers: - args: - -c - echo $(ALT_GREETING) $(ENABLE_RISKY) $SW_AGENT_TRACE_IGNORE_PATH; sleep 36000 command: - /bin/sh env: - name: ALT_GREETING valueFrom: configMapKeyRef: key: ALT_GREETING name: fat-envinpod-8hbm9d86m9 - name: ENABLE_RISKY valueFrom: configMapKeyRef: key: ENABLE_RISKY name: fat-envinpod-8hbm9d86m9 - name: SW_AGENT_TRACE_IGNORE_PATH valueFrom: configMapKeyRef: key: SW_AGENT_TRACE_IGNORE_PATH name: fat-envinpod-8hbm9d86m9 image: alpine:3.12.12 imagePullPolicy: Always name: test ports: - containerPort: 8080而后run起来查看镜像的内容你可以使用kubectl -k 或者如我这样使用[root@linuxea.com ~/kustomize/env-path]# kustomize build overlays/dev/ | kubectl --kubeconfig /root/.kube/marksugar-dev-1 apply -f - configmap/fat-envinpod-8hbm9d86m9 created deployment.apps/fat-test createdrun起来后直接查看日志，环境变量有没有被传入[root@linuxea.com ~/kustomize/env-path]# kubectl --kubeconfig /root/.kube/marksugar-dev-1 logs -f fat-test-f9d967c4-vsc28 Hiya false GET:/health,GET:/aggreg/health,/eureka/**,xxl-job/**除此之外使用envsubst 可以参考变量实值与文件变量替换配置清单在应用之前必须先通过kustomize进行重新转换成k8s的原始格式文件并且可以通过环境变量传递给kubectl，如下env is=true kubectl apply -k 推荐阅读Kustomize command to add environment variables to containers in a kustomizationDocument the environment variable substitution feature of kustomize configMapGeneratorCreate environment variables from env file with KustomizeUse ConfigMap-defined environment variables in Pod commands combineConfigskustomize-with-multiple-envsGenerating ResourcesKustomize Vars exampleIntroduce a ReplacementTransformer to replace the vars feature.#3492kustomize vars - enhance or replace?#2052Replacement poc#1631airshipctlkv.gohttps://github.com/kubernetes-sigs/kustomize/pull/3737/files#diff-c3d1278453f2a6fb229ec8998df0f109d8605b5e46ba2a84d067083f5a543761R194Using Kustomize for per-environment deployment of cert-manager resourcesHow To Manage Your Kubernetes Configurations with KustomizeIntroduce a ReplacementTransformer to replace the vars feature.#3492Kustomize PluginsChanging 'imagePullPolicy' of all containers in all deployments#1493valueAdd.mdkustomize-with-multiple-envs使用 Kustomize 对 Kubernetes 对象进行声明式管理Using system environment variables with KustomizeDemo: combining config data from devops and developershttps://github.com/kubernetes-sigs/kustomize/issues/2052https://github.com/kubernetes-sigs/kustomize/blob/master/examples/valueAdd.md

linuxea: helm3变量/命名模板/共享模板等常见用法(3) 在helm中除了此前的那些内置函数和定义的一些valumes的值替换，这些对象和控制语句，还有一些管道之外，仍然提供了变量。大多数编程语言中，都有变量。如，在golang中以及groovy中都提到了全局变量和局部变量。 他们的作用域范围不同全局变量全局变量，顾名思义，所有都可以使用局部变量在某一块的代码内被使用，而代码块作用域外无法使用有了这些的支撑，更好理解变量的出现以及使用的问题变量而变量在helm中的用的和大多数编程语言不一样，helm中的变量能够用来简化代码。这通常配合with已经range。但是在helm中，定义一个变量的方式却非常有意思，似乎结合了shell和golang。$mark := .Release.Name示例1 变量仍然存在一个局部和全局的意思，比如：with {{ with .Values.mydata -}} {{ if eq .status "isok" -}} status: ok {{- else if eq .names "marksugar" -}} namestatus: "true" {{ else -}} defaultstaus: true {{- end }} {{- end }}在如上代码块中是不能使用Release.Name的，解决这个问题的办法就是将对象分配给with的作用域之外。上面提到，定义一个变量的方式和shell，golang类似，随即定义一个Release.Name{{ $linuxea := .Release.Name -}}并将塔引入到with必须将变量定于在with之外，在with中的if之外 {{ $linuxea := .Release.Name -}} {{ with .Values.mydata -}} releasename: {{ $linuxea }} {{ if eq .status "isok" -}} status: ok {{- else if eq .names "marksugar" -}} namestatus: "true" {{ else -}} defaultstaus: true {{- end }} {{- end }}如下apiVersion: v1 kind: ConfigMap metadata: name: {{ .Release.Name }}-cmp labels: app: {{ .Release.Name }}-cmp data: name: {{ .Values.mydata.names | default "supper" }} test: {{ .Values.mydata.data | upper | repeat 5 }} {{ $linuxea := .Release.Name -}} {{ with .Values.mydata -}} releasename: {{ $linuxea }} {{ if eq .status "isok" -}} status: ok {{- else if eq .names "marksugar" -}} namestatus: "true" {{ else -}} defaultstaus: true {{- end }} {{- end }} {{ indent 2 "test: ok" }} mylist: |- {{- range .Values.mylist }} - {{ . | title | quote }} {{- end }}执行PS H:\k8s-1.20.2\helm> helm template linuxea.com .\liunuxea\ --- # Source: linuxea/templates/configmap.yaml apiVersion: v1 kind: ConfigMap metadata: name: linuxea.com-cmp labels: app: linuxea.com-cmp data: name: marksugar test: HI MARK, MY WWW.LINUXEA.COMHI MARK, MY WWW.LINUXEA.COMHI MARK, MY WWW.LINUXEA.COMHI MARK, MY WWW.LINUXEA.COMHI MARK, MY WWW.LINUXEA.COM releasename: linuxea.com status: ok test: ok mylist: |- - "Mark" - "Edwin" - "Sean"示例2 range在golang里面，range是可以将索引和值进行打印的了，而在helm中也可以这样使用。如下 mylist: |- {{- range $index,$value := .Values.mylist }} - {{ $index }}:{{ $value | title | quote }} {{- end }} values.yamlmydata: names: marksugar data: hi mark, my www.linuxea.com status: isok mylist: - mark - edwin - sean 如下apiVersion: v1 kind: ConfigMap metadata: name: {{ .Release.Name }}-cmp labels: app: {{ .Release.Name }}-cmp data: name: {{ .Values.mydata.names | default "supper" }} test: {{ .Values.mydata.data | upper | repeat 5 }} {{ $linuxea := .Release.Name -}} {{ with .Values.mydata -}} releasename: {{ $linuxea }} {{ if eq .status "isok" -}} status: ok {{- else if eq .names "marksugar" -}} namestatus: "true" {{ else -}} defaultstaus: true {{- end }} {{- end }} {{ indent 2 "test: ok" }} mylist: |- {{- range .Values.mylist }} - {{ . | title | quote }} {{- end }} {{- range $index,$value := .Values.mylist }} - {{ $index }}:{{ $value | title | quote }} {{- end }}渲染PS H:\k8s-1.20.2\helm> helm template linuxea.com .\liunuxea\ --- # Source: linuxea/templates/configmap.yaml apiVersion: v1 kind: ConfigMap metadata: name: linuxea.com-cmp labels: app: linuxea.com-cmp data: name: marksugar test: HI MARK, MY WWW.LINUXEA.COMHI MARK, MY WWW.LINUXEA.COMHI MARK, MY WWW.LINUXEA.COMHI MARK, MY WWW.LINUXEA.COMHI MARK, MY WWW.LINUXEA.COM releasename: linuxea.com status: ok test: ok mylist: |- - "Mark" - "Edwin" - "Sean" - 0:"Mark" - 1:"Edwin" - 2:"Sean"map同样，可以渲染一个字典 {{- range $index,$value := .Values.mydata }} - {{ $index }}:{{ $value | title | quote }} {{- end }} values.yamlmydata: names: marksugar data: hi mark, my www.linuxea.com status: isok mylist: - mark - edwin - sean 如下apiVersion: v1 kind: ConfigMap metadata: name: {{ .Release.Name }}-cmp labels: app: {{ .Release.Name }}-cmp data: name: {{ .Values.mydata.names | default "supper" }} test: {{ .Values.mydata.data | upper | repeat 5 }} {{ $linuxea := .Release.Name -}} {{ with .Values.mydata -}} releasename: {{ $linuxea }} {{ if eq .status "isok" -}} status: ok {{- else if eq .names "marksugar" -}} namestatus: "true" {{ else -}} defaultstaus: true {{- end }} {{- end }} {{ indent 2 "test: ok" }} mylist: |- {{- range .Values.mylist }} - {{ . | title | quote }} {{- end }} {{- range $index,$value := .Values.mylist }} - {{ $index }}:{{ $value | title | quote }} {{- end }} {{- range $index,$value := .Values.mydata }} - {{ $index }}:{{ $value | title | quote }} {{- end }} 渲染PS H:\k8s-1.20.2\helm> helm template linuxea.com .\liunuxea\ --- # Source: linuxea/templates/configmap.yaml apiVersion: v1 kind: ConfigMap metadata: name: linuxea.com-cmp labels: app: linuxea.com-cmp data: name: marksugar test: HI MARK, MY WWW.LINUXEA.COMHI MARK, MY WWW.LINUXEA.COMHI MARK, MY WWW.LINUXEA.COMHI MARK, MY WWW.LINUXEA.COMHI MARK, MY WWW.LINUXEA.COM releasename: linuxea.com status: ok test: ok mylist: |- - "Mark" - "Edwin" - "Sean" - 0:"Mark" - 1:"Edwin" - 2:"Sean" - data:"Hi Mark, My Www.Linuxea.Com" - names:"Marksugar" - status:"Isok"示例3 全局变量变量通常而言不是全局的，都在作用域之内，在range和with中作用域也只是在{{ rage }}和{{end}}之间的。为了解决这个问题，可以从顶部开始定义一个，或者使用$来引用外部变量$表示最外层的上下文顶部定义Release.Name有两种方式。第一种，在顶部定义，如下{{ $linuxea := .Release.Name -}} apiVersion: v1 kind: ConfigMap metadata: name: {{ .Release.Name }}-cmp labels: app: {{ .Release.Name }}-cmp data: ... releasename: {{ $linuxea }} ...$引用Chart.Name{{ $linuxea := .Release.Name -}} apiVersion: v1 kind: ConfigMap metadata: name: {{ .Release.Name }}-cmp labels: app: {{ .Release.Name }}-cmp data: ... chartname: {{ $.Chart.Name }} ...如下{{ $linuxea := .Release.Name -}} apiVersion: v1 kind: ConfigMap metadata: name: {{ .Release.Name }}-cmp labels: app: {{ .Release.Name }}-cmp data: name: {{ .Values.mydata.names | default "supper" }} test: {{ .Values.mydata.data | upper | repeat 5 }} {{ with .Values.mydata -}} releasename: {{ $linuxea }} chartname: {{ $.Chart.Name }} {{ if eq .status "isok" -}} status: ok {{- else if eq .names "marksugar" -}} namestatus: "true" {{ else -}} defaultstaus: true {{- end }} {{- end }} 渲染PS H:\k8s-1.20.2\helm> helm template linuxea.com .\liunuxea\ --- # Source: linuxea/templates/configmap.yaml apiVersion: v1 kind: ConfigMap metadata: name: linuxea.com-cmp labels: app: linuxea.com-cmp data: name: marksugar test: HI MARK, MY WWW.LINUXEA.COMHI MARK, MY WWW.LINUXEA.COMHI MARK, MY WWW.LINUXEA.COMHI MARK, MY WWW.LINUXEA.COMHI MARK, MY WWW.LINUXEA.COM releasename: linuxea.com chartname: linuxea status: ok到现在为止，一直都是使用一个模板来进行操作的，如果要进行多个模板，就需要使用到命名模板。命名模板命名模板也称为子模板。在文件内部定义，通常有两种方式创建，而使用方式较多一旦定义了命名模板，名称就是全局的，如果名称重复，最后被加载的模板会被使用。因此，命名需要注意不要冲突，而为了避免此事，大家开始约定，在定义的模板前添加chart名称，如{{ define "mylinuxea.name" }}通过这种特定的前缀避免相同名称带来的冲突问题如果希望渲染的是一段，那么template将很适用，否则include可以更适合嵌套helm模板允许创建命名进行嵌入式的方式在其他位置进行访问。命名约定：templates/ 目录下的文件被视为kubernetes资源清单以_开头的文件不会被当作资源清单文件以_下划线开头的文件可以被其他chart模板调用partials_开头的文件是helm中的partials，默认的名称模板就在_helpers.tpl文件中，而我们自己定义的也可以在其中。这些文件就是partials文件。而在chart的包中也能找到这个文件... templates/ | `-- tests ....... `-- _helpers.tpl ...._helpers.tpl中就定义了被其他chart调用define and templatedefine关键字可以在模板文件中创建命名模板，而后使用template来引用示例{{ define "linuxea.name" }} # describe {{ end }}定义一个标签{{- define "linuxea.labels" }} labels: generator: helm data: {{ now | htmlDate }} {{ end }}now为当前时间的函数，而htmlDate是格式化而使用则是使用template引用{{- template "linuxea.labels" }}如下{{ $linuxea := .Release.Name -}} {{- define "linuxea.labels" }} labels: generator: helm data: {{ now | htmlDate -}} {{ end }} apiVersion: v1 kind: ConfigMap metadata: name: {{ .Release.Name }}-cmp {{- template "linuxea.labels" }} data: name: {{ .Values.mydata.names | default "supper" }} test: {{ .Values.mydata.data | upper | repeat 5 }} {{ with .Values.mydata -}} releasename: {{ $linuxea }} chartname: {{ $.Chart.Name }} {{ if eq .status "isok" -}} status: ok {{- else if eq .names "marksugar" -}} namestatus: "true" {{ else -}} defaultstaus: true {{- end }} {{- end }} 渲染PS H:\k8s-1.20.2\helm> helm template linuxea.com .\liunuxea\ --- # Source: linuxea/templates/configmap.yaml apiVersion: v1 kind: ConfigMap metadata: name: linuxea.com-cmp labels: generator: helm data: 2022-04-21 data: name: marksugar test: HI MARK, MY WWW.LINUXEA.COMHI MARK, MY WWW.LINUXEA.COMHI MARK, MY WWW.LINUXEA.COMHI MARK, MY WWW.LINUXEA.COMHI MARK, MY WWW.LINUXEA.COM releasename: linuxea.com chartname: linuxea status: ok_helpers.tpl那么按照刚说的，我们应该把她放在创建的_helpers.tpl文件中{{/ 注释 /}} 注释{{/* 注释 */}} {{- define "linuxea.labels" }} labels: generator: helm data: {{ now | htmlDate -}} {{ end }}其他不需要修改，仍然是可以进行调用的PS H:\k8s-1.20.2\helm> helm template linuxea.com .\liunuxea\ --- # Source: linuxea/templates/configmap.yaml apiVersion: v1 kind: ConfigMap metadata: name: linuxea.com-cmp labels: generator: helm data: 2022-04-21 data: name: marksugar test: HI MARK, MY WWW.LINUXEA.COMHI MARK, MY WWW.LINUXEA.COMHI MARK, MY WWW.LINUXEA.COMHI MARK, MY WWW.LINUXEA.COMHI MARK, MY WWW.LINUXEA.COM releasename: linuxea.com chartname: linuxea status: ok访问范围那么此时，如果想要在_helpers.tpl中使用helm的内置函数对象，如:Release.Name类似的对象，就需要在template中传递一个..作为作用域，在_helpers.tpl中需要从template中传递过来，否则是无法正常被使用_helpers.tplchartName: {{ .Release.Name }}_helpers.tpl中的.是从template的尾部的.传递的，也就是作用域中的传递引用使用template的尾部加.{{- template "linuxea.labels" . }}如下{{/* labels */}} {{- define "linuxea.labels" }} labels: generator: helm data: {{ now | htmlDate }} chartName: {{ .Release.Name }} {{- end }}引用{{ $linuxea := .Release.Name -}} apiVersion: v1 kind: ConfigMap metadata: name: {{ .Release.Name }}-cmp {{- template "linuxea.labels" . }} data: name: {{ .Values.mydata.names | default "supper" }} test: {{ .Values.mydata.data | upper | repeat 5 }} {{ with .Values.mydata -}} releasename: {{ $linuxea }} chartname: {{ $.Chart.Name }} {{ if eq .status "isok" -}} status: ok {{- else if eq .names "marksugar" -}} namestatus: "true" {{ else -}} defaultstaus: true {{- end }} {{- end }} 渲染PS H:\k8s-1.20.2\helm> helm.exe template test .\liunuxea\ --- # Source: linuxea/templates/configmap.yaml apiVersion: v1 kind: ConfigMap metadata: name: test-cmp labels: generator: helm data: 2022-04-23 chartName: test data: name: marksugar test: HI MARK, MY WWW.LINUXEA.COMHI MARK, MY WWW.LINUXEA.COMHI MARK, MY WWW.LINUXEA.COMHI MARK, MY WWW.LINUXEA.COMHI MARK, MY WWW.LINUXEA.COM releasename: test chartname: linuxea status: okincludetemplate作为一个函数用来渲染结果，是无法将模板调用给其他函数使用的，而要完成这样的操作就需要include，include可以将模板内容导入到当前的管道，传递给其他函数template只是将结果渲染，原封不动的渲染过来，而include可以做一些操作，比如，添加空格。而这在yaml格式中尤为重要定义一个mark.labels，如下{{- define "mark.labels" }} app: linuxea.com version: v0.1 {{- end }}引用metadata: name: {{ .Release.Name }}-cmp labels: {{- include "mark.labels" . | indent 4 }} include 后，indent 4 空出4个空格如下{{ $linuxea := .Release.Name -}} apiVersion: v1 kind: ConfigMap metadata: name: {{ .Release.Name }}-cmp labels: {{- include "mark.labels" . | indent 4 }} data: name: {{ .Values.mydata.names | default "supper" }} test: {{ .Values.mydata.data | upper | repeat 5 }} {{ with .Values.mydata -}} releasename: {{ $linuxea }} chartname: {{ $.Chart.Name }} {{ if eq .status "isok" -}} status: ok {{- else if eq .names "marksugar" -}} namestatus: "true" {{ else -}} defaultstaus: true {{- end }} {{- end }} 渲染如下PS H:\k8s-1.20.2\helm> helm.exe template test .\liunuxea\ --- # Source: linuxea/templates/configmap.yaml apiVersion: v1 kind: ConfigMap metadata: name: test-cmp labels: app: linuxea.com version: v0.1 data: name: marksugar test: HI MARK, MY WWW.LINUXEA.COMHI MARK, MY WWW.LINUXEA.COMHI MARK, MY WWW.LINUXEA.COMHI MARK, MY WWW.LINUXEA.COMHI MARK, MY WWW.LINUXEA.COM releasename: test chartname: linuxea status: ok.Files.Files可以更好的导入文件中的内容，上面的几种都是来使得模板使用变得容易，而.Files是注入内容，但是并不是通过模板渲染获得。但是.files在使用的时候需要注意一下3点：1，helm chart被限制必须不能大于1M，而.files的文件也会被打包，这意味着不能什么都加进去helm chart最终是以secret形式存放，而secret是存放在etcd中，etcd本身的限制2，出于安全考虑.Files不能访问template/和.helmgnore的排除文件3，chart不会与uninx系统信息挂钩，.files的文件权限不影响使用注意：文件内容必须是yaml格式或者json格式文件必须在template目录之上，和values.yaml同级别在templates目录之上创建三个文件否则会抛出: Error: INSTALLATION FAILED: unable to build kubernetes objects from release manifest: error validating "": error validating data: [apiVersion not set, kind not set]错误所以目录结构必须正确Test.json test2.json test3.json如下PS H:\k8s-1.20.2\helm> dir .\liunuxea\ 目录: H:\k8s-1.20.2\helm\liunuxea Mode LastWriteTime Length Name ---- ------------- ------ ---- d----- 2022-04-26 0:28 templates -a---- 2022-04-26 0:21 0 .helmignore -a---- 2022-04-26 0:24 102 Chart.yaml -a---- 2022-04-26 0:22 10 Test.json -a---- 2022-04-26 0:38 11 test2.json -a---- 2022-04-26 0:38 12 test3.json -a---- 2022-04-26 0:12 120 values.yaml而后使用range来遍历如下apiVersion: v1 kind: Secret metadata: name: mysecret namespace: default type: Opaque data: {{- $files := .Files }} {{- range tuple "Test.json" "test2.json" "test3.json"}} {{ . }}: |- {{ $files.Get . }} {{- end }} --- apiVersion: v1 kind: ConfigMap metadata: name: special-config data: {{- $files := .Files }} {{- range tuple "Test.json" "test2.json" "test3.json"}} {{ . }}: |- {{ $files.Get . }} {{- end }}$files赋值的值是.Files对象的引用，tuple函数来循环文件列表，而后打印每个文件夹的{{ . }}: 文件名 |- 换行，并且去掉行尾空格 {{ $files.Get . }} 打印文件内容渲染PS H:\k8s-1.20.2\helm> helm.exe install test --dry-run .\liunuxea\ NAME: test LAST DEPLOYED: Tue Apr 26 00:39:14 2022 NAMESPACE: default STATUS: pending-install REVISION: 1 TEST SUITE: None HOOKS: MANIFEST: --- # Source: linuxea/templates/configmap.yaml apiVersion: v1 kind: Secret metadata: name: mysecret namespace: default type: Opaque data: Test.json: |- names: 123 test2.json: |- names: 1234 test3.json: |- names: 12345 --- # Source: linuxea/templates/configmap.yaml apiVersion: v1 kind: ConfigMap metadata: name: special-config data: Test.json: |- names: 123 test2.json: |- names: 1234 test3.json: |- names: 12345configmap和secretspod中的configmap和secrets如果把文件内容同时加入到configmap和secrets中，在helm中有另外一个方法https://helm.sh/docs/chart_template_guide/accessing_files/#configmap-and-secrets-utility-functionsasconfigassecrets如果不想用这两个还可使用base64编码在templates目录之上创建三个文件否则会抛出: Error: INSTALLATION FAILED: unable to build kubernetes objects from release manifest: error validating "": error validating data: [apiVersion not set, kind not set]错误所以目录结构必须正确{{ $linuxea := .Release.Name -}} apiVersion: v1 kind: ConfigMap metadata: name: {{ .Release.Name }}-cmp labels: {{- include "mark.labels" . | indent 4 }} data: token: |- {{ .Files.Get "test3.json" | b64enc }} --- apiVersion: v1 kind: Secret metadata: name: {{ .Release.Name }}-linuxea type: Opaque data: my.cnf: |- {{ .Files.Get "test2.json" | b64enc }}调试PS H:\k8s-1.20.2\helm> helm.exe install test --dry-run .\liunuxea\ NAME: test LAST DEPLOYED: Tue Apr 26 00:42:37 2022 NAMESPACE: default STATUS: pending-install REVISION: 1 TEST SUITE: None HOOKS: MANIFEST: --- # Source: linuxea/templates/configmap.yaml apiVersion: v1 kind: Secret metadata: name: test-linuxea type: Opaque data: my.cnf: |- bmFtZXM6IDEyMzQ= --- # Source: linuxea/templates/configmap.yaml apiVersion: v1 kind: ConfigMap metadata: name: test-cmp labels: app: linuxea.com version: v0.1 data: token: |- bmFtZXM6IDEyMzQ1NOTES.txt 文件NOTES.txt被用来打印一些信息的，比如在安装完成或者升级完成后，而使用模板可以自定义这些信息。而要使用这些，必须创建一个NOTES.txt的文件在templates目录下，纯文本内容。但是可以像模板一样进行处理并且和模板一样进行使用。在template模板下创建一个NOTES.txt，内容随便，如下：================================== hello thank you install {{ .Chart.Name }} describe: $ helm status {{ .Release.Name }} $ helm get {{ .Release.Name }}因为是安装后或者升级才能看到的，所以需要安装调试下才能看到PS H:\k8s-1.20.2\helm> helm.exe install test --dry-run .\liunuxea\ NAME: test LAST DEPLOYED: Tue Apr 26 23:02:27 2022 NAMESPACE: default STATUS: pending-install REVISION: 1 TEST SUITE: None HOOKS: MANIFEST: --- # Source: linuxea/templates/configmap.yaml apiVersion: v1 kind: Secret metadata: name: test-linuxea type: Opaque data: my.cnf: |- bmFtZXM6IDEyMzQ= --- # Source: linuxea/templates/configmap.yaml apiVersion: v1 kind: ConfigMap metadata: name: test-cmp labels: app: linuxea.com version: v0.1 data: token: |- bmFtZXM6IDEyMzQ1 NOTES: ================================== hello thank you install linuxea describe: $ helm status test $ helm get testsubcharts和全局值charts是可以有一些依赖项的，这种依赖的方式是subcharts，也可以称为子chart子chart注意信息：subcharts是独立的，不能依赖父chart子chart无法访问父级的值父级可以覆盖子chart的值helm中可以定义全局值，在父，子的chart都可以被访问子chart因此，我们在当前的liunxea的charts目录下下创建一个mycharthelm create mychart rm mychart/templates/*而后在templates下添加一个values.yaml，内容如下mark: "hello mark"在template下创建一个configmap.yaml，如下apiVersion: v1 kind: ConfigMap metadata: name: {{ .Release.Name }}-mychart data: mychart: {{ .Values.mark }}在当前子chart中渲染两个chart是独立的，因此可以进行渲染PS H:\k8s-1.20.2\helm\liunuxea\mychart> helm.exe template test ./ --- # Source: mychart/templates/configmap.yaml apiVersion: v1 kind: ConfigMap metadata: name: test-mychart data: mychart: hello mark如果此时要在主chart，也就是liunxea中覆盖子chart，就需要在主chart中添加同样的参数mychart: mark: "is master chart"而后直接渲染即可顶层的主chart覆盖了子chart的值，但是这种覆盖方式在模板引擎传递values值的时候就会配置作用域，当覆盖的时候，liunxea中的values的mychart值就会传递到mychart， 对于mychart模板中的.Vaules仅仅用域该子chart的值PS H:\k8s-1.20.2\helm\liunuxea> helm.exe template test . --- # Source: linuxea/templates/configmap.yaml apiVersion: v1 kind: Secret metadata: name: test-linuxea type: Opaque data: my.cnf: |- bmFtZXM6IDEyMzQ= --- # Source: linuxea/charts/mychart/templates/configmap.yaml apiVersion: v1 kind: ConfigMap metadata: name: test-mychart data: mychart: is master chart --- # Source: linuxea/templates/configmap.yaml apiVersion: v1 kind: ConfigMap metadata: name: test-cmp labels: app: linuxea.com version: v0.1 data: token: |- bmFtZXM6IDEyMzQ1全局值全局值是可以从任何 chart 或子 chart 中都可以访问的值，全局值需要显示的声明，不能将现有的非全局对象当作全局对象使用。Values 数据类型具有一个名为 Values.global 的保留部分，可以在其中设置全局值，我们在 values.yaml 文件中添加一个全局值：global: name: marksugar分别在文件中引用 name2: {{ .Values.global.name }}渲染PS H:\k8s-1.20.2\helm> helm template test .\liunuxea\ # Source: linuxea/templates/configmap.yaml apiVersion: v1 kind: Secret metadata: name: test-linuxea name2: marksugar type: Opaque data: my.cnf: |- bmFtZXM6IDEyMzQ= --- # Source: linuxea/charts/mychart/templates/configmap.yaml apiVersion: v1 kind: ConfigMap metadata: name: test-mychart data: mychart: is master chart name: marksugar --- # Source: linuxea/templates/configmap.yaml apiVersion: v1 kind: ConfigMap metadata: name: test-cmp labels: app: linuxea.com version: v0.1 data: token: |- bmFtZXM6IDEyMzQ1共享模板父级 chart 和子 chart 可以共享模板，任何 chart 中已定义的块都可以用于其他 chart。比如，我们可以定义一个简单的模板，如下所示：{{- define "labels" }}from: mychart{{ end }}前面我们提到过可以使用在模板中使用 include 和 template，但是使用 include 的一个优点是可以动态引入模板的内容：{{ include $mytemplate }}hookhelm提供一个钩子，可以在某个时间点进行干预，比如：在加载charts之前执行一个job备份数据库，安装完成后执行第二个job还原删除之前运行一个Job来关闭一些服务Hooks 的工作方式类似于普通的模板，但是他们具有特殊的注解，这些注解使 Helm 可以用不同的方式来使用他们。hooks在 Helm 中定义了如下一些可供我们使用的 Hooks：预安装pre-install：在模板渲染后，kubernetes 创建任何资源之前执行安装后post-install：在所有 kubernetes 资源安装到集群后执行预删除pre-delete：在从 kubernetes 删除任何资源之前执行删除请求删除后post-delete：删除所有 release 的资源后执行升级前pre-upgrade：在模板渲染后，但在任何资源升级之前执行升级后post-upgrade：在所有资源升级后执行预回滚pre-rollback：在模板渲染后，在任何资源回滚之前执行回滚后post-rollback：在修改所有资源后执行回滚请求测试test：在调用 Helm test 子命令的时候执行生命周期Hooks 允许开发人员在 release 的生命周期中的一些关键节点执行一些钩子函数，我们正常安装一个 chart 包的时候的生命周期如下所示：用户运行 helm install fooHelm 库文件调用安装 API经过一些验证，Helm 库渲染 foo 模板Helm 库将产生的资源加载到 kubernetes 中去Helm 库将 release 对象和其他数据返回给客户端Helm 客户端退出如果开发人员在 install 的生命周期中定义了两个 hook：pre-install和post-install，那么我们安装一个 chart 包的生命周期就会多一些步骤了：用户运行helm install fooHelm 库文件调用安装 API在 crds/ 目录下面的 CRDs 被安装经过一些验证，Helm 库渲染 foo 模板Helm 库将 hook 资源加载到 kubernetes 中，准备执行pre-install hooksHelm 库会根据权重对 hooks 进行排序（默认分配权重0，权重相同的 hook 按升序排序）Helm 库然后加载最低权重的 hookHelm 库会等待，直到 hook 准备就绪Helm 库将产生的资源加载到 kubernetes 中，注意如果添加了 --wait 参数，Helm 库会等待所有资源都准备好，在这之前不会运行 post-install hookHelm 库执行 post-install hook（加载 hook 资源）Helm 库等待，直到 hook 准备就绪Helm 库将 release 对象和其他数据返回给客户端Helm 客户端退出等待 hook 准备就绪，这是一个阻塞的操作，如果 hook 中声明的是一个 Job 资源，Helm 将等待 Job 成功完成，如果失败，则发布失败，在这个期间，Helm 客户端是处于暂停状态的。对于所有其他类型，只要 kubernetes 将资源标记为加载（添加或更新），资源就被视为就绪状态，当一个 hook 声明了很多资源是，这些资源是被串行执行的。另外需要注意的是 hook 创建的资源不会作为 release 的一部分进行跟踪和管理，一旦 Helm 验证了 hook 已经达到了就绪状态，它就不会去管它了。所以，如果我们在 hook 中创建了资源，那么不能依赖 helm uninstall 去删除资源，因为 hook 创建的资源已经不受控制了，要销毁这些资源，你需要将 helm.sh/hook-delete-policy 这个 annotation 添加到 hook 模板文件中，或者设置 Job 资源的生存（TTL）字段。编写hookHooks 就是 Kubernetes 资源清单文件，在元数据部分带有一些特殊的注解，因为他们是模板文件，所以你可以使用普通模板所有的功能，包括读取 .Values、.Release 和 .Template。例如，在 templates/post-install-job.yaml 文件中声明一个 post-install 的 hook：这个hook以job的形式在安装完成后触发，执行成功就删掉apiVersion: batch/v1 kind: Job metadata: .... annotations: # 因为添加了这个 hook，所以我们这个资源被定义为了 hook # 如果没有这行，则当前这个 Job 会被当成 release 的一部分内容。 "helm.sh/hook": post-install "helm.sh/hook-weight": "-5" "helm.sh/hook-delete-policy": hook-succeeded详细这个模板成为 hook 的原因就是添加这个注解：annotations: "helm.sh/hook": post-install一种资源也可以实现多个 hooks：annotations: "helm.sh/hook": post-install,post-upgrade类似的，实现给定 hook 的资源数量也没有限制，比如可以将 secret 和一个 configmap 都声明为 pre-install hook。当子 chart 声明 hooks 的时候，也会对其进行调用，顶层的 chart 无法禁用子 chart 所声明的 hooks。可以为 hooks 定义权重，这将有助于确定 hooks 的执行顺序：annotations: "helm.sh/hook-weight": "5"hook 权重可以是正数也可以是负数，但是必须用字符串表示，当 Helm 开始执行特定种类的 hooks 的时候，它将以升序的方式对这些 hooks 进行排序。渲染Chart.yamlapiVersion: v1 name: linuxea description: A Helm chart for Kubernetes type: application version: 0.1.2 appVersion: "1.16.0"job.yamlapiVersion: batch/v1 kind: Job metadata: name: "{{ .Release.Name }}" labels: app.kubernetes.io/managed-by: {{ .Release.Service | quote }} app.kubernetes.io/instance: {{ .Release.Name | quote }} app.kubernetes.io/version: {{ .Chart.AppVersion }} helm.sh/chart: "{{ .Chart.Name }}-{{ .Chart.Version }}" annotations: # 因为添加了这个 hook，所以我们这个资源被定义为了 hook # 如果没有这行，则当前这个 Job 会被当成 release 的一部分内容。 "helm.sh/hook": post-install "helm.sh/hook-weight": "-5" "helm.sh/hook-delete-policy": hook-succeeded spec: template: metadata: name: "{{ .Release.Name }}" labels: app.kubernetes.io/managed-by: {{ .Release.Service | quote }} app.kubernetes.io/instance: {{ .Release.Name | quote }} helm.sh/chart: "{{ .Chart.Name }}-{{ .Chart.Version }}" spec: restartPolicy: Never containers: - name: post-install-job image: "alpine:3.3" command: ["/bin/sleep","{{ default "10" .Values.sleepyTime }}"]configmap.yaml{{ $linuxea := .Release.Name -}} apiVersion: v1 kind: ConfigMap metadata: name: {{ .Release.Name }}-cmp labels: {{- include "mark.labels" . | indent 4 }} data: token: |- {{ .Files.Get "test3.json" | b64enc }} --- apiVersion: v1 kind: Secret metadata: name: {{ .Release.Name }}-linuxea type: Opaque data: my.cnf: |- {{ .Files.Get "test2.json" | b64enc }}values.yamlmydata: names: marksugar data: hi mark, my www.linuxea.com status: isok mylist: - mark - edwin - sean mychart: mark: "is master chart" global: name: marksugar开始安装PS H:\k8s-1.20.2\helm> helm install mark .\liunuxea\此时打开另外一个终端观察PS C:\WINDOWS\system32> kubectl.exe get pod -w NAME READY STATUS RESTARTS AGE dpment-linuxea-6bdfbd7b77-fr4pn 1/1 Running 16 40d dpment-linuxea-a-5b98f7fb86-9ff2f 1/1 Running 24 52d mark-hlrpf 0/1 ContainerCreating 0 14s nfs-client-provisioner-597f7dd4b-h2nsg 1/1 Running 86 277d testv1-9c974bd5d-gl52m 1/1 Running 16 40d testv2-5767685995-mjd6c 1/1 Running 23 52d traefik-6866c896d5-dqlv6 1/1 Running 16 40d whoami-7d666f84d8-8wmk4 1/1 Running 22 50d whoami-7d666f84d8-vlgb9 1/1 Running 16 40d whoamitcp-744cc4b47-24prx 1/1 Running 16 40d whoamitcp-744cc4b47-xrgqp 1/1 Running 16 40d whoamiudp-58f6cf7b8-b6njt 1/1 Running 16 40d whoamiudp-58f6cf7b8-jnq6c 1/1 Running 22 50d mark-hlrpf 1/1 Running 0 21s mark-hlrpf 0/1 Completed 0 31s mark-hlrpf 0/1 Terminating 0 32s mark-hlrpf 0/1 Terminating 0 32s可以看到执行完成后就被删除了这个job当这里执行完成，渲染也结束PS H:\k8s-1.20.2\helm> helm install mark .\liunuxea\ NAME: mark LAST DEPLOYED: Sun May 15 22:45:38 2022 NAMESPACE: default STATUS: deployed REVISION: 1 TEST SUITE: None NOTES: ================================== hello thank you install linuxea describe: $ helm status mark $ helm get mark通过ls查看PS H:\k8s-1.20.2\helm> helm ls NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION mark default 1 2022-05-15 22:45:38.6000054 +0800 CST deployed linuxea-0.1.2 1.16.0查看configmapPS H:\k8s-1.20.2\helm> kubectl.exe get configmap NAME DATA AGE istio-ca-root-cert 1 55d kube-root-ca.crt 1 339d mark-cmp 1 5m50s mark-mychart 2 5m50sdescribe查看内容PS H:\k8s-1.20.2\helm> kubectl.exe describe configmap mark-mychart Name: mark-mychart Namespace: default Labels: app.kubernetes.io/managed-by=Helm Annotations: meta.helm.sh/release-name: mark meta.helm.sh/release-namespace: default Data ==== mychart: ---- is master chart name: ---- marksugar Events: <none> PS H:\k8s-1.20.2\helm> kubectl.exe describe configmap mark-cmp Name: mark-cmp Namespace: default Labels: app=linuxea.com app.kubernetes.io/managed-by=Helm version=v0.1 Annotations: meta.helm.sh/release-name: mark meta.helm.sh/release-namespace: default Data ==== token: ---- bmFtZXM6IDEyMzQ1 Events: <none>hook删除策略还可以定义确定何时删除相应 hook 资源的策略，hook 删除策略可以使用下面的注解进行定义：annotations: "helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded我们也可以选择一个或多个已定义的注解：before-hook-creation：运行一个新的 hook 之前删除前面的资源（默认）hook-succeeded：hook 成功执行后删除资源hook-failed：hook 如果执行失败则删除资源如果未指定任何 hook 删除策略注解，则默认情况下会使用 before-hook-creation 策略。

linuxea:kubernetes中skywalking9.0部署使用 8.9.0是skywalking发布的最后一个功能版本，从2018年开始，skywalking一直是在服务，端点，实例间依赖的关系和拓扑结构，基于代理跟踪监控发展到全栈，包括日志，跟踪，指标和事件等。也添加了更多，如vm，k8s监控，服务网格。同时也引入了更多的方式来观测，如：ebpf但是在8.x的版本中使用了组的概念来解决混合的问题，但在v9核心中最重要的概念是LAYER层代表计算机科学中的一个抽象框架，例如操作系统（VM 层）、Kubernetes（k8s 层）、Service Mesh（典型的 Isto+Envoy 层），这种层将是从不同技术检测到的不同服务的所有者。相比较v8的组。显然，一个新layer概念要好得多。此外，group概念将被保留，因为它在每个layer，组将被设计为最终用户在内部对他们的服务进行分组。使用no group，它将在默认组中。这些在可视化UI中已经作为一个管理后台的dashboard的样式可视化（UI）databaseskubernetesservice meshgeneral servicebrowser这样导致SkyWalking 9.0.0 看起来是一个全栈 APM 系统，查看更多讨论部署skywalking9.0创建名称空间apiVersion: v1 kind: Namespace metadata: name: skywalking给es创建pvcapiVersion: apps/v1 kind: Deployment metadata: name: nfs-client-provisioner labels: app: nfs-client-provisioner # replace with namespace where provisioner is deployed namespace: default spec: replicas: 1 strategy: type: Recreate selector: matchLabels: app: nfs-client-provisioner template: metadata: labels: app: nfs-client-provisioner spec: serviceAccountName: nfs-client-provisioner containers: - name: nfs-client-provisioner image: quay.io/external_storage/nfs-client-provisioner:latest imagePullPolicy: IfNotPresent volumeMounts: - name: nfs-client-root mountPath: /persistentvolumes env: - name: PROVISIONER_NAME value: fuseim.pri/ifs - name: NFS_SERVER value: 192.168.3.19 - name: NFS_PATH value: /data/nfs-k8s volumes: - name: nfs-client-root nfs: server: 192.168.3.19 path: /data/nfs-k8s --- apiVersion: v1 kind: ServiceAccount metadata: name: nfs-client-provisioner # replace with namespace where provisioner is deployed namespace: default --- kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: nfs-client-provisioner-runner rules: - apiGroups: [""] resources: ["persistentvolumes"] verbs: ["get", "list", "watch", "create", "delete"] - apiGroups: [""] resources: ["persistentvolumeclaims"] verbs: ["get", "list", "watch", "update"] - apiGroups: ["storage.k8s.io"] resources: ["storageclasses"] verbs: ["get", "list", "watch"] - apiGroups: [""] resources: ["events"] verbs: ["create", "update", "patch"] --- kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: run-nfs-client-provisioner subjects: - kind: ServiceAccount name: nfs-client-provisioner # replace with namespace where provisioner is deployed namespace: default roleRef: kind: ClusterRole name: nfs-client-provisioner-runner apiGroup: rbac.authorization.k8s.io --- kind: Role apiVersion: rbac.authorization.k8s.io/v1 metadata: name: leader-locking-nfs-client-provisioner # replace with namespace where provisioner is deployed namespace: default rules: - apiGroups: [""] resources: ["endpoints"] verbs: ["get", "list", "watch", "create", "update", "patch"] --- kind: RoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: leader-locking-nfs-client-provisioner # replace with namespace where provisioner is deployed namespace: default subjects: - kind: ServiceAccount name: nfs-client-provisioner # replace with namespace where provisioner is deployed namespace: default roleRef: kind: Role name: leader-locking-nfs-client-provisioner apiGroup: rbac.authorization.k8s.io创建pvcapiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: nfs-storage namespace: default provisioner: fuseim.pri/ifs # or choose another name, must match deployment's env PROVISIONER_NAME' parameters: archiveOnDelete: "false" # Supported policies: Delete、 Retain ， default is Delete reclaimPolicy: Retain --- kind: PersistentVolumeClaim apiVersion: v1 metadata: name: pvc-skywalking namespace: skywalking spec: accessModes: - ReadWriteMany storageClassName: nfs-storage resources: requests: storage: 10Gi创建es pod# Source: skywalking/charts/elasticsearch/templates/statefulset.yaml apiVersion: v1 kind: Service metadata: name: elasticsearch namespace: skywalking labels: app: elasticsearch spec: type: ClusterIP ports: - name: elasticsearch port: 9200 protocol: TCP selector: app: elasticsearch --- apiVersion: apps/v1 kind: Deployment metadata: name: elasticsearch namespace: skywalking labels: app: elasticsearch spec: selector: matchLabels: app: elasticsearch replicas: 1 template: metadata: name: elasticsearch labels: app: elasticsearch spec: initContainers: - name: configure-sysctl securityContext: runAsUser: 0 privileged: true image: "docker.elastic.co/elasticsearch/elasticsearch:6.8.6" imagePullPolicy: "IfNotPresent" # command: ["sysctl", "-w", "vm.max_map_count=262144"] command: ["/bin/sh"] args: ["-c", "sysctl -w DefaultLimitNOFILE=65536; sysctl -w DefaultLimitMEMLOCK=infinity; sysctl -w DefaultLimitNPROC=32000; sysctl -w vm.max_map_count=262144"] resources: {} containers: - name: "elasticsearch" securityContext: capabilities: drop: - ALL runAsNonRoot: true runAsUser: 1000 image: "docker.elastic.co/elasticsearch/elasticsearch:6.8.6" imagePullPolicy: "IfNotPresent" livenessProbe: failureThreshold: 3 initialDelaySeconds: 30 periodSeconds: 2 successThreshold: 1 tcpSocket: port: 9300 timeoutSeconds: 2 readinessProbe: failureThreshold: 3 initialDelaySeconds: 30 periodSeconds: 2 successThreshold: 2 tcpSocket: port: 9300 timeoutSeconds: 2 ports: - name: http containerPort: 9200 - name: transport containerPort: 9300 resources: limits: cpu: 1000m memory: 2Gi requests: cpu: 100m memory: 2Gi env: - name: node.name valueFrom: fieldRef: fieldPath: metadata.name - name: cluster.name value: "elasticsearch" - name: network.host value: "0.0.0.0" - name: ES_JAVA_OPTS value: "-Xmx1g -Xms1g -Duser.timezone=Asia/Shanghai" - name: discovery.type value: single-node # value: "-Xmx1g -Xms1g -Duser.timezone=Asia/Shanghai MAX_OPEN_FILES=655350 MAX_LOCKED_MEMORY=unlimited" # - name: node.data # value: "true" # - name: node.ingest # value: "true" # - name: node.master # value: "true" # - name: http.cors.enabled # value: "true" # - name: http.cors.allow-origin # value: "*" # - name: http.cors.allow-headers # value: "X-Requested-With,X-Auth-Token,Content-Type,Content-Length,Authorization" # - name: bootstrap.memory_lock # value: "true" volumeMounts: - mountPath: /usr/share/elasticsearch/data name: elasticsearch-data restartPolicy: Always volumes: - name: elasticsearch-data persistentVolumeClaim: claimName: pvc-skywalking创建kabanakabana使用来管理es的，也可以使用其他的套件，如果有的话apiVersion: v1 kind: Service metadata: labels: app: kibana name: kibana namespace: skywalking spec: ports: - name: http port: 5601 protocol: TCP targetPort: 5601 selector: app: kibana type: ClusterIP --- apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: kibana-ui namespace: skywalking spec: ingressClassName: nginx rules: - host: local.kabana.com http: paths: - path: / pathType: Prefix backend: service: name: kibana port: number: 5601 --- apiVersion: apps/v1 kind: Deployment metadata: labels: app: kibana name: kibana namespace: skywalking spec: replicas: 1 selector: matchLabels: app: kibana template: metadata: labels: app: kibana spec: containers: - env: - name: ELASTICSEARCH_HOSTS value: http://elasticsearch:9200 image: kibana:6.8.6 imagePullPolicy: IfNotPresent livenessProbe: failureThreshold: 3 initialDelaySeconds: 10 periodSeconds: 2 successThreshold: 1 tcpSocket: port: 5601 timeoutSeconds: 2 name: kibana ports: - containerPort: 5601 name: http protocol: TCP readinessProbe: failureThreshold: 3 initialDelaySeconds: 10 periodSeconds: 2 successThreshold: 2 tcpSocket: port: 5601 timeoutSeconds: 2 resources: limits: cpu: "2" memory: 512Mi requests: cpu: 100m memory: 128Mi创建alarm configmap文件并且配置一个简单的告警模板apiVersion: v1 kind: ConfigMap metadata: name: alarm-configmap namespace: skywalking data: alarm-settings.yml: |- rules: # Rule unique name, must be ended with `_rule`. service_resp_time_rule: metrics-name: service_resp_time op: ">" threshold: 1000 period: 5 count: 3 silence-period: 5 message: Response time of service {name} is more than 1000ms in 3 minutes of last 10 minutes. service_sla_rule: # Metrics value need to be long, double or int metrics-name: service_sla op: "<" threshold: 8000 # The length of time to evaluate the metrics period: 10 # How many times after the metrics match the condition, will trigger alarm count: 2 # How many times of checks, the alarm keeps silence after alarm triggered, default as same as period. silence-period: 3 message: Successful rate of service {name} is lower than 80% in 2 minutes of last 10 minutes service_resp_time_percentile_rule: # Metrics value need to be long, double or int metrics-name: service_percentile op: ">" threshold: 1000,1000,1000,1000,1000 period: 10 count: 3 silence-period: 5 message: Percentile response time of service {name} alarm in 3 minutes of last 10 minutes, due to more than one condition of p50 > 1000, p75 > 1000, p90 > 1000, p95 > 1000, p99 > 1000 service_instance_resp_time_rule: metrics-name: service_instance_resp_time op: ">" threshold: 1000 period: 10 count: 2 silence-period: 5 message: Response time of service instance {name} is more than 1000ms in 2 minutes of last 10 minutes database_access_resp_time_rule: metrics-name: database_access_resp_time threshold: 1000 op: ">" period: 10 count: 2 message: Response time of database access {name} is more than 1000ms in 2 minutes of last 10 minutes endpoint_relation_resp_time_rule: metrics-name: endpoint_relation_resp_time threshold: 1000 op: ">" period: 10 count: 2 message: Response time of endpoint relation {name} is more than 1000ms in 2 minutes of last 10 minutes dingtalkHooks: textTemplate: |- { "msgtype": "text", "text": { "content": "Apache SkyWalking Alarm: \n %s." } } webhooks: - url: https://oapi.dingtalk.com/robot/send?access_token=0ca06927f1cd962ed8b47086 secret: SEC4c70c124f6148869de3285配置skywalking#ServiceAccount apiVersion: v1 kind: ServiceAccount metadata: labels: app: skywalking name: skywalking-oap namespace: skywalking --- kind: Role apiVersion: rbac.authorization.k8s.io/v1 metadata: name: skywalking namespace: skywalking labels: app: skywalking rules: - apiGroups: [""] resources: ["pods","configmaps"] verbs: ["get", "watch", "list"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: skywalking namespace: skywalking labels: app: skywalking roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: skywalking subjects: - kind: ServiceAccount name: skywalking-oap namespace: skywalking --- # es # apiVersion: v1 # kind: Service # metadata: # name: elasticsearch-master # namespace: skywalking # labels: # app: "elasticsearch-master" # spec: # type: ClusterIP # ports: # - name: elasticsearch-master # port: 9200 # protocol: TCP # --- # apiVersion: v1 # kind: Endpoints # metadata: # name: elasticsearch-master # namespace: skywalking # labels: # app: "elasticsearch-master" # subsets: # - addresses: # - ip: 192.168.0.13 # ports: # - name: elasticsearch-master # port: 9200 # protocol: TCP --- # oap apiVersion: v1 kind: Service metadata: name: skywalking-oap namespace: skywalking labels: app: skywalking-oap spec: type: ClusterIP ports: - port: 11800 name: grpc - port: 12800 name: rest selector: app: skywalking-oap chart: skywalking-4.2.0 --- apiVersion: apps/v1 kind: Deployment metadata: labels: app: skywalking-oap name: skywalking-oap namespace: skywalking spec: replicas: 1 selector: matchLabels: app: skywalking-oap template: metadata: labels: app: skywalking-oap chart: skywalking-4.2.0 spec: serviceAccountName: skywalking-oap affinity: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - weight: 1 podAffinityTerm: topologyKey: kubernetes.io/hostname labelSelector: matchLabels: app: "skywalking" release: "skywalking" component: "oap" initContainers: - name: wait-for-elasticsearch image: busybox:1.30 imagePullPolicy: IfNotPresent command: ['sh', '-c', 'for i in $(seq 1 60); do nc -z -w3 elasticsearch 9200 && exit 0 || sleep 5; done; exit 1'] containers: - name: oap image: skywalking.docker.scarf.sh/apache/skywalking-oap-server:9.0.0 # docker pull apache/skywalking-oap-server:8.8.1 imagePullPolicy: IfNotPresent livenessProbe: tcpSocket: port: 12800 initialDelaySeconds: 15 periodSeconds: 20 readinessProbe: tcpSocket: port: 12800 initialDelaySeconds: 15 periodSeconds: 20 ports: - containerPort: 11800 name: grpc - containerPort: 12800 name: rest env: - name: JAVA_OPTS value: "-Dmode=no-init -Xmx2g -Xms2g" - name: SW_CLUSTER value: kubernetes - name: SW_CLUSTER_K8S_NAMESPACE value: "default" - name: SW_CLUSTER_K8S_LABEL value: "app=skywalking,release=skywalking,component=oap" # 记录数据。 - name: SW_CORE_RECORD_DATA_TTL value: "2" # Metrics数据 - name: SW_CORE_METRICS_DATA_TTL value: "2" - name: SKYWALKING_COLLECTOR_UID valueFrom: fieldRef: fieldPath: metadata.uid - name: SW_STORAGE value: elasticsearch - name: SW_STORAGE_ES_CLUSTER_NODES value: "elasticsearch:9200" volumeMounts: - name: alarm-settings mountPath: /skywalking/config/alarm-settings.yml subPath: alarm-settings.yml volumes: - configMap: name: alarm-configmap name: alarm-settings --- # ui apiVersion: v1 kind: Service metadata: labels: app: skywalking-ui name: skywalking-ui namespace: skywalking spec: type: ClusterIP ports: - port: 80 targetPort: 8080 protocol: TCP selector: app: skywalking-ui --- apiVersion: apps/v1 kind: Deployment metadata: name: skywalking-ui namespace: skywalking labels: app: skywalking-ui spec: replicas: 1 selector: matchLabels: app: skywalking-ui template: metadata: labels: app: skywalking-ui spec: affinity: containers: - name: ui image: skywalking.docker.scarf.sh/apache/skywalking-ui:9.0.0 # docker pull apache/skywalking-ui:9.0.0 imagePullPolicy: IfNotPresent ports: - containerPort: 8080 name: page env: - name: SW_OAP_ADDRESS value: http://skywalking-oap:12800 --- # job apiVersion: batch/v1 kind: Job metadata: name: "skywalking-es-init" namespace: skywalking labels: app: skywalking-job spec: template: metadata: name: "skywalking-es-init" labels: app: skywalking-job spec: serviceAccountName: skywalking-oap restartPolicy: Never initContainers: - name: wait-for-elasticsearch image: busybox:1.30 imagePullPolicy: IfNotPresent command: ['sh', '-c', 'for i in $(seq 1 60); do nc -z -w3 elasticsearch 9200 && exit 0 || sleep 5; done; exit 1'] containers: - name: oap image: skywalking.docker.scarf.sh/apache/skywalking-oap-server:9.0.0 # docker pull apache/skywalking-oap-server:9.0.0 imagePullPolicy: IfNotPresent env: - name: JAVA_OPTS value: "-Xmx2g -Xms2g -Dmode=init" - name: SW_STORAGE value: elasticsearch - name: SW_STORAGE_ES_CLUSTER_NODES value: "elasticsearch:9200" # 记录数据。 # - name: SW_CORE_RECORD_DATA_TTL # value: "2" # Metrics数据 # - name: SW_CORE_METRICS_DATA_TTL # value: "2" volumeMounts: volumes: # --- # apiVersion: v1 # kind: Pod # metadata: # name: "skywalking-qyouc-test" # annotations: # "helm.sh/hook": test-success # spec: # containers: # - name: "skywalking-ggmnx-test" # image: "docker.elastic.co/elasticsearch/elasticsearch:6.8.6" # command: # - "sh" # - "-c" # - | # #!/usr/bin/env bash -e # curl -XGET --fail 'elasticsearch-master:9200/_cluster/health?wait_for_status=green&timeout=1s' # restartPolicy: Never --- apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: skywalking-ui namespace: skywalking spec: ingressClassName: nginx rules: - host: local.skywalking.com http: paths: - path: / pathType: Prefix backend: service: name: skywalking-ui port: number: 80创建skywalking service groupkubectl apply -f ns.yaml kubectl apply -f nas-to-es.yaml kubectl apply -f es.yaml kubectl apply -f kabana.yaml kubectl apply -f alarm.yaml kubectl apply -f 9.0.yamlpod引入在skywalking的donwload页面中选择Agent-> java agent。如，下载8.10.0https://www.apache.org/dyn/closer.cgi/skywalking/java-agent/8.10.0/apache-skywalking-java-agent-8.10.0.tgz将agent解压并添加到Dockerfile中，启动并指定jar包位置，如下..... COPY ./skywalking-agent /devops/skywalking-agent .... CMD java -javaagent:/devops/skywalking-agent/skywalking-agent.jar .....而后在pod的环境变量中配置必要的参数服务自动分组，根据${服务名称} = [${组名称}::]${逻辑名称}，一旦服务名称包含双冒号（::），冒号之前的文字字符串将被视为组名。在最新的 GraphQL 查询中，组名已作为选项参数提供。value: mark::test1 mark是组，test1是应用名称 - name: SW_AGENT_NAME value: mark::test1 - name: SW_AGENT_COLLECTOR_BACKEND_SERVICES value: skywalking-oap.skywalking:11800 而在第二个应用程序的时候就变成，mark::test2这样一来test1和test2都属于mark组，在界面中展示其他agent envirnment variable见Table of Agent Configuration Properties忽略url并不是所有的url都值得被关注，因此出于各方面考虑，配置忽略是有必要的。有两种方法可以配置忽略模式。通过系统环境设置具有更高的优先级。通过系统环境变量设置，需要添加skywalking.trace.ignore_path到系统变量中，值为需要忽略的路径，多个路径之间用,复制/agent/optional-plugins/apm-trace-ignore-plugin/apm-trace-ignore-plugin.config到/agent/config/目录，并添加规则以过滤跟踪trace.ignore_path=/your/path/1/**,/your/path/2/**实际中，在optional-plugins下将apm-trace-ignore-plugin-8.10.0.jar复制到plugins即可# cp optional-plugins/apm-trace-ignore-plugin-8.10.0.jar plugins/配置文件和环境变量任选其一，环境变量优先添加配置文件# cat config/apm-trace-ignore-plugin.config trace.ignore_path=${SW_AGENT_TRACE_IGNORE_PATH:GET:/health,/eureka/**}添加环境变量忽略GET:/health和/eureka/**的路径 - name: SW_AGENT_TRACE_IGNORE_PATH value: GET:/health,/eureka/**但是，忽略的URL不一定会马上生效，极大可能会延迟生效。参考忽略模式

linuxea:kube-prometheus远程存储victoriametrics 我们知道，在使用promentheus的过程中，默认的数据量一旦到一个量级后，查询区间的数据会非常缓慢，甚至一个查询就可能导致promentheus的崩溃，尽管我们不需要存储多久的数据，但是集群pod在一定的数量后，短期的数据仍然非常多，对于Promentheus本身的存储引擎来讲，仍是一个不小的问题，而使用外部存储就显得很有必要。早期流行的influxDB，由于社区对Promentheus并不友好，因此早些就放弃。此前，尝试了Prometheus远程存储Promscale和TimescaleDB测试，而后在讨论中发现VictoriaMetrics是更可取的方式。而VictoriaMetrics也有自己的一套系统监控。而在官方的介绍中，VictoriaMetrics强烈diss了TimescaleDBIt provides high data compression, so up to 70x more data points may be crammed into limited storage comparing to TimescaleDB and up to 7x less storage space is required compared to Prometheus, Thanos or Cortex.VictoriaMetrics可用于 Prometheus 监控数据做长期远程存储的时序数据库之一，而在github上是这样介绍的，截取部分如下可以直接用于 Grafana 作为 Prometheus 数据源使用指标数据摄取和查询具备高性能和良好的可扩展性，性能比 InfluxDB 和 TimescaleDB 高出 20 倍内存方面也做了优化，比 InfluxDB 少 10x 倍，比 Prometheus、Thanos 或 Cortex 少 7 倍其他有能够理解的部分话术针对具有高延迟 IO 和低 IOPS 的存储进行了优化提供全局的查询视图，多个 Prometheus 实例或任何其他数据源可能会将数据摄取到 VictoriaMetricsVictoriaMetrics 由一个没有外部依赖的小型可执行文件组成所有的配置都是通过明确的命令行标志和合理的默认值完成的所有数据都存储在 - storageDataPath 命令行参数指向的目录中可以使用 vmbackup/vmrestore 工具轻松快速地从实时快照备份到 S3 或 GCS 对象存储中支持从第三方时序数据库获取数据源由于存储架构原因，它可以保护存储在非正常关机（即 OOM、硬件重置或 kill -9）时免受数据损坏同样支持指标的 relabel 操作注意VictoriaMetrics 不支持prometheus本身读取，但是为了解决报警的问题，开发人员建议配置--storage.tsdb.retention.time=24h保留24小时的数据在prometheus中，而其他的数据写入到远程VictoriaMetrics ，通过grafana展示。VictoriaMetrics wiki说不支持prometheus读取，因为它发送的数据量很大； remote_read api 可以解决警报问题。我们可以启动一个 prometheus 实例，它只有 remote_read 配置部分和规则部分。victoriaMetrics 警报非常好！由于Prometheus中的这个问题，Prometheus 远程读取 API 不是为读取由其他 Prometheus 实例写入远程存储的数据而设计的。至于 Prometheus 中的警报，则将 Prometheus 本地存储保留设置为涵盖所有已配置警报规则的持续时间。通常 24 小时就足够了：--storage.tsdb.retention.time=24h. 在这种情况下，Prometheus 将对本地存储的数据执行警报规则，同时remote_write像往常一样将所有数据复制到配置的 url。而这些在github的wiki中以及为什么 VictoriaMetrics 不支持Prometheus 远程读取 API？有过说明远程读取 API 需要在给定时间范围内传输所有请求指标的所有原始数据。例如，如果一个查询包含 1000 个指标，每个指标有 10K 个值，那么远程读取 API 必须1000*10K向 Prometheus 返回 =10M 个指标值。这是缓慢且昂贵的。Prometheus 的远程读取 API 不适用于查询外部数据——也就是global query view. 有关详细信息，请参阅此问题。因此，只需通过vmui、Prometheus Querying API 或Grafana 中的 Prometheus 数据源直接查询 VictoriaMetrics 。VictoriaMetrics在VictoriaMetrics 中介绍如下VictoriaMetrics uses their modified version of LSM tree (Logging Structure Merge Tree). All the tables and indexes on the disk are immutable once created. When it's making the snapshot, they just create the hard link to the immutable files.VictoriaMetrics stores the data in MergeTree, which is from ClickHouse and similar to LSM. The MergeTree has particular design decision compared to canonical LSM.MergeTree is column-oriented. Each column is stored separately. And the data is sorted by the "primary key", and the "primary key" doesn't have to be unique. It speeds up the look-up through the "primary key", and gets the better compression ratio. The "parts" is similar to SSTable in LSM; it can be merged into bigger parts. But it doesn't have strict levels.The Inverted Index is built on "mergeset" (A data structure built on top of MergeTree ideas). It's used for fast lookup by given the time-series selector.提到的技术点， LSM 树，以及MergeTreeVictoriaMetrics 将数据存储在 MergeTree 中，MergeTree 来自 ClickHouse，类似于 LSM。与规范 LSM 相比，MergeTree 具有特定的设计决策。MergeTree 是面向列的。每列单独存储。并且数据按“主键”排序，“主键”不必是唯一的。它通过“主键”加快查找速度，获得更好的压缩比。“部分”类似于 LSM 中的 SSTable；它可以合并成更大的部分。但它没有严格的等级。倒排索引建立在“mergeset”（建立在 MergeTree 思想之上的数据结构）之上。通过给定时间序列选择器，它用于快速查找。为了能够有更多的理解，可以参考LSM Tree原理详解)：https://www.jianshu.com/p/b43b856e09bb应用到kube-prometheus对照如下kubernetes版本安装对应的kube-prometheus版本kube-prometheus stackKubernetes 1.19Kubernetes 1.20Kubernetes 1.21Kubernetes 1.22Kubernetes 1.23release-0.7✔✔✗✗✗release-0.8✗✔✔✗✗release-0.9✗✗✔✔✗release-0.10✗✗✗✔✔main✗✗✗✔✔Quickstart找到符合集群对应的版本进行安装，如果你是ack，需要卸载ack-arms-prometheus替换镜像k8s.gcr.io/prometheus-adapter/prometheus-adapter:v0.9.1 v5cn/prometheus-adapter:v0.9.1k8s.gcr.io/kube-state-metrics/kube-state-metrics:v2.4.2 bitnami/kube-state-metrics:2.4.2quay.io/brancz/kube-rbac-proxy:v0.12.0 bitnami/kube-rbac-proxy:0.12.0开始部署$ cd kube-prometheus $ git checkout main kubectl.exe create -f .\manifests\setup\ kubectl.exe create -f .\manifests配置ingress-nginx> kubectl.exe -n monitoring get svc NAME TYPE CLUSTER-IP PORT(S) alertmanager-main ClusterIP 192.168.31.49 9093/TCP,8080/TCP alertmanager-operated ClusterIP None 9093/TCP,9094/TCP,9094/UDP blackbox-exporter ClusterIP 192.168.31.69 9115/TCP,19115/TCP grafana ClusterIP 192.168.130.3 3000/TCP kube-state-metrics ClusterIP None 8443/TCP,9443/TCP node-exporter ClusterIP None 9100/TCP prometheus-adapter ClusterIP 192.168.13.123 443/TCP prometheus-k8s ClusterIP 192.168.118.39 9090/TCP,8080/TCP prometheus-operated ClusterIP None 9090/TCP prometheus-operator ClusterIP None 8443/TCP ingress-nginxapiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: monitoring-ui namespace: monitoring spec: ingressClassName: nginx rules: - host: local.grafana.com http: paths: - path: / pathType: Prefix backend: service: name: grafana port: number: 3000 --- apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: prometheus-ui namespace: monitoring spec: ingressClassName: nginx rules: - host: local.prom.com http: paths: - path: / pathType: Prefix backend: service: name: prometheus-k8s port: number: 9090配置nfs测试apiVersion: apps/v1 kind: Deployment metadata: name: nfs-client-provisioner labels: app: nfs-client-provisioner # replace with namespace where provisioner is deployed namespace: default spec: replicas: 1 strategy: type: Recreate selector: matchLabels: app: nfs-client-provisioner template: metadata: labels: app: nfs-client-provisioner spec: serviceAccountName: nfs-client-provisioner containers: - name: nfs-client-provisioner image: quay.io/external_storage/nfs-client-provisioner:latest imagePullPolicy: IfNotPresent volumeMounts: - name: nfs-client-root mountPath: /persistentvolumes env: - name: PROVISIONER_NAME value: fuseim.pri/ifs - name: NFS_SERVER value: 192.168.3.19 - name: NFS_PATH value: /data/nfs-k8s volumes: - name: nfs-client-root nfs: server: 192.168.3.19 path: /data/nfs-k8s --- apiVersion: v1 kind: ServiceAccount metadata: name: nfs-client-provisioner # replace with namespace where provisioner is deployed namespace: default --- kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: nfs-client-provisioner-runner rules: - apiGroups: [""] resources: ["persistentvolumes"] verbs: ["get", "list", "watch", "create", "delete"] - apiGroups: [""] resources: ["persistentvolumeclaims"] verbs: ["get", "list", "watch", "update"] - apiGroups: ["storage.k8s.io"] resources: ["storageclasses"] verbs: ["get", "list", "watch"] - apiGroups: [""] resources: ["events"] verbs: ["create", "update", "patch"] --- kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: run-nfs-client-provisioner subjects: - kind: ServiceAccount name: nfs-client-provisioner # replace with namespace where provisioner is deployed namespace: default roleRef: kind: ClusterRole name: nfs-client-provisioner-runner apiGroup: rbac.authorization.k8s.io --- kind: Role apiVersion: rbac.authorization.k8s.io/v1 metadata: name: leader-locking-nfs-client-provisioner # replace with namespace where provisioner is deployed namespace: default rules: - apiGroups: [""] resources: ["endpoints"] verbs: ["get", "list", "watch", "create", "update", "patch"] --- kind: RoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: leader-locking-nfs-client-provisioner # replace with namespace where provisioner is deployed namespace: default subjects: - kind: ServiceAccount name: nfs-client-provisioner # replace with namespace where provisioner is deployed namespace: default roleRef: kind: Role name: leader-locking-nfs-client-provisioner apiGroup: rbac.authorization.k8s.iovm配置创建一个pvc-victoriametricsapiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: nfs-storage namespace: default provisioner: fuseim.pri/ifs # or choose another name, must match deployment's env PROVISIONER_NAME' parameters: archiveOnDelete: "false" # Supported policies: Delete、 Retain ， default is Delete reclaimPolicy: Retain --- kind: PersistentVolumeClaim apiVersion: v1 metadata: name: pvc-victoriametrics namespace: monitoring spec: accessModes: - ReadWriteMany storageClassName: nfs-storage resources: requests: storage: 10Gi准备pvc[linuxea.com ~/victoriametrics]# kubectl apply -f pvc.yaml storageclass.storage.k8s.io/nfs-storage created persistentvolumeclaim/pvc-victoriametrics created [linuxea.com ~/victoriametrics]# kubectl get pv NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM ... pvc-97bea5fe-0131-4fb5-aaa9-66eee0802cb4 10Gi RWX Retain Bound monitoring/pvc-victoriametrics ... [linuxea.com ~/victoriametrics]# kubectl get pvc -A NAMESPACE NAME STATUS VOLUME CAPACITY ... monitoring pvc-victoriametrics Bound pvc-97bea5fe-0131-4fb5-aaa9-66eee0802cb4 10Gi 创建victoriametrics，并配置上面的pvc1w : 一周# vm-grafana.yaml apiVersion: apps/v1 kind: Deployment metadata: name: victoria-metrics namespace: monitoring spec: selector: matchLabels: app: victoria-metrics template: metadata: labels: app: victoria-metrics spec: containers: - name: vm image: victoriametrics/victoria-metrics:v1.76.1 imagePullPolicy: IfNotPresent args: - -storageDataPath=/var/lib/victoria-metrics-data - -retentionPeriod=1w ports: - containerPort: 8428 name: http resources: limits: cpu: "1" memory: 2048Mi requests: cpu: 100m memory: 512Mi readinessProbe: httpGet: path: /health port: 8428 initialDelaySeconds: 30 timeoutSeconds: 30 livenessProbe: httpGet: path: /health port: 8428 initialDelaySeconds: 120 timeoutSeconds: 30 volumeMounts: - mountPath: /var/lib/victoria-metrics-data name: victoriametrics-storage volumes: - name: victoriametrics-storage persistentVolumeClaim: claimName: nas-csi-pvc-oms-fat-victoriametrics --- apiVersion: v1 kind: Service metadata: name: victoria-metrics namespace: monitoring spec: ports: - name: http port: 8428 protocol: TCP targetPort: 8428 selector: app: victoria-metrics type: ClusterIPapply[linuxea.com ~/victoriametrics]# kubectl apply -f vmctoriametrics.yaml deployment.apps/victoria-metrics created service/victoria-metrics created [linuxea.com ~/victoriametrics]# kubectl -n monitoring get pod NAME READY STATUS RESTARTS AGE alertmanager-main-0 2/2 Running 88 268d blackbox-exporter-55c457d5fb-6rc8m 3/3 Running 114 260d grafana-756dc9b545-b2skg 1/1 Running 38 260d kube-state-metrics-76f6cb7996-j2hx4 3/3 Running 153 260d node-exporter-4hxzp 2/2 Running 120 316d node-exporter-54t9p 2/2 Running 124 316d node-exporter-8rfht 2/2 Running 120 316d node-exporter-hqzzn 2/2 Running 126 316d prometheus-adapter-59df95d9f5-7shw5 1/1 Running 78 260d prometheus-k8s-0 2/2 Running 89 268d prometheus-operator-7775c66ccf-x2wv4 2/2 Running 115 260d promoter-66f6dd475c-fdzrx 1/1 Running 3 8d victoria-metrics-56d47f6fb-qmthh 0/1 Running 0 15s [linuxea.com ~/victoriametrics]# kubectl -n monitoring get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE alertmanager-main NodePort 10.68.30.147 <none> 9093:30092/TCP 316d alertmanager-operated ClusterIP None <none> 9093/TCP,9094/TCP,9094/UDP 316d blackbox-exporter ClusterIP 10.68.25.245 <none> 9115/TCP,19115/TCP 316d etcd-k8s ClusterIP None <none> 2379/TCP 316d external-node-k8s ClusterIP None <none> 9100/TCP 315d external-pve-k8s ClusterIP None <none> 9221/TCP 305d external-windows-node-k8s ClusterIP None <none> 9182/TCP 316d grafana NodePort 10.68.133.224 <none> 3000:30091/TCP 316d kube-state-metrics ClusterIP None <none> 8443/TCP,9443/TCP 316d node-exporter ClusterIP None <none> 9100/TCP 316d prometheus-adapter ClusterIP 10.68.138.175 <none> 443/TCP 316d prometheus-k8s NodePort 10.68.207.185 <none> 9090:30090/TCP 316d prometheus-operated ClusterIP None <none> 9090/TCP 316d prometheus-operator ClusterIP None <none> 8443/TCP 316d promoter ClusterIP 10.68.26.69 <none> 8080/TCP 11d victoria-metrics ClusterIP 10.68.225.139 <none> 8428/TCP 18s修改prometheus的远程存储配置，我们主要修改如下，其他参数可在官方文档查看首先修改远程写如到vm remoteWrite: - url: "http://victoria-metrics:8428/api/v1/write" queueConfig: capacity: 5000 remoteTimeout: 30s并且prometheus的存储时间为1天retention: 1d一天的本地存储只是为了应对告警，而远程写入到vm后通过grafana来看Prometheus-prometheus.yamlapiVersion: monitoring.coreos.com/v1 kind: Prometheus metadata: labels: app.kubernetes.io/component: prometheus app.kubernetes.io/instance: k8s app.kubernetes.io/name: prometheus app.kubernetes.io/part-of: kube-prometheus app.kubernetes.io/version: 2.35.0 name: k8s namespace: monitoring spec: retention: 1d alerting: alertmanagers: - apiVersion: v2 name: alertmanager-main namespace: monitoring port: web enableFeatures: [] externalLabels: {} image: quay.io/prometheus/prometheus:v2.35.0 nodeSelector: kubernetes.io/os: linux podMetadata: labels: app.kubernetes.io/component: prometheus app.kubernetes.io/instance: k8s app.kubernetes.io/name: prometheus app.kubernetes.io/part-of: kube-prometheus app.kubernetes.io/version: 2.35.0 podMonitorNamespaceSelector: {} podMonitorSelector: {} probeNamespaceSelector: {} probeSelector: {} replicas: 1 resources: requests: memory: 400Mi remoteWrite: - url: "http://victoria-metrics:8428/api/v1/write" queueConfig: capacity: 5000 remoteTimeout: 30s ruleNamespaceSelector: {} ruleSelector: {} securityContext: fsGroup: 2000 runAsNonRoot: true runAsUser: 1000 serviceAccountName: prometheus-k8s serviceMonitorNamespaceSelector: {} serviceMonitorSelector: {} version: 2.35.0而此时的配置不出意外会被应用到URL/configremote_write: - url: http://victoria-metrics:8428/api/v1/write remote_timeout: 5m follow_redirects: true queue_config: capacity: 5000 max_shards: 200 min_shards: 1 max_samples_per_send: 500 batch_send_deadline: 5s min_backoff: 30ms max_backoff: 100ms metadata_config: send: true send_interval: 1m查看日志level=info ts=2022-04-28T15:26:12.047Z caller=main.go:944 msg="Loading configuration file" filename=/etc/prometheus/config_out/prometheus.env.yaml ts=2022-04-28T15:26:12.053Z caller=dedupe.go:112 component=remote level=info remote_name=1a1964 url=http://victoria-metrics:8428/api/v1/write msg="Starting WAL watcher" queue=1a1964 ts=2022-04-28T15:26:12.053Z caller=dedupe.go:112 component=remote level=info remote_name=1a1964 url=http://victoria-metrics:8428/api/v1/write msg="Starting scraped metadata watcher" ts=2022-04-28T15:26:12.053Z caller=dedupe.go:112 component=remote level=info remote_name=1a1964 url=http://victoria-metrics:8428/api/v1/write msg="Replaying WAL" queue=1a1964 .... totalDuration=55.219178ms remote_storage=85.51µs web_handler=440ns query_engine=719ns scrape=45.6µs scrape_sd=1.210328ms notify=4.99µs notify_sd=352.209µs rules=47.503195ms回到nfs查看[root@Node-172_16_100_49 /data/nfs-k8s/monitoring-pvc-victoriametrics-pvc-97bea5fe-0131-4fb5-aaa9-66eee0802cb4]# ll total 0 drwxr-xr-x 4 root root 48 Apr 28 22:37 data -rw-r--r-- 1 root root 0 Apr 28 22:37 flock.lock drwxr-xr-x 5 root root 71 Apr 28 22:37 indexdb drwxr-xr-x 2 root root 43 Apr 28 22:37 metadata drwxr-xr-x 2 root root 6 Apr 28 22:37 snapshots drwxr-xr-x 3 root root 27 Apr 28 22:37 tmp修改grafana的配置此时看到的数据是用promenteus中获取到的，修改grefana来从vm读取数据 datasources.yaml: |- { "apiVersion": 1, "datasources": [ { "access": "proxy", "editable": false, "name": "prometheus", "orgId": 1, "type": "prometheus", "url": "http://victoria-metrics:8428", "version": 1 } ] }顺便修改时区stringData: # 修改 时区 grafana.ini: | [date_formats] default_timezone = CST如下apiVersion: v1 kind: Secret metadata: labels: app.kubernetes.io/component: grafana app.kubernetes.io/name: grafana app.kubernetes.io/part-of: kube-prometheus app.kubernetes.io/version: 8.5.0 name: grafana-datasources namespace: monitoring stringData: # 修改链接的地址 datasources.yaml: |- { "apiVersion": 1, "datasources": [ { "access": "proxy", "editable": false, "name": "prometheus", "orgId": 1, "type": "prometheus", "url": "http://victoria-metrics:8428", "version": 1 } ] } type: Opaque --- apiVersion: v1 kind: Secret metadata: labels: app.kubernetes.io/component: grafana app.kubernetes.io/name: grafana app.kubernetes.io/part-of: kube-prometheus app.kubernetes.io/version: 8.5.0 name: grafana-config namespace: monitoring stringData: # 修改 时区 grafana.ini: | [date_formats] default_timezone = CST type: Opaque # grafana: # sidecar: # datasources: # enabled: true # label: grafana_datasource # searchNamespace: ALL # defaultDatasourceEnabled: false # additionalDataSources: # - name: Loki # type: loki # url: http://loki-stack.loki-stack:3100/ # access: proxy # - name: VictoriaMetrics # type: prometheus # url: http://victoria-metrics-single-server.victoria-metrics-single:8428 # access: proxy而此时的datasources就变成了vm，远程写入到了vm，grafana读取的是vm，而Prometheus还是读的是prometheus监控vmdashboards与版本有关，https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/dashboards并且添加监控# victoriametrics-metrics apiVersion: v1 kind: Service metadata: name: victoriametrics-metrics namespace: monitoring labels: app: victoriametrics-metrics annotations: prometheus.io/port: "8428" prometheus.io/scrape: "true" spec: type: ClusterIP ports: - name: metrics port: 8428 targetPort: 8428 protocol: TCP selector: # 对应victoriametrics的service app: victoria-metrics --- apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: victoriametrics-metrics namespace: monitoring spec: endpoints: - interval: 15s port: metrics path: /metrics namespaceSelector: matchNames: - monitoring selector: matchLabels: app: victoriametrics-metrics参考Prometheus远程存储Promscale和TimescaleDB测试victoriametricsLSM Tree原理详解

linuxea:helm3调试和基础函数if/with/range(3) 在开发一个chart的时候就需要对整个模板进行了解，从使用角度来说，是简单的。但是去开发的时候就需要一定的认识才能够完成一个模板的开发。而在其中用的较多的就是内置的一些函数，或者说是对象，通常，在helm的模板变量中，以大写开头的都是系统提供的，这由Go的函数约定。内置对象releaserelease：release是作为顶级对象的，与下面的其他不同的是她是内置函数。字面上release是发布的意思，这个对象描述了也是关于发布的一些信息，如下：Release.Name: 顾名思义，release名称Release.Namespace: release名称空间Release.IsUpgrade: release当在升级或者回滚的时候，值为trueRelease.IsInstall: 如果当前是安装则为trueRelease.Revision: release版本号，第一次安装是1，升级和回滚都会增加Release.Service: 渲染helm的服务values.yaml除此之外，还有values.yaml，这个文件的提供信息是传递到values的，而默认情况下是空的, 所以我们可以进去定制。values.yaml也是用的最多的文件Chart.yaml而chart.yaml的内容的值最终是被渲染到helm中的，这些使用helm ls的时候就可以看到，比如name和version等，这些信息可以进行定制值name: linuea Version: 1.1.1渲染后就变成了linuxea-1.1.1FilesFiles可以访问chart的非特殊文件，无法访问模板，提供以下参数Files.Get 或许文件，如： .Files.Get confuig.iniFiles.GetBytes 以bytes数组获取文件内函数Files.Glob 用于返回名称给到shell glob模式匹配的文件列表Files.Lines 逐行读取文件的函数，遍历每行内容Files.Assecrets 以Base64编码字符串返回Files.AsConfig 以yaml字典返回CapabilitiesCapabilities用于获取与kubernetes集群支持功能信息的对象Capabilities.APIVersion: 支持的版本Capabilities.APIVersion.Has.version 判断版本或者资源是否可用Capabilities.Kube.Version k8s版本，同时也是Capabilities.Kube 的缩写Capabilities.Kube.Major k8s主版本Capabilities.Kube.Minor k8s次版本TemplateTemplate包含当前正在执行的模板信息Name: 当前模板文件路径BasePath: 当前chart模板目录的路径创建chart使用create就可以进行创建，helm create --help可以看到帮助信息中有想要的参数，直接创建的将会是一个nginx的Deployment的yaml的清单，尝试创建即可helm create linuxea[root@linuxea.com /data/helm/mysql]# helm create linuxea WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /root/.kube/config WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /root/.kube/config Creating linuxea目录结构如下[root@linuxea.com /data/helm/mysql]# tree linuxea/ linuxea/ ├── charts ├── Chart.yaml ├── templates │   ├── deployment.yaml │   ├── _helpers.tpl │   ├── hpa.yaml │   ├── ingress.yaml │   ├── NOTES.txt │   ├── serviceaccount.yaml │   ├── service.yaml │   └── tests │   └── test-connection.yaml └── values.yaml 3 directories, 10 filesvalues.yaml实际上我们不需要创建的这些文件，只需要一个目录结构即可，最重要的是values.yaml这个文件可以使用-f传递给helm install或helm upgradehelm install -f values.yaml linuxea再者，使用--set传递各个参数helm install --set image=123 linuxeavalues.yaml的是被用来渲染模板，而--set的值是优先于values.yaml的，因此--set可以覆盖values.yaml的值既然默认创建的chart并不是期望的，可以删除模板文件和vlaues.yaml的内容，或者手动创建目录结构即可手动创建目录mkdir -p liunuxea/{templates,charts} touch values.yaml liunuxea/ touch configmap.yaml liunuxea/templates/configmap.yamlcat > liunuxea/Chart.yaml << EOF apiVersion: v2 name: linuxea description: A Helm chart for Kubernetes type: application version: 0.1.0 appVersion: "1.16.0" EOF结构如下[root@linuxea.com /data/helm/mysql]# tree liunuxea/ liunuxea/ ├── charts ├── Chart.yaml ├── templates │   └── configmap.yaml └── values.yaml 2 directories, 3 files示例1此时渲染一个参数的值，比如，有一个configmapapiVersion: v1 kind: ConfigMap metadata: name: linuxea-cmp labels: app: linuxea-cmp data: test: 这里的test是空的，在values.yaml中定义一行mydata: hi mark, my www.linuxea.com而现在要想在configmap中使用values.yaml的mydata，就需要使用{{ .Vaules }}来使用apiVersion: v1 kind: ConfigMap metadata: name: linuxea-cmp labels: app: linuxea-cmp data: test: {{ .Values.mydata }}{{ .Values.mydata }}这种方式是go的模板方式helm使用template可以进行渲染，直接在windows就可以渲染模板以共查看PS H:\k8s-1.20.2\helm> helm.exe template test .\liunuxea\ --- # Source: linuxea/templates/configmap.yaml apiVersion: v1 kind: ConfigMap metadata: name: linuxea-cmp labels: app: linuxea-cmp data: test: hi mark, my www.linuxea.com除此之外，当我们输入的是test的时候，我们希望test被替换城什么值的时候就可以使用Release比如，Release.NameapiVersion: v1 kind: ConfigMap metadata: name: {{ .Release.Name }}-cmp labels: app: {{ .Release.Name }}-cmp data: test: {{ .Values.mydata }}这次chart名称换 marksugarhelm.exe template marksugar .\liunuxea\PS H:\k8s-1.20.2\helm> helm.exe template marksugar .\liunuxea\ --- # Source: linuxea/templates/configmap.yaml apiVersion: v1 kind: ConfigMap metadata: name: marksugar-cmp labels: app: marksugar-cmp data: test: hi mark, my www.linuxea.com这里引用{{ .Release.Name }}的地方就被替换城了marksugar--set除此之外，还可以使用helm的--set来覆盖 --set mydata=linuxea.comPS H:\k8s-1.20.2\helm> helm.exe template marksugar --set mydata=linuxea.com .\liunuxea\ --- # Source: linuxea/templates/configmap.yaml apiVersion: v1 kind: ConfigMap metadata: name: marksugar-cmp labels: app: marksugar-cmp data: test: linuxea.com--dry-runtemplate只是用来本地模板渲染，但是--dry-run是用来模拟安装，这两个在调试的时候有一些差别，--dry-run也可以使用--debughelm install marksugar --dry-run --set mydata=linuxea.com liunuxea/如下[root@linuxea.com /data/helm]# helm install marksugar --dry-run --set mydata=linuxea.com liunuxea/ WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /root/.kube/config WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /root/.kube/config NAME: marksugar LAST DEPLOYED: Sat Apr 16 02:09:37 2022 NAMESPACE: default STATUS: pending-install REVISION: 1 TEST SUITE: None HOOKS: MANIFEST: --- # Source: linuxea/templates/configmap.yaml apiVersion: v1 kind: ConfigMap metadata: name: marksugar-cmp labels: app: marksugar-cmp data: test: linuxea.com示例2而values.yaml有多个值的时候，如mydata: names: linuxea data: hi mark, my www.linuxea.com引用的时候就发生了变化apiVersion: v1 kind: ConfigMap metadata: name: {{ .Release.Name }}-cmp labels: app: {{ .Release.Name }}-cmp data: name: {{ .Values.mydata.names }} test: {{ .Values.mydata.data }}本地渲染PS H:\k8s-1.20.2\helm> helm.exe template marksugar .\liunuxea\ --- # Source: linuxea/templates/configmap.yaml apiVersion: v1 kind: ConfigMap metadata: name: marksugar-cmp labels: app: marksugar-cmp data: name: linuxea test: hi mark, my www.linuxea.com--set覆盖也是一样PS H:\k8s-1.20.2\helm> helm.exe template marksugar --set mydata.data=linuxea.com .\liunuxea\ --- # Source: linuxea/templates/configmap.yaml apiVersion: v1 kind: ConfigMap metadata: name: marksugar-cmp labels: app: marksugar-cmp data: name: linuxea test: linuxea.com基本语法函数和管道函数values.yaml有些托拉，有时候更希望有一些函数更快的方式转换来提供数据，而这些函数大多数都由golang的template提供。这里面包含了一些逻辑和运算。另外，还有一些模板在sprig的包，如日期，时间等。阅读模板函数能更快了解quote: 转换成字符串，加上双引号upper: 转换大写with： 作用域.Values中的点(.)其实就是作用域。而with可以控制变量的作用域并且重新使用，调用就是对当前作用域的引用，.values是在当前作用域下查找values对象indent: 空格indent的用户是要顶行进行配置，比如想让test: ok前有两个空格，如下data: name: {{ .Values.mydata.names | default "supper" }} test: {{ .Values.mydata.data | upper | repeat 5 }} status: ok namestatus: "true" defaultstaus: true {{ indent 2 "test: ok" }}管道管道非常常用，并没有什么特别的区别参数 | 函数 参数 | 函数 | 函数2如上：转换字符串的话{{ .Values.mydata.name | quote | upper }} {{ .Values.mydata.name | upper | quote }}repeat : repeat COUNT STRING 将字符串重复多少次如下apiVersion: v1 kind: ConfigMap metadata: name: {{ .Release.Name }}-cmp labels: app: {{ .Release.Name }}-cmp data: name: {{ .Values.mydata.names }} test: {{ .Values.mydata.data | upper | repeat 5 }}渲染PS H:\k8s-1.20.2\helm> helm.exe template marksugar .\liunuxea\ --- # Source: linuxea/templates/configmap.yaml apiVersion: v1 kind: ConfigMap metadata: name: marksugar-cmp labels: app: marksugar-cmp data: name: linuxea test: HI MARK, MY WWW.LINUXEA.COMHI MARK, MY WWW.LINUXEA.COMHI MARK, MY WWW.LINUXEA.COMHI MARK, MY WWW.LINUXEA.COMHI MARK, MY WWW.LINUXEA.COMdefault ： 如果没有给定参数就使用默认参数这个在shell中也很常用，在helm也非常常用，特别是在你的values.yaml中不能使用一个静态配置的时候，default就排上用场了举个例子，如果names变量为空，就默认赋值supperapiVersion: v1 kind: ConfigMap metadata: name: {{ .Release.Name }}-cmp labels: app: {{ .Release.Name }}-cmp data: name: {{ .Values.mydata.names | default "supper" }} test: {{ .Values.mydata.data | upper | repeat 5 }}values.yaml中把值删掉mydata: names: data: hi mark, my www.linuxea.com渲染下PS H:\k8s-1.20.2\helm> helm.exe template marksugar .\liunuxea\ --- # Source: linuxea/templates/configmap.yaml apiVersion: v1 kind: ConfigMap metadata: name: marksugar-cmp labels: app: marksugar-cmp data: name: supper test: HI MARK, MY WWW.LINUXEA.COMHI MARK, MY WWW.LINUXEA.COMHI MARK, MY WWW.LINUXEA.COMHI MARK, MY WWW.LINUXEA.COMHI MARK, MY WWW.LINUXEA.COMprintfprintf和在go中的printf类似，用来打印字符串或者其他的数组，如下test: {{ .Vaules.test | default (printf "%s-1" (include "test" ))}}条件语句流程控制在go template中也是非常重要的一部分，和大多数判断语法一样，不但如此，还有一些声明和命名模板的操作，如下：if/else 条件with 作用域范围range 遍历的一种，类似与for eachdefine 在模板内声明新的命名模板template 导入一个命名模板blok 声明特殊的可填充模块区域if/else if 条件判断不管是在什么语言类型中，都是有一个结束标记的，而在helm中使用的是end结束{{ if TEST }} test {{ else if TEST2 }} test2 {{ elese }} default test {{ end }}如果test为真则test，如果test为test2则test2，否则defalut test判断的是管道而非values值，这里控制结构可以执行这个判断语句管道，而不仅仅是一个值，如果结果是如下几项，则为false，否则则是true:布尔 false0空字符串nil或者null空的集合，如map,slice,tuple,dict,array示例1 if以上个configmap为例，如果此时希望用一个判断语句来确定是否添加一个字段的话，就可以使用if来进行操作{{ if eq .Values.mydata.status "isok" }}status: ok{{ end }}如果此时使用的是换行的if，如{{ if eq .Values.mydata.status "isok" }} status: ok {{ end }}那么很有可能不在一行，解决这个问题的版本是加一个--是用来删除if之后和end之前的空格的，不可以加载if之前，加在前面就跳到上一行了{{ if eq .Values.mydata.status "isok" -}} status: ok {{- end }}如果Values.mydata.status等于"isok"，就添加status: okapiVersion: v1 kind: ConfigMap metadata: name: {{ .Release.Name }}-cmp labels: app: {{ .Release.Name }}-cmp data: name: {{ .Values.mydata.names | default "supper" }} test: {{ .Values.mydata.data | upper | repeat 5 }} {{ if eq .Values.mydata.status "isok" }}status: ok{{ end }}values.yamlmydata: names: marksugar data: hi mark, my www.linuxea.com status: isok渲染PS H:\k8s-1.20.2\helm> helm.exe template marksugar .\liunuxea\ --- # Source: linuxea/templates/configmap.yaml apiVersion: v1 kind: ConfigMap metadata: name: marksugar-cmp labels: app: marksugar-cmp data: name: marksugar test: HI MARK, MY WWW.LINUXEA.COMHI MARK, MY WWW.LINUXEA.COMHI MARK, MY WWW.LINUXEA.COMHI MARK, MY WWW.LINUXEA.COMHI MARK, MY WWW.LINUXEA.COM status: okelse if除此之外，还可以进行嵌套else {{ if eq .Values.mydata.status "isok" -}} status: ok {{ else if eq .Values.mydata.names "marksugar" -}} namestatus: "true" {{ else -}} defaultstaus: true {{- end }}如果等于isok，就结束，否则就继续判断第二个else if，如果第二个else if等于marksugar就结束，否则就else的默认值。如下NAME -}}{{- NAME这里的-号是删除空格的意思apiVersion: v1 kind: ConfigMap metadata: name: {{ .Release.Name }}-cmp labels: app: {{ .Release.Name }}-cmp data: name: {{ .Values.mydata.names | default "supper" }} test: {{ .Values.mydata.data | upper | repeat 5 }} {{ if eq .Values.mydata.status "isok" -}} status: ok {{ else if eq .Values.mydata.names "marksugar" -}} namestatus: "true" {{ else -}} defaultstaus: true {{- end }}执行PS H:\k8s-1.20.2\helm> helm template .\liunuxea\ --- # Source: linuxea/templates/configmap.yaml apiVersion: v1 kind: ConfigMap metadata: name: release-name-cmp labels: app: release-name-cmp data: name: marksugar test: HI MARK, MY WWW.LINUXEA.COMHI MARK, MY WWW.LINUXEA.COMHI MARK, MY WWW.LINUXEA.COMHI MARK, MY WWW.LINUXEA.COMHI MARK, MY WWW.LINUXEA.COM status: ok由于第一个if执行完成就是true，逻辑结束示例2 with上述的.Values是对当前模板引擎下的作用域内查找.vaules对象，而with是控制变量的作用域，并且和if语句类似with中默认是不能使用内置的函数，比如：.Release.Name之类的，如果要使用，需要另外的办法使用示例1的文件重新配置。如下 {{ with .Values.mydata -}} {{ if eq .status "isok" -}} status: ok {{- else if eq .names "marksugar" -}} namestatus: "true" {{ else -}} defaultstaus: true {{- end }} {{- end }}一旦 {{ with .Values.mydata -}}配置之后，在whth中，就不在使用.Vaules.mydata.names来调用了，而是.namesapiVersion: v1 kind: ConfigMap metadata: name: {{ .Release.Name }}-cmp labels: app: {{ .Release.Name }}-cmp data: name: {{ .Values.mydata.names | default "supper" }} test: {{ .Values.mydata.data | upper | repeat 5 }} {{ with .Values.mydata -}} {{ if eq .status "isok" -}} status: ok {{- else if eq .names "marksugar" -}} namestatus: "true" {{ else -}} defaultstaus: true {{- end }} {{- end }} {{ indent 2 "test: ok" }}示例3 range通过在helm中使用range来遍历，比如使用range遍历一个列表或者字典首先定义一个列表mylistmylist: - mark - edwin - sean 而后使用range mylist: |- {{- range .Values.mylist }} - {{ . | title | quote }} {{- end }}这里的|-的作用，|是将以下的数据放在一个字符串中，如果不加|就不是一个字符串了。而-是换行而这里的.的作用域是在range中的，作用域与range开始，end结束|-和|+|-：|-的意思是将文末的换行符删掉|+|+的意思是将文末的换行符保留参考helm3的简单使用(1)helm3模板渲染(2)kubernetes helm概述(49)kubernetes helm简单使用(50)kubernetes 了解chart(51)kubernetes helm安装efk(52)

linuxea:kubernetes检测pod部署状态简单实现 通常，无状态的应用多数情况下以deployment控制器下运行，在deployment更新中，当配置清单发生变化后，应用这些新的配置。我们假设一些都ok，也成功拉取镜像，并且以默认的25%进行滚动更新，直到更新完成。kubectl apply -f test1.yaml然而这一切按照预期进行，没有任何问题。kubectl只是将配置推送到k8s后，只要配置清单没有语法或者冲突问题，返回的是0，状态就是成功的而整个过程有很多不确定性，比如，不存在的镜像，没有足够的资源调度，配置错误导致的一系列问题，而捕捉这种问题也是比较关键的事情之一。这并不单纯的简单观测问题，pod并不是拉取镜像就被running起，一旦runing就意味着接收到流量，而程序准备需要时间，如果此时程序没有准备好，流量就接入，势必会出现错误。为了解决这个问题，就需要配置就绪检测或者Startup检测pod在被真正的处于ready起来之前，通常会做就绪检测，或者启动检测。在之前的几篇中，我记录了就绪检测和健康检测的重要性，而在整个就绪检测中，是有一个初始化时间的问题。如果此时，配置清单发送变化，调度器开始执行清单任务。假设此时的初始化准备时间是100秒，有30个pod，每次至少保证有75%是正常运行的，默认按照25%滚动更新Updating a Deployment。此时的准备时间(秒)至少是30 / 25% * (100)readiness probe time 如果pod越多，意味着等待所有pod就绪完成的总时间就越长，如果放在cd管道中去运行，势必会让反馈时间越久。当一个重量级的集群中，每一条全局遍历都非常消耗资源，因此操作非常昂贵。整个集群有可能因此产生大的延迟，在集群外部调用API间隔去获取远比实时获取消耗资源要少。如果pod并不多，这个问题不值得去考量。使用rollout足以解决。获取清单被推送到节点的方式，有如下：事件监控pod在更新的时候，如果有问题会触发事件watch状态以及其他第三方的编码来达到这个需求，比如由额外的程序来间隔时间差去检测状态，而不是一直watch通常，使用kubectl rollout或者helm的--wait，亦或者argocd的平面控制来观测rollout在kubernetes的文档中，rollout的页面中提到的检查状态rollout能够管理资源类型如：部署，守护进程，状态阅读rolling-back-a-deployment中的status watch，得到以下配置kubectl -n NAMESPACE rollout status deployment NAME --watch --timeout=Xmrollout下的其他参数history列出deployment/testv1的历史kubectl -n default rollout history deployment/testv1查看历史记录的版本1信息kubectl -n default rollout history deployment/testv1 --revision=1pause停止，一旦停止，更新将不会生效kubectl rollout pause deployment/testv1需要恢复，或者重启resume恢复，恢复后延续此前暂停的部署kubectl rollout resume deployment/testv1status此时可以配置status查看更新过程的状态kubectl rollout status deployment/testv1status提供了一下参数，比如常用的超时比如，--timeout=10m，最长等待10m，超过10分钟就超时kubectl rollout status deployment/testv1 --watch --timeout=10m 其他参数如下NameShorthandDefaultUsagefilenamef[]Filename, directory, or URL to files identifying the resource to get from a server.kustomizek Process the kustomization directory. This flag can't be used together with -f or -R.recursiveRfalseProcess the directory used in -f, --filename recursively. Useful when you want to manage related manifests organized within the same directory.revision 0Pin to a specific revision for showing its status. Defaults to 0 (last revision).timeout 0sThe length of time to wait before ending watch, zero means never. Any other values should contain a corresponding time unit (e.g. 1s, 2m, 3h).watchwtrueWatch the status of the rollout until it's done.undo回滚回滚到上一个版本kubectl rollout undo deployment/testv1 回滚到指定的版本1,查看已有版本# kubectl -n default rollout history deployment/testv1 deployment.apps/testv1 REVISION CHANGE-CAUSE 1 <none> 2 <none> 3 <none>查看版本信息# kubectl rollout history deployment/testv1 deployment.apps/testv1 REVISION CHANGE-CAUSE 2 <none> 7 <none> 8 <none>2，回滚到2# kubectl rollout undo deployment/testv1 --to-revision=2 deployment.apps/testv1 rolled backhelm--waitif set, will wait until all Pods, PVCs, Services, and minimum number of Pods of a Deployment, StatefulSet, or ReplicaSet are in a ready state before marking the release as successful. It will wait for as long as --timeout只有在控制器的pod出于就绪状态才会结束，默认时间似乎是600秒·看起来像是这样helm upgrade --install --namespace NAMESPACE --create-namespace --wait APP FILEAPI上面两种方式能够完成大部分场景，但是watch是非常占用资源，如果希望通过一个脚本自己的逻辑去处理，可以使用clent-go的包手动for循环查看状态clent-go手动去for循环package main import ( "context" "flag" "fmt" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/client-go/kubernetes" typev1 "k8s.io/client-go/kubernetes/typed/apps/v1" "k8s.io/client-go/rest" "k8s.io/client-go/util/retry" "os" "time" ) type args struct { namespace string image string deployment string } const ( numberOfPoll = 200 pollInterval = 3 ) func parseArgs() *args { namespace := flag.String("n", "", "namespace") deployment := flag.String("deploy", "", "deployment name") image := flag.String("image", "", "image for update") flag.Parse() var _args args if *namespace == "" { fmt.Fprintln(os.Stderr, "namespace must be specified") os.Exit(1) } _args.namespace = *namespace if *deployment == "" { fmt.Fprintln(os.Stderr, "deployment must be specified") os.Exit(1) } _args.deployment = *deployment if *image == "" { fmt.Fprintln(os.Stderr, "image must be specified") os.Exit(1) } _args.image = *image return &_args } func main() { _args := parseArgs() // creates the in-cluster config config, err := rest.InClusterConfig() if err != nil { panic(err.Error()) } // creates the clientset clientset, err := kubernetes.NewForConfig(config) if err != nil { panic(err.Error()) } deploymentsClient := clientset.AppsV1().Deployments(_args.namespace) ctx := context.Background() retryErr := retry.RetryOnConflict(retry.DefaultRetry, func() error { // Retrieve the latest version of Deployment before attempting update // RetryOnConflict uses exponential backoff to avoid exhausting the apiserver result, getErr := deploymentsClient.Get(ctx, _args.deployment, metav1.GetOptions{}) if getErr != nil { fmt.Fprintf(os.Stderr, "Failed to get latest version of Deployment %s: %v", _args.deployment, getErr) os.Exit(1) } result.Spec.Template.Spec.Containers[0].Image = _args.image _, updateErr := deploymentsClient.Update(ctx, result, metav1.UpdateOptions{}) return updateErr }) if retryErr != nil { fmt.Fprintf(os.Stderr, "Failed to update image version of %s/%s to %s: %v", _args.namespace, _args.deployment, _args.image, retryErr) os.Exit(1) } _args.pollDeploy(deploymentsClient) fmt.Println("Updated deployment") } // watch 太浪费资源了，而且时间太长，还是轮询吧 func (p *args) pollDeploy(deploymentsClient typev1.DeploymentInterface) { ctx := context.Background() for i := 0; i <= numberOfPoll; i++ { time.Sleep(pollInterval * time.Second) result, getErr := deploymentsClient.Get(ctx, p.deployment, metav1.GetOptions{}) if getErr != nil { fmt.Fprintf(os.Stderr, "Failed to get latest version of Deployment %s: %v", p.deployment, getErr) os.Exit(1) } resourceStatus := result.Status fmt.Printf("%s -> replicas: %d, ReadyReplicas: %d, AvailableReplicas: %d, UpdatedReplicas: %d, UnavailableReplicas: %d\n", result.Name, resourceStatus.Replicas, resourceStatus.ReadyReplicas, resourceStatus.AvailableReplicas, resourceStatus.UpdatedReplicas, resourceStatus.UnavailableReplicas) if resourceStatus.Replicas == resourceStatus.ReadyReplicas && resourceStatus.ReadyReplicas == resourceStatus.AvailableReplicas && resourceStatus.AvailableReplicas == resourceStatus.UpdatedReplicas { return } } fmt.Fprintf(os.Stderr, "应用在 %d 秒内没有启动成功，视作启动失败，请查看日志。\n", numberOfPoll*pollInterval) os.Exit(1) }其他参考kubernetes-deployment-status-in-jenkinsKubernetes探针补充Kubernetes Liveness 和 Readiness 探测避免给自己挖坑续集重新审视kubernetes活跃探针和就绪探针 如何避免给自己挖坑2Kubernetes Startup Probes避免给自己挖坑3

linuxea:helm3模板渲染(2) helm说白了其实就是一个模板渲染系统，核心就在templates和values，模板使用了go template编写的，并且增加了sprig库，共计50个左右的附加模板函数和其他的一些函数。如果要进行使用，必然要遵循template的模板的约定)，比如{{ if pipeline }} T1 {{ else }} {{ if pipeline }} T0 {{end}} { end} {{ range pipeline }} T1 {{ end }}if或者range开头的都需要end结尾，并且可以指定引用a模板{{ templete "a" }}这些，可以在go文档库中找到，但是这些还不够，sprig还能解决一些大小写等的问题。除此之外，helm的docs中也有自己的一些函数。而这些模板存储在templates目录下, 当helm渲染charts，就会通过模板引擎传递目录中文件，而values可以通过两种方式提供：Chart 开发人员可以在 chart 内部提供一个名为 values.yaml 的文件，该文件可以包含默认的 values 值内容。Chart 用户可以提供包含 values 值的 YAML 文件，可以在命令行中通过 helm install 来指定该文件。当用户提供自定义 values 值的时候，这些值将覆盖 chart 中 values.yaml 文件中的相应的值。简单示例mysql的包，目录结构如下[root@linuxea.com /data/helm/mysql]# tree ./ ./ ├── Chart.yaml ├── README.md ├── templates │   ├── configurationFiles-configmap.yaml │   ├── deployment.yaml │   ├── _helpers.tpl │   ├── initializationFiles-configmap.yaml │   ├── NOTES.txt │   ├── pvc.yaml │   ├── secrets.yaml │   ├── serviceaccount.yaml │   ├── servicemonitor.yaml │   ├── svc.yaml │   └── tests │   ├── test-configmap.yaml │   └── test.yaml └── values.yaml 2 directories, 15 files打开svc.conf你会看到如下的格式，这种就是template的样式，这些值是通过value来进行替换成value中的值[root@linuxea.com /data/helm/mysql]# cat templates/svc.yaml apiVersion: v1 kind: Service metadata: name: {{ template "mysql.fullname" . }} namespace: {{ .Release.Namespace }} labels: app: {{ template "mysql.fullname" . }} chart: "{{ .Chart.Name }}-{{ .Chart.Version }}" release: "{{ .Release.Name }}" heritage: "{{ .Release.Service }}" annotations: {{- if .Values.service.annotations }} {{ toYaml .Values.service.annotations | indent 4 }} {{- end }} {{- if and (.Values.metrics.enabled) (.Values.metrics.annotations) }} {{ toYaml .Values.metrics.annotations | indent 4 }} {{- end }} spec: type: {{ .Values.service.type }} {{- if (and (eq .Values.service.type "LoadBalancer") (not (empty .Values.service.loadBalancerIP))) }} loadBalancerIP: {{ .Values.service.loadBalancerIP }} {{- end }} ports: - name: mysql port: {{ .Values.service.port }} targetPort: mysql {{- if .Values.service.nodePort }} nodePort: {{ .Values.service.nodePort }} {{- end }} {{- if .Values.mysqlx.port.enabled }} - name: mysqlx port: 33060 targetPort: mysqlx protocol: TCP {{- end }} {{- if .Values.metrics.enabled }} - name: metrics port: 9104 targetPort: metrics {{- end }} selector: app: {{ template "mysql.fullname" . }} 以上面port: {{ .Values.service.port }}为例 ，values.yaml的值如下，意思就是替换城3306端口service: annotations: {} type: ClusterIP port: 3306而这种方式也可以通过--set 来进行替换预定义 Values在模板中用 .Values 可以获取到 values.yaml 文件（或者 --set 参数）提供的 values 值，此外，还可以在模板中访问其他预定义的数据。预定义可用于每个模板、并且不能被覆盖的 values 值，与所有 values 值一样，名称都是区分大小写的，而预定义的都是大写开头的，如下的values都可以在模板中获取到Release.Name：release 的名称（不是 chart），通过helm ls查看到的Release.Namespace：release 被安装到的命名空间Release.Service：渲染当前模板的服务，在 Helm 上，实际上该值始终为 HelmRelease.IsUpgrade：如果当前操作是升级或回滚，则该值为 trueRelease.IsInstall：如果当前操作是安装，则该值为 trueChart：Chart.yaml 文件的内容，可以通过 Chart.Version 来获得 Chart 的版本，通过 Chart.Maintainers 可以获取维护者信息Files： 一个包含 chart 中所有非特殊文件的 map 对象，这不会给你访问模板的权限，但是会给你访问存在的其他文件的权限（除非使用 .helmignore 排除它们），可以使用 {{ index .Files "file.name" }} 或者 {{ .Files.Get name }} 或者 {{ .Files.GetString name }} 函数来访问文件，你还可以使用 {{ .Files.GetBytes }} 以 []byte 的形式获取访问文件的内容Capabilities：也是一个类 map 的对象，其中包含有关 Kubernetes 版本（{{ .Capabilities.KubeVersion }}）和支持的 Kubernetes API 版本（{{ .Capabilities.APIVersions.Has "batch/v1" }}）信息。常被用来判断版本信息等注意任何未知的 Chart.yaml 字段都会被删除，在 Chart 对象内部无法访问他们，所以，Chart.yaml 不能用于将任意结构化的数据传递到模板中，但是可以使用 values 文件来传递。模板渲染当我们有一些模板是想渲染而不是执行到kubernetes的时候，就可以使用templatehelm template mysql stable/mysqlhelm install只是将上面的渲染直接安装了而已除次之外，也可以使用--dry-run --debug来查看整个渲染和执行的过程helm install --dry-run --debug mysql123 stable/mysql当然，他们都不会真的运行CRDhelm3中， 当定义后，CRD被使用之前都会先安装CRD目录下所有的CRD，而这些CRD不能使用模板。而一旦CRD被使用，就会等到CRD安装成功，否则是不会渲染模板并安装的因为CRD是全局安装的，所以在卸载的时候需要手动去卸载，并且CRD只有在安装操作的时候才会被创建，如果helm中的CRDS已经存在，且无论是什么版本，helm都不会重新安装或者升级。而一旦删除CRD，将会自动删除集群中所有namespace的CRD

linuxea:ingress-nginx basic auth认证的优雅实现 ingress-nginx basic auth认证是最简单和基础的，要使用它，需要安装httpd，提供一个htpasswd，而后使用htpasswd -c auth NAME 的方式创建一个文件，而后是以哦那个create来创建，如下kubectl create secret generic bauth --from-file=NAME这通常能解决使用，但是需要额外安装一个软件包，或许需要另外一个方式来解决通过htaccesstools.com 或者https://wtools.io/generate-htpasswd-online来生成加密的密钥信息，如下我通过https://wtools.io/generate-htpasswd-online生成用户名： linuxea 密码： OpSOQKs,qDJ1dSvzs生成generation如下linuxea:$apr1$btmgi74s$JEKIq8dTE3OI8o5a1qQvq0手动用base64加密[root@linuxea.com ~]# echo 'linuxea:$apr1$btmgi74s$JEKIq8dTE3OI8o5a1qQvq0' |base64 bGludXhlYTokYXByMSRidG1naTc0cyRKRUtJcThkVEUzT0k4bzVhMXFRdnEwCg==而后直接复制这串字符串添加到配置清单中apiVersion: v1 data: auth: bGludXhlYTokYXByMSRidG1naTc0cyRKRUtJcThkVEUzT0k4bzVhMXFRdnEwCg== kind: Secret metadata: name: basic-auth namespace: monitoring type: Opaque在应用到ingress-nginx的配置中即可apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: monitoring-ui namespace: monitoring annotations: nginx.ingress.kubernetes.io/auth-type: basic nginx.ingress.kubernetes.io/auth-secret: basic-auth nginx.ingress.kubernetes.io/auth-realm: "Authentication Required - input: Trump " spec: ingressClassName: nginx rules: - host: local.prom.com http: paths: - path: / pathType: Prefix backend: service: name: prometheus-k8s port: number: 9090配置完成打开这个域名参考ingress-nginx的rewrite与canaryingress-nginx应用常见的两种方式k8s下kube-prometheus监控ingress-nginx

linuxea:helm3的简单使用(1) 无论是debian还是redhat，亦或者其他linux发行版，都有一个包管理用来解决依赖问题，而在kubernetes中，helm是用来管理kubernetes应用程序，其中charts是可以定义一个可以进行安装升级的应用程序，同时也容易创建起来，并且进行版本管理。而在越复杂的应用程序来讲，helm可以作为一个开箱即用的，单单从使用角度来看，类似于yum或者apt的，使用起来，会更加流行。比如：我们创建一个应用程序，控制器使用deployment，同时需要一个service和关联其他的对象，并且还需配置一个ingress配置域名等作为入口，还可能需要部署一个有状态的，类似mysql的后端数据存储等等。这些如果需要一个个去安装维护将会麻烦很多，特别对于一个使用者来讲，更多时候，无需关注里面发生了什么，而更多的时候只想拿来即用的，helm就是用来打包这些程序。一个流行kubernetes生态的组件库中，你会发现必然会提供一个helm的方式。这正是因为helm的特色，得益于这种的易用性使得helm愈发的普及。作为一个charts提供的参数，对其中的内容进行渲染，从而生成yaml文件，安装到kubernetes中。helm就是解决这些事情的除此之外，我们还有一个kustomize也可以进行配置清单管理，kustomize解决的是另外一个问题，有机会在写这个kustomize。而helm2和helm3是有些不同的。安装helm是读取kubeconfig文件来访问集群的，因此，你至少能够使用kubectl访问集群才能使用helm在使用版本上v3版本比v2更好用一些，简化了集群内的一个服务换城了kubernetes CRD, 在v2中需要大量的权限控制，这样也会带来一个安全问题，而在v3中变成了一个客户端， 因此，我们使用v3稳定版本即可如果需要了解更多的概念，可以参考helm2的时候的一些文章对于helm2，可以查看如下kubernetes helm概述(49)kubernetes helm简单使用(50)kubernetes 了解chart(51)kubernetes helm安装efk(52)在helm的github下载对应系统的版本，比如：3.8.1的amd版本wget https://get.helm.sh/helm-v3.8.1-linux-amd64.tar.gz tar xf helm-v3.8.1-linux-amd64.tar.gz cp helm /usr/local/sbin/查看版本信息这里温馨的提示说我们的配置文件的权限太高# helm version WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /root/.kube/config WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /root/.kube/config version.BuildInfo{Version:"v3.8.1", GitCommit:"5cb9af4b1b271d11d7a97a71df3ac337dd94ad37", GitTreeState:"clean", GoVersion:"go1.17.5"}常用的命令- helm search: 搜索以恶个 charts - helm pull: 下载 chart - helm install: 安装到 Kubernetes - helm list: 查看 chartshelm installhelm install可以通过多个源进行安卓，大致如下chart仓库本地chart压缩包本地解开的压缩包的目录中的路径在线url总之要能够访问的到，首先通过在线安装1.添加chart仓库源我们需要安装一个chart源来使用，这类似于yum的源一样，我们使用azure的仓库helm repo add stable http://mirror.azure.cn/kubernetes/charts/ helm repo list[root@linuxea.com ~]# helm repo add stable "stable" has been added to your repositories [root@linuxea.com ~]# helm repo list NAME URL stable http://mirror.azure.cn/kubernetes/charts/我们可以使用 helm search repo stable查看当前的包于此同时，使用helm repo update更新到最新的状态[root@linuxea.com ~]# helm repo update WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /root/.kube/config WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /root/.kube/config Hang tight while we grab the latest from your chart repositories... ...Successfully got an update from the "stable" chart repository Update Complete. ⎈Happy Helming!⎈ [root@linuxea.com ~]# helm search repo stable WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /root/.kube/config WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /root/.kube/config NAME CHART VERSION APP VERSION DESCRIPTION stable/acs-engine-autoscaler 2.2.2 2.1.1 DEPRECATED Scales worker nodes within agent pools stable/aerospike 0.3.5 v4.5.0.5 DEPRECATED A Helm chart for Aerospike in Kubern... stable/airflow 7.13.3 1.10.12 DEPRECATED - please use: https://github.com/air... stable/ambassador 5.3.2 0.86.1 DEPRECATED A Helm chart for Datawire Ambassador stable/anchore-engine 1.7.0 0.7.3 Anchore container analysis and policy evaluatio... stable/apm-server 2.1.7 7.0.0 DEPRECATED The server receives data from the El... stable/ark 4.2.2 0.10.2 DEPRECATED A Helm chart for ark stable/artifactory 7.3.2 6.1.0 DEPRECATED Universal Repository Manager support... stable/artifactory-ha 0.4.2 6.2.0 DEPRECATED Universal Repository Manager support... stable/atlantis 3.12.4 v0.14.0 DEPRECATED A Helm chart for Atlantis https://ww... ......2.安装 chart安装一个mysql，在安装之前我们可以show一下 helm show chart stable/mysql查看它的版本号等信息更详细的信息可以通过helm show all stable/mysql ,all来查看[root@linuxea.com ~]# helm show chart stable/mysql WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /root/.kube/config WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /root/.kube/config apiVersion: v1 appVersion: 5.7.30 deprecated: true description: DEPRECATED - Fast, reliable, scalable, and easy to use open-source relational database system. home: https://www.mysql.com/ icon: https://www.mysql.com/common/logos/logo-mysql-170x115.png keywords: - mysql - database - sql name: mysql sources: - https://github.com/kubernetes/charts - https://github.com/docker-library/mysql version: 1.6.9安装--generate-name生成一个名称# helm install stable/mysql --generate-name WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /root/.kube/config WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /root/.kube/config WARNING: This chart is deprecated NAME: mysql-1649580933 LAST DEPLOYED: Sun Apr 10 04:55:35 2022 NAMESPACE: default STATUS: deployed REVISION: 1 NOTES: MySQL can be accessed via port 3306 on the following DNS name from within your cluster: mysql-1649580933.default.svc.cluster.local To get your root password run: MYSQL_ROOT_PASSWORD=$(kubectl get secret --namespace default mysql-1649580933 -o jsonpath="{.data.mysql-root-password}" | base64 --decode; echo) To connect to your database: 1. Run an Ubuntu pod that you can use as a client: kubectl run -i --tty ubuntu --image=ubuntu:16.04 --restart=Never -- bash -il 2. Install the mysql client: $ apt-get update && apt-get install mysql-client -y 3. Connect using the mysql cli, then provide your password: $ mysql -h mysql-1649580933 -p To connect to your database directly from outside the K8s cluster: MYSQL_HOST=127.0.0.1 MYSQL_PORT=3306 # Execute the following command to route the connection: kubectl port-forward svc/mysql-1649580933 3306 mysql -h ${MYSQL_HOST} -P${MYSQL_PORT} -u root -p${MYSQL_ROOT_PASSWORD}而后我们可以观察到pod的状态[root@linuxea.com ~]# kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES ... mysql-1649580933-8466b76578-gphkp 0/1 Pending 0 106s <none> <none> <none> <none> ...和svc[root@linuxea.com ~]# kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE ... mysql-1649580933 ClusterIP 10.68.106.229 <none> 3306/TCP 2m23s ...以及一个pvc[root@linuxea.com ~]# kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE mysql-1649580933 Pending 4m33s一旦安装完成可以通过ls查看她的版本[root@linuxea.com ~]# helm ls WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /root/.kube/config WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /root/.kube/config NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION mysql-1649580933 default 1 2022-04-10 04:55:35.427415297 -0400 EDT deployed mysql-1.6.9 5.7.30 当我们能看到这个name的时候，就可以使用uninstall删除uninstalll 会删除这个包下的所有相关的这个包的资源。同时，可以使用--keep-history参数保留release的记录而使用了--keep-history的时候就可以使用helm ls -a查看被卸载掉的记录[root@linuxea.com ~]# helm uninstall mysql-1649580933 WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /root/.kube/config WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /root/.kube/config release "mysql-1649580933" uninstalled参数配置如果直接install是默认的配置，但是更多时候，我们需要调整一下配置的参数，比如类似于端口等其他的选项参数，当然，这些参数必须是可以配置的的，一旦配置后，就会覆盖掉默认的值，通过helm show values来查看这些参数[root@linuxea.com ~]# helm show values stable/mysql比如配置密码，初始化，参数，调度，容忍，是否持久化等等。既然，我们要修改这些参数，那就需要一个覆盖的文件来进行操作，于是，创建一个文件，比如mvalule.yaml，在文件中配置想要修改的值, 如下指定用户和密码，并创建一个linuea的库，并且不进行数据持久化mysqlUser: linuxea mysqlPassword: linuxea.com mysqlDatabase: linuxea persistence: enabled: false而后只需要指定这个配置文件即可当你不使用 --generate-name的时候，只需要指定名称即可helm install mysql -f mvalule.yaml stable/mysql[root@linuxea.com /data/helm]# helm install -f mvalule.yaml stable/mysql --generate-name WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /root/.kube/config WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /root/.kube/config WARNING: This chart is deprecated NAME: mysql-1649582722 LAST DEPLOYED: Sun Apr 10 05:25:23 2022 NAMESPACE: default STATUS: deployed REVISION: 1 NOTES: MySQL can be accessed via port 3306 on the following DNS name from within your cluster: mysql-1649582722.default.svc.cluster.local To get your root password run: MYSQL_ROOT_PASSWORD=$(kubectl get secret --namespace default mysql-1649582722 -o jsonpath="{.data.mysql-root-password}" | base64 --decode; echo) To connect to your database: 1. Run an Ubuntu pod that you can use as a client: kubectl run -i --tty ubuntu --image=ubuntu:16.04 --restart=Never -- bash -il .....此时，可以通过kubectl describe 来查看传入的变量[root@linuxea.com ~]# kubectl describe pod mysql-1649582722-dbcdcb895-tjvsr Name: mysql-1649582722-dbcdcb895-tjvsr Namespace: default .... Environment: MYSQL_ROOT_PASSWORD: <set to the key 'mysql-root-password' in secret 'mysql-1649582722'> Optional: false MYSQL_PASSWORD: <set to the key 'mysql-password' in secret 'mysql-1649582722'> Optional: false MYSQL_USER: linuxea MYSQL_DATABASE: linuxea ...pod启动完成，我们通过上面的提示进入到mysql[root@linuxea.com /data/helm]# MYSQL_ROOT_PASSWORD=$(kubectl get secret --namespace default mysql-1649582722 -o jsonpath="{.data.mysql-root-password}" | base64 --decode; echo) [root@linuxea.com /data/helm]# echo $MYSQL_ROOT_PASSWORD 8FFSmw66je [root@linuxea.com /data/helm]# kubectl run -i --tty ubuntu --image=ubuntu:16.04 --restart=Never -- bash -il If you don't see a command prompt, try pressing enter. root@ubuntu:/# root@ubuntu:/# apt-get update && apt-get install mysql-client -y ... Setting up mysql-client-5.7 (5.7.33-0ubuntu0.16.04.1) ... Setting up mysql-client (5.7.33-0ubuntu0.16.04.1) ... Processing triggers for libc-bin (2.23-0ubuntu11.3) ... ... root@ubuntu:/# mysql -h mysql-1649582722 -p Enter password: Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 12 Server version: 5.7.30 MySQL Community Server (GPL) Copyright (c) 2000, 2021, Oracle and/or its affiliates. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. mysql> show databases; +--------------------+ | Database | +--------------------+ | information_schema | | linuxea | | mysql | | performance_schema | | sys | +--------------------+ 5 rows in set (0.00 sec) mysql> 环境变量基本上使用两种方式来传递配置信息：value那么除了使用value 或者-f指定yaml文件来覆盖values的值外，还可以指定多个值set直接在命令行指定需要覆盖的配置，但是对于深度嵌套的不建议使用--set通常--set优先于-f，-f将值持久化在configmap中如果我们使用--value配置文件已经配置了enabled: false,同时有配置了--set persistence.enabled: true, 而此时的enabled是等于true的，--set优先于--value对于value可以通过get value查看[root@linuxea.com /data/helm]# helm get values mysql-1649582722 WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /root/.kube/config WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /root/.kube/config USER-SUPPLIED VALUES: mysqlDatabase: linuxea mysqlPassword: linuxea.com mysqlUser: linuxea persistence: enabled: false对于一个已经运行的chart而言，使用helm upgrade来更新，或者使用--reset来删除--set--set可以接受0个或者多个键值对，最直接的常用的修改镜像，就是--set image:11，而对于多个，使用逗号隔开即可--set name:linuxea,image:11，如果在yaml文件中就换行，如下name: linuxea image: 11如果参数配置中所示，假如我们要修改的是mysql的参数[root@linuxea.com /data/helm]# cat mvalule.yaml mysqlUser: linuxea mysqlPassword: linuxea.com mysqlDatabase: linuxea persistence: enabled: false两种方式sethelm install mysql -f mvalule.yaml stable/mysql --set mysqlUser:linuxea,mysqlPassword:linuxea.com,mysqlDatabase:linuxea,persistence.enabled:false对于有换行的空格，使用.来拼接, persistence.enabled:false对应如下persistence: enabled: false其他1,如果有更多的参数，比如：--set args={run,/bin/start,--devel}args: - run - /bin/start - --devel2,除此之外，我们可以借用索引的方式，如下metadata: name: etcd-k8s namespace: monitoring这样的话，就变成了metadata[0].name=etcd-k8s,metadata[0].namespace=monitoring3,对于特殊字符可以使用反斜杠和双引号来做name: "a,b"这样的set明天就变成了--set name=a\,b4,其他包含反斜杠的nodeSelector: kubernetes.io/role: master这时候的--set就需要转义：--set nodeSelector."kubernetes\.io/role"=master本地安装通过fetch可以将chart放到本地[root@linuxea.com /data/helm]# helm fetch stable/mysql WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /root/.kube/config WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /root/.kube/config [root@linuxea.com /data/helm]# ls mysql-1.6.9.tgz -ll -rw-r--r-- 1 root root 11589 Apr 10 06:14 mysql-1.6.9.tgz而后就可以直接使用helm安装[root@linuxea.com /data/helm]# helm install mysql mysql-1.6.9.tgz 或者解压[root@linuxea.com /data/helm]# tar xf mysql-1.6.9.tgz [root@linuxea.com /data/helm]# ls mysql Chart.yaml README.md templates values.yaml [root@linuxea.com /data/helm]# tree mysql mysql ├── Chart.yaml ├── README.md ├── templates │   ├── configurationFiles-configmap.yaml │   ├── deployment.yaml │   ├── _helpers.tpl │   ├── initializationFiles-configmap.yaml │   ├── NOTES.txt │   ├── pvc.yaml │   ├── secrets.yaml │   ├── serviceaccount.yaml │   ├── servicemonitor.yaml │   ├── svc.yaml │   └── tests │   ├── test-configmap.yaml │   └── test.yaml └── values.yaml 2 directories, 15 files安装[root@linuxea.com /data/helm]# helm install mysql ./mysql升级与回滚helm的upgrade命令会更新你提供的信息，并且只会更新上一个版本，这种较小的更新更快捷每，进行一次upgrade都会生成新的配置版本，比如secret，默认似乎有15个版本，这将会是一个问题。添加一个mysqlRootPassword: www.linuxea.com 进行upgrademysqlUser: linuxea mysqlPassword: linuxea.com mysqlDatabase: linuxea mysqlRootPassword: www.linuxea.com persistence: enabled: falseupgradehelm upgrade mysql-1649582722 stable/mysql -f mvalule.yaml如下[root@linuxea.com /data/helm]# helm upgrade mysql-1649582722 stable/mysql -f mvalule.yaml WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /root/.kube/config WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /root/.kube/config WARNING: This chart is deprecated Release "mysql-1649582722" has been upgraded. Happy Helming! NAME: mysql-1649582722 LAST DEPLOYED: Sun Apr 10 06:29:00 2022 NAMESPACE: default STATUS: deployed REVISION: 2 NOTES: MySQL can be accessed via port 3306 on the following DNS name from within your cluster: mysql-1649582722.default.svc.cluster.local ...更新完成后REVISION已经变成2了通过helm ls查看[root@linuxea.com /data/helm]# helm ls WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /root/.kube/config WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /root/.kube/config NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION mysql-1649582722 default 2 2022-04-10 06:29:00.717252842 -0400 EDT deployed mysql-1.6.9 5.7.30 而后可以通过helm get values mysql-1649582722 查看[root@linuxea.com /data/helm]# helm get values mysql-1649582722 WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /root/.kube/config WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /root/.kube/config USER-SUPPLIED VALUES: mysqlDatabase: linuxea mysqlPassword: linuxea.com mysqlRootPassword: www.linuxea.com mysqlUser: linuxea persistence: enabled: false此时的mysql的新密码已经更新到secret，但是并没有在mysql生效的 ，我们就进行回滚下[root@linuxea.com /data/helm]# kubectl get secret --namespace default mysql-1649582722 -o jsonpath="{.data.mysql-root-password}" | base64 --decode; echo www.linuxea.comrollbackls查看helm的名称[root@linuxea.com /data/helm]# helm ls WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /root/.kube/config WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /root/.kube/config NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION mysql-1649582722 default 2 2022-04-10 06:29:00.717252842 -0400 EDT deployed mysql-1.6.9 5.7.30 查看mysql-1649582722历史版本[root@linuxea.com /data/helm]# helm history mysql-1649582722 WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /root/.kube/config WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /root/.kube/config REVISION UPDATED STATUS CHART APP VERSION DESCRIPTION 1 Sun Apr 10 05:25:23 2022 superseded mysql-1.6.9 5.7.30 Install complete 2 Sun Apr 10 06:29:00 2022 deployed mysql-1.6.9 5.7.30 Upgrade complete进行rollback[root@linuxea.com /data/helm]# helm rollback mysql-1649582722 1 WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /root/.kube/config WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /root/.kube/config Rollback was a success! Happy Helming!在来查看values[root@linuxea.com /data/helm]# helm get values mysql-1649582722 WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /root/.kube/config WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /root/.kube/config USER-SUPPLIED VALUES: mysqlDatabase: linuxea mysqlPassword: linuxea.com mysqlUser: linuxea persistence: enabled: false在查看密码[root@linuxea.com /data/helm]# kubectl get secret --namespace default mysql-1649582722 -o jsonpath="{.data.mysql-root-password}" | base64 --decode; echo 8FFSmw66je而现在的history就是三个版本了[root@linuxea.com /data/helm]# helm history mysql-1649582722 WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /root/.kube/config WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /root/.kube/config REVISION UPDATED STATUS CHART APP VERSION DESCRIPTION 1 Sun Apr 10 05:25:23 2022 superseded mysql-1.6.9 5.7.30 Install complete 2 Sun Apr 10 06:29:00 2022 superseded mysql-1.6.9 5.7.30 Upgrade complete 3 Sun Apr 10 06:39:44 2022 deployed mysql-1.6.9 5.7.30 Rollback to 1 这是因为版本是一直在新增，而3的版本就是回滚到1了，Rollback to 1 其他参数在整个helm中有一些非常有意思且重要的参数，比如常见的install和upgrade，当我们不确定一个程序是否被安装的时候，我们就需要安装，否则就是更新，于是可以使用upgrade --install，一般而言，我们可能还需要一个名称空间，那么就有了另外一个参数--create-namesapce，如果不存在就创建helm upgrade --install --create-namespace --namespace linuxea hmysql ./mysql如果名称空间不存在就创建，如果mysql没有install就install，否则就upgrade同时，当helm执行完成后, list列表中的状态已经为deployed，但是并不能说明pod已经装备好了，这两者之间并没有直接关系的，此时需要一些配置参数辅助--wait等待所有pod就绪，包含共享存储的pvc，就绪状态准备情况，以及svc，如果超过五分钟，这个版本就会标记失败-- timeout等待kubernetes命令完成，默认五分钟--no-hooks跳过命令的运行的hooks--recreate-pods仅适用于upgrade和rollback，在helm3中这个标志将导致重新创建所有的pod参考kubernetes helm概述(49)kubernetes helm简单使用(50)kubernetes 了解chart(51)kubernetes helm安装efk(52)