linuxea:kubeadm 1.13 高可用 使用kubeadm安装配置kubernetes HA，etcd外放，使用VIP做故障转移，其中不同的是，这个VIP还做了域名解析。此前尝试使用keepalived+haproxy发现有一些问题。恰巧内部有内部的DNS服务器，这样一来，两台master通过域名和VIP做转移，实现了kubernetes的高可用，如下图环境如下：[root@linuxea.com ~]# kubectl version Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.1",[root@linuxea.com ~]# docker -v Docker version 18.06.1-ce, build e68fc7a先决条件hostscat >> /etc/hosts << EOF 172.25.50.13 master-0.k8s.org 172.25.50.14 master-1.k8s.org 127.0.0.1 www.linuxea.com EOFhostname[root@linuxea.com ~]# hostnamectl set-hostname master-0.k8s.org [root@host-172-25-50-13 ~]# echo "DHCP_HOSTNAME=master-0.k8s.org" >> /etc/sysconfig/network-scripts/ifcfg-eth0 [root@linuxea.com ~]# systemctl restart network修改后重启下，在重启前，关闭防火墙[root@linuxea.com ~]# systemctl disable iptables firewalld.service [root@linuxea.com ~]# systemctl stop iptables firewalld.service [root@linuxea.com ~]# reboot当然了，我这里此前安装的就是iptablesswap[root@master-0 ~]# swapoff -a可以打开ipvscat << EOF > /etc/sysconfig/modules/ipvs.modules #!/bin/bash ipvs_modules_dir="/usr/lib/modules/\`uname -r\`/kernel/net/netfilter/ipvs" for i in \`ls \$ipvs_modules_dir | sed -r 's#(.*).ko.*#\1#'\`; do /sbin/modinfo -F filename \$i &> /dev/null if [ \$? -eq 0 ]; then /sbin/modprobe \$i fi done EOFchmod +x /etc/sysconfig/modules/ipvs.modules bash /etc/sysconfig/modules/ipvs.modulesecho "1" > /proc/sys/net/bridge/bridge-nf-call-iptables确保模块安装,nf_nat_ipv4也是关键之一[root@master-0 ~]# lsmod|grep ip_vs ip_vs_wrr 16384 0 ip_vs_wlc 16384 0 ip_vs_sh 16384 0 ip_vs_sed 16384 0 ip_vs_rr 16384 0 ip_vs_pe_sip 16384 0 nf_conntrack_sip 28672 1 ip_vs_pe_sip ip_vs_ovf 16384 0 ip_vs_nq 16384 0 ip_vs_mh 16384 0 ip_vs_lc 16384 0 ip_vs_lblcr 16384 0 ip_vs_lblc 16384 0 ip_vs_ftp 16384 0 ip_vs_fo 16384 0 ip_vs_dh 16384 0 ip_vs 151552 30 ip_vs_wlc,ip_vs_rr,ip_vs_dh,ip_vs_lblcr,ip_vs_sh,ip_vs_ovf,ip_vs_fo,ip_vs_nq,ip_vs_lblc,ip_vs_pe_sip,ip_vs_wrr,ip_vs_lc,ip_vs_mh,ip_vs_sedip_vs_ftp nf_nat 32768 2 nf_nat_ipv4,ip_vs_ftp nf_conntrack 135168 8 xt_conntrack,nf_conntrack_ipv4,nf_nat,ipt_MASQUERADE,nf_nat_ipv4,nf_conntrack_sip,nf_conntrack_netlink,ip_vs libcrc32c 16384 4 nf_conntrack,nf_nat,xfs,ip_vs如果觉得上面的步骤太繁琐，可以参考这里的脚本：curl -Lk https://raw.githubusercontent.com/marksugar/kubeadMHA/master/systeminit/chenage_hostname|bash curl -Lk https://raw.githubusercontent.com/marksugar/kubeadMHA/master/systeminit/ip_vs_a_init|bashkeepalivedinstall keepalived bash <(curl -s https://raw.githubusercontent.com/marksugar/lvs/master/keepliaved/install.sh|more)如下：输入Master或者BACKUP和VIP[root@master-0 ~]# bash <(curl -s https://raw.githubusercontent.com/marksugar/lvs/master/keepliaved/install.sh|more) You install role MASTER/BACKUP ? please enter(block letter):MASTER Please enter the use VIP: 172.25.50.15安装kubeadmcat <<EOF > /etc/yum.repos.d/kubernetes.repo [kubernetes] name=Kubernetes baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64 enabled=1 gpgcheck=1 repo_gpgcheck=1 gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg EOF setenforce 0 yum install -y kubelet kubeadm systemctl enable kubelet && systemctl start kubeletmaster-0部署kubeadm initmaster-0.k8s.orgapiVersion: kubeadm.k8s.io/v1beta1 kind: InitConfiguration nodeRegistration: name: master-0.k8s.org # taints: # - key: "kubeadmNode" # value: "master" # effect: "NoSchedule" localapiEndpoint: advertiseAddress: https://172.25.50.13 bindPort: 6443 --- apiVersion: kubeadm.k8s.io/v1beta1 kind: ClusterConfiguration kubernetesVersion: "v1.13.1" #controlPlaneEndpoint: 172.25.50.15:6444 controlPlaneEndpoint: master-vip.k8s.org:6443 apiServer: CertSANs: - master-vip.k8s.org timeoutForControlPlane: 5m0s etcd: external: endpoints: - "https://172.25.50.16:2379" - "https://172.25.50.17:2379" - "https://172.25.50.18:2379" caFile: /etc/kubernetes/pki/etcd/ca.pem certFile: /etc/kubernetes/pki/etcd/client.pem keyFile: /etc/kubernetes/pki/etcd/client-key.pem networking: serviceSubnet: 172.25.50.0/23 podSubnet: 172.25.56.0/22 dnsDomain: cluster.local imageRepository: k8s.gcr.io clusterName: "Acluster" #dns: # type: CoreDNS --- apiVersion: kubeproxy.config.k8s.io/v1alpha1 kind: KubeProxyConfiguration mode: "ipvs"开始初始化[root@master-0 ~]# kubeadm init --config ./kubeadm-init.yaml ... Your Kubernetes master has initialized successfully! To start using your cluster, you need to run the following as a regular user: mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config You should now deploy a pod network to the cluster. Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: https://kubernetes.io/docs/concepts/cluster-administration/addons/ You can now join any number of machines by running the following on each node as root: kubeadm join master-vip.k8s.org:6443 --token gjflgc.jg9i5vyrmiv295h3 --discovery-token-ca-cert-hash sha256:9b7943a35e4b6199b5f9fe50473bd336e28c184975d90e3a0f3076c25b694a18[root@master-0 ~]# mkdir -p $HOME/.kube [root@master-0 ~]# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config [root@master-0 ~]# sudo chown $(id -u):$(id -g) $HOME/.kube/config[root@master-0 ~]# kubectl get cs,nodes NAME STATUS MESSAGE ERROR componentstatus/scheduler Healthy ok componentstatus/controller-manager Healthy ok componentstatus/etcd-0 Healthy {"health":"true"} componentstatus/etcd-1 Healthy {"health":"true"} componentstatus/etcd-2 Healthy {"health":"true"} NAME STATUS ROLES AGE VERSION node/master-0.k8s.org NotReady master 75s v1.13.1[root@master-0 ~]# kubectl get pods -n kube-system NAME READY STATUS RESTARTS AGE coredns-86c58d9df4-br2x2 0/1 Pending 0 71s coredns-86c58d9df4-kcm42 0/1 Pending 0 71s kube-apiserver-master-0.k8s.org 1/1 Running 0 28s kube-controller-manager-master-0.k8s.org 1/1 Running 0 29s kube-proxy-rp8dg 1/1 Running 0 71s kube-scheduler-master-0.k8s.org 1/1 Running 0 31s安装calico[root@master-0 ~]# wget https://docs.projectcalico.org/v3.4/getting-started/kubernetes/installation/hosted/calico.yaml[root@master-0 ~]# cd calico/ [root@master-0 calico]# ls calicoctl calico.yamlapply[root@master-0 calico]# kubectl apply -f ./ configmap/calico-config created secret/calico-etcd-secrets created daemonset.extensions/calico-node created serviceaccount/calico-node created deployment.extensions/calico-kube-controllers created serviceaccount/calico-kube-controllers created clusterrole.rbac.authorization.k8s.io/calico-kube-controllers created clusterrolebinding.rbac.authorization.k8s.io/calico-kube-controllers created clusterrole.rbac.authorization.k8s.io/calico-node created clusterrolebinding.rbac.authorization.k8s.io/calico-node created[root@master-0 calico]# kubectl get pods -n kube-system NAME READY STATUS RESTARTS AGE calico-kube-controllers-5d94b577bb-7k7jn 1/1 Running 0 27s calico-node-bdkgn 1/1 Running 0 27s coredns-86c58d9df4-br2x2 1/1 Running 0 3m8s coredns-86c58d9df4-kcm42 1/1 Running 0 3m8s kube-apiserver-master-0.k8s.org 1/1 Running 0 2m25s kube-controller-manager-master-0.k8s.org 1/1 Running 0 2m26s kube-proxy-rp8dg 1/1 Running 0 3m8s kube-scheduler-master-0.k8s.org 1/1 Running 0 2m28s你可能需要修改两个地方，一个网卡接口- name: IP value: "autodetect" - name: IP_AUTODETECTION_METHOD value: "interface=eth.*" 或者，直接写成ip- name: CALICO_IPV4POOL_CIDR value: "172.25.56.0/22"另外，如果有必要，你还要修改容忍度来容忍master的污点tolerations: # Mark the pod as a critical add-on for rescheduling. - key: CriticalAddonsOnly operator: Exists - key: node-role.kubernetes.io/master effect: NoSchedule可以参考我的github上的文件：https://github.com/marksugar/kubeadMHA/tree/master/calico延伸阅读：https://docs.projectcalico.org/v3.4/getting-started/kubernetes/installation/calico安装metrics-server我们创建 一个目录来存放metrics-server[root@master-0 ~]# mkdir deploy/metrics-server/ [root@master-0 ~]# cd ~/deploy/metrics-server/而后我们只下载metrics-server的部署文件即可[root@master-0 metrics-server]# for i in \ aggregated-metrics-reader.yaml \ auth-delegator.yaml \ auth-reader.yaml \ metrics-apiservice.yaml \ metrics-server-deployment.yaml \ metrics-server-service.yaml \ resource-reader.yaml \ ;do curl -Lks https://raw.githubusercontent.com/kubernetes-incubator/metrics-server/master/deploy/1.8%2B/$i -o "${i}";done[root@master-0 metrics-server]# ll total 28 -rw-r--r-- 1 root root 384 Jan 1 21:37 aggregated-metrics-reader.yaml -rw-r--r-- 1 root root 308 Jan 1 21:37 auth-delegator.yaml -rw-r--r-- 1 root root 329 Jan 1 21:37 auth-reader.yaml -rw-r--r-- 1 root root 298 Jan 1 21:37 metrics-apiservice.yaml -rw-r--r-- 1 root root 815 Jan 1 21:37 metrics-server-deployment.yaml -rw-r--r-- 1 root root 249 Jan 1 21:37 metrics-server-service.yaml -rw-r--r-- 1 root root 502 Jan 1 21:37 resource-reader.yaml而后部署即可[root@master-0 metrics-server]# kubectl apply -f ./ clusterrole.rbac.authorization.k8s.io/system:aggregated-metrics-reader created clusterrolebinding.rbac.authorization.k8s.io/metrics-server:system:auth-delegator created rolebinding.rbac.authorization.k8s.io/metrics-server-auth-reader created apiservice.apiregistration.k8s.io/v1beta1.metrics.k8s.io created serviceaccount/metrics-server created deployment.extensions/metrics-server created service/metrics-server created clusterrole.rbac.authorization.k8s.io/system:metrics-server created clusterrolebinding.rbac.authorization.k8s.io/system:metrics-server created[root@master-0 metrics-server]# kubectl get pods -n kube-system NAME READY STATUS RESTARTS AGE calico-kube-controllers-5d94b577bb-7k7jn 1/1 Running 0 64s calico-node-bdkgn 1/1 Running 0 64s coredns-86c58d9df4-br2x2 1/1 Running 0 3m45s coredns-86c58d9df4-kcm42 1/1 Running 0 3m45s kube-apiserver-master-0.k8s.org 1/1 Running 0 3m2s kube-controller-manager-master-0.k8s.org 1/1 Running 0 3m3s kube-proxy-rp8dg 1/1 Running 0 3m45s kube-scheduler-master-0.k8s.org 1/1 Running 0 3m5s metrics-server-54f6f996dc-kr5wz 1/1 Running 0 7s稍等片刻就可以看到节点资源状态等[root@master-0 metrics-server]# kubectl top node NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% master-0.k8s.org 292m 14% 1272Mi 35% [root@master-0 metrics-server]# kubectl top pods -n kube-system NAME CPU(cores) MEMORY(bytes) calico-kube-controllers-5d94b577bb-7k7jn 2m 10Mi calico-node-bdkgn 20m 19Mi coredns-86c58d9df4-br2x2 2m 11Mi coredns-86c58d9df4-kcm42 3m 11Mi kube-apiserver-master-0.k8s.org 79m 391Mi kube-controller-manager-master-0.k8s.org 33m 67Mi kube-proxy-rp8dg 2m 15Mi kube-scheduler-master-0.k8s.org 10m 13Mi metrics-server-54f6f996dc-kr5wz 1m 11Mi master-1部署复制密钥到master-1[root@master-0 metrics-server]# cd /etc/kubernetes/ && scp -r ./pki 172.25.50.14:/etc/kubernetes/ && scp ./admin.conf 172.25.50.14:/etc/kubernetes/ ca.pem 100% 1371 101.9KB/s 00:00 client.pem 100% 997 430.3KB/s 00:00 client-key.pem 100% 227 132.2KB/s 00:00 ca.key 100% 1675 366.5KB/s 00:00 ca.crt 100% 1025 1.5MB/s 00:00 apiserver-kubelet-client.key 100% 1679 817.6KB/s 00:00 apiserver-kubelet-client.crt 100% 1099 602.7KB/s 00:00 apiserver.key 100% 1679 170.8KB/s 00:00 apiserver.crt 100% 1261 266.2KB/s 00:00 front-proxy-ca.key 100% 1675 796.2KB/s 00:00 front-proxy-ca.crt 100% 1038 595.1KB/s 00:00 front-proxy-client.key 100% 1679 816.4KB/s 00:00 front-proxy-client.crt 100% 1058 512.4KB/s 00:00 sa.key 100% 1679 890.5KB/s 00:00 sa.pub 100% 451 235.0KB/s 00:00 admin.conf 100% 5450 442.9KB/s 00:00使用kubeadm token create --print-join-command获取当前token[root@master-0 kubernetes]# kubeadm token create --print-join-command kubeadm join master-vip.k8s.org:6443 --token qffwr6.4dqd3hshvfbxn3f8 --discovery-token-ca-cert-hash sha256:9b7943a35e4b6199b5f9fe50473bd336e28c184975d90e3a0f3076c25b694a18[root@master-1 ~]# echo "1" > /proc/sys/net/bridge/bridge-nf-call-iptables && bash /etc/sysconfig/modules/ipvs.modules[root@master-1 ~]# kubeadm join master-vip.k8s.org:6443 --token qffwr6.4dqd3hshvfbxn3f8 --discovery-token-ca-cert-hash sha256:9b7943a35e4b6199b5f9fe50473bd336e28c184975d90e3a0f3076c25b694a18 --experimental-control-plane ... This node has joined the cluster and a new control plane instance was created: * Certificate signing request was sent to apiserver and approval was received. * The Kubelet was informed of the new secure connection details. * Master label and taint were applied to the new node. * The Kubernetes control plane instances scaled up. To start administering your cluster from this node, you need to run the following as a regular user: mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config Run 'kubectl get nodes' to see this node join the cluster. ...我们在备用的master节点使用experimental-control-plane延伸阅读：https://kubernetes.io/docs/setup/independent/high-availability/#external-etcd[root@master-1 ~]# mkdir -p $HOME/.kube [root@master-1 ~]# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config [root@master-1 ~]# sudo chown $(id -u):$(id -g) $HOME/.kube/config[root@master-1 ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION master-0.k8s.org Ready master 11m v1.13.1 master-1.k8s.org Ready master 70s v1.13.1 [root@master-1 ~]# kubectl get pods -n kube-system NAME READY STATUS RESTARTS AGE calico-kube-controllers-5d94b577bb-7k7jn 1/1 Running 0 8m13s calico-node-bdkgn 1/1 Running 0 8m13s calico-node-nk6xw 1/1 Running 0 79s coredns-86c58d9df4-br2x2 1/1 Running 0 10m coredns-86c58d9df4-kcm42 1/1 Running 0 10m kube-apiserver-master-0.k8s.org 1/1 Running 0 10m kube-apiserver-master-1.k8s.org 1/1 Running 0 79s kube-controller-manager-master-0.k8s.org 1/1 Running 0 10m kube-controller-manager-master-1.k8s.org 1/1 Running 0 79s kube-proxy-cz8h8 1/1 Running 0 79s kube-proxy-rp8dg 1/1 Running 0 10m kube-scheduler-master-0.k8s.org 1/1 Running 0 10m kube-scheduler-master-1.k8s.org 1/1 Running 0 79s metrics-server-54f6f996dc-kr5wz 1/1 Running 0 7m16s[root@master-1 ~]# kubectl top nodes NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% master-0.k8s.org 226m 11% 1314Mi 36% master-1.k8s.org 120m 6% 788Mi 21% 添加node添加node主机，做好修改主机名，开启ipvs等。这里写在一个文件里curl -Lk https://raw.githubusercontent.com/marksugar/kubeadMHA/master/systeminit/chenage_hostname|bash主机名你也可以这样修改echo $(ip addr show eth0 | grep -Po 'inet \K[\d.]+'|awk -F. '{print $0}') > /etc/hostname CHOSTNAME=node-$(echo `sed 's@\.@-@g' /etc/hostname`).k8s.org CHOSTNAME_pretty='k8s node' sysctl -w kernel.hostname=$CHOSTNAME hostnamectl set-hostname $CHOSTNAME --static hostnamectl set-hostname "$CHOSTNAME_pretty" --pretty sysctl kernel.hostname=$CHOSTNAME echo -e "\033[31m\033[01m[ `hostnamectl` ]\033[0m"主机名修改后，还需要在做一些操作curl -Lk https://raw.githubusercontent.com/marksugar/kubeadMHA/master/systeminit/ip_vs_a_init|bash这个脚本可以在github打开查看即可而后添加即可[root@node-172-25-50-19.k8s.org ~]# kubeadm join master-vip.k8s.org:6443 --token qffwr6.4dqd3hshvfbxn3f8 --discovery-token-ca-cert-hash sha256:9b7943a35e4b6199b5f9fe50473bd336e28c184975d90e3a0f3076c25b694a18 [preflight] Running pre-flight checks [discovery] Trying to connect to API Server "master-vip.k8s.org:6443" [discovery] Created cluster-info discovery client, requesting info from "https://master-vip.k8s.org:6443" [discovery] Requesting info from "https://master-vip.k8s.org:6443" again to validate TLS against the pinned public key [discovery] Cluster info signature and contents are valid and TLS certificate validates against pinned roots, will use API Server "master-vip.k8s.org:6443" [discovery] Successfully established connection with API Server "master-vip.k8s.org:6443" [join] Reading configuration from the cluster... [join] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml' [kubelet] Downloading configuration for the kubelet from the "kubelet-config-1.13" ConfigMap in the kube-system namespace [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" [kubelet-start] Activating the kubelet service [tlsbootstrap] Waiting for the kubelet to perform the TLS Bootstrap... [patchnode] Uploading the CRI Socket information "/var/run/dockershim.sock" to the Node API object "node-172-25-50-19.k8s.org" as an annotation This node has joined the cluster: * Certificate signing request was sent to apiserver and a response was received. * The Kubelet was informed of the new secure connection details. Run 'kubectl get nodes' on the master to see this node join the cluster.[root@master-0 ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION master-0.k8s.org Ready master 133m v1.13.1 master-1.k8s.org Ready master 123m v1.13.1 node-172-25-50-19.k8s.org Ready <none> 15s v1.13.1故障测试我们关掉keepalived，模拟master-0宕机[root@master-0 kubernetes]# systemctl stop keepalived.service 此时eth0上的172.25.10.15就会飘逸到master-1上，master-0如下[root@master-0 kubernetes]# ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether fa:16:3e:ed:d3:21 brd ff:ff:ff:ff:ff:ff inet 172.25.50.13/16 brd 172.25.255.255 scope global dynamic eth0 valid_lft 82951sec preferred_lft 82951secmaster-1如下：[root@master-1 ~]# ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether fa:16:3e:ae:76:b9 brd ff:ff:ff:ff:ff:ff inet 172.25.50.14/16 brd 172.25.255.255 scope global dynamic eth0 valid_lft 85332sec preferred_lft 85332sec inet 172.25.50.15/16 brd 172.25.255.255 scope global secondary eth0:vip valid_lft forever preferred_lft forever[root@master-1 ~]# kubectl get node NAME STATUS ROLES AGE VERSION master-0.k8s.org Ready master 15m v1.13.1 master-1.k8s.org Ready master 5m7s v1.13.1 [root@master-1 ~]# kubectl get pod -n kube-system NAME READY STATUS RESTARTS AGE calico-kube-controllers-5d94b577bb-7k7jn 1/1 Running 0 12m calico-node-bdkgn 1/1 Running 0 12m calico-node-nk6xw 1/1 Running 0 5m17s coredns-86c58d9df4-br2x2 1/1 Running 0 14m coredns-86c58d9df4-kcm42 1/1 Running 0 14m kube-apiserver-master-0.k8s.org 1/1 Running 0 14m kube-apiserver-master-1.k8s.org 1/1 Running 0 5m17s kube-controller-manager-master-0.k8s.org 1/1 Running 0 14m kube-controller-manager-master-1.k8s.org 1/1 Running 0 5m17s kube-proxy-cz8h8 1/1 Running 0 5m17s kube-proxy-rp8dg 1/1 Running 0 14m kube-scheduler-master-0.k8s.org 1/1 Running 0 14m kube-scheduler-master-1.k8s.org 1/1 Running 0 5m17s metrics-server-54f6f996dc-kr5wz 1/1 Running 0 11m添加Master我们使用的是keepalived跑的vip，测试发现使用VIP更妥当一些，DNS轮询不见得好用。于是乎将域名解析到这个VIP上，依靠VIP飘逸做HA关于 keepalived，可以参考上面的安装方式，非常简单，运行以下脚本即可bash <(curl -s https://raw.githubusercontent.com/marksugar/lvs/master/keepliaved/install.sh|more)如下：输入Master或者BACKUP和VIP[root@master-0 ~]# bash <(curl -s https://raw.githubusercontent.com/marksugar/lvs/master/keepliaved/install.sh|more) You install role MASTER/BACKUP ? please enter(block letter):MASTER Please enter the use VIP: 172.25.50.15这里需要注意的是，如果是三台keepalived，需要手动修改权重了。如果我们要添加一台Master只需要kubeadm token create --print-join-command获取到token，而后使用--experimental-control-plane，大致如下：kubeadm join master-vip.k8s.org:6443 --token qffwr6.4dqd3hshvfbxn3f8 --discovery-token-ca-cert-hash sha256:9b7943a35e4b6199b5f9fe50473bd336e28c184975d90e3a0f3076c25b694a18 --experimental-control-plane延伸阅读：https://kubernetes.io/docs/setup/independent/high-availability/#external-etcdhttps://godoc.org/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1alpha3https://godoc.org/k8s.io/kube-proxy/config/v1alpha1#KubeProxyIPVSConfigurationhttps://k8smeetup.github.io/docs/setup/independent/high-availability/如果你是需要频繁的安装测试，那下面的这些命令或许有 用：kubeadm reset \rm -rf /var/lib/cni/ \rm -rf /var/lib/kubelet/* \rm -rf /etc/cni/ ip link delete cni0 ip link delete flannel.1 ip link delete docker0 ip link delete dummy0 ip link delete kube-ipvs0 ip link delete tunl0@NONE ip link delete tunl0 ip addr add IP/32 brd IP dev eth0 ip addr del IP/32 brd IP dev tunl0@NONE echo "1" > /proc/sys/net/bridge/bridge-nf-call-iptables && bash /etc/sysconfig/modules/ipvs.modules

linuxea: 基于kubernetes的etcd 3.3.10外部集群 etcd是一个分布式键值存储，它提供了一种在一组机器上存储数据的可靠方法。它是开源的，可在GitHub上获得。etcd在网络分区期间优雅地处理leader选举，并且可以容忍机器故障，包括leader。应用程序可以将数据读写到etcd中。一个简单的用例是将etcd中的数据库连接详细信息或功能标记存储为键值对。可以监视这些值，允许您的应用在更改时重新配置。高级用法利用一致性保证来实现数据库leader选举或跨工作集群进行分布式锁定etcd是用Go编写的，它具有出色的跨平台支持，较小二进制文件和活跃的社区。etcd机器之间的通信通过Raft一致性算法处理。此处的ETCD主要用来部署kubernetes高可用集群，此后的使用都是基于kubernetes。参考kubernetes官网：https://kubernetes.io/docs/tasks/administer-cluster/configure-upgrade-etcd/etcd配置：https://github.com/etcd-io/etcd/blob/master/Documentation/op-guide/hardware.md#hardware-recommendations集群文档：https://github.com/etcd-io/etcd/blob/master/Documentation/op-guide/clustering.md示例参考：https://github.com/etcd-io/etcd/tree/master/hack/tls-setup参考：https://github.com/etcd-io/etcd/tree/master/hack/tls-setup/config参考：https://k8smeetup.github.io/docs/setup/independent/high-availability/我们首先配置etcd证书，etcd我们将会用在kubernetes上。这一步是必须的相比较github上etcd的示例，我们简单修改下在这之前 ，我们有必要修改一下主机名，并且配通所有的sshhostnamectl set-hostname etcd1 hostnamectl set-hostname etcd2 hostnamectl set-hostname etcd3cat >> /etc/hosts << EOF 172.25.50.16 etcd1 172.25.50.17 etcd2 172.25.50.18 etcd3 EOF[root@linuxea.com-16 /etc/etcda]# ssh-keygen -t rsa [root@linuxea.com-16 /etc/etcda]# for i in 172.25.50.{17,18};do ssh-copy-id $i; done安装cfssl和cfssljson[root@linuxea.com-16 ~]# curl -so /usr/local/bin/cfssl https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 [root@linuxea.com-16 ~]# curl -o /usr/local/bin/cfssljson https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 [root@linuxea.com-16 ~]# chmod +x /usr/local/bin/cfssl*生成证书[root@linuxea.com-16 ~]# mkdir -p /etc/kubernetes/pki/etcd [root@linuxea.com-16 ~]# cd /etc/kubernetes/pki/etcdca-config.jsoncat > cat /etc/kubernetes/pki/etcd/ca-config.json << EOF { "signing": { "default": { "expiry": "876000h" }, "profiles": { "server": { "expiry": "876000h", "usages": [ "signing", "key encipherment", "server auth", "client auth" ] }, "client": { "expiry": "876000h", "usages": [ "signing", "key encipherment", "client auth" ] }, "peer": { "expiry": "876000h", "usages": [ "signing", "key encipherment", "server auth", "client auth" ] } } } } EOFca-csr.jsoncat > ca-csr.json << EOL { "CN": "etcd", "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "Shanghai", "L": "Shanghai", "O": "etcd", "OU": "Etcd Security" } ] } EOL生成CA证书[root@linuxea.com-16 /etc/kubernetes/pki/etcd]# cfssl gencert -initca ca-csr.json | cfssljson -bare ca - 2018/12/25 16:29:14 [INFO] generating a new CA key and certificate from CSR 2018/12/25 16:29:14 [INFO] generate received request 2018/12/25 16:29:14 [INFO] received CSR 2018/12/25 16:29:14 [INFO] generating key: rsa-2048 2018/12/25 16:29:14 [INFO] encoded CSR 2018/12/25 16:29:14 [INFO] signed certificate with serial number 472142876620060394898834048533122419461412171471[root@linuxea.com-16 /etc/kubernetes/pki/etcd]# ll total 20 -rw-r--r-- 1 root root 905 Dec 25 16:28 ca-config.json -rw-r--r-- 1 root root 1005 Dec 25 16:29 ca.csr -rw-r--r-- 1 root root 212 Dec 25 16:29 ca-csr.json -rw------- 1 root root 1679 Dec 25 16:29 ca-key.pem -rw-r--r-- 1 root root 1371 Dec 25 16:29 ca.pem生成 etcd 客户端证书[root@linuxea.com-16 /etc/kubernetes/pki/etcd]# cat client.json { "CN": "client", "key": { "algo": "ecdsa", "size": 256 } }[root@linuxea.com-16 /etc/kubernetes/pki/etcd]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=client client.json | cfssljson -bare client 2018/12/25 16:29:56 [INFO] generate received request 2018/12/25 16:29:56 [INFO] received CSR 2018/12/25 16:29:56 [INFO] generating key: ecdsa-256 2018/12/25 16:29:56 [INFO] encoded CSR 2018/12/25 16:29:56 [INFO] signed certificate with serial number 644510971695673396838569226835778482472560755733 2018/12/25 16:29:56 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for websites. For more information see the Baseline Requirements for the Issuance and Management of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org); specifically, section 10.2.3 ("Information Requirements").如下[root@linuxea.com-16 /etc/kubernetes/pki/etcd]# ll total 36 -rw-r--r-- 1 root root 905 Dec 25 16:28 ca-config.json -rw-r--r-- 1 root root 1005 Dec 25 16:29 ca.csr -rw-r--r-- 1 root root 212 Dec 25 16:29 ca-csr.json -rw------- 1 root root 1679 Dec 25 16:29 ca-key.pem -rw-r--r-- 1 root root 1371 Dec 25 16:29 ca.pem -rw-r--r-- 1 root root 351 Dec 25 16:29 client.csr -rw-r--r-- 1 root root 95 Dec 25 16:29 client.json -rw------- 1 root root 227 Dec 25 16:29 client-key.pem -rw-r--r-- 1 root root 997 Dec 25 16:29 client.pemconfig.json对于config.json有两种方式,第一种,使用官网的，如下[root@linuxea.com-16 /etc/kubernetes/pki/etcd]# cfssl print-defaults csr > config.json[root@linuxea.com-16 /etc/kubernetes/pki/etcd]# sed -i '0,/CN/{s/example\.net/'"$PEER_NAME"'/}' config.json[root@linuxea.com-16 /etc/kubernetes/pki/etcd]# sed -i 's/www\.example\.net/'"$PRIVATE_IP"'/' config.json[root@linuxea.com-16 /etc/kubernetes/pki/etcd]# sed -i 's/example\.net/'"$PUBLIC_IP"'/' config.json[root@linuxea.com-16 /etc/kubernetes/pki/etcd]# cat config.json { "CN": "etcd1", "hosts": [ "", "172.25.50.16" ], "key": { "algo": "ecdsa", "size": 256 }, "names": [ { "C": "US", "L": "CA", "ST": "San Francisco" } ] }第二种方式，直接在这里编辑， 填写参与集群的ipcat > /etc/kubernetes/pki/etcd/config.json << EOF { "CN": "etcd1", "hosts": [ "127.0.0.1", "172.25.50.16", "172.25.50.17", "172.25.50.18" ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "Shanghai", "L": "Shanghai", "O": "etcd", "OU": "Etcd Security" } ] } EOF运行 cfssl 命令，将会生成peer.pem、peer-key.pem、server.pem、server-key.pem。[root@linuxea.com-16 /etc/kubernetes/pki/etcd]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=server config.json | cfssljson -bare server 2018/12/25 16:37:53 [INFO] generate received request 2018/12/25 16:37:53 [INFO] received CSR 2018/12/25 16:37:53 [INFO] generating key: rsa-2048 2018/12/25 16:37:54 [INFO] encoded CSR 2018/12/25 16:37:54 [INFO] signed certificate with serial number 397776469717117599117003178668354588092528739871 2018/12/25 16:37:54 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for websites. For more information see the Baseline Requirements for the Issuance and Management of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org); specifically, section 10.2.3 ("Information Requirements").[root@linuxea.com-16 /etc/kubernetes/pki/etcd]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=peer config.json | cfssljson -bare peer 2018/12/25 16:37:59 [INFO] generate received request 2018/12/25 16:37:59 [INFO] received CSR 2018/12/25 16:37:59 [INFO] generating key: rsa-2048 2018/12/25 16:37:59 [INFO] encoded CSR 2018/12/25 16:37:59 [INFO] signed certificate with serial number 453856739993256449551996181659627954567417235192 2018/12/25 16:37:59 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for websites. For more information see the Baseline Requirements for the Issuance and Management of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org); specifically, section 10.2.3 ("Information Requirements").如下[root@linuxea.com-16 /etc/kubernetes/pki/etcd]# ll total 64 -rw-r--r-- 1 root root 905 Dec 25 16:28 ca-config.json -rw-r--r-- 1 root root 1005 Dec 25 16:29 ca.csr -rw-r--r-- 1 root root 212 Dec 25 16:29 ca-csr.json -rw------- 1 root root 1679 Dec 25 16:29 ca-key.pem -rw-r--r-- 1 root root 1371 Dec 25 16:29 ca.pem -rw-r--r-- 1 root root 351 Dec 25 16:29 client.csr -rw-r--r-- 1 root root 95 Dec 25 16:29 client.json -rw------- 1 root root 227 Dec 25 16:29 client-key.pem -rw-r--r-- 1 root root 997 Dec 25 16:29 client.pem -rw-r--r-- 1 root root 375 Dec 25 16:37 config.json -rw-r--r-- 1 root root 1078 Dec 25 16:37 peer.csr -rw------- 1 root root 1679 Dec 25 16:37 peer-key.pem -rw-r--r-- 1 root root 1456 Dec 25 16:37 peer.pem -rw-r--r-- 1 root root 1078 Dec 25 16:37 server.csr -rw------- 1 root root 1679 Dec 25 16:37 server-key.pem -rw-r--r-- 1 root root 1456 Dec 25 16:37 server.pem证书传递将这些生成的证书复制到etcd2和etcd3上[root@linuxea.com-16 /etc/kubernetes/pki/etcd]# for i in 172.25.50.{17,18} ;do scp -r /etc/kubernetes $i:/etc ;done ca-config.json 100% 905 635.1KB/s 00:00 ca-csr.json 100% 212 174.5KB/s 00:00 ca.pem 100% 1371 1.0MB/s 00:00 ca-key.pem 100% 1679 1.3MB/s 00:00 ca.csr 100% 1005 773.7KB/s 00:00 client.json 100% 95 76.1KB/s 00:00 client.pem 100% 997 751.7KB/s 00:00 client-key.pem 100% 227 171.9KB/s 00:00 client.csr 100% 351 256.1KB/s 00:00 config.json 100% 375 96.5KB/s 00:00 server.pem 100% 1456 365.4KB/s 00:00 server-key.pem 100% 1679 425.8KB/s 00:00 server.csr 100% 1078 276.8KB/s 00:00 peer.pem 100% 1456 366.5KB/s 00:00 peer-key.pem 100% 1679 439.1KB/s 00:00 peer.csr 100% 1078 289.2KB/s 00:00 ca-config.json 100% 905 569.4KB/s 00:00 ca-csr.json 100% 212 134.9KB/s 00:00 ca.pem 100% 1371 944.5KB/s 00:00 ca-key.pem 100% 1679 1.1MB/s 00:00 ca.csr 100% 1005 605.7KB/s 00:00 client.json 100% 95 63.3KB/s 00:00 client.pem 100% 997 748.6KB/s 00:00 client-key.pem 100% 227 151.1KB/s 00:00 client.csr 100% 351 244.9KB/s 00:00 config.json 100% 375 90.4KB/s 00:00 server.pem 100% 1456 322.2KB/s 00:00 server-key.pem 100% 1679 372.1KB/s 00:00 server.csr 100% 1078 253.3KB/s 00:00 peer.pem 100% 1456 325.0KB/s 00:00 peer-key.pem 100% 1679 394.1KB/s 00:00 peer.csr 100% 1078 259.3KB/s 00:00 安装etcd这里的环境变量在三台参与集群的机器分别运行版本3.3.10export ETCD_VERSION=v3.3.10 curl -sSL https://github.com/coreos/etcd/releases/download/${ETCD_VERSION}/etcd-${ETCD_VERSION}-linux-amd64.tar.gz | tar -xzv --strip-components=1 -C /usr/local/bin/ rm -rf etcd-$ETCD_VERSION-linux-amd64*先决变量主机名和ip地址，这里的eth0应该与机器的网卡相符export PEER_NAME=$(hostname) export PRIVATE_IP=$(ip addr show eth0 | grep -Po 'inet \K[\d.]+')环境变量写入到/etc/etcd.envtouch /etc/etcd.env echo "PEER_NAME=$PEER_NAME" >> /etc/etcd.env echo "PRIVATE_IP=$PRIVATE_IP" >> /etc/etcd.env启动脚本即将用到的变量，这里标记的是参与etcd集群的三个ip地址export etcd0_ip_address=172.25.50.16 export etcd1_ip_address=172.25.50.17 export etcd2_ip_address=172.25.50.18启动脚本这里面的变量就是上面设置的cat > /etc/systemd/system/etcd.service << EOL [Unit] Description=etcd Documentation=https://github.com/coreos/etcd Conflicts=etcd.service Conflicts=etcd2.service [Service] EnvironmentFile=/etc/etcd.env Type=notify Restart=always RestartSec=5s LimitNOFILE=40000 TimeoutStartSec=0 ExecStart=/usr/local/bin/etcd --name ${PEER_NAME} \ --data-dir /var/lib/etcd \ --listen-client-urls https://${PRIVATE_IP}:2379 \ --advertise-client-urls https://${PRIVATE_IP}:2379 \ --listen-peer-urls https://${PRIVATE_IP}:2380 \ --initial-advertise-peer-urls https://${PRIVATE_IP}:2380 \ --cert-file=/etc/kubernetes/pki/etcd/server.pem \ --key-file=/etc/kubernetes/pki/etcd/server-key.pem \ --client-cert-auth \ --trusted-ca-file=/etc/kubernetes/pki/etcd/ca.pem \ --peer-cert-file=/etc/kubernetes/pki/etcd/peer.pem \ --peer-key-file=/etc/kubernetes/pki/etcd/peer-key.pem \ --peer-client-cert-auth \ --peer-trusted-ca-file=/etc/kubernetes/pki/etcd/ca.pem \ --initial-cluster etcd1=https://${etcd0_ip_address}:2380,etcd2=https://${etcd1_ip_address}:2380,etcd3=https://${etcd2_ip_address}:2380 \ --initial-cluster-state new [Install] WantedBy=multi-user.target EOLsystemctl daemon-reload systemctl enable etcd.service systemctl start etcd集群状态查看集群状态是需要证书的，我们配置一个环境变量CMD='--cacert=/etc/kubernetes/pki/etcd/ca.pem --cert=/etc/kubernetes/pki/etcd/server.pem --key=/etc/kubernetes/pki/etcd/server-key.pem' CMD1='https://172.25.50.16:2379,https://172.25.50.17:2379,https://172.25.50.18:2379' CMD2='--ca-file=/etc/kubernetes/pki/etcd/ca.pem --cert-file=/etc/kubernetes/pki/etcd/server.pem --key-file=/etc/kubernetes/pki/etcd/server-key.pem'[root@linuxea.com-16 /etc/kubernetes/pki/etcd]# for i in 172.25.50.{16,17,18}; do ETCDCTL_API=3 etcdctl --endpoints=https://${i}:2379 $CMD endpoint health; done https://172.25.50.16:2379 is healthy: successfully committed proposal: took = 1.984026ms https://172.25.50.17:2379 is healthy: successfully committed proposal: took = 3.357136ms https://172.25.50.18:2379 is healthy: successfully committed proposal: took = 3.55185ms[root@linuxea.com-16 /etc/kubernetes/pki/etcd]# ETCDCTL_API=3 etcdctl --endpoints=https://172.25.50.16:2379 $CMD member list 2e70a124f01a4a5, started, etcd3, https://172.25.50.18:2380, https://172.25.50.18:2379 5fba4c5d1e214899, started, etcd2, https://172.25.50.17:2380, https://172.25.50.17:2379 b55bca6849256d2d, started, etcd1, https://172.25.50.16:2380, https://172.25.50.16:2379 [root@linuxea.com-16 /etc/kubernetes/pki/etcd]# [root@linuxea.com-16 /etc/kubernetes/pki/etcd]# etcdctl -C $CMD1 $CMD2 cluster-health member 2e70a124f01a4a5 is healthy: got healthy result from https://172.25.50.18:2379 member 5fba4c5d1e214899 is healthy: got healthy result from https://172.25.50.17:2379 member b55bca6849256d2d is healthy: got healthy result from https://172.25.50.16:2379 cluster is healthy[root@linuxea.com-16 /etc/kubernetes/pki/etcd]# curl -Lk --cert ./server.pem --key ./server-key.pem https://172.25.50.16:2379/metrics|grep -v debugging延伸阅读：https://coreos.com/etcd/docs/latest/metrics.htmldocker安装如果安装此前的配置，docker的配置应该如下172.25.50.16docker run --net=host -d -v /etc/kubernetes/pki/etcd/:/etc/kubernetes/pki/etcd/ -p 4001:4001 -p 2380:2380 -p 2379:2379 \ --name etcd quay.io/coreos/etcd:v3.3.10 \ etcd -name etcd1 \ --data-dir /var/lib/etcd \ -advertise-client-urls https://172.25.50.16:2379,https://172.25.50.16:4001 \ -listen-client-urls https://0.0.0.0:2379,https://0.0.0.0:4001 \ -initial-advertise-peer-urls https://172.25.50.16:2380 \ -listen-peer-urls https://0.0.0.0:2380 \ --cert-file=/etc/kubernetes/pki/etcd/server.pem \ --key-file=/etc/kubernetes/pki/etcd/server-key.pem \ --client-cert-auth \ --trusted-ca-file=/etc/kubernetes/pki/etcd/ca.pem \ --peer-cert-file=/etc/kubernetes/pki/etcd/peer.pem \ --peer-key-file=/etc/kubernetes/pki/etcd/peer-key.pem \ --peer-client-cert-auth \ --peer-trusted-ca-file=/etc/kubernetes/pki/etcd/ca.pem \ -initial-cluster-token etcd-cluster \ -initial-cluster etcd1=https://172.25.50.16:2380,etcd2=https://172.25.50.17:2380,etcd3=https://172.25.50.18:2380 \ -initial-cluster-state new172.25.50.17docker run --net=host -d -v /etc/kubernetes/pki/etcd/:/etc/kubernetes/pki/etcd/ -p 4001:4001 -p 2380:2380 -p 2379:2379 \ --name etcd quay.io/coreos/etcd:v3.3.10 \ etcd -name etcd2 \ --data-dir /var/lib/etcd \ -advertise-client-urls https://172.25.50.17:2379,https://172.25.50.17:4001 \ -listen-client-urls https://0.0.0.0:2379,https://0.0.0.0:4001 \ -initial-advertise-peer-urls https://172.25.50.17:2380 \ -listen-peer-urls https://0.0.0.0:2380 \ --cert-file=/etc/kubernetes/pki/etcd/server.pem \ --key-file=/etc/kubernetes/pki/etcd/server-key.pem \ --client-cert-auth \ --trusted-ca-file=/etc/kubernetes/pki/etcd/ca.pem \ --peer-cert-file=/etc/kubernetes/pki/etcd/peer.pem \ --peer-key-file=/etc/kubernetes/pki/etcd/peer-key.pem \ --peer-client-cert-auth \ --peer-trusted-ca-file=/etc/kubernetes/pki/etcd/ca.pem \ -initial-cluster-token etcd-cluster \ -initial-cluster etcd1=https://172.25.50.16:2380,etcd2=https://172.25.50.17:2380,etcd3=https://172.25.50.18:2380 \ -initial-cluster-state new 172.25.50.18docker run --net=host -d -v /etc/kubernetes/pki/etcd/:/etc/kubernetes/pki/etcd/ -p 4001:4001 -p 2380:2380 -p 2379:2379 \ --name etcd quay.io/coreos/etcd:v3.3.10 \ etcd -name etcd3 \ --data-dir /var/lib/etcd \ -advertise-client-urls https://172.25.50.18:2379,https://172.25.50.18:4001 \ -listen-client-urls https://0.0.0.0:2379,https://0.0.0.0:4001 \ -initial-advertise-peer-urls https://172.25.50.18:2380 \ -listen-peer-urls https://0.0.0.0:2380 \ --cert-file=/etc/kubernetes/pki/etcd/server.pem \ --key-file=/etc/kubernetes/pki/etcd/server-key.pem \ --client-cert-auth \ --trusted-ca-file=/etc/kubernetes/pki/etcd/ca.pem \ --peer-cert-file=/etc/kubernetes/pki/etcd/peer.pem \ --peer-key-file=/etc/kubernetes/pki/etcd/peer-key.pem \ --peer-client-cert-auth \ --peer-trusted-ca-file=/etc/kubernetes/pki/etcd/ca.pem \ -initial-cluster-token etcd-cluster \ -initial-cluster etcd1=https://172.25.50.16:2380,etcd2=https://172.25.50.17:2380,etcd3=https://172.25.50.18:2380 \ -initial-cluster-state new 这样比较麻烦，我们简化一下先决条件：配置各个主机的hostname[root@DT_Node-172_25_50_16 /etc/kubernetes/pki/etcd]# hostnamectl set-hostname etcd1 [root@DT_Node-172_25_50_17 /etc/kubernetes/pki/etcd]# hostnamectl set-hostname etcd2 [root@DT_Node-172_25_50_18 /etc/kubernetes/pki/etcd]# hostnamectl set-hostname etcd3环境变量export PEER_NAME=$(hostname) export PRIVATE_IP=$(ip addr show eth0 | grep -Po 'inet \K[\d.]+') export etcd0_ip_address=172.25.50.16 export etcd1_ip_address=172.25.50.17 export etcd2_ip_address=172.25.50.18现在docker的启动命令就如下所示了：docker run --net=host -d -v /etc/kubernetes/pki/etcd/:/etc/kubernetes/pki/etcd/ -p 4001:4001 -p 2380:2380 -p 2379:2379 \ -v /data/etcd:/data/etcd \ --name etcd quay.io/coreos/etcd:v3.3.10 \ etcd -name ${PEER_NAME} \ --data-dir /data/etcd \ -advertise-client-urls https://${PRIVATE_IP}:2379,https://${PRIVATE_IP}:4001 \ -listen-client-urls https://0.0.0.0:2379,https://0.0.0.0:4001 \ -initial-advertise-peer-urls https://${PRIVATE_IP}:2380 \ -listen-peer-urls https://0.0.0.0:2380 \ --cert-file=/etc/kubernetes/pki/etcd/server.pem \ --key-file=/etc/kubernetes/pki/etcd/server-key.pem \ --client-cert-auth \ --trusted-ca-file=/etc/kubernetes/pki/etcd/ca.pem \ --peer-cert-file=/etc/kubernetes/pki/etcd/peer.pem \ --peer-key-file=/etc/kubernetes/pki/etcd/peer-key.pem \ --peer-client-cert-auth \ --peer-trusted-ca-file=/etc/kubernetes/pki/etcd/ca.pem \ -initial-cluster-token etcd-cluster \ -initial-cluster etcd1=https://${etcd0_ip_address}:2380,etcd2=https://${etcd1_ip_address}:2380,etcd3=https://${etcd2_ip_address}:2380 \ -initial-cluster-state new但是这样还是不太方便，我们写成docker-compose即可docker-composeversion: '2.2' services: etcd: image: marksugar/coreos-etcd:v3.3.10 container_name: etcd3 restart: always privileged: true network_mode: "host" volumes: - /data/etcd:/data/etcd - /etc/kubernetes/pki/etcd/:/etc/kubernetes/pki/etcd/ command: "etcd -name ${PEER_NAME} --data-dir /data/etcd -advertise-client-urls https://${PRIVATE_IP}:2379,https://${PRIVATE_IP}:4001 -listen-client-urls https://0.0.0.0:2379,https://0.0.0.0:4001 -initial-advertise-peer-urls https://${PRIVATE_IP}:2380 -listen-peer-urls https://0.0.0.0:2380 --cert-file=/etc/kubernetes/pki/etcd/server.pem --key-file=/etc/kubernetes/pki/etcd/server-key.pem --client-cert-auth --trusted-ca-file=/etc/kubernetes/pki/etcd/ca.pem --peer-cert-file=/etc/kubernetes/pki/etcd/peer.pem --peer-key-file=/etc/kubernetes/pki/etcd/peer-key.pem --peer-client-cert-auth --peer-trusted-ca-file=/etc/kubernetes/pki/etcd/ca.pem -initial-cluster-token etcd-cluster -initial-cluster etcd1=https://${etcd0_ip_address}:2380,etcd2=https://${etcd1_ip_address}:2380,etcd3=https://${etcd2_ip_address}:2380 -initial-cluster-state new " cpu_shares: 90 mem_limit: 2048m logging: driver: "json-file" options: max-size: "200M" labels: SERVICE_TAGS: etcd必须：-v /data/etcd:/data/etcd如果丢失数据-dir ==永远丢失成员。附上三个快速重置的脚本：脚本运行后会自动删除集群内的数据存储目录，而后重启当前节点的etcd#!/bin/bash ######################################################################### # File Name: start.sh # Author: www.linuxea.com # Email: usertzc@163.com # Version: # Created Time: Wed 02 Jan 2019 11:14:26 AM CST ######################################################################### for i in 172.25.50.{16,17,18};do ssh $i "docker rm -f etcd && \rm -rf /data/etcd && ls /data";done hostnamectl set-hostname etcd1 export PEER_NAME=$(hostname) export PRIVATE_IP=$(ip addr show eth0 | grep -Po 'inet \K[\d.]+') export etcd0_ip_address=172.25.50.16 export etcd1_ip_address=172.25.50.17 export etcd2_ip_address=172.25.50.18 hostname docker-compose -f /opt/docker-compose.yaml up -d #CMD='--cacert=/etc/kubernetes/pki/etcd/ca.pem --cert=/etc/kubernetes/pki/etcd/server.pem --key=/etc/kubernetes/pki/etcd/server-key.pem' #CMD1='https://172.25.50.16:2379,https://172.25.50.17:2379,https://172.25.50.18:2379' #CMD2='--ca-file=/etc/kubernetes/pki/etcd/ca.pem --cert-file=/etc/kubernetes/pki/etcd/server.pem --key-file=/etc/kubernetes/pki/etcd/server-key.pem' #for i in 172.25.50.{16,17,18}; do ETCDCTL_API=3 etcdctl --endpoints=https://${i}:2379 $CMD endpoint health; done #cd /etc/kubernetes/pki/etcd/ && scp -P22992 ca.pem client.pem client-key.pem 172.25.50.13:/etc/kubernetes/pki/etcd/脚本2中仅仅设置了环境变量和启动的docker-compose.yaml#!/bin/bash ######################################################################### # File Name: start.sh # Author: www.linuxea.com # Email: usertzc@163.com # Version: # Created Time: Wed 02 Jan 2019 11:15:02 AM CST ######################################################################### #docker rm -f etcd && \rm -rf /data/etcd hostnamectl set-hostname etcd2 export PEER_NAME=$(hostname) export PRIVATE_IP=$(ip addr show eth0 | grep -Po 'inet \K[\d.]+') export etcd0_ip_address=172.25.50.16 export etcd1_ip_address=172.25.50.17 export etcd2_ip_address=172.25.50.18 hostname docker-compose -f /opt/docker-compose.yaml up -d脚本3和2几乎 一样，除了名称外#!/bin/bash ######################################################################### # File Name: start.sh # Author: www.linuxea.com # Email: usertzc@163.com # Version: # Created Time: Wed 02 Jan 2019 11:15:26 AM CST ######################################################################### #docker rm -f etcd && \rm -rf /data/etcd hostnamectl set-hostname etcd3 export PEER_NAME=$(hostname) export PRIVATE_IP=$(ip addr show eth0 | grep -Po 'inet \K[\d.]+') export etcd0_ip_address=172.25.50.16 export etcd1_ip_address=172.25.50.17 export etcd2_ip_address=172.25.50.18 hostname docker-compose -f /opt/docker-compose.yaml up -d延伸阅读：https://coreos.com/etcd/docs/latest/v2/docker_guide.html https://github.com/etcd-io/etcd/blob/master/Documentation/op-guide/container.md监控https://coreos.com/etcd/docs/latest/metrics.htmlhttps://etcd.readthedocs.io/en/latest/operate.html#v3-3如果要在kubernetes中监控外部etcd，可参考：https://github.com/marksugar/k8s-pgmon

linuxea:kubernetes helm安装efk(52) efkElasticSearch 是一个搜索引擎，由java编写，Logstash用于生成日志，收集日志，转换日志，而后输入到ElasticSearch 。Logstash扮演agent用来收集日志，因此使用DaemonSet，日志收集完成发送到Logstash server端，Logstash统一整理之后注入到ElasticSearch 当中。ElasticSearch 一般情况下会是一个集群。Logstash也可能不是一个节点，如果两者直接速度不协调，可以使用消息队列，比如：redis组件。一般使用的情况如下图:其中Logstash扮演的角色也包括将所有节点的日志格式化(统一的格式转换)等额外操作，而后注入到ElasticSearch cluster。但是Logstash作为agent来讲，会显得重量级。因此，使用Filebeat来替代。但是，我们并不使用Filebeat。在节点上工作并收集日志的工具不单单有Logstash，Filebeat，还有Fluentd，还有其他的等。在k8s集群上，每个节点运行了很多pod，每个pod内也有很多容器，这些容器的日志信息，就需要统一平台进行管理。尽管我们可以使用kuberctl logs进行查看日志内容，但是在k8s中，pod是 以群体存在的，这种方式并不适用。其次，如果一个pod中的容器崩溃，那么崩溃前的日志内容很可能就无法收集，这是无法解决的。当一个pod容器宕掉，想要查看日志，就需要提前实时进行收集，在云环境平台，尤其需要日志统一平台进行收集日志。在一个完整的k8s平台中,有4个重要附件：coredns,ingress,metrics-server+prometheus,dashboard，而EFK也作为一个基础附件，不过，EFK在k8s集群中大多情况下是必须要提供的。ElasticSearch而在helm中efk的部署方式，特别是ElasticSearch的部署方式和常规的elk方式是有所不同的。为了能让ElasticSearch运行在k8s环境下，ElasticSearch官方制作了类型的相关镜象，在其镜象中打包了相关组件，直接能够运行在k8s之上，一般而言可以运行三种格式：ElasticSearch拆分为两部分，由master和data组成，master节点负责轻量化的查询请求，data节点负责重量级的索引构建等请求。ElasticSearch能够完成既能查询也能构建索引功能，分为两层实现。master作为接入到ElasticSearch的唯一入口，而master和data都可作为分布式，data层可随量横向扩展，master也做多台冗余。在data层是需要持久数据存储，也就是存储卷。这样一来就意味着有两个集群，分别是master和data集群。如下图：除此之外，还有client集群。摄入节点（如上图），收入任何的日志收集工具，如Fluentd发来的日志，由摄入节点统一生成特定格式以后发给master节点，由此，我们可以当作它为Logstash节点。在ELK中Logstash也就是作为清洗日志，而后转交给ElasticSearch。而摄入节点也就是有此功能。这也是x-pack的方式。master 和 data都是有状态的，意味着需要存储卷，持久存储卷。k8s日志收集方式收集日志可以分为两种方式：pod内部和外部内部：每个pod单独发送自己的日志给日志存储的，这样一来就需要在pod内部署几个容器来进行收集日志，随着pod增多，那就需要部署很多。也可以在docker中部署日志收集器，但是这种方式并不被推荐。外部：一般而言， 会在节点上部署一个统一的插件，节点上所有容器，以及节点自身。统一收集发往日志平台，而这套日志收集系统，可以部署在集群内，也可以部署在集群之外。我们暂且不管ElasticSearch部署在哪里。假如使用Fluentd收集，Fluentd本身是部署在集群外和集群内是需要考量的。如果在集群内，只需要挂载存储卷并且将主机日志目录关联到pod中，运行DaemonSet即可。而运行主机上，如果Fluentd出现问题就不再集群控制范围内。Fluentd本身是通过本地/var/log来获取日志，而节点上每一个容器的日志是输出到/var/log/containers/下部署elasticsearch[root@linuxea helm]# helm fetch stable/elasticsearch --version 1.13.3[root@linuxea helm]# tar xf elasticsearch-1.13.3.tgz tar: elasticsearch/Chart.yaml: implausibly old time stamp 1970-01-01 01:00:00 tar: elasticsearch/values.yaml: implausibly old time stamp 1970-01-01 01:00:00 tar: elasticsearch/templates/NOTES.txt: implausibly old time stamp 1970-01-01 01:00:00 tar: elasticsearch/templates/_helpers.tpl: implausibly old time stamp 1970-01-01 01:00:00 tar: elasticsearch/templates/client-deployment.yaml: implausibly old time stamp 1970-01-01 01:00:创建一个名称空间[root@linuxea elasticsearch]# kubectl create namespace efk namespace/efk created[root@linuxea elasticsearch]# kubectl get ns -n efk NAME STATUS AGE default Active 8d efk Active 13s而后install stable/elasticsearch，并且指定名称空间，-f指定/values.yaml文件values文件部分要将持久存储关闭 podDisruptionBudget: enabled: false persistence: enabled: false安装[root@linuxea elasticsearch]# helm install --name els-1 --namespace=efk -f ./values.yaml stable/elasticsearch NAME: els-1 LAST DEPLOYED: Mon Nov 19 06:45:40 2018 NAMESPACE: efk STATUS: DEPLOYED RESOURCES: ==> v1/ConfigMap NAME AGE els-1-elasticsearch 1s ==> v1/ServiceAccount els-1-elasticsearch-client 1s els-1-elasticsearch-data 1s els-1-elasticsearch-master 1s ==> v1/Service els-1-elasticsearch-client 1s els-1-elasticsearch-discovery 1s ==> v1beta1/Deployment els-1-elasticsearch-client 1s ==> v1beta1/StatefulSet els-1-elasticsearch-data 1s els-1-elasticsearch-master 1s ==> v1/Pod(related)初始化状态NAME READY STATUS RESTARTS AGE els-1-elasticsearch-client-779495bbdc-5d22f 0/1 Init:0/1 0 1s els-1-elasticsearch-client-779495bbdc-tzbps 0/1 Init:0/1 0 1s els-1-elasticsearch-data-0 0/1 Init:0/2 0 1s els-1-elasticsearch-master-0 0/1 Init:0/2 0 1sNOTES信息，这些信息可通过stats复现NOTES: The elasticsearch cluster has been installed. Elasticsearch can be accessed: * Within your cluster, at the following DNS name at port 9200: els-1-elasticsearch-client.efk.svc * From outside the cluster, run these commands in the same shell: export POD_NAME=$(kubectl get pods --namespace efk -l "app=elasticsearch,component=client,release=els-1" -o jsonpath="{.items[0].metadata.name}") echo "Visit http://127.0.0.1:9200 to use Elasticsearch" kubectl port-forward --namespace efk $POD_NAME 9200:9200当镜象拉取完成，都准备好就会启动[root@linuxea ~]# kubectl get pods -n efk NAME READY STATUS RESTARTS AGE els-1-elasticsearch-client-779495bbdc-5d22f 0/1 Running 0 32s els-1-elasticsearch-client-779495bbdc-tzbps 0/1 Running 0 32s els-1-elasticsearch-data-0 0/1 Running 0 32s els-1-elasticsearch-master-0 0/1 Running 0 32s稍等后全是Running状态，READY满载，也就是client，client，master都运行了2个[root@linuxea ~]# kubectl get pods -n efk NAME READY STATUS RESTARTS AGE els-1-elasticsearch-client-779495bbdc-5d22f 1/1 Running 0 1m els-1-elasticsearch-client-779495bbdc-tzbps 1/1 Running 0 1m els-1-elasticsearch-data-0 1/1 Running 0 1m els-1-elasticsearch-data-1 1/1 Running 0 47s els-1-elasticsearch-master-0 1/1 Running 0 1m els-1-elasticsearch-master-1 1/1 Running 0 53s验证我们下载一个cirror镜象，验证elasticsearch是否正常运行[root@linuxea elasticsearch]# kubectl run cirror-$RANDOM --rm -it --image=cirros -- /bin/sh If you don't see a command prompt, try pressing enter. /#查看是否能够被解析/ # nslookup els-1-elasticsearch-client.efk.svc Server: 10.96.0.10 Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local Name: els-1-elasticsearch-client.efk.svc Address 1: 10.104.57.197 els-1-elasticsearch-client.efk.svc.cluster.local是否能够访问9200端口/ # curl els-1-elasticsearch-client.efk.svc.cluster.local:9200 { "name" : "els-1-elasticsearch-client-779495bbdc-tzbps", "cluster_name" : "elasticsearch", "cluster_uuid" : "ROD_0h1vRiW_5POVBwz3Nw", "version" : { "number" : "6.4.3", "build_flavor" : "oss", "build_type" : "tar", "build_hash" : "fe40335", "build_date" : "2018-10-30T23:17:19.084789Z", "build_snapshot" : false, "lucene_version" : "7.4.0", "minimum_wire_compatibility_version" : "5.6.0", "minimum_index_compatibility_version" : "5.0.0" }, "tagline" : "You Know, for Search" }_cat/ # curl els-1-elasticsearch-client.efk.svc.cluster.local:9200/_cat/ =^.^= /_cat/allocation /_cat/shards /_cat/shards/{index} /_cat/master /_cat/nodes /_cat/tasks /_cat/indices /_cat/indices/{index} /_cat/segments /_cat/segments/{index} /_cat/count /_cat/count/{index} /_cat/recovery /_cat/recovery/{index} /_cat/health /_cat/pending_tasks /_cat/aliases /_cat/aliases/{alias} /_cat/thread_pool /_cat/thread_pool/{thread_pools} /_cat/plugins /_cat/fielddata /_cat/fielddata/{fields} /_cat/nodeattrs /_cat/repositories /_cat/snapshots/{repository} /_cat/templates以及节点/ # curl els-1-elasticsearch-client.efk.svc.cluster.local:9200/_cat/nodes 172.16.4.92 21 85 2 0.32 0.27 0.16 i - els-1-elasticsearch-client-779495bbdc-tzbps 172.16.5.55 19 76 2 0.08 0.11 0.12 mi * els-1-elasticsearch-master-1 172.16.3.109 21 40 5 0.15 0.13 0.10 di - els-1-elasticsearch-data-0 172.16.4.91 21 85 2 0.32 0.27 0.16 mi - els-1-elasticsearch-master-0 172.16.5.54 21 76 2 0.08 0.11 0.12 i - els-1-elasticsearch-client-779495bbdc-5d22f 172.16.4.93 21 85 1 0.32 0.27 0.16 di - els-1-elasticsearch-data-1或者索引/ # curl els-1-elasticsearch-client.efk.svc.cluster.local:9200/_cat/indices部署fluenetd[root@linuxea helm]# helm fetch stable/fluentd-elasticsearch[root@linuxea helm]# tar xf fluentd-elasticsearch-1.1.0.tgz tar: fluentd-elasticsearch/Chart.yaml: implausibly old time stamp 1970-01-01 01:00:00 tar: fluentd-elasticsearch/values.yaml: implausibly old time stamp 1970-01-01 01:00:00 tar: fluentd-elasticsearch/templates/NOTES.txt: implausibly old time stamp 1970-01-01 01:00:00 tar: fluentd-elasticsearch/templates/_helpers.tpl: implausibly old time stamp 1970-01-01 01:00:00 tar: fluentd-elasticsearch/templates/clusterrole.yaml: implausibly old time stamp 1970-01-01 01:00:00修改elasticsearch: host: 'elasticsearch-client' port: 9200 buffer_chunk_limit: 2M buffer_queue_limit: 8 为elasticsearch: host: 'els-1-elasticsearch-client.efk.svc.cluster.local' port: 9200 buffer_chunk_limit: 2M buffer_queue_limit: 8这个els-1-elasticsearch-client.efk.svc.cluster.local地址是集群内部的地址，并不是pod地址。prometheus启用annotations: prometheus.io/scrape: "true" prometheus.io/port: "24231" ------------- service: type: ClusterIP ports: - name: "monitor-agent" port: 24231污点容忍度tolerations: - key: node-role.kubernetes.io/master operator: Exists effect: NoSchedule而后install[root@linuxea helm]# helm install --name fluentd1 --namespace=efk -f fluentd-elasticsearch/values.yaml stable/fluentd-elasticsearch观察pod running[root@linuxea ~]# kubectl get pods -n efk -w NAME READY STATUS RESTARTS AGE els-1-elasticsearch-client-779495bbdc-5d22f 1/1 Running 0 1h els-1-elasticsearch-client-779495bbdc-tzbps 1/1 Running 0 1h els-1-elasticsearch-data-0 1/1 Running 0 1h els-1-elasticsearch-data-1 1/1 Running 0 1h els-1-elasticsearch-master-0 1/1 Running 0 1h els-1-elasticsearch-master-1 1/1 Running 0 1h fluentd1-fluentd-elasticsearch-28wnb 0/1 ContainerCreating 0 16s fluentd1-fluentd-elasticsearch-77qmr 0/1 ContainerCreating 0 16s fluentd1-fluentd-elasticsearch-885fc 0/1 ContainerCreating 0 16s fluentd1-fluentd-elasticsearch-9kzfm 0/1 ContainerCreating 0 16s fluentd1-fluentd-elasticsearch-lnbvg 0/1 ContainerCreating 0 16s fluentd1-fluentd-elasticsearch-9kzfm 1/1 Running 0 2m fluentd1-fluentd-elasticsearch-77qmr 1/1 Running 0 2m fluentd1-fluentd-elasticsearch-28wnb 1/1 Running 0 2m fluentd1-fluentd-elasticsearch-885fc 1/1 Running 0 2m fluentd1-fluentd-elasticsearch-lnbvg 1/1 Running 0 3m验证验证索引/ # curl els-1-elasticsearch-client.efk.svc.cluster.local:9200/_cat/indices green open logstash-2018.11.15 trf07L62QHaAIG5oym73kw 5 1 12796 0 5.9mb 3mb green open logstash-2018.11.16 thookgxbS86mNmbLPCeOQA 5 1 9964 0 4.8mb 2.4mb green open logstash-2018.11.18 Y0pbmu7RSQizbvoH7V6Cig 5 1 9597 0 5.3mb 2.7mb green open logstash-2018.11.17 pIhgyn-7TaeYOzLbH-dSJA 5 1 13302 0 9.8mb 4.8mb green open logstash-2018.11.14 JirkQqsUSmqUnn0bRPVJLA 5 1 12402 0 5.8mb 2.8mb green open logstash-2018.11.10 4zOpjsVFSMmF0hpY13jryg 5 1 179246 0 128.8mb 65.8mb green open logstash-2018.11.11 ZF6V9DETQlCBsJPSMdc6ww 5 1 34778 0 15.7mb 7.8mb green open logstash-2018.11.13 0qZgntkHTRiuK2_S3Rtvnw 5 1 12679 0 6.3mb 3.1mb green open logstash-2018.11.09 8lSv0UvxQHWTPBZVx9EMVg 5 1 7229 0 7.3mb 4mb green open logstash-2018.11.12 rJiKEdFdTzqE3ovKecn4kw 5 1 11983 0 5.3mb 2.6mb green open logstash-2018.11.19 MhasgEdtS3KWR-KpS30E7A 5 1 38671 0 73.3mb 40mb部署kibanakibana版本要和es一致[root@linuxea helm]# helm fetch stable/kibana --version 0.18.0[root@linuxea helm]# tar xf kibana-0.18.0.tgz tar: kibana/Chart.yaml: implausibly old time stamp 1970-01-01 01:00:00 tar: kibana/values.yaml: implausibly old time stamp 1970-01-01 01:00:00 tar: kibana/templates/NOTES.txt: implausibly old time stamp 1970-01-01 01:00:00 tar: kibana/templates/_helpers.tpl: implausibly old time stamp 1970-01-01 01:00:00修改values的elasticsearch.url地址，这个地址可使用helm list查看，并且使用helm status HELMNAME查看files: kibana.yml: ## Default Kibana configuration from kibana-docker. server.name: kibana server.host: "0" # elasticsearch.url: http://elasticsearch:9200 elasticsearch.url: http://els-1-elasticsearch-client.efk.svc:9200使用NodePort。便于集群外访问service: type: NodePort externalPort: 443 internalPort: 5601另外，如果有必要，这里的image版本必须和ElasticSearch一样image: repository: "docker.elastic.co/kibana/kibana-oss" tag: "6.4.3" pullPolicy: "IfNotPresent"安装[root@linuxea helm]# helm install --name kibana1 --namespace=efk -f kibana/values.yaml stable/kibana --version 0.18.0 NAME: kibana1 LAST DEPLOYED: Mon Nov 19 08:13:34 2018 NAMESPACE: efk STATUS: DEPLOYED RESOURCES: ==> v1/ConfigMap NAME AGE kibana1 1s ==> v1/Service kibana1 1s ==> v1beta1/Deployment kibana1 1s ==> v1/Pod(related) NAME READY STATUS RESTARTS AGE kibana1-578f8d68c7-dvq2z 0/1 ContainerCreating 0 1s NOTES: To verify that kibana1 has started, run: kubectl --namespace=efk get pods -l "app=kibana" Kibana can be accessed: * From outside the cluster, run these commands in the same shell: export NODE_PORT=$(kubectl get --namespace efk -o jsonpath="{.spec.ports[0].nodePort}" services kibana1) export NODE_IP=$(kubectl get nodes --namespace efk -o jsonpath="{.items[0].status.addresses[0].address}") echo http://$NODE_IP:$NODE_PORT经过漫长的等待kibana1已经running，不过由于资源不足，fluentd有被驱逐Evicted的现象[root@linuxea helm]# kubectl get pods -n efk -o wide -w NAME READY STATUS RESTARTS AGE IP NODE els-1-elasticsearch-client-779495bbdc-9rg4x 1/1 Running 0 7m 172.16.3.124 linuxea.node-2.com els-1-elasticsearch-client-779495bbdc-bhq2f 1/1 Running 0 7m 172.16.2.28 linuxea.node-1.com els-1-elasticsearch-data-0 1/1 Running 0 7m 172.16.5.65 linuxea.node-4.com els-1-elasticsearch-data-1 1/1 Running 0 6m 172.16.2.29 linuxea.node-1.com els-1-elasticsearch-master-0 1/1 Running 0 7m 172.16.3.125 linuxea.node-2.com els-1-elasticsearch-master-1 1/1 Running 0 6m 172.16.5.66 linuxea.node-4.com els-1-elasticsearch-master-2 1/1 Running 0 5m 172.16.4.97 linuxea.node-3.com fluentd1-fluentd-elasticsearch-2bllt 1/1 Running 0 3m 172.16.4.98 linuxea.node-3.com fluentd1-fluentd-elasticsearch-7pkvl 1/1 Running 0 3m 172.16.2.30 linuxea.node-1.com fluentd1-fluentd-elasticsearch-cnhk6 1/1 Running 0 3m 172.16.0.26 linuxea.master-1.com fluentd1-fluentd-elasticsearch-mk9m2 1/1 Running 0 3m 172.16.5.67 linuxea.node-4.com fluentd1-fluentd-elasticsearch-wm2kw 1/1 Running 0 3m 172.16.3.126 linuxea.node-2.com kibana1-bfbbf89f6-4tkzb 0/1 ContainerCreating 0 45s <none> linuxea.node-2.com kibana1-bfbbf89f6-4tkzb 1/1 Running 0 1m 172.16.3.127 linuxea.node-2.com通过NodePort端口进行访问[root@linuxea helm]# kubectl get svc -n efk NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE fluentd1-fluentd-elasticsearch ClusterIP 10.106.221.12 <none> 24231/TCP 4m kibana1 NodePort 10.98.70.188 <none> 443:32725/TCP 2m

linuxea:kubernetes 了解chart(51) 在helm的体系中，有helm，tiller server，以及chart。tiller作为服务端一般运行在k8s集群之上，helm能够通过tiller server在k8s之上部署应用程序，这些应用程序来在helm能够访问到的仓库当中的chart。此前我们自定义了一些简单的值文件，实现实例化chart可安装的，可运行的release。也可以通过chart生成release，其中配置文件config来自chart中的values.yaml文件。对于chart来讲，事实上内部没有太复杂的地方，只是将此前我们所使用的编写的配置清单，改写为可复用的格式。此前我们的配置清单的都是为了完成某一个特定的功能而开发，一旦编辑完成，如果需要配置变更就需要重新修改配置清单文件，或者重新更新配置文件。而helm chart引用了一种机制，能够把此前配置清单中特定的内容，尤其是属性值，配置为模板内容。这类似于ansibel当中的playbook模板。只不过其中调用的变量值来自于不同的位置，或是helm内建的，或者是来自部署的release，也有一些属性值是由用户通过值文件(config)提供。chart组成我们先获取一个chart[root@linuxea helm]# helm fetch stable/redis [root@linuxea helm]# ls redis-4.2.10.tgz values.yaml这些文件在打包的时候将时间改为unix元年的时间[root@linuxea helm]# tar xf redis-4.2.10.tgz tar: redis/Chart.yaml: implausibly old time stamp 1970-01-01 01:00:00 tar: redis/values.yaml: implausibly old time stamp 1970-01-01 01:00:00 tar: redis/templates/NOTES.txt: implausibly old time stamp 1970-01-01 01:00:00 tar: redis/templates/_helpers.tpl: implausibly old time stamp 1970-01-01 01:00:00 tar: redis/templates/configmap.yaml: implausibly old time stamp 1970-01-01 01:00:00一个chart组成部分：https://docs.helm.sh/developing_charts/#the-chart-file-structure，翻译如下wordpress/ Chart.yaml #YAML文件，包含有关chart的信息,名称，版本等等信息 LICENSE #OPTIONAL：包含chart许可证的纯文本文件 README.md #OPTIONAL：一个人类可读的README文件 requirements.yaml #OPTIONAL：列出chart依赖关系的YAML文件。如，lnmp的依赖关系。自动处理依赖 values.yaml ＃此chart的默认配置值。对模板中自定义变量的引用设定的默认值。 charts/ ＃包含此chart所依赖的任何chart的目录。charts于requirements类似。目录下存放每一个被当前charts所依赖的其他的charts的打包tgz格式文件。一旦tgz存放在此处，就会被依赖。不管requirements是否存在。手动依赖关系 templates/ ＃模板目录，当与值组合时。这取决于模板的语法，比如清单中定义的service,pvc等，这些便于复用的属性或者字段就会改造成能够通过模板编程语言，基于模板编程结果生成的信息并且自动替换到代码所在处。这需要对go模板语法理解。 ＃将生成有效的Kubernetes清单文件。 templates/NOTES.txt #OcTIONAL：包含简短使用说明的纯文本文件redis文件树如下：yaml定义chartapiVersion：chartAPI版本，始终为“v1”（必填） name：chart的名称（必填），与目录名称保持一致 Version：SemVer 2版本（必填），版本格式 kubeVersion：SemVer系列兼容的Kubernetes版本（可选） description：该项目的单句描述（可选） keywords： - 关于此项目的关键字列表（可选） home：该项目主页的URL（可选） sources： - 此项目源代码的URL列表（可选） maintainers：＃（可选） - name：维护者的名字（每个维护者都需要） email：维护者的电子邮件（每个维护者可选） url：维护者的URL（每个维护者可选） engine：gotpl＃模板引擎的名称（可选，默认为gotpl） icon：要用作图标的SVG或PNG图像的URL（可选）。 appVersion：包含的应用程序版本（可选）。这不一定是SemVer。 deprecated：是否弃用此chart（可选，布尔值） tillerVersion：此chart所需的Tiller版本。这应该表示为SemVer范围：“> 2.0.0”（可选）requirementsdependencies: - name: apache version: 1.2.3 repository: http://example.com/charts - name: mysql version: 3.2.1 repository: http://another.example.com/charts在dependencies中会引用一个列表，每一个列表是一个对象，每一个对象就定义一个被依赖的chart该name字段是您想要的chart的名称。该version字段是您想要的chart的版本。该repository字段是chart存储库的完整URL。请注意，您还必须使用helm repo add在本地添加该repo。打包一个当前的chart，就会分析requirements文件语法格式，将每一个依赖的chart从指定的仓库中下载至本地完成打包。运行helm dependency update CHARTS_NAME ，将使用的requirements文件将所有指定的依赖项下载到charts/目录中。这样在执行一个应用的时候，打包内的依赖都被自动安装完成。当然，如果我此前已知依赖关系，只需要将依赖的包下载到charts目录下，也是可以的。alias也有一些依赖是通过特殊的名称进行依赖，这时候就可以用alias别名来定义依赖关系即可dependencies: - name: subchart repository: http://localhost:10191 version: 0.1.0 alias: new-subchart-1templates模板文件遵循编写Go模板的标准约定，详情参阅:https://golang.org/pkg/text/template/变量模板文件中{{.Values.imageRegistry}}变量，这里的.values就是指明来在values.yaml文件。而imageRegistry和dockerTag才是第一个字段image: {{.Values.imageRegistry}}/postgres:{{.Values.dockerTag}}而在values中的值，分别如下：imageRegistry: "quay.io/deis" dockerTag: "latest"也可以设置默认值预设值，如果.Values.storage存在就是用存在values值文件的值，如果不存在就是用默认default "minio"，如下：value: {{default "minio" .Values.storage}}而以往使用的定义的配置清单，都可以引用模板语法格式，对一些可变动的属性值，可以做成变量引用，一些复杂的场景可以使用条件判断等，如：设置为ture或者false来启用或者关闭一段配置。此后，在安装的时候使用--values=VALUES.yaml即可预定义的值可以从模板中的对象访问通过values.yaml文件（或通过--set标志）提供的值.Values。也可以在模板中访问其他预定义的数据。以下值是预定义的，可用于每个模板，并且无法覆盖。与所有值一样，名称区分大小写。Release.Name：发布的名称（不是chart）Release.Time：chart发布上次更新的时间。这将匹配Last ReleasedRelease对象上的时间。Release.Namespace：chart发布到的名称空间。Release.Service：进行发布的服务。通常这是Tiller。Release.IsUpgrade：如果当前操作是升级或回滚，则设置为true。Release.IsInstall：如果当前操作是安装，则设置为true。Release.Revision：修订号。它从1开始，每个都递增helm upgrade。Chart：内容Chart.yaml。因此，chart版本可以Chart.Version和维护者一起获得Chart.Maintainers。Files：类似于地图的对象，包含chart中的所有非特殊文件。这不会授予您访问模板的权限，但可以访问存在的其他文件（除非使用它们除外.helmignore）。可以使用{{index .Files "file.name"}}或使用{{.Files.Get name}}或 {{.Files.GetString name}}函数访问文件。您也可以访问该文件的内容，[]byte使用{{.Files.GetBytes}}Capabilities：类似于地图的对象，包含有关Kubernetes（{{.Capabilities.KubeVersion}}，Tiller（{{.Capabilities.TillerVersion}}和支持的Kubernetes API）版本（{{.Capabilities.APIVersions.Has "batch/v1"）的版本的信息注意：将删除任何未知的Chart.yaml字段。它们将无法在Chart对象内部访问。因此，Chart.yaml不能用于将任意结构化数据传递到模板中。但是，值文件可以用于此目的。如果要修改一个值就可以使用set修改。创建charthelm create创建一个linuxea.com的chartChart.yaml[root@linuxea helm]# helm create linuxea Creating linuxea而后生成如下清单而后修改一下Chart文件[root@linuxea linuxea]# cat Chart.yaml apiVersion: v1 appVersion: "1.0" description: A Helm chart for linuxea name: linuxea version: 0.1.0 maintainer: - name: mark email: linuxea@gmail.com依赖文件requirements，先不定义NOTES.txt这个文件是在安装生成chart之后，提示给用户的使用信息，也在helm status命令中显示_helpers.tpltpl模板语法用法ymldeployment.yaml ingress.yaml service.yaml在deployment.yaml中语法格式，有1级字段和二级字段，如下：[root@linuxea templates]# grep Values *.yaml deployment.yaml: replicas: {{ .Values.replicaCount }} deployment.yaml: image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"并且还有一些复杂语法模式{{ toYaml .Values.resources | indent 12 }} {{- with .Values.nodeSelector }} nodeSelector: {{ toYaml . | indent 8 }} {{- end }} {{- with .Values.affinity }} affinity: {{ toYaml . | indent 8 }} {{- end }} {{- with .Values.tolerations }} tolerations: {{ toYaml . | indent 8 }} {{- end }}在service.yaml中可以赋值大多数的可变变量[root@linuxea templates]# cat service.yaml apiVersion: v1 kind: Service metadata: name: {{ include "linuxea.com.fullname" . }} labels: app.kubernetes.io/name: {{ include "linuxea.name" . }} helm.sh/chart: {{ include "linuxea.chart" . }} app.kubernetes.io/instance: {{ .Release.Name }} app.kubernetes.io/managed-by: {{ .Release.Service }} spec: type: {{ .Values.service.type }} ports: - port: {{ .Values.service.port }} targetPort: http protocol: TCP name: http selector: app.kubernetes.io/name: {{ include "linuxea.name" . }} app.kubernetes.io/instance: {{ .Release.Name }}以及ingress.yaml[root@linuxea templates]# cat ingress.yaml {{- if .Values.ingress.enabled -}} {{- $fullName := include "linuxea.fullname" . -}} {{- $ingressPath := .Values.ingress.path -}} apiVersion: extensions/v1beta1 kind: Ingress metadata: name: {{ $fullName }} labels: app.kubernetes.io/name: {{ include "linuxea.name" . }} helm.sh/chart: {{ include "linuxea.chart" . }} app.kubernetes.io/instance: {{ .Release.Name }} app.kubernetes.io/managed-by: {{ .Release.Service }} {{- with .Values.ingress.annotations }} annotations: {{ toYaml . | indent 4 }} {{- end }} spec: {{- if .Values.ingress.tls }} tls: {{- range .Values.ingress.tls }} - hosts: {{- range .hosts }} - {{ . | quote }} {{- end }} secretName: {{ .secretName }} {{- end }} {{- end }} rules: {{- range .Values.ingress.hosts }} - host: {{ . | quote }} http: paths: - path: {{ $ingressPath }} backend: serviceName: {{ $fullName }} servicePort: http {{- end }} {{- end }}修改values参数简单修改几项replicaCount: 2打开resources，这里需要将{}删掉，如果启用了的话resources: limits: cpu: 100m memory: 128Mi requests: cpu: 100m memory: 128Mi检查语法而后检查语法是否错误，在父级目录检查[root@linuxea helm]# helm lint linuxea ==> Linting linuxea [INFO] Chart.yaml: icon is recommended 1 chart(s) linted, no failures打包如果没有错误，就可以进行打包，如果有必要，也可以上传到仓库中[root@linuxea helm]# helm package linuxea/ Successfully packaged chart and saved it to: /root/linuxea/manifests/helm/linuxea-0.1.0.tgz[root@linuxea helm]# ll total 40 drwxr-xr-x. 4 root root 93 Nov 18 12:08 linuxea -rw-r--r--. 1 root root 2614 Nov 18 12:09 linuxea-0.1.0.tgz本地仓库我们打开本地的仓库，本地仓库需要运行才能正常使用[root@linuxea linuxea]# helm repo list NAME URL stable https://kubernetes-charts.storage.googleapis.com local http://127.0.0.1:8879/charts 打开一个临时简单的仓库服务，也可以用nginx当作仓库服务使用[root@linuxea prometheus]# helm serve Regenerating index. This may take a moment. Now serving you on 127.0.0.1:8879[root@linuxea linuxea]# ss -tlnp|grep 8879 LISTEN 0 128 127.0.0.1:8879 *:* users:(("helm",pid=3816,fd=3))但是，helm仓库不允许使用命令行上传，但是当创建之后就会被记录下来，可通过helm search查看[root@linuxea helm]# helm search linuxea NAME CHART VERSION APP VERSION DESCRIPTION local/linuxea 0.1.0 1.0 A Helm chart for linuxea.com尤为注意的是NOTES.txt 文件需要真正反映安装相关配置参数，需要进行定义成具体相关的内容安装chart安装[root@linuxea ~]# helm install --name linuxe local/linuxea NAME: linuxe LAST DEPLOYED: Sun Nov 18 12:10:27 2018 NAMESPACE: default STATUS: DEPLOYED RESOURCES: ==> v1/Service NAME AGE linuxe-linuxea 1s ==> v1beta2/Deployment linuxe-linuxea 1s ==> v1/Pod(related) NAME READY STATUS RESTARTS AGE linuxe-linuxea-666bf7dc5b-476gm 0/1 Pending 0 1s linuxe-linuxea-666bf7dc5b-xgs8h 0/1 Pending 0 1s NOTES: 1. Get the application URL by running these commands: export POD_NAME=$(kubectl get pods --namespace default -l "app.kubernetes.io/name=linuxea,app.kubernetes.io/instance=linuxe" -o jsonpath="{.items[0].metadata.name}") echo "Visit http://127.0.0.1:8080 to use your application" kubectl port-forward $POD_NAME 8080:80[root@linuxea ~]# kubectl get pods NAME READY STATUS RESTARTS AGE linuxe-linuxea-666bf7dc5b-476gm 1/1 Running 0 1m linuxe-linuxea-666bf7dc5b-xgs8h 1/1 Running 0 1m linuxea-redis-master-0 1/1 Running 0 6h linuxea-redis-slave-77f4768cd8-rf4l2 1/1 Running 1 6h这里安装的信息是存储在NOTES.txt ，可以在此获取[root@linuxea helm]# helm status linuxe LAST DEPLOYED: Sun Nov 18 12:10:27 2018 NAMESPACE: default STATUS: DEPLOYED RESOURCES: ==> v1/Pod(related) NAME READY STATUS RESTARTS AGE linuxe-linuxea-666bf7dc5b-476gm 1/1 Running 0 3m linuxe-linuxea-666bf7dc5b-xgs8h 1/1 Running 0 3m ==> v1/Service NAME AGE linuxe-linuxea 3m ==> v1beta2/Deployment linuxe-linuxea 3m NOTES: 1. Get the application URL by running these commands: export POD_NAME=$(kubectl get pods --namespace default -l "app.kubernetes.io/name=linuxea,app.kubernetes.io/instance=linuxe" -o jsonpath="{.items[0].metadata.name}") echo "Visit http://127.0.0.1:8080 to use your application" kubectl port-forward $POD_NAME 8080:80卸载chart--purge移除[root@linuxea helm]# helm del --purge linuxe2 release "linuxe2" deleted [root@linuxea helm]# helm del --purge linuxe21 Error: release: "linuxe21" not found [root@linuxea helm]# helm del --purge linuxe1 release "linuxe1" deleted [root@linuxea helm]# helm del --purge linuxe release "linuxe" deleted添加仓库添加incubator，incubator是不稳定的版本[root@linuxea helm]# helm repo add incubator https://kubernetes-charts-incubator.storage.googleapis.com/ "incubator" has been added to your repositories [root@linuxea helm]# helm repo list NAME URL stable https://kubernetes-charts.storage.googleapis.com local http://127.0.0.1:8879/charts incubator https://kubernetes-charts-incubator.storage.googleapis.com/

linuxea:kubernetes helm简单使用(50) 核心组件与程序架构核心组件Chart一个helm程序包，包含了k8s上部署应用程序的清单的定义(并不包含镜像，镜像是由镜像仓库来提供)，镜像的引用，依赖关系，资源定义等等。必要时，包含其他的定义，如：Service.Repository而Chart是有多个，放置到Repository，也可以说是一个Chart。同时也是一个http/https服务器。ReleaseChart实例化后部署在kubernetes cluster之上的Chart实例，就叫做Release。之所以称作为Release，是因为将Chart拿来提供了配置信息。在chart中，会提供一些配置信息，也就是值文件，这些文件最终用来给模板赋值，而后将Chart模板当中的属性使用config或者值文件赋值后生成Release。而config就是值文件，也是应用程序实例化安装运行时候使用的配置信息。程序架构而helm程序由两部分组成，helm客户端和tiller服务端。服务端可以运行在kubernetes之上，也可以运行在kubernetes集群之外，如果运行在kubernetes集群外配置起来是非常麻烦的，多数情况下tiller都会运行在kubernetes集群之上。而helm只是作为本地的客户端，helm基于go语言编写，基于GRPC协议与tiller server进行交互，能够实现管理本地的chart仓库，也可以管理chart，也要与tiller交互，用于发送chart，实现实例安装，查询，卸载等操作。 tiller作为服务端，监听来自helm的请求，而后将helm发来的chart和config合并在一起(将config赋值到chart当中)，完成部署应用程序，生成release。helm安装helm是工作在本地的客户端，不必要是k8s集群内的主机来进行安装部署。[root@linuxea ~]# wget https://storage.googleapis.com/kubernetes-helm/helm-v2.11.0-linux-amd64.tar.gz[root@linuxea ~]# tar xf helm-v2.11.0-linux-amd64.tar.gz -C /usr/local/ [root@linuxea ~]# ln -s /usr/local/linux-amd64/helm /usr/local/bin/helm[root@linuxea ~]# helm version Client: &version.Version{SemVer:"v2.11.0", GitCommit:"2e55dbe1fdb5fdb96b75ff144a339489417b146b", GitTreeState:"clean"} Error: could not find tiller [root@linuxea ~]# tiller生成tiller使用init本身就可以自动生成。helm本身第一次init的时候会联系到api server,api server只会着部署tiller的pod，这也就意味着helm需要链接api server，且需要认认证，并还要管理员权限。helm会获取当前系统的上的kuberctl的权限，在~/.kube/下的config文件是可以被helm获取到。helm会扮演成kubectl客户端，进行初始化和部署安装。[root@linuxea ~]# ls ~/.kube/ cache config http-cachetiller部署在K8s集群的时候，tiller需要获取集群的管理权限，以便于完成安装，卸载等的管理权限。.RBAC权限部署完成helm之后在进行部署tiller的时候，需要在k8s有RBAC集群上，设置RBAC的配置，依赖的用户名(service accout)就是叫做tiller。这个tiller如果要很大的权限的话，就需要绑定到clusterrobingd之上。如下：此前部署k8s集群的时候是强制启用RBAC的，那就需要配置RBAC[root@linuxea manifests]# cat ~/linuxea/manifests/tiller-rbac.yaml apiVersion: v1 kind: ServiceAccount metadata: name: tiller namespace: kube-system --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: tiller roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: cluster-admin subjects: - kind: ServiceAccount name: tiller namespace: kube-system如果是名称空间就需要其他的配置。其他配置参考：https://github.com/helm/helm/blob/master/docs/rbac.md#example-service-account-with-cluster-admin-roleapply[root@linuxea manifests]# kubectl apply -f ~/linuxea/manifests/tiller-rbac.yaml serviceaccount/tiller created clusterrolebinding.rbac.authorization.k8s.io/tiller created[root@linuxea manifests]# kubectl get sa -n kube-system|grep tiller tiller 1 28s生成在第一次生成的时候会在家目录下生成一个/root/.helm 的目录，其中包含很多本地参考的文件。最后在kubernetes-charts.storage.googleapis.com获取一个stable repo[root@linuxea manifests]# helm init --service-account tiller Creating /root/.helm Creating /root/.helm/repository Creating /root/.helm/repository/cache Creating /root/.helm/repository/local Creating /root/.helm/plugins Creating /root/.helm/starters Creating /root/.helm/cache/archive Creating /root/.helm/repository/repositories.yaml Adding stable repo with URL: https://kubernetes-charts.storage.googleapis.com Adding local repo with URL: http://127.0.0.1:8879/charts $HELM_HOME has been configured at /root/.helm. Tiller (the Helm server-side component) has been installed into your Kubernetes Cluster. Please note: by default, Tiller is deployed with an insecure 'allow unauthenticated users' policy. To prevent this, run `helm init` with the --tiller-tls-verify flag. For more information on securing your installation see: https://docs.helm.sh/using_helm/#securing-your-helm-installation Happy Helming! [root@linuxea [root@linuxea manifests]# kubectl get pods -n kube-system NAME READY STATUS RESTARTS AGE coredns-78fcdf6894-9q7cp 1/1 Running 0 2d coredns-78fcdf6894-t8vjc 1/1 Running 0 2d etcd-linuxea.master-1.com 1/1 Running 0 2d kube-apiserver-linuxea.master-1.com 1/1 Running 0 2d kube-controller-manager-linuxea.master-1.com 1/1 Running 0 2d kube-flannel-ds-amd64-j54qv 1/1 Running 0 2d kube-flannel-ds-amd64-kq995 1/1 Running 0 2d kube-flannel-ds-amd64-m2m6t 1/1 Running 0 2d kube-flannel-ds-amd64-sx5cz 1/1 Running 0 2d kube-flannel-ds-amd64-xfvst 1/1 Running 0 2d kube-proxy-8tvxw 1/1 Running 0 2d kube-proxy-9vr46 1/1 Running 0 2d kube-proxy-fmlfq 1/1 Running 0 2d kube-proxy-gt6sk 1/1 Running 0 2d kube-proxy-vtzm2 1/1 Running 0 2d kube-scheduler-linuxea.master-1.com 1/1 Running 0 2d metrics-server-v0.3.1-57ddf848d8-wdx6l 2/2 Running 0 2d tiller-deploy-57f988f854-g8hw6 1/1 Running 0 27s现在已经安装完成Client和Server端版本都是v2.11.0[root@linuxea ~]# helm version Client: &version.Version{SemVer:"v2.11.0", GitCommit:"2e55dbe1fdb5fdb96b75ff144a339489417b146b", GitTreeState:"clean"} Server: &version.Version{SemVer:"v2.11.0", GitCommit:"2e55dbe1fdb5fdb96b75ff144a339489417b146b", GitTreeState:"clean"}此刻可以安装一些应用程序，只需要install即可。由于tiller是由deploy控制器控制，如果要卸载 tiller服务端，只需要删除deploy控制器即可。使用helm远程更新仓库，helm repo update。这条命令会跳过本地而链接远程仓库。[root@linuxea ~]# helm repo update Hang tight while we grab the latest from your chart repositories... ...Skip local chart repository ...Successfully got an update from the "stable" chart repository Update Complete. ⎈ Happy Helming!⎈ [root@linuxea ~]# 我们可以通过https://hub.kubeapps.com/查找一些helm资源。对一些资源选中就可以查看到详情页并且，也可以使用helm使用search搜索，这和docker类似。helm search --help查看帮助。如果没有填写关键词就会列出所有，helm search nginx搜索当前内所有的nginx可以用。如下：[root@linuxea ~]# helm search nginx NAME CHART VERSION APP VERSION DESCRIPTION stable/nginx-ingress 0.31.0 0.20.0 An nginx Ingress controller that uses ConfigMap to store ... stable/nginx-ldapauth-proxy 0.1.2 1.13.5 nginx proxy with ldapauth stable/nginx-lego 0.3.1 Chart for nginx-ingress-controller and kube-lego stable/gcloud-endpoints 0.1.2 1 DEPRECATED Develop, deploy, protect and monitor your APIs... [root@linuxea ~]# 并且可以探索内部的详细信息。这些信息和网页上说差不多[root@linuxea ~]# helm inspect stable/nginx-ingress其次，在repo list中可以看到有稳定版路径和本地路径，如果需要添加incubator，就需要将路径添加[root@linuxea ~]# helm repo list NAME URL stable https://kubernetes-charts.storage.googleapis.com local http://127.0.0.1:8879/charts repo add 添加如下，参考：https://github.com/helm/charts[root@linuxea ~]# helm repo add incubator https://kubernetes-charts-incubator.storage.googleapis.com/ "incubator" has been added to your repositories示例如果我们要安装一个应用，可以在搜索到后进行安装，并且配合values文件进行赋值。一般来讲，不赋值也会有默认参数。安装一个memcached为例[root@linuxea ~]# helm search memcached NAME CHART VERSION APP VERSION DESCRIPTION stable/memcached 2.3.1 1.5.6 Free & open source, high-performance, distributed memory ... stable/mcrouter 0.1.1 0.36.0 Mcrouter is a memcached protocol router for scaling memca...[root@linuxea ~]# helm inspect stable/memcached安装 helm install --name linuxea-men stable/memcached[root@linuxea ~]# helm install --name linuxea-men stable/memcached NAME: linuxea-men 名称 LAST DEPLOYED: Fri Nov 16 13:44:46 2018 部署时间 NAMESPACE: default 部署的名称空间 STATUS: DEPLOYED 当前状态 RESOURCES: ==> v1/Service NAME AGE linuxea-men-memcached 1s ==> v1beta1/StatefulSet linuxea-men-memcached 1s ==> v1beta1/PodDisruptionBudget pod回收机制 linuxea-men-memcached 1s ==> v1/Pod(related) NAME READY STATUS RESTARTS AGE linuxea-men-memcached-0 0/1 Pending 0 1s NOTES: Memcached can be accessed via port 11211 on the following DNS name from within your cluster: linuxea-men-memcached.default.svc.cluster.local If you'd like to test your instance, forward the port locally: export POD_NAME=$(kubectl get pods --namespace default -l "app=linuxea-men-memcached" -o jsonpath="{.items[0].metadata.name}") kubectl port-forward $POD_NAME 11211 In another tab, attempt to set a key: $ echo -e 'set mykey 0 60 5\r\nhello\r' | nc localhost 11211 You should see: STORED 并且可以使用 export POD_NAME=$(kubectl get pods --namespace default -l "app=linuxea-men-memcached" -o jsonpath="{.items[0].metadata.name}")验证删除使用[root@linuxea ~]# helm delete linuxea-menhelm常用命令： delete 删除 history 历史命令 install 安装 list 查看 rollback 回滚 search 查找 upgrade 修改 history 查看部署历史 status 获取状态信息chary管理命令create 创建 delete 删除 fetch 获取，从仓库下载到本地，并且展开 get 获取，仅仅下载 inspect 查看chart详细信息，包括使用，接收值，传递参数等等。 package 将本地开发的包打包 verify 校验示例2我们试图创建一个redis[root@linuxea ~]# helm search redis NAME CHART VERSION APP VERSION DESCRIPTION stable/prometheus-redis-exporter 0.3.4 0.21.2 Prometheus exporter for Redis metrics stable/redis 4.2.10 4.0.11 Open source, advanced key-value store. It is often referr... stable/redis-ha 3.0.1 4.0.11 Highly available Kubernetes implementation of Redis stable/sensu 0.2.3 0.28 Sensu monitoring framework backed by the Redis transport 安装[root@linuxea ~]# helm install --name redis1 stable/redis创建完成可以使用helm list进行查看[root@linuxea ~]# helm list NAME REVISION UPDATED STATUS CHART APP VERSION NAMESPACE redis1 1 Sat Nov 17 06:30:50 2018 DEPLOYED redis-4.2.10 4.0.11 default right-molly 1 Fri Nov 16 13:50:01 2018 DEPLOYED nginx-ingress-0.31.0 0.20.0 default 所有安装的应用都会被get到本地来进行 应用，存放/root/.helm/cache/archive路径下[root@linuxea archive]# cd /root/.helm/cache/archive [root@linuxea archive]# ls memcached-2.3.1.tgz nginx-ingress-0.31.0.tgz redis-4.2.10.tgz这些taz包可以拆开[root@linuxea archive]# tar xf redis-4.2.10.tgz tar: redis/Chart.yaml: implausibly old time stamp 1970-01-01 01:00:00 tar: redis/values.yaml: implausibly old time stamp 1970-01-01 01:00:00 tar: redis/templates/NOTES.txt: implausibly old time stamp 1970-01-01 01:00:00 tar: redis/templates/_helpers.tpl: implausibly old time stamp 1970-01-01 01:00:00 ...其中这些Yaml文件与我们之前所写的大不一样[root@linuxea archive]# tree redis redis ├── Chart.yaml 描述信息 ├── README.md 叙述文件 ├── templates 模板文件 │ ├── configmap.yaml │ ├── health-configmap.yaml │ ├── _helpers.tpl │ ├── metrics-deployment.yaml │ ├── metrics-svc.yaml │ ├── networkpolicy.yaml │ ├── NOTES.txt │ ├── redis-master-statefulset.yaml │ ├── redis-master-svc.yaml │ ├── redis-rolebinding.yaml │ ├── redis-role.yaml │ ├── redis-serviceaccount.yaml │ ├── redis-slave-deployment.yaml │ ├── redis-slave-svc.yaml │ └── secret.yaml ├── values-production.yaml └── values.yaml 1 directory, 19 files [root@linuxea archive]# 这些yaml文件大多调用go模板语法进行赋不同的值传递参数来完成不同的部署，而之前所说的config就是为模板中的属性提供值的配置信息，也可以定义在values.yaml中，并且在install 的时候进行加载即可。其中双##号上注释，单#号表示可以启用的选项示例values我们修改values的一些配置信息，使得redis正常启动[root@linuxea archive]# cp redis/values.yaml ~/linuxea/manifests/helm/ [root@linuxea archive]# cd ~/linuxea/manifests/helm/我们修改下参数，使得使用本地目录存储 persistence: enabled: false删掉此前的创建的redis，在进行部署一次[root@linuxea templates]# helm delete right-molly redis1 release "right-molly" deleted release "redis1" deleted重新部署一次[root@linuxea helm]# helm install --name linuxea-redis -f values.yaml stable/redis[root@linuxea helm]# kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE linuxea-redis-master-0 1/1 Running 0 1m 172.16.4.64 linuxea.node-3.com linuxea-redis-slave-77f4768cd8-rf4l2 1/1 Running 1 1m 172.16.3.78 linuxea.node-2.com使用helm status linuxea-redis可以查看历史中的创建状态信息[root@linuxea helm]# helm status linuxea-redis LAST DEPLOYED: Sun Nov 18 05:16:38 2018 NAMESPACE: default STATUS: DEPLOYED RESOURCES: ==> v1beta2/StatefulSet NAME AGE linuxea-redis-master 2m ==> v1/Pod(related) NAME READY STATUS RESTARTS AGE linuxea-redis-slave-77f4768cd8-rf4l2 1/1 Running 1 2m linuxea-redis-master-0 1/1 Running 0 2m ==> v1/Secret NAME AGE linuxea-redis 2m ==> v1/ConfigMap linuxea-redis-health 2m ==> v1/Service linuxea-redis-master 2m linuxea-redis-slave 2m ==> v1beta1/Deployment linuxea-redis-slave 2m NOTES: ** Please be patient while the chart is being deployed ** Redis can be accessed via port 6379 on the following DNS names from within your cluster: linuxea-redis-master.default.svc.cluster.local for read/write operations linuxea-redis-slave.default.svc.cluster.local for read-only operations To get your password run: export REDIS_PASSWORD=$(kubectl get secret --namespace default linuxea-redis -o jsonpath="{.data.redis-password}" | base64 --decode) To connect to your Redis server: 1. Run a Redis pod that you can use as a client: kubectl run --namespace default linuxea-redis-client --rm --tty -i \ --env REDIS_PASSWORD=$REDIS_PASSWORD \ --image docker.io/bitnami/redis:4.0.11 -- bash 2. Connect using the Redis CLI: redis-cli -h linuxea-redis-master -a $REDIS_PASSWORD redis-cli -h linuxea-redis-slave -a $REDIS_PASSWORD To connect to your database from outside the cluster execute the following commands: kubectl port-forward --namespace default svc/linuxea-redis 6379:6379 & redis-cli -h 127.0.0.1 -p 6379 -a $REDIS_PASSWORD我们run一个client试图登录[root@linuxea helm]# kubectl run --namespace default linuxea-redis-client --rm --tty -i \ > --env REDIS_PASSWORD=$REDIS_PASSWORD \ > --image docker.io/bitnami/redis:4.0.11 -- bash If you don't see a command prompt, try pressing enter. I have no name!@linuxea-redis-client-79859b8d88-2zczc:/$ 而后直接登录redisI have no name!@linuxea-redis-client-79859b8d88-2zczc:/$ redis-cli -h linuxea-redis-master -a $REDIS_PASSWORD Warning: Using a password with '-a' option on the command line interface may not be safe.可info查看linuxea-redis-master:6379> info # Server redis_version:4.0.11 redis_git_sha1:00000000 redis_git_dirty:0 redis_build_id:a3b8846cf74959a5 redis_mode:standalone os:Linux 4.9.12-1.el7.centos.x86_64 x86_64 arch_bits:64 multiplexing_api:epoll atomicvar_api:atomic-builtin gcc_version:6.3.0 process_id:1 run_id:d4288c1f28433ba066538fb02d33bcb0038a9f97 tcp_port:6379 uptime_in_seconds:246 uptime_in_days:0 hz:10 lru_clock:15791815 executable:/opt/bitnami/redis/bin/redis-server config_file:/opt/bitnami/redis/etc/redis.conf # Clients connected_clients:1 client_longest_output_list:0 client_biggest_input_buf:0 blocked_clients:0 # Memory used_memory:1917896 used_memory_human:1.83M used_memory_rss:11833344 used_memory_rss_human:11.29M查看主从状态# Replication role:master connected_slaves:1 slave0:ip=172.16.3.78,port=6379,state=online,offset=322,lag=1 master_replid:6f6a4fe934510bc0187e62c519fc28ee53cf0d8d master_replid2:0000000000000000000000000000000000000000 master_repl_offset:322 second_repl_offset:-1 repl_backlog_active:1 repl_backlog_size:1048576 repl_backlog_first_byte_offset:1 repl_backlog_histlen:322 # CPU used_cpu_sys:0.11 used_cpu_user:0.12 used_cpu_sys_children:0.04 used_cpu_user_children:0.02 # Cluster cluster_enabled:0 # Keyspace linuxea-redis-master:6379>