Kubernetes搭建(ubutu)-master集群

环境设置

1.关闭防火墙

2.关闭selinux

3.关闭swap分区

4.配置本地地址解析

5.修改hostname文件

容器进行时

1.每个节点安装docker

2.配置开机自启、配置Docker守护进程

sudo mkdir /etc/docker
cat <<EOF | sudo tee /etc/docker/daemon.json
{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "storage-driver": "overlay2"
}
EOF

3.重新启动Docker

sudo systemctl enable docker
sudo systemctl daemon-reload
sudo systemctl restart docker

安装 kubeadm、kubectl、kubelet

1.前提：开启某些端口，swap分区必须禁用

2.确保每个节点上 MAC 地址和 product_uuid 的唯一性

#检查mac地址和 product_uuid 
ip link  |awk  '/link\/ether/ {print $2}'
cat /sys/class/dmi/id/product_uuid

3.修改内核参数

创建模块配置文件

#官方基础配置
cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
br_netfilter
EOF
#可尝试使用下列配置
cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
br_netfilter
ip_vs
ip_vs_rr
ip_vs_wrr
ip_vs_sh
nf_conntrack_ipv4
EOF

重新加载配置模块

modprobe br_netfilter

检查模块是否加载成功

lsmod | grep br_netfilter

4.允许 iptables 检查桥接流量

cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sudo sysctl --system

5.检查所需端口

6.安装 kubeadm、kubelet 和 kubectl

• kubeadm：用来初始化集群的指令。
• kubelet：在集群中的每个节点上用来启动 Pod 和容器等。
• kubectl：用来与集群通信的命令行工具

添加 Kubernetes apt 仓库：

sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg

echo "deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list

更新 apt 包索引，安装 kubelet、kubeadm 和 kubectl，并锁定其版本：

sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl

7.设置kubelet服务自启动

sudo systemctl start kubelet
sudo systemctl enable kubelet

sudo systemctl start kubelet

8.执行脚本，下载镜像(仅在master节点下载)

vim /opt/kubeadm-image-pull.sh

aliyun_registry=registry.cn-hangzhou.aliyuncs.com/google_containers
images_arry=$(kubeadm config images list |grep k8s)
for k8s_image in ${images_arry[@]}
do
    image=$(basename $k8s_image)
    docker pull ${aliyun_registry}/${image}
    if [ $? -eq 0 ];then
       docker tag ${aliyun_registry}/${image} ${k8s_image}
       docker rmi ${aliyun_registry}/${image}
       echo "${k8s_image}" is success
    else
       echo "${k8s_image} is fail"
    fi
done

9.确认镜像版本与kubeadm 版本是否为一致

kubeadm初始化集群

1.设置虚拟ip（vip），使用 keepalived

安装keepalived

sudo apt-get install keepalived

修改配置文件,如果没有请创建

sudo vim /etc/keepalived/keepalived.conf

vrrp_instance VI_1 {
    state MASTER # 主MASTER 从BACKUP
    interface ens15f0 # 网卡名称
    virtual_router_id 51 #路由器标识，MASTER和BACKUP必须是一致的
    priority 100 #定义优先级，数字越大，优先级越高，在同一个vrrp_instance下，MASTER的优先级必须大于BACKUP的优先级。这样MASTER故障恢复后，就可以将VIP资源再次抢回来
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass memoriae123456
    }
    virtual_ipaddress {
        10.10.8.60 # 虚拟ip
    }
}

mater从节点keepalived配置文件

vrrp_instance VI_1 {
    state BACKUP # 主MASTER 从BACKUP
    interface ens15f1 # 网卡名称
    virtual_router_id 51 #路由器标识，MASTER和BACKUP必须是一致的
    priority 90 #定义优先级，数字越大，优先级越高，在同一个vrrp_instance下，MASTER的优先级必须大于BACKUP的优先级。这样MASTER故障恢复后，就可以将VIP资源再次抢回来
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass memoriae123456
    }
    virtual_ipaddress {
        10.10.8.60 # 虚拟ip
    }
}

启动keepalived，并设置开机自启

sudo systemctl start keepalived.service
sudo systemctl enable keepalived.service

2.kubuadm init

单节点

只需启动一台master节点，其余节点作为node节点参与；

kubeadm init --kubernetes-version=v1.23.6 --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=10.10.8.60:80 --ignore-preflight-errors=Swap --control-plane-endpoint=memo.com --upload-certs

多节点

启动多个master节点；

kubeadm init --kubernetes-version=v1.23.6 --pod-network-cidr=10.244.0.0/16  --ignore-preflight-errors=Swap --control-plane-endpoint=memo.com --upload-certs

参数解释：

• --node-name 增加该参数设置节点名称，默认名称为当前系统名，--node-name master1-memo

• --kubernetes-version 参数与kubeadm版本一致，使用kubeadm查看当前版本

• --pod-network-cidr=10.244.0.0/16 用来规定pod的网络使用范围

• --apiserver-advertise-address，定义当前网络ip地址用于为控制平面节点的Apiserver设置广播地址，向集群成员发布apiserver的ip地址

• --control-plane-endpoint 允许ip地址和DNS名称，用来添加master节点，此处应在/etc/hosts文件配置本地解析10.10.9.64 memo.com

3.查看节点状态

sudo kubectl get nodes
NAME           STATUS   ROLES                  AGE   VERSION
master1-memo   Ready    control-plane,master   22m   v1.23.6

4.网络插件-flannel

下载网络插件

git clone https://github.com/blackmed/kubernetes-kubeadm.git

进入下载目录，启动配置文件

sudo kubectl apply -f kubernetes-kubeadm/flannel.yaml

停止网络插件

sudo kubectl delete-f kubernetes-kubeadm/flannel.yaml

5.查看节点状态

查看组件pod状态

sudo kubectl get pods -A
NAMESPACE     NAME                                   READY   STATUS    RESTARTS   AGE
kube-system   coredns-64897985d-hfg9p                1/1     Running   0          26m
kube-system   coredns-64897985d-kjzvg                1/1     Running   0          26m
kube-system   etcd-master1-memo                      1/1     Running   0          26m
kube-system   kube-apiserver-master1-memo            1/1     Running   0          26m
kube-system   kube-controller-manager-master1-memo   1/1     Running   0          26m
kube-system   kube-flannel-ds-nkrlt                  1/1     Running   0          7m3s
kube-system   kube-proxy-w4txh                       1/1     Running   0          26m
kube-system   kube-scheduler-master1-memo            1/1     Running   0          26m

检查pod容器日志

sudo kubectl logs kube-flannel-ds-ppgp5 -n kube-system

检查集群scheduler和controller-manager组件健康状态

sudo kubectl get cs
scheduler            Healthy   ok
controller-manager   Healthy   ok
etcd-0               Healthy   {"health":"true","reason":""}

如果检查集群状态显示为：Unhealthy

将两个配置文件中–port注释掉

vim /etc/kubernetes/manifests/kube-scheduler.yaml

vim /etc/kubernetes/manifests/kube-controller-manager.yaml

重新启动kubelet服务,再次查看状态

sudo systemctl restart kubelet

6.生成Token及证书

注意Token时效，如果过期，请重新生成

# kubeadm token create --print-join-command
kubeadm join memo.com:6443 --token wdbmng.cq2l3sxu7gdvwc8v --discovery-token-ca-cert-hash sha256:72156db92a1e71d14eb644f5260ab095d68096fe316e3c982680d58d552ce35e
# kubeadm init phase upload-certs --upload-certs
[upload-certs] Storing the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace
[upload-certs] Using certificate key:
40720d8b9e322b752e2d1cc9b24efd6d5129769245ff5b301210a7d7b0866172

7.加入节点

master从节点执行(加入master集群)

kubeadm join memo.com:6443 --token wdbmng.cq2l3sxu7gdvwc8v --discovery-token-ca-cert-hash sha256:72156db92a1e71d14eb644f5260ab095d68096fe316e3c982680d58d552ce35e --control-plane --certificate-key 40720d8b9e322b752e2d1cc9b24efd6d5129769245ff5b301210a7d7b0866172

node节点执行(加入工作节点)

kubeadm join memo.com:6443 --token wdbmng.cq2l3sxu7gdvwc8v --discovery-token-ca-cert-hash sha256:72156db92a1e71d14eb644f5260ab095d68096fe316e3c982680d58d552ce35e

8.查看当前master集群状态

sudo kubectl get nodes
NAME           STATUS     ROLES                  AGE   VERSION
master1-memo   Ready      control-plane,master   15h   v1.23.6
master2-memo   Ready      control-plane,master   15h   v1.23.6

9.检查etcd状态

master集群部署3个，首先获取集群中的 etcd pod 列表

sudo kubectl get pods -n kube-system | grep etcd
etcd-master1-memo                      1/1     Running   0              17h
etcd-master2-memo                      1/1     Running   0              17h
etcd-master3-memo                      1/1     Running   0              157m

进入任意一个 pod，通过 kubectl 工具进入 pod 内部：

sudo kubectl exec -it kube-flannel-ds-ppgp5 sh -n kube-system

进入后执行

#配置环境
export ETCDCTL_API=3
alias etcdctl='etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key'
查看 etcd 集群成员列表
etcdctl member list
3a82a2a54a23544d, started, master1-memo, https://10.10.9.60:2380, https://10.10.9.60:2379, false
40be7a5531c2ce6e, started, master2-memo, https://10.10.9.48:2380, https://10.10.9.48:2379, false
a0589eb911d55583, started, master3-memo, https://10.10.9.49:2380, https://10.10.9.49:2379, false

清除节点

node节点

sudo kubectl drain <node name> --delete-local-data --force --ignore-daemonsets
sudo kubectl delete node <node name>
#执行完卸载和剔除节点之后，重置node节点，node节点执行
sudo kubeadm reset

master节点

sudo kubeadm reset

执行kubeadm reset之后，请删除执行kubeadm命令用户的 .kube集群文件，防止再次加入集群时报错

sudo rm -r $HOME/.kube

污点(Taints)

kubectl describe node <NODE_NAME> | grep Taint

kubectl describe node master | grep Taint
Taints:             node-role.kubernetes.io/master:NoSchedule  
Taints:             node-role.kubernetes.io/master:NoSchedule  
Taints:             node-role.kubernetes.io/master:NoSchedule