文章目录

• 前言
• 一、基础环境
• 二、开始
• • 1.关闭防火墙
  • 2.禁用SELINUX
  • 3禁用所有swap交换分区
  • 4设置主机名
  • 5创建配置文件
  • 6配置服务器支持开启ipvs的前提条件
• 三、使用kubeadm安装k8s 1.26.0
• • 1、安装containerd
  • 2 拉取镜像，测试containerd是否能创建和启动成功
• 四、安装kubeadm、kubelet、kubectl
• • 更新证书
  • 更新源
  • yum方式
  • apt方式
  • 使用kubeadm init初始化集群
  • 安装包管理器helm 3
  • 部署Pod Network组件Calico
  • 验证k8s DNS是否可用
  • 添加节点
• 五、闭坑和遇到的问题
• • 参考

---

前言

本篇来讲解如何在ubuntu20.4下使用kubeadm安装部署k8s 1.26集群，并使用containerd。

---

一、基础环境

准备1台虚拟机（别惊慌，后面会教你克隆的）。

主机IP	配置
192.168.2.5	master节点，能连外网，ubuntu 20.04版本，至少2核CPU，2G内存

安装基础软件例如vim、wget、netstat,curl，方便修改ymal，查看端口，排查错误等等。
设置root用户。
还有就是换源！！！！更换国内镜像源方法

二、开始

1.关闭防火墙

root@master:~# ufw status					#ufw查看当前的防火墙状态：inactive状态是防火墙关闭状态 active是开启状态。
root@master:~# ufw enable | disable	        #启动、关闭防火墙

2.禁用SELINUX

setenforce 0

vim /etc/selinux/config
SELINUX=disabled

3禁用所有swap交换分区

注：最好是安装虚拟机时就不要创建swap交换分区

root@master:~#  swapoff -a							#禁用所有swap交换分区
root@master:~#  free -h
              total        used        free      shared  buff/cache   available
Mem:           1.8G        280M        1.2G        9.6M        286M        1.4G
Swap:            0B          0B          0B
root@master:~#  vim /etc/fstab			#永久禁用swap,删除或注释掉/etc/fstab里的swap设备的挂载命令即可
#/swap.img      none    swap    sw      0       0

4设置主机名

cat >> /etc/hosts <<EOF
192.168.2.5 K8Smaster
EOF

后续克隆之后在执行

cat >> /etc/hosts <<EOF
192.168.2.6 K8sNode1  #在第二台虚拟机执行
192.168.2.6 K8sNode2  #在第三台虚拟机执行
EOF

注：如果提前配置了3台虚拟机需要同步时间否则会出error

root@master:~# date										#查看时区，时间
root@master:~# timedatectl set-timezone Asia/Shanghai	#先查看时区是否正常，不正确则替换为上海时区
root@master:~# yum -y install ntp	 					#安装ntp服务
root@master:~# systemctl start ntp						#开始ntpd服务,或者做定时任务如：*/5 * * * * /usr/sbin/ntpdate -u 192.168.2.5
root@master:~# systemctl enable ntp

5创建配置文件

创建/etc/modules-load.d/containerd.conf配置文件:

cat << EOF > /etc/modules-load.d/containerd.conf
overlay
br_netfilter
EOF

执行以下命令使配置生效:

modprobe overlay
modprobe br_netfilter

创建/etc/sysctl.d/99-kubernetes-cri.conf配置文件：

cat << EOF > /etc/sysctl.d/99-kubernetes-cri.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
user.max_user_namespaces=28633
EOF

执行以下命令使配置生效:

sysctl -p /etc/sysctl.d/99-kubernetes-cri.conf

6配置服务器支持开启ipvs的前提条件

kube-proxy开启ipvs的前提需要加载以下的内核模块:

ip_vs
ip_vs_rr
ip_vs_wrr
ip_vs_sh
nf_conntrack_ipv4

那么执行脚本

cat > /etc/sysconfig/modules/ipvs.modules <<EOF
#!/bin/bash
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack_ipv4
EOF
chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules && lsmod | grep -e ip_vs -e nf_conntrack_ipv4

上面脚本创建了的/etc/sysconfig/modules/ipvs.modules文件，保证在节点重启后能自动加载所需模块。 使用lsmod | grep -e ip_vs -e nf_conntrack_ipv4命令查看是否已经正确加载所需的内核模块。

接下来还需要确保各个节点上已经安装了ipset软件包，为了便于查看ipvs的代理规则，最好安装一下管理工具ipvsadm。
如果不满足以上前提条件，则即使kube-proxy的配置开启了ipvs模式，也会退回到iptables模式。

三、使用kubeadm安装k8s 1.26.0

kubeadm是官方社区推出的一个用于快速部署kubernetes集群的工具，这个工具能通过两条指令完成一个kubernetes集群的部署。

1、创建一个master节点，kubeadm init。
2、将node节点加入kubernetes集群，kubeadm join <master_IP:port >。

我当然是用yaml啦！kubeadm init --config kubeadm.yaml 一键三连！

1、安装containerd

（在所有节点服务器上都执行，因为k8s 1.24版本默认CRI为containerd，cri称之为容器运行时插件）
containerd的官网
containerd官网安装教程，官网安装文档提供了源码包安装和普通的yum、apt-get安装，这里使用源码包安装。
下载Containerd的二进制包:

#安装containerd
wget https://github.com/containerd/containerd/releases/download/v1.6.14/containerd-1.6.14-linux-amd64.tar.gz #巨慢！建议浏览器下载，然后CP到/usr/local 然后执行解压
tar Cxzvf /usr/local containerd-1.6.8-linux-amd64.tar.gz	#解压到/usr/local/bin目录下了
bin/														#解压到/usr/local/bin目录下了
bin/containerd-shim-runc-v2									#这6个可执行文件就是解压出来的containerd相关命令
bin/containerd-shim
bin/ctr
bin/containerd-shim-runc-v1
bin/containerd
bin/containerd-stress

#使用systemcd来管理containerd
wget https://raw.githubusercontent.com/containerd/containerd/main/containerd.service 
mv containerd.service  /usr/lib/systemd/system/
systemctl daemon-reload && systemctl enable --now containerd 
systemctl  status containerd		               #containerd已经是running状态了

#安装runc
#runc是容器运行时，runc实现了容器的init，run，create，ps...我们在运行容器所需要的cmd：
curl -LO https://github.com/opencontainers/runc/releases/download/v1.1.4/runc.amd64 && \
install -m 755 runc.amd64 /usr/local/sbin/runc

修改containerd的配置，因为containerd默认从k8s官网拉取镜像

mkdir -p /etc/containerd				#创建一个目录用于存放containerd的配置文件
containerd config default | sudo tee /etc/containerd/config.toml	#把containerd配置导出到文件
vim /etc/containerd/config.toml			#修改配置文件
[plugins."io.containerd.grpc.v1.cri"]
..................................
sandbox_image = "registry.aliyuncs.com/google_containers/pause:3.9"	 #搜索sandbox_image，把原来的k8s.gcr.io/pause:3.6改为"registry.aliyuncs.com/google_containers/pause:3.9" 
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
..........................
SystemdCgroup = true					#搜索SystemdCgroup，把这个false改为true		

[plugins."io.containerd.grpc.v1.cri".registry]
     config_path = "/etc/containerd/certs.d"			#搜索config_path，配置镜像加速地址（这是一个目录下面创建）

#创建镜像加速的目录 
mkdir /etc/containerd/certs.d/docker.io -pv
#配置加速
cat > /etc/containerd/certs.d/docker.io/hosts.toml << EOF
server = "https://docker.io"
[host."https://b9pmyelo.mirror.aliyuncs.com"]
  capabilities = ["pull", "resolve"]
EOF

#加载containerd的内核模块
cat <<EOF | sudo tee /etc/modules-load.d/containerd.conf
overlay
br_netfilter
EOF
sudo modprobe overlay
sudo modprobe br_netfilter

#重启containerd
systemctl restart containerd 
systemctl status containerd

2 拉取镜像，测试containerd是否能创建和启动成功

ctr i pull docker.io/library/nginx:alpine			#能正常拉取镜像说明没啥问题
ctr images ls										#查看镜像
ctr c create --net-host docker.io/library/nginx:alpine nginx #创建容器
ctr task start -d nginx								#启动容器，正常说明containerd没啥问题
ctr containers ls 									#查看容器
ctr tasks kill -s SIGKILL  nginx					#终止容器
ctr containers rm nginx								#删除容器

四、安装kubeadm、kubelet、kubectl

更新证书

curl https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg | sudo apt-key add

更新源

apt-get update

yum方式

cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=0
gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
        http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

apt方式

sudo apt update
sudo apt install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl
#apt-mark 用于将软件包标记/取消标记为自动安装。 hold 选项用于将软件包标记为保留，以防止软件包被自动安装、升级或删除。

使用kubeadm init初始化集群

systemctl enable kubelet.service

使用kubeadm config print init-defaults --component-configs KubeletConfiguration可以打印集群初始化默认的使用的配置：

apiVersion: kubeadm.k8s.io/v1beta3
bootstrapTokens:
- groups:
  - system:bootstrappers:kubeadm:default-node-token
  token: abcdef.0123456789abcdef
  ttl: 24h0m0s
  usages:
  - signing
  - authentication
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 1.2.3.4    #需要替换master节点IP
  bindPort: 6443
nodeRegistration:
  criSocket: unix:///var/run/containerd/containerd.sock
  imagePullPolicy: IfNotPresent
  name: node
  taints: null
---
apiServer:
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta3
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controllerManager: {}
dns: {}
etcd:
  local:
    dataDir: /var/lib/etcd
imageRepository: registry.k8s.io  #这个源要换成阿里的！registry.aliyuncs.com/google_containers
kind: ClusterConfiguration
kubernetesVersion: 1.26.0
networking:
  dnsDomain: cluster.local
  serviceSubnet: 10.96.0.0/12
scheduler: {}
---
apiVersion: kubelet.config.k8s.io/v1beta1
authentication:
  anonymous:
    enabled: false
  webhook:
    cacheTTL: 0s
    enabled: true
  x509:
    clientCAFile: /etc/kubernetes/pki/ca.crt
authorization:
  mode: Webhook
  webhook:
    cacheAuthorizedTTL: 0s
    cacheUnauthorizedTTL: 0s
cgroupDriver: systemd
clusterDNS:
- 10.96.0.10
clusterDomain: cluster.local
cpuManagerReconcilePeriod: 0s
evictionPressureTransitionPeriod: 0s
fileCheckFrequency: 0s
healthzBindAddress: 127.0.0.1
healthzPort: 10248
httpCheckFrequency: 0s
imageMinimumGCAge: 0s
kind: KubeletConfiguration
logging:
  flushFrequency: 0
  options:
    json:
      infoBufferSize: "0"
  verbosity: 0
memorySwap: {}
nodeStatusReportFrequency: 0s
nodeStatusUpdateFrequency: 0s
resolvConf: /run/systemd/resolve/resolv.conf
rotateCertificates: true
runtimeRequestTimeout: 0s
shutdownGracePeriod: 0s
shutdownGracePeriodCriticalPods: 0s
staticPodPath: /etc/kubernetes/manifests
streamingConnectionIdleTimeout: 0s
syncFrequency: 0s
volumeStatsAggPeriod: 0s

从默认的配置中可以看到，可以使用imageRepository定制在集群初始化时拉取k8s所需镜像的地址。基于默认配置定制出本次使用kubeadm初始化集群所需的配置文件kubeadm.yaml：

apiVersion: kubeadm.k8s.io/v1beta3
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 192.168.2.5  #master节点IP
  bindPort: 6443
nodeRegistration:
  criSocket: unix:///run/containerd/containerd.sock
  taints:
  - effect: PreferNoSchedule
    key: node-role.kubernetes.io/master
---
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
kubernetesVersion: 1.26.0
imageRepository: registry.aliyuncs.com/google_containers #阿里的源
networking:
  podSubnet: 10.244.0.0/16
---
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
cgroupDriver: systemd
failSwapOn: false
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: ipvs

这里定制了imageRepository为阿里云的registry，避免因gcr被墙，无法直接拉取镜像。criSocket设置了容器运行时为containerd。 同时设置kubelet的cgroupDriver为systemd，设置kube-proxy代理模式为ipvs。

在开始初始化集群之前可以使用kubeadm config images pull --config kubeadm.yaml预先在各个服务器节点上拉取所k8s需要的容器镜像。

kubeadm config images pull --config kubeadm.yaml  #提前拉取镜像
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-apiserver:v1.26.0
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-controller-manager:v1.26.0
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-scheduler:v1.26.0
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-proxy:v1.26.0
[config/images] Pulled registry.aliyuncs.com/google_containers/pause:3.9
[config/images] Pulled registry.aliyuncs.com/google_containers/etcd:3.5.6-0
[config/images] Pulled registry.aliyuncs.com/google_containers/coredns:v1.9.3

接下来使用kubeadm初始化集群执行下面的命令：

kubeadm init --config kubeadm.yaml

W0104 01:55:08.531622   29428 common.go:84] your configuration file uses a deprecated API spec: "kubeadm.k8s.io/v1beta2". Please use 'kubeadm config migrate --old-config old.yaml --new-config new.yaml', which will write the new, similar spec using a newer API version.
[init] Using Kubernetes version: v1.26.0
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Using existing ca certificate authority
[certs] Using existing apiserver certificate and key on disk
[certs] Using existing apiserver-kubelet-client certificate and key on disk
[certs] Using existing front-proxy-ca certificate authority
[certs] Using existing front-proxy-client certificate and key on disk
[certs] Using existing etcd/ca certificate authority
[certs] Using existing etcd/server certificate and key on disk
[certs] Using existing etcd/peer certificate and key on disk
[certs] Using existing etcd/healthcheck-client certificate and key on disk
[certs] Using existing apiserver-etcd-client certificate and key on disk
[certs] Using the existing "sa" key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Using existing kubeconfig file: "/etc/kubernetes/admin.conf"
[kubeconfig] Using existing kubeconfig file: "/etc/kubernetes/kubelet.conf"
[kubeconfig] Using existing kubeconfig file: "/etc/kubernetes/controller-manager.conf"
[kubeconfig] Using existing kubeconfig file: "/etc/kubernetes/scheduler.conf"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 9.503362 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node node1 as control-plane by adding the labels: [node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]
[mark-control-plane] Marking the node node1 as control-plane by adding the taints [node-role.kubernetes.io/master:PreferNoSchedule]
[bootstrap-token] Using token: o4pn35.3zjs1udp6uw1eg8w
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:
#注意：这段kubeamd join命令的token只有24h，24h就过期，需要执行kubeadm token create --print-join-command 重新生成。
kubeadm join 192.168.2.5:6443 --token shrl66.4tkloy7gnukj03l4 \
    --discovery-token-ca-cert-hash sha256:f82eb61a39fbc0164f0b91e2d2619fc3d8f9b2d1b235ed3a60c4cc4e97b68dc9
    

上面记录了完成的初始化输出的内容，根据输出的内容基本上可以看出手动初始化安装一个Kubernetes集群所需要的关键步骤。 其中有以下关键内容：

[certs]生成相关的各种证书
[kubeconfig]生成相关的kubeconfig文件
[kubelet-start] 生成kubelet的配置文件"/var/lib/kubelet/config.yaml"
[control-plane]使用/etc/kubernetes/manifests目录中的yaml文件创建apiserver、controller-manager、scheduler的静态pod
[bootstraptoken]生成token记录下来，后边使用kubeadm join往集群中添加节点时会用到
[addons]安装基本插件:CoreDNS, kube-proxy
下面的命令是配置常规用户如何使用kubectl访问集群：

#我们根据输入的提示信息复制粘贴照着做即可
root@master:~# mkdir -p $HOME/.kube										#复制上面提示照着做即可
root@master:~# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config	#复制上面提示照着做即可
root@master:~# sudo chown $(id -u):$(id -g) $HOME/.kube/config			#复制上面提示照着做即可
root@master:~#  export KUBECONFIG=/etc/kubernetes/admin.conf

查看一下集群状态，确认个组件都处于healthy状态

kubectl get cs

Warning: v1 ComponentStatus is deprecated in v1.19+
NAME                 STATUS    MESSAGE                         ERROR
controller-manager   Healthy   ok
scheduler            Healthy   ok
etcd-0               Healthy   {"health":"true","reason":""}

如果出现问题，就执行kubeadm reset命令进行清理，然后执行执行rm -rf $HOME/.kube，再init。（坑后面会补充）

安装包管理器helm 3

Helm是Kubernetes的包管理器，后续流程也将使用Helm安装Kubernetes的常用组件。 这里先在master节点node1上安装helm。

wget https://get.helm.sh/helm-v3.10.3-linux-amd64.tar.gz
tar -zxvf helm-v3.10.3-linux-amd64.tar.gz
mv linux-amd64/helm  /usr/local/bin/

执行helm list确认没有错误输出。

部署Pod Network组件Calico

选择calico作为k8s的Pod网络组件，下面使用helm在k8s集群中安装calico。

下载tigera-operator的helm chart:

wget https://github.com/projectcalico/calico/releases/download/v3.24.5/tigera-operator-v3.24.5.tgz

查看这个chart的中可定制的配置:

helm show values tigera-operator-v3.24.5.tgz

imagePullSecrets: {}

installation:
  enabled: true
  kubernetesProvider: ""

apiServer:
  enabled: true

certs:
  node:
    key:
    cert:
    commonName:
  typha:
    key:
    cert:
    commonName:
    caBundle:


#Resource requests and limits for the tigera/operator pod.
resources: {}

# Tolerations for the tigera/operator pod.
tolerations:
- effect: NoExecute
  operator: Exists
- effect: NoSchedule
  operator: Exists

# NodeSelector for the tigera/operator pod.
nodeSelector:
  kubernetes.io/os: linux

# Custom annotations for the tigera/operator pod.
podAnnotations: {}

# Custom labels for the tigera/operator pod.
podLabels: {}

# Image and registry configuration for the tigera/operator pod.
tigeraOperator:
  image: tigera/operator
  version: v1.28.5
  registry: quay.io
calicoctl:
  image: docker.io/calico/ctl
  tag: v3.24.5

定制的values.yaml如下:

# 可针对上面的配置进行定制,例如calico的镜像改成从私有库拉取。
# 这里只是个人本地环境测试k8s新版本，这里只有下面几行配置
apiServer:
  enabled: false

使用helm安装calico：

helm install calico tigera-operator-v3.24.5.tgz -n kube-system  --create-namespace -f values.yaml
等待并确认所有pod处于Running状态:


kubectl get pod -n kube-system | grep tigera-operator
tigera-operator-5fb55776df-wxbph   1/1     Running   0             5m10s

kubectl get pods -n calico-system
NAME                                       READY   STATUS    RESTARTS   AGE
calico-kube-controllers-68884f975d-5d7p9   1/1     Running   0          5m24s
calico-node-twbdh                          1/1     Running   0          5m24s
calico-typha-7b4bdd99c5-ssdn2              1/1     Running   0          5m24s

查看一下calico向k8s中添加的api资源:

kubectl api-resources | grep calico
bgpconfigurations                              crd.projectcalico.org/v1               false        BGPConfiguration
bgppeers                                       crd.projectcalico.org/v1               false        BGPPeer
blockaffinities                                crd.projectcalico.org/v1               false        BlockAffinity
caliconodestatuses                             crd.projectcalico.org/v1               false        CalicoNodeStatus
clusterinformations                            crd.projectcalico.org/v1               false        ClusterInformation
felixconfigurations                            crd.projectcalico.org/v1               false        FelixConfiguration
globalnetworkpolicies                          crd.projectcalico.org/v1               false        GlobalNetworkPolicy
globalnetworksets                              crd.projectcalico.org/v1               false        GlobalNetworkSet
hostendpoints                                  crd.projectcalico.org/v1               false        HostEndpoint
ipamblocks                                     crd.projectcalico.org/v1               false        IPAMBlock
ipamconfigs                                    crd.projectcalico.org/v1               false        IPAMConfig
ipamhandles                                    crd.projectcalico.org/v1               false        IPAMHandle
ippools                                        crd.projectcalico.org/v1               false        IPPool
ipreservations                                 crd.projectcalico.org/v1               false        IPReservation
kubecontrollersconfigurations                  crd.projectcalico.org/v1               false        KubeControllersConfiguration
networkpolicies                                crd.projectcalico.org/v1               true         NetworkPolicy
networksets                                    crd.projectcalico.org/v1               true         NetworkSet

这些api资源是属于calico的，因此不建议使用kubectl来管理，推荐按照calicoctl来管理这些api资源。 将calicoctl安装为kubectl的插件:

cd /usr/local/bin
curl -o kubectl-calico -O -L  "https://github.com/projectcalico/calicoctl/releases/download/v3.21.5/calicoctl-linux-amd64" 
chmod +x kubectl-calico

也可以自行下来，然后cp到/usr/local/bin，改名字，改成可执行。

验证插件正常工作:

kubectl calico -h

验证k8s DNS是否可用

kubectl run curl --image=radial/busyboxplus:curl -it
If you don't see a command prompt, try pressing enter.
[ root@curl:/ ]$

进入后执行nslookup kubernetes.default确认解析正常:

nslookup kubernetes.default
Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

Name:      kubernetes.default
Address 1: 10.96.0.1 kubernetes.default.svc.cluster.local

添加节点

克隆出2台K8sNode1, K8sNode1添加到Kubernetes集群中，分别在node2, node3上执行:

kubeadm join 192.168.2.5:6443 --token shrl66.4tkloy7gnukj03l4 \
    --discovery-token-ca-cert-hash sha256:f82eb61a39fbc0164f0b91e2d2619fc3d8f9b2d1b235ed3a60c4cc4e97b68dc9

node2和node3加入集群很是顺利，在master节点上执行命令查看集群中的节点：

kubectl get node
NAME    STATUS   ROLES           AGE   VERSION
master   Ready    control-plane   47m   v1.26.0
K8sNode1   Ready    <none>          31s   v1.26.0
K8sNode1   Ready    <none>          19s   v1.26.0

---

五、闭坑和遇到的问题

在安装过程中遇到的一些问题和操作时的流程注意事项，也欢迎各位总结汇总，我也会定期更新
1、在kubeadm init成功之后，如果想kubeadm reset ，那么需要在执行kubeadm reset之前手动删除 $HOME/.kube。否则会报：

Unable to connect to the server: x509: certificate signed by unknown authority

不要犹豫直接删rm -rf $HOME/.kube，可以解决。
2、kubeadm init 时出现卡在

[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.

然后通过journalctl -f -u containerd看容器引擎的日志

failed, error" error="failed to get sandbox image \"k8s.gcr.io/pause:3.6\": failed to pull image \"k8s.gcr.io/pause:3.6\": failed to pull and unpack image \"k8s.gcr.io/pause:3.6\": failed to resolve reference \"k8s.gcr.io/pause:3.6\": failed to do request: Head \"https://k8s.gcr.io/v2/pause/manifests/3.6\": dial tcp 108.177.125.82:443: i/o timeout"
Jul 05 19:08:30 k8s-testing01-190 containerd[13788]: time="2023-3-01T19:08:30.696324518+08:00" level=info msg="trying next host" error="failed to do request: Head \"https://k8s.gcr.io/v2/pause/manifests/3.6\": dial tcp 108.177.125.82:443: i/o timeout" host=k8s.gcr.io

因为containerd默认从k8s官网拉取镜像，其次默认配置中是pause:3.6版本，而实际我们需要的3.9版本。
failed to pull and unpack image \"k8s.gcr.io/pause:3.6\
所以我们安装完containerd一定修改containerd的配置，改源，改pause版本号。

参考

参考