草庐IT

k8s-v1.18.1证书过期处理

前浪浪奔浪流 2023-03-28 原文

很久没开的k8s测试环境，今天打开发现在master节点查看node发现node2 notready 状态
在node2节点查看发现kubelet停止运行了

kubelet报错：

part of the existing bootstrap client certificate is expired: 2022-06-04

通过查看/etc/kubernetes/kubelet.conf 发现证书路径/var/lib/kubelet/pki/kubelet-client-current.pem

cat /etc/kubernetes/kubelet.conf

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: LS0tLS1......UtLS0tLQo=
    server: https://192.168.100.201:6443
  name: default-cluster
contexts:
- context:
    cluster: default-cluster
    namespace: default
    user: default-auth
  name: default-context
current-context: default-context
kind: Config
preferences: {}
users:
- name: default-auth
  user:
    client-certificate: /var/lib/kubelet/pki/kubelet-client-current.pem
    client-key: /var/lib/kubelet/pki/kubelet-client-current.pem

然后切换到/var/lib/kubelet/pki/ 路径下查看证书日期

cd /var/lib/kubelet/pki
ll
总用量 20
-rw------- 1 root root 1061 9月  14 2020 kubelet-client-2020-09-14-18-00-01.pem
-rw------- 1 root root 1061 6月   4 2021 kubelet-client-2021-06-04-19-03-23.pem
-rw------- 1 root root 1066 6月  10 11:00 kubelet-client-2022-06-10-11-00-15.pem
lrwxrwxrwx 1 root root   59 6月  10 11:00 kubelet-client-current.pem -> /var/lib/kubelet/pki/kubelet-client-2021-06-04-19-03-23.pem
-rw-r--r-- 1 root root 2144 9月  14 2020 kubelet.crt
-rw------- 1 root root 1679 9月  14 2020 kubelet.key

可以看出kubelet-client-current.pem指向的是kubelet-client-2021-06-04-19-03-23.pem 现在是2022-06-10 所以证书已经过期了。

在node2上查看证书有效期

# openssl x509 -in /var/lib/kubelet/pki/kubelet-client-current.pem -noout -text | grep Not
            Not Before: Jun  4 10:58:23 2021 GMT
            Not After : Jun  4 10:58:23 2022 GMT

由于我的 master节点和node1节点都正常；
我可以用之前的kubeadm.yaml配置文件重新生成下证书

#备份之前的证书
# cp -rp /etc/kubernetes /etc/kubernetes.bak

#生成新的证书
# kubeadm alpha certs renew all --config=kubeadm.yaml
W0610 09:24:36.851093   26346 configset.go:202] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself renewed
certificate for serving the Kubernetes API renewed
certificate the apiserver uses to access etcd renewed
certificate for the API server to connect to kubelet renewed
certificate embedded in the kubeconfig file for the controller manager to use renewed
certificate for liveness probes to healthcheck etcd renewed
certificate for etcd nodes to communicate with each other renewed
certificate for serving etcd renewed
certificate for the front proxy client renewed
certificate embedded in the kubeconfig file for the scheduler manager to use renewed

#备份之前的配置文件
# mkdir  /root/backconf
# mv /etc/kubernetes/*.conf    /root/backconf/
# ll backconf/
总用量 32
-rw------- 1 root root 5451 6月  10 09:24 admin.conf
-rw------- 1 root root 5491 6月  10 09:24 controller-manager.conf
-rw------- 1 root root 5463 9月   1 2021 kubelet.conf
-rw------- 1 root root 5439 6月  10 09:24 scheduler.conf

#重新生成配置文件
# kubeadm init phase kubeconfig all --config kubeadm.yaml
W0610 09:26:59.426236   27497 configset.go:202] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file

# ll /etc/kubernetes/
总用量 52
-rw------- 1 root root 5451 6月  10 09:27 admin.conf
-rw-r--r-- 1 root root 1025 3月  23 2021 ca.crt
-rw-r--r-- 1 root root 3117 3月  23 2021 cert.pfx
-rw-r--r-- 1 root root 1082 3月  23 2021 client.crt
-rw-r--r-- 1 root root 1679 3月  23 2021 client.key
-rw------- 1 root root 5487 6月  10 09:27 controller-manager.conf
-rw------- 1 root root 5459 6月  10 09:27 kubelet.conf
drwxr-xr-x 2 root root  113 10月  6 2021 manifests
drwxr-xr-x 3 root root 4096 9月  14 2020 pki
-rw------- 1 root root 5439 6月  10 09:27 scheduler.conf

# 将新生成的admin.conf文件覆盖掉.kube/config文件:
mv $HOME/.kube/config $HOME/.kube/config.old
cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
chown $(id -u):$(id -g) $HOME/.kube/config
chmod 644 $HOME/.kube/config

# 重启kube-apiserver,kube-controller,kube-scheduler,etcd这4个容器：(一定要ps -a要不有可能服务容器没启动)
# docker ps -a | grep -v pause | grep -E "etcd|scheduler|controller|apiserver" | awk '{print $1}' | awk '{print "docker","restart",$1}' | bash

# 各节点重启kubelet或相关组件:
systemctl restart kubelet

master节点就更新完成了，然后获取token在更新slave节点时要用

# kubeadm token create --print-join-command
W0610 09:40:30.975578    2435 configset.go:202] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
kubeadm join 192.168.100.201:6443 --token 6co5f1.g8wnog41jopfchp8     --discovery-token-ca-cert-hash sha256:8adf630dbe900681db88950f0877faa7be4308f6fd837029ab7e9e41dd0eafd6

# kubeadm token list
TOKEN                     TTL         EXPIRES                     USAGES                   DESCRIPTION                                                EXTRA GROUPS
6co5f1.g8wnog41jopfchp8   23h         2022-06-11T09:40:31+08:00   authentication,signing   <none>                                                     system:bootstrappers:kubeadm:default-node-token

node节点添加进集群（需删除原先kubelet配置文件，否则加入失败）

先备份下配置文件的存放目录

cp -r /etc/kubernetes /etc/kubernetes.bak
# ll /etc/kubernetes*
/etc/kubernetes:
总用量 4
-rw------- 1 root root 1856 9月  14 2020 kubelet.conf
drwxr-xr-x 2 root root    6 4月   9 2020 manifests
drwxr-xr-x 2 root root   20 9月  14 2020 pki

/etc/kubernetes.bak:
总用量 4
-rw------- 1 root root 1856 6月  10 10:58 kubelet.conf
drwxr-xr-x 2 root root    6 6月  10 10:58 manifests
drwxr-xr-x 2 root root   20 6月  10 10:58 pki

然后删除旧的kubelet配置文件

#  rm -rf /etc/kubernetes/kubelet.conf
#  rm -rf /etc/kubernetes/pki/ca.crt
# rm -rf /etc/kubernetes/bootstrap-kubelet.conf     #这个文件我没有
# systemctl stop kubelet
# systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
   Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
  Drop-In: /usr/lib/systemd/system/kubelet.service.d
           └─10-kubeadm.conf
   Active: inactive (dead) since 五 2022-06-10 09:38:04 CST; 1h 20min ago
     Docs: https://kubernetes.io/docs/
  Process: 31448 ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS (code=exited, status=0/SUCCESS)
 Main PID: 31448 (code=exited, status=0/SUCCESS)

6月 10 09:37:59 node2 kubelet[31448]: E0610 09:37:59.469934   31448 reflector.go:178] object-"loki"/"loki": Failed to list *v1.Secret: secrets "loki" is forb...this object
6月 10 09:37:59 node2 kubelet[31448]: W0610 09:37:59.676710   31448 status_manager.go:572] Failed to update status for pod "loki-0_loki(e0ea4379-7e48-4107-83...\"Initializ
6月 10 09:38:00 node2 kubelet[31448]: W0610 09:38:00.077588   31448 status_manager.go:572] Failed to update status for pod "sentinel-0_default(49b3d865-37ae-...type\":\"In
6月 10 09:38:00 node2 kubelet[31448]: W0610 09:38:00.476110   31448 status_manager.go:572] Failed to update status for pod "usercenter-deployment-7bf4744f58-...ementOrder/
6月 10 09:38:00 node2 kubelet[31448]: W0610 09:38:00.877862   31448 status_manager.go:572] Failed to update status for pod "getaway-deployment-6595fb8444-ztf...ntOrder/con
6月 10 09:38:02 node2 kubelet[31448]: I0610 09:38:02.721843   31448 kubelet_node_status.go:294] Setting node annotation to enable volume controller attach/detach
6月 10 09:38:02 node2 kubelet[31448]: I0610 09:38:02.849726   31448 kubelet_node_status.go:70] Attempting to register node node2
6月 10 09:38:02 node2 kubelet[31448]: E0610 09:38:02.859581   31448 kubelet_node_status.go:92] Unable to register node "node2" with API server: nodes "node2"...ode "node2"
6月 10 09:38:04 node2 systemd[1]: Stopping kubelet: The Kubernetes Node Agent...
6月 10 09:38:04 node2 systemd[1]: Stopped kubelet: The Kubernetes Node Agent.
Hint: Some lines were ellipsized, use -l to show in full.

node2重新加入集群

# kubeadm join 192.168.100.201:6443 --token 6co5f1.g8wnog41jopfchp8     --discovery-token-ca-cert-hash sha256:8adf630dbe900681db88950f0877faa7be4308f6fd837029ab7e9e41dd0eafd6
W0610 11:00:11.849573    5754 join.go:346] [preflight] WARNING: JoinControlPane.controlPlane settings will be ignored when control-plane flag is not set.
[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.18" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

验证结果

[root@master ~]# kubectl get node
NAME     STATUS   ROLES    AGE    VERSION
master   Ready    master   633d   v1.18.1
node1    Ready    <none>   633d   v1.18.1
node2    Ready    <none>   633d   v1.18.1

[root@master ~]# kubectl get pods -n kube-system
NAME                                        READY   STATUS    RESTARTS   AGE
coredns-7ff77c879f-629sv                    1/1     Running   15         633d
coredns-7ff77c879f-hk25m                    1/1     Running   15         633d
default-http-backend-55fb564b-rrddj         1/1     Running   3          146d
etcd-master                                 1/1     Running   15         633d
kube-apiserver-master                       1/1     Running   8          386d
kube-controller-manager-master              1/1     Running   7          281d
kube-flannel-ds-amd64-g885t                 1/1     Running   15         633d
kube-flannel-ds-amd64-nm5xp                 1/1     Running   14         633d
kube-flannel-ds-amd64-zd56s                 1/1     Running   15         633d
kube-proxy-rdf9s                            1/1     Running   16         633d
kube-proxy-rsm5n                            1/1     Running   14         633d
kube-proxy-wc7zr                            1/1     Running   15         633d
kube-scheduler-master                       1/1     Running   17         633d
kube-state-metrics-99d76dd5d-srlvt          1/1     Running   8          300d
metrics-server-7b75fd6bfb-4prml             1/1     Running   9          386d
nginx-ingress-controller-5cf88d6db5-mqp8c   1/1     Running   3          146d
k8s-vk8kubeletrootnode

有关k8s-v1.18.1证书过期处理的更多相关文章

  1. ruby - i18n Assets 管理/翻译 UI - 2

    我正在使用i18n从头开始​​构建一个多语言网络应用程序，虽然我自己可以处理一大堆yml文件，但我说的语言(非常)有限，最终我想寻求外部帮助帮助。我想知道这里是否有人在使用UI插件/gem(与django上的django-rosetta不同)来处理多个翻译器，其中一些翻译器不愿意或无法处理存储库中的100多个文件，处理语言数据。谢谢&问候，安德拉斯(如果您已经在ruby​​onrails-talk上遇到了这个问题，我们深表歉意) 最佳答案 有一个rails3branchofthetolkgem在github上。您可以通过在Gemfi

  2. ruby - 如何指定 Rack 处理程序 - 2

    Rackup通过Rack的默认处理程序成功运行任何Rack应用程序。例如:classRackAppdefcall(environment)['200',{'Content-Type'=>'text/html'},["Helloworld"]]endendrunRackApp.new但是当最后一行更改为使用Rack的内置CGI处理程序时，rackup给出“NoMethodErrorat/undefinedmethod`call'fornil:NilClass”:Rack::Handler::CGI.runRackApp.newRack的其他内置处理程序也提出了同样的反对意见。例如Rack

  3. ruby-on-rails - Rails 3 I18 : translation missing: da. datetime.distance_in_words.about_x_hours - 2

    我看到这个错误:translationmissing:da.datetime.distance_in_words.about_x_hours我的语言环境文件:http://pastie.org/2944890我的看法:我已将其添加到我的application.rb中:config.i18n.load_path+=Dir[Rails.root.join('my','locales','*.{rb,yml}').to_s]config.i18n.default_locale=:da如果我删除I18配置，帮助程序会处理英语。更新:我在config/enviorments/devolpment

  4. ruby-on-rails - ActionController::RoutingError: 未初始化常量 Api::V1::ApiController - 2

    我有用于控制用户任务的Rails5API项目，我有以下错误，但并非总是针对相同的Controller和路由。ActionController::RoutingError:uninitializedconstantApi::V1::ApiController我向您描述了一些我的项目，以更详细地解释错误。应用结构路线scopemodule:'api'donamespace:v1do#=>Loginroutesscopemodule:'login'domatch'login',to:'sessions#login',as:'login',via::postend#=>Teamroutessc

  5. ruby-on-rails - 如果我将 ruby​​ 版本 2.5.1 与 rails 版本 2.3.18 一起使用会怎样？ - 2

    如果我使用ruby​​版本2.5.1和Rails版本2.3.18会怎样？我有基于rails2.3.18和ruby​​1.9.2p320构建的rails应用程序，我只想升级ruby的版本，而不是rails，这可能吗？我必须面对哪些挑战？ 最佳答案 GitHub维护apublicfork它有针对旧Rails版本的分支，有各种变化，它们一直在运行。有一段时间，他们在较新的Ruby版本上运行较旧的Rails版本，而不是最初支持的版本，因此您可能会发现一些关于需要向后移植的有用提示。不过，他们现在已经有几年没有使用2.3了，所以充其量只能让更

  6. ruby-on-rails - Ruby on Rails I18n 插值 - 2

    大家好!我对我的:username字段进行了一个小的验证，它应该是4到30个字符。我写了一个验证::length=>{:within=>4..30,:message=>I18n.t('activerecord.errors.range')-我想显示一个错误各种错误的消息(不像，太长或太短)，但这里有一个问题-我可以将最小值和最大值都传递给翻译，以便有类似的东西:用户名应该在4到30个字符之间。目前我有:range:"shouldbebetween%{count}and%{count}characters"，这显然不起作用(只是为了检查)。是否可以从范围中获取这些值？谢谢大家的指教!

  7. 区块链之加解密算法&数字证书 - 2

    目录一.加解密算法数字签名对称加密DES(DataEncryptionStandard)3DES(TripleDES)AES(AdvancedEncryptionStandard)RSA加密法DSA(DigitalSignatureAlgorithm)ECC(EllipticCurvesCryptography)非对称加密签名与加密过程非对称加密的应用对称加密与非对称加密的结合二.数字证书图解一.加解密算法加密简单而言就是通过一种算法将明文信息转换成密文信息，信息的的接收方能够通过密钥对密文信息进行解密获得明文信息的过程。根据加解密的密钥是否相同，算法可以分为对称加密、非对称加密、对称加密和非

  8. Ruby-vips 图像处理库。有什么好的使用示例吗？ - 2

    我对图像处理完全陌生。我对JPEG内部是什么以及它是如何工作一无所知。我想知道，是否可以在某处找到执行以下简单操作的ruby​​代码:打开jpeg文件。遍历每个像素并将其颜色设置为fx绿色。将结果写入另一个文件。我对如何使用ruby​​-vips库实现这一点特别感兴趣https://github.com/ender672/ruby-vips我的目标-学习如何使用ruby​​-vips执行基本的图像处理操作(Gamma校正、亮度、色调……)任何指向比“helloworld”更复杂的工作示例的链接——比如ruby​​-vips的github页面上的链接，我们将不胜感激!如果有ruby​​-

  9. ruby - Faye WebSocket，关闭处理程序被触发后重新连接到套接字 - 2

    我有一个super简单的脚本，它几乎包含了FayeWebSocketGitHub页面上用于处理关闭连接的内容:ws=Faye::WebSocket::Client.new(url,nil,:headers=>headers)ws.on:opendo|event|p[:open]#sendpingcommand#sendtestcommand#ws.send({command:'test'}.to_json)endws.on:messagedo|event|#hereistheentrypointfordatacomingfromtheserver.pJSON.parse(event.d

  10. ruby - 如何使用 Ruby HTTP::Net 处理 404 错误？ - 2

    我正在尝试解析网页，但有时会收到404错误。这是我用来获取网页的代码:result=Net::HTTP::getURI.parse(URI.escape(url))如何测试result是否为404错误代码？ 最佳答案 像这样重写你的代码:uri=URI.parse(url)result=Net::HTTP.start(uri.host,uri.port){|http|http.get(uri.path)}putsresult.codeputsresult.body这将打印状态码和正文。

随机推荐