k8s 部署 minio

Jerry00713 2023-04-15 原文

一、k8s部署老版本minio

老版本使用9000端口进行业务通信＋提供web页面＋ localhost:9000/minio/health/ready提供健康检查

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    name: minio
  name: minio
  namespace: armory
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 7
  selector:
    matchLabels:
      name: minio
  template:
    metadata:
      labels:
        app: minio
        name: minio
    spec:
      containers:
      - name: minio
        image: harbor.od.com:180/armory/minio:latest
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 9000
          protocol: TCP
        args:
        - server
        - /data
        env:
        - name: MINIO_ACCESS_KEY
          value: admin
        - name: MINIO_SECRET_KEY
          value: admin123
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /minio/health/ready
            port: 9000
            scheme: HTTP
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 5
        volumeMounts:
        - mountPath: /data
          name: data
      imagePullSecrets:
      - name: harbor
      volumes:
      - nfs:
          server: hdss7-200
          path: /data/nfs-volume/minio
        name: data

---

apiVersion: v1
kind: Service
metadata:
  name: minio
  namespace: armory
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 9000
  selector:
    app: minio

---

apiVersion: extensions/v1beta1
kind: Ingress
metadata: 
  name: minio
  namespace: armory
spec:
  rules:
  - host: minio.od.com
    http:
      paths:
      - path: /
        backend: 
          serviceName: minio
          servicePort: 80

二、新版本minio特性

2.1、新版本特性

１、新版本使用9000端口进行业务通信
２、新版本把可视化独立出来，默认用动态端口进行通信，或--console-address ":5000"　自己设置端口，但主动访问IP:9000 依旧是可以访问web页面
３、localhost:9000/minio/health/ready提供健康检查

2.2、使用老版本的配置清单引入的问题

假如使用了老版本的资源配置清单，使用新的minio镜像，会出现：

1、访问容器IP:9000是依旧可以访问web页面
2、但是如果使用ingress域名访问，比如是minio.od.com，会出现页面返回的是minio.od.com:32154，并且页面无任何数据。
3、如果直接访问容器IP:32154，可以直接访问web页面
4、通过kubectl describe pod的名字，会发现有一条是Console，端口是32154
5、总上所述，对比新版本minio特性，说明了32154是Console的端口，而且在kill掉这个pod，在启动一个新的，发现Console端口会变化，说明Console端口是随机的。
6、按照次特性，可以修改deployment资源，增加Nodename固定在那个节点运行，修改svc资源设置nodeport，共享网络名称空间，让pod的9000端口映射宿主机的某一个端口，直接访问宿主机的端口，通过即可。

2.3、新版本如何编写资源配置清单

1、deployment 中需要使用命令固定Console端口，deployment 开放9000端口、Console端口
2、service 资源中代理9000端口、Console端口
3、ingress 资源中代理Console端口

三、k8s部署新版本minio

3.1、docker 部署minio

docker run -p 9000:9000 -p 9090:9090 --name minio \
-d --restart=always \
-e "MINIO_ACCESS_KEY=admin" \
-e "MINIO_SECRET_KEY=admin123456" \
-v /data/minio/data:/data \
-v /data/minio/config:/root/.minio \minio/minio \
server /data --console-address ":9090"

3.2、k8s 部署版本minio

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: minio
spec:
  replicas: 1
  selector:
    matchLabels:
      app: minio
  template:
    metadata:
      labels:
        app: minio
    spec:
      containers:
      - name: minio
        env:
        - name: MINIO_ROOT_USER
          value: "admin"
        - name: MINIO_ROOT_PASSWORD
          value: "root123456"
        image: minio/minio:latest
        imagePullPolicy: IfNotPresent
        command:
          - /bin/sh
          - -c
          - minio server /data --console-address ":5000"
        ports:
        - name: data
          containerPort: 9000
          protocol: "TCP"
        - name: console
          containerPort: 5000
          protocol: "TCP"

---

apiVersion: v1
kind: Service
metadata:
  name: minio
spec:
  ports:
  - name: data
    port: 9000
    targetPort: 9000
    protocol: TCP
  - name: console
    port: 5000
    targetPort: 5000
    protocol: TCP
  selector:
    app: minio

---

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: minio
spec:
  rules:
  - host: minio.od.com
    http:
      paths:
        - path: /
          backend:
            serviceName: minio
            servicePort: 5000

注释：

1、是MINIO_ROOT_USER、 MINIO_ROOT_PASSWORD 还是MINIO_ACCESS_KEY、MINIO_SECRET_KEY 具体没研究过，可以去官网看，或者直接容器启动后，查看环境变量。MINIO_ROOT_PASSWORD不得少于8位
2、修改容器启动的第一命令/bin/sh -c minio server /data --console-address ":5000"，使其固定Console端口5000

应用配置清单后，配置访问minio.od.com，流量可以调度给ingress-controller即可

3.3、k8s 部署版本minio集群

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: minio
spec:
  serviceName: minio
  replicas: 4
  selector:
    matchLabels:
      app: minio
  template:
    metadata:
      labels:
        app: minio
    spec:
      containers:
      - name: minio
        env:
        - name: MINIO_ROOT_USER
          value: "admin"
        - name: MINIO_ROOT_PASSWORD
          value: "root123456"
        image: minio/minio
        imagePullPolicy: IfNotPresent
        command:
          - /bin/sh
          - -c
          - minio server --console-address ":5000" http://minio-{0...3}.minio.default.svc.cluster.local/data
        ports:
        - name: data
          containerPort: 9000
          protocol: "TCP"
        - name: console
          containerPort: 5000
          protocol: "TCP"
        volumeMounts:
        - name: data
          mountPath: /data
        - name: date-config
          mountPath: /etc/localtime
      volumes:
        - name: date-config
          hostPath:
            path: /usr/share/zoneinfo/Asia/Shanghai
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes:
        - ReadWriteMany
      resources:
        requests:
          storage: 10Gi
      storageClassName: little

---

apiVersion: v1
kind: Service
metadata:
  name: minio
  labels:
    app: minio
spec:
  clusterIP: None
  ports:
    - port: 9000
      name: data
    - port: 5000
      name: console
  selector:
    app: minio
---
apiVersion: v1
kind: Service
metadata:
  name: minio-service
spec:
  type: NodePort
  ports:
   - name: data
     port: 9000
     targetPort: 9000
     protocol: TCP
     nodePort:
   - name: console
     port: 5000
     targetPort: 5000
     protocol: TCP
     nodePort:
  selector:
    app: minio


---

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: minio
  namespace: default
spec:
  rules:
    - host: minio.od.com
      http:
        paths:
          - path: /
            backend:
              serviceName: minio-service
              servicePort: 5000