ceph 与 kubernetes 集成

  • ceph 与 kubernetes 集成
    • ceph 与 kubernetes 集成概述
    • ceph 与 volumes 结合
    • ceph 与 pv pvc 结合
    • ceph 与 storageClass 结合

1.0 ceph 与 kubernetes 集成

1.1 ceph 与 kubernetes 集成概述

  • 在 K8S 集群中，比较麻烦的是: 有状态服务持久化，并且保证高可用，面临的问题:
      1. 有状态服务 Pod 消亡时，持久化数据能够立刻转移到新拉起的 Pod
      1. 持久化数据本身读写的高可用
  • ceph 分布式存储就是解决问题2。同时使用独立的分布式存储集群，分离 PV 与 工作节点的耦合，确保 PV 能够转移给新的 Pod，解决问题1
  • kubernetes 和 ceph 集成提供了三种实现方式:

2.0 ceph 与 volumes 结合

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
// 所有 k8s 节点都需要安装
# yum -y install ceph-common

// 创建 pool
[root@ceph-node01 ~]# ceph osd pool create kubernetes 8 8
[root@ceph-node01 ~]# ceph osd pool application enable kubernetes rbd
[root@ceph-node01 ~]# ceph osd pool application get kubernetes
[root@ceph-node01 ~]# ceph osd lspools

// 创建用户
[root@ceph-node01 ~]# ceph auth get-or-create client.kubernetes mon 'profile rbd' osd 'profile rbd pool=kubernetes'
[client.kubernetes]
	key = AQBblHdormInORAAp3q8n66Pj4f9Gtn0DLYihw==

// 将生成的 key 使用 base64 加密
[root@ceph-node01 ~]# echo 'AQBblHdormInORAAp3q8n66Pj4f9Gtn0DLYihw==' | base64
QVFCYmxIZG9ybUluT1JBQXAzcThuNjZQajRmOUd0bjBETFlpaHc9PQo=

2.1 创建 secrets 对象

1
2
3
4
5
6
7
8
9
10
11
[root@k8s-master01 ~]# cat ceph-secret.yaml 
apiVersion: v1
kind: Secret
metadata:
  name: ceph-secret
type: "kubernetes.io/rbd"
data:
  key: QVFCYmxIZG9ybUluT1JBQXAzcThuNjZQajRmOUd0bjBETFlpaHc9PQo=

[root@k8s-master01 ~]# kubectl apply -f ceph-secret.yaml 
[root@k8s-master01 ~]# kubectl get secret

2.2 创建 rbd 块

1
2
3
4
5
6
7
8
[root@ceph-node01 ~]# rbd create kubernetes/volumes-rbd.img --size 10G
[root@ceph-node01 ~]# rbd info kubernetes/volumes-rbd.img
[root@ceph-node01 ~]# rbd -p kubernetes ls

// 去掉它的不支持的相关特性
[root@ceph-node01 ~]# rbd feature disable kubernetes/volumes-rbd.img deep-flatten
[root@ceph-node01 ~]# rbd feature disable kubernetes/volumes-rbd.img fast-diff
[root@ceph-node01 ~]# rbd feature disable kubernetes/volumes-rbd.img exclusive-lock
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
// pod 中引用 RBD volumes
[root@k8s-master01 ~]# cat rbd-pod.yaml 
apiVersion: v1
kind: Pod
metadata:
  name: volume-rbd-pod
spec:
  containers:
    - name: pod-rbd-rw
      image: nginx:1.18
      imagePullPolicy: IfNotPresent
      ports:
        - name: www
          containerPort: 80
          protocol: TCP
      volumeMounts:
        - name: volume-rbd
          mountPath: /data
  volumes:
    - name: volume-rbd
      rbd:
        monitors:
          - '10.16.41.136:6789'
          - '10.16.41.30:6789'
          - '10.16.41.169:6789'
        pool: kubernetes
        image: volumes-rbd.img
        fsType: ext4
        user: kubernetes
        secretRef:
          name: ceph-secret

// 查看pod所在节点的挂载情况
[root@k8s-master01 ~]# kubectl get pod -o wide                # 查看 pod 落在那个节点上
[root@k8s-node03 ~]# rbd showmapped                              # 在所在节点查看挂载情况

upload successful

upload successful

3.0 ceph 与 pv pvc 结合

  • 预先将存储空间划分好
  • pool、镜像、用户认证、secrets 参考 2.0

3.1 创建 pv、pvc

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
// 创建块
[root@ceph-node01 ~]# rbd create kubernetes/ceph-pv-rbd.img --image-feature layering --size 10G
[root@ceph-node01 ~]# rbd -p kubernetes ls
[root@k8s-master01 ceph-kubernetes]# cat ceph-pvpvc-rbd.yaml 
apiVersion: v1
kind: PersistentVolume
metadata:
  name: ceph-pv-rbd
spec:
  accessModes:
    - ReadWriteOnce
  capacity:
    storage: 10G
  rbd:
    monitors:
      - '10.16.41.136:6789'
      - '10.16.41.30:6789'
      - '10.16.41.169:6789'
    pool: kubernetes
    image: ceph-pv-rbd.img
    fsType: ext4
    user: kubernetes
    secretRef:
      name: ceph-secret
  persistentVolumeReclaimPolicy: Retain
  storageClassName: rbd

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: ceph-pvc-rbd
spec:
  accessModes:
    - ReadWriteOnce
  volumeName: ceph-pv-rbd
  resources:
    requests:
      storage: 10G
  storageClassName: rbd

[root@k8s-master01 ceph-kubernetes]# kubectl apply -f ceph-pvpvc-rbd.yaml
[root@k8s-master01 ceph-kubernetes]# kubectl get pv,pvc

// 创建的 pv-rbd.img 不支持的功能需要取消。创建时可以直接加参数"--image-feature layering
"则不需要操作取消的相关命令了
[root@ceph-node01 ~]# rbd info kubernetes/ceph-pv-rbd.img
[root@ceph-node01 ~]# ceph osd pool application enable kubernetes rbd            # 不启用会报错，根据提示输入命令 <volumes 已经启用了，此处不再需要 enable>

upload successful

3.2 容器引用 pvc 存储

  • 容器通过 pvc 调用存储
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
// 需要确认 kubernetes/ceph-pv-rbd.img 是否为 "features: layering"
[root@k8s-master01 ceph-kubernetes]# cat ceph-pvpvc-pod.yaml 
apiVersion: v1
kind: Pod
metadata:
  name: pvc-pod
spec:
  containers:
    - name: ceph-pvc-demo
      image: nginx:1.18
      imagePullPolicy: IfNotPresent
      ports:
        - name: www
          protocol: TCP
          containerPort: 80
      volumeMounts:
        - name: ceph-pvc-rbd
          mountPath: /data
  volumes:
    - name: ceph-pvc-rbd
      persistentVolumeClaim:
        claimName: ceph-pvc-rbd			# 实际PVC名称

[root@k8s-master01 ceph-kubernetes]# kubectl apply -f ceph-pvpvc-pod.yaml

upload successful

4.0 ceph 与 storageClass 结合

  • 准备工作: pool 创建、用户创建 <2.0已经创建好了>

upload successful

4.1 ceph csi 驱动安装

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
// 生成 ceph-csi configmap 文件
[root@ceph-node01 ~]# ceph mon dump                 # 获取 fsid: 9504ad3b-d89d-4bba-9bd2-94be5c8d3f75
[root@k8s-master01 ceph-kubernetes]# cat csi-config-map.yaml 
apiVersion: v1
kind: ConfigMap
metadata:
  name: ceph-csi-config
data:
  config.json: |-
    [
      {
        "clusterID": "9504ad3b-d89d-4bba-9bd2-94be5c8d3f75",
        "monitors": [
          "10.16.41.136:6789",
          "10.16.41.30:6789",
          "10.16.41.169:6789"
        ]
      }
    ]

[root@k8s-master01 ceph-kubernetes]# kubectl apply -f csi-config-map.yaml 
[root@k8s-master01 ceph-kubernetes]# kubectl get cm
4.1.1 认证 key 配置
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
[root@ceph-node01 ~]# ceph auth get client.kubernetes                    # 查询 key
[client.kubernetes]
	key = AQBblHdormInORAAp3q8n66Pj4f9Gtn0DLYihw==
	caps mon = "profile rbd"
	caps osd = "profile rbd pool=kubernetes"

[root@k8s-master01 ceph-kubernetes]# cat csi-rbd-secret.yaml 
apiVersion: v1
kind: Secret
metadata:
  name: csi-rbd-secret
  namespace: default
stringData:
  userID: kubernetes
  userKey: AQBblHdormInORAAp3q8n66Pj4f9Gtn0DLYihw==

[root@k8s-master01 ceph-kubernetes]# kubectl apply -f csi-rbd-secret.yaml
[root@k8s-master01 ceph-kubernetes]# kubectl get secret

upload successful

4.1.2 创建 sa 和 rbac

upload successful

  • 上面 6 个文件中,只有 “csi-rbdplugin.yaml” 文件需要添加几行内容,如下。
csi-rbdplugin.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
[root@k8s-master01 csi]# cat csi-rbdplugin.yaml 
---
kind: DaemonSet
apiVersion: apps/v1
metadata:
  name: csi-rbdplugin
  # replace with non-default namespace name
  namespace: default
spec:
  selector:
    matchLabels:
      app: csi-rbdplugin
  template:
    metadata:
      labels:
        app: csi-rbdplugin
    spec:
      tolerations:				# 解决 master 不创建 node 的问题
        - key: node-role.kubernetes.io/master
          effect: NoSchedule
      serviceAccountName: rbd-csi-nodeplugin
      hostNetwork: true
      hostPID: true
      priorityClassName: system-node-critical
......
1
2
3
[root@k8s-master01 csi]# ls
ceph-conf.yaml  csi-nodeplugin-rbac.yaml  csi-provisioner-rbac.yaml  csi-rbdplugin-provisioner.yaml  csi-rbdplugin.yaml  kms-config.yaml
[root@k8s-master01 csi]# kubectl apply -f .

upload successful

4.2 pvc 动态申请空间

  • Kubernetes StorageClass定义了一类存储。可以创建多个 StorageClass 对象来映射到不同的服务质量级别（即 NVMe 与基于 HDD 的池）和功能
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
[root@ceph-node01 ~]# ceph -s
[root@k8s-master01 csi]# cat csi-rbd-sc.yaml 
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
   name: csi-rbd-sc
provisioner: rbd.csi.ceph.com
parameters:
   clusterID: 9504ad3b-d89d-4bba-9bd2-94be5c8d3f75                        # ceph -s 查看集群 ID
   pool: kubernetes                                                       # 访问什么 pool
   imageFeatures: layering
   csi.storage.k8s.io/provisioner-secret-name: csi-rbd-secret             # 什么 key
   csi.storage.k8s.io/provisioner-secret-namespace: default
   csi.storage.k8s.io/controller-expand-secret-name: csi-rbd-secret
   csi.storage.k8s.io/controller-expand-secret-namespace: default               # 空间
   csi.storage.k8s.io/node-stage-secret-name: csi-rbd-secret
   csi.storage.k8s.io/node-stage-secret-namespace: default
reclaimPolicy: Delete
allowVolumeExpansion: true
mountOptions:
   - discard

[root@k8s-master01 csi]# kubectl apply -f csi-rbd-sc.yaml 
[root@k8s-master01 csi]# kubectl get storageclass
    # 名称: csi-rbd-sc
    # 驱动: rbd.csi.ceph.com
    # 策略: Delete

upload successful

upload successful

4.2.1 块存储
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
[root@k8s-master01 ceph-kubernetes]# cat raw-block-pvc.yaml 
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: raw-block-pvc
spec:
  accessModes:
    - ReadWriteOnce
  volumeMode: Block
  resources:
    requests:
      storage: 5Gi
  storageClassName: csi-rbd-sc

---

apiVersion: v1
kind: Pod
metadata:
  name: pod-with-raw-block-volume
spec:
  containers:
    - name: fc-container
      image: fedora:26
      command: ["/bin/sh", "-c"]
      args: ["tail -f /dev/null"]
      volumeDevices:
        - name: data
          devicePath: /dev/xvda
  volumes:
    - name: data
      persistentVolumeClaim:
        claimName: raw-block-pvc

[root@k8s-master01 ceph-kubernetes]# kubectl apply -f raw-block-pvc.yaml
4.2.2 文件系统
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
[root@k8s-master01 ceph-kubernetes]# cat fs-pvc.yaml 
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: rbd-pvc
spec:
  storageClassName: csi-rbd-sc
  accessModes:
    - ReadWriteOnce
  volumeMode: Filesystem
  resources:
    requests:
      storage: 5Gi
---
apiVersion: v1
kind: Pod
metadata:
  name: csi-rbd-demo-pod
spec:
  containers:
    - name: web-server
      image: nginx
      volumeMounts:
        - name: mypvc
          mountPath: /var/lib/www/html
  volumes:
    - name: mypvc
      persistentVolumeClaim:
        claimName: rbd-pvc
        readOnly: false

[root@k8s-master01 ceph-kubernetes]# kubectl apply -f fs-pvc.yaml

upload successful

4.2.3 sts 的 pvc
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
[root@k8s-master01 ceph-kubernetes]# cat nginx.yaml 
apiVersion: v1
kind: Service
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  ports:
    - port: 80
      name: web
  clusterIP: None
  selector:
    app: nginx
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: web
spec:
  selector:
    matchLabels:
      app: nginx
  serviceName: "nginx"
  replicas: 3
  template:
    metadata:
      labels:
        app: nginx
    spec:
      terminationGracePeriodSeconds: 10
      containers:
        - name: nginx
          image: nginx
          ports:
            - containerPort: 80
              name: web
          volumeMounts:
            - name: www
              mountPath: /usr/share/nginx/html
  volumeClaimTemplates:
    - metadata:
        name: www
      spec:
        accessModes: [ "ReadWriteOnce" ]
        storageClassName: "csi-rbd-sc"
        resources:
          requests:
            storage: 6Gi

[root@k8s-master01 ceph-kubernetes]# kubectl apply -f nginx.yaml 
[root@k8s-master01 ceph-kubernetes]# kubectl get pod,pv,pvc -o wide

upload successful

微信

加微信,入技术群

微信

扫一扫，分享到微信

tag:

    缺失模块。
    1、请确保node版本大于6.2
    2、在博客根目录（注意不是yilia-plus根目录）执行以下命令：
    npm i hexo-generator-json-content --save

    3、在根目录_config.yml里添加配置： 

      jsonContent:
    meta: false
    pages: false
    posts:
      title: true
      date: true
      path: true
      text: false
      raw: false
      content: false
      slug: false
      updated: false
      comments: false
      link: false
      permalink: false
      excerpt: false
      categories: false
      tags: true

运维技术面：
1. Linux 基础、脚本、服务部署、虚拟化
2. Linux 自动化、服务器硬件
3. Kubernetes、kube 服务部署
4. Windows 客户端,服务端的便捷应用

微信： wnn_chat

写博客的动力,源自于个人爱好!