k8s使用NFS作为sc

Posted on Edited on In Views:
k8s环境下学习很多应用服务会很方便，特别是集群搭建方便，高可用性强。搭建高可用的过程中，一定会用到持久化存储，比较常见的是通过NFS实现storageclasses。

准备sc

选择一个k8s节点安装nfs服务，作为sc的提供者

1
2
3
4
5
6
7
// 安装服务
# yum install nfs-utils rpcbind -y
# mkdir /data

// nfs配置
# cat /etc/exports
/data 192.168.111.0/24(rw,sync,insecure,no_subtree_check,no_root_squash) // 集群网络

启动服务

1
2
3
4
5
6
# systemctl start nfs && systemctl enable nfs
# systemctl start rpcbind && systemctl enable rpcbind

# showmount -e localhost
Export list for localhost:
/data 192.168.111.0/24

所有节点都安装nfs-common，没有安装时，nfs-storage-nfs-client-provisioner-dcf475cdb-22zm9状态无法正常启动，报错日志如下：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
Events:
  Type     Reason       Age               From               Message
  ----     ------       ----              ----               -------
  Normal   Scheduled    32s               default-scheduler  Successfully assigned default/nfs-storage-nfs-client-provisioner-dcf475cdb-z5jn6 to node02
  Warning  FailedMount  0s (x7 over 31s)  kubelet            MountVolume.SetUp failed for volume "nfs-client-root" : mount failed: exit status 32
Mounting command: mount
Mounting arguments: -t nfs 192.168.111.128:/data /var/lib/kubelet/pods/54c0d134-37bc-4661-a1a1-34691f914224/volumes/kubernetes.io~nfs/nfs-client-root
Output: mount: wrong fs type, bad option, bad superblock on 192.168.111.128:/data,
       missing codepage or helper program, or other error
       (for several filesystems (e.g. nfs, cifs) you might
       need a /sbin/mount.<type> helper program)

       In some cases useful info is found in syslog - try
       dmesg | tail or so.

1
yum install nfs-common  nfs-utils -y

手动安装sc的过程涉及权限相关，虽然不复杂，但是如果出错排查不方便，这里选择通过helm安装nfs的sc，方案更加成熟。

安装helm，helm安装

1
2
3
# wget https://get.helm.sh/helm-v3.9.0-linux-amd64.tar.gz
# tar zxvf helm-v3.9.0-linux-amd64.tar.gz
# cp helm /usr/local/bin/helm

增加helm库

1
2
3
4
5
6
# helm repo add azure http://mirror.azure.cn/kubernetes/charts/
"azure" has been added to your repositories
# helm search repo nfs
NAME                            CHART VERSION	APP VERSION	DESCRIPTION
azure/nfs-client-provisioner    1.2.11       	3.1.0      	DEPRECATED - nfs-client is an automatic provisi...
azure/nfs-server-provisioner    1.1.3        	2.3.0      	DEPRECATED - nfs-server-provisioner is an out-o...

安装helm

1
2
# helm pull azure/nfs-client-provisioner
# tar zxvf nfs-client-provisioner-1.2.11.tgz

修改变量

1
2
3
4
5
nfs:
     server: 192.168.111.128
  path: /data
storageClass:
     defaultClass: true

安装

1
# helm install nfs-storage -f values.yaml ./

查看结果

1
2
3
4
5
6
# kubectl get pod -o wide
NAME                                                 READY   STATUS    RESTARTS   AGE     IP            NODE     NOMINATED NODE   READINESS GATES
nfs-storage-nfs-client-provisioner-dcf475cdb-22zm9   1/1     Running   0          2m3s    10.244.2.21   node02   <none>           <none>
# kubectl get sc
NAME                   PROVISIONER                                        RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
nfs-client (default)   cluster.local/nfs-storage-nfs-client-provisioner   Delete          Immediate           true                   12m

踩坑

通过nfs的sc无法创建pvc，pod日志

1
E0710 06:40:34.902614       1 controller.go:1004] provision "default/mysql-pv-claim" class "nfs-client": unexpected error getting claim reference: selfLink was empty, can't make reference

解决方案:nfs-provisioner报错问题:”selfLink was empty”

1
2
3
# vim /etc/kubernetes/manifests/kube-apiserver.yaml
// 增加
    - --feature-gates=RemoveSelfLink=false　　#添加此行