6、kubernetes（k8s）安装

本文内容以语雀为准

文档

1. 等等，Docker 被 Kubernetes 弃用了?
2. 容器运行时
3. 端口和协议
4. kubeadm init
5. kubeadm config
6. 安装网络策略驱动
7. 使用 kubeadm 创建集群
   1. 控制平面节点隔离
8. 持久卷
9. 为容器设置环境变量
10. 在CentOS上安装Docker引擎
11. Pod 网络无法访问排查处理

说明

1. 本文以 CentOS 7.9、k8s 1.25.3（文章首次发布于2022-10-30，是当时的最新版）为例
2. 本文固定了 k8s 的版本，防止不同版本存在差异，当你了解了某一版本的安装与使用，自己就可以尝试其他版本的安装了
3. 2022-11-18，经过测试，当前时间的最新版：1.25.4，同样适用于本文章
4. 由于 k8s 1.24 及之后的版本使用的是 containerd，之前的版本是 docker，故此文都安装并配置了，可以修改 k8s 的版本号进行学习、测试。
   | | 控制面板 | node 节点 |
   | — | — | — |
   | 主机名 | k8s | node-1 |
   | IP | 192.168.80.60 | 192.168.80.16 |

安装

1. 安装所需工具

sudo yum -y install vim
sudo yum -y install wget

2. 将主机名指向本机IP，主机名只能包含：字母、数字、-（横杠）、.（点）
   1. 获取主机名

hostname

2. 临时设置主机名

hostname 主机名

3. 永久设置主机名

sudo echo '主机名' > /etc/hostname

4. 编辑 hosts

sudo vim /etc/hosts

控制面板：设置IP

192.168.80.60 	k8s

node 节点：设置IP

192.168.80.16 	node-1

3. 安装并配置 ntpdate，同步时间

sudo yum -y install ntpdate
sudo ntpdate ntp1.aliyun.com
sudo systemctl status ntpdate
sudo systemctl start ntpdate
sudo systemctl status ntpdate
sudo systemctl enable ntpdate

4. 安装并配置 bash-completion，添加命令自动补充

sudo yum -y install bash-completion
source /etc/profile

5. 关闭防火墙、或者开通指定端口

sudo systemctl stop firewalld.service 
sudo systemctl disable firewalld.service

# 控制面板
firewall-cmd --zone=public --add-port=6443/tcp --permanent # Kubernetes API server	所有
firewall-cmd --zone=public --add-port=2379/tcp --permanent # etcd server client API	kube-apiserver, etcd
firewall-cmd --zone=public --add-port=2380/tcp --permanent # etcd server client API	kube-apiserver, etcd
firewall-cmd --zone=public --add-port=10250/tcp --permanent # Kubelet API	自身, 控制面
firewall-cmd --zone=public --add-port=10259/tcp --permanent # kube-scheduler	自身
firewall-cmd --zone=public --add-port=10257/tcp --permanent # kube-controller-manager	自身
firewall-cmd --zone=trusted --add-source=192.168.80.60 --permanent # 信任集群中各个节点的IP
firewall-cmd --zone=trusted --add-source=192.168.80.16 --permanent # 信任集群中各个节点的IP
firewall-cmd --add-masquerade --permanent # 端口转发
firewall-cmd --reload
firewall-cmd --list-all
firewall-cmd --list-all --zone=trusted

# 工作节点
firewall-cmd --zone=public --add-port=10250/tcp --permanent # Kubelet API	自身, 控制面
firewall-cmd --zone=public --add-port=30000-32767/tcp --permanent # NodePort Services?	所有
firewall-cmd --zone=trusted --add-source=192.168.80.60 --permanent # 信任集群中各个节点的IP
firewall-cmd --zone=trusted --add-source=192.168.80.16 --permanent # 信任集群中各个节点的IP
firewall-cmd --add-masquerade --permanent # 端口转发
firewall-cmd --reload
firewall-cmd --list-all
firewall-cmd --list-all --zone=trusted

6. 关闭交换空间

sudo swapoff -a
sudo sed -i 's/.*swap.*/#&/' /etc/fstab

7. 关闭 selinux

getenforce
cat /etc/selinux/config
sudo setenforce 0
sudo sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config
cat /etc/selinux/config

8. 安装 Containerd、Docker
   Docker 不是必须的，k8s 1.24.0 开始使用 Containerd 替代 Docker，但还是推荐安装 Docker，原因：在k8s中构建Docker镜像时使用，需要在GitLab Runner 中配置如下，详情参见：https://www.yuque.com/xuxiaowei-com-cn/gitlab-k8s/gitlab-runner-k8s

[[runners]]
  ...
  [runners.kubernetes]
    ...
    [runners.kubernetes.volumes]
    [[runners.kubernetes.volumes.host_path]]
      name = "docker"
      mount_path = "/var/run/docker.sock"
      host_path = "/var/run/docker.sock"

/etc/containerd/config.toml 中的 SystemdCgroup = true 的优先级高于 /etc/docker/daemon.json 中的 cgroupdriver

# https://docs.docker.com/engine/install/centos/
sudo yum remove docker docker-client docker-client-latest docker-common docker-latest docker-latest-logrotate docker-logrotate docker-engine
sudo yum install -y yum-utils device-mapper-persistent-data lvm2
sudo yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo 
# yum --showduplicates list docker-ce
sudo yum install -y docker-ce docker-ce-cli containerd.io docker-compose-plugin
sudo yum install -y containerd

# 启动 docker 时，会启动 containerd
# sudo systemctl status containerd.service
sudo systemctl stop containerd.service

sudo cp /etc/containerd/config.toml /etc/containerd/config.toml.bak
sudo containerd config default > $HOME/config.toml
sudo cp $HOME/config.toml /etc/containerd/config.toml
# 修改 /etc/containerd/config.toml 文件后，要将 docker、containerd 停止后，再启动
sudo sed -i "s#registry.k8s.io/pause#registry.cn-hangzhou.aliyuncs.com/google_containers/pause#g" /etc/containerd/config.toml
# https://kubernetes.io/zh-cn/docs/setup/production-environment/container-runtimes/#containerd-systemd
# 确保 /etc/containerd/config.toml 中的 disabled_plugins 内不存在 cri
sudo sed -i "s#SystemdCgroup = false#SystemdCgroup = true#g" /etc/containerd/config.toml

sudo systemctl enable --now containerd.service
# sudo systemctl status containerd.service

# sudo systemctl status docker.service
sudo systemctl start docker.service
# sudo systemctl status docker.service
sudo systemctl enable docker.service
sudo systemctl enable docker.socket
sudo systemctl list-unit-files | grep docker

sudo mkdir -p /etc/docker

sudo tee /etc/docker/daemon.json <<-'EOF'
{
  "registry-mirrors": ["https://hnkfbj7x.mirror.aliyuncs.com"],
    "exec-opts": ["native.cgroupdriver=systemd"]
}
EOF

sudo systemctl daemon-reload
sudo systemctl restart docker
sudo docker info

sudo systemctl status docker.service

sudo systemctl status containerd.service

9. 添加阿里云 k8s 镜像仓库

cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
# 是否开启本仓库
enabled=1
# 是否检查 gpg 签名文件
gpgcheck=0
# 是否检查 gpg 签名文件
repo_gpgcheck=0
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

10. 安装 k8s 1.25.3 所需依赖

# 设置所需的 sysctl 参数，参数在重新启动后保持不变
cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1
EOF

# 应用 sysctl 参数而不重新启动
sudo sysctl --system

# yum --showduplicates list kubelet --nogpgcheck
# yum --showduplicates list kubeadm --nogpgcheck
# yum --showduplicates list kubectl --nogpgcheck

# 2023-02-07，经过测试，版本号：1.24.0，同样适用于本文章
# sudo yum install -y kubelet-1.24.0-0 kubeadm-1.24.0-0 kubectl-1.24.0-0 --disableexcludes=kubernetes --nogpgcheck

# 如果你看到有人说 node 节点不需要安装 kubectl，其实这种说法是错的，kubectl 会被当做依赖安装，如果安装过程没有指定 kubectl 的版本，则会安装最新版的 kubectl，可能会导致程序运行异常
sudo yum install -y kubelet-1.25.3-0 kubeadm-1.25.3-0 kubectl-1.25.3-0 --disableexcludes=kubernetes --nogpgcheck

# 2022-11-18，经过测试，版本号：1.25.4，同样适用于本文章
# sudo yum install -y kubelet-1.25.4-0 kubeadm-1.25.4-0 kubectl-1.25.4-0 --disableexcludes=kubernetes --nogpgcheck

# 2023-02-07，经过测试，版本号：1.25.5，同样适用于本文章
# sudo yum install -y kubelet-1.25.5-0 kubeadm-1.25.5-0 kubectl-1.25.5-0 --disableexcludes=kubernetes --nogpgcheck

# 2023-02-07，经过测试，版本号：1.25.6，同样适用于本文章
# sudo yum install -y kubelet-1.25.6-0 kubeadm-1.25.6-0 kubectl-1.25.6-0 --disableexcludes=kubernetes --nogpgcheck

# 2023-02-07，经过测试，版本号：1.26.0，同样适用于本文章
# sudo yum install -y kubelet-1.26.0-0 kubeadm-1.26.0-0 kubectl-1.26.0-0 --disableexcludes=kubernetes --nogpgcheck

# 2023-02-07，经过测试，版本号：1.26.1，同样适用于本文章
# sudo yum install -y kubelet-1.26.1-0 kubeadm-1.26.1-0 kubectl-1.26.1-0 --disableexcludes=kubernetes --nogpgcheck

# 安装最新版，生产时不建议
# sudo yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes --nogpgcheck

systemctl daemon-reload
sudo systemctl restart kubelet
sudo systemctl enable kubelet

11. 查看kubelet日志

# k8s 未初始化时，kubelet 可能无法启动
journalctl -xefu kubelet

12. 查看kubelet状态

# k8s 未初始化时，kubelet 可能无法启动
sudo systemctl status kubelet

13. 已上命令需要在控制面板与node节点执行，并确保没有错误与警告

已上命令需要在控制面板与node节点执行，并确保没有错误与警告
已上命令需要在控制面板与node节点执行，并确保没有错误与警告

14. 控制面板：初始化

kubeadm init --image-repository=registry.aliyuncs.com/google_containers
# 指定集群的IP
# kubeadm init --image-repository=registry.aliyuncs.com/google_containers --apiserver-advertise-address=192.168.80.60

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

kubectl cluster-info

# 初始化失败后，可进行重置，重置命令：kubeadm reset

# 执行成功后，会出现类似下列内容：
# kubeadm join 192.168.80.60:6443 --token f9lvrz.59mykzssqw6vjh32 \
# --discovery-token-ca-cert-hash sha256:4e23156e2f71c5df52dfd2b9b198cce5db27c47707564684ea74986836900107 	

#
# kubeadm token create --print-join-command

15. node 节点：加入集群

# 运行的内容来自上方执行结果
kubeadm join 192.168.80.60:6443 --token f9lvrz.59mykzssqw6vjh32 \
--discovery-token-ca-cert-hash sha256:4e23156e2f71c5df52dfd2b9b198cce5db27c47707564684ea74986836900107 

#
# kubeadm token create --print-join-command

# kubeadm join 192.168.80.60:6443 --token f9lvrz.59mykzssqw6vjh32 \
# --discovery-token-unsafe-skip-ca-verification

16. 控制面板：

kubectl get pods --all-namespaces -o wide

可以查看到 coredns-* 的状态是 Pending，nodes 为 NotReady，原因是网络还未配置

[root@k8s ~]# kubectl get pods --all-namespaces -o wide
NAMESPACE     NAME                          READY   STATUS    RESTARTS   AGE     IP              NODE            NOMINATED NODE   READINESS GATES
kube-system   coredns-c676cc86f-4lncg       0/1     Pending   0          3m19s   <none>          <none>          <none>           <none>
kube-system   coredns-c676cc86f-7n9wv       0/1     Pending   0          3m19s   <none>          <none>          <none>           <none>
kube-system   etcd-k8s                      1/1     Running   0          3m26s   192.168.80.60   k8s             <none>           <none>
kube-system   kube-apiserver-k8s            1/1     Running   0          3m23s   192.168.80.60   k8s             <none>           <none>
kube-system   kube-controller-manager-k8s   1/1     Running   0          3m23s   192.168.80.60   k8s             <none>           <none>
kube-system   kube-proxy-87lx5              1/1     Running   0          81s     192.168.0.18    centos-7-9-16   <none>           <none>
kube-system   kube-proxy-rctn6              1/1     Running   0          3m19s   192.168.80.60   k8s             <none>           <none>
kube-system   kube-scheduler-k8s            1/1     Running   0          3m23s   192.168.80.60   k8s             <none>           <none>
[root@k8s ~]#

kubectl get nodes -o wide

[root@k8s ~]# kubectl get nodes -o wide
NAME            STATUS     ROLES           AGE     VERSION   INTERNAL-IP     EXTERNAL-IP   OS-IMAGE                KERNEL-VERSION           CONTAINER-RUNTIME
centos-7-9-16   NotReady   <none>          7m58s   v1.25.3   192.168.0.18    <none>        CentOS Linux 7 (Core)   3.10.0-1160.el7.x86_64   containerd://1.6.9
k8s             NotReady   control-plane   10m     v1.25.3   192.168.80.60   <none>        CentOS Linux 7 (Core)   3.10.0-1160.el7.x86_64   containerd://1.6.9
[root@k8s ~]#

17. 控制面板：配置网络，选择 Calico 配置
    | Kubernetes 版本 | Calico 版本 | Calico 文档 | |
    | — | — | — | — |
    | 1.18、1.19、1.20 | 3.18 | https://projectcalico.docs.tigera.io/archive/v3.18/getting-started/kubernetes/requirements | https://projectcalico.docs.tigera.io/archive/v3.18/manifests/calico.yaml |
    | 1.19、1.20、1.21 | 3.19 | https://projectcalico.docs.tigera.io/archive/v3.19/getting-started/kubernetes/requirements | https://projectcalico.docs.tigera.io/archive/v3.19/manifests/calico.yaml |
    | 1.19、1.20、1.21 | 3.20 | https://projectcalico.docs.tigera.io/archive/v3.20/getting-started/kubernetes/requirements | https://projectcalico.docs.tigera.io/archive/v3.20/manifests/calico.yaml |
    | 1.20、1.21、1.22 | 3.21 | https://projectcalico.docs.tigera.io/archive/v3.21/getting-started/kubernetes/requirements | https://projectcalico.docs.tigera.io/archive/v3.21/manifests/calico.yaml |
    | 1.21、1.22、1.23 | 3.22 | https://projectcalico.docs.tigera.io/archive/v3.22/getting-started/kubernetes/requirements | https://projectcalico.docs.tigera.io/archive/v3.22/manifests/calico.yaml |
    | 1.21、1.22、1.23 | 3.23 | https://projectcalico.docs.tigera.io/archive/v3.23/getting-started/kubernetes/requirements | https://projectcalico.docs.tigera.io/archive/v3.23/manifests/calico.yaml |
    | 1.22、1.23、1.24 | 3.24 | https://projectcalico.docs.tigera.io/archive/v3.24/getting-started/kubernetes/requirements | https://projectcalico.docs.tigera.io/archive/v3.24/manifests/calico.yaml |
    | 1.22、1.23、1.24 | 3.25 | https://projectcalico.docs.tigera.io/archive/v3.25/getting-started/kubernetes/requirements | https://projectcalico.docs.tigera.io/archive/v3.25/manifests/calico.yaml |

# 下载
wget --no-check-certificate https://projectcalico.docs.tigera.io/archive/v3.25/manifests/calico.yaml

# 修改 calico.yaml 文件
vim calico.yaml

# 在 - name: CLUSTER_TYPE 下方添加如下内容
- name: CLUSTER_TYPE
  value: "k8s,bgp"
  # 下方为新增内容
- name: IP_AUTODETECTION_METHOD
  value: "interface=网卡名称"

# 配置网络
kubectl apply -f calico.yaml

18. 控制面板：查看 pods、nodes

kubectl get nodes -o wide

[root@k8s ~]# kubectl get nodes -o wide
NAME            STATUS     ROLES           AGE     VERSION   INTERNAL-IP     EXTERNAL-IP   OS-IMAGE                KERNEL-VERSION           CONTAINER-RUNTIME
centos-7-9-16   NotReady   <none>          7m58s   v1.25.3   192.168.0.18    <none>        CentOS Linux 7 (Core)   3.10.0-1160.el7.x86_64   containerd://1.6.9
k8s             NotReady   control-plane   10m     v1.25.3   192.168.80.60   <none>        CentOS Linux 7 (Core)   3.10.0-1160.el7.x86_64   containerd://1.6.9
[root@k8s ~]# 

kubectl get pods --all-namespaces -o wide

[root@k8s ~]# kubectl get pods --all-namespaces -o wide
NAMESPACE     NAME                                      READY   STATUS     RESTARTS   AGE     IP              NODE            NOMINATED NODE   READINESS GATES
kube-system   calico-kube-controllers-f79f7749d-rkqgw   0/1     Pending    0          11s     <none>          <none>          <none>           <none>
kube-system   calico-node-7698p                         0/1     Init:0/3   0          11s     192.168.80.60   k8s             <none>           <none>
kube-system   calico-node-tvhnb                         0/1     Init:0/3   0          11s     192.168.0.18    centos-7-9-16   <none>           <none>
kube-system   coredns-c676cc86f-4lncg                   0/1     Pending    0          8m14s   <none>          <none>          <none>           <none>
kube-system   coredns-c676cc86f-7n9wv                   0/1     Pending    0          8m14s   <none>          <none>          <none>           <none>
kube-system   etcd-k8s                                  1/1     Running    0          8m21s   192.168.80.60   k8s             <none>           <none>
kube-system   kube-apiserver-k8s                        1/1     Running    0          8m18s   192.168.80.60   k8s             <none>           <none>
kube-system   kube-controller-manager-k8s               1/1     Running    0          8m18s   192.168.80.60   k8s             <none>           <none>
kube-system   kube-proxy-87lx5                          1/1     Running    0          6m16s   192.168.0.18    centos-7-9-16   <none>           <none>
kube-system   kube-proxy-rctn6                          1/1     Running    0          8m14s   192.168.80.60   k8s             <none>           <none>
kube-system   kube-scheduler-k8s                        1/1     Running    0          8m18s   192.168.80.60   k8s             <none>           <none>
[root@k8s ~]# 

控制面板：等待几分钟后，再次查看 pods、nodes

kubectl get nodes -o wide

[root@k8s ~]# kubectl get nodes -o wide
NAME            STATUS   ROLES           AGE   VERSION   INTERNAL-IP     EXTERNAL-IP   OS-IMAGE                KERNEL-VERSION           CONTAINER-RUNTIME
centos-7-9-16   Ready    <none>          23m   v1.25.3   192.168.80.16   <none>        CentOS Linux 7 (Core)   3.10.0-1160.el7.x86_64   containerd://1.6.9
k8s             Ready    control-plane   25m   v1.25.3   192.168.80.60   <none>        CentOS Linux 7 (Core)   3.10.0-1160.el7.x86_64   containerd://1.6.9
[root@k8s ~]# 

kubectl get pods --all-namespaces -o wide

[root@k8s ~]# kubectl get pods --all-namespaces -o wide
NAMESPACE     NAME                                      READY   STATUS    RESTARTS            AGE   IP              NODE            NOMINATED NODE   READINESS GATES
kube-system   calico-kube-controllers-f79f7749d-rkqgw   1/1     Running   2 (52s ago)         17m   172.16.77.9     k8s             <none>           <none>
kube-system   calico-node-7698p                         0/1     Running   2 (52s ago)         17m   192.168.80.60   k8s             <none>           <none>
kube-system   calico-node-tvhnb                         0/1     Running   0                   17m   192.168.80.16   centos-7-9-16   <none>           <none>
kube-system   coredns-c676cc86f-4lncg                   1/1     Running   2 (52s ago)         25m   172.16.77.8     k8s             <none>           <none>
kube-system   coredns-c676cc86f-7n9wv                   1/1     Running   2 (52s ago)         25m   172.16.77.7     k8s             <none>           <none>
kube-system   etcd-k8s                                  1/1     Running   2 (52s ago)         25m   192.168.80.60   k8s             <none>           <none>
kube-system   kube-apiserver-k8s                        1/1     Running   2 (52s ago)         25m   192.168.80.60   k8s             <none>           <none>
kube-system   kube-controller-manager-k8s               1/1     Running   2 (52s ago)         25m   192.168.80.60   k8s             <none>           <none>
kube-system   kube-proxy-87lx5                          1/1     Running   1 (<invalid> ago)   23m   192.168.80.16   centos-7-9-16   <none>           <none>
kube-system   kube-proxy-rctn6                          1/1     Running   2 (52s ago)         25m   192.168.80.60   k8s             <none>           <none>
kube-system   kube-scheduler-k8s                        1/1     Running   2 (52s ago)         25m   192.168.80.60   k8s             <none>           <none>
[root@k8s ~]# 

19. 至此，k8s安装与配置已完成，下面内容是测试。
20. 控制面板：创建 nginx 服务

vim nginx.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.23.2
        ports:
        - containerPort: 80

kubectl apply -f nginx.yaml

# 编辑
# kubectl edit deployment nginx-deployment

kubectl get pods --all-namespaces -o wide

[root@k8s ~]# kubectl get pods --all-namespaces -o wide
NAMESPACE     NAME                                      READY   STATUS              RESTARTS            AGE   IP              NODE            NOMINATED NODE   READINESS GATES
default       nginx-deployment-86956f97b8-nfv2l         0/1     ContainerCreating   0                   15s   <none>          centos-7-9-16   <none>           <none>
default       nginx-deployment-86956f97b8-x26kx         0/1     ContainerCreating   0                   15s   <none>          centos-7-9-16   <none>           <none>
kube-system   calico-kube-controllers-f79f7749d-rkqgw   1/1     Running             2 (6m22s ago)       23m   172.16.77.9     k8s             <none>           <none>
kube-system   calico-node-7698p                         0/1     Running             2 (6m22s ago)       23m   192.168.80.60   k8s             <none>           <none>
kube-system   calico-node-tvhnb                         0/1     Running             0                   23m   192.168.80.16   centos-7-9-16   <none>           <none>
kube-system   coredns-c676cc86f-4lncg                   1/1     Running             2 (6m22s ago)       31m   172.16.77.8     k8s             <none>           <none>
kube-system   coredns-c676cc86f-7n9wv                   1/1     Running             2 (6m22s ago)       31m   172.16.77.7     k8s             <none>           <none>
kube-system   etcd-k8s                                  1/1     Running             2 (6m22s ago)       31m   192.168.80.60   k8s             <none>           <none>
kube-system   kube-apiserver-k8s                        1/1     Running             2 (6m22s ago)       31m   192.168.80.60   k8s             <none>           <none>
kube-system   kube-controller-manager-k8s               1/1     Running             2 (6m22s ago)       31m   192.168.80.60   k8s             <none>           <none>
kube-system   kube-proxy-87lx5                          1/1     Running             1 (<invalid> ago)   29m   192.168.80.16   centos-7-9-16   <none>           <none>
kube-system   kube-proxy-rctn6                          1/1     Running             2 (6m22s ago)       31m   192.168.80.60   k8s             <none>           <none>
kube-system   kube-scheduler-k8s                        1/1     Running             2 (6m22s ago)       31m   192.168.80.60   k8s             <none>           <none>
[root@k8s ~]#

kubectl get pods -o wide

[root@k8s ~]# kubectl get pods -o wide
NAME                                READY   STATUS              RESTARTS   AGE   IP       NODE            NOMINATED NODE   READINESS GATES
nginx-deployment-86956f97b8-nfv2l   0/1     ContainerCreating   0          35s   <none>   centos-7-9-16   <none>           <none>
nginx-deployment-86956f97b8-x26kx   0/1     ContainerCreating   0          35s   <none>   centos-7-9-16   <none>           <none>
[root@k8s ~]# 

控制面板：**几分钟后再查看**

kubectl get pods --all-namespaces -o wide

[root@k8s ~]# kubectl get pods --all-namespaces -o wide
NAMESPACE     NAME                                      READY   STATUS    RESTARTS            AGE     IP               NODE            NOMINATED NODE   READINESS GATES
default       nginx-deployment-86956f97b8-nfv2l         1/1     Running   0                   3m30s   172.16.132.193   centos-7-9-16   <none>           <none>
default       nginx-deployment-86956f97b8-x26kx         1/1     Running   0                   3m30s   172.16.132.194   centos-7-9-16   <none>           <none>
kube-system   calico-kube-controllers-f79f7749d-rkqgw   1/1     Running   2 (9m37s ago)       26m     172.16.77.9      k8s             <none>           <none>
kube-system   calico-node-7698p                         0/1     Running   2 (9m37s ago)       26m     192.168.80.60    k8s             <none>           <none>
kube-system   calico-node-tvhnb                         0/1     Running   0                   26m     192.168.80.16    centos-7-9-16   <none>           <none>
kube-system   coredns-c676cc86f-4lncg                   1/1     Running   2 (9m37s ago)       34m     172.16.77.8      k8s             <none>           <none>
kube-system   coredns-c676cc86f-7n9wv                   1/1     Running   2 (9m37s ago)       34m     172.16.77.7      k8s             <none>           <none>
kube-system   etcd-k8s                                  1/1     Running   2 (9m37s ago)       34m     192.168.80.60    k8s             <none>           <none>
kube-system   kube-apiserver-k8s                        1/1     Running   2 (9m37s ago)       34m     192.168.80.60    k8s             <none>           <none>
kube-system   kube-controller-manager-k8s               1/1     Running   2 (9m37s ago)       34m     192.168.80.60    k8s             <none>           <none>
kube-system   kube-proxy-87lx5                          1/1     Running   1 (<invalid> ago)   32m     192.168.80.16    centos-7-9-16   <none>           <none>
kube-system   kube-proxy-rctn6                          1/1     Running   2 (9m37s ago)       34m     192.168.80.60    k8s             <none>           <none>
kube-system   kube-scheduler-k8s                        1/1     Running   2 (9m37s ago)       34m     192.168.80.60    k8s             <none>           <none>
[root@k8s ~]# 

kubectl get pods -o wide

[root@k8s ~]# kubectl get pods -o wide
NAME                                READY   STATUS    RESTARTS   AGE     IP               NODE            NOMINATED NODE   READINESS GATES
nginx-deployment-86956f97b8-nfv2l   1/1     Running   0          4m31s   172.16.132.193   centos-7-9-16   <none>           <none>
nginx-deployment-86956f97b8-x26kx   1/1     Running   0          4m31s   172.16.132.194   centos-7-9-16   <none>           <none>
[root@k8s ~]# 

# 控制面板：查看pod,svc
kubectl get pod,svc -o wide

[root@k8s ~]# kubectl get pod,svc -o wide
NAME                                    READY   STATUS    RESTARTS   AGE     IP               NODE            NOMINATED NODE   READINESS GATES
pod/nginx-deployment-86956f97b8-nfv2l   1/1     Running   0          4m59s   172.16.132.193   centos-7-9-16   <none>           <none>
pod/nginx-deployment-86956f97b8-x26kx   1/1     Running   0          4m59s   172.16.132.194   centos-7-9-16   <none>           <none>

NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE   SELECTOR
service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   36m   <none>
[root@k8s ~]# 

# 控制面板：设置服务
kubectl expose deployment nginx-deployment --type=NodePort --name=nginx-service

# 控制面板：查看pod,svc
kubectl get pod,svc -o wide

[root@k8s ~]# kubectl get pod,svc -o wide
NAME                                    READY   STATUS    RESTARTS   AGE     IP               NODE            NOMINATED NODE   READINESS GATES
pod/nginx-deployment-86956f97b8-nfv2l   1/1     Running   0          7m58s   172.16.132.193   centos-7-9-16   <none>           <none>
pod/nginx-deployment-86956f97b8-x26kx   1/1     Running   0          7m58s   172.16.132.194   centos-7-9-16   <none>           <none>

NAME                    TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE   SELECTOR
service/kubernetes      ClusterIP   10.96.0.1       <none>        443/TCP        39m   <none>
service/nginx-service   NodePort    10.109.120.77   <none>        80:30593/TCP   55s   app=nginx
[root@k8s ~]# 

# 重启控制面板、node节点
# 控制面板：查看pod,svc
kubectl get pod,svc -o wide

[root@k8s ~]# kubectl get pod,svc -o wide
NAME                                    READY   STATUS    RESTARTS            AGE   IP               NODE            NOMINATED NODE   READINESS GATES
pod/nginx-deployment-86956f97b8-nfv2l   1/1     Running   1 (<invalid> ago)   11m   172.16.132.196   centos-7-9-16   <none>           <none>
pod/nginx-deployment-86956f97b8-x26kx   1/1     Running   1 (<invalid> ago)   11m   172.16.132.195   centos-7-9-16   <none>           <none>

NAME                    TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE    SELECTOR
service/kubernetes      ClusterIP   10.96.0.1       <none>        443/TCP        42m    <none>
service/nginx-service   NodePort    10.109.120.77   <none>        80:30593/TCP   4m8s   app=nginx
[root@k8s ~]# 

可以看到：重启前后 pod/nginx-deployment- IP 发生了变化，service/nginx-service 的 IP 与 端口没有发生变化，可在后面使用 service/nginx-service 的 端口*

Token 相关命令

1. 控制平面节点上运行以下命令来获取令牌

kubeadm token list

2. 默认情况下，令牌会在 24 小时后过期，可以通过在控制平面节点上运行以下命令来创建新令牌

kubeadm token create

相关命令

1. 查看更多信息

 -o wide

2. 查看所有命名空间

--all-namespaces

3. 查看指定命名空间

-n 命名空间

4. 查看所有 pod

kubectl get pods --all-namespaces -o wide

5. 查看 pod 描述

kubectl -n 命名空间 describe pod 名称

6. 删除 pod

kubectl -n 命名空间 delete pod 名称

7. 进入 pod

kubectl exec -it pod名称 bash

8. 查看 Service Account

kubectl get sa --all-namespaces

kubectl -n 命名空间 get sa

9. 查看 pv

kubectl get pv

10. 查看 pvc

kubectl get pvc

11. 查看角色绑定

kubectl get rolebinding --all-namespaces -o wide

错误说明

1. 提示：/proc/sys/net/bridge/bridge-nf-call-iptables

error execution phase preflight: [preflight] Some fatal errors occurred:
	[ERROR FileContent--proc-sys-net-bridge-bridge-nf-call-iptables]: /proc/sys/net/bridge/bridge-nf-call-iptables contents are not set to 1
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
To see the stack trace of this error execute with --v=5 or higher

# 执行命令
# 如果报错 sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-iptables: No such file or directory，可以先执行 modprobe br_netfilter
sysctl -w net.bridge.bridge-nf-call-iptables=1

2. 提示：/proc/sys/net/ipv4/ip_forward

error execution phase preflight: [preflight] Some fatal errors occurred:
	[ERROR FileContent--proc-sys-net-ipv4-ip_forward]: /proc/sys/net/ipv4/ip_forward contents are not set to 1
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
To see the stack trace of this error execute with --v=5 or higher

# 执行命令
sysctl -w net.ipv4.ip_forward=1

3. 控制面板（master）作为node使用（去污）
   注意：此处的命令可能和你在网上看到去污命令不同，原因是k8s的版本不同

# https://kubernetes.io/zh-cn/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/#control-plane-node-isolation
kubectl taint nodes --all node-role.kubernetes.io/control-plane-

# 1.24.0 版本需要使用下列命令去污
# kubectl taint nodes --all node-role.kubernetes.io/master-

**可使用下列命令查看当前软件的去污的命令参数**

kubectl get no -o yaml | grep taint -A 10