centos7.9部署K8S

系统环境

Centos7.9

集群规划

主机名可以不用改强制修改为这样，只需要主机名，/etc/hosts 和 /etc/ansible/hosts 都一致即可

主机名	IP	用途
master1	192.168.181.11	集群主节点 1
master2	192.168.181.12	集群主节点 2
node1	192.168.181.13	工作节点 1
node2	192.168.181.14	工作节点 2

检查网络环境

四个节点都检查一遍，确保网络没有问题，涉及到后面拉取镜像

[root@master1 ~]# ping pkgs.k8s.io
PING redirect.k8s.io (34.107.204.206) 56(84) bytes of data.
64 bytes from 206.204.107.34.bc.googleusercontent.com (34.107.204.206): icmp_seq=1 ttl=128 time=158 ms

[root@master1 ~]# ping registry.aliyuncs.com
PING registry.aliyuncs.com (120.55.105.209) 56(84) bytes of data.
64 bytes from 120.55.105.209 (120.55.105.209): icmp_seq=1 ttl=128 time=30.4 ms

手动修改所有节点 hostname(请注意: 主机名，ansible 节点，/etc/hosts 要保持一致)

[root@master1 ~]# hostnamectl set-hostname master1
[root@master2 ~]# hostnamectl set-hostname master2
[root@node1 ~]# hostnamectl set-hostname node1
[root@node2 ~]# hostnamectl set-hostname node2

手动修改主节点 1 /etc/hosts(请注意: 主机名，ansible 节点，/etc/hosts 要保持一致)

[root@master1 ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6


192.168.181.11 master1
192.168.181.12 master2
192.168.181.13 node1
192.168.181.14 node2

配置免密 (主节点 1)

生成密钥

[root@master1 ~]# ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Created directory '/root/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:IhGroFT22rFJKOQTECWGm2SfPbvw8IW+Q0MvgAtDEBE root@master1
The key's randomart image is:
+---[RSA 2048]----+
|EX.o.            |
|==+ oo           |
|=Bo.=+           |
|Boo=+=+          |
|o.o.++* S        |
| . o B +         |
|    B =          |
|     *           |
|     .o          |
+----[SHA256]-----+

复制密钥到各个节点，包括自己循环四次依次输入 yes 和密码

[root@master1 ~]# for i in {master1,master2,node1,node2}; do  ssh-copy-id root@$i; done

/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
The authenticity of host 'master1 (192.168.152.200)' can't be established.
ECDSA key fingerprint is SHA256:1QncUYX+qzfiSSNgIiU7NQtEBZEuv6+sHOwb7gGdseY.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@master1's password:

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh'root@master1'"
and check to make sure that only the key(s) you wanted were added.
... 其他三次省略

确认可以使用以下方式远程到四台机

[root@master1 ~]# ssh root@master1
[root@master1 ~]# ssh root@master2
[root@master1 ~]# ssh root@node1
[root@master1 ~]# ssh root@node2

更换主节点 1 系统源

https://help.mirrors.cernet.edu.cn/epel/

[root@master1 ~]# sudo yum install epel-release -y
[root@master1 ~]# sudo sed -e 's!^metalink=!#metalink=!g'     -e 's!^#baseurl=!baseurl=!g'     -e 's!https\?://download\.fedoraproject\.org/pub/epel!https://mirrors.cernet.edu.cn/epel!g'     -e 's!https\?://download\.example/pub/epel!https://mirrors.cernet.edu.cn/epel!g'     -i /etc/yum.repos.d/epel{,-testing}.repo

[root@master1 ~]# yum update -y

安装 ansible(在主节点安装)

[root@master1 ~]# yum -y install ansible
[root@master1 ~]# mkdir -p /etc/ansible/

复制整个 k8s_kubeadm_install 到主节点任意位置

https://github.com/AYYQ127/k8s_kubeadm_install

[root@master1 k8s_kubeadm_install]# tree
.
├── files
│   ├── ansible
│   │   ├── ansible.cfg
│   │   └── hosts
│   ├── calico
│   │   ├── custom-resources_v3.26.4.yaml
│   │   ├── custom-resources_v3.27.0.yaml
│   │   ├── tigera-operator_v3.26.4.yaml
│   │   └── tigera-operator_v3.27.0.yaml
│   ├── ingress
│   │   ├── deploy_v1.9.4.yaml
│   │   └── deploy_v1.9.5.yaml
│   ├── k8s_pkgs
│   │   ├── docker-ce.repo
│   │   ├── kubernetes-apt-keyring.gpg
│   │   ├── kubernetes-lock.repo
│   │   ├── kubernetes-nolock.repo
│   │   ├── repomd.xml.key
│   │   └── source.list
│   ├── metrics
│   │   └── components.yaml
│   ├── rancher
│   ├── test-ingress.yaml
│   └── vars.yaml
├── How_to_run.md
├── How_to_run_redhat_release.md
├── LICENSE
├── playbooks
│   ├── dashboard_install.yaml
│   ├── harbor_install.yaml
│   ├── main_redhat_release.yaml
│   ├── main.yaml
│   ├── metrics_server_install.yaml
│   └── prometheus_install.yaml
└── README.md

在主节点 1 准备 ansible 环境

[root@master1 k8s_kubeadm_install~]# vim files/ansible/hosts
[root@master1 k8s_kubeadm_install~]# cat files/ansible/hosts
# 修改 hosts 节点名, 分组不能修改, 只加 /etc/hosts 中对应主机名 

###                                                           
#
# 如果不想使用 22 端口 ssh 连接, 可以添加变量 ansible_ssh_port=port_num
# 例如：
# [manage_node]
# master1 ansible_ssh_port=2222
#
#
###

# 执行安装的节点, 第一台 master
[manage_node]
master1 

# 其他主节点在此添加, 不要再加 manage_node
[other_masters]
master2 

# 工作节点在此添加 
[nodes]
node1 
node2 


# **************** 以下内容不要修改 *****************

# 除了操作节点的所有节点 
[except_manage_node:children]
other_masters
nodes

# 所有主节点 (请勿修改)
[masters:children]
manage_node
other_masters

# 所有节点分组 (请勿修改)
[k8s:children]
manage_node
other_masters
nodes

# 修改 hosts 节点名, 分组不能修改 (请注意: 主机名，ansible 节点，/etc/hosts 要保持一致)
[root@master1 k8s_kubeadm_install~]# \cp -r files/ansible /etc/

使用 ansible 统一修改 hosts 和 yum 源

[root@master1 k8s_kubeadm_install~]# ansible k8s -m copy -a "src=/etc/hosts dest=/etc/hosts"
[root@master1 k8s_kubeadm_install~]# ansible k8s -m yum -a "update_cache=yes"
[root@master1 k8s_kubeadm_install~]# ansible k8s -m yum -a "name=epel-release state=latest"
[root@master1 k8s_kubeadm_install~]# ansible k8s -m copy -a "src=/etc/yum.repos.d/epel.repo dest=/etc/yum.repos.d/"
[root@master1 k8s_kubeadm_install~]# ansible k8s -m copy -a "src=/etc/yum.repos.d/epel-testing.repo dest=/etc/yum.repos.d/"
[root@master1 k8s_kubeadm_install~]# ansible k8s -m yum -a "update_cache=yes"

修改 files/vars.yaml(主节点 1)

指定版本，master 节点 ip 等信息

[root@master1 k8s_kubeadm_install~]# cat files/vars.yaml
# 时间服务器 
NTP: 192.168.181.11

# 控制面板主机名，不需要修改，固定为 [manage_node]
control_plane_endpoint: master1

# kubeadm init 使用，大版本号，不需要后缀 
kubernetes_version: v1.28.4

# 详细版本请看 README.md
# apt 版本修改这个 
kube_3tools_version: 1.28.4-1.1
# yum 版本修改这个 
kube_3tools_version_yum: 1.28.4-150500.1.1

# 定义 pod 的 cidr 网络，kubeadm init 使用 
pod_network_cidr: 10.244.0.0/16
# kubeadm init 使用 
apiserver_advertise_address: 192.168.181.11

# 修改 calico 使用 custom-resources.yaml 使用 
pod_network: 10.244.0.0

# pause 所使用镜像版本，需要替换阿里源，官方源国内无法拉取 
sandbox_image: pause:3.9

# 有版本依赖，请参考 README.md 指引更换版本 
calico_version: v3.26.4

# 有版本依赖，请参考 README.md 指引更换版本 
ingress_version: v1.9.4

# 有版本依赖，请参考 README.md 指引更换版本 
metrics_server_version: v0.6.4

# 有版本依赖，请参考 README.md 指引更换版本 
kubernetes_dashboard_version: v3.0.0-alpha0
# v3.0.0-alpha0 版本 ingress 中指定的 host，用于部署完成后浏览器访问 
dashboard_host: k8s.dashboard.example
# 版本低于 v3 的需要指定端口，请前往 k8s_kubeadm_install/files/dashboard/README.md 查看 
#nodePoart: 30443

# 有版本依赖，请参考 README.md 指引更换版本 
# dashboard 版本 v3 以后必须安装 
cert_manager_version: v1.13.3

Run

必须确认 vars.yaml 变量是否修改 control_plane_endpoint 需要与 hosts 文件格式一致

安装过程分为三步，第一步会重启所有节点，重启后再次进入主节点 1 目录运行相同命令 (总共执行两次)

# 默认只安装集群基础功能 
[root@master1 k8s_kubeadm_install~]# ansible-playbook playbooks/main_redhat_release.yaml

PLAY [第一步初始化系统] ***********************************************************************************************************************************************************************************************
...
...

PLAY [第二步安装 kubeadmin] ***************************************************************************************************************************************************************************************
...
...
PLAY [第三步初始化集群, 添加工作节点] ***************************************************************************************************************************************************************************************
...
PLAY RECAP ****************************************************************************************************************************************************************************************************
localhost                  : ok=26   changed=18   unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
master1                    : ok=20   changed=16   unreachable=0    failed=0    skipped=17   rescued=0    ignored=0
master2                    : ok=20   changed=15   unreachable=0    failed=0    skipped=17   rescued=0    ignored=0
node1                      : ok=20   changed=15   unreachable=0    failed=0    skipped=17   rescued=0    ignored=0
node2                      : ok=20   changed=16   unreachable=0    failed=0    skipped=17   rescued=0    ignored=0

# 等 4 - 5 分钟执行以下命令，集群安装完毕 
[root@master1 k8s_kubeadm_install~]# kubectl get nodes
NAME      STATUS   ROLES           AGE     VERSION
master1   Ready    control-plane   6m39s   v1.28.4
master2   Ready    control-plane   5m3s    v1.28.4
node1     Ready    node            6m16s   v1.28.4
node2     Ready    node            6m13s   v1.28.4

# 选装其他插件,harbor 需要修改 ansible/hosts 中分组和 /etc/hosts 解析 
[root@master1 k8s_kubeadm_install~]# ansible-playbook playbooks/main_redhat_release.yaml -t [metrics | harbor | dashboard | prometheus]

# 暂时只支持 metrics
[root@master1 k8s_kubeadm_install]# ansible-playbook playbooks/main_redhat_release.yaml  -t metrics

PLAY [第一步初始化系统] 
********************************

PLAY [第二步 安装 kubeadmin] 
********************************

PLAY [第三步 初始化集群, 添加工作节点] 
********************************

PLAY [创建 metrics-server 资源对象] 
********************************

TASK [检查 metrics.lock 文件是否存在]  
********************************
ok: [localhost]

TASK [创建 metrics-server 资源对象]  
********************************
changed: [localhost]

PLAY [开启聚合 API]  
********************************

TASK [检查 metrics.lock 文件是否存在]  
********************************
ok: [master2]
ok: [master1]

TASK [开启聚合 API]  
********************************
changed: [master2]
changed: [master1]

TASK [重启 kubelet]  
********************************
changed: [master2]
changed: [master1]

PLAY [为 kubelet 签发证书]  
********************************

TASK [检查 metrics.lock 文件是否存在]  
********************************
ok: [master2]
ok: [node2]
ok: [node1]
ok: [master1]

TASK [在最后一行插入]  
********************************
changed: [node2]
changed: [master2]
changed: [node1]
changed: [master1]

TASK [重启 kubelet]  
********************************
changed: [node2]
changed: [master2]
changed: [node1]
changed: [master1]

PLAY [签发证书]  
********************************

TASK [检查 metrics.lock 文件是否存在]  
********************************
ok: [master1]

TASK [暂停 15 秒]  
********************************
Pausing for 15 seconds
(ctrl+C then 'C' = continue early, ctrl+C then 'A' = abort)
ok: [master1]

TASK [获取新发证书名]  
********************************
changed: [master1]

TASK [正式签发证书]  
********************************
changed: [master1] => (item=csr-2q9wb)
changed: [master1] => (item=csr-pwtj6)
changed: [master1] => (item=csr-wll9j)
changed: [master1] => (item=csr-z7wbp)

PLAY [创建 lock 文件]  
********************************

TASK [检查 metrics.lock 文件是否存在]  
********************************
ok: [node2]
ok: [node1]
ok: [master2]
ok: [master1]

TASK [file]  
********************************
changed: [master2]
changed: [node1]
changed: [node2]
changed: [master1]

PLAY RECAP  
********************************
localhost                  : ok=2    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
master1                    : ok=12   changed=7    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
master2                    : ok=8    changed=5    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
node1                      : ok=5    changed=3    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
node2                      : ok=5    changed=3    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

# 等待 pod 创建完成 
[root@master1 k8s_kubeadm_install]# kubectl get pod -A | grep metrics
kube-system        metrics-server-5b779d9499-znctk            1/1     Running     0              2m5s
[root@master1 k8s_kubeadm_install]# kubectl top node
NAME      CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
master1   120m         6%     1852Mi          50%
master2   115m         5%     1947Mi          53%
node1     46m          2%     1512Mi          41%
node2     46m          2%     1558Mi          42%

# dashboard
[root@master1 k8s_kubeadm_install]# ansible-playbook playbooks/main_redhat_release.yaml  -t dashboard

PLAY [创建 dashboard 资源对象] ************************************************************************************************************************************************************************

TASK [检查 dashboard.lock 文件是否存在] *****************************************************************************************************************************************************************
ok: [localhost]

TASK [请确认 cert_manager 已经安装] ********************************************************************************************************************************************************************
ok: [localhost] => {"msg": "\" 请确认 cert_manager 已经安装, 如果没有, 请终止 playboook, 执行 ansible-playbook playbooks/main.yaml  -t cert_manager\"\n"
}

TASK [请确认 cert_manager 已经安装, 暂停 15 秒] **************************************************************************************************************************************************************
Pausing for 15 seconds
(ctrl+C then 'C' = continue early, ctrl+C then 'A' = abort)
Press 'C' to continue the play or 'A' to abort
ok: [localhost]

TASK [判断版本是不是 v2.7.0] **************************************************************************************************************************************************************************
skipping: [localhost]

TASK [创建 dashboard 资源对象] ************************************************************************************************************************************************************************
changed: [localhost]

TASK [授权] *************************************************************************************************************************************************************************************
changed: [localhost]

TASK [生成 token] ********************************************************************************************************************************************************************************
changed: [localhost]

TASK [将 token 写入一个文件保存起来] ***********************************************************************************************************************************************************************
changed: [localhost] => (item=eyJhbGciOiJSUzI1NiIsImtpZCI6IlhDVUhuNEVIR3VublplRlAwa0NZOUY1bWNoZXhJWnRVRElyYUM5UmVjME0ifQ.eyJhdWQiOlsiaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwiXSwiZXhwIjoyMDIxNjkyMTc4LCJpYXQiOjE3MDYzMzIxNzgsImlzcyI6Imh0dHBzOi8va3ViZXJuZXRlcy5kZWZhdWx0LnN2Yy5jbHVzdGVyLmxvY2FsIiwia3ViZXJuZXRlcy5pbyI6eyJuYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsInNlcnZpY2VhY2NvdW50Ijp7Im5hbWUiOiJrdWJlLWRhc2hib2FyZC1hZG1pbi1zYSIsInVpZCI6IjM4NDk1MjhjLWU0ODktNDQ5NC04MGQzLTExMjM4MTY3MTZkMCJ9fSwibmJmIjoxNzA2MzMyMTc4LCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6a3ViZS1zeXN0ZW06a3ViZS1kYXNoYm9hcmQtYWRtaW4tc2EifQ.VrzelIRT_JF2TnBlV3Fvg0YPfXGRLu_48IX7QkVFgyFMpMd_nk7rBSqZJwULNOS-e-n07oiEv4gzzCUNknsLsAFmA8CgqCRELGQX_fviOKJHZ-S38nFIVS0TeI-BZvTSFnJo9zUFSFAHKOZ0zmjFhIORwsnRGkJyS9u7kvHWNDFMccd15WgmtO9jh9NjBsoR838P8LWsn2c48-G8nsBsP3TtUTy1rpZkbTBSPvfLgGEulSMQUms_51Q5GNDk1sUpLVeIy8ZxLcWyeyvlKYbH_qPyRUzH5yaDW6KmQiPb0PftR7Ip6vQd-xOc7GdjagV2wv8OV5kEhOTHcvS2jRDAKw)

TASK [dashboard 的 token 请查看 /root/k8s_install/dashboard_token] ************************************************************************************************************************************
ok: [localhost] => {"msg": "cat /root/k8s_install/dashboard_token\n"
}

TASK [创建安装 lock 文件] *****************************************************************************************************************************************************************************
changed: [localhost]

PLAY RECAP ************************************************************************************************************************************************************************************
localhost                  : ok=9    changed=5    unreachable=0    failed=0    skipped=1    rescued=0    ignored=0


[root@master1 k8s_kubeadm_install]# kubectl -n kubernetes-dashboard  get service
NAME                        TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)         AGE
dashboard-metrics-scraper   ClusterIP   10.96.210.243   <none>        8000/TCP        2m48s
kubernetes-dashboard        NodePort    10.97.130.151   <none>        443:30443/TCP   40s