Install Kubernetes Cluster with Kubeadm on VMware Photon OS

Install Kubernetes Cluster dengan Kubeadm pada VMware Photon OS 5.0

  • Minimum requirement: cpu: 2 cpu & ram: 2 GB
  • Master : 10.10.1.201 : photon-1.sys-ops.id
  • Worker : 10.10.1.202 : photon-2.sys-ops.id

Setting Hostname

  • Setting hostname pada semua node
# node master
hostnamectl set-hostname photon-1.sys-ops.id

# node worker
hostnamectl set-hostname photon-2.sys-ops.id
  • Setting file hosts pada semua node
nano /etc/hosts

10.10.1.201     photon-1.sys-ops.id photon-1
10.10.1.202     photon-2.sys-ops.id photon-2
  • Disable iptables pada semua node
systemctl stop iptables
systemctl disable iptables
  • Jika ingin menggunakan firewall pastikan allow port berikut pada node master dan node worker
##Photon Master
iptables -A INPUT -p icmp --icmp-type echo-request -j ACCEPT
iptables -A INPUT -p tcp -m tcp --dport 2379:2380 -j ACCEPT
iptables -A INPUT -p tcp -m tcp --dport 6443 -j ACCEPT
iptables -A INPUT -p tcp -m tcp --dport 10250:10252 -j ACCEPT
iptables -A INPUT -p tcp -m tcp --dport 179 -j ACCEPT
iptables -A INPUT -p tcp -m tcp --dport 4789 -j ACCEPT

##Photon Worker
iptables -A INPUT -p icmp --icmp-type echo-request -j ACCEPT
iptables -A INPUT -p tcp -m tcp --dport 10250:10252 -j ACCEPT
iptables -A INPUT -p tcp -m tcp --dport 30000:32767 -j ACCEPT
iptables -A INPUT -p tcp -m tcp --dport 179 -j ACCEPT
iptables -A INPUT -p tcp -m tcp --dport 4789 -j ACCEPT

##Save config iptables
iptables-save > /etc/systemd/scripts/ip4save

Install Kubernetes Cluster

  • Update system dan install kubernetes pada semua node
tdnf update -y
tdnf install -y vim nano wget curl htop docker cri-tools kubernetes-kubeadm apparmor-parser
  • Enable IPv4 IP forwarding pada semua node
nano /etc/sysctl.d/kubernetes.conf

net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net/bridge/bridge-nf-call-arptables = 1
  • Load br_netfilter dan apply konfigurasi
modprobe br_netfilter && sysctl --system
  • Konfigurasi Runtime Containerd pada semua node
nano /etc/crictl.yaml

runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock
timeout: 2
debug: false
pull-image-on-create: false
disable-pull-on-run: false
  • Edit file config.toml seperti berikut
#disabled_plugins = ["cri"]

#root = "/var/lib/containerd"
#state = "/run/containerd"
#subreaper = true
#oom_score = 0
version = 2

#[grpc]
#  address = "/run/containerd/containerd.sock"
#  uid = 0
#  gid = 0

[plugins."io.containerd.grpc.v1.cri"]
enable_selinux = true
  [plugins."io.containerd.grpc.v1.cri".containerd]
      [plugins."io.containerd.grpc.v1.cri".containerd.runtimes]
        [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
          runtime_type = "io.containerd.runc.v2"
          [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
            SystemdCgroup = true

#[debug]
#  address = "/run/containerd/debug.sock"
#  uid = 0
#  gid = 0
#  level = "info"
  • Reload systemd dan restart service containerd pada semua node
systemctl daemon-reload
systemctl restart containerd
systemctl enable containerd
systemctl status containerd
  • Aktifkan service kubelet dan download config image pada semua node
systemctl enable --now kubelet
kubeadm config images pull

Initialize Kubernetes Cluster

  • Initialize kubernetes cluster pada node Master
kubeadm init --pod-network-cidr=192.168.0.0/16
---<output>---
Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 10.10.1.201:6443 --token ei7vis.jjqgxcovrn7gyhm0 \
	--discovery-token-ca-cert-hash sha256:0f4524f20398bd8da976025775bd935ea98f937966a686db1b7bec32ecbb33a8 
  • Setting kubernetes cluster pada node master
mkdir -p $HOME/.kube
cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
chown $(id -u):$(id -g) $HOME/.kube/config
  • Download dan Install pod network pada node master
kubectl taint nodes --all node-role.kubernetes.io/control-plane-
curl  https://raw.githubusercontent.com/projectcalico/calico/master/manifests/calico.yaml -o network.yaml

systemctl restart docker

docker pull calico/cni
docker pull calico/node
docker pull flannelcni/flannel
docker pull calico/kube-controllers

kubectl apply -f network.yaml
  • Cek status cluster pada node master
kubectl cluster-info
---<output>---
Kubernetes control plane is running at https://10.10.1.201:6443
CoreDNS is running at https://10.10.1.201:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
--------------------------------------------------------------------------------------------------------
kubectl get nodes
---<output>---
NAME       STATUS     ROLES           AGE   VERSION
photon-1   NotReady   control-plane   11m   v1.27.3
  • Cek status namespace pastikan semua sudah running
kubectl get pods --all-namespaces
---<output>---
NAMESPACE     NAME                                      READY   STATUS    RESTARTS   AGE
kube-system   calico-kube-controllers-8787c9999-hz4hm   1/1     Running   0          2m44s
kube-system   calico-node-vhcwg                         1/1     Running   0          2m44s
kube-system   coredns-5d78c9869d-4lrd5                  1/1     Running   0          13m
kube-system   coredns-5d78c9869d-ctfrl                  1/1     Running   0          13m
kube-system   etcd-photon-1                             1/1     Running   0          14m
kube-system   kube-apiserver-photon-1                   1/1     Running   0          14m
kube-system   kube-controller-manager-photon-1          1/1     Running   0          14m
kube-system   kube-proxy-xpw4g                          1/1     Running   0          13m
kube-system   kube-scheduler-photon-1                   1/1     Running   0          14m

Add Worker Node

  • Tambahkan worker ke dalam cluster pada semua node worker
  • Perintah berikut otomatis ketika selesai intitialize cluster pada node master
kubeadm join 10.10.1.201:6443 --token ei7vis.jjqgxcovrn7gyhm0 \
	--discovery-token-ca-cert-hash sha256:0f4524f20398bd8da976025775bd935ea98f937966a686db1b7bec32ecbb33a8 
  • Download pod network pada node worker
systemctl restart docker

docker pull calico/cni
docker pull calico/node
docker pull flannelcni/flannel
docker pull calico/kube-controllers

  • Cek status cluster node pada node master
kubectl get nodes -o wide
---<output>---
NAME       STATUS   ROLES           AGE     VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                 KERNEL-VERSION     CONTAINER-RUNTIME
photon-1   Ready    control-plane   20m     v1.27.3   10.10.1.201   <none>        VMware Photon OS/Linux   6.1.41-1.ph5-esx   containerd://1.6.21
photon-2   Ready    <none>          4m30s   v1.27.3   10.10.1.202   <none>        VMware Photon OS/Linux   6.1.41-1.ph5-esx   containerd://1.6.21

Pengujian

  • Deploy aplikasi web service ke dalam cluster pada node master
kubectl create deploy web-test --image=sysopsid/web-test
  • Cek status deployment
kubectl get deploy -o wide
---<output>---
NAME       READY   UP-TO-DATE   AVAILABLE   AGE   CONTAINERS   IMAGES              SELECTOR
web-test   1/1     1            1           80s   web-test     sysopsid/web-test   app=web-test
  • Cek status Pod pada node master
kubectl get pods -o wide
---<output>---
NAME                        READY   STATUS    RESTARTS   AGE    IP                NODE       NOMINATED NODE   READINESS GATES
web-test-5fdf749d97-f889s   1/1     Running   0          109s   192.168.118.129   photon-2   <none>           <none>
  • Expose service web-test agar dapat di akses dari luar jaringan menggunakan ip node
kubectl create service nodeport web-test --tcp=80:80
  • Cek service, external port web-test menggunakan port 30794
kubectl get services -o wide
---<output>---
NAME         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE   SELECTOR
kubernetes   ClusterIP   10.96.0.1        <none>        443/TCP        26m   <none>
web-test     NodePort    10.104.178.227   <none>        80:30242/TCP   12s   app=web-test
  • Verifiaksi service pod web-test menggunakan curl dengan ip node
  • Pod web-test sudah bisa di akses dari luar jaringan
curl 10.10.1.201:30242
---<output>---
<!DOCTYPE html>
<html>
<head>
  <title>sys-ops.id</title>
  <link rel="stylesheet" href="css/styles.css">
</head>
<body>
    <header>
    <h1>Welcome To My Page!</h1>
  </header>
  <main>
    <img src="img/cat1.png" alt="cat" />
    <p>Server hostname: web-test-5fdf749d97-f889s</p>
    <p>Server IP address: 192.168.118.129</p>
    <p>PHP version: 8.2.5</p>
  </main>
  <footer>
    <p>© 2023 sys-ops.id</p>
  </footer>
</body>
</html>

Scale Up/Down Deployment

  • Scale Up/Down Deployment menjadi 4 container
kubectl scale --replicas=4 deployment web-test
  • Cek status Pod pada node master
kubectl get pods -o wide
---<output>---
NAME                        READY   STATUS    RESTARTS   AGE     IP                NODE       NOMINATED NODE   READINESS GATES
web-test-5fdf749d97-4cz7b   1/1     Running   0          32s     192.168.118.130   photon-2   <none>           <none>
web-test-5fdf749d97-f889s   1/1     Running   0          4m23s   192.168.118.129   photon-2   <none>           <none>
web-test-5fdf749d97-s7mt5   1/1     Running   0          32s     192.168.248.197   photon-1   <none>           <none>
web-test-5fdf749d97-vkzw6   1/1     Running   0          32s     192.168.248.196   photon-1   <none>           <none>
  • Cek keseluruhan pod dan service secara details
kubectl get all -o wide
---<output>---
NAME                            READY   STATUS    RESTARTS   AGE     IP                NODE       NOMINATED NODE   READINESS GATES
pod/web-test-5fdf749d97-4cz7b   1/1     Running   0          51s     192.168.118.130   photon-2   <none>           <none>
pod/web-test-5fdf749d97-f889s   1/1     Running   0          4m42s   192.168.118.129   photon-2   <none>           <none>
pod/web-test-5fdf749d97-s7mt5   1/1     Running   0          51s     192.168.248.197   photon-1   <none>           <none>
pod/web-test-5fdf749d97-vkzw6   1/1     Running   0          51s     192.168.248.196   photon-1   <none>           <none>

NAME                 TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE    SELECTOR
service/kubernetes   ClusterIP   10.96.0.1        <none>        443/TCP        28m    <none>
service/web-test     NodePort    10.104.178.227   <none>        80:30242/TCP   2m4s   app=web-test

NAME                       READY   UP-TO-DATE   AVAILABLE   AGE     CONTAINERS   IMAGES              SELECTOR
deployment.apps/web-test   4/4     4            4           4m42s   web-test     sysopsid/web-test   app=web-test

NAME                                  DESIRED   CURRENT   READY   AGE     CONTAINERS   IMAGES              SELECTOR
replicaset.apps/web-test-5fdf749d97   4         4         4       4m42s   web-test     sysopsid/web-test   app=web-test,pod-template-hash=5fdf749d97
  • Masuk ke dalam bash container
kubectl exec --stdin --tty pod/web-test-5fdf749d97-4cz7b -- /bin/bash
---<output>---
root@web-test-5fdf749d97-4cz7b:/var/www/html# cat /etc/hosts
# Kubernetes-managed hosts file.
127.0.0.1	localhost
::1	localhost ip6-localhost ip6-loopback
fe00::0	ip6-localnet
fe00::0	ip6-mcastprefix
fe00::1	ip6-allnodes
fe00::2	ip6-allrouters
192.168.118.130	web-test-5fdf749d97-4cz7b
  • Cek detail status deployment
kubectl describe deployment web-test
---<output>---
Name:                   web-test
Namespace:              default
CreationTimestamp:      Sat, 29 Jul 2023 13:35:48 +0000
Labels:                 app=web-test
Annotations:            deployment.kubernetes.io/revision: 1
Selector:               app=web-test
Replicas:               4 desired | 4 updated | 4 total | 4 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:
  Labels:  app=web-test
  Containers:
   web-test:
    Image:        sysopsid/web-test
    Port:         <none>
    Host Port:    <none>
    Environment:  <none>
    Mounts:       <none>
  Volumes:        <none>
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Progressing    True    NewReplicaSetAvailable
  Available      True    MinimumReplicasAvailable
OldReplicaSets:  <none>
NewReplicaSet:   web-test-5fdf749d97 (4/4 replicas created)
Events:
  Type    Reason             Age    From                   Message
  ----    ------             ----   ----                   -------
  Normal  ScalingReplicaSet  6m15s  deployment-controller  Scaled up replica set web-test-5fdf749d97 to 1
  Normal  ScalingReplicaSet  2m24s  deployment-controller  Scaled up replica set web-test-5fdf749d97 to 4 from 1
  • Hapus service dan deployment web-test
kubectl delete service web-test
kubectl delete deployment web-test

herdiana3389

A system administrator with skills in system administration, virtualization, linux, windows, networking, cloud computing, container, etc.