Kubernetes Cluster Setup on 3 Ubuntu Servers

0. Planning & Prerequisites

Hardware (per node, minimums for production-leaning)

Naming & networking

Pick hostnames and static IPs before you start. Example:

Reserved (non-overlapping) CIDRs for this guide:

• Pod network CIDR: 192.168.0.0/16 (Calico default)
• Service CIDR: 10.96.0.0/12 (kubeadm default)
• Node subnet: whatever your LAN uses (must not overlap with the two above)

Firewall ports

Open these between nodes before bootstrapping:

Control plane

• 6443/tcp — Kubernetes API server
• 2379-2380/tcp — etcd
• 10250/tcp — kubelet API
• 10257/tcp — kube-controller-manager
• 10259/tcp — kube-scheduler

Workers

• 10250/tcp — kubelet API
• 30000-32767/tcp — NodePort services

Calico (all nodes)

• 179/tcp — BGP
• 4789/udp — VXLAN
• 5473/tcp — Typha (if enabled)

1. Prepare ALL 3 nodes (run on every server)

SSH in as a user with sudo. Run every command in this section on cp-1, node-1, and node-2.

1.1 Set a unique hostname and /etc/hosts

# On cp-1:
sudo hostnamectl set-hostname cp-1
# On node-1:
sudo hostnamectl set-hostname node-1
# On node-2:
sudo hostnamectl set-hostname node-2

On every node, add all three entries to /etc/hosts (edit to your real IPs):

sudo tee -a /etc/hosts <<EOF
10.0.0.10 cp-1
10.0.0.11 node-1
10.0.0.12 node-2
EOF

1.2 Update the system

sudo apt-get update && sudo apt-get upgrade -y
sudo apt-get install -y curl gnupg2 ca-certificates apt-transport-https software-properties-common

1.3 Disable swap (kubelet requires this)

sudo swapoff -a
sudo sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab

Verify: free -h should show Swap: 0B.

1.4 Load required kernel modules

cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF

sudo modprobe overlay
sudo modprobe br_netfilter

1.5 Configure sysctl for Kubernetes networking

cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1
EOF

sudo sysctl --system

1.6 Install containerd (container runtime)

# Docker's apt repo ships the most up-to-date containerd.io package
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | \
  sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
sudo chmod a+r /etc/apt/keyrings/docker.gpg

echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] \
  https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

sudo apt-get update
sudo apt-get install -y containerd.io

Write a proper default config and enable the systemd cgroup driver (required for kubelet on systemd-based distros):

sudo mkdir -p /etc/containerd
containerd config default | sudo tee /etc/containerd/config.toml > /dev/null
sudo sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml

sudo systemctl restart containerd
sudo systemctl enable containerd

1.7 Install kubeadm, kubelet, kubectl

Pin to a specific minor version so all 3 nodes stay in lockstep. This example uses v1.30 — adjust if a newer stable is out when you run this.

K8S_MINOR="v1.30"

sudo mkdir -p /etc/apt/keyrings
curl -fsSL https://pkgs.k8s.io/core:/stable:/${K8S_MINOR}/deb/Release.key | \
  sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg

echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/${K8S_MINOR}/deb/ /" | \
  sudo tee /etc/apt/sources.list.d/kubernetes.list

sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl   # prevent accidental upgrades
sudo systemctl enable --now kubelet

2. Bootstrap the control plane (run on cp-1 ONLY)

Pre-pull images to fail fast if anything is wrong:

sudo kubeadm config images pull

Initialize the control plane. Use the control-plane node's IP for --apiserver-advertise-address.

sudo kubeadm init \
  --apiserver-advertise-address=10.0.0.10 \
  --pod-network-cidr=192.168.0.0/16 \
  --service-cidr=10.96.0.0/12 \
  --upload-certs

When it finishes you'll see two important things:

1. A kubeadm join ... command for workers — save it, you'll need it in step 4.
2. Instructions to set up kubectl. Run them as your regular (non-root) user on cp-1:

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Sanity check:

kubectl get nodes
# cp-1 will show NotReady until the CNI is installed (next step)

3. Install Calico CNI (run on cp-1)

Use the operator-based install (current recommended path):

kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.28.0/manifests/tigera-operator.yaml

Create the Installation CR with the pod CIDR matching kubeadm init:

cat <<EOF | kubectl apply -f -
apiVersion: operator.tigera.io/v1
kind: Installation
metadata:
  name: default
spec:
  calicoNetwork:
    ipPools:
      - blockSize: 26
        cidr: 192.168.0.0/16
        encapsulation: VXLANCrossSubnet
        natOutgoing: Enabled
        nodeSelector: all()
---
apiVersion: operator.tigera.io/v1
kind: APIServer
metadata:
  name: default
spec: {}
EOF

Wait for Calico to come up:

watch kubectl get pods -n calico-system
# Ctrl+C once all pods are Running

kubectl get nodes should now show cp-1 as Ready.

4. Join the workers (run on node-1 and node-2)

Run the kubeadm join command that kubeadm init printed, as root. It looks like:

sudo kubeadm join 10.0.0.10:6443 \
  --token <token> \
  --discovery-token-ca-cert-hash sha256:<hash>

Lost the command? Regenerate it on cp-1:

kubeadm token create --print-join-command

Back on cp-1, verify:

kubectl get nodes -o wide
# All three nodes should be Ready within ~30-60s

5. Smoke test

kubectl create deployment nginx --image=nginx --replicas=3
kubectl expose deployment nginx --port=80 --type=NodePort
kubectl get pods -o wide        # pods should spread across workers
kubectl get svc nginx           # note the NodePort (30000-32767)
curl http://<any-node-ip>:<nodeport>

Clean up: kubectl delete deployment nginx && kubectl delete svc nginx.

6. Production hardening (do these before putting real workloads on it)

6.1 Back up etcd regularly

# On cp-1
sudo ETCDCTL_API=3 etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  snapshot save /var/backups/etcd-$(date +%F-%H%M).db

Put it on a cron / systemd timer and ship snapshots off-box.

6.2 Understand your single-point-of-failure

With 1 control plane, losing cp-1 means losing the API server (workloads keep running, but you can't schedule or change anything). For real production, plan to expand to 3 control plane nodes behind a load balancer (kube-vip or an external LB). The cluster is designed so you can do this later with kubeadm join --control-plane.

6.3 RBAC & auth

• Don't distribute /etc/kubernetes/admin.conf to humans — generate per-user certs or hook up OIDC.
• Apply default-deny NetworkPolicy per namespace; Calico enforces them.
• Enable audit logging on the API server (edit /etc/kubernetes/manifests/kube-apiserver.yaml).

6.4 Node hardening

• Enable ufw or stick with your cloud SG, but verify the ports in section 0 are open node-to-node.
• Enable unattended-upgrades for security patches only (not kernel/containerd without a maintenance plan).
• Run kube-bench against each node to check CIS compliance.

6.5 Ingress & storage (next steps)

• Ingress: install ingress-nginx or Traefik and put a TCP LB (MetalLB for bare metal, your cloud LB otherwise) in front.
• Storage: for dynamic PVs on bare metal, Longhorn, OpenEBS, or Rook/Ceph. For single-node lab PVs, local-path-provisioner is fine.
• Metrics: install metrics-server so kubectl top works.
• Observability: kube-prometheus-stack (Prometheus + Grafana + Alertmanager).

6.6 Upgrades

Upgrade one minor version at a time, control plane first, then workers, using kubeadm upgrade plan / kubeadm upgrade apply. Always snapshot etcd first.

7. Troubleshooting quick-reference

Useful one-liners:

kubectl get events -A --sort-by=.lastTimestamp | tail -50
kubectl describe node <node>
sudo crictl ps -a                  # containers via containerd
sudo journalctl -u kubelet -f      # live kubelet logs

8. Reset / start over

If something goes sideways and you want a clean slate:

# On every node
sudo kubeadm reset -f
sudo rm -rf /etc/cni/net.d /var/lib/cni /var/lib/kubelet/* ~/.kube
sudo iptables -F && sudo iptables -t nat -F && sudo iptables -t mangle -F && sudo iptables -X
sudo systemctl restart containerd

Then start again from section 2.