Kubernetes primer on the Tinys

The Tinys cluster is easier to learn in a browser than in a terminal. Treat this page as the first real reading surface. The terminal is for checking the cluster after the terms make sense.

The one idea

Kubernetes is a control system: Git and operators describe the desired state, the API server stores it in etcd, and controllers keep the machines moving toward that state.

flowchart LR
  You[You] --> Laptop[Laptop]
  You --> Spark[Spark jumpbox]
  Spark --> Ansible[Ansible]
  Spark --> Kubectl[kubectl]
  Spark --> Git[GitHub fos-workbench]
  Git --> Flux[Flux controllers]
  Kubectl --> API[Kubernetes API server]
  Flux --> API
  Ansible --> Nodes[Tinys Ubuntu hosts]
  API --> Etcd[etcd cluster memory]
  API --> Controllers[Kubernetes controllers]
  Controllers --> Pods[Pods and services]
  Nodes --> Pods

Read the diagram left to right:

You operate from the laptop and Spark.
Ansible prepares the Tinys as Ubuntu machines.
k3s turns the Tinys into a Kubernetes cluster.
Flux reads fos-workbench from GitHub.
Kubernetes stores cluster state in etcd.
Controllers keep pods, services, jobs, and storage matching the desired state.

What exists right now

Thing	Concrete example	What it means
Operator laptop	Your laptop	Human-facing control surface: browser, terminal, approvals, dashboards.
Jumpbox	Spark	The workstation with repo, SSH, Ansible, `kubectl`, Helm, Flux, and AI workloads.
Node	`eigil`, `ingvild`, `dicte`	One machine Kubernetes can use.
k3s server	All three Tinys	A node that runs the Kubernetes control plane.
etcd member	All three Tinys	A node that participates in the cluster database.
GitOps source	`fos-workbench`	The GitHub repo Flux reads from.
GitOps path	`clusters/tinys`	The path Flux applies to the cluster.
Workload namespace	`fos-workbench`	The first namespace created from Git.

The first command to read

Run this from Spark:

KUBECONFIG=~/.kube/tinys.yaml kubectl get nodes -o wide

The important part should look like this:

NAME      STATUS   ROLES
eigil     Ready    control-plane,etcd
ingvild   Ready    control-plane,etcd
dicte     Ready    control-plane,etcd

Read that as:

Ready means the node can participate in the cluster.
control-plane means it is part of the Kubernetes brain.
etcd means it is part of the cluster memory.

The basic nouns

Node

Bottom line: a node is one machine Kubernetes can use.

Your nodes are the three Tinys:

eigil     192.168.0.24
ingvild   192.168.0.97
dicte     192.168.0.112

When a node is healthy, Kubernetes can schedule pods there. When a node is not healthy, start by checking the host, the k3s service, disk, and network before debugging the app.

Pod

Bottom line: a pod is the smallest workload Kubernetes schedules.

A pod usually contains one container. It gets a pod IP, a node assignment, and any mounted storage. During the smoke test, the HTTP pod ran on ingvild and the PVC writer pod ran on dicte.

Run:

KUBECONFIG=~/.kube/tinys.yaml kubectl get pods -A -o wide

The NODE column tells you where each pod landed.

Deployment

Bottom line: a Deployment keeps a chosen number of pods alive.

If a raw pod dies, it stays dead. If a Deployment wants one replica, Kubernetes creates a replacement pod when the old one disappears.

Flux itself runs as Deployments:

KUBECONFIG=~/.kube/tinys.yaml kubectl -n flux-system get deploy

You should see:

source-controller
kustomize-controller
helm-controller
notification-controller

Service

Bottom line: a Service gives replaceable pods a stable name and cluster IP.

Pods move. Services stay. During the smoke test, this name reached the HTTP pod:

smoke-http.fos-smoke.svc.cluster.local

That name means:

service name: smoke-http
namespace:    fos-smoke
cluster DNS:  svc.cluster.local

Job

Bottom line: a Job runs work until it completes.

Deployments are for long-running processes. Jobs are for run-to-completion work, such as a batch pipeline. The smoke test job printed:

smoke-job-ok

The real repo-shaped version is:

tutorial/deployment/k8s/dlt-dbt-job.yaml

PersistentVolumeClaim

Bottom line: a PVC asks Kubernetes for storage.

Pods are replaceable, but data often should survive the pod. The Tinys cluster currently uses k3s local-path storage. That is good for learning and local rehearsal, but it is not highly available storage.

If a local-path volume is created on dicte, the pod using that volume is tied to dicte unless the data is moved or restored.

The control plane

Bottom line: the control plane is the part of Kubernetes that receives, records, and acts on desired state.

Part	What it does
API server	The front door. `kubectl`, Flux, and controllers talk to it.
Scheduler	Picks which node should run a new pod.
Controllers	Keep desired state true.
etcd	Stores cluster state.

k3s packages these pieces into a smaller operational shape. You interact with one k3s service on each Tiny instead of installing every component by hand.

etcd

Bottom line: etcd is the database where Kubernetes stores cluster memory.

etcd stores facts such as:

which nodes exist;
which namespaces exist;
which Deployments, Services, Jobs, PVCs, and Secrets exist;
which Git revision Flux has applied;
what the desired replica counts are.

That is why etcd snapshots matter. If all etcd data is lost, Kubernetes loses its memory of the cluster.

You already have snapshots copied to Spark:

~/.local/share/fos-workbench/tinys/etcd-snapshots/

HA and quorum

Bottom line: HA means the cluster can survive the failure you designed for; quorum is the majority etcd needs to keep making safe decisions.

With three etcd members:

members:  eigil, ingvild, dicte
majority: 2

Available etcd members	Result
3 of 3	Healthy.
2 of 3	Still has quorum.
1 of 3	No quorum. Control-plane writes stop.
0 of 3	Cluster is down.

The practical rule is simple:

You can lose one Tiny temporarily.
You cannot casually power off two of the three.

HA does not mean nothing breaks. If a pod is running on the failed node, that pod is affected. HA means the cluster has enough remaining brain and memory to recover from one expected failure.

Flux and GitOps

Bottom line: Flux reads Git and makes the cluster match it.

Run:

KUBECONFIG=~/.kube/tinys.yaml flux get sources git -A
KUBECONFIG=~/.kube/tinys.yaml flux get kustomizations -A

The current meaning is:

Flux object	What it means
`GitRepository/flux-system`	Fetch `fos-workbench` from GitHub.
`Kustomization/flux-system`	Apply `./clusters/tinys`.
`Ready=True`	Flux fetched and applied the Git revision.

Because fos-workbench is private from the cluster’s point of view, Flux uses a read-only deploy key. The public key is registered in GitHub. The private key is stored as a Kubernetes Secret in flux-system.

Reconciliation

Bottom line: reconciliation means “compare actual state with desired state, then move actual state closer.”

Example:

Git says Namespace/fos-workbench should exist.
Flux fetches Git.
Flux applies that namespace through the API server.
Kubernetes stores it in etcd.
If someone deletes it manually, Flux recreates it because Git still says it should exist.

That is the operator shift:

Before Flux: run commands and rely on terminal history.
After Flux: change Git, let controllers converge, inspect status.

Networking

Bottom line: Kubernetes gives pods and services their own internal network.

In this cluster:

pod IPs:      10.42.x.x
service IPs:  10.43.x.x

k3s uses Flannel for pod networking by default. CoreDNS gives services stable names such as:

service.namespace.svc.cluster.local

That lets pods call services without knowing which node the backing pod is on.

Azure route

Bottom line: Azure is the scale and reliability target, but the Tinys teach the runtime model first.

Tinys k3s + Flux
  -> Azure reliability bridge
  -> Azure Container Apps for simple hosted runtime
  -> AKS + Flux for enterprise Kubernetes parity

Container Apps is the simpler cloud runtime. AKS + Flux is the enterprise Kubernetes route.

Power posture

Bottom line: lower power first by reducing work, then tune hosts carefully.

Safe first moves:

Measure wall power with a smart plug or meter.
Keep workload replicas low.
Prefer Jobs that finish over idle long-running services.
Avoid unnecessary DaemonSets.
Tune one node before all three.

Risky moves:

powertop --auto-tune can change device power settings that are hard to debug remotely.
NIC power saving can affect SSH and Kubernetes heartbeats.
Suspending two etcd members removes quorum.

Practice path

Run these commands slowly and explain each line back to yourself:

export KUBECONFIG=~/.kube/tinys.yaml

kubectl get nodes -o wide
kubectl get pods -A -o wide
kubectl get namespaces
kubectl get storageclass
kubectl get --raw='/readyz?verbose' | tail

flux get sources git -A
flux get kustomizations -A

For every command, ask:

Am I looking at host state, Kubernetes state, or GitOps state?
Which controller is responsible for keeping it true?
Where would this state be restored from if the cluster broke?