Taints and Tolerations

You have a node with GPUs. Most Pods should not run there: they do not need GPUs and would waste the capacity. Only specific machine learning workloads should be scheduled on this node. Taints let you mark the node as restricted and tolerations let specific Pods declare they are allowed through.

A taint is placed on a node. It repels Pods. A toleration is placed on a Pod. It allows the Pod to be scheduled on nodes with a matching taint. The toleration does not attract Pods to tainted nodes: it only removes the repulsion. The scheduler still decides where to place the Pod based on resources and other constraints.

Adding a taint to a node

kubectl taint node sim-worker gpu=true:NoSchedule

The taint format is key=value:effect. This adds a taint with key gpu, value true, and effect NoSchedule.

kubectl describe node sim-worker | grep Taint

You see Taints: gpu=true:NoSchedule. Any Pod without a matching toleration will not be scheduled on this node.

Create a Pod without a toleration:

kubectl run no-toleration --image=busybox:1.36 --restart=Never \
  --command -- sleep 3600
kubectl describe pod no-toleration | grep -A3 Events

The Events section shows Warning FailedScheduling ... 1 node(s) had untolerated taint {gpu: true}. The Pod stays Pending.

The three taint effects

NoSchedule: new Pods without a matching toleration are not scheduled on this node. Existing Pods on the node are not affected.

PreferNoSchedule: the scheduler tries to avoid this node but will use it if no other node is available. A soft restriction.

NoExecute: the strongest effect. New Pods are blocked, and any existing Pod without a toleration for this taint is evicted. This is what Kubernetes uses automatically for node conditions like node.kubernetes.io/not-ready.

Adding a toleration to a Pod

nano toleration-pod.yaml

apiVersion: v1
kind: Pod
metadata:
  name: toleration-pod
spec:
  tolerations:
    - key: gpu
      value: 'true'
      operator: Equal
      effect: NoSchedule
  containers:
    - name: app
      image: busybox:1.36
      command: ['sh', '-c', 'sleep 3600']

kubectl apply -f toleration-pod.yaml
kubectl get pods -o wide

The toleration-pod is scheduled on sim-worker (the tainted node) because its toleration matches the taint. The no-toleration Pod is still Pending.

The Exists operator

The Exists operator in a toleration matches any taint with the specified key, regardless of value:

tolerations:
  - key: gpu
    operator: Exists
    effect: NoSchedule

This Pod tolerates any gpu=*:NoSchedule taint. Useful when the value varies across nodes but you want to target all GPU nodes regardless of the GPU type.

An empty key with Exists is a wildcard that tolerates all taints on a node:

tolerations:
  - operator: Exists

This Pod can be scheduled anywhere. Use this carefully: it bypasses all taints on all nodes.

kubectl delete pod no-toleration toleration-pod
kubectl taint node sim-worker gpu=true:NoSchedule-

The trailing - after the taint specification removes it.

Taints mark nodes as restricted; tolerations are the key that unlocks that restriction. The next lesson covers nodeSelector, the simpler affinity mechanism for directing workloads to specific nodes.