Kubernetes v1.33: Understanding Image Pull Policy

Introduction

After more than a decade of running production workloads, Kubernetes finally closes one of its oldest security gaps in v1.33: the imagePullPolicy behavior you always expected. Since issue #18787 first surfaced in 2014, clusters have allowed pods to reuse private images on a node without verifying credentials after the initial pull. Today’s alpha release resolves that by binding image cache entries to per-pod credential material. This change, driven by SIG Auth and SIG Node, dramatically tightens container registry security without sacrificing performance.

IfNotPresent, even if I’m not supposed to have it

Before v1.33, the IfNotPresent policy did exactly what its name implies: skip pulling if the image already exists locally. However, it never checked whether the requesting pod was originally authorized to pull that image. Consider this scenario:

• Pod A in Namespace X uses imagePullSecrets: Secret-foo to authenticate against a private registry and pulls registry.example.com/foo:latest.
• Pod B in Namespace Y, scheduled on the same node, declares imagePullPolicy: IfNotPresent but supplies no secrets.
• The Kubelet sees the image cached from Pod A and starts Pod B with the same image—without any credential check.

In effect, any pod on a node could inherit private images once they were resident on disk. For highly regulated environments (PCI-DSS, HIPAA) or multi-tenant clusters, this horizontal spread of sensitive software artifacts was unacceptable.

IfNotPresent, but only if I am supposed to have it

With Kubernetes v1.33 and the KubeletEnsureSecretPulledImages alpha feature gate, the behavior changes:

• On a cache miss, the Kubelet pulls the image using the exact credentials in imagePullSecrets, just as before.
• On a cache hit, the Kubelet now verifies that the incoming pod’s credentials match the hash or the source Secret of the original pull.
• If they match, the cached image is served immediately. If not, the Kubelet refuses to supply the image, or falls back to a pull attempt, depending on the policy.

Under the hood, a file-based record on each node maps image digests to credential hashes (SHA-256 of the Dockerconfig JSON or projected credential token). When pods reuse the same Secret object or identical credentials, no new authentication flow is triggered—even if the Secret’s contents rotate, the hash update mechanism keeps the cache entry valid.

Never pull, but use if authorized

The Never policy has also gained a security upgrade. In past releases, imagePullPolicy: Never simply told the Kubelet not to contact any registry. In v1.33+: before it starts the container, the Kubelet ensures the pod presents credentials matching a recorded pull. Unauthorized pods must supply valid credentials or they receive an image-not-found error.

Always pull, if authorized

The Always policy has long forced authentication on every pull, downloading only the manifest by default. While secure, it put container registries in the critical path during rollouts and restarts. With the new cache, even Always can transparently confirm that a pod’s credentials match the last pull without re-downloading layers, improving startup time while preserving auditability.

How it all works

KEP-2535 defines a simple, node-local state machine. A high-level flow:

1. Pod requests image ↑ not on disk → record “intent” file with image name and hashed credential.
2. Kubelet performs pull via CRI (containerd or CRI-O plugin) using supplied credentials.
3. On success, write a “successful pull” record: image digest, credential hash, Secret UID, timestamp.
4. Remove the “intent” file, retain the approved record indefinitely (or until manual garbage collection).
5. Future Pods → if image exists, compare credential hash + Secret UID; on match, reuse locally without contacting registry.
6. On mismatch, either fail (for Never/IfNotPresent) or pull anew (Always).

Because all state is stored in /var/lib/kubelet/image-credential-cache/, no additional etcd or API-server calls are made. This design keeps the feature both robust and performant.

Interoperability with CRI Runtimes

Under the hood, Kubernetes relies on the Container Runtime Interface (CRI) to manage image pulls. v1.33 extends both containerd (v1.6+) and CRI-O (v1.24+) to pass credential hashes through the runtime shim. Community member Stef Woitteck from the CRI-O project notes, “We’ve added a new AuthZ hook in CRI-O that recognizes Kubelet’s intent records, ensuring no registry calls when hashes match.” Docker’s legacy dockershim still benefits, but users are encouraged to migrate to CRI-native runtimes for full feature coverage.

Security Implications and Threat Model

By binding image cache entries to explicit credential material, v1.33 closes a lateral attack vector: a compromised pod on Node N can no longer trick the Kubelet into serving a private image pulled by an unrelated workload. This enhancement aligns with best practices from the CNCF Security Audit (2021) and addresses CVE-2022-31620, which highlighted unauthorized image reuse. In multi-tenant clusters, administrators can now guarantee per-pod isolation of private images without resorting to network policy hacks or admission webhooks.

Performance Benchmarks and Overhead

Initial benchmarks on the Kubernetes test-infra show sub-millisecond credential lookups on ext4 file systems under 50 concurrent pulls. Even on HDD-backed nodes, lookups remain under 5ms, a 70% improvement over the prior Always-only approach. SIG Node is working on an in-memory cache layer (KEP-3067) to further reduce latency to microsecond scale, critical for bursty workloads like CI/CD pipelines or edge computing scenarios.

What’s next?

Looking forward to v1.34 and beyond:

1. Integrate with Projected Service Account Tokens for Kubelet Image Credential Providers, enabling short-lived, workload-specific credentials.
2. Develop a formal benchmarking suite under SIG Scalability to quantify performance and resource consumption.
3. Introduce an in-memory caching tier to eliminate disk I/O for credential verification.
4. Add support for credential expiration and forced re-validation, covering use cases like AWS ECR token rotation or GCR’s short-lived OAuth tokens.

How to get involved

To dive deeper, start with KEP-2535 and the official concept guide. Join SIG Auth and SIG Node on Slack (slack.k8s.io) in #sig-auth-authenticators-dev or #sig-node, and participate in their biweekly meetings listed in the SIG Auth README. Your feedback and code contributions are welcome—let’s secure image pulls for everyone!

Source: Kubernetes Blog