18.2 Container Image Security¶

Container images have become the dominant packaging format for modern applications, with over 20 billion container images pulled from Docker Hub alone each month. This ubiquity makes container images a high-value target for supply chain attacks. When you run docker pull nginx or deploy a Kubernetes workload, you inherit not just the application code but every library, binary, and configuration file layered into that image. A vulnerability or backdoor in any layer—from the base operating system to application dependencies—becomes your vulnerability to manage.

The container image supply chain introduces risks distinct from traditional software distribution. Images are opaque binaries that obscure their contents, they accumulate vulnerabilities over time as they sit unused in registries, and their layered architecture means a single compromised base image can affect thousands of derived images. Securing this supply chain requires attention at every stage: selecting trustworthy base images, minimizing attack surface, scanning for vulnerabilities, signing to ensure integrity, enforcing policies at deployment, and monitoring at runtime.

The Container Image Layer Model and Inherited Risk¶

Understanding how container images work is essential for understanding their security properties. A container image consists of a series of read-only layers, each representing a set of filesystem changes from the previous layer. When you build an image with a Dockerfile, each instruction creates a new layer:

FROM ubuntu:22.04           # Layer 1: Base OS (~77MB)
RUN apt-get update && \     # Layer 2: Package updates
    apt-get install -y python3
COPY requirements.txt .     # Layer 3: Requirements file
RUN pip install -r requirements.txt  # Layer 4: Python dependencies
COPY app.py .               # Layer 5: Application code

Each layer is immutable and identified by a cryptographic hash. The final image is the union of all layers, with later layers able to shadow (but not delete) files from earlier layers.

This model has profound security implications. First, inherited risk means your image inherits every vulnerability present in its base image. If ubuntu:22.04 contains a vulnerable version of OpenSSL, your image contains it too—regardless of whether your application uses it. Second, layer accumulation means that files added in early layers remain in the image even if later layers appear to delete them. Running rm /secrets/api-key.txt creates a new layer marking the file as deleted, but the file remains accessible in the earlier layer. Third, update propagation means that when a base image receives a security update, all derived images must be rebuilt to incorporate the fix.

Research from Sysdig's 2023 Cloud-Native Security and Usage Report found that 87% of container images contain at least one high or critical vulnerability, with an average of 15 high/critical vulnerabilities per image. Many of these stem from base images that have not been updated.

Base Image Selection and Hardening¶

The choice of base image establishes the security foundation for everything built on top of it. We recommend evaluating base images against the following criteria:

Provenance and trust: Use images from verified publishers with clear ownership and maintenance commitments. Docker Hub's "Docker Official Images" and "Verified Publisher" programs provide some assurance, but you should verify that images are built from auditable sources. The docker-library/official-images repository documents build processes for official images.

Update frequency and support lifecycle: Choose base images with regular security updates and defined support timelines. Alpine Linux, for example, provides security updates for approximately two years per release (main repository), while Debian stable versions receive five years of support (three years standard plus two years LTS). Abandoned or infrequently updated base images accumulate vulnerabilities.

Size and attack surface: Larger images contain more packages, which means more potential vulnerabilities and more code that could be exploited. A full Ubuntu image contains approximately 77MB in its base form, while Alpine contains roughly 5MB.

Package management and transparency: Prefer base images where you can audit installed packages and their versions. This enables vulnerability scanning and reproducible builds.

Common base image families include:

Hardening a base image involves removing unnecessary packages, applying security configurations, and ensuring updates are applied. Organizations with strict requirements often create golden images—internally maintained base images that incorporate organizational security policies and are regularly rebuilt with the latest patches.

Minimal Images: Distroless and Scratch-Based¶

Distroless images, pioneered by Google, contain only the application and its runtime dependencies—no package manager, no shell, no extraneous utilities. This dramatically reduces attack surface and eliminates entire categories of post-exploitation techniques.

Google publishes distroless images for common runtimes:

# Java application using distroless
FROM gcr.io/distroless/java17-debian11
COPY target/myapp.jar /app.jar
CMD ["app.jar"]

The gcr.io/distroless/java17-debian11 image contains only the JRE and essential libraries—no apt-get, no bash, no curl. An attacker who gains code execution cannot easily download additional tools, establish a reverse shell, or explore the filesystem.

The benefits of distroless images include:

• Reduced vulnerability count: Fewer packages means fewer CVEs to track and remediate
• Smaller image size: Faster pulls, reduced storage costs, smaller blast radius
• Limited post-exploitation options: No shell or package manager for attackers to leverage
• Simplified compliance: Fewer components to audit and document

The limitations are equally important to understand:

• Debugging difficulty: No shell means you cannot kubectl exec into a running container for troubleshooting
• Build complexity: Multi-stage builds required to compile application then copy to distroless runtime
• Compatibility issues: Some applications require files or utilities not present in distroless images

For even more minimal images, scratch-based images contain literally nothing—only what you explicitly add:

# Go application from scratch
FROM golang:1.22 AS builder
WORKDIR /app
COPY . .
RUN CGO_ENABLED=0 go build -o myapp

FROM scratch
COPY --from=builder /app/myapp /myapp
COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
ENTRYPOINT ["/myapp"]

Scratch-based images work well for statically compiled languages like Go and Rust. They require careful attention to dependencies—you must explicitly include CA certificates, timezone data, and any other runtime requirements.

Chainguard Images offer a middle ground: minimal images with shell access available in "dev" variants for debugging, while production variants are distroless. All Chainguard Images are signed with Sigstore and include SBOMs.

Image Scanning: Vulnerabilities, Secrets, and Misconfigurations¶

Container image scanning examines image contents to identify security issues before deployment. Modern scanners detect three primary categories of issues:

Vulnerability scanning identifies known CVEs in operating system packages and application dependencies by comparing installed software versions against vulnerability databases. This is the most mature scanning capability.

Secret detection finds credentials, API keys, and other sensitive material accidentally embedded in image layers. Even if secrets are deleted in later layers, they remain accessible in the image history.

Misconfiguration detection identifies security anti-patterns like running as root, exposing unnecessary ports, or using insecure environment variables.

Major scanning tools include:

Trivy has emerged as the de facto standard for open source scanning, supporting container images, filesystems, Git repositories, and Kubernetes clusters:

# Scan a container image
trivy image myregistry.io/myapp:v1.0.0

# Scan with severity filtering
trivy image --severity HIGH,CRITICAL myregistry.io/myapp:v1.0.0

# Generate SBOM during scan
trivy image --format spdx-json --output sbom.json myregistry.io/myapp:v1.0.0

# Scan for secrets
trivy image --scanners secret myregistry.io/myapp:v1.0.0

Integrate scanning into CI/CD pipelines to catch issues before images reach registries:

# GitHub Actions example
- name: Scan image with Trivy
  uses: aquasecurity/trivy-action@0.33.1
  with:
    image-ref: ${{ env.IMAGE }}
    exit-code: '1'
    severity: 'CRITICAL,HIGH'

While scanning provides critical visibility into your security posture, its true value lies in acting on the findings. A scan report that sits unread provides no protection.

We recommend scanning at multiple points: during CI/CD builds, continuously in registries, and before deployment via admission control.

Image Signing with Cosign and Notary¶

As covered in Section 17.7, cryptographic signing proves that images have not been tampered with and come from trusted sources. Two primary tools serve this purpose:

Cosign (part of Sigstore) provides keyless signing using OIDC identity:

# Sign an image (keyless mode)
cosign sign myregistry.io/myapp@sha256:abc123...

# Sign with attestations (SBOM, SLSA provenance)
cosign attest --predicate sbom.spdx.json --type spdx \
  myregistry.io/myapp@sha256:abc123...

# Verify signature
cosign verify myregistry.io/myapp@sha256:abc123... \
  --certificate-identity "https://github.com/myorg/myapp/.github/workflows/build.yml@refs/heads/main" \
  --certificate-oidc-issuer "https://token.actions.githubusercontent.com"

Notary (the technology behind Docker Content Trust) uses long-lived keys managed by the publisher:

# Enable Docker Content Trust
export DOCKER_CONTENT_TRUST=1

# Push automatically signs
docker push myregistry.io/myapp:v1.0.0

# Pull automatically verifies
docker pull myregistry.io/myapp:v1.0.0

We recommend Cosign for most use cases due to its simpler key management (no keys to manage) and broader ecosystem adoption. Notary remains relevant in environments requiring specific key custody arrangements.

Admission Control: Enforcing Image Policies¶

Signing and scanning provide visibility, but admission control enforces policy by preventing non-compliant images from running. In Kubernetes, several tools implement admission control for images:

Kyverno uses Kubernetes-native policies:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: verify-image-signatures
spec:
  validationFailureAction: Enforce
  rules:
  - name: verify-cosign-signature
    match:
      resources:
        kinds:
        - Pod
    verifyImages:
    - imageReferences:
      - "myregistry.io/*"
      attestors:
      - entries:
        - keyless:
            issuer: "https://token.actions.githubusercontent.com"
            subjectRegExp: "https://github.com/myorg/.*"

Sigstore Policy Controller focuses specifically on Sigstore verification:

apiVersion: policy.sigstore.dev/v1beta1
kind: ClusterImagePolicy
metadata:
  name: require-signatures
spec:
  images:
  - glob: "myregistry.io/**"
  authorities:
  - keyless:
      url: https://fulcio.sigstore.dev
      identities:
      - issuer: https://token.actions.githubusercontent.com
        subjectRegExp: https://github.com/myorg/.*

OPA Gatekeeper provides general-purpose policy enforcement that can include image verification rules.

Admission control policies typically enforce:

• Images must be signed by trusted identities
• Images must come from approved registries
• Images must not contain critical vulnerabilities (verified against recent scan)
• Images must include required attestations (SBOM, provenance)

Runtime Monitoring and Drift Detection¶

Security does not end at deployment. Runtime monitoring detects anomalous behavior that might indicate compromise, while drift detection identifies unauthorized changes to running containers.

Runtime security tools like Falco, Sysdig Secure, and Aqua Runtime Protection monitor system calls and container behavior against expected patterns:

# Falco rule detecting shell in container
- rule: Terminal shell in container
  desc: Detect shell opened in container
  condition: >
    spawned_process and container and
    shell_procs and not user_known_shell_spawn
  output: Shell opened (container=%container.name command=%proc.cmdline)
  priority: WARNING

Drift detection alerts when container filesystems change at runtime. Since container images are immutable, any file modification indicates either legitimate application behavior (in designated writable paths) or potential compromise. Tools like Sysdig and Falco can detect file modifications outside expected paths.

We recommend combining runtime monitoring with read-only root filesystems where possible:

# Kubernetes pod with read-only root filesystem
securityContext:
  readOnlyRootFilesystem: true

Image Update and Patching Strategies¶

Container images require ongoing maintenance as new vulnerabilities are discovered in base images and dependencies. Effective patching strategies include:

1. Automated rebuilds: Trigger image rebuilds when base images are updated. Tools like Dependabot and Renovate support container image dependencies.

2. Regular rebuild schedules: Even without detected vulnerabilities, rebuild images weekly or monthly to incorporate latest security patches.

3. Image lifecycle policies: Automatically delete old images from registries after defined periods, preventing deployment of stale, vulnerable images.

4. Immutable tags: Never reuse version tags (v1.0.0). Use digest-based references (@sha256:...) or unique build identifiers for traceability.

We recommend treating container images like any other software artifact: version them, sign them, scan them continuously, and replace them when they exceed their security shelf life.