17.1 CI/CD Security Principles¶

The SolarWinds attack succeeded because attackers compromised the build system—not the source code repository, not the deployed application, but the infrastructure that transformed code into software. CI/CD pipelines are uniquely powerful: they have access to source code, secrets, signing keys, and deployment credentials. They execute arbitrary code as part of normal operation. A compromised build pipeline can inject malicious code into every artifact it produces. Securing these systems requires principles that recognize their unique position in the software supply chain.

This section establishes foundational security principles for CI/CD pipelines that apply across platforms, from GitHub Actions to self-hosted Jenkins, providing the basis for secure build infrastructure.

The Unique Risk of CI/CD Systems¶

CI/CD systems combine characteristics that make them high-value targets:

The OWASP CI/CD Security Top 10 identifies the most critical risks:

1. Insufficient Flow Control Mechanisms
2. Inadequate Identity and Access Management
3. Dependency Chain Abuse
4. Poisoned Pipeline Execution
5. Insufficient PBAC (Pipeline-Based Access Controls)
6. Insufficient Credential Hygiene
7. Insecure System Configuration
8. Ungoverned Usage of Third-Party Services
9. Improper Artifact Integrity Validation
10. Insufficient Logging and Visibility

These risks share a common thread: CI/CD systems are often treated as trusted infrastructure rather than attack surface requiring defense. Many security teams discover that hardening applications means little if an attacker can compromise the build system and deploy arbitrary code directly to production.

Least Privilege for Build Systems¶

Least privilege means granting only the minimum permissions necessary for a task. For CI/CD, this applies to every component.

Pipeline Permissions:

# GitHub Actions: Minimal permissions
permissions:
  contents: read      # Only read source code
  packages: write     # Write to package registry
  # All other permissions implicitly denied

# NOT this:
permissions: write-all  # Excessive

Credential Scoping:

Implementation by Platform:

GitHub Actions:

# Job-level permissions
jobs:
  build:
    permissions:
      contents: read
    # Cannot push, cannot access secrets by default

  deploy:
    permissions:
      contents: read
      id-token: write  # For OIDC
    environment: production  # Requires approval

GitLab CI:

# Use protected variables for sensitive credentials
variables:
  DEPLOY_KEY:
    value: ""  # Actual value in CI/CD settings
    protected: true  # Only on protected branches
    masked: true     # Hidden in logs

# Restrict job to specific branches
deploy:
  only:
    - main
  environment:
    name: production

Jenkins:

// Use credentials binding with minimal scope
pipeline {
  stages {
    stage('Deploy') {
      steps {
        withCredentials([
          usernamePassword(
            credentialsId: 'deploy-creds',
            usernameVariable: 'DEPLOY_USER',
            passwordVariable: 'DEPLOY_PASS'
          )
        ]) {
          // Credentials only available in this block
          sh 'deploy.sh'
        }
        // Credentials no longer available here
      }
    }
  }
}

Least Privilege Checklist:

# CI/CD Least Privilege Audit

### Repository Access
- [ ] Build jobs have read-only source access where possible
- [ ] Write access limited to specific branches/paths
- [ ] No admin access to repositories from pipelines

### Credential Access
- [ ] Each job accesses only credentials it needs
- [ ] Credentials scoped to specific environments
- [ ] No shared "super" credentials across pipelines

### Infrastructure Access
- [ ] Build systems cannot access production directly
- [ ] Deployment requires separate, scoped credentials
- [ ] No persistent admin access from build systems

### Network Access
- [ ] Outbound access limited to required destinations
- [ ] No inbound access except from authorized sources
- [ ] Build systems isolated from corporate network

Isolation Between Build Jobs¶

Build isolation ensures that one job cannot interfere with or access data from another job.

Why Isolation Matters:

Without isolation: - Malicious job can steal secrets from other jobs - Compromised build can persist and affect future builds - One project can attack another project's build

Isolation Mechanisms:

Container Isolation:

# GitHub Actions: Container job
jobs:
  build:
    runs-on: ubuntu-latest
    container:
      image: node:18
      options: --user 1001  # Non-root
    steps:
      - run: npm ci && npm run build

# GitLab CI: Container isolation
build:
  image: node:18
  script:
    - npm ci && npm run build
  # Each job runs in fresh container

VM-Level Isolation:

# GitHub Actions: Self-hosted with VM isolation
jobs:
  build:
    runs-on: [self-hosted, vm-isolated]
    # Each job gets fresh VM, destroyed after

Cross-Job Protection:

# Prevent jobs from accessing each other's data
jobs:
  job-a:
    runs-on: ubuntu-latest
    steps:
      - run: echo "secret-a" > /tmp/secret

  job-b:
    runs-on: ubuntu-latest
    needs: job-a
    steps:
      # This should NOT find job-a's secret
      - run: cat /tmp/secret || echo "Properly isolated"

Isolation Failures to Watch For:

Ephemeral Build Environments¶

Ephemeral environments are created fresh for each build and destroyed afterward, eliminating persistence attacks.

Ephemeral vs. Persistent Runners:

The Persistence Problem:

On persistent runners: - Malware can hide and persist - Cached credentials can be stolen - Build artifacts can be tampered with - Debugging artifacts may contain secrets

Implementing Ephemeral Runners:

GitHub Actions (Self-hosted):

# Ephemeral self-hosted runner
# Runner deregisters after one job
runs-on: [self-hosted, ephemeral]

# Configuration in runner setup:
# ./config.sh --ephemeral

GitLab CI (Autoscaling):

# config.toml for GitLab Runner
[[runners]]
  name = "ephemeral-runner"
  executor = "docker+machine"
  [runners.machine]
    IdleCount = 0
    IdleTime = 0  # Destroy immediately after use
    MaxBuilds = 1  # One build per machine

Jenkins (Kubernetes):

# Kubernetes plugin: Pod template
apiVersion: v1
kind: Pod
metadata:
  labels:
    jenkins: agent
spec:
  containers:
    - name: jnlp
      image: jenkins/inbound-agent
  # Pod destroyed after build
  restartPolicy: Never

Cache Security with Ephemeral Runners:

# GitHub Actions: Secure caching pattern
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: actions/cache@v3
        with:
          path: ~/.npm
          key: npm-${{ hashFiles('**/package-lock.json') }}
          # Cache keyed to lockfile - changes trigger new cache

      # Even with caching, runner is ephemeral
      # Compromised cache detected by hash mismatch

Audit Logging and Traceability¶

Every action in CI/CD should be logged, traceable, and tamper-evident.

What to Log:

Log Requirements:

## CI/CD Audit Logging Requirements

### Mandatory Fields
- Timestamp (UTC, high precision)
- Actor (user, service, token ID)
- Action (what was done)
- Resource (what was affected)
- Result (success, failure, error)
- Source (IP, trigger source)

### Retention
- Minimum: 90 days online, 1 year archived
- Security incidents: 7 years

### Protection
- Logs shipped to immutable storage
- Tamper detection enabled
- Separate access controls from CI/CD

Platform-Specific Logging:

GitHub Actions:

# GitHub provides built-in audit logs
# Access via: Settings > Audit log

# Additional logging in workflows
- name: Log deployment
  run: |
    echo "::notice::Deployed ${{ github.sha }} to production"

# Send to external SIEM
- name: Ship logs
  uses: some-org/siem-action@v1
  with:
    endpoint: ${{ secrets.SIEM_ENDPOINT }}

GitLab CI:

# GitLab provides audit events
# Access via: Admin Area > Audit Events

# Job-level logging
script:
  - echo "Build started by ${GITLAB_USER_LOGIN} at $(date -u)"
  - # Commands here are logged in job output

# Artifact audit
artifacts:
  paths:
    - build/
  reports:
    junit: test-results.xml

Jenkins:

// Enable audit logging plugin
// Manage Jenkins > Configure Global Security > Audit Trail

pipeline {
  options {
    buildDiscarder(logRotator(numToKeepStr: '100'))
    timestamps()
  }
  stages {
    stage('Build') {
      steps {
        echo "Build ${BUILD_NUMBER} started by ${BUILD_USER}"
      }
    }
  }
}

SLSA Provenance:

SLSA (Supply-chain Levels for Software Artifacts) defines provenance requirements:

{
  "_type": "https://in-toto.io/Statement/v0.1",
  "subject": [
    {
      "name": "artifact.tar.gz",
      "digest": {"sha256": "abc123..."}
    }
  ],
  "predicateType": "https://slsa.dev/provenance/v0.2",
  "predicate": {
    "builder": {"id": "https://github.com/Attestations/GitHubHostedActions@v1"},
    "buildType": "https://github.com/Attestations/GitHubActionsWorkflow@v1",
    "invocation": {
      "configSource": {
        "uri": "git+https://github.com/org/repo",
        "digest": {"sha1": "abc123..."},
        "entryPoint": ".github/workflows/release.yml"
      }
    },
    "materials": [
      {"uri": "git+https://github.com/org/repo", "digest": {"sha1": "..."}}
    ]
  }
}

Network Segmentation¶

Build infrastructure should be network-isolated from both development and production environments.

Segmentation Architecture:

┌─────────────────────────────────────────────────────────────────┐
│                    NETWORK SEGMENTATION                          │
├─────────────────────────────────────────────────────────────────┤
│                                                                  │
│  ┌──────────────────┐    ┌──────────────────┐                  │
│  │   DEVELOPMENT    │    │   PRODUCTION     │                  │
│  │     NETWORK      │    │     NETWORK      │                  │
│  └────────┬─────────┘    └────────┬─────────┘                  │
│           │                       │                             │
│           │ (no direct access)    │ (no direct access)         │
│           │                       │                             │
│  ┌────────▼───────────────────────▼─────────┐                  │
│  │              BUILD NETWORK                │                  │
│  │                                          │                  │
│  │   ┌─────────┐  ┌─────────┐  ┌─────────┐ │                  │
│  │   │ Runners │  │ Artifact│  │ Registry│ │                  │
│  │   │         │  │ Storage │  │         │ │                  │
│  │   └─────────┘  └─────────┘  └─────────┘ │                  │
│  │                                          │                  │
│  └──────────────────────────────────────────┘                  │
│                         │                                       │
│              (controlled egress only)                           │
│                         │                                       │
│  ┌──────────────────────▼──────────────────┐                   │
│  │           EXTERNAL SERVICES              │                   │
│  │  (package registries, APIs, etc.)       │                   │
│  └─────────────────────────────────────────┘                   │
│                                                                  │
└─────────────────────────────────────────────────────────────────┘

Egress Controls:

# Kubernetes NetworkPolicy for build runners
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: build-runner-egress
spec:
  podSelector:
    matchLabels:
      role: build-runner
  policyTypes:
    - Egress
  egress:
    # Allow package registries
    - to:
        - ipBlock:
            cidr: 104.16.0.0/12  # npm, etc.
      ports:
        - port: 443
    # Allow internal artifact storage
    - to:
        - namespaceSelector:
            matchLabels:
              name: artifact-storage
      ports:
        - port: 443
    # Deny all other egress by default

Cloud-Specific Segmentation:

Minimal Build Authority¶

Minimal build authority means the build system should have only the capabilities required to build software—nothing more.

Authority Boundaries:

Separation of Concerns:

BUILD AUTHORITY          vs.         DEPLOY AUTHORITY
─────────────────────────────────────────────────────
Compile code                         Push to production
Run tests                            Modify infrastructure
Create artifacts                     Access production data
Sign packages                        Change production config
Upload to staging                    Access customer data

Implementation Pattern:

# GitHub Actions: Separated build and deploy
jobs:
  build:
    # Build has minimal authority
    permissions:
      contents: read
    outputs:
      artifact-digest: ${{ steps.build.outputs.digest }}
    steps:
      - uses: actions/checkout@v4
      - run: npm ci && npm run build
      - id: build
        run: echo "digest=$(sha256sum dist.tar.gz)" >> $GITHUB_OUTPUT
      - uses: actions/upload-artifact@v3
        with:
          name: dist
          path: dist.tar.gz

  deploy:
    # Deploy has scoped authority, requires approval
    needs: build
    permissions:
      contents: read
      id-token: write
    environment: production  # Requires manual approval
    steps:
      - uses: actions/download-artifact@v3
      # Verify artifact integrity before deploy
      - run: |
          echo "${{ needs.build.outputs.artifact-digest }}" | sha256sum -c
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: ${{ secrets.DEPLOY_ROLE }}

Recommendations¶

For DevOps Engineers:

1. Default to minimal permissions. Start with no permissions and add only what's required. Document every permission and its justification.

2. Use ephemeral runners. Persistent runners accumulate risk. Ephemeral runners eliminate persistence attacks and simplify security.

3. Segment build networks. Build systems should not have direct access to production. Use separate networks and controlled pathways.

For Platform Engineers:

1. Implement comprehensive logging. Every CI/CD action should be logged to immutable storage. You cannot investigate what you didn't record.

2. Separate build and deploy authority. Building software and deploying it are different functions requiring different permissions. Separate them.

3. Design for isolation. Each build job should be isolated from others. Container or VM isolation prevents cross-job attacks.

For Security Practitioners:

1. Audit CI/CD like production. Apply the same security rigor to build systems as production systems. They're equally critical.

2. Verify isolation regularly. Test that jobs cannot access each other's data, that runners are truly ephemeral, and that network controls work.

3. Assume compromise. Design build systems assuming they will be targeted. Minimize blast radius through segmentation and least privilege.

CI/CD security principles form the foundation for everything that follows. Without least privilege, isolation, ephemeral environments, logging, segmentation, and minimal authority, no amount of tooling or process will secure your build pipeline. These principles—applied consistently across platforms—create the conditions for secure software delivery.