16.2 Secrets Management and Credential Hygiene¶

In 2019, a security researcher discovered that Samsung engineers had accidentally committed AWS credentials and private keys to a public GitLab repository. The exposure included SmartThings source code and internal project credentials. Samsung was fortunate the researcher reported responsibly rather than exploiting the access. According to GitGuardian's 2023 State of Secrets Sprawl report, over 10 million new secrets were exposed in public GitHub repositories in 2022 alone—and private repositories fare little better. Secrets management isn't a nice-to-have; it's a fundamental requirement for supply chain security.

This section provides comprehensive guidance on managing secrets throughout the development lifecycle, from local development through CI/CD to production deployment, including what to do when things go wrong.

Types of Secrets¶

Secrets are any sensitive data that grants access, proves identity, or enables cryptographic operations. Understanding the types helps prioritize protection.

Secret Types Taxonomy:

Risk Hierarchy:

Not all secrets are equal. Prioritize protection:

CRITICAL: Signing keys, root credentials, production database
    ↓
HIGH: CI/CD tokens, cloud admin credentials, encryption keys
    ↓
MEDIUM: Service API keys, internal service tokens
    ↓
LOWER: Development-only credentials, test environment tokens

Common Exposure Vectors¶

Secrets leak through predictable patterns. Understanding these helps prevent exposure.

Vector 1: Secrets in Source Code

The most common exposure—hardcoding secrets in application code:

# DON'T: Hardcoded API key
AWS_ACCESS_KEY = "AKIAIOSFODNN7EXAMPLE"
AWS_SECRET_KEY = "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"

# DON'T: Hardcoded database connection
db_conn = "postgresql://admin:supersecret@prod-db.internal:5432/main"

Real Example: Uber's 2016 breach began when attackers accessed a private GitHub repository and found AWS credentials stored there. The exposed credentials provided access to an S3 bucket containing data on 57 million users.

Vector 2: Secrets in Git History

Even deleted secrets persist in Git history:

# Secret committed, then "removed" - still in history!
git log -p --all -S "AKIA"  # Finds all commits containing AWS keys

Vector 3: Secrets in .env Files

Environment files committed accidentally:

# .env file committed to repository
DATABASE_URL=postgres://admin:supersecret@db.example.com/prod
STRIPE_SECRET_KEY=sk_live_0011223344556677889900
JWT_SECRET=super_secret_jwt_signing_key_2026

Vector 4: Secrets in Logs

Application logging that captures secrets:

# DON'T: Logging request with auth header
logger.debug(f"Request: {request.headers}")  # Includes Authorization header

# DON'T: Logging configuration
logger.info(f"Starting with config: {config}")  # May include secrets

Vector 5: Secrets in CI/CD Artifacts

Build outputs containing secrets:

# Build log exposing secrets
Running: npm install
> Setting NPM_TOKEN=npm_xxxxxxxxxxxxx
> Authenticating to registry...

Vector 6: Secrets in Container Images

Secrets baked into images or layers:

# DON'T: Secret in environment
ENV API_KEY=secret_key_value

# DON'T: Secret in build arg persists in layer
ARG DATABASE_PASSWORD
RUN echo $DATABASE_PASSWORD > /tmp/setup  # Visible in image history

Exposure Statistics:

According to GitGuardian's 2023 State of Secrets Sprawl report (analyzing 2022 data): - 1 in 10 code authors exposed a secret in 2022 - Generic secrets and passwords remain the most frequently exposed type - AWS keys continue to be frequently exposed despite widespread tooling

Organizations frequently discover that production credentials have been committed to repositories for months or years, only uncovered when secrets scanning tools are first deployed. These credentials often remain valid, representing an ongoing security risk.

Secrets Scanning¶

Implement multiple layers of scanning to catch secrets before and after commit.

Pre-Commit Scanning:

Stop secrets before they enter the repository.

gitleaks Setup:

# Install gitleaks
brew install gitleaks  # macOS
# or download from [gitleaks][gitleaks]

[gitleaks]: https://github.com/gitleaks/gitleaks

# Create configuration
cat > .gitleaks.toml << 'EOF'
[extend]
useDefault = true

[allowlist]
description = "Allowlisted files"
paths = [
    '''\.gitleaks\.toml$''',
    '''(.*?)(test|spec)(.*?)\.py$''',
]
EOF

# Set up pre-commit hook
cat > .git/hooks/pre-commit << 'EOF'
#!/bin/bash
gitleaks protect --staged --verbose
if [ $? -ne 0 ]; then
    echo "Secrets detected! Commit blocked."
    exit 1
fi
EOF
chmod +x .git/hooks/pre-commit

detect-secrets Setup:

# Install detect-secrets
pip install detect-secrets

# Create baseline (for existing repos)
detect-secrets scan > .secrets.baseline

# Pre-commit hook configuration (.pre-commit-config.yaml)
repos:
  - repo: https://github.com/Yelp/detect-secrets
    rev: v1.4.0
    hooks:
      - id: detect-secrets
        args: ['--baseline', '.secrets.baseline']

Team-Wide Enforcement:

# .pre-commit-config.yaml
repos:
  - repo: https://github.com/gitleaks/gitleaks
    rev: v8.18.0
    hooks:
      - id: gitleaks

  - repo: https://github.com/Yelp/detect-secrets
    rev: v1.4.0
    hooks:
      - id: detect-secrets
        args: ['--baseline', '.secrets.baseline']

CI/CD Integration:

Scan on every push and pull request:

# GitHub Actions
name: Secrets Scan
on: [push, pull_request]

jobs:
  gitleaks:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - uses: gitleaks/gitleaks-action@v2
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

  trufflehog:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - name: TruffleHog Scan
        uses: trufflesecurity/trufflehog@main
        with:
          extra_args: --only-verified

Historical Scanning:

Scan entire repository history for previously committed secrets:

# Full history scan with gitleaks
gitleaks detect --source . --verbose --report-path secrets-report.json

# Full history scan with truffleHog
trufflehog git file://. --only-verified --json > trufflehog-report.json

Secrets Management Solutions¶

Centralized secrets management provides secure storage, access control, and audit logging.

Solution Comparison:

HashiCorp Vault Example:

# Store a secret
vault kv put secret/myapp/database \
    username="app_user" \
    password="secure_password_123"

# Retrieve a secret
vault kv get -field=password secret/myapp/database

# Generate dynamic database credentials
vault read database/creds/my-role
# Returns: username: v-token-my-role-xxx, password: xxx (auto-rotated)

AWS Secrets Manager Integration:

import boto3
import json

def get_secret(secret_name):
    client = boto3.client('secretsmanager')
    response = client.get_secret_value(SecretId=secret_name)
    return json.loads(response['SecretString'])

# Usage
db_creds = get_secret('prod/database/credentials')
connection = connect(
    host=db_creds['host'],
    user=db_creds['username'],
    password=db_creds['password']
)

Selection Criteria:

Short-Lived Credentials¶

Replace long-lived secrets with short-lived, automatically rotated credentials.

OIDC for CI/CD:

Modern CI/CD platforms support OIDC authentication, eliminating stored cloud credentials:

# GitHub Actions with OIDC (no stored AWS credentials)
jobs:
  deploy:
    permissions:
      id-token: write
      contents: read
    steps:
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789:role/GitHubActionsRole
          aws-region: us-east-1
      # No AWS_ACCESS_KEY_ID or AWS_SECRET_ACCESS_KEY needed!

AWS STS for Temporary Credentials:

import boto3

# Assume role for temporary credentials
sts = boto3.client('sts')
response = sts.assume_role(
    RoleArn='arn:aws:iam::123456789:role/DeployRole',
    RoleSessionName='deployment-session',
    DurationSeconds=900  # 15 minutes
)

# Use temporary credentials
credentials = response['Credentials']
# These expire automatically

Service Account Tokens (Kubernetes):

# Short-lived service account token
apiVersion: v1
kind: Pod
spec:
  serviceAccountName: my-app
  containers:
    - name: app
      volumeMounts:
        - name: token
          mountPath: /var/run/secrets/tokens
  volumes:
    - name: token
      projected:
        sources:
          - serviceAccountToken:
              path: token
              expirationSeconds: 3600  # 1 hour
              audience: my-service

Benefits of Short-Lived Credentials:

• Reduced blast radius: Compromised credential expires quickly
• No rotation needed: New credential issued each use
• Better audit trail: Each credential tied to specific action
• Compliance friendly: Meets credential rotation requirements

Credential Rotation Automation¶

For secrets that must be long-lived, implement automated rotation.

AWS Secrets Manager Rotation:

# Lambda function for database password rotation
import boto3
import secrets

def lambda_handler(event, context):
    secret_id = event['SecretId']
    step = event['Step']

    sm = boto3.client('secretsmanager')

    if step == 'createSecret':
        # Generate new password
        new_password = secrets.token_urlsafe(32)
        sm.put_secret_value(
            SecretId=secret_id,
            SecretString=json.dumps({'password': new_password}),
            VersionStage='AWSPENDING'
        )

    elif step == 'setSecret':
        # Update password in database
        update_database_password(new_password)

    elif step == 'testSecret':
        # Verify new password works
        test_database_connection(new_password)

    elif step == 'finishSecret':
        # Promote pending to current
        sm.update_secret_version_stage(
            SecretId=secret_id,
            VersionStage='AWSCURRENT',
            MoveToVersionId=event['ClientRequestToken']
        )

Vault Dynamic Secrets:

# Configure Vault database secrets engine
vault secrets enable database

vault write database/config/mydb \
    plugin_name=mysql-database-plugin \
    connection_url="{{username}}:{{password}}@tcp(mysql.internal:3306)/" \
    allowed_roles="my-role" \
    username="vault_admin" \
    password="admin_password"

vault write database/roles/my-role \
    db_name=mydb \
    creation_statements="CREATE USER '{{name}}'@'%' IDENTIFIED BY '{{password}}'; GRANT SELECT ON *.* TO '{{name}}'@'%';" \
    default_ttl="1h" \
    max_ttl="24h"

# Application gets new credentials each time
vault read database/creds/my-role
# Credentials automatically revoked after TTL

Rotation Schedule Recommendations:

Emergency Response: What to Do When Secrets Leak¶

Despite best efforts, secrets leak. Rapid response limits damage.

Secret Leak Response Playbook:

# Secret Exposure Response Playbook

### IMMEDIATE (Within 15 minutes)

#### 1. Confirm Exposure
- [ ] Verify the secret is real (not placeholder/test)
- [ ] Determine exposure scope (public repo, logs, etc.)
- [ ] Identify what systems the secret accesses

#### 2. Revoke/Rotate Immediately
- [ ] Revoke the exposed credential
- [ ] Generate new credential
- [ ] Update dependent systems
- [ ] Verify services still function

**Do not wait to investigate before revoking. Revoke first.**

### SHORT-TERM (Within 1 hour)

#### 3. Assess Exposure Window
- [ ] When was secret first exposed?
- [ ] When was it discovered?
- [ ] Who/what could have accessed it?

#### 4. Check for Unauthorized Use
- [ ] Review access logs for the credential
- [ ] Check for anomalous activity
- [ ] Identify any data accessed

#### 5. Contain Blast Radius
- [ ] Identify other secrets potentially at risk
- [ ] Check for lateral movement indicators
- [ ] Lock down related accounts if needed

### MEDIUM-TERM (Within 24 hours)

#### 6. Root Cause Analysis
- [ ] How did the secret get exposed?
- [ ] What controls failed?
- [ ] Who was involved (not for blame, for process improvement)

#### 7. Remediation
- [ ] Remove secret from source (including history if applicable)
- [ ] Add detection for this exposure pattern
- [ ] Update training/documentation

### DOCUMENTATION

#### Required Information
- Secret type and scope
- Exposure vector
- Timeline (exposure → discovery → rotation)
- Impact assessment
- Remediation actions
- Preventive measures implemented

Platform-Specific Rotation:

Git History Remediation:

If secrets are committed, removing them from history is complex:

# Remove file from history (USE WITH CAUTION)
git filter-branch --force --index-filter \
  "git rm --cached --ignore-unmatch path/to/secret-file" \
  --prune-empty --tag-name-filter cat -- --all

# Force push (requires coordination)
git push origin --force --all

# Better: Use BFG Repo-Cleaner
bfg --delete-files secret-file.txt
bfg --replace-text passwords.txt  # Replace text patterns

Important: Rewriting history affects all collaborators and doesn't guarantee the secret wasn't already copied. Always rotate the secret regardless of history cleanup.

Recommendations¶

For Developers:

1. We recommend never committing secrets. Use environment variables, secrets managers, or configuration outside of version control. When in doubt, don't commit.

2. We recommend installing pre-commit hooks. Configure gitleaks or detect-secrets as pre-commit hooks. Make it impossible to accidentally commit secrets.

3. Assume exposure. Treat any secret that touches a developer machine as potentially exposed. Use short-lived credentials wherever possible.

For DevOps Engineers:

1. We recommend implementing secrets managers. HashiCorp Vault or cloud-native solutions should be the only source of production secrets. No more shared password files.

2. We recommend using OIDC for CI/CD. Eliminate stored cloud credentials in CI/CD systems. Modern platforms support identity federation.

3. We recommend automating rotation. Set up automated rotation for any secrets that can't be short-lived. Manual rotation doesn't happen reliably.

For Security Practitioners:

1. We recommend scanning continuously. Run secrets scanning in CI/CD, on repositories, and on build outputs. Assume scanning will find things.

2. We recommend having a response plan. When secrets leak (they will), fast response limits damage. Practice the playbook before you need it.

3. We recommend measuring and improving. Track metrics: secrets found, time to rotation, exposure duration. Use data to drive improvement.

Secrets management is one of the most impactful supply chain security controls. A single exposed credential can undo all other security investments. The goal is defense in depth: prevent exposure through tooling, detect exposure quickly, respond immediately when it happens, and continuously improve based on findings.