20.2 Internal Communication During Incidents¶

When a supply chain incident unfolds, internal communication determines whether your organization responds as a coordinated unit or as disconnected teams working at cross-purposes. Industry analyses of the SolarWinds response suggest that organizations with established escalation paths, regular briefing cadences, and cross-functional coordination mechanisms contained the impact more effectively than those improvising communication on the fly. Internal communication is not administrative overhead—it is the connective tissue that enables effective response.

Supply chain incidents present particular communication challenges. They often affect multiple applications and teams simultaneously. The technical complexity requires translation for non-technical stakeholders. The scope may be uncertain for extended periods. And the involvement of external parties—upstream maintainers, downstream customers, potentially law enforcement—creates coordination requirements beyond typical security incidents. Establishing communication practices before incidents occur enables your organization to respond effectively when they do.

Escalation Procedures and Chains¶

Escalation procedures define who is notified when, based on incident characteristics. Clear escalation prevents both under-response (where leadership learns about major incidents too late) and over-escalation (where every minor issue reaches the C-suite).

Escalation matrix template:

Supply chain-specific escalation triggers:

Beyond standard severity criteria, supply chain incidents warrant escalation based on:

• Scope uncertainty: When you cannot determine how many systems are affected, escalate early. Scope often grows during investigation.
• Upstream involvement: Incidents requiring coordination with external maintainers or vendors need leadership awareness for potential public implications.
• Customer impact: Any potential impact to customer data or systems triggers customer-facing team involvement.
• Media attention: If the vulnerability is receiving public attention (X/Twitter, tech news, security blogs), communications and executive teams need immediate notification.

Escalation should be automatic, not discretionary. Define clear criteria and empower responders to escalate without seeking permission. The cost of over-escalation is a few unnecessary notifications; the cost of under-escalation can be existential.

Executive Briefing: What Leadership Needs to Know¶

Executives need different information than technical responders. They must make resource allocation decisions, assess business risk, prepare for external communication, and potentially inform the board. Technical details matter less than business impact and response status.

Executive briefing format:

# Incident Status Briefing
**Prepared for**: Executive Leadership
**Date/Time**: 2024-03-15 14:00 UTC
**Incident**: Compromised dependency in payment processing service
**Current Status**: Containment in progress

# Situation Summary (2-3 sentences)
A malicious package was discovered in our payment service dependencies. 
We are currently assessing scope and have implemented initial containment 
measures. No confirmed data exfiltration at this time.

## Business Impact
- Payment processing: Operating normally, under enhanced monitoring
- Customer data: Assessment in progress, no confirmed exposure
- Revenue impact: None currently; potential if we must take systems offline
- Regulatory: GDPR/PCI notification assessment underway with Legal

## Response Status
- [x] Incident confirmed and classified
- [x] Initial containment measures implemented
- [ ] Scope assessment complete (ETA: 16:00 UTC)
- [ ] Root cause identified
- [ ] Remediation complete

## Key Decisions Needed
1. Approve emergency change window for credential rotation (tonight, 02:00-04:00 UTC)
2. Authorize external forensic engagement if internal assessment inconclusive

## Next Briefing
Scheduled for 18:00 UTC or upon significant status change

Briefing cadence:

Adjust cadence based on incident dynamics. Fast-moving situations warrant more frequent updates; stable situations need fewer. Always communicate the next scheduled update time so executives know when to expect information.

What executives do not need:

• Technical implementation details
• Full packet captures or log excerpts
• Play-by-play of every investigative step
• Speculation about attribution (unless confirmed)

Provide these details in appendices or separate technical reports for those who want them, but keep core briefings focused on business-relevant information.

Engineering Coordination: Technical Response Teams¶

Technical response requires coordination across multiple engineering teams—security, infrastructure, application developers, SRE, and potentially specialized groups. Without clear coordination mechanisms, teams duplicate effort, miss critical steps, or work at cross-purposes.

Technical coordination mechanisms:

1. Dedicated incident channel: Create a private Slack/Teams channel for each significant incident. Include all active responders and relevant stakeholders.

Channel naming: #inc-2024-03-15-supply-chain
Pinned: Incident summary, current status, key contacts
Topic: "Supply chain incident - payment service | IC: @jane | Status: Containing"

1. Incident commander role: Designate a single incident commander (IC) responsible for coordinating response. The IC does not need to be the most senior person; they need to be available, organized, and empowered to make decisions.

2. Regular sync calls: For complex incidents, schedule brief sync calls (15-30 minutes) at regular intervals. These provide synchronization points and reduce ad-hoc interruptions.

3. Work stream leads: For large incidents, designate leads for distinct work streams:

4. Containment lead: Stopping the bleeding
5. Investigation lead: Understanding what happened
6. Remediation lead: Fixing affected systems

7. Communication lead: Internal and external messaging

8. Handoff procedures: Incidents lasting more than a work day require shift handoffs. Document current status, pending actions, and open questions at each handoff.

Anti-patterns to avoid:

• Multiple people investigating the same artifact without coordination
• Key findings shared verbally but not documented
• Responders working in isolation without regular synchronization
• Too many people in calls/channels (creates noise; limit to active responders)

Legal and Compliance Involvement¶

Legal and compliance teams require early engagement for incidents with potential regulatory, contractual, or litigation implications. Their involvement shapes response decisions, communication, and documentation.

Legal team engagement triggers:

Working with legal during incidents:

• Early notification: Inform legal counsel when incidents meet any trigger criteria, even if details are uncertain. Legal teams prefer early heads-up over late surprises.

• Privileged channels: For matters where attorney-client privilege is important, use clearly marked privileged communication channels. Not all incident communication should be privileged—only legal advice and analysis.

• Documentation review: Legal may review external communications before publication. Build this into your timeline rather than treating it as an afterthought.

• Notification tracking: Legal tracks notification deadlines (GDPR's 72 hours, state law requirements, contractual SLAs). Provide them with the information they need to assess obligations.

HR Considerations: Insider Threat Scenarios¶

Some supply chain incidents involve insider threats—employees who intentionally introduced vulnerabilities, exfiltrated code, or facilitated external attackers. These scenarios require HR involvement from the earliest indication.

HR involvement triggers:

• Evidence suggesting intentional employee action
• Compromised employee credentials (distinguishing compromise from misuse)
• Social engineering targeting specific employees
• Need to interview employees about their actions
• Potential termination or disciplinary action

Coordination with HR:

When insider involvement is suspected:

1. Notify HR immediately: Before confronting or interviewing the employee, coordinate with HR on appropriate procedures.

2. Preserve evidence carefully: Work with legal on evidence handling that will support potential employment action or criminal referral.

3. Maintain confidentiality: Limit knowledge of insider investigation to those with need-to-know. Premature disclosure can compromise the investigation.

4. Consider access revocation timing: Coordinate the timing of access revocation with evidence preservation needs and HR procedures.

5. Document objectively: Record observations and evidence without conclusions about intent until investigation is complete.

Even when insider involvement is not suspected, HR may need involvement if the incident affects employees (e.g., compromised HR systems, exposed employee data).

Documentation During the Incident¶

Real-time documentation serves multiple purposes: it enables shift handoffs, supports post-incident analysis, provides evidence for legal proceedings, and creates institutional memory. Under-documenting during incidents forces reconstruction from memory later—an unreliable approach.

Real-time documentation practices:

1. Incident log: Maintain a running log of significant events, decisions, and findings:

## Incident Log

### 2024-03-15 14:32 UTC - @jane
Confirmed compromised package version 2.3.1 in production payment-service
SBOM scan shows deployment on 2024-03-10

### 2024-03-15 14:45 UTC - @bob
Network logs show outbound connections to 192.0.2.100 from payment-service pods
Starting correlation with other services

### 2024-03-15 15:03 UTC - @jane
DECISION: Rotating all payment-service credentials immediately
Rationale: Cannot rule out credential exfiltration; cost of rotation < risk
Approved by: @alice (IC)

1. Decision log: Explicitly record significant decisions with rationale and approver. This supports post-incident review and demonstrates due diligence.

2. Action tracking: Maintain a list of assigned actions, owners, and status:

## Action Items
- [ ] Rotate database credentials (@bob, ETA 16:00)
- [x] Block egress to known C2 IP (@network-team, completed 15:15)
- [ ] Notify affected customers (@comms-team, pending legal review)

1. Artifact collection: Save copies of relevant logs, configurations, and evidence to a designated incident folder. Reference these in the incident log.

Documentation tools:

• Incident management platforms: PagerDuty, Opsgenie, FireHydrant provide structured incident tracking
• Collaborative documents: Google Docs, Notion, Confluence for narrative documentation
• Chat archives: Ensure incident channel history is preserved and exportable
• Ticketing systems: Create incident tickets for tracking and historical reference

Communication Channel Security¶

During security incidents, assume that attackers may have visibility into your normal communication channels. Compromised credentials, insider threats, or infrastructure breaches could expose incident response discussions.

Secure communication considerations:

• Out-of-band channels: For sensitive incidents, use communication channels separate from potentially compromised infrastructure. Mobile phones, separate messaging apps, or in-person communication may be appropriate.

• Need-to-know access: Limit incident channels to active responders and essential stakeholders. Broad access increases exposure risk.

• Avoid discussing attribution: Speculation about attackers in written channels can leak and complicate response.

• Protect customer names: When incidents affect specific customers, consider using pseudonyms or identifiers in broad channels.

For most incidents, standard enterprise communication channels are appropriate. Reserve high-security measures for incidents where those channels themselves may be compromised.

Recommendations¶

We recommend the following internal communication practices:

1. Establish escalation matrices before incidents: Define criteria and notification chains now, not during a crisis. Review and update quarterly.

2. Train incident commanders: Invest in incident command training for team leads. Coordination skills are distinct from technical skills.

3. Standardize briefing formats: Use consistent templates for executive updates. Familiarity enables efficient information processing during stressful situations.

4. Engage legal early: Err on the side of earlier legal notification. Legal teams can advise on what is not needed; they cannot undo delayed engagement.

5. Document in real-time: Assign a scribe if the incident commander cannot document while coordinating. The log is invaluable; memory is unreliable.

6. Define communication channel protocols: Establish where different conversations happen (incident channel, executive updates, technical deep-dives) before you need them.

7. Practice through exercises: Tabletop exercises reveal communication gaps that are invisible until tested under pressure.

Internal communication is infrastructure. Build it before you need it, maintain it between incidents, and rely on it when crises occur. Organizations that communicate effectively internally are far better positioned to manage external communication and recover successfully from supply chain compromises.