Preserve Digital Evidence During an Incident

Operating objective

Preservation must keep enough trustworthy context to answer a defined incident question. It must also avoid unnecessary service damage, evidence destruction, personal-data collection, and delay to urgent containment. The evidence plan states the question, authorized collector, systems, time window, volatility order, method, storage, access, retention, and known limits.

Do not collect every available source without a purpose. Start with the facts needed to bound authority, affected resources, actions, persistence, and recovery.

Signals and evidence

For an identity-aware request, preserve identity-provider events, authenticator and recovery changes, session and token metadata, issuer and key state, Pomerium access and decision records, route and policy versions, deployment and control-plane changes, DNS and network context, time state, upstream application records, and final object actions. Capture configuration, schema, software version, and documentation needed to interpret each field.

Order collection by volatility and consequence. Record source identifier, owner, authorization, collection time, event time, clock source, tool and version, exact command or API request, filters, result count, errors, size, digest, destination, and every transformation. Preserve originals. Analyze verified working copies. For managed services that regenerate exports, record the query, export job, provider metadata, and limitation.

Response and recovery

1. State the incident question and immediate containment limit.
2. Identify volatile and high-value sources. Preserve them before they rotate, expire, or change when delay is safe.
3. Record collection authority and start an evidence register with unique identifiers.
4. Capture time, configuration, version, and schema context with each source.
5. Transfer evidence through authenticated, access-controlled storage and verify digests where meaningful.
6. Preserve the original. Record parsing, filtering, decompression, normalization, redaction, and derived outputs.
7. Build a timeline that distinguishes event time, receipt time, processing time, and analyst time.
8. Corroborate important conclusions with independent evidence and record contradictions.
9. Apply access, retention, deletion, and disclosure rules to originals and working copies.
10. After response, verify custody, reproducibility, disposition, and remaining evidence gaps.

Design tradeoffs and residual risk

Live collection preserves volatile state and changes the system. Isolation protects evidence and can interrupt service. Full images preserve options and collect unrelated sensitive data. Fast provider exports are convenient and can omit records, normalize fields, or change between requests. Strong access limits protect evidence and can slow an urgent investigation.

Residual risk includes compromised source logs, missing direct-path records, bad clocks, short retention, incomplete provider exports, encrypted data without recoverable keys, and application actions that infrastructure evidence cannot reconstruct.

Pomerium boundary

Pomerium can emit records about traffic and decisions that it processes. Operators own export, transport, storage, time correlation, custody, retention, access, privacy, and correlation with other systems. Preserve the applicable Pomerium configuration, policy, route, key, version, and deployment state with the records. A digest of a log file does not prove that the original event source was complete or truthful.

Exercise

In staging, simulate an unauthorized request followed by a policy change and a session revocation. Preserve the relevant identity-provider, Pomerium, configuration, and application evidence. Make one source arrive late and one clock differ by three minutes.

Give the collection to another analyst. They must reproduce the timeline, find the same supported conclusion, identify the same gap, and verify every transformation without access to your working notes.

Evaluation checklist

• Does collection start from a defined incident question and authorized scope?
• Are volatile sources, original state, time context, configuration, schema, and collection errors recorded?
• Can every transfer, access, transformation, redaction, and derived output be reproduced?
• Are material conclusions corroborated and contrary evidence preserved?
• Are sensitive data, retention, access, disposal, and unavoidable evidence gaps explicit?