Replace stored secrets with workload credentials

Protection need

A workload needs to authenticate to a database, cloud API, gateway, queue, or peer. A long-lived secret copied into a manifest, environment variable, image, node, or generic secret store can be read and replayed outside the workload. The design must minimize how many durable secrets exist, who can retrieve them, where plaintext appears, how long credentials work, and what authority they carry.

Separate the secret record from the credential delivered to a workload. A secret manager can store a database password and still give the workload a reusable password. A projected token or Workload API can instead deliver a short-lived credential bound to current workload identity.

Security objectives and requirements

• Inventory every credential by issuer, subject, audience, authority, storage, delivery, consumer, lifetime, rotation, revocation, and owner.
• Prefer workload-attested, short-lived, audience-bound credentials when the target supports them.
• Keep bootstrap authority narrower than the credentials it can request.
• Deliver credentials through a protected workload-local path with correct file, process, and node permissions.
• Rotate before expiry without writing credentials to logs, metrics, command lines, crash dumps, or backups.
• Remove obsolete credentials and prove the old value fails.
• Define safe behavior when the issuer or delivery service is unavailable.

Kubernetes Secrets are base64-encoded data objects, not encrypted by that encoding. Cluster storage encryption, API authorization, node access, Pod permissions, and application handling determine their protection.

Security invariants and evidence

Only the intended running workload can retrieve or use the credential. The credential is valid only for the intended target and short interval. Deployment source, images, logs, telemetry, and backups contain no credential. Rotation does not require a service restart when the mechanism supports live updates. A terminated workload cannot keep minting or using new credentials beyond the stated bound.

Evidence records identity, credential type, issuer, audience, issue and expiry times, policy version, rotation result, and target decision. Store a safe fingerprint, not the credential.

Failure cases

• A secret appears in source control, a container image layer, shell history, environment dump, or support bundle.
• Any Pod with namespace read access can retrieve another workload's Secret.
• A projected file rotates, but the application reads it only at startup.
• A sidecar or node agent exposes one workload's credential to another process.
• A credential has no audience and works at several services.
• A short-lived token can be refreshed forever through a long-lived bootstrap credential.
• Rotation deploys a new credential but leaves the old one valid indefinitely.
• Issuer outage causes fallback to a shared static key.
• Logs record full tokens during failed authentication.

Scan artifacts and runtime surfaces. Test a terminated Pod, another ServiceAccount, another node process, wrong audience, expired credential, issuer outage, and old credential after rotation.

Design tradeoffs and residual risk

Static secrets work with legacy targets and have high replay and distribution cost. Secret managers improve central custody and audit but do not change the credential's lifetime or portability. Projected and brokered credentials reduce exposure while adding issuer, agent, clock, and renewal dependencies.

Very short lifetimes reduce replay windows and can amplify an issuer outage. Caching improves resilience and extends stale authority. Define the acceptable overlap for each target.

Residual risk includes a compromised workload using its live credential, node or agent compromise, target-side over-permission, unsafe application memory handling, and bootstrap authority. Credential delivery does not replace target authorization.

Pomerium boundary

Pomerium service accounts and route credentials support documented programmatic and upstream access patterns. Operators own credential scope, storage, delivery, rotation, and target validation. Use a Pomerium credential only for its documented audience and route. Do not treat it as a general workload-identity credential for unrelated systems.

Exercise

Inventory the credentials of one Kubernetes workload. Include ServiceAccount token, image-pull credential, database password, cloud credential, TLS key, Pomerium credential, and recovery path. Mark storage and plaintext exposure.

Replace one long-lived key with a projected or federated short-lived credential. Test live rotation, wrong audience, Pod deletion, issuer outage, application reload, log redaction, and old-value rejection. Record the measured last-use time after revocation.

Evaluation checklist

• Is every durable secret justified by a target that cannot accept short-lived identity?
• Can only the intended workload retrieve and use each credential?
• Are audience, authority, lifetime, refresh, and revocation bounded?
• Does the application consume rotations without preserving old values?
• Is outage behavior safe and free of shared static-key fallback?