Recovery model

In the HSM profile the secret kernel terminates inside a hardware module (Kontinuum or a compatible HSM). Secret material — key shares, nonce state, presign artifacts, permit records — lives inside the device across power cycles. The HSM maintains its own protected storage; a daemon restart or a paired-runtime crash does not erase that state. Recovery re-establishes the wire connection and re-derives the node binding from the device identity and the active epoch. Secret state stays inside the hardware boundary throughout.

The device is the single source of truth, and its failure modes are covered by the HSM certification rather than by application-level journaling.

The MPC threshold is the primary recovery mechanism. Because key material is split t-of-n across independent devices, losing one HSM node keeps the key intact: as long as the remaining nodes still form a quorum, signing continues and skips the failed device. RefreshShares issues a re-share to a replacement device, transferring participation without changing the public key or requiring a full key rotation. This is the preferred path when the cluster can still reach threshold.

If the device cannot participate and re-share is not possible (for example, a full rack failure below threshold), the fallback is an encrypted HSM state export produced before the failure. The device wraps its internal secret state — shares, nonce handles, presign artifacts, permit records — under a transport key known only to a set of key custodians. The resulting blob is inert ciphertext; it can be stored in cold storage, replicated across air-gapped sites, or handed to custodians as M-of-N wrapped fragments. Restoring on a replacement HSM requires the custodians to reconstruct the transport key and import the blob into a device that has been admitted to the same security domain and epoch. The epoch binding is re-derived from the device identity after import, and a new DomainManifest records the replacement as an epoch transition.

In the software profile the secret kernel process owns an isolated private store. FileSecretKernelLocalStore commits every LocalDelta atomically to a file before the step result returns to the runtime. On daemon restart the file is read back and the kernel resumes from the last committed checkpoint. Raw secret values stay out of the runtime-visible DeltaSigma; the file store and the kernel process are the boundary.

The operational envelope — permits and step-permit lifecycle records — lives on a separate file at DVM_SECRET_KERNEL_OPERATIONAL_ENVELOPE_PATH, committed atomically alongside the local store but kept on distinct storage. The two files together give a consistent view of which steps committed and which permits were consumed at the point of failure.

The software profile has no hardware to hold state across a physical failure. If the server disk is lost, the secret kernel file store goes with it — and with it the node's key share. As long as the remaining nodes form a quorum, RefreshShares recovers participation the same way as the HSM path: issue a new share to a replacement node without touching the public key. But if the cluster drops below threshold, the only recourse is a prior backup.

The backup discipline mirrors the HSM encrypted export: the local store file is sealed under a separate key before it leaves the server boundary, producing an encrypted blob that can be stored in cold storage or replicated across sites. The sealing key stays separate from the blob. What holds that sealing key depends on the deployment: in cloud setups it is a cloud KMS with IAM-bound access; in on-prem setups it is a local KMS (HashiCorp Vault, network HSM) or a custodian quorum that holds key fragments offline. Restoration brings the blob to a replacement server, unseals it via whichever mechanism was used to seal it, and starts the secret kernel daemon pointing at the recovered file. As with the HSM path, the node binding is re-derived and the replacement is admitted via an epoch transition.

Regardless of how the secret boundary stores its state, operation-level resume follows the same deterministic function. recover_step(k, d) takes a step key k = (operation_id, session_id, phase, slot) and an input digest d. If k is committed and d matches, the committed effect is returned directly. If k is still pending, the step executes and commits. If k is committed but d conflicts, the session closes to Poisoned.

Non-terminal states — PendingApproval, PendingRemote, Rotating, Degraded, Recovered — allow idempotent replay. Terminal states — Aborted, Poisoned, Failed — are final. The outer runtime re-enters via recover_step after any restart; the secret kernel's storage backend determines whether the committed state is still present, but the recovery protocol is identical across HSM and software profiles.

If the outer runtime journal, the secret kernel journal, and the local session descriptors carry different hash descriptors for the same step key, the session closes to Poisoned with a PoisonedSessionWitness rather than attempting a merge. The witness records the conflicting hashes, the last confirmed transition, and the originating boundary — a portable artifact that external parties can verify without access to raw secret state.

The recovery test suite ships with v1 and covers: runtime resumption after a local effect at every commit point, secret kernel crash in a secret step, journal divergence with Poisoned transition, and equivocation from a sender. Each scenario produces a DRTestReceipt confirming the expected artifacts are present and verifiable.