Two Bad Extremes

Talk to operations teams running regulated software and you will find two opposite philosophies, both wrong.

The first: ‘never touch production automatically.’ Every anomaly, however routine, waits for a human. This sounds conservative but produces its own failure mode — long outages for problems a script could fix in seconds, weekend escalations that burn out the team, and pressure to grant broad ad-hoc access during incidents, which is precisely when mistakes happen. Manual heroics are not a control; they are an uncontrolled variable with a login.

The second: ‘automate everything.’ Modern orchestration platforms make it easy to auto-restart, auto-scale, auto-rollback, auto-repair. Applied indiscriminately in a GxP context, this is worse — automation making consequential changes to a validated system with no record of what it did or why, and no human judgment applied to anomalies that were actually symptoms of something serious.

We run DnXT — a multi-tenant platform where the workloads are regulatory submissions and controlled documents — on a third model. It fits in one sentence: automate only remediation that is safe and reversible, write an audit record for every automated action, and route everything else to a human. The rest of this post is what that sentence means in practice.

The Boundary: What Qualifies for Automatic Healing

A remediation qualifies for automation only if it passes all three tests:

  • It is safe: the action cannot make the situation worse, cannot destroy data, and cannot mask a defect that needs fixing.
  • It is reversible or reconstructive: it rebuilds derived state from a source of truth, or restores a known-good configuration — it never modifies the source of truth itself.
  • It is deterministic: given the same trigger, it does the same thing. No judgment calls encoded in scripts.

Concretely, the things our platform heals automatically are all in the category of derived state:

  • Cache rebuilds. Cumulative views and lookup caches are projections of the underlying records. If a cache is stale or inconsistent, rebuilding it from the platform data layer is always safe — the worst case is spent compute.
  • Configuration and tenant reloads. When a service’s in-memory view of tenant configuration drifts from the stored configuration, reloading from source is safe by construction.
  • Schema conformance on startup. Services verify expected structural properties at startup and add what is missing — additive, idempotent operations only. Anything that would modify or remove existing structure is out of bounds.
  • Container restarts on failed health checks. The classic, bounded by restart budgets so a crash loop escalates instead of cycling forever.

Note what is absent: nothing on this list edits a regulatory record, resolves a data conflict, or decides which of two disagreeing values is correct. Those require judgment, and judgment is a human job.

Every Heal Leaves a Record

The second half of the model is non-negotiable: an automated action that leaves no trace is indistinguishable from tampering. Every heal — every cache rebuild, every configuration reload, every startup conformance fix — writes an audit entry: what was detected, what action ran, what changed, and when.

This does three jobs at once:

  • Compliance. Under 21 CFR Part 11 and ALCOA+ expectations, system-initiated changes are still changes. Our audit trail treats the platform itself as an actor whose actions are recorded exactly like a user’s.
  • Diagnostics. Frequency is signal. A cache that self-heals once a quarter is healthy resilience; one that self-heals hourly has an upstream defect the healing is hiding. Because heals are recorded, we can see the pattern and fix the cause — automation that conceals its own activity converts defects into permanent invisible overhead.
  • Trust. When a customer or auditor asks ‘has anything touched this tenant’s data pathways?’, the answer is a query, not an investigation.

Heal Endpoints: Remediation as a Product Surface

A practice we have come to consider essential: remediation logic lives in first-class, named endpoints — not in shell scripts on an operator’s laptop. A cumulative-view rebuild, for example, is an authenticated administrative operation with defined scope, permission checks, and audit output.

The contrast matters. Ad-hoc incident-time scripts are unreviewed, unrepeatable, and invisible to the quality system. A heal endpoint is code-reviewed once, tested, permissioned, and then every subsequent use is consistent and recorded. When an incident does need human intervention, the human’s toolbox is the same governed surface the automation uses — nobody is hand-editing state at 2 a.m. This aligns the operational reality with the security and access model the platform promises on paper.

Everything Else Escalates

Outside the safe-and-reversible boundary, the automation’s only job is to notice loudly. Data integrity anomalies, authentication irregularities, validation failures, anything touching the record itself — these page a human with context attached: what was observed, what the automation did not do, and why.

We are deliberate about keeping this list conservative. Every incident retrospective asks whether some remediation could move into the automated tier — and the answer is yes only when we can prove the three tests hold. The boundary moves slowly, on evidence, in one direction. That is how confidence in automation is supposed to accumulate in a validated environment: the same change-controlled way anything else does.

What to Ask Your Vendor

  • What does your platform remediate automatically, and what is the explicit rule that bounds the list?
  • Where do I see the record of automated actions affecting my tenant?
  • Are remediation operations governed endpoints, or scripts run by whoever is on call?
  • How does a recurring self-heal get noticed and root-caused rather than normalized?

A mature answer describes a boundary and an audit surface. An immature one describes either heroics or magic.

Resilience and compliance are usually framed as a tradeoff. Built this way, they are the same discipline: the system keeps itself healthy within limits it can prove are safe, and tells the truth about everything it does. If you want to see the audit trail of a platform that operates like this, talk to a regulatory expert.