The Regression Problem Nobody in RegTech Talks About

Every eCTD publishing platform faces the same uncomfortable question: when you change the software, how do you know you did not subtly break the output? Unit tests verify functions. Integration tests verify workflows. But the artifact that actually matters — the published sequence, with its folder structure, its backbone XML, its checksums, its lifecycle operations — is judged by exactly one authority that counts: the health agency’s validator, months later, when the submission is filed.

That feedback loop is far too slow and far too expensive to serve as your quality signal. A regression that quietly changes how a lifecycle operation resolves, or drops a leaf reference under rare conditions, may not surface until a sponsor’s submission bounces. (For a primer on what these structures are, see What Is eCTD?.)

Our answer to this problem borrows an old idea from software engineering — characterization testing — and applies it at the scale of a real regulatory archive. The premise: a correctly published eCTD sequence is its own test oracle. It already encodes the right answer. You just have to use it.

The Harness: Harvest, Replay, Diff

The characterization harness works in three stages:

1. Harvest

We collect real, previously published sequences — outputs that were accepted as correct, with their complete on-disk structure: index files, regional backbones, checksum manifests, and folder trees. Each becomes a frozen reference: for these inputs, this is the known-good output.

2. Replay

The same source inputs are pushed through the current build of the platform — import, lifecycle resolution, publication — producing a fresh sequence with today’s code.

3. Semantic diff

The naive approach — byte-comparing old and new output — fails immediately, because legitimate differences exist (timestamps, generated identifiers). The diff has to be semantic: parse both outputs into their regulatory meaning and compare that. Same leaf set? Same lifecycle resolutions? Same folder placement? Same checksum integrity? Differences that matter surface; differences that do not are filtered.

We ran this approach across more than 3,000 harvested sequences. At that scale, you stop testing the happy path and start testing the archive’s full weirdness: unusual submission types, deep lifecycle chains, study tagging files, edge-case regional structures that no hand-written test plan would think to include.

Fifteen Invariants: What ‘Intact’ Means, Precisely

Alongside the diff, every replayed sequence is checked against a set of fifteen integrity invariants — properties that must hold for any valid output regardless of content. A sample of the family:

  • Every leaf reference in the backbone resolves to a file that exists on disk — no dangling references.
  • Every file on disk is reachable from the backbone — no orphaned content.
  • Checksums in the manifest match the actual files.
  • Lifecycle operations resolve coherently: a replaced document’s predecessor exists; the cumulative view reflects the operation.
  • Folder structure conforms to the regional specification, including the study-tagging-file substructure.

Invariants are the safety net under the safety net. A semantic diff can only compare against the reference; invariants catch cases where the reference itself masked a latent issue, or where new functionality produces structures the archive never contained.

What It Caught

The honest measure of any testing approach is its catch record. Two examples from ours:

  • A dangling leaf reference produced under a specific lifecycle edge case — the backbone cited a file that was not in the output. That defect class maps directly to a well-known FDA validation failure for missing referenced files. Our harness flagged it in minutes on a build that had passed conventional tests; the agency’s validator would have flagged it in a sponsor’s submission. (Our guide to eCTD validation rules covers why these errors are rejection-grade.)
  • A folder-naming collision in study tagging file handling, where two studies with similar identifiers could contend for the same folder slug — the kind of defect that appears only with real-world data variety, which is exactly what a harvested archive provides.

Both were fixed before any customer or agency ever saw them. That is the entire economic argument for the harness in two bullet points.

Why This Matters More in Regulated Software

Characterization testing is a known technique, but it earns special status in RegTech for three reasons:

  • The cost asymmetry is extreme. A rejected sequence costs a sponsor weeks of delay on a program where every week is expensive. Catching the same defect internally costs a bug ticket.
  • The oracle really is authoritative. In most domains, deciding what ‘correct output’ means is itself hard. Here, an accepted published sequence is as close to ground truth as software gets — and we additionally benchmark our validation engine against the industry-standard external validator to keep our definition of correct calibrated.
  • It compounds with releases. The whole point of a modern platform is to ship improvements continuously. Continuous shipping without archive-scale regression safety is how publishing platforms quietly break; with it, every release is re-proven against thousands of real sequences.

What Buyers Should Take From This

You will rarely see ‘characterization harness’ on a vendor’s feature list, because it is not a feature — it is the quality system behind the features. But you can probe for it. Ask a vendor: how do you know a new release still publishes yesterday’s submissions identically? How many real sequences does that proof run against? What integrity properties are checked on every output? The answers separate platforms engineered for regulatory stakes from platforms that test like ordinary software and let sponsors discover the difference.

If you want to see the output side of this discipline — sequences that publish clean and validate clean — talk to a regulatory expert. We are happy to go as deep on the engineering as you want to go.