In shadow mode the AI runs beside your normal process and its outputs are recorded and assessed, but nobody acts on them. People carry on exactly as before. After about 90 days you have evidence of how the AI behaves on your own data: how often it agrees with your people, where it refuses, and how often a reviewer would have overturned it.
What is shadow mode?
It is a way of trying AI without betting on it. The AI receives the same inputs as the live process and produces its outputs. Those outputs are checked, tiered and recorded exactly as they would be in production, but they are not released to anyone who could act on them. The existing process continues to make every real decision.
Why start here?
Because the usual alternative is to rely on a vendor's claims, or on a short demonstration, and then discover problems in live use. Regulators on both sides of the Atlantic are moving towards the same question: is there enough evidence to trust this model for this specific use? Shadow mode produces that evidence from your own documents, your own questions and your own reviewers, before anything depends on it.
How a 90 day pilot is usually shaped
- Weeks 1 to 3: discover and classify. Inventory where AI is used, tier each use by consequence, and pick one use case with a willing process owner.
- Weeks 4 to 6: define and own. Write the intended use statement, agree who is responsible, identify the data sources and set acceptance criteria.
- Weeks 7 to 10: configure and test. Prepare the approved content, encode the rules, set the tier mapping and test against the acceptance criteria.
- Weeks 11 to 13: operate in shadow. Run live alongside the real process with qualified reviewers, then present the evidence pack to leadership.
What to measure
- Concordance: how often the AI output agrees with what your people actually did or decided.
- Reviewer overturn rate: how often a qualified reviewer would have rejected or changed the output.
- Refusals and escalations: how often the controls stopped an output, and why. A refusal is evidence that a control works.
- Corpus currency: whether the documents the AI relies on are the current effective versions.
- Audit trail completeness: whether every output, refusal and decision can be reconstructed.
Write the exit criteria first
Decide before you start what result would justify going live, what would send you back to redesign, and what would make you stop. If the criteria are written after the results arrive, they will bend to fit the results, and an inspector will see that.
Mistakes we see
- Choosing a high consequence use case for the first pilot. Start with tier 1 or tier 2.
- No named process owner, so nobody acts on what the pilot finds.
- Measuring only accuracy, and ignoring refusals, overturns and record completeness.
- Letting shadow outputs leak into live work, which turns an experiment into an uncontrolled deployment.
- Having no baseline, so there is nothing to compare the AI against.
Frequently asked questions
How long should a shadow mode pilot run?
Long enough to see a representative range of cases. We plan 13 weeks in total, of which about three are live shadow operation, but the right length depends on how often the process produces the cases that matter.
Does shadow mode need validation?
The controls that assess and record the outputs should be specified and tested. Because no live decision depends on the AI during shadow mode, the formal validation effort can be proportionate, and the results become the basis for deciding what validation a live deployment needs.
What happens at the end of a pilot?
You decide to proceed, redesign or stop, using the criteria you set at the start. If you proceed, validation is completed through your own change control and residual risk is accepted by named approvers.
Sources and further reading
- ICH quality guidelines, including Q9(R1) Quality Risk Management
- ISPE GAMP 5 Guide, Second Edition
- European Commission: consultation on draft Annex 22 (Artificial Intelligence), revised Annex 11 and Chapter 4
This article is general information, not legal or regulatory advice. Regulatory texts change, and some discussed here are drafts: check the current official text before relying on any detail.
