# Fit and Readiness Score validation plan, and shadow mode

**Protocol v2 · 2026-09-05 · implements Part III of
`docs/fit-and-readiness-scores-brief.md`.** v2 is v1 with one change, made
before any data were collected under v1: JointVoyage operates independently
of Total Joint Specialists, so the development cohort is JointVoyage's own
patients and no outcome data arrives from anyone else. What changed is marked
**[v2]**; §10 logs it. v1's text is preserved in git history rather than
duplicated here. This document is the analysis
plan. It is written before the data it describes exist, and the code that
collects that data (v1.147.0.0) is the same release. Changing an endpoint, a
threshold, a candidate model or an analysis after data are in requires a new
protocol version with the change and its reason stated; the old version stays
in this file.

What this plan is not: a promise that the scores will be found valid. The
brief's last principle is that every consequential result returns to human
judgment, and the point of a prespecified plan is that the judgment is made on
evidence that was named in advance.

---

## 1. What "shadow mode" means here

The brief asks that the revised system run "in shadow mode without authority
to release a case for surgery". JointVoyage has never released a case: a
surgeon's named acceptance is what the readiness status ladder waits for, and
the Fit Score has never been an eligibility decision. So shadow mode is not a
switch that turns authority off. It is the set of records written beside the
live product, read by nobody but the validation report:

| Record | Written when | Written by | Read by |
|---|---|---|---|
| `fit_shadow` | every saved assessment | the platform, from the stored answers | the validation report only |
| `blinded_triage` | a surgeon records their own read of a case before the Fit Score is shown, or chooses to see the score first | the surgeon | the validation report only |
| `readiness_audit` | a person audits a record the arithmetic called complete | the surgeon or a clinical administrator | the validation report only |
| comprehension counters | a patient answers the one-question check on a result page | anonymous counter, keyed by what the page claimed | the validation report only |

The one visible effect of shadow mode is the blinded read: the physician
facesheet holds the Fit Score and its claim back until the surgeon has
recorded their read or chosen to look first. Either choice is one click and
both are recorded; a read taken after looking is kept and marked unblinded.

Nothing in shadow mode changes a score, a claim, a readiness status, or any
sentence a patient or surgeon is shown. `lib/validation.ts` and
`lib/fit-shadow.ts` hold the code; `validation_records` holds the rows;
`tests/validation.test.mjs` pins that no rendering surface imports either
module's candidate scores.

## 2. Intended use, stated per output (brief §11.1)

The four outputs the patient sees answer four different questions and are
validated against four different standards. Conflating them is the error the
brief's Part I exists to prevent, so it is refused here at the outset.

| Output | Intended use | Validated against |
|---|---|---|
| **Joint Impact** (100 − HOOS JR / KOOS JR) | describes current impairment | nothing new: the instrument is published and validated; we check only that our scoring matches the crosswalk (`tests/fit-score.test.mjs`) |
| **Indication Evidence** (four checks) and the **claim** the page leads with | referral triage: whether the usual replacement indication is established, and what would settle it | blinded independent surgeon assessment (§5) |
| **Fit Score** (the number) | a descriptive summary, shown with its parts; **not** an outcome prediction and **not** an eligibility decision | descriptive only in this protocol; its relationship to outcomes is measured (§6) but no probability claim is made or planned from these data alone |
| **Readiness Score** and the **status ladder** | preparation progress, and whether the surgical team has confirmed | workflow and safety audit (§7); no prediction model |

The Fit Score will **not** be presented as a probability of benefit under this
protocol. §6 describes what would have to be true before a later protocol
could even propose one.

## 3. Cohorts

- **Every episode** with a saved assessment enters the shadow cohort. There is
  no consent gate on `fit_shadow` records: they are JointVoyage's own
  operating data about its own arithmetic and are never shown to a surgeon or
  published as a cohort (the same rule the pathway dashboard states).
- **Hip and knee are separate cohorts** for every analysis in this plan
  (brief §11.2). No pooled estimate is reported without its two halves.
- **Patients who do and do not undergo surgery** both stay in (brief §11.4).
  A "did not proceed" is an outcome, recorded from the episode (no surgery
  date twelve months after the assessment), not an exclusion.
- **Outcome analyses** (§6) use only episodes with research consent on file
  (`research-consent.ts`), because they join to postoperative PROMs, which are
  registry data.
- **[v2] Every outcome is JointVoyage's own.** There is no partner EMR to draw
  from and no practice whose follow-up can stand in for ours: a PROM exists
  because a JointVoyage patient recorded one. That makes outcome COLLECTION a
  first-class part of the product rather than an analysis detail, and §6's
  gate is a countdown the platform reports on itself
  (`cohortReadiness` in `lib/score-ledger.ts`, `/api/validation-export`).

## 4. Candidate Fit models, pre-registered (brief §13)

Scored on every save by `lib/fit-shadow.ts`, `SHADOW_MODELS_VERSION` 1. Shown
to nobody. The live score is the control.

| id | Origin | Rule |
|---|---|---|
| `live_v2` | the live score, `FIT_SCORE_VERSION` 2 | weighted mean over answered domains; a dropped domain's weight re-shared; recall imaging at half weight |
| `cap_unread_imaging` | external memo (declined for display, brief §3.9) | live score capped at 49 on patient-recall imaging, 39 with no grade |
| `no_renormalize` | brief §1.3 taken as arithmetic | value × nominal weight over the full 100; a dropped domain contributes nothing and its weight is not re-shared |
| `treatment_doubled` | external memo (declined, brief §3.7) | treatment history at twice its weight, taken from joint health |
| `joint_impact_only` | brief §3.1 | 100 − the JR instrument, nothing else |

A candidate added after any outcome data exist is not a hypothesis and will
be reported as exploratory, in its own section, never beside these five.

## 5. Triage validation: blinded surgeon reads (brief §11.5)

**Question.** Does the claim the result page leads with agree with what a
surgeon reads from the same records, before seeing the score?

**Data.** `blinded_triage` records. The surgeon chooses one of five reads —
candidate on what is here; not yet; confirm the pain source first; a different
pathway; cannot say — and the record stores whether the Fit Score was held
back until the read was taken.

**Primary analysis.** Agreement between the read and the claim, over
comparable pairs (a read of "cannot say" is reported, never scored):

| Read | Agrees with claim |
|---|---|
| candidate | `good` |
| not yet | `borderline`, `not-yet`, `provisional` |
| confirm the pain source | `confirm-pain-source` |
| a different pathway | `other-pathway`, `reassess-imaging` |

Reported as percent agreement and Cohen's κ, blinded and unblinded reads
apart, hip and knee apart. Unblinded reads are a sensitivity analysis, not
the estimate.

**Sample size.** To estimate agreement to within ±10 percentage points
(95% CI) at an expected agreement of 70%: n = 1.96² × 0.7 × 0.3 / 0.1² ≈ 81
comparable blinded reads per joint. To detect a difference between two
claim gates' agreement rates is not planned at this size; that is a later
protocol.

**What a result means.** Low agreement is information about the gates
before it is an argument about the cut points: the first question asked of a
disagreement is which read was right, and that is answered by the outcome
data in §6, not by moving a threshold.

## 6. Outcome measurement (brief §11.6–§11.10)

**Endpoints, collected at the published PROM windows** (`promWindows` in
`lib/prom-instruments.ts`): HOOS JR / KOOS JR change from baseline; the
proportion reaching the published PASS and MCID thresholds (reported as
population benchmarks, brief §3.8); satisfaction and "would choose it again"
as recorded in the outcomes rail; complications and revision as recorded on
the episode; and, for the non-operative half of the cohort, the same JR
instrument at the same intervals.

**Analyses, per joint.** For each candidate model and for the claim gates:
discrimination for reaching PASS at 12 months (c-statistic), calibration
(calibration slope and intercept, with the calibration curve plotted), and
decision-curve net benefit across the threshold range 0.3–0.8. Missingness is
reported by domain and by claim gate before any imputation; the primary
analysis is complete-case with a multiple-imputation sensitivity analysis.

**Subgroups, prespecified.** Joint (always), age band (<55, 55–69, ≥70),
sex, imaging tier at assessment (recall / report / confirmed), treatment
barrier answer (none / contraindicated / inaccessible), and whether the
assessment was saved through the preparation front door.

**Sample size.** A validation of calibration for a binary 12-month outcome
needs a minimum of 100 events and 100 non-events per model per joint (Collins
et al.; Riley et al.). At an expected 65% PASS rate that is ≈ 290 operated
patients per joint with a 12-month PROM; at the observed 12-month completion
rate this is the number that decides when §6 can be run, and it is reported on
the validation dashboard as a countdown, not assumed.

**Causal claims.** No claim of benefit from surgery relative to continued
nonoperative care is made from this observational cohort (brief §11.7). If a
later protocol proposes one, it names its comparison cohort and design first.

**Locking.** Any model that a later protocol proposes for patient-facing use
must be locked, then validated temporally (on episodes saved after the lock
date) or externally before any probability or eligibility claim is shown
(brief §11.10). **[v2]** External now means outside JointVoyage entirely — a
second site, a published cohort, or a registry — not "outside TJS". Temporal
validation is the one JointVoyage can run alone, and the ledger
(`score_observations`) is what makes it possible: the answer sheet is kept
with every score, so a model locked today can be scored over records saved
before the lock without asking a single patient anything again.

## 7. Readiness validation: workflow and safety (brief §12)

Readiness is not a prediction model and is not validated as one.

**Line-by-line review (§12.1).** The requirement definitions
(`lib/readiness-requirements.ts`), the validity windows
(`lib/readiness-validity.ts`, which states that its windows are local policy)
and the teach-back prompts (`lib/education-preop.ts`) are to be reviewed with
arthroplasty surgery, anesthesia, nursing, PT, infection prevention, primary
care, operations and patients. **[v2]** Those reviewers are convened by
JointVoyage; where a window was inherited from a practice protocol it is
re-derived from published guidance or marked as JointVoyage's own local
policy, and none of them is described as anyone else's standard. Each review is recorded as a dated entry in
§10. Until the first review is recorded, the validity windows carry their
existing "illustrative local policy" label.

**Shadow mode (§12.2).** Already the case: no readiness surface releases a
case. The status ladder's top rung requires a named acceptance by a person.

**Audit of every apparent completion (§12.3).** Every record whose eight
parts reach complete, or whose status reaches Ready for Review or Ready for
Surgery, is to be audited by a person against the requirement rows and the
record, with the finding recorded as `readiness_audit`: confirmed, or a
false completion with what was open. The validation dashboard shows how many
apparent completions have and have not been audited; the target is every one.
The **false-completion rate** is the primary readiness safety measure.

**Workflow measures (§12.4), read off the record.**

| Measure | Source |
|---|---|
| requirements still open on Day Zero, and which | the final OTIF each surgery day records (`otif_final`) |
| items open at the lock date | `lockReview` at the lock, from the readiness sweep |
| expired testing at arrival | rows stale by `TESTING_VALID_DAYS` on Day Zero |
| missing transportation / caregiver | `order:transportation`, `order:caregiver` open on Day Zero |
| last-minute cancellations, discharge failures, staff rework | **not on the record today** — to be collected by TJS operations in a dated log and reported beside these; a gap, stated as one |

**Equity monitoring (§12.5).** Time to complete the eight parts and the
readiness order, and which item blocks longest, reported by joint, age band,
sex, insurance (from the financial step), postal-code rurality, portal access
(signed-in vs texted one-tap link), and caregiver availability. Language is
not recorded today and is listed as a gap.

**Windows and exceptions (§12.6).** Every signed exception and every stale
row at arrival is a case for the §12.1 review of the window that produced it.

## 8. Comprehension (brief §11.11, §12.7)

One question on each result page, answered once, anonymous, keyed by what the
page claimed:

- **Fit:** "What does this result tell you?" — probability / approval /
  grade / eligibility / *how closely my situation fits the usual case, and
  what is still missing*.
- **Readiness:** "Does 100 on this score mean you are cleared for surgery?" —
  yes / *no, my preparation is done and my team confirms readiness* / not sure.

The right reading is stated after the answer, so the question teaches as it
measures; only the first answer is counted. **Success criterion:** at least
80% of answers give the intended reading, for every claim kind separately. A
claim kind below 80% is a copy defect on that headline and is fixed as one.

## 9. Governance

- The candidate list, the endpoints, the subgroups and the success criteria
  above are locked with this version.
- The validation dashboard (`/physician/admin/pathway`, "Validation, in
  shadow") is the only reader of shadow data and reports every figure with
  its denominator.
- **[v2] The cohort is exported from one place**, `/api/validation-export`,
  which assembles the ledger, the outcomes and the shadow records into the one
  row-per-episode table these analyses are run on. It carries no name, no
  contact detail and no answer sheet; the answers stay on the platform, where
  re-scoring happens. Access is `JV_CLINICAL_ADMIN_EMAILS`.
- No cut point in `lib/fit-claim.ts`, no weight in `lib/fit-score.ts`, and no
  band boundary moves until §5 and §6 have been run under this or a later
  protocol version and the result is recorded in §10.
- A protocol change is a new numbered version below, never an edit above.

## 10. Log

| Date | Version | Entry |
|---|---|---|
| 2026-09-05 | v1 | Protocol prespecified; shadow records, blinded reads, audits and comprehension checks shipped in v1.147.0.0. No outcome data exist. |
| 2026-09-05 | v2 | JointVoyage operates independently of TJS. Cohort definition amended: the development cohort is JointVoyage's own patients and every outcome is collected by the platform; external validation now means outside JointVoyage entirely; inherited protocol windows are re-derived or marked as local policy. Shipped with the score ledger (v1.148.0.0), which keeps the answer sheet with every score so a model locked later can be validated temporally over earlier records. Changed before any data were collected under v1; no analysis was run under the superseded text. |
| 2026-09-06 | v2 | Export columns added, no protocol change: the recovery loop as counts — check-ins (total, red-flag, from the texted screen, texts sent, first and last postoperative day), Help me now flows (total, same-day, emergency, by flow id), exercises (completed, not completed, paused) and whether the surgeon contact card was confirmed (v1.191.0.0). Counts and enumerated ids only; no pain figure, incision word, note or phone number leaves the platform. No analysis, gate or cut point changes. |
| 2026-09-08 | v2 | Published, no protocol change: the rationale and this plan are now readable at `/physician/validation`, with this document served in full at `/physician/validation/protocol.txt` from the repository copy rather than a duplicate. Prespecification is only checkable if the plan can be read before the results, so the page is indexable and linked from the surgeon's page and from the blinded-read panel itself. No candidate, endpoint, subgroup, gate or success criterion changes. |