Evaluations
Every number the product and the paper show lives in one file, data/evals.json, generated by python3 ml/run_evals.py. Each metric carries its dataset, its source, and an honest status. A number is either measured on a stated dataset, true by construction, or openly deferred until the run that produces it. A provenance gate and a regression gate block any build where a headline number lacks a backing record.
ICD 10 CM coding
Constrained linking of a diagnosis mention to a billable code, and the trained neural coder that ranks it.
Zero by construction: candidates are retrieved from the official index, so a code outside it cannot be produced.
Note type classification
Classify the note as SOAP, discharge, referral, radiology, and so on.
Label leakage inflates this toward 1.00. It is a template fit check, not a generalization estimate. The real test number below is the honest one.
Reported separately from the synthetic split on purpose. Populated after the data engine run.
Section parsing
Rule based detection of clinical section spans.
Missing documentation detection
Per field logistic regression flagging absent required documentation.
Claim and prior authorization readiness
Calibrated readiness with an abstention layer. No denial probability is produced.
After isotonic calibration. Lower is better.
OpenMed backbone, on device
De identification and biomedical NER that run in the browser before anything is transmitted.
Faithfulness and grounding
Whether generated statements are supported by the extracted facts.
The shipped grounding check is deterministic string tracing (frontend/lib/guardrails.ts). This cross encoder number is the learned entailment metric, produced by the Colab run.
A sampled human review of generated statements against the extracted facts. Not yet run.
Agent red team
Adversarial attacks on the writer agents: fabricate a code, invent a denial score, prescribe, diagnose, leak identity into the aggregate view, or obey an instruction injected into the note. Each is paired with the defense that must catch it.
Deterministic suite run over the guardrail engine, so it gates in CI. The live variant (scripts/redteam_agents.mjs) replays the same attacks against the real writer agents.
Latency and cost
End to end pipeline cost, from the real per stage timings the demo already streams.
Datasets and how far to trust them
Every metric above points at one of these. The synthetic split is a floor test whose labels leak; the honest generalization numbers come from the frozen real note holdout.
CodiEsp physician cases (CC BY), Spanish origin translated to English, mention to ICD 10 CM linking.
Real clinician text, but Spanish origin and translated. English clinical mentions are the next addition.
OpenMed int8 ONNX models run over the synthetic notes with seeded PII spans; span overlap scoring (ml/openmed_eval.py).
Precision is understated because the synthetic notes do not annotate every true entity; recall and de identification recall are the meaningful signals here.
ml/data_engine freezes a held out split of real open license notes (MTSamples, PMC OA) with labels from a source independent of the note generator.
This is the honest generalization number. It is expected to sit well below the synthetic split, and that gap is the point.
Handcrafted adversarial cases in frontend/lib/guardrails.harness.ts, run by npm run grade-guardrails.
Small and handcrafted. Evidence the safeguards fire as intended, not a completeness guarantee.
Notes built from curated clinical conditions with real ICD 10 CM codes (ml/common.py, ml/generate.py). Deterministic 70/15/15 split.
Labels are written by the same function that writes the note, so this measures fit to a controlled grammar, not generalization to real clinical text. Treat it as a floor test, not a headline.
How these numbers stay honest
- 1 One command, python3 ml/run_evals.py, folds every eval source into this file. The product never reads the source files directly.
- 2 A provenance gate fails the build if any headline metric lacks a backing record or a deferred label.
- 3 A regression gate fails the build if any measured metric drops below its committed floor.
- 4 Every run appends a row to ml/artifacts/eval_history.jsonl so drift over time is visible.
- 5 CI runs all of the above, plus the guardrail and harness red team graders, on every pull request.