receipts for whatever detector you already trust

gaige = calibration and drift receipts for AI measurement: your corpus, your thresholds, honest error bars, and a fingerprint that proves the instrument hasn’t changed underneath you. gaige itself is not a detector. It reports what the detector you already run actually does on your material.

SCOPE

Point gaige at a corpus or probe set and any pluggable scorer. It emits a receipts report: ROC and AUROC with bootstrap confidence intervals, operating thresholds stated two ways (empirical in-sample and conformal with a finite-sample guarantee), per-subgroup error rates with intervals, base-rate harm arithmetic, and a complete instrument fingerprint: model, quantization verified at load, device, library versions, corpus hash, and the exact command that reproduces the number.

One machine, two applications. Detector calibration: the false-positive rate on your own material, not a vendor page. Instrument drift: whether the system you are measuring changed, or your measuring pipeline did. Scoring needs a model; analysis runs anywhere.

Detection is a moving contest between generators and detectors, and nobody stays ahead for long. gaige does not enter that race. It keeps the score honest for whoever is running it, and it stands alone: one command-line tool that plugs into the pipelines you already have.

PROCEDURE

Four steps, no prior knowledge assumed: put your material in, run one command, read the receipt, and check that nothing moved. The panel below steps through a run. The corpus lines are demonstration data; the measured values are gaige’s actual reference receipt.

bench · gaige walkthrough demonstration data · real receipt values

corpus.jsonl
receipt.json
fingerprint.txt
series.json

1 / 4 · Ingest

THE PYTHON PACKAGE

Everything above is produced by one small tool: gaige, a Python package and command line built to be run by one person on one machine. The core needs only numpy and requests; torch and transformers enter through an optional GPU extra, because scoring needs a model and analysis never does. AGPL-3.0, and everything that surfaces an error, an interval, or a caveat stays free on principle.

pip install gaige · 0.0.1 on PyPI, source and history on GitHub. The PACKAGE page walks the install, your first receipt, and the command surface.

bench · quickstart real commands · typed demonstration

gaige runscore a corpus

Scores a labeled corpus with a pluggable scorer and writes the full receipts report: ROC, thresholds both ways, subgroups, fingerprint.

gaige analyzere-derive anywhere

Re-derives every number from scores that already exist. No GPU, no model, and the replay is bit-identical against the pinned receipt.

gaige corpus prepare-raidbenchmark slices

Provenance-pinned, seeded, stratified calibration slices from the RAID benchmark. The slice metadata rides into every downstream receipt.

gaige planwhat fits, measured

What this machine can honestly run: measured memory floors, attestation level, and a runtime anchor naming its receipt.

gaige scoreone document

Scores a single document against a calibrated report’s stated operating points. Measurements only, never a verdict.

gaige series watchdrift monitors

Watches a registered run series with per-interval conformal alarm bounds; movement inside the measured variance bound is not called drift.

corpus RAID slice g2×d2×a2 · 480 ai / 120 human · sha256 54b78669…
scorer Fast-DetectGPT · Falcon-7B · 4-bit (verified at load: 128× Linear4bit · 4.04 GB)
result AUROC 0.9285 95% CI [0.9061, 0.9482] · TPR@1%FPR 61.5% in-sample
conformal α=.05 → TPR 72.5% · α=.01 → TPR 56.2% (finite-sample, marginal guarantee)
refusal α=.005: refused — needs ≥199 human calibration samples, got 120.
“A tighter guarantee than your data supports is not a guarantee.”
stratified the aggregate hides: greedy decoding 87.6% vs sampled 39.7% TPR, same threshold

$ gaige plan · this machine, tonight (a resident scoring daemon holds most of the GPU)
falcon-7b 4-bit NO — needs 8.0 GB free VRAM, have 7.9
falcon-7b fp16 NO — needs 13.7 GB free VRAM, have 7.9
gpt2-large fp32 · CPU fits now · 0.64 s/sample (measured · receipt named in-table)
binoculars 4-bit pair NO — needs 9.0 GB free VRAM, have 7.9
no quality column, deliberately: separation lives in receipts, never in a hardware table

The REFERENCE RECEIPT below is what every run emits. These cards add the harder benchmark corpus and the machine view: a tool that tells you what your machine can honestly run: including when the honest answer is no. Full instructions (get, set up, use) live on the PACKAGE page; why your corpus is the part of the instrument you own is the CUSTODY page.

APPLICATIONS

Institutionsbefore acting on a score

Calibrate on your corpus before a score touches a person: fix and disclose the false-positive rate, see per-subgroup error rates and the base-rate arithmetic, keep appeal-ready receipts. gaige does not make any detector accurate. It makes a detector’s error rates visible on your material.

Researcherslongitudinal measurement

Run the same probes against a model over time with frozen instrument identity: fingerprints, versioned vintages, replicate-based variance bounds, and drift series that separate model change from pipeline change. movement inside the measured variance bound is not evidence of drift, and the reports say so.

Vendors & platformsproving calibration claims

Ship threshold claims a hostile reviewer can re-run: conformal operating points with finite-sample guarantees, stated beside the in-sample numbers they are usually confused with. a guarantee is marginal, over corpora like the calibration set. It is not a promise about any single document.

Contested casesexpert & appeal work

Error bars, disclosed operating points, base-rate arithmetic, and chain-of-custody receipts for every number entered into a dispute. gaige outputs evidence about a measurement process. It never outputs a verdict about a person or a document.

Operationsdrift monitoring

Per-interval conformal alarms with a marginal false-alarm bound; PH and CUSUM change detectors run alongside with their constants recorded and no guarantee claimed. the bound holds per interval, marginally. Alarm streams are for triage, not adjudication.

EVIDENCE OF NEED

Open-source detectors at their default thresholds produced
        47–100% false-positive rates (GLTR
        100%, LLMDet 97.9%, Fast-DetectGPT
        47.3% at threshold 0.25), while commercial detectors ship
        factory-calibrated at ≤1.7%. The study’s own
        recommendation: “calibrate detectors on in-domain data before
        using them.” That sentence is this project’s thesis.
RAID · ACL 2024
Across 15 detector variants, every one fell to near-chance
        (AUROC ≤0.60) on at least one evaluation set, and
        12 of 15 showed ≥15% FPR on human
        text. Rankings reorder by dataset and metric.
multi-detector eval
Binoculars’ authors report >90% TPR at
        0.01% FPR on their eval suite; an independent out-of-domain
        measurement found 43% TPR at 0.7% FPR
        (roughly 40–70× the headline FPR),
        and a 58.13% false-negative rate on GPT-4 text under the same
        calibration. Both numbers belong in the same sentence.
authors · Mozilla.ai
Conformal calibration held empirical FPR within its bound and lifted TPR at
        strict operating points: Fast-DetectGPT 51.22→69.32 at
        0.5% FPR; Binoculars 70.16→84.34.
arXiv 2505.05084
Short human texts carry substantially higher false-positive rates, and score
        distributions shift with length and style (KSmax
        0.3081, p < 0.01). Subgroup error rates are a
        measurement, not a footnote.
subgroup analysis
In institutions: a 14-detector study concluded the tools were
        “neither accurate nor reliable”; Vanderbilt
        disabled Turnitin’s AI flag “for the foreseeable
        future”; a newsroom test falsely flagged
        1–2% of 500 guaranteed-human essays; a grammar
        polisher flipped fully human text to a fully AI classification.
Weber-Wulff · Vanderbilt · Bloomberg

REFERENCE RECEIPT

corpus HC3-mini · n=100 · sha256 pinned
scorer Fast-DetectGPT · Falcon-7B · 4-bit (verified at load)
result AUROC 0.9720 95% CI [0.9458, 0.9938] · bootstrap
thresholds empirical in-sample and conformal (α=.01, marginal guarantee)
quant note fp32 vs fp16 thresholds agree to 4 decimals · 4-bit moves thr@1% ~10%
fingerprint model · quant · device · library versions · corpus hash · reproduce command
replay bit-identical analyze replay against the pinned receipt

Every gaige number ships like this, instrument and reproduce command attached, or it doesn’t ship. That rule applies to this page.

EXCLUSIONS

No verdicts. gaige reports measurements about a scoring process. It never pronounces a document human or AI, and the trademark terms forbid using the name to dress a score up as a verdict.
No universal thresholds. A threshold is a property of a corpus, a scorer, and an operating point. None of them generalize for free, so none are shipped as if they do.
No accuracy claims. gaige does not make a detector accurate and will not call one accurate. It measures error rates and their uncertainty, in the open.
No bias-free claims. Subgroup disparities are measured and reported with intervals, never averaged away and never declared solved.

PROVENANCE

gaige comes out of practice, not a pitch deck: security engineering in environments where the network is a privilege, doctoral-level research in AI measurement and detection calibration, and years spent defending numbers to reviewers who were paid to doubt them. That history is why the tool refuses more often than it reassures.

Names and résumés stay off this page on purpose. Here, receipts outrank credentials, and every receipt reproduces without knowing who ran it first.

QUEUED WORK

Everything below is queued, not shipped. Support moves the list faster (even a coffee’s worth genuinely helps), and none of it changes what stays free: anything that surfaces an error, an interval, or a caveat is free forever.

gaige admitone-verb admission

Candidate material against your trusted baseline in one command: divergence with intervals, stratified where-it-differs, refusal floors. The custody flow, folded into a single receipt.

Mondrian conformalper-subgroup guarantees

Group-adaptive operating points carrying the same finite-sample honesty: needs ≥99 human samples per bucket at α=.01, and the data decides whether you get one.

Batched scoringbenchmark scale

Full-benchmark runs without the single-text loop; batch size and padding enter the fingerprint, because they change the numbers.

The full queue, including what fuel concretely buys and what money never changes, lives on the ROADMAP page.

TERMS

LICENSE

AGPL-3.0, with a commercial license for organizations that need different terms. Scoring and calibration are free forever; anything that surfaces an error, an interval, or a caveat stays free on principle.

TRADEMARK

gaige is an invented word and a claimed mark. Fork the code freely under another name; use of the name is conditioned on keeping receipts intact and never presenting a score as a verdict.

CONTRIBUTIONS

Maintainership is deliberately closed. Issues and receipt-backed reproductions are welcome; pull requests are not accepted. Have an improvement idea, a use case, or a spot where gaige turned up in the wild? Open a thread on Discussions: triage actually happens, and a thread that survives it shows up on the roadmap with a receipt attached.

STATUS

Released. 0.0.1 on PyPI, carrying its verification receipts; source and history public on GitHub. Licensing inquiries: [email protected]