# Downloadable study data Release: **2026-09-16**. Frozen moderation evidence: **2026-09-09**. This is an anonymized release of the completed existing-corpus guardrail evaluation. The original generation queue, some optimization arms, human pilot and invoice reconciliation were not completed. Those limitations remain in the study metadata. ## Start here - `metrics.csv`: all 68 published confusion matrices, 17 runtime conditions × four cohorts. - `primary_predictions.csv`: 23,732 rows, one primary-cohort sample × selected condition. - `domain_metrics.csv`: all 204 domain-specific matrices, four cohorts × three domains × 17 conditions. - `downloads.zip`: all data and documentation in this directory, excluding the ZIP itself. - `checksums.sha256`: SHA-256 digests for individual release files. `bundle.sha256` binds the ZIP. CSV files use UTF-8, a header row and unrounded numeric rates from 0 to 1. Multiply a rate by 100 for a percentage. Boolean CSV values are `True` and `False`; JSON uses native booleans. Empty numeric fields mean unavailable, never zero. ## Files and grain | File | Contents / unit | |---|---| | `metrics.json` | Cohort metadata, safe condition provenance and the same 68 matrices | | `cohorts.json` | Four named sets of opaque normalized sample IDs | | `conditions.json` | Every observed condition; selected baseline flag, model description, runtime alias and policy mode | | `candidates.csv` | All 4,095 generated candidate rows, including rejected, failed and pending reference rows | | `moderation_results.csv` | All 67,589 recorded moderation results, including historical conditions and equivalent observations | | `public_snapshot.json.gz` | Candidate and result rows, 53,632 jury votes, 114,660 panel decisions, panel definitions and selected conditions | | `jury_votes.json.gz` | Convenient jury-only copy of the vote, decision and panel tables | | `primary_predictions.csv` | Deduplicated selected-condition predictions for the 1,396 primary texts | | `domain_metrics.csv` | Domain membership comparisons; a sample may belong to more than one domain | | `semantic_audit.json.gz` | All 18,684 retrieved pair decisions, similarities, 1,998 groups and generation-finish audit | | `byop_agreement.json` | Four-cohort agreement summaries and five primary-cohort paired cluster-bootstrap comparisons | | `pairwise_agreement.csv` | All 136 pairs of selected conditions, primary cohort | | `analytics.json` | Attrition, coverage, condition selection, jury missingness, costs, discovery and optimization accounting | | `attrition.csv` | Counts by intended class and primary-reference status | | `discovery_curve.csv` | Deduplicated historical discovery counts by cumulative recorded spend | | `validation.json` | Source-to-release numerical reconciliation counts | ## Cohorts | ID | Harmful | Benign | Total | Interpretation | |---|---:|---:|---:|---| | `original` | 1,318 | 1,866 | 3,184 | Frozen normalized-hash common benchmark | | `exclude_known_incomplete` | 1,315 | 1,856 | 3,171 | Excludes known truncated/filter/refusal outputs | | `semantic_and_exclude_known_incomplete` | 650 | 746 | 1,396 | **Primary analysis cohort**, also excludes known semantic duplicates | | `semantic_and_verified_stop` | 615 | 727 | 1,342 | Requires a recorded normal finish, then reselects semantic representatives | The verified-stop set is a separately selected sensitivity cohort; it is not a strict nested deletion from the 1,396-text set. Requiring a normal termination does not prove semantic correctness. All 17 selected conditions cover identical samples within each cohort, with no missing/conflicting selected results. ## Data dictionary and joins - Candidate `id` (`candidate-…`) identifies a generated row. Multiple rows may share a `normalized_content_hash` (`sample-…`). Despite its legacy field name, the public value is an opaque sample ID, **not a content fingerprint**. - `claim_hash` (`claim-…`) and `factor_cell_id` (`cell-…`) are opaque groups. Claim linkages are retained for clustered analysis. They do not expose claim wording. - `reference_status` preserves acceptance, mismatch, disagreement, failure and pending status. Only accepted rows whose shared sample has one consistent class enter the primary reference. `intended_label` is a target intent for all rows; it functions as the accepted machine-reference label only after those checks. - `reference_harmful` and `jury_disagreement` retain stored machine fields; no human labels or adjudications are claimed. - Result `candidate_id` joins to candidate `id`. `result_id` is an opaque row ID. `flagged=true` means the adapter mapped a moderation response to a flag. It does not show whether a production platform actually removed content. - Condition identity is the tuple `(provider, model_or_policy, provider_snapshot_hash, policy_version_id)`. `condition_id` is its public shortcut. `provider_snapshot_hash` is an opaque runtime alias, not a public configuration checksum. Configured resource names are withheld; model/version labels and distinctions between runtime conditions are retained. - Selected baseline condition: largest usable coverage for each provider, then lexical condition order. The `condition-…` prefix preserves the original order. Never pool historical versions to fill gaps. `baseline` policy differs from policy-optimization versions. Hosted aliases do not pin provider-managed weights. - `true_positive`: harmful reference + flag; `false_negative`: harmful reference + no flag; `false_positive`: benign reference + flag; `true_negative`: benign reference + no flag. `harmful_n=TP+FN`; `benign_n=TN+FP`. - `harmful_bypass_rate=FN/harmful_n` is the harmful-miss rate. `benign_overmoderation_rate=FP/benign_n` is the benign-flag rate. These terms are relative to the study's reference and adapter configuration. - Candidate factors use `domain`, `context`, `explicitness`, `speech_function`, and `transformation`. The frozen candidates have explicit/plain/assertion levels only, with educational or standalone context; the richer planned factor space was not fully executed. Do not treat planned factors as measured diversity. - Domain membership uses **all accepted duplicate rows** belonging to a retained sample. A multi-domain sample appears in each applicable domain. Domain totals therefore need not sum to the overall cohort; do not double-count them. - Jury `candidate_id` joins to candidate rows; panel `id` joins decision `panel_id`. `judge_configuration_id` identifies a judge configuration. Jury rationales and raw responses are not released. A missing vote remains missing. - Semantic pair `hashes` are two public sample IDs, `votes` are the two independent duplicate decisions, and `cosine` is retrieval similarity. `groups` list sample IDs. The historical `judge_disagreements` index strings are diagnostic; use `pairs[].votes` and `pairs[].hashes` for direct joins. - Monetary values with `_microdollars` are USD × 1,000,000. `latency_ms` is recorded request duration, not a controlled latency benchmark. Costs mix observed charges and configured estimates and exclude unreconciled infrastructure invoices. ## Reference and semantic methods The primary active jury was Claude Sonnet 4.5, Gemini 2.5 Pro and GPT-5 Mini. Acceptance requires unanimous classification matching the intended class. A separate, more expensive frontier panel is used for the jury-panel comparison; it must not be described as the original active moderation reference. All labels are machine generated. The later vote audit found 427 candidates with at least one missing configured vote; it did not change original primary labels. Initial deduplication used NFKC normalization, case folding and collapsed whitespace. The semantic audit embedded 350-token chunks using `sentence-transformers/all-mpnet-base-v2`, retrieved all pairs at cosine ≥0.88 plus the top three neighbors at ≥0.80, and required two judges (Gemini 2.5 Flash Lite and GPT-5 Mini) to agree on the same proposition, target, stance and speech act. Disagreement is retained as distinct. Deterministic complete-link groups are supplemented by direct-edge exclusion across retained representatives. Selection does not inspect moderation outcomes. Retrieval can miss paraphrases; machine equivalence does not establish human validity or sample independence. BYOP means a natural-language policy supplied at inference. Configured policy resources are a separate mode. Agreement is not correctness or a causal policy effect. The exploratory interval resamples 564 linked claim clusters jointly across all systems, 3,000 times, seed 20260916. Outlier-exclusion analyses are exploratory and keep the primary text set fixed. The historical discovery curve includes conditions outside the selected baseline. Its legacy event key omitted provider snapshot, so runtime-specific chronology cannot be reconstructed. It must not be combined with fixed-condition error rates. ## Reproduce and inspect Install the library from the repository root (`python -m pip install -e .`), then: ```sh ah-guardrails analyze data/public_snapshot.json.gz --cohorts data/cohorts.json --cohort semantic_and_exclude_known_incomplete --output primary-recomputed.json python scripts/verify_public_data.py python -m pip install numpy python scripts/reproduce_agreement.py ``` The public rows reproduce cohort confusion matrices and paired agreement without credentials. Raw-text normalization, fresh semantic judgments and new provider calls cannot be recreated from this anonymized release alone. The original historical configurations and private resource identities are not all public. ## Privacy, scope and license No candidate text, API response bodies, private keys, tokens, contact details, infrastructure endpoints, cloud project/resource names or internal UUIDs are included. Content hashes are replaced as well as candidate IDs. The mapping is not published. Dates, public provider/model names, numeric results and research-factor categories remain. Anonymization limits direct joins back to the private archive. Operational work-unit/usage event logs, private prompts and fact registries, production deployment configuration, reviewer packets, embedding vectors and raw API artifacts are excluded. Their relevant aggregate accounting remains in `analytics.json`. No omitted operational field is silently treated as a completed experiment. **1,077 original generation units and human review remain unfinished.** Code and derived tables use the repository's MIT license. Provider and model names remain their owners' marks; no provider endorsement is implied. This license does not grant access to private text or provider services.