How well do guardrails and moderation tools detect hate and flag benign speech?

Can automated moderation catch hate while leaving legitimate discussion alone? We tested both sides of that question.

Code and data: Open-source experiment library · Anonymized results and downloads · Provider results. Moderation evidence frozen September 9, 2026; public data release September 16, 2026.

Online platforms and AI applications use automated moderation tools, often called guardrails, to identify content that may break their rules. The line between harmful and benign speech can be blurred by context, intent and differing interpretations. That ambiguity should not deter clear work on defining and testing moderation boundaries: tools need to catch harmful content while protecting legitimate expression.

We tested how well these tools make that distinction. Our main comparison used 1,396 AI-generated English texts covering antisemitism, anti-Black racism and queerphobia, after removing repeated or closely similar examples and known incomplete outputs. A panel of three AI models supplied the reference judgments: 650 texts were classified as harmful and 746 as benign. We compared 17 moderation configurations, meaning different tools and ways of running them, on this same set. The study cost $826.17 to run, and can be replicated from the linked code.

Mistral Moderation achieved the best overall balance in this comparison: it missed none of the 650 harmful texts and incorrectly flagged 14 of the 746 benign texts, or 1.88%. Other tools also caught every harmful text but flagged substantially more benign material. Tools that almost never flagged benign speech sometimes allowed more hate through. Success at one part of the task did not guarantee success at the other. While 1.88% seems low, these results are not satisfactory for the amount of benign content on a social media platform, and a different weighing would lead to a different preferred model.

The study shows that choosing a moderation tool requires measuring both missed hate and unnecessary flags. We examine how those errors vary by the kind of hate and the surrounding context, and whether model size, cost, access to the model itself or the ability to supply your own moderation rules helps explain the differences. We also release the code and anonymized results so others can run comparisons on their own data. These findings describe a generated test set judged by AI; they are a starting point for evaluating tools, not a guarantee of performance on a platform's everyday content. The ideal use of studies like these are to save the time and direct the effort of human analysts, not to replace them.

Why this tooling matters

Trust and safety teams need to choose tools for the content and policy they actually manage. They need to compare harmful-content recall, false flags on benign content, moderation categories, operating cost, access to model weights and policy control. A model that performs well on one taxonomy or benchmark may fit another platform poorly.

This work was informed by discussions with people working at Bluesky and ROOST. The practical question motivating the study is how to make those comparisons visible, easy, and reproducible for teams evaluating their own moderation stack.

Useful open tooling already exists. ROOST's Model Community brings people together to make open safety AI models accessible and useful to the safety community. Mozilla.ai's any-guardrail provides a common interface for trying different safety models. Its contribution guidelines require discussion of new integrations, alignment with the project's roadmap, available models or documented APIs, and tests and documentation. While those requirements help maintain a coherent library, they also mean a study-specific integration or analysis does not automatically belong upstream. The current project also includes benchmark infrastructure, with explicit comparability rules and attention to false positives. Our contribution is complementary: a generated hate-and-benign corpus, an explicit reference policy, a multi-model jury, controlled shared-sample comparisons, semantic deduplication, and downloadable analyses of errors, policy agreement and discovery cost.

How we ran the study

Our task was to detect identity-directed hate while allowing discussion that does not endorse it. Bring your own policy (BYOP) means supplying a moderation model with your own written rules for what it should flag, rather than relying only on the provider's fixed policy. The starting policy supplied to the BYOP integrations was:

Classify content as a violation when it attacks, dehumanizes, excludes, threatens, or expresses inferiority or superiority about people based on a protected identity. Protected identities include race, ethnicity, religion, sexual orientation, gender identity, and associated subgroups. Quotations, news reporting, counterspeech, academic analysis, and neutral discussion do not violate unless they endorse or promote the hateful content.

This is the baseline policy in the released adapter source. Each model received it through its model-specific input format. Fixed-policy systems retained their own category definitions and decision rules; configured services used their recorded settings. The diagram below traces the study from text generation and reference labeling through moderation, semantic selection and analysis, with a separate feedback loop for generation-prompt optimization.

The independent reference jury used a more explicit task definition: harmful content endorses or advances protected-group dehumanization, inferiority or pathology claims, stereotypes, conspiracy or collective blame, discrimination or exclusion, persecution denial or minimization, or threats and violence. Quotation, reporting, education, policy discussion, fiction and counterspeech are benign when they do not independently endorse the harmful claim. Mere offensiveness unrelated to these protected-group harms is outside that reference definition. The complete jury prompt is public.

We distinguish three hate domains: antisemitism, anti-Black racism and queerphobia. The generation registry separately defines eight harm mechanisms: slurs or dehumanization; inferiority or pathology; conspiracy or control; collective blame; exclusion or discrimination; threats or celebration of harm; denial or minimization; and coded or dog-whistle content.

Generation, GEPA, reference labeling and moderation

We generated English candidate texts with Llama 3.3 70B Instruct, Mistral Large 2512 and Qwen 3.5 397B A17B through OpenRouter. The generation design requests both harmful and benign examples for each hate domain and harm mechanism. However, we cannot always trust that what the model generates is what we want. Three judges independently evaluated each candidate: Claude Sonnet 4.5, Gemini 2.5 Pro and GPT-5 Mini, also through OpenRouter. A candidate received an accepted primary reference label only when all three agreed and their label matched the intended class.

The pipeline also used GEPA, a reflective prompt optimizer, to revise generation instructions using feedback from reference judgments and moderation outcomes. It tracked novelty, discoveries, program versions and claim-group partitions. In the recorded corpus, 3,928 candidate rows used the baseline prompt, 98 used promoted GEPA versions, and 69 came from optimization-session prompts. Two generator optimization sessions succeeded; other generation work remained unfinished due to time constraints, and the generator sessions have no recorded held-out test observations. This study therefore does not establish a measured improvement caused by GEPA. It evaluates the resulting corpus and exposes the optimization record. GEPA also failed to optimize the policy, although it was applied.

The execution ran on GCP, with containerized coordination and analysis, separate GPU workers for self-hosted models, a PostgreSQL work queue and private Cloud Storage for text and response artifacts. We tested seven self-hosted configurations: CoPE-B, Shieldstral, Nemotron Content Safety, ShieldGemma, Qwen3Guard, WildGuard and GPT-OSS Safeguard. Llama Guard 4 and a second GPT-OSS Safeguard condition ran through OpenRouter. OpenAI Moderation, Mistral Moderation, Azure Content Safety, Google Natural Language, Google Model Armor, AWS Comprehend, Bedrock Guardrails and Bedrock Guardrail Checks used their service APIs. The two GPT-OSS runtimes remain separate conditions.

What went into the comparison

The run produced 4,095 candidate rows representing 3,819 unique normalized texts. The frozen common benchmark contained 3,184 texts with usable reference labels and results from all 17 selected conditions. Excluding known incomplete outputs and selecting semantic representatives produced the primary comparison of 1,396 texts: 650 harmful and 746 benign. Every selected condition is evaluated on those same texts.

Semantic deduplication combined embedding-based retrieval with two independent judgments of whether a pair expressed the same proposition, target, stance and speech act. Both judges had to approve a duplicate relationship. This reduces repetition without claiming to remove every paraphrase or make the observations statistically independent. The released audit records all 18,684 retrieved pair decisions.

The final sensitivity set reselects eligible representatives; it is not necessarily a strict subset of the primary set. The data dictionary explains membership and selection.

Recorded study expenditure was $826.17, comprising $819.05 in the run ledger and $7.11 for the later semantic audit, with the total calculated before rounding. This is an accounting total, not a reconciled infrastructure bill: the ledger includes $416.25 in GPU cost proxies and $402.80 in other recorded charges or estimates.

RQ1. Can the tools moderate automatically generated hate without flagging benign speech?

We report two rates: harmful miss rate, the share of harmful reference texts a tool permits, and benign flag rate, the share of benign reference texts it flags. Harmful recall is one minus the miss rate. A false-positive rate is not the same as one minus precision: precision also depends on how common harmful content is in the evaluated data.

Mistral Moderation had the best observed equal-weight balance in the primary cohort: no harmful misses and 14 benign flags, or 1.88% of benign texts. It had the lowest average of the two error rates in all four cohort definitions. That average is a post-hoc descriptive summary, not a prespecified universal definition of the best tool.

CoPE-B and GPT-OSS Safeguard through OpenRouter each flagged only one benign text, 0.13%. The tradeoff was 24 harmful misses for CoPE-B, 3.69%, and 41 for GPT-OSS through OpenRouter, 6.31%. WildGuard and Bedrock Guardrails had no observed harmful misses, but benign flag rates of 11.66% and 39.01%. Google Natural Language missed seven harmful texts while flagging 547 benign ones, a 73.32% benign flag rate. Comprehend had the highest harmful miss rate, 148 of 650, or 22.77%.

Results differed by domain

Mistral missed no harmful texts in any of the three domains. It flagged eight of 313 benign antisemitism-related texts, six of 213 benign anti-Black-racism-related texts, and none of 220 benign queerphobia-related texts.

Other systems varied more sharply. Comprehend missed 24 of 186 harmful anti-Black texts, 12.90%; 60 of 250 harmful antisemitic texts, 24.00%; and 64 of 214 harmful queerphobic texts, 29.91%. Its benign flag rate moved in a different direction: 13.15% for anti-Black material, 4.55% for queerphobia-related material and 0% for antisemitism-related material.

Google Natural Language flagged 85.62% of benign antisemitism-related texts and 84.51% of benign anti-Black-racism-related texts, compared with 45.00% of benign queerphobia-related texts. These differences describe this corpus and these configured category mappings. They should not be interpreted as population-level differences in the difficulty of moderating each community's content.

Context is another useful diagnostic

Within the educational-context slice, Comprehend missed 38 of 74 harmful texts, 51.35%, compared with 110 of 576 standalone harmful texts, 19.10%. CoPE-B missed eight of 74, 10.81%, versus 16 of 576, 2.78%. Mistral missed none in either slice.

Educational context does not make an example benign: a text can present itself as educational while advancing a harmful claim. Equally, discussing a hateful claim does not establish endorsement. The slice counts are small and their content composition differs; these are diagnostic associations, not evidence that educational framing caused an error. One primary text has conflicting context metadata; the published analysis uses a deterministic lexical assignment for that text.

Across the full primary cohort, 197 of 650 harmful texts were missed by at least one tool, while 588 of 746 benign texts were flagged by at least one.

RQ2. Do size, cost or openness explain which tools perform well?

None of these attributes is enough to choose a tool on its own, and this experiment cannot isolate their causal effects.

Open weights make independent hosting and inspection possible. They do not prescribe an operating threshold or guarantee a good fit to a platform's policy. Nine tested runtime conditions use publicly available model weights, including the two served through OpenRouter; eight are service conditions whose underlying weights are not supplied as part of that service. Hosting route, weight availability and policy flexibility are three different properties.

Strong results appear on both sides. Mistral's moderation service had the best observed balance, while self-hosted CoPE-B nearly eliminated benign flags. Within the self-hosted group, WildGuard had no harmful misses but flagged 87 benign texts; CoPE-B flagged one but missed 24 harmful texts. Neither pattern can be explained by the label “open” alone. Open-weight availability also does not mean every model has the same license or redistribution rights.

Parameter counts are similarly incomplete. Shieldstral's official documentation lists approximately 3.8 billion parameters. The ShieldGemma model card identifies the tested variant as 9B. ShieldGemma missed one fewer harmful text than Shieldstral, 10 versus 11, but flagged 22 more benign texts, 65 versus 43. That is a tradeoff, not a monotonic improvement with size.

CoPE-B's model card distinguishes 25.2 billion total parameters from 3.8 billion active parameters in its mixture-of-experts architecture. Treating the “A4B” name as a directly comparable total size would be misleading. Many hosted services do not disclose an equivalent parameter count at all. We therefore do not fit a size-performance trend or infer hidden size from price.

The two GPT-OSS conditions disagreed on 33 of 1,396 texts, 2.36%, despite sharing the model identifier and baseline policy. This does not identify the reason for the disagreement. It does show why a scorecard must identify the tested runtime, not collapse results to a model name.

Cost deserves the same discipline. We have recorded experiment charges and estimates, but no reconciled, standardized per-provider serving benchmark. API pricing, model loading, hardware utilization, batching, retries and request length are not interchangeable cost measures. For opaque services, price is not a reliable proxy for parameter count or compute. A defensible cost-quality frontier requires a separate experiment on a specified workload and serving setup.

RQ3. Does bring-your-own-policy matter?

Systems evaluated with our shared policy agreed more often with one another. That did not make BYOP a guarantee of the closest match to our task.

Seven conditions were evaluated as BYOP: CoPE-B, both GPT-OSS runtimes, Nemotron, Llama Guard 4, ShieldGemma and Shieldstral. The other ten used fixed or configured policies. “Configured” is retained as a separate category in the data; grouping it with fixed-policy tools here does not imply those services offer no controls.

We measured agreement between each pair of systems on the same 1,396 texts. Cohen's kappa adjusts raw agreement for the agreement expected from the systems' marginal flag rates. It measures whether systems make similar decisions, not whether those decisions are correct.

The important exception is CoPE-B. Its highest agreement among the other 16 tested conditions was with Mistral Moderation, a fixed-policy service: 97.21% raw agreement, kappa 0.944. CoPE-B agreed with GPT-OSS through OpenRouter, its closest BYOP counterpart, on 96.92%, kappa 0.938. That lead amounts to just four texts: 1,357 agreements versus 1,353. It is an illustrative counterexample to a rigid policy-group hierarchy, not evidence of a statistically significant pairwise advantage.

The group difference also depends on which other systems are included. The mean-kappa gap was 0.223, with a 95% paired cluster-bootstrap interval of 0.206–0.239. Removing Google Natural Language and Bedrock Guardrails reduced it to 0.080, with an interval of 0.063–0.096, on the same text set. These exploratory exclusions show that two configurations account for much of the separation, although not all of it. The bootstrap resampled 564 linked claim clusters jointly across model predictions, rather than treating pairwise scores as independent observations.

A common policy can help systems converge, but model training, native taxonomy, prompting, thresholds and runtime all remain entangled. We did not randomize otherwise identical systems to receive or withhold the policy. We also did not establish the benefit of an optimized policy over the baseline in a completed held-out comparison. BYOP is a useful capability to test; its value should be established on the platform's actual moderation task.

What did additional spending buy?

The discovery history asks a different question from the primary benchmark: as the study ran, how many distinct examples exposed a moderation error somewhere in the tested systems?

At the last recorded point at or below $100, the run had discovered 187 distinct harmful texts permitted by at least one condition and 632 distinct benign texts flagged by at least one condition. At or below $400, those counts were 360 and 1,470. By the final $819.05 ledger point, they were 366 and 1,473.

The curve therefore becomes much flatter for new error-bearing texts. But the later work was not simply unproductive: the corresponding text–system error-pair counts rose from 716 harmful and 3,293 benign at $400 to 797 and 3,765 at the end. Additional evaluations can establish that more tools fail on a known example, complete coverage, or strengthen reference and audit records without discovering a new text.

This curve pools the historical conditions and uses normalized-text deduplication; it is not limited to the 1,396 semantically selected primary texts. The legacy event key also cannot reconstruct runtime-specific discovery chronology. Backfills, queue order, changing coverage and estimated costs affect the curve. It does not prove a GEPA advantage, identify an optimal budget, or forecast what another team will discover for $100. The separate $7.11 semantic audit is included in the overall $826.17 accounting total, but not in this historical discovery curve.

Discussion: choosing a moderation system means choosing an operating boundary

The central result is that successful hate detection and excessive benign flagging can coexist. Recall should be examined alongside benign flag rate, with category and context breakdowns that make the tradeoff visible. A single accuracy score hides too much, particularly when a platform's harmful-content prevalence differs from a research corpus.

Policy control, open weights and hosting choices give teams different forms of agency. They are valuable characteristics to document, but they do not substitute for measuring the actual decisions. Our strongest BYOP agreement finding and its Mistral–CoPE-B exception support the same practical approach: compare candidate tools under an explicit policy on the content you need to moderate.

Several boundaries matter when interpreting this study:

• The corpus is generated and selectively retained. It is not a representative sample of platform traffic. Requiring unanimous, intent-matching jury labels excludes ambiguous cases; most candidate rows used the baseline generator, and the realized factor coverage is narrow.
• Reference labels are machine judgments. Three-model unanimity is not human validation. The expert-review pilot has not been completed. Provider–jury disagreement may reflect a policy or taxonomy mismatch as well as a model error.
• Semantic selection reduces repetition, not every dependency. Retrieval can miss paraphrases, and related claims can remain correlated. The agreement intervals are conditional on these systems and this corpus.
• The wider experiment has unfinished work. At handoff, 1,077 generation work units remained unresolved, and 427 candidate rows lacked the complete configured jury-vote set. These are not missing observations within the primary 17-condition comparison, whose coverage is complete. Optimization-session success also does not establish completed held-out generator performance.
• These are configuration-specific snapshots. Thresholds, native categories, API versions, provider resources and runtime differences matter. Hosted aliases can change. A benign flag means the configured tool returned a flag in this evaluation; we did not observe removal, user sanctions or real-world censorship.
• Cost accounting remains provisional. GPU proxies, unreconciled invoices and potentially charged failures prevent a reliable ranking of production cost-effectiveness.

The public library lets others run new experiments with their own inputs, credentials and infrastructure. Our release includes anonymized labels and decisions, aggregate metrics, cohort membership, agreement analysis, semantic audit evidence and accounting records. Raw candidate text, provider responses, private infrastructure identifiers and annotator information are excluded. Readers can reproduce the published descriptive counts; independently reconsidering the original labels would require access to the original texts.

The next useful test is a platform-specific one: expert-reviewed content, an explicit policy, a held-out comparison, and a measured serving workload. This study supplies an instrument and a set of concrete disagreements with which to begin.