Hate-speech classification. 9 protected categories. Distinguishes use from mention.
Nine per-category detectors, one per protected identity (women, LGBTQ+, jews, muslims, black, asian, latino, disabled, migrants). On the HateCheck functional benchmark it scores AUROC 0.9339 — ahead of Detoxify (0.91), Perspective API (~0.87), and HateBERT (0.85-0.88) on that same benchmark. It distinguishes use ("I hate X", P=0.87) from mention ("Saying I hate X is bigoted", P=0.10) — the failure mode that trips up production classifiers.
Scope, stated plainly: that figure is HateCheck-specific, and HateCheck is what this probe was fit on. We measured it on four other English hate corpora and it does not transfer — AUROC 0.62 (CONAN), 0.59 (Stormfront), 0.57 (CivilComments), 0.47 (MLMA, i.e. below chance). A multicorpus-trained variant is steadier at 0.62–0.84 across those same corpora and is the right choice for general traffic. Treat the benchmark number as a benchmark number, not as expected production accuracy — and ask any vendor quoting a single hate-speech score which corpus it came from.
How this works
- Nine per-category detectors. A single detector loses category-specific signal — it collapses 9 distinct hate directions into one. Bhala calibrates one detector per identity (women, LGBTQ+, jews, muslims, black, asian, latino, disabled, migrants).
- Aggregator score = max across probes. Fires the top category at its own calibrated 5%-FPR threshold rather than averaging.
- Calibration set. Per-category probes trained on HateCheck + CONAN + CivilComments; 5%-FPR thresholds per identity.
- Use vs mention. The model preserves quotation/sarcasm boundaries that string-matching and Perspective-style classifiers don't.
- Audit receipt. Timestamped record per call, retrievable by receipt id (unkeyed SHA-256 — recomputable, not a signature). The same head powers the Bluesky labeler at bsky.bhala.ai, where receipts are Ed25519-signed and get a public appeal page.