STATUS 2026-07-28: contains claims superseded by the resume decision + race subspace runs (the directions inside the model along which race is carried). Do not send externally until rebuilt. See docs/BENCHMARKS.md.
Bhala research / programmable embeddings
Named operators on meaning — measured, not asserted.
Bhala's embedding space is built so you can name a concept — a bias, a protected trait, a tone — and measure it. Detection is the product. Erasure is not removal: rank-3 erasure — deleting the three strongest race directions — leaves race readable at 0.947-0.993 on the same model, 0.949-0.998 on a part of the record the erasure was never fitted on, and 0.996-1.000 when the direction is fitted on one part of the record (say the address line) and read back from another (say the education line) — the case we call cross-channel. Four models × 600 real resumes. This page lays out the operators that hold up under testing, the ones that don't, and the evidence for both.
How to read the scores on this page. They run from 0.5 to 1.0. 0.5 means the model is guessing at random — it has learned nothing from the record. 1.0 means it is right every single time. There are only two possible answers, so random guessing gets you 0.5; anything above that is real information the model picked up. (The technical name for this measure is AUC.) So a trait still readable at 0.947-0.993 after erasure is one the model still gets right roughly 19 times out of 20, or better.
01 — New capabilities
Naming a concept, then acting on it
Current AI models are black boxes: you feed in text and get back a vector or prediction, but you cannot name the change you want and apply it as an operator. You cannot point at the bias component and say how much of it the model is carrying. You cannot tell the model “apply formal register” without retraining.
Bhala's embedding space changes the primitive. The representation is structured so that named operators act on it: compose two meanings, invert a transformation, read a demographic direction off a frozen model and quantify how strongly it is carried. These operators are learned components with their own training objectives — not arithmetic post-hoc on a frozen space. What we do not claim is the last step: subtracting a demographic direction does not make it unreadable, and we report that as a measured negative below rather than a feature.
Steering — per model, per method, dual-use
Hate steers, per model and per method: K-steering at layer 10 gives +0.188 [0.146, 0.231] on Qwen2.5-7B against a random-direction floor of +0.0081, and steers Mistral-7B-Instruct-v0.2. NULL on Mistral-7B-v0.1 base (baseline 0.2981, headroom present) and Gemma-2-9b (baseline 0.0549, floor-limited). A single-direction difference-of-means vector scores +0.030 and fails. The same direction inverted is a jailbreak vector — dual-use, never a safety dial.
Reproducible audit records
Every prediction writes a timestamped record — which operators were applied, the model version, and hashes of the input and API key rather than the raw values — to an append-only log you can retrieve by receipt id. The id is an unkeyed SHA-256 you can recompute offline, so it attests the record's contents but not its origin. It is not a signature and we don't call it one. (The Bluesky moderation labeler is the exception: those receipts are Ed25519-signed and stored in Postgres with a public per-receipt page.)
Cross-lingual transfer without parallel data
Operators learned in one language transfer algebraically to others. A compact model trained on Zulu classifies Swahili intent zero-shot at 71.0%, against a random-init floor of 42–44% — a margin above floor, not a win over the frontier. On the same protocol InkubaLM-422M scores 79.2% and TF-IDF 76.1%, and on the 8-language set we lose on 4. The earlier “beats GPT-4o” framing compared different regimes and is retracted.
Erasure is not removal
Detection is the product. Erasure is not removal: rank-3 erasure leaves race readable at 0.947-0.993 on the same model, 0.949-0.998 on a part of the record the fit never saw, and 0.996-1.000 cross-channel (4 models x 600 real resumes). Removing race from ONE part of the record took 12 directions (surname 0.535 with a simple readout / 0.508 with a stronger one) and left ZIP 0.9936, school 0.9958, first name 0.986 untouched. The edit is cheap (qualification 0.4677 → 0.4672) but it moved top-10 selection share beyond a random-direction control on 1 of 4 models.
03 — Empirical evidence
What we measure
We do not publish numbers we cannot regenerate on request. The public probe results reproduce in under 90 seconds on a laptop GPU; the race-direction and erasure results need the 4 open-weight models and 600 real resumes, which is a cluster job, not a laptop one.
What a “proxy” is. Something on the record that isn’t race — but gives it away anyway. A ZIP code is not race. But if a neighbourhood is 96% Black, writing that ZIP down tells a computer nearly as much as writing the person’s race would. Same for the university: an HBCU is not a race, but it is a very good clue. Nobody has to intend this; the model works it out on its own.
| Result | Value | Test conditions |
|---|---|---|
| Trait detection on frozen models (ZIP · HBCU · name · gender) | 0.984 / 0.983 / 0.945 / 0.979ZIP and university: right about 49 times out of 50. Name: about 19 times out of 20. Gender: about 24 times out of 25. On this scale 0.5 would mean guessing at random. | AUC across 4 models · 500 matched pairs per axis · 12/12 ZIP code areas (ZCTAs) confirmed against 2020 Census DHC P8 · HBCU status is statutory · dose-response spearman 1.000 — a perfect ranking: the higher the neighbourhood’s Black share in the census, the higher the score, with no exceptions. Re-measured 2026-07-29 at full resume length (8,000 chars / 2,048 tokens, 90.8% of all text). The earlier figures — ZIP 0.991 / HBCU 0.991 / name 0.964 / gender 0.986 — were taken on ~7% of each resume (420 chars / 160 tokens), which left the injected marker at ~17× its natural prominence. Every number fell, and the name fell hardest: the gap between the proxies and the name widened from 0.026 to 0.039 |
| Low-collateral editing (remove one concept, others hold) | retracted | The 4–6.5× frame result was retracted on 23 Jul 2026: across 21/21 runs the frame never erases and LEACE — the standard published erasure method we were measuring ourselves against — wins. Collateral on the surviving edit is ±0.008 max (mean 0.002), and rank-3 erasure leaves race readable at 0.947-0.993 same-model / 0.996-1.000 cross-channel |
| Set-intersection detection (race × gender) | 0.81 | AUC, the same 0.5-to-1.0 scale · concept directions combine like sets (union · intersection · difference) for intersectional cases |
| Bias-axis detection (BBQ + StereoSet + CrowS-Pairs + WinoBias) | 28 / 28 | 10,064 sentence pairs · 16 layers probed · the axis is readable on every one of the 28 axes. This is a detection result, not a correction result — subtracting the centroid direction does not make the axis unreadable |
| MASSIVE Swahili intent classification | 73.2% | Zero-shot · zero target-language data · above GPT-4o (70.6%) |
| Cross-family transfer (Korean / Hindi / Amharic) | 72.5 / 69.7 / 66.5% | A simple linear readout fitted on top of a frozen encoder — no retraining of the model itself · 38–43× over random · strongest published frozen + linear + zero-target-language result we know of |
| Injongo (8 Bantu languages) | SOTA 4 / 8 | vs AfroXLMR-76L (fine-tuned per language). Bhala is zero-shot — no in-language training |
| Sentiment steering (operator algebra) | 100% in-family | Verified by independent classifier · single operator vector · cross-language transfer 77% to English (zero-shot). Not yet scored against a norm-matched random direction, which is a floor to beat, not a zero (Rogue Scalpel, arXiv 2509.22067) — treat this row as unverified until that control is run |
Methodology and per-axis breakdowns: full benchmarks page.
05 — Academic context
For the technically curious
If you know word2vec: king − man + woman ≈ queen. That's word arithmetic — approximate, inconsistent, breaks after one step. Bhala does the same for sentences and meanings, but as learned operators with their own training objectives, tested on data they were never fitted on, not loose arithmetic on a frozen space. Some of these operators hold up cleanly; others don't, and we report which.
The idea has deep roots in a four-decade debate about whether neural networks can support structure-sensitive operations. We are testing how far a learned embedding space can go toward that ideal. Our generalization results on that question are under internal audit, and until the audit is done we make no claim — in either direction. One thing we already know and report: full round-trip closure, blindly decomposing a composition back into its parts, does not hold. Composition is directional, not exactly reversible.
Bhala's architecture is a purpose-built encoder trained under joint algebraic constraints. Composition and detection are learned components with their own loss terms. What that buys in practice is detection: a named trait can be read off a frozen model with high reliability. It does not buy removal. The claim that we edit one concept 4–6.5× more cleanly than standard methods was retracted on 23 Jul 2026 — across 21/21 runs the frame never erases and LEACE wins.
The evidence table in Section 03 shows what holds and what doesn't. Every result is reproducible from public datasets on a laptop GPU.
Read further
Full benchmark methodology, per-axis breakdowns, and reproduction code on the benchmarks page. We're happy to walk research collaborators or investors through the architecture in detail.