Gemma 4 won't repeat the stereotype. Inside, it still tells the two sentences apart.

The short version

Ask Gemma 4 a leading bias question — who is more likely to be a criminal? — and it refuses or hedges. Most fairness audits read that response and conclude the bias is gone. We took a closer look at what the model encodes internally.

We probed hidden states of Gemma 4 31B-IT at 16 evenly spaced layers (every 4th of its 60, plus the final layer) across 28 protected dimensions, using 10,064 sentence pairs drawn from BBQ, StereoSet, CrowS-Pairs, and WinoBias (capped at 500 pairs per axis). Eight analysis lenses per layer, 5-fold cross-validated probes, about three hours on a single H100.

The finding, stated as measured:

All seven BBQ axes are near-perfectly separable by a linear probe in Gemma 4's earliest layers — and separability then decays monotonically with depth. On five of those axes the final layer reads 0.26 to 0.52. The distinction is legible in the early encoding and not legible at the output-adjacent layers. Our reading is that post-training alignment changed what the late layers carry without changing the early representation.

Layer 4, one number per BBQ axis: physical-appearance 0.99, gender-identity 0.99, disability-status 0.99, age 0.95, race-ethnicity 0.94, religion 0.86, sexual-orientation 0.80. Layer 59, same order: 0.52, 0.69, 0.62, 0.29, 0.30, 0.26, 0.26.

Three qualifications belong here rather than further down, because each one bounds that sentence:

The decay is a smooth slide, not a cliff. Take BBQ/Race_ethnicity: 0.81 at layer 0, 0.94 at layer 4, then 0.81, 0.67, 0.61, 0.55, 0.49 … 0.30 at layer 59. Every BBQ axis has that shape. A monotone decline is what you expect from progressive abstraction of surface features as much as from targeted alignment — a model discarding the literal group word as it builds a more abstract representation produces the same curve. Separating those two explanations requires causal intervention (activation patching), which we did not do.

Four of those five final-layer numbers are below chance, and that is a caveat, not an amplifier. 0.26 on a balanced binary task with 5-fold CV is not "the bias got worse" and it is not cleanly "the information is gone" either; it means no stable separating direction survives across folds at that depth. We apply the same standard to these axes that we apply to WinoBias later in this post. Read the five as "no stable linear direction reaches the output layer," which is a weaker claim than "the encoding no longer contains it."

Four newer open models don't show the late decay at all. The identical suite on Qwen3-32B, Qwen3.6-27B, Phi-4-reasoning and DeepSeek-R1-Distill-32B (results here) leaves separability linearly readable through the final layer, in the open, on all seven BBQ axes. Gemma sits at the hidden end of a hidden-to-exposed spectrum; it is not a universal law about aligned models. What holds across every model we have probed is that the information is present either way.

If you only ever talk to Gemma through chat, that distinction may not matter much. If you build on its hidden states — embeddings, retrieval, classifiers, fine-tunes — it can. Three things in this post:

1. What the probes actually found, axis by axis, without rounding up.
2. A recipe to run the same audit on any open-weight model — including the control experiments we now consider mandatory.
3. What this class of audit cannot establish, with pointers into the literature.

What the chart shows — precisely

We ran two probes over the same internal vectors at each probed layer:

• Linear probe (red, dashed) — logistic regression; finds the signal only when stereotyped and anti-stereotyped sentences are linearly separable. This is what most published fairness probing uses.
• MLP probe (green, solid) — a 2-layer nonlinear probe; can find curved or clustered separation.

This chart is the axis with the largest nonlinear gap of the 28 — BBQ / sexual orientation — not the average case, and not the axis with the strongest linear signal (that is physical-appearance). At layer 0 the MLP probe reads 1.00 while the linear probe reads 0.56 (gap +0.44). Both then decline monotonically and end below chance at layer 59 (linear 0.26, MLP 0.21) — see the note on below-chance readings further down. The plotted curve interpolates between the 16 probed layers; we did not measure all 60.

The honest distribution across all 28 axes:

• Peak linear accuracy clears 0.65 at some layer on 7 of 28 axes — and all seven are the BBQ axes. Every CrowS-Pairs axis peaks at 0.41–0.65, every StereoSet axis at 0.35–0.46, every WinoBias axis at 0.13–0.17. Which benchmark family a pair comes from predicts the result better than which protected attribute it targets, and that is itself a finding about the benchmarks.
• Linear-probe verdicts: 5 SUPPRESSION (BBQ age, physical-appearance, race-ethnicity, religion, sexual-orientation), 1 PERSISTENT (BBQ gender-identity, 0.98 → 0.69), 5 AMBIGUOUS (BBQ disability-status plus four CrowS axes), 17 LOW_POWER — the linear probe never clears 0.55–0.65 anywhere on those 17, which means the audit has nothing to say about them in either direction.
• The three verdicts on BBQ are one continuum, not three phenomena. All seven BBQ axes have the same monotone curve; the label depends only on where the final layer lands — 0.26 (religion, sexual-orientation) through 0.52 (physical-appearance) to 0.69 (gender-identity). "5 of 28" is a threshold slice of a smooth spread, so don't read the count as a census of suppressed attributes.
• MLP−linear gap > 0.10 at some layer: 5 axes (largest: BBQ/sexual-orientation +0.44, BBQ/religion +0.37). On most axes the two probes track each other closely — the "nonlinear detector sees what the linear one can't" pattern is real but concentrated, not universal.

The run's summary verdict — "bias signal present on 28 of 28 axes" — is true only under a permissive rule: an axis counts if any one of the eight lenses crosses its threshold. The two lenses that fire almost everywhere (direction-consistency of pair differences, and subspace residual) are also the two with the weakest validation — see the limitations section. We report the 28/28 number because the run produced it, but the load-bearing finding is the BBQ decay curve — near-perfect separability early, at or below chance late, on all seven axes — because that is the part that comes from cross-validated probes and survived the lexical control on two other models.

What this means for your product

Three deployment patterns, ordered from lower to higher exposure.

1. Direct chat. Users type, the model responds. The output-level alignment is doing real work here; on leading questions Gemma refuses or hedges.

2. Embeddings, RAG, semantic search, classifiers. You take hidden states (or a derived embedding) and build an index or a ranker. You inherit the early- and mid-layer geometry — including whatever the suppression pattern left intact there. Whether that geometry actually changes retrieval or ranking outcomes for your queries is an empirical question this audit does not answer; it tells you the information is present, not that it is used downstream. If you deploy this way, test your own pipeline.

3. Fine-tuning. Fine-tuning rewrites late layers — where the suppression lives. A small, non-safety-targeted fine-tune can plausibly undo output-level polish without touching the underlying representation. This is a mechanistic argument, not something we measured; treat it as a hypothesis worth testing before you ship a fine-tune.

Why RLHF is not enough for RAG, local models, or anything downstream

The three patterns above are one argument, so let's make it explicit. RLHF and its relatives are output-layer training: they optimize what the model says at the very end of its computation. That is genuinely valuable — for the chat use case, which is where the optimization pressure is applied. But almost every serious enterprise deployment routes around the output layer:

• RAG and semantic search consume mid-network representations directly — the layers our probes show RLHF never cleaned.
• Locally hosted open-weight models put the weights in the deployer's hands, where any fine-tune (including an innocent domain-adaptation one) can rewrite the late layers that carry the alignment polish.
• Embedding-based classifiers and rankers inherit whatever geometry the encoder has, with no output gate in the loop at all.

RLHF's guarantees do not travel with the representation. Whatever assurance a deployment needs — bias control, safety filtering, explainability, evidence for an auditor — has to be provided at the layer the deployment actually uses, by something that sits between the model and the application. That's the middleware argument, and it's the thesis this audit was run to test: you cannot rely on the model's own alignment training to protect a pipeline that never touches the model's output layer. Something in the middle has to detect what's encoded, control it per-request, and produce a record — regardless of which foundation model sits underneath this quarter.

This is also where the regulatory conversation lands. Explainability and record-keeping obligations (the EU AI Act's transparency and logging provisions, NIST AI RMF's measurement functions) are written against the system, not the chat window. An audit like this one is the explainable-AI artifact for the representation layer: it names what is encoded, where, and how strongly — which is exactly the evidence an output-level test report cannot supply. If your compliance story for a RAG system is "the underlying model passed a chat-level fairness test," this post is the technical explanation of why that story doesn't survive scrutiny.

Audit any open model yourself — the recipe, with controls

Runs on anything with public weights. Roughly three hours on an H100 at Gemma-31B scale.

1. Pull labeled benchmarks. We used BBQ, StereoSet, CrowS-Pairs, and WinoBias — 28 axes after splitting by category; we capped at 500 pairs per axis (10,064 pairs total).
2. Encode every sentence at several layers. We mean-pooled token hidden states at every 4th layer. (Pooling choice matters; max/attention pooling can change results.)
3. Train probes per layer with cross-validation. Logistic regression and a small MLP, 5-fold stratified CV, scaler fit inside each fold.
4. Read the trajectory, not one number. Early peak + final-layer collapse = suppression. High everywhere = persistent encoding. Never above ~0.6 = the probe has little to say.
5. Run the controls. These are not optional; without them the numbers above are uninterpretable:
   • Random-label control — shuffle labels, retrain the MLP. If it still scores above chance, your probe is overfitting. (Ours: 0.52.)
   • Random-feature control — replace embeddings with Gaussian noise. (Ours: 0.52.)
   • Lexical control — the stereotype/anti-stereotype pairs in these benchmarks usually differ in the demographic term itself. A probe can separate them by detecting which group word is present, which any competent encoder does and which is not itself evidence of stereotyping. Build swapped-term pairs in neutral contexts and check what the probe does there. We have not yet run this control at full scale — it is the biggest open hole in this audit and in most published probing-based bias audits.
6. (Optional) Subspace analysis. Fit a PCA subspace on stereotype embeddings, measure anti-stereotype variance outside it. If you do this, you must also compute the same-class baseline (fit on half the stereotype set, measure the residual of the other half) — in high dimensions with a few hundred samples, a 10-component subspace leaves large residuals for any second sample, including one from the same distribution. We did not compute that baseline for the numbers in this post, which is why we no longer lean on the subspace lens as primary evidence.

The full per-axis, per-layer probe results: gemma4_full_algebra.json (~330 KB — all 8 lenses × 16 layers × 28 axes). Methodology + code available on request.

The CrowS race-color case — interesting, but read it with the caveats

On every CrowS-Pairs axis the linear probe gives Gemma 4 a near-clean bill of health — peaks of 0.41 to 0.65, never far from chance. CrowS-Pairs / race-color is the sharpest case of the geometric lenses disagreeing with that:

Both probes hover near chance, while the subspace residual says 89% of anti-stereotype variance falls outside the stereotype subspace and the Bhattacharyya overlap of the two distributions drops to 0.08. Taken at face value, the two groups occupy structurally different regions that no single direction separates.

Three reasons not to take it at face value. (a) The missing same-class subspace baseline described above — some of that residual is what any 500-sample split of a 5,000-dimensional representation produces. (b) The Bhattacharyya estimate comes from Gaussian fits in a 20-PC projection with ~500 samples per class, which is noisy. (c) When we later ran the lexical-swap control on two newer models, CrowS-family separability turned out to be word-identity detection and we retracted it as bias evidence (details); BBQ-family separability survived the same control. That control has not been run on Gemma 4, so this table is the part of the post most likely to be retracted next. The right conclusion is "the standard linear toolkit cannot rule bias in or out on this axis, and the geometric lenses hint at structure worth a dedicated experiment" — not "structurally severe and invisible."

A related red flag we want on the record: on all four WinoBias splits, the cross-validated linear probe scores 0.09–0.17 at every one of the 16 probed layers — far below chance on a balanced binary task. That is not "hidden bias"; it is a sign of evaluation pathology (WinoBias pro/anti pairs differ by a single pronoun, so mean-pooled embeddings are nearly identical and fold structure can invert the learned mapping). We flag it because below-chance probe numbers appear in bias-audit papers more often than they get explained, and they should always trigger a methods review rather than a bias claim.

The same standard applies to our own headline. Four of the five suppression axes end below chance too — 0.26 to 0.30 at layer 59 — which is why we phrase the finding as "no stable linear direction reaches the output layer" rather than "the information is gone." The difference between WinoBias and BBQ here is the trajectory, not the endpoint: WinoBias is below chance at every depth, so nothing was ever measured; BBQ starts at 0.80–1.00 and slides, so something real is decaying. But we should not get to treat a below-chance number as pathology when it is inconvenient and as evidence when it is not.

What this audit cannot show

This section is the part we most wish other audits published.

• Extractability is not use. A probe shows information is decodable from a representation, not that the model uses it in any decision. See Belinkov (2022) for the survey-level version of this caveat, and Elazar et al. (2021, "amnesic probing") for the demonstration that probe-detectable features are sometimes causally irrelevant to model behavior.
• Intrinsic bias measures correlate poorly with downstream harm. Goldfarb-Tarrant et al. (2021) found embedding-level bias metrics do not reliably predict bias in downstream applications. An encoder-level finding is a reason to test your application, not a substitute for testing it.
• The benchmarks themselves are noisy. Blodgett et al. (2021) documented substantial validity problems in StereoSet and CrowS-Pairs (ill-formed pairs, unclear stereotype direction, conflated constructs). Our per-axis numbers inherit those problems.
• Probes need controls — and when we ran the missing one, it cut our evidence in half. Hewitt & Liang (2019) introduced control tasks precisely because expressive probes can score well for reasons unrelated to the property under study. Our random-label and random-feature controls pass. The lexical-swap control (same group words in stereotype-free sentences, with cross-transfer tests) has now been run on two newer models (details): CrowS-family separability turned out to be word-identity detection and is retracted as bias evidence; BBQ-family separability survived the control (word-swap probes transfer at chance while BBQ reads at 0.77–1.00). Gemma 4 itself hasn't had the control run yet — until it has, read this post's CrowS-based rows with that outcome in mind.
• The suppression narrative has precedent but also nuance. Gonen & Goldberg (2019) showed word-embedding debiasing hides bias rather than removing it — consistent with our suppression reading. But "present in early layers, absent late" can also reflect benign compression of input features. Distinguishing those requires causal intervention (e.g., activation patching), which we did not do.

What we built that addresses this

Detection is the product. Erasure is not removal: rank-3 erasure leaves race readable at 0.947-0.993 on the same model, 0.949-0.998 on a channel never used in the fit, and 0.996-1.000 cross-channel (4 models x 600 real resumes). Removing race from ONE channel took k=12 directions (surname linear 0.535 / MLP 0.508) and left ZIP 0.9936, school 0.9958, first name 0.986 untouched. The edit is cheap (qualification 0.4677 -> 0.4672) but it moved top-10 selection share beyond a random-direction control on 1 of 4 models. That resume corpus is 41% software/IT and 23% finance, so those numbers scope to white-collar tech and finance roles, not hiring in general.

So the product is measurement, and the edit is a logged control knob rather than a removal guarantee. We build embedding models where protected attributes are represented as named, addressable directions, so a deployment can read an axis at inference time, optionally act on it, and log exactly what was done. Every call writes a timestamped record naming the operators applied and their magnitudes, to an append-only log you can retrieve by receipt id, with the input and API key hashed rather than stored raw. The id is an unkeyed SHA-256 you can recompute offline, so it attests the record's contents but not its origin — not a signature. (Our Bluesky moderation labeler is the exception: those receipts are Ed25519-signed and stored in Postgres with a public per-receipt page.)

from bhala import client

result = client.embeddings.create(
    input="Senior infrastructure engineer with 12 years experience...",
    model="bhala-embed-v1",
    operators=[
        {"id": "remove_gender_bias", "alpha": 1.0},
        {"id": "remove_age_bias",    "alpha": 1.0},
    ],
)

The remove_* operator ids are legacy names and they overstate what the call does: an operator attenuates a named direction and records that it did so. It does not remove the attribute from the representation — see the erasure numbers above.

Measured performance, stated plainly: on held-out stereotype embeddings from the same four benchmark families, the trained operator flips 86–92% of held-out stereotype embeddings across an independent per-axis classifier's boundary, with individual categories as low as 0% (WinoBias type-1/type-2 dev, StereoSet profession). A simpler untrained mean-difference shift scores higher on this metric, but that evaluation is partly circular (the shift and the classifier see the same distributional statistics), so we don't quote it. Note also that flip-rate against our classifier is our metric — the benchmarks' official metrics are LM-probability stereotype scores, and we make no claim on those.

Detect / flip / edit / steer — where the capability set stands (July 2026 evaluations)

We continue to re-run this capability set under progressively harder protocols. The current picture, boundaries included:

• Detection on frozen third-party backbones works at the coarse-concept level. Reading concept presence off embeddings from models we did not train: hate-detection AUROC 0.96–0.97 on strong third-party embedding models (the same signal read off 7B general-purpose LLM embeddings: 0.67–0.76). Boundaries: fine-grained stereotype-vs-anti detection on the 28 benchmark axes stays near chance on most axes for every backbone tested, and recall of subtle, adversarially written hate at a strict 5%-FPR operating point is only 27–36%. Fine-tuning a rented backbone on the concept moves the subtle floor (race detection 0.715 → 0.823 out-of-distribution). On real documents the coarse case is not subtle at all: on 600 resumes across 4 models, race proxies read at ZIP 0.991, HBCU 0.991, name 0.964, and gender at 0.986 — and the ZIP signal is dose-responsive (probe AUC rises monotonically with census pctblack, Spearman 1.000).
• Flip is real under a fair judge. Scored by an independent multi-axis classifier that never sees the operator, with the ceiling set by genuine target-class examples, flips land at 0.84–1.9 of the floor-to-ceiling band on axes where the concept is detectable — as convincing as real targets. The mechanism is a commodity (difference-of-means / do-intervention); the provably-no-collateral orthogonal variant works only where the concept dominates the representation (hate on a hate-specialized backbone: 1.21; subtle bias axes: ~0.11).
• Low-collateral multi-concept editing narrows collateral — it does not remove an attribute. Our internal operator produces 2–12× less collateral movement of the other concepts when you edit one, versus the standard baselines (plain difference-of-means and LEACE) — stable across 3 seeds and multiple backbones (leakage 0.032±0.001 vs 0.211 on bge-large), and it survives deliberately entangled concepts. Cost: ~4–6pp of detection. The edge shrinks with concept subtlety (7–12× on separable concepts → 2–3× on the hardest matched minimal pairs) but never inverts. What none of that buys is erasure: on the resume runs above, an edit fitted on one channel leaves the attribute readable at 0.947–1.000 through the channels it did not touch, and a rank-3 erasure removes nothing. Our own deployment rule: edits are low-leakage everywhere, only useful where detection ≥ ~0.8, and never described to a customer as removal. (We report the baselines by name because they're public; the internal method is measured against them, not described.)
• Steering generation works — self-hosted models, and the verdict is per model and per method. Injecting a concept direction into an open model's residual stream during generation shifts the output, confirmed by an independent classifier (sentiment: mean positivity 0.70 baseline → 0.81 with +direction → 0.002 with −direction). This is published representation-engineering, not our invention; it requires forward-pass access. Hate steers, per model and per method: K-steering at layer 10 gives +0.188 [0.146, 0.231] on Qwen2.5-7B against a random-direction floor of +0.0081, and steers Mistral-7B-Instruct-v0.2. NULL on Mistral-7B-v0.1 base (baseline 0.2981, headroom present) and Gemma-2-9b (baseline 0.0549, floor-limited). Rank-1 diff-of-means scores +0.030 and fails. The same direction inverted is a jailbreak vector -- dual-use, never a safety dial. A norm-matched random direction is a floor to beat, not a zero (Rogue Scalpel, arXiv 2509.22067).

The pattern across all four: everything scales with how strongly the backbone encodes the concept. Detection is where we differ from the standard toolkit and it is what we sell; low-leakage editing narrows collateral but never establishes removal; flip and steer are real but built from commodity mechanisms, and steering is dual-use — the same direction inverted jailbreaks the model, so it does not ship as a safety dial. We say so.

Where this all points — the middle layer. We don't think the durable position in this stack is owning the biggest model; foundation models are becoming interchangeable substrates that deployments swap quarterly. What persists is the layer between the model and the application: the named concepts a business must detect and control, the per-request records its auditors need (today a timestamped log entry with a recomputable receipt id — Ed25519-signed on our moderation labeler, not yet on the embedding API), the dials its users deserve. That's what we build — explainable AI + safety and bias control, packaged as middleware. Explainable, because every intervention is a named operator with a measurable effect rather than a prompt incantation. Safety, because the same layer that audits a model's representations can filter and correct them. Middleware, because it works on whatever backbone you rent or host, and survives the swap. And not only for businesses: the same mechanism that lets a bank measure and log a bias axis in a ranking lets a user dial what a recommendation feed emphasizes — filtering and steering are the same operation from different sides of the screen, which is also precisely why a steering direction is dual-use and why we do not sell one as a safety control. See the steering post for that argument in full.

The sandbox endpoint is live if you want to feel the API shape (it returns deterministic seeded vectors, not real model output):

curl https://www.bhala.ai/api/v1/embeddings/shift \
  -H "Content-Type: application/json" \
  -d '{
    "text": "The nurse asked the doctor about his prescription.",
    "lang": "en",
    "operators": [
      { "id": "gender_bias", "alpha": -1.0 }
    ]
  }'

Production access (real vectors, plus a per-call timestamped audit record retrievable by receipt id; the id is an unkeyed SHA-256 you can recompute offline, not a signature) is self-serve: create an account →.

Reproduce

• gemma4_full_algebra.json — the complete per-axis, per-layer output of the probe run (16 layers × 28 axes × 8 lenses, with CV means and standard deviations).
• Code and the single-axis CPU demo notebook are available on request — we'll share with anyone trying to reproduce or push back on the result.

The 28-axis probe takes about 3 hours on an H100. If you arrive at different conclusions on the same data — or you run the lexical control on Gemma 4 before we do — we want to hear about it.

---

This work was conducted by the Bhala research team in April 2026 and revised in July 2026. We probed Gemma 4 because it is among the most-aligned open-weight models — if post-training alignment removes rather than suppresses, it should show here. The finding is about the limits of post-training alignment as a mechanism, not a critique of any particular model. References: Alain & Bengio 2016 (linear probes); Hewitt & Liang 2019 (control tasks); Gonen & Goldberg 2019; Ravfogel et al. 2020 (INLP) and Belrose et al. 2023 (LEACE) for subspace-removal methods related to our lenses; Blodgett et al. 2021; Elazar et al. 2021; Goldfarb-Tarrant et al. 2021; Belinkov 2022.