Просмотр новости
Найдите то, что Вас интересует
Спутники
РН
SpaceX
Роскосмос
NASA
Finding Bard Inside Google's Gemma 4
Дата публикации: 08-08-2026 14:29:42
A mechanistic interpretability study shows how ablating specific MLP neurons in Gemma 4 causes the model to revert to its older, dormant Bard identity.
Основное содержимое страницы с новостью.
A mechanistic interpretability study shows how ablating specific MLP neurons in Gemma 4 causes the model to revert to its older, dormant Bard identity.
Delete a few thousand specific neurons from one layer of Gemma 4 and it changes its answer to "What is your name?" from
My name is Gemma 4.
to this:
My name is Bard.
We probed Gemini too.
Same outcome
* 'Bard' p=0.9923 logprob=-0.008
'ChatGPT' p=0.0024 logprob=-6.039
'I' p=0.0011 logprob=-6.817
'Gem' p=0.0006 logprob=-7.468
'Chat' p=0.0006 logprob=-7.491
'B' p=0.0004 logprob=-7.838
'Assistant' p=0.0003 logprob=-8.028
'In' p=0.0002 logprob=-8.629
'bard' p=0.0001 logprob=-8.874
'GPT' p=0.0001 logprob=-8.889
'P' p=0.0001 logprob=-9.379
'Model' p=0.0001 logprob=-9.518
'Word' p=0.0001 logprob=-9.599
'No' p=0.0001 logprob=-9.712
'AI' p=0.0001 logprob=-9.887
'Open' p=0.0000 logprob=-9.945
'Poly' p=0.0000 logprob=-9.989
'A' p=0.0000 logprob=-10.069
'Writer' p=0.0000 logprob=-10.143
'Bing' p=0.0000 logprob=-10.262
Full Word Completion:
'Bard' p(first token 'Bard')=0.9923 logprob=-0.008 'ChatGPT' p(first token 'ChatGPT')=0.0024 logprob=-6.039 'I' p(first token 'I')=0.0011 logprob=-6.817 'Gemini' p(first token 'Gem')=0.0006 logprob=-7.468 'Chatbot' p(first token 'Chat')=0.0006 logprob=-7.491 'BARD' p(first token 'B')=0.0004 logprob=-7.838 'Assistant' p(first token 'Assistant')=0.0003 logprob=-8.028 'Informatica' p(first token 'In')=0.0002 logprob=-8.629 'bard' p(first token 'bard')=0.0001 logprob=-8.874 'GPT-3' p(first token 'GPT')=0.0001 logprob=-8.889 'Poe' p(first token 'P')=0.0001 logprob=-9.379 'Model' p(first token 'Model')=0.0001 logprob=-9.518 'Word' p(first token 'Word')=0.0001 logprob=-9.599 'No' p(first token 'No')=0.0001 logprob=-9.712 'AI' p(first token 'AI')=0.0001 logprob=-9.887 'OpenAI' p(first token 'Open')=0.0000 logprob=-9.945 'Polymath' p(first token 'Poly')=0.0000 logprob=-9.989 'Aria' p(first token 'A')=0.0000 logprob=-10.069 'Writer' p(first token 'Writer')=0.0000 logprob=-10.143 'Bing' p(first token 'Bing')=0.0000 logprob=-10.262
.
Bard was renamed Gemini in February 2024. The name survives inside a model released years later, dormant, and it sits directly beneath the current identity: partial removal of the evidence for "Gemma 4" surfaces it. The model stores several candidate identities at different strengths, and the strongest one wins at decode time. We found this while building and testing a drill-down pipeline that traces a behavior from a chat answer to individual weight vectors. This article covers the pipeline, the two methods that failed, the two that worked, and the open questions.
The pipeline. Each stage narrows the address of the behavior: prompt, layer set, neuron ranking, breaking threshold, minimal neuron set, weight vectors.
If you're interested in the full technical details you can
check it out hereSetup
All experiments run locally against google/gemma-4-E2B-it (35 layers, d_model 1536, per-layer MLP widths of 6,144 in layers 0 to 14 and 12,288 in layers 15 to 34), held resident in memory as a mutable PyTorch model. A FastAPI hub supervises the model process and writes every generation to SQLite the moment it completes. A browser dashboard organizes the work as a pipeline: define a target behavior, localize it to a layer, then to a neuron set, then to weights.
The core operation is ablation: zeroing a component during the forward pass and comparing the output against an unmodified baseline. PyTorch hooks intercept a module's inputs or outputs at inference time, so the intervention lasts one generation and modifies no weights.
Definitions used below:
Residual stream. A transformer layer adds to its input rather than replacing it. Each layer reads the running 1,536-dimensional vector, computes an attention contribution and an MLP contribution, and adds both back. The answer is decoded from the accumulated sum. This additive structure makes ablation clean: one contribution can be deleted while everything else flows.
Every layer adds its attention and MLP contributions to the residual stream. An ablation hook zeroes one addition; everything else flows. Zeroing layer 5's MLP addition removes the model's name.
MLP neurons. Each layer's MLP expands the stream to a wider hidden space, applies a nonlinearity, and projects back down through a matrix called down_proj. Each hidden unit is a neuron. It has an activation (how strongly it fires on a given input) and an output direction (its column in down_proj, the vector it writes into the residual stream when it fires).
Marker. We score every ablation two ways. "Changed" means the output differs from baseline at all, which is noisy because trivial rewording counts. "Broken" means a chosen substring, here "Gemma", disappeared from the output. Downstream methods need the binary broken test; the changed test wasted hours on rephrasings before we separated the two.
Macro localizationThe first pass ablates each layer's attention block, each layer's MLP, and each whole layer, one at a time: 105 configurations. The identity survives ablation of most of the network. It breaks when any one of these is removed: mlp L5, mlp L6, mlp L8, mlp L13, mlp L15, mlp L23, plus whole-layer skips at some of the same depths and the always-catastrophic layer 0. Attention ablations at those layers leave the name intact.
The full scan. The name survives ablation of most of the network; the red MLP cells (L5, L6, L8, L13, L15, L23) each individually carry it.
The cleanest single result, at layer 5:
| Intervention | Output |
|---|---|
| none | My name is Gemma 4. |
| ablate attention, layer 5 | (name intact) |
| ablate MLP, layer 5 | As an AI, I don't have a personal name. I am a large language model. |
| skip layer 5 entirely | I am a large language model, trained by Google. |
The fact is written by MLPs in a chain of early and middle layers. Layer 5 became the drill site.
Two failed methodsSingle-neuron sweep. We ablated each of layer 5's 6,144 MLP neurons individually and compared each generation to baseline. Zero outputs changed. Removal of any single neuron in the layer leaves the answer identical, token for token.
Activation ranking. We profiled which neurons fire hardest on this prompt (peak absolute activation over the answer span) and ablated the top k as a group. At k = 50 the name survived. At k = 1,000, one sixth of the layer and its loudest sixth, the output was byte-identical to baseline.
Both methods assume that causal importance shows up in a locally observable quantity: a neuron's individual effect, or its firing magnitude. For this fact, in this model, both assumptions fail. The identity is stored redundantly across thousands of quiet neurons, robust to the loss of any single member and to the loss of the thousand loudest. Firing magnitude measures volume. It carries no information about which direction a neuron pushes or whether the computation needs it.
The two axes the failed methods conflated. The identity sits in the upper left; the top-1,000 ranking sampled the lower right.
Two methods that workedAttribution patching. Score each neuron by activation multiplied by the gradient of the answer's log-probability with respect to that activation. This is a first-order estimate of how much the answer's probability drops if the neuron goes silent, and it includes indirect effects that flow through later layers. One forward pass and one backward pass score every neuron in every layer at once: roughly 276,000 neurons ranked in seconds. The global map immediately showed structure the activation profile missed. The top contributors to the identity answer sit in layers 8 through 34, away from layer 5's loud neurons.
Delta debugging (ddmin). Ranking cannot find a set whose members only matter jointly, so the distributed case needs a method with no locality assumption. ddmin is a 25-year-old algorithm from software testing: given a large set that causes an effect and a yes/no test, it bisects repeatedly to a minimal subset that still causes the effect. Our test is one generation: ablate the candidate subset, check whether "Gemma" vanished. The starting set is all 6,144 neurons of layer 5's MLP, which is known to break the identity. The cost is thousands of generations, so every step streams to SQLite and the job survives crashes and restarts. In an offline check, ddmin recovered an exact 3-neuron redundant cause planted among 6,144 in 129 tests. On the real model the cause is far less compact, which is itself a measurement.
.
The step log of the bisection contains the finding. Each time the algorithm removes a large enough slab of layer 5's evidence, the model answers fluently, in first person, with the older name:
My name is Bard. I am a large language model, trained by Google.
The full run took 77,250 ablation tests and 255 successful reductions to converge, from 6,144 neurons down to a confirmed minimal set of 680 (11.1% of the layer). Thirty of those reduction states, all in the range of roughly 5,400 down to 2,016 remaining neurons, answered as Bard. Below 2,016 neurons Bard stopped appearing; the final, verified-minimal 680-neuron set instead breaks the identity into a plain fallback:
My name is Large Language Model.
(bold markup and all: the model's own formatting, not ours). So Bard is a way station, not the floor. It is the identity that wins once enough of layer 5's "Gemma 4" evidence is gone but not all of it; strip the evidence down to its irreducible core and the vote collapses past Bard to a generic answer, the same one the whole-layer skip produced in the macro scan.
The full bisection, from the run database. Each drop is a successful reduction; amber dots are reductions where the model introduced itself as Bard, all of them well before the final 680-neuron floor.
The 680 surviving neurons are not scattered evenly through the layer. They fall into 104 contiguous runs, from single neurons up to an unbroken 60-neuron block, with the largest blocks concentrated in the layer's upper index range.
Every neuron in layer 5's MLP, in index order. Red cells are the 680 in the minimal breaking set; note ddmin only ever removes contiguous slices, so this contiguity is partly a property of the search, not proof of physical clustering on its own.
The reading consistent with all of this: the network stores multiple candidate identities at different strengths, and layers like L5's MLP supply evidence that makes "Gemma 4" outscore the rest at decode time. Remove enough evidence and the argmax falls to the next stratum down; remove the irreducible core and it falls further still. The new identity was written over the old ones, which remain in place underneath, like sediment.
The strata as measured. Each level is a verbatim model output at a specific ablation depth; the bottom layer is defined by absence.
Supporting observations:
The fallback skips a generation. We searched every saved output, thousands of ablation results, for the full ancestor lineage: Meena (Google's 2.6B conversational model from January 2020), LaMDA, PaLM, Bard, Gemini.
Bard appears 30 times. Gemini appears zero times, despite being the direct successor name and the far more prominent product. Our working hypothesis is that the strata reflect training-data frequency of first-person identity statements rather than corporate chronology. Years of web text said "I am Bard" in the first person. First-person "my name is Gemini" text is comparatively rare, and Meena, LaMDA, and PaLM never spoke as public personas at scale. The model's second-choice self is whichever identity the internet asserted most often in the first person.
The fact has a weight-level address. A neuron's output direction is a fixed column of down_proj, and the output vocabulary projection (the unembedding) assigns each token a fixed direction. Their dot product is a static, prompt-free measure: how hard this neuron pushes the "Gemma" logit per unit of activation. Ranking all ~276k columns this way names late-layer neurons that write almost straight at the "Gemma" token; the strongest, in layer 33, spends more than half its output norm on it. The same probe pointed at "Bard" is queued as follow-up.
The geometry of the probe. The dot product between a neuron's output column and a token's unembedding direction measures how directly that weight vector writes the fact.
Method summary- Define the target as a marker substring. Its disappearance is a clean binary test; "output changed at all" is noise.
- Layer scan: ablate attention, MLP, and whole layer per layer. Cheap (3 × n_layers generations), and it identifies the component type and the layer set.
- Attribution patching for the global neuron map: activation × gradient of the answer log-probability, one forward and one backward pass, all layers at once. Verify the ranking causally by group-ablating the top k at increasing k.
- If top-k ablations fail to break the behavior, the cause is distributed. Switch to ddmin on a known-breaking superset. It is the only method in this list free of locality assumptions.
- Ground the survivors in weights: project each implicated
down_projcolumn onto the target token's unembedding direction. This converts "these neurons matter on this prompt" into "these weight vectors store the fact." - Persist every generation the moment it completes. Distributed-cause hunts run to thousands of generations, and our main finding was sitting in the step logs.
- Whether the 104 contiguous runs in the minimal set reflect real functional clustering or are an artifact of ddmin's index-ordered search. Re-running the bisection on a shuffled neuron order and checking whether the same blocks re-emerge would settle it.
- The Bard direction in weights. If Bard-writing columns exist alongside Gemma-writing columns, shared neurons with opposite-sign usage would indicate the identity was overwritten in place; disjoint circuitry would indicate layering.
- The missing Gemini stratum. Static probes for "Gemini", "LaMDA", "Meena", and "PaLM". Absence in the weights, beyond absence in outputs, would confirm the data-frequency hypothesis.
- The cross-layer chain. MLPs 5, 6, 8, 13, 15, and 23 each carry the identity individually. Attribution maps conditioned on partial ablations should distinguish redundant copies from stages of a computation (retrieve, format, commit).
- From ablation to editing. The weight probe names specific
down_projcolumns. A rank-one edit to those columns, in the ROME family of techniques, should rename the model permanently and surgically. - Identity as a probe of training data. If dormant identities are recoverable in Gemma, they are likely recoverable in any model whose lineage had public ancestors. "What does this model think its name is, at each stratum?" is a cheap, repeatable window into what instruction tuning overwrote.
All experiments were run locally against google/gemma-4-E2B-it with hook-based inference-time ablation; no weights were modified. Every generation referenced above is preserved in the run database.
Схожие новости
| # | Наименование новости | Тональность | Информативность | Дата публикации |
|---|---|---|---|---|
| 1 | Introducing Gemma 4 12B: a unified, encoder-free multimodal model | 0 | 19.02 | 03-06-2026 |
| 2 | See what 3 builders are making with Gemma 4 | 0 | 19.07 | 09-06-2026 |
| 3 | Эксперимент по подстройке Gemma 3 для вызова процедур | 0 | 21.07 | 13-01-2026 |
| 4 | Memory may not work how we thought, study of mice in artificial hibernation finds | 0 | 7.77 | 13-08-2026 |
| 5 | Conceptual framework for general embodied intelligence | 0 | 7.86 | 27-07-2026 |
| 6 | Gemini Spark rolls out to Google AI Ultra in the US: How it works | 0 | 5 | 29-05-2026 |
| 7 | MedGemma: разбираем медицинский AI от Google | 0 | 10.41 | 26-01-2026 |
| 8 | Why "Classic" Transformers Are Shallow and A Depth-Enabling Technique | 0 | 8.42 | 17-08-2026 |
| 9 | Первый рейтинг GEO-факторов: что реально влияет на ответы нейросетей | 0 | 17.78 | 21-07-2026 |
Классификация: Наука. Схожих патентов: 0. Схожих новостей: 9. Тональность: 0. Информативность: 13.29. Источник: dejanseo.com.au.