Okay, here is the abstract and summary for the provided research paper transcript.
Abstract:
This paper introduces Circuit Tracing, a mechanistic interpretability methodology designed to reveal the computational mechanisms underlying language model behaviors. Addressing the challenge posed by polysemantic neurons and superposition, the method employs Cross-Layer Transcoders (CLTs) to learn sparse, interpretable features that replace the standard Multi-Layer Perceptron (MLP) blocks of an LLM, forming an interpretable "replacement model." For analyzing specific prompts, a "local replacement model" is constructed by incorporating reconstruction error adjustments and freezing attention patterns and normalization denominators, ensuring it exactly replicates the original model's output for that prompt. The core analytical tool is the "attribution graph," derived from this local model, which precisely maps the linear causal influence between input embeddings, active features, reconstruction errors, and output logits for the targeted output token. The paper details the construction, pruning, interactive visualization, and validation of these graphs through perturbation experiments (interventions). Case studies on an 18-layer transformer model (acronym generation, factual recall, small number addition) illustrate how attribution graphs reveal interpretable computational pathways and heuristics. Quantitative evaluations demonstrate that CLTs offer Pareto-improvements over per-layer transcoders and thresholded neurons in terms of model fidelity and graph simplicity, while validation experiments show that graph-derived influence predicts intervention outcomes. The paper discusses significant limitations of the current approach, such as the inability to explain attention circuit formation and handling inactive features, and previews a companion paper applying these methods to a frontier model, Claude 3.5 Haiku.
Circuit Tracing: Revealing Computational Graphs in Language Models
§ 1 Introduction: Deep learning models' computations occur across many "neurons." Mechanistic interpretability aims to describe these computations in human-understandable terms. Standard neurons are often polysemantic due to superposition, hindering interpretation, especially in language models. The paper proposes using sparse coding methods like Cross-Layer Transcoders (CLTs) to identify interpretable "features" as building blocks for understanding model "circuits." The core approach involves building a replacement model with CLT features, analyzing prompt-specific attribution graphs, and validating with interventions.
§ 2 Building an Interpretable Replacement Model:
§ 2.1 Architecture: Cross-Layer Transcoders (CLTs) consist of sparse features designed to reconstruct MLP outputs. Each feature at layer ℓ reads from the residual stream at layer ℓ but can write to (decode to reconstruct MLP output in) all subsequent layers (ℓ through L). They are trained using a reconstruction error loss and a sparsity penalty, employing the JumpReLU nonlinearity.
§ 2.2 From CLT to Replacement Model: A replacement model is constructed by substituting the original model's MLPs with the CLT's reconstructions. Attention layers remain. This model approximates the original model's next-token predictions (~50% match for a large 18-layer model CLT), demonstrating better fidelity than per-layer transcoders or thresholded neurons at comparable sparsity levels.
§ 2.3 The Local Replacement Model: For analyzing a specific prompt, a "local replacement model" is created. It uses the CLT substitution but incorporates the original model's attention patterns, normalization denominators (frozen), and adds the CLT's reconstruction error as a constant adjustment term. This model exactly reproduces the original model's activations and output logits for that prompt and serves as the basis for constructing attribution graphs, enabling linear attribution between features.
§ 3 Attribution Graphs:
§ 3.1 Constructing an Attribution Graph for a Prompt: Causal graphs are built for a specific prompt's output token. Nodes include input embeddings, active CLT features at each token position, error nodes (unexplained MLP output), and output logit nodes. Edges represent direct, linear causal influence in the local replacement model, computed efficiently via backwards Jacobians with stop-gradients on nonlinearities. Graphs are pruned to manage complexity, retaining the most influential paths (e.g., reducing nodes by ~10x while keeping ~80% of influence).
§ 3.2 Learning from Attribution Graphs: An interactive visualization interface is developed to explore and trace key paths through the often complex pruned graphs.
§ 3.3 Understanding and Labeling Features: Features are manually interpreted and labeled based on their activation patterns across dataset examples ("feature visualizations"). Examples include input features (specific tokens), output features (promoting specific tokens/categories like "say X"), and more abstract features.
§ 3.4 Grouping Features into Supernodes: Similar features with related roles on a prompt are manually grouped into "supernodes" to simplify analysis and visualization of key mechanisms.
§ 3.5 Validating Attribution Graph Hypotheses with Interventions: Hypotheses derived from graph structure are tested by perturbing (e.g., inhibiting or amplifying) feature activations in the original model (with fixed attention patterns). "Constrained patching" modifies a feature's decoding across a specified range of layers. Perturbation effects on downstream features and model outputs are measured and compared to graph predictions.
§ 3.7 Factual Recall Case Study: Analysis of "Fact: Michael Jordan plays the sport of" shows parallel computational paths from entity information ("Michael Jordan" features) and task context ("sport of" features) converging to promote the correct output ("basketball").
§ 3.8 Addition Case Study: Analysis of "calc: 36+59=" (using Haiku 3.5) reveals distinct parallel processing pathways related to computing the final digit (modulo 10) and the approximate magnitude of the sum, mediated by "add," "lookup table," and "sum" features. This supports a view of addition as a "Graph of Heuristics."
§ 4 Global Weights:
Analyzing prompt-independent feature interactions ("virtual weights" between CLT encoders/decoders) provides a more global view of mechanisms.
A major challenge is "interference weights" – spurious large weights between features that rarely co-activate on data.
Weighting virtual weights by feature co-activation statistics (e.g., Target-Weighted Expected Residual Attribution - TWERA) helps filter interference and improve interpretability of global connections.
§ 4.1 Global Weights in Addition: Global weight analysis on addition features reveals interpretable connections consistent with the "Graph of Heuristics" structure observed in prompt-specific graphs.
§ 5.1 Cross-Layer Transcoder Evaluation: CLTs achieve Pareto-improvements over baselines (per-layer transcoders, thresholded neurons) in terms of reconstruction error/KL divergence vs. sparsity/interpretability (measured via LLM-based sort/contrastive evals). Feature quality generally improves with scale.
§ 5.2 Attribution Graph Evaluation: CLT-based graphs are quantitatively simpler (shorter average path length) and more complete (lower influence from uninterpretable error nodes) compared to graphs from baselines. Pruning is effective at reducing complexity while retaining influence.
§ 5.3 Evaluating Mechanistic Faithfulness: Intervention experiments show that graph-derived node influence on logits and pairwise feature influence are reasonably predictive of ablation effects in the underlying model. While local replacement model perturbation responses match the underlying model well shortly after intervention, discrepancies compound significantly over layers.
§ 6 Biology: A companion paper applies Circuit Tracing methods to study a diverse range of behaviors (multi-step reasoning, planning, multilingualism, hallucinations, safety refusals, etc.) in the Claude 3.5 Haiku model.
§ 7 Limitations: Key limitations include: inability to explain attention pattern formation (missing QK-circuits); impact of CLT reconstruction errors ("dark matter"); difficulty in analyzing mechanisms involving inactive or inhibitory features; inherent complexity of graphs even after pruning; challenges with features being at an awkward level of abstraction (splitting/absorption); difficulty in understanding prompt-independent "global circuits" fully due to interference and attention mediation; and gaps in mechanistic faithfulness between the replacement and underlying models.
§ 8 Discussion: Summarizes the rationale behind core methodological choices (learned features, transcoders, CLTs, linear attribution) and reiterates the current limitations as key directions for future research. Highlights the insights gained from the addition case study regarding complex, heuristic computations.
Appendix: Provides extensive technical details on CLT implementation, compute requirements, graph computation formulas, pruning algorithms, validation experiment setups and results, notes on the interactive interface, residual stream norm growth, further analysis of interference weights and number output weights, and details on evaluation datasets.