Professor Nir Shavit presents "combinatorial interpretability," a novel approach to understanding the internal logic of neural networks. Moving beyond analyzing activations, this method simplifies the problem by focusing on the signs (+, -, 0) of the weights and biases within a network trained on a well-defined task: computing boolean expressions. The central discovery is that neural networks, through standard gradient descent, spontaneously learn to implement "feature channel coding." In this scheme, the computation of a single logical feature (e.g., an AND clause) is distributed across a specific set of neurons, forming a non-random "code." The research reveals that the network learns to compute positive instances of a feature using AND-like logic and negative instances using XOR-like logic. This framework provides a mechanistic explanation for complex phenomena like neural network scaling laws, where larger models achieve lower error by forming more robust and distinct feature codes. The talk concludes by introducing "cascading feature disentanglement," a proposed technique to extend this analysis to larger, more complex networks, with the ultimate goal of understanding computation in both artificial and biological neural systems.
How Neural Networks Might Think: A Combinatorial Approach to Interpretability
0:02:45 The Challenge of Interpretability: Professor Shavit explains that understanding how neural networks compute is difficult due to "polysemanticity," where a single neuron is involved in representing many different features. This is explained by the "superposition hypothesis," which suggests networks pack more features than they have neurons into a high-dimensional space.
0:07:02 A Simplified, Controllable Model: The proposed "combinatorial interpretability" approach simplifies the problem by analyzing the signs (positive, negative, zero) of network weights rather than their exact values. It focuses on a controlled task—training a network to compute a specific boolean formula (a Disjunctive Normal Form, or DNF)—to know precisely which features the network is supposed to learn.
0:13:31 Key Hypothesis: Feature Channel Coding: The core idea is that a network represents a single feature (e.g., x2 AND x4) not with one neuron, but by distributing its computation across a group of neurons. This group forms a "feature channel code." This allows the network to compute many features simultaneously, as long as their codes are sufficiently distinct.
0:18:05 A Key Discovery with Trained Networks: When a simple two-layer network is trained to compute a boolean formula, its weight matrix initially appears random. However, by reordering the matrix based on the features and separating neurons by their positive or negative contribution, a clear, learned structure emerges: the network spontaneously discovers and implements feature channel coding.
0:22:00 The Network's Learned Logic: The analysis reveals that gradient descent teaches the network a specific computational strategy:
Positive Logic (AND): Neurons that positively contribute to the final output learn to compute the AND of the input variables for a given feature.
Negative Logic (XOR): Neurons that negatively contribute learn an XOR-like function, identifying cases where only some, but not all, required variables are present, thus providing negative evidence.
0:34:48 Explaining Scaling Laws: This model provides a concrete explanation for why larger networks perform better. With more neurons, the network can create more robust and less overlapping feature channel codes for each feature, which directly leads to a reduction in computational error.
0:37:58 Visualizing How Networks Learn: The talk shows how these feature codes emerge during training. The network first quickly learns the primary positive (AND) codes to get a basic result, and then refines the negative (XOR) codes over subsequent epochs to eliminate the remaining error.
0:42:21 Scaling to Larger Networks: Disentanglement: To apply this to real-world networks where feature representations are "entangled," Professor Shavit introduces "cascading feature disentanglement." This is a proposed method to mathematically unravel the jumbled weight matrices of larger models to reveal the clean, underlying feature codes.
0:51:45 Vision for the Future: This research provides a new lens to understand the fundamental mechanics of neural computation. The ultimate goal is to scale these techniques to analyze large AI models and potentially apply the same principles to understand computation in biological neural tissue.