Yao Fu | Website | Twitter / X

Dec 27 2024

I have been thinking about the basic ingredients of language models’ reasoning for a while. In 2022 I and a group of concurrent researchers proposed the hypothesis that coding improves reasoning, and that hypothesis elicits quite a few inspiring research. So I feel like thought-provoking hypothesis should be beneficial for the LLM / AI research community. Asking the correct question is equally important as answering it.

Today I discuss three unit hypotheses that I have been pondering upon. Despite the lack of concrete evidence, I tend to believe all of them are true. To verify them it would require rigorous design and execution of controlled experiments, but may not require much compute. So I would like to share them and hopefully encourage the pursuit of science of LLMs. Let’s turn this alchemy into rigorous science.

In short: we hypothesis there simultaneously exist memorization and reasoning circuits within the model architecture. The structure of the reasoning circuits within the model should match the computational graph of the sub problem. The complexity of the graph that can be computed within one single forward pass is about 10 - 100 nodes and edges, and a model less than 10B activated parameters, if trained well, should be able to execute most reasoning graphs within known human knowledge.

Disclaimer: the content of this doc is purely fundamental science from a personal perspective. The formulation and the thinking process does not require any knowledge about any proprietary recipe of building LLMs from any company. Our discussion is solely based on open-source models, research papers, and first-principle thinking. This article does not discuss popular topics like scaling inference time compute.

Table of content

0 - Preliminaries: reasoning as DAG

A reasoning problem is a directed acyclic computational graph

• A reasoning problem = a directed acyclic computational graph
• A model that does reasoning = the model executes an algorithm that traverses the reasoning graph
  • This means reasoning = executing the algorithm defined by the DAG = traversing the DAG and on-the-fly compute the intermediate variables until output
  • The execution of reasoning can be either implicitly within the model architecture (after grokking) or explicitly with CoT. Here grokking means internally the model has constructed a circuit that has the same structure of the reasoning problem. CoT means the model internally does not have this DAG so it construct the DAG on-the-fly through decoding.
• A model that does memorization means = the model memorizes all possible input-output pair of this DAG, but does not actually execute the DAG
  • This means we cannot probe the intermediate variables, because there are none.

An example of a reasoning problem formulated in a DAG

Question: John has 3 apples, and each apple weighs 150 grams. He gives half of his apples to his friend Mary. Mary already has 2 apples, each weighing 200 grams. What is the total weight of the apples Mary has now?