🔬 TorchDynamo: Deep Dive

Table of Contents

• 1. The Python Bytecode Model
  • 1.1 CPython Code Objects and the Value Stack
  • 1.2 PEP 523: The Frame-Eval API
  • 1.3 Hook Installation via set_eval_frame
  • 1.4 The Two Fast Paths
• 2. InstructionTranslator: Symbolic Bytecode Execution
  • 2.1 The Symbolic Value Stack
  • 2.2 Opcode Dispatch
  • 2.3 OutputGraph: FX Node Accumulation
  • 2.4 Inlining: InliningInstructionTranslator
• 3. VariableTracker Taxonomy
  • 3.1 Design Principle
  • 3.2 Major Subtypes
  • 3.3 Worked Example: y = x + 1
• 4. Guard Generation and GuardBuilder
  • 4.1 Source Annotations
  • 4.2 Guard Types and GuardBuilder Methods
  • 4.3 Guard Code Generation
  • 4.4 The guard_fn Structure
• 5. Graph Breaks: Mechanics and Implications
  • 5.1 Exhaustive Trigger Catalogue
  • 5.2 What Dynamo Does on a Break
  • 5.3 fullgraph=True Mode
• 6. The Compilation Cache in Depth
  • 6.1 The Per-Code-Object Linked List
  • 6.2 Cache Size and Fallback Policy
  • 6.3 The Persistent FX Graph Cache
  • 6.4 Clearing State
• 7. Debugging and Observability
  • 7.1 Backend Isolation Modes
  • 7.2 torch._dynamo.explain
  • 7.3 Logging and Bytecode Inspection
• References

1. The Python Bytecode Model

1.1 CPython Code Objects and the Value Stack

Every Python function is compiled by CPython into a code object (PyCodeObject), a C struct containing:

When CPython calls a function it allocates a frame (PyFrameObject) from the code object. The frame holds:

• Value stack — a fixed-size array of PyObject* pointers. The program counter (f_lasti) and stack pointer (f_stacktop) track execution state.
• f_locals — a C array of local variable slots, indexed by position in co_varnames.
• A pointer to the enclosing frame (f_back) forming the call stack.

The eval loop (_PyEval_EvalFrameDefault) fetches one instruction at a time, dispatches to a switch/computed-goto block keyed on the opcode, and manipulates the value stack. For instance:

• LOAD_FAST i — pushes f_locals[i] onto the stack.
• BINARY_ADD — pops two objects, calls PyNumber_Add, pushes the result.
• CALL_FUNCTION n — pops the top \(n+1\) stack slots (callee + \(n\) args), calls _PyObject_Call, pushes the return value.
• RETURN_VALUE — pops the stack top and returns it as the frame’s result.

Definition (Execution Frame). Formally, a frame \(F\) is the tuple \((C, \sigma, \text{pc}, \ell)\) where \(C\) is the code object, \(\sigma : \text{Varnames} \to \text{PyObject*}\) is the local variable store, \(\text{pc} \in [0, |C.\texttt{co\_code}|)\) is the instruction pointer, and \(\ell\) is the value stack.

1.2 PEP 523: The Frame-Eval API

PEP 523 (CPython 3.6+) added an eval_frame function pointer to PyInterpreterState. Before evaluating any frame, CPython checks:

// Simplified from CPython/ceval.c
if (tstate->interp->eval_frame != NULL) {
    return tstate->interp->eval_frame(tstate, frame, throwflag);
}
return _PyEval_EvalFrameDefault(tstate, frame, throwflag);

This gives any C extension the ability to intercept every function call and either handle it or delegate back to the default evaluator. Dynamo registers custom_eval_frame_shim (in torch/csrc/dynamo/eval_frame.c) as this hook.

Definition (Custom Frame Evaluator). A frame evaluator has the C signature:

\[ \texttt{eval\_fn}(\texttt{PyThreadState}^*, \texttt{PyFrameObject}^*, \texttt{int}) \to \texttt{PyObject}^* \]

The throwflag argument signals whether an exception is being propagated into the frame; Dynamo passes it through unchanged when falling back to the default evaluator.

Ansel et al. (2024): TorchDynamo installs a custom frame-evaluation function via PEP 523. On the first call, it compiles the frame; on subsequent calls with passing guards, it swaps in the compiled code object directly.

1.3 Hook Installation via set_eval_frame

torch._dynamo.optimize and torch.compile ultimately call into torch/_dynamo/eval_frame.py, which wraps the C function set_eval_frame(callback). This function stores the callback in PyThreadState (thread-local), so the hook is per-thread, not global:

# Simplified from torch/_dynamo/eval_frame.py
@contextmanager
def enable_dynamo(compiler_fn):
    prior = set_eval_frame(compiler_fn)   # installs hook, returns old value
    try:
        yield
    finally:
        set_eval_frame(prior)             # restores previous hook (or None)

1.4 The Two Fast Paths

When custom_eval_frame_shim is invoked for a frame, it immediately checks the compilation cache attached to the frame’s code object:

flowchart TD
    A["Frame arrives at shim"]
    B["Iterate cache linked list"]
    C{"guard_fn check"}
    D["Swap code object
run compiled code
fast path 1: guard hit"]
    E["Call _compile_frame()
fast path 2: cache miss"]
    F["Append CacheEntry(guard_fn, compiled_code)"]
    G["Run compiled code"]

    A --> B --> C
    C -->|pass| D
    C -->|fail: all entries exhausted| E --> F --> G

Fast path 1 (guard hit): guard evaluation is pure Python predicate evaluation with no allocations. The compiled code object is swapped in and the default evaluator runs it — Dynamo is entirely out of the hot path.

Fast path 2 (cache miss): _compile_frame is called, which instantiates InstructionTranslator, performs symbolic execution, emits an FX graph, compiles it via the backend, generates a guard function, and appends a new CacheEntry to the linked list.

2. InstructionTranslator: Symbolic Bytecode Execution

2.1 The Symbolic Value Stack

InstructionTranslator (defined in torch/_dynamo/symbolic_convert.py) is a bytecode interpreter that re-executes the function’s bytecode symbolically rather than concretely. The key difference from CPython’s eval loop:

• CPython’s value stack holds PyObject* — pointers to real Python objects.
• Dynamo’s symbolic value stack holds VariableTracker instances — abstract representations that carry both the object’s identity (for guard generation) and its FX proxy (for graph construction).

Definition (Symbolic Value Stack). The symbolic stack \(\hat{\ell}\) is a sequence of VariableTracker objects. Each tracker \(v \in \hat{\ell}\) carries: 1. A source annotation — a path expression describing how to reach the underlying Python object from the frame locals (used by GuardBuilder). 2. Optionally, an fx_proxy — a torch.fx.Proxy node in the OutputGraph representing this value’s computation.

2.2 Opcode Dispatch

InstructionTranslator contains one method per CPython opcode (or inherits it from a base class). The dispatch loop:

# Simplified from symbolic_convert.py
def step(self):
    inst = self.instructions[self.instruction_pointer]
    self.instruction_pointer += 1
    handler = getattr(self, inst.opname, None)
    if handler:
        handler(inst)
    else:
        self.unimplemented(f"opcode {inst.opname}")

Key opcode handlers:

2.3 OutputGraph: FX Node Accumulation

OutputGraph (in torch/_dynamo/output_graph.py) is the accumulator that builds the torch.fx graph incrementally as InstructionTranslator processes instructions. It wraps an fx.Graph and exposes three primary emission methods:

Definition (FX Node Emission). During symbolic execution, OutputGraph emits nodes of three kinds:

Each emission returns an fx.Proxy object, which is wrapped in a new TensorVariable and pushed onto the symbolic stack. This is how tensor computations propagate through the symbolic interpreter without actually running the computation.

2.4 Inlining: InliningInstructionTranslator

When the callee of a CALL_FUNCTION is a Python def that Dynamo can inline (i.e., not a graph-break-causing builtin), Dynamo does not emit a call_function node. Instead, it inlines the callee by spawning a child InliningInstructionTranslator:

flowchart TD
    P["Parent InstructionTranslator
sees CALL_FUNCTION targeting a Python def"]
    C["InliningInstructionTranslator (child)
shares same OutputGraph
receives args from parent's symbolic stack
executes callee bytecode symbolically"]
    R["Returns result VariableTracker
to parent stack"]

    P --> C --> R

Definition (Inlining). InliningInstructionTranslator is a subclass of InstructionTranslator that overrides RETURN_VALUE to return the top-of-stack tracker to the parent rather than finalizing the graph. It shares the parent’s OutputGraph, so any FX nodes emitted by the inlined function appear in the same graph fragment.

Inlining is critical for capturing helper functions, nn.functional calls that are implemented in Python, and user-defined utility functions that operate only on tensors. If inlining encounters an operation that would cause a graph break, the break propagates up to the parent.

def helper(t):
    if t.shape[0] > 10:   # data-independent: shape is static
        return t * 2
    return t + 1

@torch.compile
def f(x):
    return helper(x)

3. VariableTracker Taxonomy

3.1 Design Principle

VariableTracker (base class in torch/_dynamo/variables/base.py) is the abstract representation of a Python value during symbolic execution. Every object that Dynamo encounters — tensors, integers, functions, modules, lists — is wrapped in an appropriate subclass. The subclass determines three things:

1. What guards are installed when the value enters the symbolic stack (via make_guards).
2. What FX node is emitted when an operation is performed on the value (via call_function, __add__, etc.).
3. Whether a graph break occurs if the operation is unsupported.

VariableBuilder is the factory that constructs VariableTracker instances from real Python objects at frame entry, attaching source annotations.

3.2 Major Subtypes

3.3 Worked Example: y = x + 1

Let x be a torch.float32 tensor of shape (3, 4) passed as a local variable. The bytecode for y = x + 1 (under CPython 3.11) is approximately:

LOAD_FAST   x          # push x
LOAD_CONST  1          # push 1
BINARY_OP   +          # pop 1, pop x; push result
STORE_FAST  y          # pop result; bind to y

Dynamo’s InstructionTranslator processes each instruction:

1. LOAD_FAST x → look up x in the symbolic locals dict. VariableBuilder has already wrapped the concrete tensor at frame entry: it calls TensorVariable.__init__, records source LocalSource("x"), and installs guards via TENSOR_MATCH (checks dtype = float32, device = cpu/cuda, ndim = 2, shape = (3, 4) under static mode). The TensorVariable (carrying FX proxy %x) is pushed onto \(\hat\ell\).

2. LOAD_CONST 1 → the literal 1 is wrapped as ConstantVariable(1) with guard EQUALS_MATCH. No source annotation needed (literals have no runtime address). Pushed onto \(\hat\ell\).

3. BINARY_OP + → pops ConstantVariable(1) then TensorVariable(%x). Dispatch: TensorVariable.__add__(tx, [ConstantVariable(1)]). The tensor variable’s __add__ method:

   • Resolves that 1 is a constant, so it can be passed directly as an FX argument.
   • Calls OutputGraph.call_function(torch.Tensor.__add__, args=(%x, 1)).
   • OutputGraph emits an FX node: %add = call_function[target=torch.Tensor.__add__](args=(%x, 1)).
   • Returns a new TensorVariable carrying proxy %add. Pushed onto \(\hat\ell\).

4. STORE_FAST y → pops TensorVariable(%add) and binds it to y in the symbolic locals.

The FX graph after tracing y = x + 1 contains exactly two nodes: a placeholder for x and a call_function for the add. The constant 1 is not a graph node — it is a literal embedded in the call_function args.

4. Guard Generation and GuardBuilder

4.1 Source Annotations

Every VariableTracker carries a source field — a source annotation that encodes the path from the frame locals to the wrapped Python object. GuardBuilder uses this path to emit guard predicates that check the correct runtime object.

Definition (Source). A source is a tree of path accessors:

Sources compose: if model is a LocalSource("model") and its attribute weight is an AttrSource(LocalSource("model"), "weight"), then the guard for weight.dtype generates L['model'].weight.dtype == torch.float32.

4.2 Guard Types and GuardBuilder Methods

GuardBuilder (in torch/_dynamo/guards.py, with a performance-critical C++ implementation in torch/csrc/dynamo/guards.cpp) contains one method per guard type. At the time of writing, over 30 guard types exist. The most important:

Definition (Guard). A guard is a dataclass with three fields: - name: identifier string for the guarded variable - source: an enum indicating the source type (distinct from VariableTracker.source) - create_fn: a callable that is a method of GuardBuilder

Guards are created via VariableTracker.make_guards(GuardBuilder.METHOD_NAME), which calls self.source.make_guard(fn).

4.3 Guard Code Generation

GuardBuilder methods emit Python code strings that are joined with and into a single check_fn predicate. For example, EQUALS_MATCH appends:

# Emitted by GuardBuilder.EQUALS_MATCH for ConstantVariable(1) at LocalSource("y")
"L['y'] == 1"

TENSOR_MATCH emits a call to the compiled C++ helper ___check_tensors:

# Emitted by GuardBuilder.TENSOR_MATCH for TensorVariable at LocalSource("x")
"___check_tensors(L['x'], dtype=torch.float32, device='cpu', ndim=2, requires_grad=False)"

Multiple guards combine:

guard_code = (
    "___guarded_code.valid and "
    "___check_type_id(L['x'], <id>) and "
    "___check_tensors(L['x'], ...) and "
    "L['x'].shape[0] == 3 and "
    "L['x'].shape[1] == 4"
)

4.4 The guard_fn Structure

The final guard function has the Python signature:

Definition (Guard Function). Let \(\mathcal{L}\) be the dict of frame locals and \(\mathcal{G}\) the dict of frame globals. The guard function is:

\[ \texttt{guard\_fn}(\mathcal{L} : \texttt{dict}, \mathcal{G} : \texttt{dict}) \to \texttt{bool} \]

It is a pure Python function (with C helpers for tensor checks) generated by exec-ing the emitted code string into a fresh namespace. The L and G bindings refer to the dictionaries of live locals and globals respectively, which are constructed from the frame at the cache-lookup site.

class Config:
    dtype = torch.float32

@torch.compile
def f(x, cfg):
    return x.to(cfg.dtype)

"L['cfg'].dtype == torch.float32"   # EQUALS_MATCH guard
"___check_type_id(L['cfg'], <id_of_Config>)"  # TYPE_MATCH / ID_MATCH guard

5. Graph Breaks: Mechanics and Implications

5.1 Exhaustive Trigger Catalogue

A graph break occurs whenever InstructionTranslator encounters an operation that cannot be represented in the FX graph or whose outcome depends on runtime data that is not symbolically available. Dynamo categorizes break sources:

Data escape — tensor value enters Python: - tensor.item() — materializes a 0-d tensor to a Python scalar - bool(tensor) — calls __bool__, which calls .item() for 0-d tensors - int(tensor) — calls __index__ or __int__ - tensor.tolist() — materializes the full tensor as a Python list

Unsupported Python operations: - print(...), logging.* calls, file I/O — side effects with no FX equivalent - id(tensor), hash(tensor), type(tensor) used as a runtime value — not graph-representable - torch.autograd.grad(...) called directly in the forward pass (not via .backward()) - torch.nn.utils.rnn.pack_padded_sequence and ops with data-dependent output shapes

Data-dependent control flow: - if tensor_valued_condition: — when the condition is a TensorVariable (not a ConstantVariable or SymNodeVariable that can be statically resolved) - for i in range(tensor.item()): — loop bound depends on tensor data

Untraced callees: - Calling a C extension function that Dynamo does not have a VariableTracker for - Calling a Python function that, when InliningInstructionTranslator attempts to inline it, itself triggers a graph break at the top level

Unsupported Python builtins (in tensor-producing context): - exec, eval, globals(), locals() — dynamic name lookup - __import__ — dynamic import

5.2 What Dynamo Does on a Break

When InstructionTranslator hits a break trigger at instruction offset \(k\) in the original bytecode, it performs the following sequence:

flowchart TD
    A["1. Finalize OutputGraph
emit RETURN_VALUE for all live tensors
compile FX fragment [0, k) with backend
append CacheEntry(guard_fn, compiled_code_0)"]
    B["2. Generate __resume_at_k code object
copy original bytecode, reset PC to offset k
receive all live frame variables as arguments"]
    C["3. Emit wrapper bytecode for parent frame:
__compiled_fn_0(*live_tensors_0)
execute break-causing instruction in eager
__resume_at_k(*live_tensors_1)"]
    D["4. Resume function re-enters Dynamo
processes bytecode [k+1, end)
possibly emits another FX fragment [k+1, m)"]

    A --> B --> C --> D

Definition (Resume Stub). A __resume_at_<offset>_<index> code object is a copy of the original bytecode with co_firstlineno and jump targets adjusted so that execution begins at byte offset <offset>. Live frame variables are passed as arguments. The stub re-enters custom_eval_frame_shim on its first call, triggering Dynamo to compile the continuation.

Importantly, a function with \(N\) graph breaks becomes \(N+1\) compiled FX graph fragments, each with its own guard set. The wrapper function that glues them together is itself compiled bytecode — it is not re-interpreted by Dynamo on every call.

def f(x, y):
    a = x + y          # captured in graph fragment 0
    print("debug")     # graph break here
    return a * 2       # captured in graph fragment 1

def f(x, y):
    a, = __compiled_fn_0(x, y)   # fragment 0: x + y
    print("debug")                # eager
    return __resume_at_38_1(a)   # fragment 1: a * 2

5.3 fullgraph=True Mode

torch.compile(fullgraph=True) instructs Dynamo to treat any graph break as a fatal error:

@torch.compile(fullgraph=True)
def f(x):
    print(x)   # raises torch._dynamo.exc.Unsupported at trace time
    return x + 1

fullgraph=True is the preferred mode for production deployment: it makes graph break mistakes visible at development time rather than silently fragmenting the graph. It is also required for torch.export, which demands a single captured graph with no Python fallback.

@torch.compile
def f(x):
    a = x.relu()
    print(a.shape)       # break 1
    b = a * 2
    if a.item() > 0:     # break 2
        return b + 1
    return b - 1

6. The Compilation Cache in Depth

6.1 The Per-Code-Object Linked List

The Dynamo compilation cache is a linked list of CacheEntry pairs, stored as an attribute on each PyCodeObject. Because code objects are shared Python objects (the same f.__code__ is referenced everywhere f appears), the cache is implicitly shared across all call sites of the same function.

Definition (Cache Entry). A CacheEntry is a pair:

\[ \texttt{CacheEntry} = (\texttt{guard\_fn} : \texttt{dict} \to \texttt{bool},\; \texttt{code} : \texttt{PyCodeObject}) \]

where guard_fn encodes the specialization conditions under which code is valid, and code is the compiled code object that replaces the original function body.

Cache lookup is \(O(n)\) in the number of entries. For the default limit of \(n \le 8\) this is a small constant — the guard functions are compiled predicates and the list is cache-friendly in memory.

Definition (Cache Lookup Protocol). On each call to a compiled function, the hook iterates the linked list from head to tail. For entry \(i\): - If entry_i.guard_fn(locals, globals) returns True: swap entry_i.code as the code object and evaluate it via _PyEval_EvalFrameDefault. Stop. - If False: advance to entry_i.next. - If the list is exhausted: call _compile_frame, append a new entry, evaluate.

Surprisingly, putting the most recently compiled specialization at the head (rather than maintaining MRU order) is not the current policy — the list is append-only. This means that if call patterns cycle through many specializations, later entries incur higher lookup cost. The cache_size_limit prevents this from becoming unbounded.

6.2 Cache Size and Fallback Policy

Definition (Cache Size Limit). torch._dynamo.config.cache_size_limit (default: 8) is the maximum number of CacheEntry objects per code object. Once this limit is reached, Dynamo stops compiling new specializations for that function and falls back to eager execution for any guard-failing call.

Common causes of excessive recompilation: - Calling a function with tensors of varying shapes without enabling dynamic=True - Passing different nn.Module instances (each fails the ID_MATCH guard on the module) - Changing requires_grad between calls

6.3 The Persistent FX Graph Cache

The Dynamo per-code-object cache is in-process and non-persistent. A separate persistent cache tier stores Inductor compilation artifacts across process restarts:

TORCHINDUCTOR_CACHE_DIR defaults to /tmp/torchinductor_<username>/. The cache key for FXGraphCache is a hash over: the FX graph structure, operator signatures, PyTorch version, and relevant compiler config flags. This means that upgrading PyTorch invalidates all cached artifacts. Validation at load time also checks that the GPU device matches the cached target.

The persistent cache means that the second process run after a cold compilation is significantly faster — Inductor skips lowering and kernel compilation, retrieving pre-compiled .cubin objects directly. This is especially impactful for large transformer models where Inductor compilation dominates startup time.

6.4 Clearing State

torch._dynamo.reset()

torch._dynamo.reset() clears the Dynamo per-code-object linked-list caches (all CacheEntry pairs on all code objects registered with Dynamo) and resets internal counters. It does not clear the Inductor persistent cache (TORCHINDUCTOR_CACHE_DIR). To force a full recompilation including Inductor, delete the cache directory:

import shutil, os
shutil.rmtree(os.environ.get("TORCHINDUCTOR_CACHE_DIR",
              f"/tmp/torchinductor_{os.getenv('USER')}"))

7. Debugging and Observability

7.1 Backend Isolation Modes

torch.compile accepts a backend argument that controls how far down the compilation stack the function is sent. Two diagnostic backends are especially useful:

# Test Dynamo alone
f_dynamo = torch.compile(f, backend="eager")

# Test Dynamo + AOTAutograd
f_aot = torch.compile(f, backend="aot_eager")

Both "eager" and "aot_eager" produce correct outputs (they fall back to eager kernels), so correctness testing against the original function is valid. Performance will be slower than eager due to Dynamo overhead without the benefit of kernel fusion.

7.2 torch._dynamo.explain

torch._dynamo.explain(fn)(*args) performs a dry run trace of fn with the given arguments and returns an ExplainOutput object summarizing all graph captures and breaks:

import torch

@torch.compile
def f(x):
    a = x.relu()
    print(a.shape)   # break
    return a * 2

x = torch.randn(4)
explanation = torch._dynamo.explain(f)(x)

print(explanation.graphs)              # list of captured FX graphs
print(explanation.graph_count)         # number of graph fragments
print(explanation.graph_break_count)   # number of breaks
print(explanation.break_reasons)       # list of (reason_str, traceback) pairs

ExplainOutput.break_reasons gives human-readable strings like "'print' is a graph-breaking function" paired with a Python traceback pointing to the source location. This is the primary tool for iteratively eliminating graph breaks from a model.

import torch

def helper(t):
    return t.item()     # data escape

@torch.compile
def f(x):
    v = helper(x)       # break triggered inside helper
    return x + v

x = torch.tensor(1.0)
explanation = torch._dynamo.explain(f)(x)

# explanation.graph_count == 2
# explanation.break_reasons[0] might read:
# ("'item' is a graph-breaking function", <traceback to helper>)

7.3 Logging and Bytecode Inspection

Environment variables for verbose tracing:

Programmatic bytecode inspection:

# See the compiled/rewritten code in Python form
import torch._dynamo
torch._dynamo.config.output_code = True
# Then call the compiled function — it prints the generated code

# Or use depyf (third-party) for decompiled source:
# pip install depyf
import depyf
with depyf.prepare_debug(f, x):
    f(x)

Checking for graph breaks in CI:

# Strict mode: raises on first graph break
torch.compile(f, fullgraph=True)(x)

This pattern is recommended for production model CI: add a test that calls the model once under fullgraph=True on representative inputs, and any future regression that introduces a break will fail the test immediately.

exp = torch._dynamo.explain(model.forward)(sample_input)
print(exp.graph_count, exp.graph_break_count)

for reason, tb in exp.break_reasons:
    print(reason)
    print(tb)

torch.compile(model.forward, fullgraph=True)(sample_input)

References