30  Lab 14 — Residual-Block Sheaf

Anchor chapter: Chapter 14 — Frontiers: Deeper Architectures, Scaling, Open Questions.

Goal. Construct the cellular sheaf for a one-block residual network (path graph plus one chord), and see exactly which parts of the path-graph story survive the skip connection and which do not.

Build the sheaf of the single-residual-block network from §Worked example: widths \([2, 2, 1]\) with a skip edge \(W^{\text{skip}}\) from the input into the output pre-activation. Compute \(\delta_\Omega\) (tall \(7 \times 6\)), verify Prop. 14.8 numerically for random weights (full column rank — so the harmonic extension exists and is unique), and solve the pinned Dirichlet problem via least squares (since back-substitution fails). Then measure the gap between the harmonic extension and forward(x) of the equivalent reference module: the forward-pass identity of Prop. 3.4 is specific to path graphs, and here it fails. As a control, delete the chord and watch the identity come back. Bonus: stack two residual blocks and investigate whether the full-rank property persists.

TipRuns in your browser

This lab uses NumPy, Matplotlib, and NetworkX, plus a small torch-like shim (nn.Module, nn.Linear, torch.relu) defined in the Setup cell below to construct the reference residual-block forward pass. PyTorch itself has no Pyodide wheel, so the shim stands in for torch.nn — it reproduces the forward pass and default weight initialisation, not autograd — and the code below reads like the paper’s PyTorch reference implementation. Everything runs directly in the page via WebAssembly, no local install needed.

Prefer a local Jupyter environment with real PyTorch? Download lab-14-residual-block-sheaf.ipynb

Install dependencies: pip install torch numpy matplotlib networkx

30.1 Setup

30.2 1. Build the object

We use the architecture of the worked example in Chapter 14 — a two-layer \([2, 2, 1]\) MLP with a skip into the output pre-activation: \[z^{(1)} = W^{(1)} x + b^{(1)}, \quad a^{(1)} = \mathrm{ReLU}(z^{(1)}), \quad z^{(2)} = W^{(2)} a^{(1)} + b^{(2)} + W^{\text{skip}} x, \quad \hat{y} = z^{(2)},\] with identity output for simplicity. The base graph is the Ch. 7 path \(v_x \to v_{z^{(1)}} \to v_{a^{(1)}} \to v_{z^{(2)}} \to v_{\hat{y}}\) plus a chord edge \(e^{\text{skip}}: v_x \to v_{z^{(2)}}\) with tail restriction map \(W^{\text{skip}} \in \mathbb{R}^{1 \times 2}\) and head restriction map \(I_1\). Pinning \(v_x\) leaves free coordinates \((c_{z^{(1)}}, c_{a^{(1)}}, c_{z^{(2)}}, c_{\hat{y}})\) of total dimension \(2+2+1+1 = 6\), while the edge stalks total \(2+2+1+1+1 = 7\): the skip adds an edge without adding free variables, so \(\delta_\Omega \in \mathbb{R}^{7 \times 6}\) is tall, not square — the unitriangular structure and the \(\det = 1\) argument of Lemma 3.2 break down.

30.3 2. Verify a theorem / run an experiment

Since \(\delta_\Omega\) is tall, there is no triangular back-substitution; the harmonic extension is the least-squares solve \(\hat{c} = (\delta_\Omega^\top \delta_\Omega)^{-1} \delta_\Omega^\top \tilde{b}\). By Prop. 14.8 it exists and is unique — that is all the paper (§7 of [1]) claims beyond path graphs, and it is worth seeing why nothing stronger can be claimed. Two edges now impose conditions on \(c_{z^{(2)}}\): the path edge wants \(c_{z^{(2)}} = W^{(2)} c_{a^{(1)}} + b^{(2)}\), the chord wants \(c_{z^{(2)}} = W^{\text{skip}} x\). The network’s forward pass satisfies their sum, not each separately — so the forward pass is not a global section, the least-squares residual \(\|\delta_\Omega \hat{c} - \tilde{b}\|\) is strictly positive, and the harmonic extension does not reproduce forward(x). This is the Prop. 3.4 coincidence breaking before your eyes. As a control, we delete the chord: the plain \([2,2,1]\) path graph snaps back to residual \(\approx 0\) and harmonic extension \(=\) forward pass, exactly as in Ch. 8.

30.4 Exercises

  1. Two-block stack. Stack two residual blocks (say widths \([2,2,2]\) then \([2,2,1]\), one skip per block) and build the sheaf on the corresponding graph. Count the free dimensions and the edge dimensions — how tall is \(\delta_\Omega\) now? Is it still full column rank for generic weights?

  2. Identity skip. Many ResNets use an identity skip (which forces matching widths). Set widths \([2,2,2]\) and fix \(W^{\text{skip}} = I_2\). Does Prop. 14.8’s genericity argument still apply when the skip restriction map is not generic? Check \(\sigma_{\min}(\delta_\Omega)\) over random \(W^{(1)}, W^{(2)}\).

  3. Weight-scale dependence. Compute \(\lambda_{\min}(\delta_\Omega^\top \delta_\Omega)\) for 50 random weight draws at scales \(\|W\| \sim 0.1\), \(1\), and \(10\). Positive definiteness holds generically (Prop. 14.8) — but how does the conditioning of the Dirichlet problem depend on the weight scale?

  4. Open problem. §7 of the paper leaves the extension beyond path graphs open: which graph topologies and restriction-map structures guarantee full column rank, and does the sheaf heat equation still converge? Simulate the state-dependent heat equation \(\dot{c} = -L_{\text{free}}(\sigma(c))\, c\) for the residual-block sheaf starting from a random cochain (input pinned, everything else free). Does the Dirichlet energy decrease monotonically? Document any counterexamples or numerical failures.