23  Lab 07 — Build the Neural Sheaf

Anchor chapter: Chapter 7 — Building a Sheaf from a ReLU Network.

Goal. Given an nn.Sequential-style ReLU network, programmatically construct the corresponding cellular sheaf on the path graph.

Construct the neural sheaf for an arbitrary \([n_0, n_1, \ldots, n_{k+1}]\) ReLU MLP as a data structure: vertex list, edge list, edge-type labels, restriction-map callables. Populate it from a small ReLU MLP, evaluate the coboundary \(\delta(\sigma)\) on a concrete input, and verify the dimensional coincidence \(\dim C^1 = \dim C^0 - n_0\) (Rem. 7.3). Includes a visualisation of the network’s path-graph base, annotated with per-vertex stalk dimensions and per-edge discord.

TipRuns in your browser

This lab uses NumPy, Matplotlib, and NetworkX, plus a small torch-like shim (nn.Linear, nn.ReLU, nn.Sequential) defined in the Setup cell below. PyTorch itself has no Pyodide wheel, so the shim stands in for torch.nn — it reproduces the forward pass and default weight initialisation, not autograd — and the code below reads like the paper’s PyTorch reference implementation. Everything runs directly in the page via WebAssembly, no local install needed.

Prefer a local Jupyter environment with real PyTorch? Download lab-07-build-neural-sheaf.ipynb

Install dependencies: pip install torch numpy matplotlib networkx

23.1 Setup

23.2 1. Build the object

We extract weight matrices and biases from an nn.Sequential (built from the torch-like shim above) and build a NeuralSheaf object that stores vertex stalks \(\mathcal{F}(\ell) = \mathbb{R}^{n_\ell}\) and the activation-pattern-dependent coboundary \(\delta_\sigma\). For a \([n_0, n_1, \ldots, n_{k+1}]\) network, edge \(e_\ell = (\ell{-}1, \ell)\) has stalk \(\mathbb{R}^{n_\ell}\) with restriction maps \(\mathcal{F}_{\ell \trianglelefteq e_\ell} = I_{n_\ell}\) (identity) and \(\mathcal{F}_{(\ell-1) \trianglelefteq e_\ell}(c) = W^{(\ell)} \Sigma^{(\ell-1)} c\) (ReLU-gated weight). The coboundary is lower-triangular with identity blocks on the diagonal, confirming Lemma 3.2: \(\det \delta_\Omega = 1\).

23.3 2. Verify a theorem / run an experiment

We verify Remark 7.3 (\(\dim C^1 = \dim C^0 - n_0\)), Lemma 3.2 (\(\det \delta_\Omega = 1\)), and Proposition 3.4 (forward pass = harmonic extension) on a concrete input. The harmonic extension solves \(\delta_\Omega c_\Omega = \tilde{b}\) by triangular back-substitution; the result should match net(x) to floating-point precision. The path-graph visualisation annotates each vertex with its stalk dimension and each edge with its type.

23.4 Exercises

  1. Deeper network. Build a NeuralSheaf for a [2, 8, 8, 1] network. Verify that \(\det \delta_\Omega = 1\) and that Proposition 3.4 holds for 50 random inputs. How does the condition number of \(\delta_\Omega\) change with depth?

  2. Singular activation patterns. When a neuron is exactly at the switching boundary (\(z_j = 0\)), the activation pattern is ambiguous. Create a synthetic input that puts one neuron on the boundary and compare the two choices of \(\Sigma\) (0 and 1 for that neuron). Do both give valid harmonic extensions?

  3. Dimensional coincidence. Remark 7.3 states \(\dim C^1 = \dim C^0 - n_0\), which is what makes \(\delta_\Omega\) square. For a ResNet-style skip connection from layer 0 to layer 2, this dimensional coincidence fails. Construct such a network, show that \(\delta_\Omega\) is no longer square, and discuss what happens to the harmonic-extension interpretation.

  4. Sheaf from biases. In the harmonic extension formulation the bias \(b^{(\ell)}\) appears as a boundary-forcing term, not in \(\delta_\Omega\) itself. Reformulate the sheaf by introducing a “bias vertex” with stalk \(\mathbb{R}^1\) pinned to 1 and edges carrying \(b^{(\ell)}\) as restriction maps. Does this change \(\det \delta_\Omega\)?