24  Lab 08 — Verify Proposition 3.4

Anchor chapter: Chapter 8 — The Forward Pass as Harmonic Extension.

Goal. Show numerically that the forward pass of a small ReLU network equals the harmonic extension on the neural sheaf built in Lab 07.

Build the restricted coboundary \(\delta_\Omega(\sigma)\) for a small pretrained MLP (e.g., a [2, 8, 1] regressor) as a dense matrix. Verify Lem. 8.2 numerically (\(\det \delta_\Omega = 1\) up to floating-point noise), solve the pinned Dirichlet problem by one triangular back-substitution, and compare the solution component-by-component against forward(x) from the network’s reference forward pass. Repeat across 100 random inputs to confirm Prop. 8.6 holds independently of which activation pattern is realised.

TipRuns in your browser

This lab uses NumPy, Matplotlib, and SciPy, plus a small torch-like shim (nn.Linear, nn.ReLU, nn.Sequential) defined in the Setup cell below. PyTorch itself has no Pyodide wheel, so the shim stands in for torch.nn — it reproduces the forward pass and default weight initialisation, not autograd — and the code below reads like the paper’s PyTorch reference implementation. Everything runs directly in the page via WebAssembly, no local install needed.

Prefer a local Jupyter environment with real PyTorch? Download lab-08-verify-prop-3-4.ipynb

Install dependencies: pip install torch numpy scipy matplotlib

24.1 Setup

24.2 1. Build the object

We initialise a [2, 8, 1] ReLU MLP, extract its weight matrices as NumPy arrays, and for any input \(x\) build the restricted coboundary \(\delta_\Omega(\sigma)\) as a dense \((n_1 + n_2) \times (n_1 + n_2)\) lower-triangular matrix. The key identity is \((δ_\Omega)_{\ell\ell} = I_{n_\ell}\): each diagonal block is an identity matrix, making \(\delta_\Omega\) unitriangular and therefore invertible with \(\det = 1\) regardless of the activation pattern \(\sigma\).

24.3 2. Verify a theorem / run an experiment

We solve the harmonic extension by triangular back-substitution (one forward solve, since \(\delta_\Omega\) is lower-triangular). The resulting cochain components are the hidden-layer pre-activations \(z^{(1)}\) and the output pre-activation \(z^{(2)}\); they should match the reference net(x) and internal hook values to within floating-point precision. We repeat across 100 random inputs and plot the maximum absolute error per input to confirm Proposition 3.4 holds for all activation patterns.

24.4 Exercises

  1. Deeper network. Extend the construction to a [2, 8, 4, 1] network (two hidden layers). Build the \(13 \times 13\) lower-triangular \(\delta_\Omega\), verify \(\det = 1\), solve by triangular substitution, and confirm the match to the reference forward pass across 50 random inputs.

  2. Condition number. Compute \(\kappa(\delta_\Omega) = \|\delta_\Omega\| \cdot \|\delta_\Omega^{-1}\|\) as a function of weight magnitude \(\alpha\) (scale all weights by \(\alpha \in \{0.1, 0.5, 1, 2, 5\}\)). Plot \(\kappa\) vs \(\alpha\) and explain the dependence.

  3. Verification via Dirichlet energy. The harmonic extension minimises \(\|\delta_\sigma c\|^2\) subject to \(c^{(0)} = x\). Verify this numerically by computing \(\|\delta_\sigma c\|^2\) for the harmonic extension and for 20 random perturbations around it, confirming the harmonic extension achieves the minimum.

  4. Bias absorption. The bias vector \(b^{(\ell)}\) appears in \(\tilde{b}\) (the RHS) rather than in \(\delta_\Omega\) itself. Reformulate the problem by absorbing the bias into an augmented cochain \(\tilde{c}^{(\ell)} = (c^{(\ell)}, 1)\) and modified weight matrices \(\tilde{W}^{(\ell)} = [W^{(\ell)}, b^{(\ell)}]\). What does \(\delta_\Omega\) look like in this augmented formulation?