10 The Forward Pass as a Harmonic Extension
Purpose. Proves the paper’s foundational identification: the forward pass equals the unique harmonic extension of the input data on the neural sheaf.
10.1 Key concepts & results
- Lemma 3.2 (unitriangular factorization): in coordinates adapted to an activation pattern, the restricted coboundary δ_free is unitriangular; det δ_free = 1 and L_free positive definite for every activation pattern.
- Proposition 3.4 (forward pass = harmonic extension): H^0(G, U; F) = 0 and the minimizer of ½‖δc‖² matches forward(x).
- Why the unitriangularity is ‘exactly the topological order of the computation graph.’
Prerequisites: Ch 5, Ch 7
10.2 Motivating example
Take the [2, 4, 1] sheaf from Ch. 7 with the input vertex pinned to a concrete value \(x_{\text{in}} = (1, -1)\). Write down the coboundary \(\delta\) in coordinates adapted to the current activation pattern — the \(0/1\) pattern of signs of \(z^{(1)}\) at \(x_{\text{in}}\). The resulting matrix is block lower-triangular with identity diagonal blocks. Solve \(\delta c = 0\) for the unknown interior cochain by back-substitution, reading values off one vertex at a time. The sequence of values you recover is, layer by layer, exactly what NumPy returns when you run forward(x_in): the pre-activation \(z^{(1)}\), the post-activation \(a^{(1)} = R_{z^{(1)}} z^{(1)}\), the pre-output \(z^{(2)} = W^{(2)} a^{(1)} + b^{(2)}\), and the prediction \(\hat{y} = z^{(2)}\). No iteration, no training loop — a single linear solve whose structure is the forward pass.
This is the content of Proposition 3.4 of [1]: once the sheaf is built, the forward pass is not a separate algorithm; it is the unique harmonic extension of the input data, made computable by the triangular structure of \(\delta\).
10.3 Intuition
A feedforward network is usually described by the order in which it runs: input first, then layer 1, then layer 2, and so on. The sheaf picture erases that order. It replaces the sequence of substitutions with a single global equation — “find the cochain whose edge discrepancies all vanish, subject to the input boundary data” — and then observes that the computation-graph topology makes this equation trivially solvable by back-substitution.
Concretely: label the vertices of the path graph in topological order — input, \(z^{(1)}\), \(a^{(1)}\), \(z^{(2)}\), output. Fix any activation pattern \(\sigma\) and write \(\delta\) in these coordinates. The key observation (Lemma 3.2) is that each edge’s restriction map on its downstream endpoint is the identity (affine edges read \(z^{(\ell)}\) directly; ReLU edges read \(a^{(\ell)}\) directly; the output edge reads \(\hat{y}\) directly). So the diagonal blocks of the restricted coboundary \(\delta_\Omega\) — the coboundary with input pinned and only the free (interior) vertices varying — are all identity matrices. The lower-triangular blocks come from the upstream restriction maps: \(W^{(\ell)}\), \(R_{z^{(\ell)}}\), or the output Jacobian. A block lower-triangular matrix with identity diagonal is unitriangular; its determinant is \(1\), so \(\delta_\Omega\) is invertible, and there is a unique cochain killing the discrepancies. Reading it off row by row is layer-by-layer evaluation.
Two reframes make the identification land. First, harmonic extension is the statement of the forward pass; sequential evaluation is the substitution order the unitriangular structure makes available. They are not two computations that happen to agree; they are the same computation named twice. Second, the determinant-\(1\) identity is not cosmetic — it is the unique-solvability certificate that upgrades “local agreement” to “global forward pass” without any \(\delta^T \delta\)-invertibility hypothesis. Ch. 14 will flag that this identity fails the moment the computation graph stops being a path (residual connections, attention), which is where most open problems live.
Intuition device (planned): Block-triangular animation showing δ assembled layer-by-layer, with diagonal blocks lighting up as identities.
10.4 Formal development
[TO FILL: formal development — definitions, statements, careful notation]
10.5 Theorem demonstrations
[TO FILL: proofs / proof sketches of the key results named above. Proofs should come *after* the intuition section, as agreed.]
10.6 Worked examples
[TO FILL: worked example(s) carried out by hand]
10.7 Coding lab
lab-08-verify-prop-3-4 —
[TO FILL: one-paragraph description of the lab's goal]
10.8 Exercises
[TO FILL: 3–6 exercises, graded from warm-up to project-level]
10.9 Further reading
[TO FILL: annotated paragraph of 3–6 references]
10.10 FAQ / common misconceptions
[TO FILL: short Q&A for things readers frequently get wrong]