19  Lab 03 — Switched Gradient Flow

Anchor chapter: Chapter 3 — Nonsmooth Dynamics: Filippov, Clarke, CPWA Flows.

Goal. Simulate the piecewise-quadratic gradient flow \(\dot{x} = -\nabla f(x)\) for a CPWA potential, and verify that trajectories satisfy the Filippov conditions at switching surfaces.

Simulate a CPWA gradient flow on a simple 2D state space with two activation regions separated by the line \(z_1 = 0\). Initialize from several starting points and observe: crossing trajectories, sliding-mode trajectories, and convergence to the fixed point. Plot the Lyapunov function \(V(t)\) on a log scale to verify monotone decrease. Then embed a single-hidden-layer neural sheaf and run the restricted heat equation, watching ReLU boundary crossings and energy decrease. This lab builds the intuition for the convergence proof in Ch. 9.

TipRuns in your browser

This lab requires only NumPy, Matplotlib, and SciPy, loaded automatically via Pyodide. Code cells run directly in the page via WebAssembly — no local Python installation needed.

Prefer a local Jupyter environment? Download lab-03-switched-gradient-flow.ipynb

19.1 Setup

19.2 1. Build the object

We build a [2,4]-layer network (four hidden neurons, 2D input) whose hidden pre-activations \(z = Wx + b\) define four switching hyperplanes that partition \(\mathbb{R}^2\) into polyhedral regions. The CPWA Lyapunov potential is \(V(x) = \tfrac{1}{2}\|\text{ReLU}(Wx+b)\|^2 = \tfrac{1}{2}\sum_j \max(0, z_j)^2\), which is non-negative and piecewise-quadratic. Its gradient \(\nabla V(x) = W^\top \operatorname{diag}(\sigma(x))(Wx+b)\) is computable from the activation pattern \(\sigma(x) = \mathbf{1}[z > 0]\); at a switching surface we use the one-sided limit from the current region, which is the Filippov convention for discontinuous right-hand sides.

19.3 2. Verify a theorem / run an experiment

We integrate \(\dot{x} = -\nabla V(x)\) from eight evenly-spaced starting points on a circle of radius 1.8 using forward Euler. The left panel shows phase-plane trajectories overlaid on level sets of \(V\) and the four switching hyperplanes (dashed); the right panel plots \(V(x(t))\) on a log scale, confirming monotone decrease. Since \(V\) is a valid Lyapunov function for the flow — \(\dot{V} = -\|\nabla V\|^2 \leq 0\) — the energy must be non-increasing along any trajectory, and the final print statement checks this numerically.

19.4 Exercises

  1. Fixed points. The gradient flow \(\dot{x} = -\nabla V(x)\) converges to a critical point of \(V\). For the CPWA potential used in this lab, find all fixed points numerically (where \(\nabla V(x) = 0\)) and classify them by their activation pattern.

  2. Sliding modes. A trajectory “slides” along a switching surface \(\{x : (Wx+b)_j = 0\}\) when the flow directions on both sides point inward. Detect such surfaces by checking whether the \(j\)-th neuron’s gradient component changes sign across the boundary. Plot at least one sliding trajectory.

  3. Step-size sensitivity. Run the same trajectory with \(dt \in \{0.001, 0.01, 0.05, 0.1\}\). At what step size does the monotone-decrease property break down? Compare to the theoretical stability limit \(dt < 2/\lambda_{\max}(\nabla^2 V|_{\text{region}})\).

  4. Connection to Ch. 9. The sheaf heat equation \(\dot{x} = -\mathcal{L}_{\mathcal{F}}(\sigma(x)) x\) is structurally similar to this CPWA flow, but the “Laplacian” \(\mathcal{L}_{\mathcal{F}}(\sigma)\) depends on the cochain \(x\) through the activation pattern \(\sigma(x)\). Sketch how a switching surface in the present lab corresponds to a region boundary in the activation-region diagram of Lab 01.