Blueprint: Norm Discretization

7.1 The proof of the Kiefer–Wolfowitz theorem

The measure of Theorem 7.1 is one whose Gram matrix has an almost maximal determinant. Mixing in a further point cannot increase that determinant by much, and the matrix determinant lemma turns this into the bound \(a(y)^* G^{-1} a(y) \leq m + \varepsilon \) of Proposition 7.5, from which the theorem follows. No topology and no measurable structure on \(\Omega \) are used until the very end: a finitely supported measure is a finite list of points with weights.

A design consists of finitely many points \(x_1, \dots , x_N \in \Omega \) and weights \(w_1, \dots , w_N \geq 0\) with \(\sum _k w_k = 1\), and its Gram matrix is \(G = \sum _k w_k \, a(x_k) a(x_k)^*\), the Gram matrix of \(a\) in \(L_2\) of the measure \(\sum _k w_k \delta _{x_k}\); in Lean it is Discretization.KieferWolfowitz.designGram. Its quadratic form is the weighted sum of squares,

\[ c^* G c = \sum _k w_k \, |\langle c, a(x_k) \rangle |^2 \qquad \text{for every } c \in \mathbb {C}^{\iota }, \]

since the quadratic form of a rank-one matrix \(u u^*\) at \(c\) is \(|\langle c, u \rangle |^2\) (Discretization.KieferWolfowitz.re_quadForm_designGram). The statements below spell out points, weights and Gram matrix.

Proposition 7.5 Almost maximal determinant
✓

Let \(\iota \) be a finite nonempty index set with \(m = \# \iota \) elements, let \(a = (a_k)_{k \in \iota }\) be bounded functions on a set \(\Omega \) whose coordinate functions are linearly independent, and let \(\varepsilon {\gt} 0\). Then there are points \(x_1, \dots , x_N \in \Omega \) and weights \(w_1, \dots , w_N \geq 0\) with \(\sum _k w_k = 1\) such that \(G = \sum _k w_k \, a(x_k) a(x_k)^*\) is positive definite and

\[ a(y)^* G^{-1} a(y) \; \leq \; m + \varepsilon \qquad \text{for all } y \in \Omega . \]
Proof ▼

First, some design has a positive definite Gram matrix (Discretization.KieferWolfowitz.exists_design_posDef). Linear independence of the coordinate functions says that \(c = 0\) is the only coefficient vector with \(\langle c, a(y) \rangle = 0\) for all \(y \in \Omega \). It therefore suffices to find points annihilating no nonzero coefficient vector, for then the uniform distribution on them has \(c^* G c {\gt} 0\) for every \(c \neq 0\) by the formula for \(c^* G c\). Descending induction on dimension supplies them: while some nonzero vector survives the points chosen so far, the hypothesis gives a point at which it does not vanish, and adding that point lowers the dimension.

The determinants of the Gram matrices of designs form a set of real numbers which is bounded above, since the entries of every Gram matrix are bounded by \(C^2\) and the Leibniz formula bounds a determinant by its entries, and which contains a positive number by the first step. Let \(S\) be its supremum and take a design whose determinant exceeds \(S / (1 + \eta )\), where \(\eta = \varepsilon ^2 / (4m(m + \varepsilon - 1))\); its Gram matrix \(G\) is positive definite because its determinant is positive.

Appending a point \(y\) with weight \(\alpha \in [0,1)\) and scaling the old weights by \(1 - \alpha \) is again a design, with Gram matrix \((1-\alpha ) G + \alpha \, a(y) a(y)^*\). By Lemma 2.9 its determinant is \(\det G\) multiplied by \((1-\alpha )^{m-1}(1 + \alpha (t-1))\) with \(t = a(y)^* G^{-1} a(y)\), and that multiple cannot exceed \(S / \det G {\lt} 1 + \eta \). If \(t\) were larger than \(m + \varepsilon \), the weight of Lemma 7.6 would make it at least \(1 + \eta \).

The proof above rests on the following estimate for the gain from mixing in one point.

Lemma 7.6 Gain from mixing in one point
✓

Let \(m \geq 1\) and \(t {\gt} m\) be real. Then \(\alpha = (t-m) / (2m(t-1))\) lies in \([0,1)\) and

\[ 1 + \frac{(t-m)^2}{4m(t-1)} \; \leq \; (1-\alpha )^{m-1} \bigl( 1 + \alpha (t-1) \bigr). \]
Proof ▼

By Bernoulli’s inequality \((1-\alpha )^{m-1} \geq 1 - (m-1)\alpha \), and the difference of the two sides of the claim is then the explicit nonnegative quantity \((t-m)^2 / (4m^2(t-1))\). The weight \(\alpha \) is the one for which the linear gain \(\alpha (t-m)\) is twice the quadratic loss \((m-1)(t-1)\alpha ^2\), so half of the gain survives.

The number of points can be cut down without changing the Gram matrix.

Lemma 7.7 Few points suffice
✓

Let \(\Omega \) be a set, let \(\iota \) be a finite set with \(m = \# \iota \) elements and let \(a : \Omega \to \mathbb {C}^{\iota }\). For all points \(x_1, \dots , x_N \in \Omega \) and weights \(w_1, \dots , w_N \geq 0\) with \(\sum _k w_k = 1\) there are at most \(m^2 + 1\) points \(y_j \in \Omega \) and weights \(v_j \geq 0\) with \(\sum _j v_j = 1\) and \(\sum _j v_j \, a(y_j) a(y_j)^* = \sum _k w_k \, a(x_k) a(x_k)^*\).

Proof ▼

The Gram matrix lies in the convex hull of the matrices \(a(y) a(y)^*\), so by Carathéodory’s theorem it is a convex combination of affinely independent ones. These matrices are Hermitian, and the Hermitian matrices in \(\mathbb {C}^{\iota \times \iota }\) form a real vector space of dimension \(m^2\), since every complex matrix is uniquely \(H + iK\) with \(H\) and \(K\) Hermitian. An affinely independent family in it has at most \(m^2 + 1\) members.

Proof of Theorem 7.1 ▼

Take the design of Proposition 7.5 for \(\varepsilon /2\). For \(s = m + \varepsilon \) the bound \(s^{-1} a(y)^* G^{-1} a(y) {\lt} 1\) is strict, so subtracting \(s^{-1} a(y) a(y)^*\) from \(G\) leaves a positive definite matrix by Lemma 2.8(b), that is \(a(y) a(y)^* \preceq s \cdot G\). Evaluating this at the coefficient vector \(c\) of \(f\), with \(c^* G c = \sum _k w_k |f(x_k)|^2\), gives \(|f(y)|^2 \leq s \sum _k w_k |f(x_k)|^2\). This inequality depends on the design only through its Gram matrix, which Lemma 7.7 preserves while cutting the design down to \(m^2 + 1\) points. Read as the measure \(\varrho = \sum _k w_k \delta _{x_k}\), the design gives the theorem: integration against \(\varrho \) is the weighted sum over the points, for every measurable integrand.

For \(\varepsilon = 0\) on a compact space, the matrices \(a(y) a(y)^*\) form a compact set, whose convex hull is compact as well and consists exactly of the Gram matrices of designs. The determinant is continuous, so it attains its maximum at a design. No mixture can then increase the determinant at all, and Lemma 7.6 applied with the factor \(1\) gives \(a(y)^* G^{-1} a(y) \leq m\) without slack. The passage to the norm inequality is the one above for every constant \(s {\gt} m\), followed by the limit \(s \to m\), which is a limit of real numbers, and the point count is again Lemma 7.7.

In Lean the design form is Discretization.KieferWolfowitz.exists_optimal_design_card_le and its compact counterpart, and Discretization.KieferWolfowitz.exists_measure_of_design turns a design into the measure.

Remark 7.8 What the count is good for
✓

The sharper count \(r + 1\), with \(r\) the dimension of the span of the products \(a_k \overline{a_l}\), is not formalised, and nothing needs it.

Remark 7.9 The identity decomposed over the design
✓

For a design with points \(x_1, \dots , x_n\), weights \(w_1, \dots , w_n\) and invertible Gram matrix \(G\), and with \(m = \# \iota \),

\[ \sum _{k=1}^n w_k \, a(x_k)^* G^{-1} a(x_k) = m \qquad \text{and} \qquad \sum _{k=1}^n w_k \bigl( a(x_k)^* G^{-1} z \bigr) a(x_k) = z \]

for every \(z \in \mathbb {C}^{\iota }\), the first being \(\operatorname {Tr}(G^{-1} G)\) and the second \(G G^{-1} z = z\) written out (Discretization.KieferWolfowitz.sum_weight_quadForm_inv_eq_card, Discretization.KieferWolfowitz.sum_weight_smul_quadForm_inv_eq_self). With \(c_k = w_k \, a(x_k)^* G^{-1} a(x_k)\) this is John’s decomposition of the identity over the design points, in the inner product \(\langle u, z \rangle = u^* G^{-1} z\) of the ellipsoid of \(G\), and it is the condition dual to the bound of Proposition 7.5. Under Theorem 7.1 with \(\varepsilon = 0\) the average forces \(a(x_k)^* G^{-1} a(x_k) = m\) at every design point, which makes the points contact points. For an arbitrary convex body the theorem is proved in the companion project Lean-SNumbers (PDF).