驗證梯度的正確性

題目（152）：如何驗證求目標函數梯度功能的正確性？

考點：微積分、Taylor expansion

近似（微積分）
根據partial derivative的定義，
$\frac{\partial L(\bm \theta)}{\partial \theta_i} = \frac{L(\theta_1, \cdots,\theta_i+h, \cdots,\theta_p) - L(\theta_1, \cdots,\theta_i-h, \cdots,\theta_p)}{2h}$

*E.g. $h=10^{-7}$

近似誤差（Taylor expansion with Lagrange remainder）
Univariate Taylor expansion on the function $\tilde{L}(x) = L(\bm \theta + x \bm e_i)$:

$L(\bm \theta+h\bm e_i) = L(\bm \theta) + (h-0)L(\bm \theta) + \frac{h^2}{2}L''(\bm \theta) + \frac{h^3}{6}L'''(\bm \theta + p \bm e_i)$

$L(\bm \theta-h\bm e_i) = L(\bm \theta) - (h-0)L(\bm \theta) + \frac{h^2}{2}L''(\bm \theta) - \frac{h^3}{6}L'''(\bm \theta + q \bm e_i)$

$\frac{L(\bm \theta+h\bm e_i) - L(\bm \theta-h\bm e_i)}{2h} = L(\bm \theta) + \frac{h^2}{12}[L'''(\bm \theta + p \bm e_i)-L'''(\bm \theta + q \bm e_i)] \hspace{3.8em}$

$|L(\bm \theta) - \frac{L(\bm \theta+h\bm e_i) - L(\bm \theta-h\bm e_i)}{2h}| = \frac{L'''(\bm \theta + p \bm e_i)-L'''(\bm \theta + q \bm e_i)}{12}h^2 = Mh^2,$
where $p,q \in (0,h)$. The last equation suggests that the approximation error is proportional to $h^2$.

Reasons and diagnosis when the error is larger than expected:

1. Large value of $M$: reduce $h$ by an order of $10^{-1}$ and check if the error is reduced by an order of $10^{-2}$.
2. Wrong calculation of gradient

Appendix
Lagrangian remainder:
$f(x) = f(x_0) + (x-x_0) f'(x_0) + \cdots + \frac{(x-x_0)^n}{n!}f^{(n)}(x_0) + R_n$

$R_n = \int_{x_0}^x f^{(n+1)}(t) \frac{(x-t)^n}{n!}dt$
Using the mean-value theorem,
$R_n = \frac{f^{(n+1)}(x^\ast)}{(n+1)!}(x-x_0)^{n+1},$
for some $x^\ast \in (x_0,x)$.

參考文獻：

1. 《百面機器學習》
2. Lagrange Remainder, http://mathworld.wolfram.com/LagrangeRemainder.html