In this paper we compute the matrix derivative of the fixed-rank least squares approximation. We describe the eigen structure of the derivative and apply the result to derive an expression for the speed of convergence of iterative nonlinear principal component algorithms.