Horseshoes (quadratic curves) routinely show up in multidimensional scaling and correspondence analysis solutions. We review some of the empirical situations in which they are known to occur, and we discuss some of the mathematical models that produce them. One particular model, discussed by Diaconis et al. in a recent paper, is the Kac-Murdock-Szegö matrix A with elements aij = ρ | i-j |. In this paper we analyze this example in some detail. We point out that A is both totally positive and Toeplitz, and that the horseshoes also occur for other matrices with these properties. It is shown that double centering of a Toeplitz matrix leads to a centro-symmetric matrix, which again will produce horseshoes.