Modeling of images is a complicated problem, a number of models [46] [80] [76] have been proposed, which work well with a select categories of images. However, there has not been a panoramic study in the literature on the structures of the whole ensemble of natural image patches. In this dissertation, we study the mathematical structures of the ensemble of natural image patches and map image patches into groups of subspaces that we call manifold. Manifolds are used to reduce the dimensionality or compress the volume of subspaces by only covering a group of wanted image patches using a set of perceptual metrics. Each manifold then can be modeled independently by choosing the best metric suited for its members. Based on the metrics, we believe there are two general types of manifolds, “explicit manifolds” and “implicit manifolds”. Explicit manifolds are build bottom-up, more complex metrics result in increase of volume, and one would find those simple and regular image primitives, such as edges, bars, corners and junctions. Implicit manifolds are build top down, more complex metrics result in volume reduction, one finds more complex image patches, such as textures and clutters. By using the ideas of the manifolds, we showed a unified framework for learning a probabilistic model on the space of image patches by pursuing both types of manifolds under a common information theoretical principle. The connection between the two types of manifolds is realized through image scaling which changes the entropy of the image patches. The explicit manifolds live in low entropy regimes while the implicit manifolds live in high entropy regimes. In experiments, we cluster the natural image patches and compare the two types of manifolds with a common information theoretical criterion. We also study the transition of the manifolds over scales and show that the complexity peak in a middle entropy regime where most objects and parts reside.