Image understanding depends critically on learning visual models which can be formulated in terms of probabilistic structured representations. But the two-dimensional nature of images makes it much hard to design efficient image understanding systems and the form of the representations is also unclear. The key assumption used to address these problems in this thesis is that the visual world is recursively compositional. This enables us to construct representations, by recursively combining elementary stochastic templates, and makes inference and unsupervised learning practical. By introducing elementary visual constituents of recursion properties, we design recursively compositional systems for two basic image understanding problems, deformable object parsing and image parsing. We first propose a recursive deformable template model to represent objects in a hierarchical form. The object template consists of a small number of small sub-templates which is composed by smaller subsub-templates, and so on. The composition of templates and their deformation are imposed at different levels to capture both short-range and long-range correlations. The recursive design enables us to perform rapid parsing by dynamic programming, efficient unsupervised learning by recursive composition and supervised training by structured -perceptron. The learnt model together with the inference algorithms are capable of performing different vision tasks simultaneously, such as object detection, segmentation and parsing (e.g. matching/alignment of object parts). Next we extend the recursive deformable template model to a novel AND/OR graph representation for parsing articulated objects into parts and recovering their poses. The recursion design in the AND/OR graph allows us to handle an enormous variety of articulated poses with a compact graphical model where the rapid inference can be performed. We present a novel structure-learning method, Max Margin AND/OR Graph (MM-AOG), to learn the parameters of the AND/OR graph model discriminatively. Finally, we present a recursive segmentation and recognition template model for 2D image parsing. This representation consists of recursive segmentation-recognition templates which account for image segmentation and object recognition simultaneously at multiple layers. The recursive templates result in a coarse-to-fine representation which is capable of capturing long-range dependency and exploiting different levels of contextual information. The recursive structure also allows us to design a rapid inference algorithm, based on dynamic programming, which enables us to parse the image rapidly in polynomial time, and learn the model efficiently in a discriminative manner. We demonstrate the significant benefits of the recursive design on several vision tasks including deformable articulated object detection, parsing and segmentation, and image segmentation and scene understanding. Our experiments on the challenging public datasets show that the recursively compositional systems achieve the state-of the-art performance.