The objective of perceptual organization (grouping, segmentation and recognition) is to parse generic natural images into their constituent components which are respectively instances of a wide variety of visual patterns. These visual patterns are fundamentally stochastic processes governed by probabilistic models which ought to be learned from the statistics of natural images. In this paper, we review research streams from several disciplines, and divide existing models into four categories according to their semantic structures: descriptive models, causal Markov models, generative models, discriminative models. The objectives, principles, theories, and typical models are reviewed in each category. The central theme of this epistemological paper is to study the relationships between the four types of models and to pursue a unified mathematical framework for the conceptualization (or definition) and modeling of various visual patterns. In representation, we point out that the effective integration of descriptive and generative models is the future direction for statistical modeling. to make visual models tractable computationally, we study the causal Markov models as approximations adn we observe that the discriminative models are computational heuristics for inferring generative models. Under this unified mathematical framework statistical models for various visual patterns should form a “”continuous”” spectrum, in the sense that they belong to a serial of probability families in the space of attributed graphs. Visual patterns and their parts are conceptualized as statistical ensembles governed by their models. These statistical models and concepts amount to a visual language with a hierarchy of vocabularies, which is essential for building effective, robust, and generic vision systems.