Quantifying the Complexity of Multilevel Confirmatory Factor Analysis Models

Yun Kyung Kim
M.S., 2026
CAI, LI
In multilevel modeling, as in any statistical modeling, model selection requires balancing goodness-of-fit (GOF) against model complexity, yet the complexity of multilevel models and how it operates differently across levels have received little systematic attention. This study quantified and decomposed the fitting propensity (FP) of multilevel confirmatory factor analysis (ML-CFA) models, a class of models widely used to validate multilevel constructs. The 12 ML-CFA models under scrutiny had comparable numbers of freely estimated parameters but differed in specification, including the number of latent factors, relationships among the latent factors, or the presence of cross-level measurement invariance constraints. The FP of each model was decomposed into level-specific components that represent the capacity of the model to fit any data at the within- and between-group levels, respectively.Two new mechanisms for generating multilevel data were developed: the covariance decomposition (CD) and marginal composition (MC) approaches, where the former was a constrained instance of the latter. Using the more flexible MC approach, multilevel data were generated across 18 conditions defined by the number of groups, expected intraclass correlation coefficient (ICC) level, and per-group sample size. In each data condition, level-specific FPs of ML-CFA models were examined by two complementary approaches: comparing the distributions of level-specific GOF indices and visualizing the data space occupied by each model. Three findings were central. First, cross-level measurement invariance constraints disproportionately reduced between-group level FP while leaving within-group level FP largely unaffected, which is a substantial asymmetry that parameter counts alone do not anticipate. Second, parameter nesting did not guarantee data-space nesting. For instance, configural models that are parametrically nested within their unconstrained counterparts were capable of representing some data better than their unconstrained counterparts. Third, models with identical parameter counts differed substantially in their level-specific FPs, and their relative ordering could vary depending on data conditions. These results contribute to the literature on multilevel measurement invariance, the relationship between parameter nesting and covariance matrix nesting, and the customization of GOF index cutoff thresholds to candidate models as well as data conditions.
2026