Analyzing Semantic Understanding in Large Language Models through Embedding and Clustering Techniques
Yutong Chen
MASDS, 2025
WU, YINGNIAN
We study whether large language model (LLM) embeddings exhibit human-interpretable cluster structure. Using the Yelp Open Dataset for topic and sentiment labels, we extract sentence embeddings (primarily all-MiniLM-L6-v2) and evaluate unsupervised clustering via intrinsic indices (Silhouette, DBI) and extrinsic alignment (NMI, ARI, Purity). On a stratified sample of 60000 reviews across ten coarse topics (Restaurants, Food, Nightlife, Hotels, Shopping, Beauty, Automotive, Home, Health), fixed-k clustering with cosine geometry yields NMI=0.239, ARI=0.108, Purity=0.580 for topics, and NMI=0.060, ARI=0.040, Purity=0.740 for sentiment (positive=41755, negative=11362, neutral=6883). Encoder comparison indicates small but consistent gains from MPNet on NMI with a lower DBI, while MiniLM attains slightly higher Silhouette. Sensitivity analyses show modest degradation under class imbalance and little change with light Gaussian noise. We conclude that modern sentence embeddings support meaningful clustering at coarse topical granularity while sentiment separation remains limited without supervision.
2025

