Document Clustering and Visualization for Large Text Collections

Abstract

The exponential growth of digital textual data across scientific, journalistic, and social me￾dia domains has created a critical need for automated document organisation and exploration tools. This study presents a comprehensive end-to-end pipeline for document clustering and visualisation applied to large text collections. Three complementary approaches are system￾atically evaluated: K-Means clustering with TF-IDF representations, Latent Dirichlet Alloca￾tion (LDA), and BERTopic, a state-of-the-art neural framework combining transformer-based embeddings with UMAP dimensionality reduction and HDBSCAN density-based clustering. Experiments are conducted on three benchmark corpora 20 Newsgroups, Reuters-21578, and a curated arXiv dataset representing diverse levels of semantic complexity and class distribu￾tion. Performance is assessed using six complementary metrics covering internal cluster qual￾ity, external label agreement, topic coherence and diversity, and computational efficiency. Results demonstrate that BERTopic achieves superior semantic coherence and topic diver￾sity, while K-Means delivers the highest structural accuracy and computational efficiency when paired with dense sentence embeddings. LDA, although interpretable, consistently un￾derperforms in capturing deep semantic structure. These findings provide practical guidelines for selecting document clustering methods in real-world information retrieval applications.