An Approach to Evaluating Clustering Quality for Network Anomaly Detection

Abstract

Clustering is a fundamental task in unsupervised learning and is important for extracting interesting patterns and structures within data. Evaluating the quality of clustering algorithms is a complex task, often requiring a balance between homogeneity and completeness. In this paper, we apply V-measure as an evaluation metric that effectively determines the clustering quality and balances these two aspects by computing their harmonic mean to find the best features.  We explore the theoretical foundations of V-measure, its calculation, and strategies for optimizing clustering performance to reduce data dimensionality, maintain low false alerts, and high detection rate for anomaly detection in binary intrusion classification. Our findings highlight significant reductions in dimensionality and data volume, coupled with low false positive and false negative rates, thereby enhancing detection accuracy.