Composite hypothesis testing for hidden markov models via joint GLRT-Em estimation Bridging pointwise and uniform optimality with robust finite‑sample performance

Abstract

This paper presents a comprehensive joint estimation detection framework for hidden Markov chains (HMCs) with unknown statistical parameters, a critical challenge in physical-layer security and adversarial communication environments. By integrating the Generalized Likelihood Ratio Test (GLRT) with the Expectation Maximization (EM) algorithm, we develop a composite hypothesis test that achieves high detection probability without prior parameter knowledge. The proposed framework addresses four fundamental gaps in the existing literature: (i) the unrealistic assumption of perfect parameter knowledge in adversarial settings; (ii) the neglect of uniform optimality and its implications for worst-case parameter attacks; (iii) the lack of finite-sample analysis with confidence intervals and sensitivity studies; and (iv) insufficient baseline comparisons against minimax and energy-based detectors. Our key contributions include: (a) rigorous derivation of the EM GLRT integration with complete algorithmic description; (b) theoretical proof of the distinction between pointwise and uniform error exponents, demonstrating that the latter is strictly lower — a vulnerability that adversaries can exploit; (c) extensive finite‑sample Monte Carlo validation on a Gilbert Elliott channel, including detailed comparisons with clairvoyant, minimax, and energy‑based detectors; (d) comprehensive discussion of estimation bias, detection performance, ROC curves, convergence behavior, and sensitivity analysis; (e) practical guidelines for threshold calibration and system design. Results show that the proposed detector attains PD>0.96 for n=500, closely approaching the clairvoyant upper bound, while maintaining computational feasibility with O(nM2) complexity. The framework is robust to initialization and converges within 15–25 EM iterations, making it suitable for real‑time applications. The uniform error exponent gap is quantified and its security implications are thoroughly discussed.