An Enhanced Intelligent Server Monitoring System for Predictive Fault Detection and Triage Prioritization

Abstract

Modern digital infrastructure relies heavily on continuous server availability, yet traditional monitoring tools remain largely reactive, relying on static thresholds that generate excessive false alerts and often miss early signs of degradation. This research presents an enhanced intelligent server monitoring system that combines real-time stream processing with a hybrid unsupervised machine learning model for predictive fault detection and triage prioritization. The system ingests server metrics (CPU, memory, disk, and network utilization) through an Apache Kafka message bus and computes a weighted hybrid anomaly score. S(x_t) that combines an Isolation Forest outlier score with an Autoencoder reconstruction error, and classifies detected anomalies into priority levels using a Random Forest classifier. Prometheus and InfluxDB provide short-term and long-term time-series storage, respectively; Grafana delivers live visualization dashboards; Telegram delivers real-time alerts. An independent Apache Spark batch layer periodically generates aggregate historical reports. The models were trained on 2,243 real records from the Alibaba Cluster Trace 2018 production dataset, selected after evaluating and rejecting two alternative public datasets (the Server Machine Dataset, whose 38 metrics are anonymized, and the NAB AWS CloudWatch collection, which lacks a memory-utilization metric). The hybrid score's decision threshold (0.4963) was derived automatically from the 98th percentile of the training score distribution. System performance was validated across 28 automated tests spanning unit, integration, and deployment levels (100% pass rate), and the anomaly detector was evaluated against a test set with known injected anomalies, achieving a 92.5% detection rate (Recall) with a 3.39% false-positive rate. The results demonstrate that a hybrid unsupervised approach, trained on real production telemetry, can deliver practical, low-noise anomaly detection suitable for real-time server health monitoring.