Approach:

The pipeline follows a modular and systematic methodology, designed to optimize performance and scalability in real-time anomaly detection. Key aspects of the approach include:

• Data Ingestion & Preprocessing: The pipeline utilizes PySpark and Apache Spark for distributed data ingestion and transformation, enabling efficient handling of large-scale datasets stored on distributed systems like Hadoop (HDFS).
• Feature Engineering: The system aggregates monthly energy consumption features and extracts statistical summaries for each Point of Delivery (PODID), ensuring that only complete and meaningful data is included in the anomaly detection process.
• Hyperparameter Optimization: To enhance model accuracy and performance, GridSearch was employed to identify the optimal hyperparameters for each unsupervised model applied. This systematic approach involved fine-tuning parameters such as the number of estimators and contamination levels for Isolation Forest, and the neighborhood size for Local Outlier Factor (LOF).
• Anomaly Detection Models: Unsupervised techniques such as PCA, Isolation Forest, and LOF were employed. Each model brought a unique perspective to anomaly detection, and their results were combined via an ensemble approach to enhance robustness and reliability.
• Production Design: The pipeline was developed with scalability in mind, leveraging PySpark for distributed processing and ensuring integration with production environments. Modular design principles facilitated easy maintenance, while the use of Apache Spark ensured real-time processing capabilities.