Resilient data-driven non-intrusive load monitoring for efficient energy management using machine learning techniques

Abstract

The integration of smart homes into smart grids presents numerous challenges, particularly in managing energy consumption efficiently. Non-intrusive load management (NILM) has emerged as a viable solution for optimizing energy usage. However, as smart grids incorporate more distributed energy resources, the complexity of demand-side management and energy optimization escalates. Various techniques have been proposed to address these challenges, but the evolving grid necessitates intelligent optimization strategies. This article explores the potential of data-driven NILM (DNILM) by leveraging multiple machine learning algorithms and neural network architectures for appliance state monitoring and predicting future energy consumption. It underscores the significance of intelligent optimization techniques in enhancing prediction accuracy. The article compares several data-driven mechanisms, including decision trees, sequence-to-point models, denoising autoencoders, recurrent neural networks, long short-term memory, and gated recurrent unit models. Furthermore, the article categorizes different forms of NILM and discusses the impact of calibration and load division. A detailed comparative analysis is conducted using evaluation metrics such as root-mean-square error, mean absolute error, and accuracy for each method. The proposed DNILM approach is implemented using Python 3.10.5 on the REDD dataset, demonstrating its effectiveness in addressing the complexities of energy optimization in smart grid environments.

1 Introduction

Smart grids are essential for modernizing and optimizing the energy distribution system. They utilize advanced technologies to intelligently manage the generation, distribution, and consumption of electricity [1]. Even though smart grid has numerous benefits, they also have certain drawbacks like the lack of granular visibility into energy consumption at the appliance level, limited ability to detect [2], diagnose energy faults or abnormalities in real-time and facing challenges in implementing demand response programs effectively. In order to overcome these issues, non-intrusive load monitoring is necessary, which helps by disaggregating the overall energy consumption data that provides detailed insights into individual appliance usage and also enabling the timely detection of abnormalities by accurately identifying appliance-level energy demands [3, 4]. NILM also enhances the functionality and efficiency of smart grids. It provides granular visibility into appliance-level energy consumption, enables real-time fault detection and diagnosis [5], and facilitates precise demand response strategies. Overall, NILM complements smart grid technology and helps overcome several limitations such as lack of granular visibility, inefficient resource allocation, difficulty in identifying energy wastage, inability to adapt to dynamic energy demands and promoting a more effective and sustainable energy management system [6].

Fig. 1

Smart grid outline with DNILM

1.1 Problem statement

NILM brings challenges in energy management, appliance identification, demand response programs, and fault detection. Accurate predictions are necessary to ensure efficient energy usage, promote sustainability, optimize demand response strategies, and proactively identify and address faults in the energy system. By analyzing historical energy consumption data and utilizing predictive modeling techniques, NILM can forecast future energy consumption patterns of appliances. This information enables utilities and grid operators to anticipate peak demand periods and allocate energy resources accordingly, ensuring a reliable and efficient energy distribution system. Consumers can also gain valuable insights into the efficiency of their devices and make informed choices about energy-efficient alternatives. This empowers consumers to optimize their energy usage, reduce waste, and contribute to overall sustainability goals. Anomalies or deviations can be identified in comparing predicted energy consumption with actual energy consumption, indicating potential equipment malfunctions or energy losses. This enables timely maintenance and repair interventions, preventing energy wastage and enhancing grid reliability. Prediction analysis, therefore, plays a crucial role in enhancing the capabilities and effectiveness of NILM in achieving efficient energy usage within the smart grid.

The present article emphasizes the importance of prediction analysis in NILM. By employing prediction techniques, it becomes possible to gain insights into the energy consumption patterns of different appliances. This information is critical for effective energy management and optimization within a smart grid.

1.2 Literature survey

Non-intrusive load monitoring (NILM) is a technology used to analyze and identify individual appliances and their energy consumption in a building without the need for intrusive hardware installations. It has gained significant interest in both academic and industrial sectors due to its potential for unlocking smart home services and opportunities. NILM can be implemented using deep learning techniques, such as convolutional neural networks and k-nearest neighbors classifiers, to process measured power transient responses and detect appliances in real time [7, 8]. While intrusive load monitoring (ILM) requires attaching low-end meter devices to appliances, NILM only requires a single point of sensing, making it a more cost-effective and flexible solution. Future developments in load monitoring are expected to combine the benefits of NILM with individual power measurement by smart plugs and appliances, creating hybrid solutions [9]. The utility of NILM extends to various applications, including energy scorekeeping, condition monitoring, and activity tracking in smart grid systems [10]. Real-time load disaggregation, which involves identifying simultaneous switching operations, remains a challenging problem in NILM. However, recent research proposes practical solutions using adaptive-window based detection and deep dictionary learning models with sparse coding algorithms [11]. Additionally, the use of convolutional neural networks with differential input has shown improved performance in appliance-level load monitoring services [12].

The evolution of prediction techniques used in non-intrusive load monitoring (NILM) has been driven by advancements in machine learning and data analysis [13,14,15]. Various machine learning algorithms such as artificial neural network (ANN), support vector machines (SVM), k-Nearest Neighbor (k-NN), hidden Markov model (HMM), decision tree (DT), random forest (RF), and deep learning (DL) have been utilized for load disaggregation methods based on pattern recognition in NILM. By incorporating appliance state transitions, HMMs can accurately infer the appliance responsible for specific energy consumption patterns. This approach has demonstrated promising results in appliance-level energy disaggregation [16] and has become a foundational technique in NILM research. Similarly, [17] focuses on exploiting the sparsity property of hidden Markov models (HMMs) to perform online real-time non-intrusive load monitoring (NILM). By leveraging the inherent sparsity structure in NILM problems, the proposed method achieves efficient and accurate energy disaggregation. The use of SVM with linear and radial basis function (RBF) kernels has been extensively examined in NILM research to analyze energy consumption. These studies incorporate features like active-reactive power and power factor [18]. Furthermore, to mitigate the impact of noise, the wavelet shrinkage method was employed. Another recent study utilized the GA method to optimize the parameters of the RBF kernel [19]. In a research investigation where data samples were collected from the overall load signal using a sliding window approach, a study utilized the multi-label k-NN method for device identification [20]. Furthermore, recent advancements have shown improved classification performance by combining k-NN and template matching techniques [21]. In another study, an enhanced algorithm called improved k-nearest neighbors (IKNN) was introduced to reduce computation time for learning and enhance classification performance [22].

Decision tree (DT) is not commonly preferred for load disaggregation due to its lower success rate compared to other pattern recognition methods. In a comparative study of three pattern recognition methods, k-nearest neighbors (k-NN) is emerged as the most successful method, while DT fell short of achieving the desired level of success [23]. However, there have been instances where DT yielded more successful results in low sampling frequency systems, outperforming the hidden Markov model (HMM) method in a study that prioritized low sampling [24]. Another study employed genetic algorithm (GA) for feature extraction and feature selection, where DT and multilayer perceptron (MLP) methods exhibited a high degree of accuracy at 99.5% [25]. The random forest (RF) method, which utilizes multiple decision trees for learning, has also been explored in some studies [26]. In a comparative study of DT, RF, k-NN, and support vector machine (SVM) methods with admittance-based feature extraction, the SVM method yielded the most successful results, with DT and RF falling short compared to the other methods [26].

However, there is a lack of comprehensive studies that compare these methods in detail regarding performance. As a result, it remains uncertain which of these methods is more effective in NILM. Furthermore, the performance of these methods can vary depending on the types of devices used and the features extracted from them. Overall, the evolution of these techniques in NILM has allowed for a more detailed estimation of appliance-level energy consumption. These advancements improve energy management, demand response strategies, fault detection, and energy conservation practices within smart grid systems (Table 1).

Table 1 Literature survey of NILM

1.3 Key contributions

• 1. Classification of appliance state using K means clustering.

• 2. Achieving load disaggregation using mapping mechanism.

• 3. Removing the impact of voltage fluctuations and measurement errors using calibration technique.

• 4. Design of combinatorial optimization to reduce the misaccounted power and facilitate the proper state assignment to mains.

• 5. Implementation of gated recurrent unit (GRU) to the appliance dataset for power consumption forecasting.

1.4 Organization

This paper addresses the challenges of load degradation in non-intrusive load monitoring (NILM) when prediction analysis is not performed, as discussed in Sect. 2. In Sect. 1.2, the literature survey explores various NILM algorithms incorporating prediction analysis. Section 3 presents the implementation of the proposed NIALM approach, illustrated through a flowchart. The results and analysis of the proposed approach are presented in Sect. 4. To conclude, Sect. 5 summarizes the current state of research, emphasizing the focus on enhancing NILM accuracy in recent studies.

2 Methodology

NILM (non-intrusive load monitoring) is an integral part of the smart grid infrastructure. By using advanced algorithms, NILM allows for the disaggregation and analysis of energy consumption at the individual appliance level without additional metering. This enables real-time monitoring of appliance-level energy usage, aiding in demand response programs, load balancing efforts, fault detection, and accurate billing. NILM enhances energy management within the smart grid ecosystem by providing detailed insights into individual appliance energy consumption. Figure 1 denotes working outline of the smart grid employing DNILM. It can be observed that there are two paths, one denoting the power flow represented in bold and other is information flow represented in dotted line. In this study, power flow assumes as unidirectional flow from utility to the consumer, whereas information transfer takes place in utility side as well as consumer side. Utility-side information exchange can be observed in communication loop. Major bridge between the utility and the consumer for the information flow is smart meter. Central server also called as SCADA receives the information from the utility as well as consumer. Server sends the request to the smart meter through a gateway for the consumer-side information regarding power consumption and scheduled loads. Once the server gets the information from all the meters, an appropriate control is applied to maintain the stability of the grid. In other way, utility sends the information regarding the tariffs and the incentives power consumption patterns to the smart meter, which enables the consumer to implement the smart scheduling process to achieve cost optimization. This architecture becomes further complex with the involvement of distributed energy resources into the grid.

The residential energy datasets (REEDs) [33] offer a widely used dataset called the REDD dataset. This dataset consists of detailed energy consumption measurements from multiple households. One of the residences included in the REDD dataset for NILM research is House 2, situated in Austin, Texas, USA. From April to May 2011, high-frequency electricity measurements were collected in this single-family residence with a sampling rate of 1 Hz. The dataset provides voltage and current waveforms recorded at the household’s mains, offering valuable insights into the energy consumption patterns.

2.1 Load division through mapping

The mapping process in non-intrusive load monitoring (NILM) involves estimating the energy consumption of individual appliances based on the aggregate energy signal. The aggregate energy signal represents the total power consumption (P) of all appliances in the building at any given time. It is denoted as A(t), where t is the time index and i represents the specific appliance or load as given in (1). NILM models often consider the aggregate energy signal as a linear combination of the individual appliance energy consumptions, with an added noise term N(t) representing the noise component, as shown in (2).

$$\begin{aligned}{} & {} P_{t} = [P_1,P_2, \ldots ,P_T] \end{aligned}$$

(1)

$$\begin{aligned}{} & {} A(t) = P_{t}+N(t) \end{aligned}$$

(2)

Relevant features, such as power, voltage, or current, are extracted from the aggregate energy signal at various time intervals. These features are denoted as $F_j{(t)}$, where j represents a specific feature. NILM algorithms use machine learning techniques to classify and associate the extracted features with individual appliances. This classification process involves learning the model’s parameters to differentiate between appliances based on their feature patterns. Once the appliances are classified, the mapping equation shown in (3) is used to estimate the energy consumption of each appliance at a given time t.

$$\begin{aligned} P_{t}=f(F_j(t)) \end{aligned}$$

(3)

Here, f represents the mapping function that maps the extracted features to the estimated energy consumption of the specific appliance, $P_{t}$. Each day is converted into sequences with time $t \in [1,T]$. If there is a load division to two mains mains1 ($L_{1})$) and mains2 ($L_{1})$), then the total power can be represented as (4) and the power measured for each mains individually is represented as (5) and (6). Equation (7) represents the set of powers measured by individual appliances in the mains at T different time instants.

$$\begin{aligned} P_{t}= & {} \alpha ^{L_1}+\alpha ^{L_2} \end{aligned}$$

(4)

$$\begin{aligned} \alpha ^{L_1}= & {} \sum _{i=1}^{b} \alpha ^{i} \end{aligned}$$

(5)

$$\begin{aligned} \alpha ^{L_2}= & {} \sum _{i=1}^{l} \alpha ^{i}, t \in [1,T] \end{aligned}$$

(6)

$$\begin{aligned} \alpha ^{i}= & {} [\alpha _{1}^{i},\alpha _{2}^{i}, \ldots ,\alpha _{1}^{T}] \end{aligned}$$

(7)

The total number of appliances (g) is the sum of appliances in mains1 (b) and mains2 (l), respectively, where b and l are the mutually exclusive sets of appliances that are subsets of g.

To make extracting patterns from the primary signal easier, the appliances are sorted in decreasing order of their peak power. Starting with the appliance having the highest peak load, the power of each primary signal is always compared.

If the $\alpha ^{L_1}> \alpha ^{i}$ at any time, then the appliance is assigned to that main1, similarly to mains 2 if $\alpha ^{L_2}> \alpha ^{i}$ it is assigned to mains2. If this approach does not result in the assignment of any $\alpha ^{i}$ all to $E_i{(t)}$, we examine the times when events (B(t)) occur in the appliance’s power series and always the condition $B_{\alpha ^{i}}(t)\subseteq B_{E}(t)$ needs to be satisfied to which appliance that is assigned to mains. An appropriate threshold ensures that minor voltage fluctuations are not considered events. Once an appliance has been assigned to a main using either of these filters, its power sequence is subtracted from the corresponding main signal. This simplifies the main assignment process for the remaining appliances.

2.2 Clustering

K-means clustering can be used as a mathematical technique for appliance classification and mapping based on the extracted features from the aggregate energy signal. After extracting the features through mapping denoted as $F_j{(t)}$, these extracted features are organized into a feature matrix, denoted as X, where each row represents a different time interval, and each column represents a specific feature. The K-means algorithm starts by randomly initializing K cluster centroids. Each centroid is represented as a vector, denoted as ${\hat{C}}(k)$, where k represents the cluster index. For each feature vector in X, the algorithm calculates the Euclidean distance between the feature vector and each cluster centroid. The feature vector is then assigned to the cluster with the closest centroid. This can be expressed mathematically in (8). For each feature vector ${\hat{x}}(i)$ in X: Assign ${\hat{x}}(i)$ to the cluster with the closest centroid.

$$\begin{aligned} {\hat{k}}(i) = \hbox {argmin}_k ||{\hat{x}}(i) - {\hat{c}}(k)||^2 \end{aligned}$$

(8)

After assigning all the feature vectors to their respective clusters, the algorithm updates the cluster centroids based on the mean of the feature vectors assigned to each cluster. This can be expressed as shown in (9). For each cluster k: Update the centroid ${\hat{C}}(k)$ to the mean of the assigned feature vectors:

$$\begin{aligned} {\hat{C}}(k) = 1/|{\hat{S}}(k)| * \sum {\hat{x}}(i) \end{aligned}$$

(9)

for ${\hat{x}}(i)$ in the cluster ${\hat{S}}(k)$

Here, $|{\hat{S}}(k)|$ represents the number of feature vectors assigned to cluster k, and $\sum {\hat{x}}(i)$ represents the summation of assigned feature vectors. The above process related to equations (3) and (4) is repeated iteratively until convergence. Convergence occurs when the cluster assignments and centroids no longer change significantly. Once K-means clustering converges, the clustering results can be used to map each feature vector to its corresponding appliance. Each cluster represents a distinct appliance or load, and the feature vectors assigned to each cluster are associated with that appliance. The cluster index can be used as the label or identifier for each appliance.

2.3 CO algorithm

Hart introduced the combinatorial optimization (CO) algorithm as a fundamental approach in the field of non-intrusive load monitoring (NILM). This algorithm assumes that each appliance can have multiple states, and the last value is considered as a relatively small value. Each state corresponds to a specific energy consumption value associated with the appliance. The algorithm’s objective is to assign the correct operating states to each appliance, aiming to minimize the difference between the measured aggregate power and the sum of the energy consumption of all appliances. An appliance can only be in a single state at any given time, which is represented as (10)

$$\begin{aligned} \sum _{s=1}^{K} G_{t,s}^{i}=1 \end{aligned}$$

(10)

At sth state at time t, the power consumption of ith appliance is given as (11):

$$\begin{aligned} {\hat{\gamma }}_{t,s}^i=\sum _{s=1}^{K} G_{t,s}^{i} * \theta _{s}^{i} \end{aligned}$$

(11)

Therefore, for all appliances, the total overall power consumption at time t is given as (12):

$$\begin{aligned} {\hat{P}}_t=\sum _{i=1}^{I} \sum _{s=1}^{K} G_{t,s}^{i} * \theta _{s}^{i} \end{aligned}$$

(12)

After load assignment, the error in power signal is given as (13):

$$\begin{aligned} {\hat{p}}_t=|P_t-\left( \sum _{i=1}^{I} \sum _{s=1}^{K} G_{t,s}^{i} * \theta _{s}^{i} \right) | \end{aligned}$$

(13)

Therefore, the error value ${\hat{p}}_t$ depends on the classified state of the appliance and the predicted power consumption of the appliance at that particular instant. The optimization should be performed to select the appropriate state and predict the power consumption that is free from voltage fluctuations and unaccounted power. Equation (14) gives the cost function for error minimization.

$$\begin{aligned} \phi _{t}=\hbox {argmin}_{G_t}|P_t- {\hat{P}}_t| \end{aligned}$$

(14)

() can be achieved by implementing the prediction methodology to forecast the $\theta _{s}^{i}$. The predicted output for the ith appliance for T time sequences with optimization in error is given as (15)

$$\begin{aligned} O^{i}=\left\{ \Theta _{\phi _{1}^{i}}^{i} \ldots \Theta _{\phi _{T}^{i}}^{i} \right\} \end{aligned}$$

(15)

Fig. 2

Workflow representation of the proposed DNILM method

2.4 Algorithm

The algorithm follows a series of steps. It consists of two phases: the training phase and the test phase. Initially, in the training phase, the algorithm receives power measurements from each individual appliance. Using the k-means clustering algorithm, the load profile is transformed into a sequence of finite states, resulting in an appliance model. Once all appliances are modeled, the Combinatorial Optimization (CO) algorithm constructs a lookup table containing all possible combinations of operating states. In the test phase, the CO algorithm takes the aggregate power as input and identifies the most probable combination of appliance states based on the lookup table.

The widely used NILMTK toolkit, which is developed in a Python environment specifically for energy disaggregation problems, was utilized to implement the Combinatorial Optimization algorithm.

As shown in Fig. 2, the proposed DNILM algorithm is implemented in four different scenarios: without calibration and load division (WoC & WoLD), without load division and with calibration (WC & WoLD), with load division and without calibration (WoC & WLD), with load division and with calibration (WC & WLD). These four modes of operations are classified based on calibration and load division. In WoC & WoLD, WC & WoLD, initial step is to classify the state of appliance operation depending on the power consumption, which is performed by the clustering technique further followed by calculating cluster centers. In WoC & WoLD process, CO algorithm is immediately employed after center calculation to find the best possible operating state of appliance, whereas in WC & WoLD, an calibration step is performed after center calculation to increase the efficiency. Calibration process is further discussed in this section. It can be observed from Fig. 2 that WoC & WLD is similar to WoC & WoLD and WC & WLD is similar to WC & WoLD except a extra load division process is carried out before dividing into the clusters. In this load division, the appliance is initially divided between the mains 1 and mains 2 depending on the load demand. This load division divides the loads among themselves; this pre-classification increases the clustering efficiency further as it reduces the range of cluster. The calibration of the measured power sequence plays a vital role in understanding the actual power consumption by the appliance. Often, the measured power is not equal to the appliance power consumption due to various scenarios such as voltage fluctuations, metering errors, inconsistency in meter measurements, electromagnetic interference and many other external factors. To compensate for this effect, a regularization term $K_{c}$ is multiplied by the measured power as shown in (16). During normal conditions, the power change in mains equals the power consumption change in the appliance at a particular time instant, but if there are any external factors involved, this ratio will change. The meter’s increased measured power is compensated by the inverse ratio of the regularization term, thus maintaining the proper power measurement.

$$\begin{aligned} \theta _{s_{c}}^{i} = \theta _{s}^{i}\frac{\Delta \alpha ^{L_{z}}}{\Delta \alpha ^{i}} \end{aligned}$$

(16)

In the case of without calibration and without load division, the error after load assignment ${\hat{p}}_{t1}$ and the estimated power consumption is given in (17). In the case of with calibration and without load division, the error term${\hat{p}}_{t2}$ is changed as shown (18), and consequently, error minimization equation is modified.

$$\begin{aligned} {\hat{p}}_{t1}= & {} \left| P_t-\left( \sum _{i=1}^{I} \sum _{s=1}^{K} G_{t,s}^{i} * \theta _{s}^{i}\right) \right| \end{aligned}$$

(17)

$$\begin{aligned} {\hat{p}}_{t2}= & {} \left| P_t-\left( \sum _{i=1}^{I} \sum _{s=1}^{K} G_{t,s}^{i} * \theta _{s_{c}}^{i}\right) \right| \end{aligned}$$

(18)

In the case of without calibration and with load division, the error ${\hat{p}}_{t3}$ is calculated as given by (19).

$$\begin{aligned} {\hat{p}}_{t3}=\left| \alpha ^{L_{z}}-\left( \sum _{i=1}^{I} \sum _{s=1}^{K} G_{t,s}^{i} * \theta _{s}^{i}\right) \right| \end{aligned}$$

(19)

In case of with calibration and with load division, the error ${\hat{p}}_{t4}$ is calculated as given by (20).

$$\begin{aligned} {\hat{p}}_{t4}=\left| \alpha ^{L_{z}}-\left( \sum _{i=1}^{I} \sum _{s=1}^{K} G_{t,s}^{i} * \theta _{s_{c}}^{i}\right) \right| \end{aligned}$$

(20)

Therefore depending on the case selected, measured power of the appliance depending on the state of operation (${\hat{P}}_{t}$) and depending on the error value, the appropriate state assignment is performed such that the next state prediction is accurate with less error.

3 Prediction

After performing the combinatorial optimization (CO) algorithm in non-intrusive load monitoring (NILM), the next step is often to employ a prediction model such as gated recurrent unit (GRU) to forecast the future energy consumption of individual appliances. A set of energy consumption of different appliances $a^{i}_{t}$, price $R_{t}$ are given as input. The input data, typically aggregate power measurements, are preprocessed to create sequences (often referred to as windows or time series) of past power values for each appliance. Later relevant features such as power values are extracted from these sequences. Now, loading the appliances data and splitting the data into $X_{\mathrm{train}}$, $Y_{\mathrm{train}}$, $X_{\mathrm{test}}$, $Y_{\mathrm{test}}$.later, initialize the weights ($w_{1},w_{2}, \ldots w_{n}$) and biases for various components of the GRU model, such as the update gate, reset gate, and candidate activation. Compute the reset gate activation: Multiply the input ($i_{1},i_{2}, \ldots i_{n}$) at the current time step t with the corresponding weight matrix $M_{r}$, add the previous hidden state $M_{n-1}$ multiplied by another weight matrix M, and apply an activation function to get the reset gate activation.

$$\begin{aligned} R_{t}&=\hbox {sigmoid}(M_{r}*(M_{n-1},M)\\ U_{t}&=\hbox {sigmoid}(M_{r}*(M_{n-1},M)\\ Z_{t}&=\hbox {tanh}((M_{r}*(R_{t}*M_{n-1},M)) \end{aligned}$$

Multiply the update gate activation with the previous hidden state $h_{n-1}$, element-wise. Multiply ($1 - U_{t}$ with the candidate activation, element-wise. Sum these two results to obtain the new hidden state as shown in (21).

$$\begin{aligned} H=(1-U_{t})*M_{n-1}+U_{t}*Z_{t} \end{aligned}$$

(21)

Iterate over each time step t in the input sequence and perform the following operations:

a. Compute the outputs ($o_{1},o_{2}, \ldots o_{n}$) of the GRU cell at that time step t, using the input ($i_{1},i_{2}, \ldots i_{n}$) at the current time step t and the previous hidden state $h_{n-1}$.

b. Save the current hidden state h for the next time step $t_{1}$.

After processing all time steps and layers, the final hidden state will be left, which represents the encoded information of the input sequence and that final layer is used to make predictions.

Once the GRU model is trained, it is tested using the test data $X_{\mathrm{test}}$, $Y_{\mathrm{test}}$. Given a sequence of past power values, the GRU model processes the sequence and generates a predicted energy consumption value for the next time step. By calculating root-mean-square error (RMSE) and mean absolute error (MAE), you can assess the accuracy and performance of your GRU model in predicting the output values on the test set.

Fig. 3

a Mains1 power consumption and b unaccounted power of Mains1

Fig. 4

a Mains2 power consumption and b unaccounted power of Mains2

Fig. 5

a Percentage power distribution in Mains1 and b percentage power distribution in Mains2

4 Results

The proposed DNILM algorithm is trained with a REDD dataset with multiple scenarios. Most importantly, the load division among the multiple mains at the customer and the calibration of the power consumption are considered. As discussed in the previous sections, these two scenarios impact the power consumption and effective utilization of the resources. The seven appliances considered in this study are subjected to DNILM under four different combinations: DNILM with calibration and without load division, DNILM without calibration and with load division, DNILM without calibration and load division, and DNILM with calibration and load division.

Fig. 6

Operating states of different appliances

Fig. 7

Operating state clustering of a dishwasher, b fridge, c kitchen1, d kitchen2, e lighting, f microwave and g stove

4.1 Load division using mapping

By using the mapping technique, load division is performed in which mains 1 is assigned with four appliances, namely dishwasher, kitchen, kitchen 2, and stove. Mains 2 is assigned with light, microwave, and refrigerator. This division through a simple mapping technique involves a lot of challenges, such as event conditions and unaccounted power. Figure 3a represents the mains 1 power with all the appliances as Fig. 3b indicates the power measured without any appliance; this power is identified as the unaccounted power. The percentage of unaccounted power calculated for mains 1 can be observed in Fig. 5a. Similarly, the mains 2 power and its unaccounted power are represented in Figs. 4a, b and 5b. These statistics emphasize the importance of calibration.

Fig. 8

Operating states classification of a dishwasher, b refrigerator, c kitchen and d kitchen2

4.2 Appliance state classification

Each appliance at the customer side operates in different modes or states, leading to variations in their power consumption. Depending upon these states of operations, the power consumption can be predicted, and the load division can be performed to attain energy optimization. Figure 6 details the statewide operation of the multiple appliances considered in this study. It is observed that some appliances has three distinct states, and some have two distinct states. The appliance selection is performed cautiously to obtain the diversity in operating states. This mismatch in the number of states requires a more complex and robust classification algorithm.

Initially, k means clustering is performed on the raw data to identify the number of clusters based on the power consumption of each appliance. Later on, these clusters are considered as the states of operation, and each sample is labeled accordingly. Figure 7 represents the individual appliance states.

Fig. 9

Operating states classification of a light, b microwave and c stove

4.3 Calibration and load division with CO scenarios

A high amount of unaccounted power and error in load assignment is observed without calibration and CO for load division. With the states obtained for each appliance using k means clustering, different cases are considered. In each case, the ability of the accuracy of the classification is analyzed. Figures 8 and 9 represent the state classification of each appliance in different scenarios in the form of confusion matrix.

It is observed that for every appliance, the classification accuracy is high with calibration and with load division case, whereas without calibration, load division suffers the most with less accuracy. Therefore, this states that with calibration and load division, there can be clear distinction, and unaccounted power can be limited. Figures 8 and 9 denote the possible combination of appliances on mains 1 and mains 2, respectively, based on their working time and the power consumption in the predefined time span.

4.4 Prediction analysis with GRU

With the accurate identification of states, with performing calibration and load division using CO, the prediction of power consumption for the next time instant is obtained using data-driven artificial intelligence methods. Multiple prediction algorithms are analyzed such as DT, sequence-to-point (S2P), long short-term memory (LSTM), recurrent neural networks (RNNs), denoising autoencoders (DAEs), and GRU. The analysis is performed with different evaluation metrics such as RMSE, MAE and accuracy to finalize the better prediction algorithm. Results for prediction analysis for each appliance are tabulated in Tables 2, 3, 4, 5, 6, 7 and 8. From the prediction analysis, it is clear that GRU, the advanced, recurrent neural network architecture, has a clear advantage compared to its counterparts. This algorithm demonstrates superior adaptability and efficiency in managing increased data volume and additional monitored devices without compromising performance, whereas in terms of runtime, the GRU algorithm exhibits accelerated processing capabilities, enabling the analysis of energy consumption data. This is complemented by its optimized memory usage, ensuring efficient handling of computational requirements. Figures 10 and 11 represent the prediction graphs of the microwave and kitchen; here, it can be observed that the proposed DNILM eliminates the voltage fluctuations and unaccounted power from the actual data and predicts only the appliance consumption.

Fig. 10

Power consumption prediction using GRU for kitchen1

Fig. 11

Power consumption prediction using GRU for microwave

Table 2 Performance analysis of ML and DL models for dishwasher appliance

Table 3 Performance analysis of ML and DL models for fridge appliance

Table 4 Performance analysis of ML and DL models for kitchen1 appliance

Table 5 Performance analysis of ML and DL models for kitchen2 appliance

Table 6 Performance analysis of ML and DL models for light appliance

Table 7 Performance analysis of ML and DL models for microwave appliance

Table 8 Performance analysis of ML and DL models for stove appliance

5 Conclusion

This paper emphasizes the significance of prediction analysis in non-intrusive load monitoring (NILM) within smart grids. The study demonstrates that prediction analysis enhances the effectiveness of NILM by enabling real-time estimation of energy consumption patterns at the appliance level. By employing advanced algorithms and leveraging historical energy consumption data, the paper showcases the utility of decision trees, sequence-to-point, denoising autoencoders, recurrent neural networks, long short-term memory, and gated recurrent unit models in accurately disaggregating energy usage. The comparison of performance metrics validates the effectiveness of prediction models in enhancing NILM. After conducting extensive simulations, it can be concluded that the gated recurrent unit (GRU) method is the most appropriate approach for predicting the power consumption profile of an individual household. These findings have important implications for energy management and offer valuable insights into how different appliances contribute to overall energy usage within smart grids. By harnessing the power of prediction analysis, energy grid operators and consumers can make more informed decisions, implement targeted conservation measures, and optimize energy allocation, ultimately leading to more efficient and sustainable energy management in smart grid systems.

5.1 Practical implications

Non-intrusive load monitoring (NILM) revolves around the need for clear guidelines on data privacy, security, and standardization. As NILM technology continues to advance, policymakers should consider establishing regulations that protect consumer data while also allowing for the collection of necessary information for load monitoring. By implementing this technology, homeowners can benefit from accurate and real-time insights into their energy usage patterns. This can lead to the development of personalized energy-saving strategies, thereby fostering efficient energy consumption and cost savings. Additionally, utility companies can leverage the data obtained through non-intrusive load monitoring to optimize grid operations, manage peak demands, and enhance overall energy distribution and conservation. Moreover, research and development in this area can pave the way for the creation of user-friendly interfaces and apps that empower individuals to actively engage in sustainable energy practices. Overall, the integration of non-intrusive load monitoring in home energy management systems using machine learning techniques holds the potential to revolutionize the way consumers understand, consume, and manage energy in everyday lives.