Preprocessing for Data-Driven Modeling with Probability Density Estimation
 

Density-Based Subset Selection

In the context of data-driven modeling, selecting a representative subset from a larger dataset is a challenging task, particularly when dealing with large amounts of data. Traditional methods such as random sampling often fail to capture the underlying distribution of the data, resulting in subsets that may not be representative. To address this challenge, a novel subset selection algorithm based on kernel density estimation (KDE) was developed in [3].

The algorithm aims to ensure that the selected subset represents either (1) the distribution of the original dataset or (2) an arbitrary distribution—such as the uniform distribution—as closely as possible; see Fig. 1. It operates using a greedy or forward selection approach, meaning that data points from the original dataset are selected one after another and added to the selected subset without replacement. The selection is based on a comparison of estimated kernel densities. For this purpose, the PDF of the original dataset is compared with the PDFs of candidate datasets. The candidate datasets consist of the selected data points and one non-selected data point each.

A key feature of this method is its computational efficiency. The algorithm evaluates the estimated density in a sophisticated manner that significantly reduces the computational load. The effectiveness of density-based subset selection is demonstrated using various benchmark datasets and real-world applications, showing that it performs on par with or better than state-of-the-art methods in capturing the essential characteristics of a dataset.

Figure 1: Two cases for selecting a desired PDF for subset selection with alternative goals. A subset is selected either to (1) fit the original dataset’s data distribution or (2) approximate an arbitrary target distribution. The minus sign, which indicates the discrepancy, should be understood symbolically and represents the comparison of two distributions.

Density-Based Data Weighting

Addressing imbalances in dynamic datasets is a significant challenge in data-driven modeling. Datasets collected from real-world processes often exhibit imbalances, where certain operational regimes are overrepresented while others are underrepresented. This occurs, for example, when regular operation takes place at only a few operating points. To mitigate this issue, a data weighting method is introduced in [2, 1], which adjusts the influence of individual data points using kernel density estimation.

The method extends the loss function by incorporating a weighting term determined by the inverse of the estimated data point density in the regressor space; see Fig. 2. In this way, data points in sparsely populated regions of the regressor space are assigned higher weights, ensuring that they have a greater influence on the model training process. Conversely, data points from overrepresented regions are assigned lower weights.

This approach is validated through extensive testing with (nonlinear) autoregressive with exogenous input ((N)ARX) models on both linear and nonlinear dynamic datasets as well as real-world applications. The results consistently demonstrate improved model performance, especially for imbalanced datasets.

Figure 2: Visualization of the density-based data weighting strategy.
 

References