General Idea of Input Selection

• Forward Selection (FS)
  • fast
  • recommended for many potential inputs but only few selected
  • ignores correlations/interactions between potential inputs
• Backward Elimination (BE)
  • slower
  • recommended if most potential inputs are selected
  • takes into account correlations/interactions between potential inputs
• Stagewise Selection
  • combination of FS and BE
• Brute Force
  • all input combinations
     

Input Selection - What for?

Input Selection in the Context of Machine Learning

For choice of optimal model complexity, input selection evaluates the merits of input subsets:

Motivation

• Optimal bias/variance trade-off
• Explore relevance of inputs

Aims

• Weakening the curse of dimensionality
• Reducing the model’s variance error
• Increase of process understanding
• More concise and transparent models
• May support future design of experiments (DoE) or active learning strategies

Local Model Networks Enhanced

Separate Inputs for Local Models and Validity Functions 

• Whole problem split into smaller sub-problems
• Training algorithm necessary
  • Here: Hierarchical Local Model Tree (HILOMOT)
     

Separation between Linear and Nonlinear Effects 

• LMNs Reveal Possibility to Separate between Linear and Nonlinear Effects
• Any physical input ui can be included in the rule premises zi and/or in the rule consequents xi
• The separation between linear and nonlinear effects is not possible for other types of models
• In the following the rule premises are referred to as the z-input space, the rule consequents are referred to as the x-input space
   

Equivalence to Fuzzy - Interpretation as Takagi-Sugeno Fuzzy System

Input Selection With the HILOMOT-Algorithm
 

Input Spaces
 

Overview: Different Selection Strategies for Input Selection Tasks

Depending on the selection strategy either subsets or all physical inputs can be assigned to the x- and z-input space.

Demonstration Example: HILOMOT Wrapper Method
 

Artificial Process 

• Superposition of:
  •  Hyperbola f(u1)
  •  Gaussian function f(u2)
  •  Linear function f(u3)
  •  Normal distributed noise, σ = 0, σ = 0.05 (for u4)

Setting

• Training samples N = 625 & N = 1296
• Placed on a 4-dimensional grid

HILOMOT Wrapper Settings 

• Evaluation criterion: Akaike’s information criterion (AICc)
• Search Strategy: Backward elimination (BE)

4-D Demonstration Example: Linked x-z-Selection
 

Results for the Linked x-z-Selection

• Because of the BE search strategy, the results have to be read from right to left
• Each ‘o’ is labeled with the input that is discarded in the corresponding BE step
• A growing sample size N reduces the bad influence of the useless input u4
• For both sample sizes N, the HILOMOT wrapper method identifies the 4th input as not useful in the linked x-z-space

4-D Demonstration Example: z-Selection
 

Results for the z-Selection

• Inputs u3 and u4 are identified as useless in the z-space
  • Because the slope in u3 direction does not change, no partitioning has to be done in that direction
• As soon as an important input in the z-space is removed the evaluation criterion gets significantly worse

4-D Demonstration Example: Separated x-z-Selection
 

Results for the Separated x-z-Selection

• Input u4 is identified as useless in the x- as well as in the z-space for both sample sizes N
• Input u3 is identified as useless in the z-space for both sample sizes N
• After the first useful input is discarded, the evaluation criterion gets slightly worse

Real-World Demonstration Examples
 

Auto Miles Per Gallon (MPG) Data Set

(From the UCI Machine Learning Repository (http://archive.ics.uci.edu/ml) 2010)

• N = 392 samples
• q = 1 physical output:
  •  Auto miles per gallon
• p = 7 physical inputs:
  • -Cylinders (u1)
  • -Displacement (u2)
  • -Horsepower (u3)
  • -Weight (u4)
  • -Acceleration (u5)
  • -Model year (u6)
  • -Country of origin (u7)
• Data Splitting
  • ¾ of data: training samples (used for network training, complexity selection, input selection)
  •  ¼ of data: test samples (only for final testing)
  •  Heuristic, deterministic data splitting strategy

Auto MPG Wrapper Input Selection Results

Results for the x-z-Input Selection

• Both search strategies yield similar results:
  • Best AICc-Value for 5 inputs
  • Same input combination
• Input selection path:
  • Indicates the input combination to a given number of inputs
  • In case of BE read from right to left

Results on the Test Data

• Mean squared error (MSE) for best AICc-Values:
  • MSE (5 inputs) = 6.0
  • MSE (all inputs) = 7.7
• Most important inputs: u4 (car weight) and u6 (model year)
• Improvement of model accuracy
• Information about very important inputs

Results for the z-Input Selection

• Best AICc-Value ES result:
  • 4 inputs
• Best AICc-Value BE result:
  • 4 inputs
  • BUT other input combination(see input selection path)

Results on the Test Data

• MSE for best AICc-Values:
  • MSE (best ES) = 5.2
  • MSE (best BE) = 5.3
  • MSE (all inputs) = 7.7
• Most important inputs: u2 (displacement) and u6 (model year)
• Improvement of model accuracy
• Information about nonlinear input influences

Climate Control
 

Process

• 2 controlled inputs (u1, u2)
  • valve positions
• 4 measured inputs (d1, d2, d3, d4)
  • temperature, rel. humidity, temp., temp. 
• 2 measured outputs (y1, y2):
  • temperature, rel. humidity
• Difficult modeling with ARX: Unstable models are possible

Selection
 

• Selection on the simulated output of validation data.
• Backward selection gives significantly better results.
• ~15 inputs/regressors are sufficient for a good model.
• Allowed delays: (k-1) … (k-10).
• Forward selection: Tries to model the process with multiple nonlinear influences.
• Backward selection: Nonlinearities only need to be considered in u1(k-1) and u2(k-2)!
   

Next Chapter: 4. Metamodeling     Back to Overview