Vision: Automatic Modeling

Local Model Network

Introduction
 

Model output is calculated by summing up the contributions of all M local models (LMs):

Advantages

• Separate input spaces for validity functions (rule premises) z and local models (rule consequents) x.
• Local estimation of local model parameters much more robust → Regularization.
• Local models can be of arbitrary type:
  • linear: number of parameters scale linearly with dim(x)
  • quadratic: optimization,
  • higher order polynomials: subset selection or ridge regression required, 
  • from first principles: prior knowledge effectively utilized.
• Exact shape of validity function not relevant.
• Efficient greedy learning strategies are available.

Drawbacks

• Undesired interpolation side effect.
• Undesired normalization side effects.
• Suboptimal.

Separate Input Spaces for Validity Functions z and Local Models x

• Additional flexibility.
• Better interpretation.
• z can be seen as scheduling variables (often low-dimensional).
• x often is high-dimensional, particularly for dynamic systems.

Interpretation as Takagi-Sugeno Fuzzy System

Partion of Unity

Two Ways to Achieve This

1. Normalization based on Gaussians: 

• Defuzzification
• Undesirable side effects

2. Hierarchy based on sigmoids

• Binary tree
• Knots weighted with Psi and 1-Psi

Algorithm 1: Local Linear Model Tree (LOLIMOT)

Properties:

• Partitioning: Axes-orthogonal.
• Structure: Flat (parallel).
• Splitting functions: Normalized Gaussian functions.
• Splitting method: Heuristically, without nonlinear optimization.

LOLIMOT Construction Algorithm

• Incremental (growing) algorithm: Adds one LM in each iteration.
• Split of the locally worst LM.
• Test of all splitting dimensions and selection of the best alternative.
• Local least squares estimation of the LM parameters.
• Use normalized Gaussian validity functions.

Demonstration Example (Hyperbola)

Algorithm 2: Hierarchical Local Model Tree (HILOMOT)

Properties:

• Partitioning: Axes-oblique.
• Structure: Hierarchical.
• Splitting functions: Sigmoid functions.
• Splitting method: Nonlinear optimization of sigmoid parameters.

HILOMOT Optimization

• Nested approach.
• Separable nonlinear least squares.
• Outer loop: Gradient-based nonlinear optimization of split.
• Inner loop: One-shot least squares (LS) optimization of local model parameters.

Enhancements 

• Regularization of LS optimization (ridge regression).
• Subset selection instead of LS can determine local structure.
• Constraint nonlinear optimization of the splits guarantees for all local models 
  • 
• Analytical gradient calculation speeds up optimization by factor dim(u).
• Derivative of inverse matrix (LS estimation) necessary.
• Automatic sophisticated smoothness adjustment of sigmoid splitting functions. 

Separable Nonlinear Least Squares

HILOMOT: Extension and Modification of LOLIMOT

• Incremental (growing) algorithm: Adds one LM in each iteration → like LOLIMOT.
• Split of the locally worst LM → like LOLIMOT.
• Optimize splitting position and angle by nonlinear optimization → new.
• Automatic smoothness adjustment algorithm → new.
• Local least squares estimation of the LM parameters → like LOLIMOT.
• Use sigmoid validity functions → different to LOLIMOT.
• Build up a hierarchical model structure → different to LOLIMOT.
   

Demonstration Example (Hyperbola)

Properties of LOLIMOT

• Axes-orthogonal splits.
• Flat structure can be computed in parallel.
• Strongly suboptimal in high dimensions.
• Faster training.
• Easier to understand and interpret.
• Normalization side effects.

Properties of HILOMOT

• Axes-oblique splits.
• Hierarchical structure fully decouples local models.
• Well suited for high dimensions.
• Analytical gradient calculation for split optimization.
• Superior model accuracy.
• No normalization numerator (defuzzification). → No reactivation effects. Hierarchy automatically guarantee a ‘partition of unity’.
• Sigmoids are easier to approximate than Gaussians for microcontroller implementation.

Summary: Advantages of Local Model Networks

• Incremental → all models from simple to complex are constructed.
• Local estimation:
  • All local model are decoupled.
  • Extremely fast.
  • Regularization effect → almost no overfitting.
• Local model are linear in their parameters:
  • Global optimum is found.
  • Robust & mature algorithms for least squares estimation are utilized.
  • Structure selection technique can be applied.
  • Adaptive models → recursive algorithms can be applied.
• Local model can be of any linearly parameterized type: constant, linear, quadratic, ...
• Different input spaces for validity functions (rule premises) and local models (rule consequents) → new approaches to dimensionality reduction.
• Complex problem is divided into smaller sub-problems (local models) → divide & conquer.
• Many concepts from the linear world can be transferred to the nonlinear world → particularly important for dynamic models.

LMNtool

• MATLAB toolbox.
• Object-oriented programming.
• Local linear and quadratic models.
• LOLIMOT (axes-orthogonal partitions).
• HILOMOT (axes-oblique partitions).
• Model complexity chosen according to corrected Akaike information criterion.
• Analytical gradient calculation for high training speed
• Reproducible results
• All fiddle parameters can keep their default values. No expertise required.

• Trivial usage:  

  

Next Chapter: 2. Design of Experiments (DoE)     Back to Overview