Vision: Automatic Modeling

Local Model Network

Introduction
 

The model output is calculated by summing the contributions of all M local models (LMs):

Advantages

• Separate input spaces for validity functions (rule premises) z and local models (rule consequents) x.
• Local estimation of local model parameters is much more robust → Regularization.
• Local models can be of any type:
  • linear: the number of parameters scales linearly with dim(x)
  • quadratic: optimization,
  • higher-order polynomials: subset selection or ridge regression required, 
  • from first principles: prior knowledge is effectively utilized.
• The exact form of the validity function is not relevant.
• Efficient greedy learning strategies are available.

Drawbacks

• Undesired interpolation side effect.
• Undesired normalization side effects.
• Suboptimal.

Separate input spaces for validity functions z and local models x

• Additional flexibility.
• Better interpretation.
• z can be viewed as scheduling variables (often low-dimensional).
• x is often high-dimensional, particularly for dynamic systems.

Interpretation as a Takagi-Sugeno Fuzzy System

Partition of Unity

Two Ways to Achieve This

1. Normalization based on Gaussians: 

• Defuzzification
• Undesirable side effects

2. Hierarchy based on sigmoid functions

• Binary tree
• Knots weighted with Psi and 1-Psi

Algorithm 1: Local Linear Model Tree (LOLIMOT)

Properties:

• Partitioning: Axes-orthogonal.
• Structure: Flat (parallel).
• Splitting functions: Normalized Gaussian functions.
• Splitting method: Heuristic, without nonlinear optimization.

LOLIMOT Construction Algorithm

• Incremental (growing) algorithm: Adds one LM in each iteration.
• Split of the locally worst LM.
• Test of all splitting dimensions and selection of the best alternative.
• Local least-squares estimation of the LM parameters.
• Use normalized Gaussian validity functions.

Demonstration Example (Hyperbola)

Algorithm 2: Hierarchical Local Model Tree (HILOMOT)

Properties:

• Partitioning: Axes-oblique.
• Structure: Hierarchical.
• Splitting functions: Sigmoid functions.
• Splitting method: Nonlinear optimization of sigmoid parameters.

HILOMOT Optimization

• Nested approach.
• Separable nonlinear least squares.
• Outer loop: Gradient-based nonlinear optimization of the split.
• Inner loop: One-shot least squares (LS) optimization of local model parameters.

Enhancements 

• Regularization of LS optimization (ridge regression).
• Subset selection instead of LS can determine local structure.
• Constrained nonlinear optimization of the splits guarantees for all local models 
  • 
• Analytical gradient calculation speeds up optimization by a factor of dim(u).
• Derivative of the inverse matrix (LS estimation) is required.
• Automatic, sophisticated smoothness adjustment of sigmoid splitting functions. 

Separable Nonlinear Least Squares

HILOMOT: Extension and Modification of LOLIMOT

• Incremental (growing) algorithm: Adds one LM in each iteration → like LOLIMOT.
• Splits the locally worst LM → like LOLIMOT.
• Optimizes the splitting position and angle using nonlinear optimization → new.
• Automatic smoothness adjustment algorithm → new.
• Local least-squares estimation of the LM parameters → like LOLIMOT.
• Uses sigmoid validity functions → different from LOLIMOT.
• Build a hierarchical model structure → different from LOLIMOT.
   

Demonstration Example (Hyperbola)

Properties of LOLIMOT

• Splits orthogonal to the axes.
• Flat structure can be computed in parallel.
• Strongly suboptimal in high dimensions.
• Faster training.
• Easier to understand and interpret.
• Normalization side effects.

Properties of HILOMOT

• Axes-oblique splits.
• The hierarchical structure fully decouples local models.
• Well suited for high dimensions.
• Analytical gradient calculation for split optimization.
• Superior model accuracy.
• No normalization numerator (defuzzification). → No reactivation effects. The hierarchy automatically guarantees a “partition of unity.”
• Sigmoid functions are easier to approximate than Gaussian functions for microcontroller implementation.

Summary: Advantages of Local Model Networks

• Incremental → all models, from simple to complex, are constructed.
• Local estimation:
  • All local models are decoupled.
  • Extremely fast.
  • Regularization effect → almost no overfitting.
• Local models are linear in their parameters:
  • The global optimum is found.
  • Robust and mature algorithms for least-squares estimation are used.
  • Structure selection techniques can be applied.
  • Adaptive models → recursive algorithms can be applied.
• Local models can be of any linearly parameterized type: constant, linear, quadratic, ...
• Different input spaces for validity functions (rule premises) and local models (rule consequents) → new approaches to dimensionality reduction.
• A complex problem is divided into smaller subproblems (local models) → divide and conquer.
• Many concepts from the linear domain can be applied to the nonlinear domain → particularly important for dynamic models.

LMNtool

• MATLAB toolbox.
• Object-oriented programming.
• Local linear and quadratic models.
• LOLIMOT (axes-orthogonal partitions).
• HILOMOT (axes-oblique partitions).
• Model complexity selected based on the corrected Akaike information criterion.
• Analytical gradient calculation for high training speed
• Reproducible results
• All Fiddle parameters can retain their default values. No expertise required.

• Simple usage:  

  

Next Chapter: 2. Design of Experiments (DoE) Back to Overview