This page is a rough summary of the various methods for optimization, curve fitting/linear regression, etc.

First, some definitions:

• In statistics, linear regression is basically a way to make a curve fit a set of data points

## Optimization

• Convex optimization

### One-Dimensional Search Methods

• Golden Section Search 91
• Fibonacci Search 95
• Newton's Method 103
• Secant Method

### Unconstrained Optimization and Neural Networks

• Descent methods
• Line search
• Descent methods with trust region
• Steepest descent
• Single-Neuron Training
• Backpropagation Algorithm

### Newton-Type Methods

• Newton’s method
• Damped Newton methods
• Quasi–Newton methods
• DFP formula
• BFGS formulas
• Quasi–Newton implementation

### Direct Search

• Simplex method
• Method of Hooke and Jeeves

### Linear Data Fitting

• “Best” fit
• Linear least squares
• Weighted least squares
• Generalized least squares
• Polynomial fit
• Spline fit
• Choice of knots

### Nonlinear Least Squares Problems

• Gauss–Newton method
• The Levenberg–Marquardt method
• Powell’s Dog Leg Method
• Secant version of the L–M method
• Secant version of the Dog Leg method

### Duality

• The Lagrange dual function
• The Lagrange dual problem