Optimization - Revision history

Patapom at 09:01, 5 August 2017

2017-08-05T09:01:20Z

Patapom at 09:05, 4 August 2017

2017-08-04T09:05:01Z

Patapom at 18:29, 30 July 2017

2017-07-30T18:29:59Z

Patapom at 18:18, 30 July 2017

2017-07-30T18:18:07Z

Patapom at 15:22, 30 July 2017

2017-07-30T15:22:42Z

Patapom at 15:22, 30 July 2017

2017-07-30T15:22:34Z

Patapom: Created page with "This page is a rough summary of the various methods for optimization, curve fitting/linear regression, etc. First, some definitions: * In statistics, '''linear regression''' ..."

2017-07-30T15:22:01Z

Created page with "This page is a rough summary of the various methods for optimization, curve fitting/linear regression, etc. First, some definitions: * In statistics, '''linear regression''' ..."

New page

This page is a rough summary of the various methods for optimization, curve fitting/linear regression, etc.

First, some definitions:
* In statistics, '''linear regression''' is basically a way to make a curve fit a set of data points
[[Linear_Regression.png]]

*

== Optimization ==

← Older revision		Revision as of 09:05, 4 August 2017
Line 6:		Line 6:

	*		*
		+
		+
		+	TODO: [https://en.wikipedia.org/wiki/Gaussian_quadrature Gaussian Quadrature]

	== Optimization ==		== Optimization ==

@@ Line 9: / Line 9: @@
 == Optimization ==
-=== Unconstrained Optimization ===
+* Convex optimization
 * Descent methods
-* Line search . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
+* Line search
-* Descent methods with trust region . . . . . . . . . . . . . . . . . 26
+* Descent methods with trust region
-* Steepest descent . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
+* Steepest descent
-* Quadratic models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
+* Quadratic models
-* Conjugate gradient methods . . . . . . . . . . . . . . . . . . . . . . . . 31
+* Conjugate gradient methods
 === Newton-Type Methods ===
-* Newton’s method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
+* Newton’s method
-* Damped Newton methods . . . . . . . . . . . . . . . . . . . . . . . . . . 48
+* Damped Newton methods
-* Quasi–Newton methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
+* Quasi–Newton methods
-* DFP formula . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
+* DFP formula
-* BFGS formulas . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
+* BFGS formulas
-* Quasi–Newton implementation . . . . . . . . . . . . . . . . . . . . . 63
+* Quasi–Newton implementation
 === Direct Search ===
-* Simplex method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
+* Simplex method
-* Method of Hooke and Jeeves . . . . . . . . . . . . . . . . . . . . . . . 70
+* Method of Hooke and Jeeves
 === Linear Data Fitting ===
-* “Best” fit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
+* “Best” fit
-* Linear least squares . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
+* Linear least squares
-* Weighted least squares . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
+* Weighted least squares
-* Generalized least squares . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
+* Generalized least squares
-* Polynomial fit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
+* Polynomial fit
-* Spline fit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
+* Spline fit
-* Choice of knots . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
+* Choice of knots
 === Nonlinear Least Squares Problems ===
-* Gauss–Newton method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116
+* Gauss–Newton method
-* The Levenberg–Marquardt method . . . . . . . . . . . . . . . . . 120
+* The Levenberg–Marquardt method
-* Powell’s Dog Leg Method . . . . . . . . . . . . . . . . . . . . . . . . . . 125
+* Powell’s Dog Leg Method
-* Secant version of the L–M method . . . . . . . . . . . . . . . . . 130
+* Secant version of the L–M method
-* Secant version of the Dog Leg method . . . . . . . . . . . . . . 134
+* Secant version of the Dog Leg method

← Older revision		Revision as of 18:18, 30 July 2017
Line 8:		Line 8:

	== Optimization ==		== Optimization ==
		+
		+	=== Unconstrained Optimization ===
		+	* Descent methods
		+	* Line search . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
		+	* Descent methods with trust region . . . . . . . . . . . . . . . . . 26
		+	* Steepest descent . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
		+	* Quadratic models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
		+	* Conjugate gradient methods . . . . . . . . . . . . . . . . . . . . . . . . 31
		+
		+	=== Newton-Type Methods ===
		+	* Newton’s method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
		+	* Damped Newton methods . . . . . . . . . . . . . . . . . . . . . . . . . . 48
		+	* Quasi–Newton methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
		+	* DFP formula . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
		+	* BFGS formulas . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
		+	* Quasi–Newton implementation . . . . . . . . . . . . . . . . . . . . . 63
		+
		+	=== Direct Search ===
		+	* Simplex method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
		+	* Method of Hooke and Jeeves . . . . . . . . . . . . . . . . . . . . . . . 70
		+
		+	=== Linear Data Fitting ===
		+	* “Best” fit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
		+	* Linear least squares . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
		+	* Weighted least squares . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
		+	* Generalized least squares . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
		+	* Polynomial fit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
		+	* Spline fit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
		+	* Choice of knots . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
		+
		+	=== Nonlinear Least Squares Problems ===
		+	* Gauss–Newton method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116
		+	* The Levenberg–Marquardt method . . . . . . . . . . . . . . . . . 120
		+	* Powell’s Dog Leg Method . . . . . . . . . . . . . . . . . . . . . . . . . . 125
		+	* Secant version of the L–M method . . . . . . . . . . . . . . . . . 130
		+	* Secant version of the Dog Leg method . . . . . . . . . . . . . . 134

@@ Line 1: / Line 1: @@
 This page is a rough summary of the various methods for optimization, curve fitting/linear regression, etc.
@@ Line 6: / Line 11: @@
+*
 == Optimization ==

@@ Line 3: / Line 3: @@
 First, some definitions:
 * In statistics, '''linear regression''' is basically a way to make a curve fit a set of data points
-[[File:Linear_Regression.png|thumbnail]]
+[[File:Linear_Regression.png]]
+*
 == Optimization ==