This page is a rough summary of the various methods for optimization, curve fitting/linear regression, etc.
First, some definitions: