This blog mainly explains the linear least squares problem, which mainly includes the following contents:

• Definition of least squares problem
• Normal equation solving
• Chomsky decomposition method
• QR decomposition method
• Singular value decomposition method
• Least squares of homogeneous equations

I. Definition of the problem

The least squares problem can usually be expressed as a fitting of a certain model by collecting some data (acquiring the obtained samples) and making the model results and samples reach a certain degree of best fit as much as possible:

Converted to a mathematical expression as:

Where x is the vector of the parameters in the model, and e is often referred to as the residual vector.
Now suppose our model function is Ax, the sample is b and the number of equations is greater than the unknown number:
Convert to a least squares expression as:
The equation can usually be solved by normal equations, QR decomposition, Cholesky decomposition and SVD.

2. Solution method

2.1. Normal Equation

willExpand to get:
To solve for the optimal solution (ie the minimum) of the equation, we can solve its partial derivative of the parameter x and make it equal to zero:
After simplification, you get:
The above is called the normal equation of least squares. Further solution can be obtained:
The result can also be expressed as a pseudo-inverse form of the matrix (the pseudo-inverse is the inverse matrix generalized form, the singular matrix or the non-square matrix does not have the inverse matrix, but the pseudo-inverse matrix can be solved)

2.2. Chomsky Decomposition

The following content reference[1].
Since the simple normal equations calculate the amount of computation, for faster and more stable solutions, Chomsky decomposition can usually be used to solve.
The Chomsky decomposition of the left side of the normal matrix is:

This Chomsky decomposition generates an upper triangular matrix R ̄ and its transposed matrix, so we can calculate the solution vector by successive forward and backward substitutions:

2.3. QR decomposition

The following picture reference[4]

QR decomposition can refer to matrix decomposition as the product of a standard orthogonal square matrix and an upper triangular matrix:

There are many ways to calculate QR decomposition. The following is an example of Householder transformation. First, we make a simple analysis of Householder-based QR decomposition. Suppose A is a 5X4 matrix. We use X to represent the unchanged elements of this transformation. The elements that represent transformations with + are:

  

By analogy, matrix A can be converted to an upper triangular matrix by four Householder transformations, and if A is not correlated, then R is a non-singular matrix, then:

So how do we calculate the QR decomposition by the Householder method in the actual calculation? First, let's review the Householder change. For the Householder transformation, we have the following definitions:

Then we use a practical calculation example to illustrate the QR decomposition of Householder. Case reference[2]. Suppose now we have the following five data:

(-1, 1), (-0.5, 0.5), (0, 0), (0.5, 0.5), (1.2)

And there is a linear system that satisfies this data as follows:

First of all, our team will handle the first column:

The √5 isValue, then we can pass the formulaCalculated

Then, and so on, the second and third columns are solved:

This completes the QR decomposition of matrix A.
Through the above method we can find the Q and R matrices, then how to solve the least squares problem through them.
For a least squares problem we have:

By performing QR decomposition on A, we can get:

Here we willSplit into two parts, Q1 represents the first n lines, and Q2 represents the remainder of the multiplication with zero.

So further we have:

The result is an expression that solves the linear least squares for QR decomposition.

2.4. Singular Value Decomposition (SVD)

The least squares problem of the above form can also be solved by the SVD decomposition method. forWe perform SVD decomposition on matrix A:

Since we have assumed that the problem is overdetermined (m>n), there are:

The solution method of SVD is summarized as follows, refer to [3].

The least squares of the homogeneous equation

The least squares problem we discussed earlier is in the form of Ax − b = 0. If the problem form changes to Ax = 0, how should such a least squares problem be solved?

Problems with the form Ax = 0 often appear in reconstruction problems. We expect to find a solution where the equation Ax = 0 and x is not equal to zero. Due to the special form of the equation we find that for a solution where x is not equal to zero, we multiply any scale factor k so that the solution becomes kx without changing the final result. Because we can convert the problem to solve x such that the ||Ax|| value is the smallest and ||x||=1.

Now the SVD decomposition of matrix A is:

Through the properties of the D matrix in the SVD decomposition we can find that D is a diagonal matrix and its diagonal elements are arranged in descending order. So the solution to the equation should be:

The solution has a non-zero entry with a value of 1 at the last position. Based on this result, according to the equation:

We can see that the value of x is the last column of matrix V.

-------------------------------------------------------------------------------------------------------------------------

Attached:

For Ax=b, A is a matrix of mXn, and the result has the following three possibilities:

• m<n, the number of unknowns is greater than the equation format, in which case there is no unique solution, but there is a vector set of solutions;
• m=n, if the matrix A is reversible, there is a unique solution;
• m>n, the number of equations is greater than the number of unknowns. Under normal circumstances, the equation will not have a solution, unless b is located in the column space of matrix A;

Please let me know if there is any delay in the above content.

 

Reference content:

[1]. Solving Linear and Nonlinear Least Squares Problems with Math.NET

[2]. Least Squares Approximation/Optimization

[3]. Multiple View Geometry in Computer Vision, Appendix 5, Least-squares Minimization

[4]. Matrix Method in Machine Learning 03: QR Decomposition