Lukas Kanade Optical flow – Idea

Assumptions

1. The neighboring points have a constant flow
2. Brightness is consistent

Idea

Take a patch (5×5) around a feature point

Feature is detected by a feature extractor such as harris corner detection (using PCA )

For each pixel in the patch calculate dI/dx, dI/dy, dI/dt

i.e. the derivatives across x,y direction of the image and also the change in time dt.

25 equations

We have 25 pixels and we can set up a vectorized equation as follows:

I_xu + I_yv + I_t = 0 where u and v are the displacement of pixels from I_t-1 to I_t

I_xu + I_yv = -I_t

This is in the form of Ax = b — (1)

Minimize eqn 1 using least square:

|Ax – b|² = 0 => (Ax – b) (Ax – b)^T

We can alternatively deduce the above equation to A^TAx = A^Tb — (2)

We can solve eqn 2 if the A^TA is invertible i.e. x = (AA^T)^-1A^Tb, and the final expression is:

Invariant to large motions

Create a pyrmaid of different image resolution. Smaller the image, the large motion of pixel can be captured better.