## Description

1 KLT Tracker

In this section you will track a specific object in an image sequence

(carSequence.mat) by template tracking.

The first groundbreaking work on template tracking was the KLT tracker.

It basically assumes that the template undergoes constant motion in a small

region. The KLT Tracker works on two frames at a time, and does not assume

any statistical motion model throughout the sequence. The algorithm estimates

the deformations between two image frames under the assumption that the

intensity of the objects has not changed significantly between the two frames.

In this homework, we assume the motion is translation (the matrix

W we learned from class is a translation matrix here). Starting with a

rectangle Rt on frame It, the KLT Tracker aims to move it by an o↵set (u, v) to

obtain another rectangle Rt+1 on frame It+1, so that the pixel squared di↵erence

in the two rectangles is minimized:

min

u,v J(u, v) = X

(x,y)2Rt

(It+1(x + u, y + v) It(x, y))2 (1)

1.1 Preliminary [14 points]

Starting with an initial guess of (u, v)(usually (0,0)), we can compute the

optimal (u⇤, v⇤) iteratively. In each iteration, the objective function is locally

linearized by first-order Taylor expansion and optimized by solving a linear

system that has the form Ap = b, where p = (u, v)T is the template o↵set.

• What is AT A?

• What conditions must AT A meet so that the template o↵set can be calculated reliably?

1.2 Iterative KLT Tracker [50 points]

Complete the script iterative KLT.m , which computes the optimal local

motion from frame It to frame It+1 that minimizes Eq. 1. Please read the

script file for details. Note that moving the template by u and v will

lead to fractional coordinates of the pixels. To deal with fractional

movement of the template, you will need to interpolate the image

using the MATLAB function interp2. You will also need to iterate the

estimation until the change in (u, v) is below a threshold. You are allowed to

use MATLAB function gradient.

1.3 Testing [25 points]

Complete the script testCarSequence.m that loads the video sequence

(carSequence.mat) and run the KLT Tracker to track the speeding car. The

rectangle in the first frame is [x1, y1, x2, y2] = [328, 213, 419, 265]. In other

2

words, the rectangle starts from (328, 213) (row 213 and column 328 in the

image) and ends at (419, 265).

Save your results as file carPosition.mat which has top-left and bottomright corners of your tracked object for us to evaluate your tracking result.

The carPosition.mat file should contain a variable called box which is a n ⇥

4 matrix; n is the number of frames and each row in the matrix contains 4

numbers: [x1, y1, x2, y2], where indicates the coordinates of top-left and bottomright corners to the object you tracked.

If you complete this script correctly, it will play a video with a rectangle

tracking the car.

In your write-up, print the tracking result (image + rectangle) at frame 1,

frame 20, frame 50 and frame 100. You can use p1 3.m to produce the figures.

And your visualization result should be like figure 1.

Figure 1: Iterative KLT tracking. The bounding box of the car in the first frame

is given, and you need to implement the KLT tracker to locate the car’s position

in the rest of the frames like this.

1.4 Failure analysis [10 points]

In your write-up, give a short analysis on the possible reasons the tracker

could fail; give a possible solution to fix some of these problems.

3

2 Questionnaire [1 point]

Approximately how many hours do you spend on this homework? Please

answer it in your write-up.

4