Description
Use the Dynamic Mode Decomposition method to take a video clip containing a foreground and
background object and separate the video stream to both the foreground video and a background.
Make several test videos to try the algorithm on. You can use your smart phone to generate appropriate videos.
The DMD spectrum of frequencies can be used to subtract background modes. Specifically,
assume that ωp, where p ∈ {1, 2, . . . , `}, satisfies ∥ωp∥ ≈ 0, and that ∥ωj∥ ∀ j ≠ p is bounded away
from zero. Thus,
XDMD = bpϕp
e
ωpt
´ ¹¹¹¹¹¹¹¹¸ ¹¹¹¹¹¹¹¹¶
Background Video
+ ∑
j≠p
bjϕj
e
ωj t
´ ¹¹¹¹¹¹¹¹¹¹¹¹¹¸ ¹¹¹¹¹¹¹¹¹¹¹¹¹¶
Foreground Video
(1)
Assuming that X ∈ R
n×m, then a proper DMD reconstruction should also produce XDMD ∈ R
n×m.
However, each term of the DMD reconstruction is complex: bjϕj
exp (ωjt) ∈ C
n×m ∀j, though they
sum to a real-valued matrix. This poses a problem when separating the DMD terms into approximate
low-rank and sparse reconstructions because real-valued outputs are desired and knowing how to
handle the complex elements can make a significant difference in the accuracy of the results.
Consider
calculating the DMD’s approximate low-rank reconstruction according to
XLow-Rank
DMD = bpϕp
e
ωpt
.
Since it should be true that
X = XLow-Rank
DMD + X
Sparse
DMD ,
then the DMD’s approximate sparse reconstruction,
X
Sparse
DMD = ∑
j≠p
bjϕj
e
ωj t
,
can be calculated with real-valued elements only as follows. . .
X
Sparse
DMD = X − ∣XLow-Rank
DMD ∣,
where ∣ ⋅ ∣ yields the modulus of each element within the matrix. However, this may result in X
Sparse
DMD
having negative values in some of its elements, which would not make sense in terms of having
negative pixel intensities.
These residual negative values can be put into a n × m matrix R and then
be added back into XLow-Rank
DMD as follows:
XLow-Rank
DMD ← R + ∣XLow-Rank
DMD ∣
X
Sparse
DMD ← X
Sparse
DMD − R
This way the magnitudes of the complex values from the DMD reconstruction are accounted for,
while maintaining the important constraints that
X = XLow-Rank
DMD + X
Sparse
DMD ,
so that none of the pixel intensities are below zero, and ensuring that the approximate low-rank and
sparse DMD reconstructions are real-valued. This method seems to work well empirically.
NOTE: it is pretty easy to produce a video on your smart phone.
