Description
In this assignment, you will be implementing an AR application step by step using planar homographies. Before we step into the implementation, we will walk you through the theory of planar homographies. In the programming section, you will first learn to find point correspondences between two images and use these to estimate the homography between them. Using this homography you will then warp images and finally implement your own AR applications. Instructions 1. Integrity and collaboration: Students are encouraged to work in groups but each student must submit their own work. If you work as a group, include the names of your collaborators in your write up. Code should NOT be shared or copied. Please DO NOT use external code unless permitted. Plagiarism is strongly prohibited and may lead to failure of this course. 2. Start early! This is a much bigger assignment than homework 1. 3. Questions: If you have any questions, please look at Piazza first. Other students may have encountered the same problem, and it may be solved already. If not, post your question on the discussion board. Teaching staff will respond as soon as possible. 4. Write-up: Your write-up should mainly consist of three parts, your answers to theory questions, resulting images of each step, and the discussions for experiments. Please note that we DO NOT accept handwritten scans for your write-up in this assignment. Please type your answers to theory questions and discussions for experiments electronically. 5. Please stick to the function prototypes mentioned in the handout. This makes verifying code easier for the TAs. 6. Submission: Create a zip file, .zip, composed of your write-up, your Python implementations (including helper functions) and results, and the implementations and results for extra credits (optional). Please make sure to remove the data/ folder, and any other temporary files you’ve generated. Your final upload should have the files arranged in this layout: 1 • .zip – / ∗ .pdf ∗ python/ · ar.py · briefRotTest.py · HarryPotterize.py · helper.py · loadVideo.py · matchPics.py · opts.py · planarH.py · q2 1 4.py · yourHelperFunctions.py (optional) ∗ result/ · ar.avi ∗ ec/ (optional for extra credit) · ar ec.py · loadVideo.py · panorama.py · the images required for generating the results. · yourHelperFunctions.py (optional) Please make sure you do follow the submission rules mentioned above before uploading your zip file to Canvas. Assignments that violate this submission rule will be penalized by up to 10% of the total score. 7. File paths: Please make sure that any file paths that you use are relative and not absolute. Not cv2.imread(’/name/Documents/subdirectory/hw2/data/xyz.jpg’) but cv2.imread(’../data/xyz.jpg’). 1 Homographies Planar Homographies as a Warp Recall that a planar homography is an warp operation (which is a mapping from pixel coordinates from one camera frame to another) that makes a fundamental assumption of the points lying on a plane in the real world. Under this particular assumption, pixel coordinates in one view of the points on the plane can be directly mapped to pixel coordinates in another camera view of the same points. 2 Figure 1: A homography H links all points xπ lying in plane π between two camera views x and x 0 in cameras C and C 0 respectively such that x 0 = Hx. [From Hartley and Zisserman] Q1.1 Homography (5 points) Prove that there exists a homography H that satisfies equation 1 given two 3×4 camera projection matrices P1 and P2 corresponding to the two cameras and a plane Π. You do not need to produce an actual algebraic expression for H. All we are asking for is a proof of the existence of H. x1 ≡ Hx2 (1) The ≡ symbol stands for identical to. The points x1 and x2 are in homogenous coordinates, which means they have an additional dimension. If x1 is a 3D vector xi yi zi T , it represents the 2D point xi zi yi zi T (called inhomogenous coordinates). This additional dimension is a mathematical convenience to represent transformations (like translation, rotation, scaling, etc) in a concise matrix form. The ≡ means that the equation is correct to a scaling factor. Note: A degenerate case happens when the plane Π contains both cameras’ centers, in which case there are infinite choices of H satisfying equation 1. You can ignore this special case in your answer. The Direct Linear Transform A very common problem in projective geometry is often of the form x ≡ Ay, where x and y are known vectors, and A is a matrix which contains unknowns to be solved. Given matching points in two images, our homography relationship clearly is an instance of such a problem. Note that the equality holds only up to scale (which means that the set of equations are of the form x = λHx0 ), which is why we cannot use an ordinary least squares solution such as what you may have used in the past to solve simultaneous equations. A standard approach to solve these kinds of problems is called the Direct Linear Transform, where we rewrite the equation as proper homogeneous equations which are then solved in the standard least 3
