Description
1. [Hadoop MapReduce, 60 points] In this problem, you are asked to write a Hadoop MapReduce
program, TwoPhase.java, that computes the multiplication of two given matrices using the twophase approach described in class.
A template file is provided to you, which contains the skeleton for mapper and reducer of phase one
and two. You only need to supply codes for the block indicated by “// fill in your code” add import
statements at the beginning of the file if necessary (you can only import packages from java.*). DO
NOT modify other parts of the template.
Note that the input matrices A and B are stored in two separate directories, e.g., mat-A and mat-B.
Each directory contains a single text file, each line of which is a tuple (row-index, column-index,
value) that specifies a non-zero value in the matrix.
For example, the following is the content of mat-A/values.txt which stores the entries for the matrix
A shown above.
0,0,2
0,1,2
0,2,1
1,0,2
1,1,1
2,0,1
2,2,2
MultileInputs class is used to specify different mappers for different input directory.
You also need to submit your jar file with name __2p.jar.
Example invocation of your program is as follows:
bin/hadoop jar __2p.jar TwoPhase mat-A
mat-B output
Your output directory (the file part-r-00000) should contain the entries of matrix C (= A * B) in the
following format (tab-separated). For example,
INF 553 – Spring 2017
2
1,1 3
1,2 7
2,1 2
2,2 4
3,1 3
3,2 3
Submissions: Submit both __TwoPhase.java and
__2p.jar. DO NOT make them into folder or zip file.
2. [40 points] Write a Spark program in Python, TwoPhase.py, which implements the same 2-phase
approach as in Problem 1.
Restrictions: you can NOT use join (and any outer join variations) in your code, but may use any
other transformations and actions provided in the lecture. This gives you a chance to think about
how to implement join using other methods in Spark. It will also maximize the benefits of this
exercise. In the end, you will implement a very similar program in Spark as you have done in Hadoop
MapReduce for Problem 1.
Your program should be invoked as follows.
bin/spark-submit __TwoPhase.py matA/values.txt mat-B/values.txt output.txt
It should produce the same output as Problem 1 and store the output in a file “output.txt”.
Submissions: Submit __TwoPhase.py.
Notes:
In both questions, you may assume that matrix entries are all integers.
Make sure to follow the output format (The order of output entries doesn’t matter) and the
naming format. If you don’t follow either one or both of them, 20% points will be deducted.
You MUST implement your program using the two-phase approach described in class, and
you CAN NOT use join transformation in your python program. If you don’t follow these
instructions, 80% points will be deducted.
Make sure that your program can be invoked correctly. If not, no more than 50% of the
points can be earned.
