Description
Problem 1 Multiclass Ridge Regression Classifier (5 pts each)
In this problem, you will derive and implement a multiclass linear classifier based on ridge regression. Suppose
you are given m training examples {(xi
, yi)}, from K classes, where xi ∈ R
d and yi ∈ {e1, . . . , eK}
1
is a
one-hot vector. One way to perform classification is to find the matrix W ∈ R
d×K that minimizes the least
squares cost function given below. You can then estimate the integer-valued class labels as
yˆi = arg max
k∈[K]
WT xi
.
(a) Derive a closed-form solution to the problem
Wc = arg min
W∈Rd×K
Xm
i=1
WT xi − yi
2
2
+ λ kWk
2
F
,
where λ > 0 is a regularization parameter.
(b) Implement your trained classifier with λ = 10−4 on the full MNIST dataset included in the homework
files. You may not use sklearn or other high-level libraries to perform the one-hot encoding, but you
may use numpy and scipy. Turn in your code, as well as the classification error on the training and
test sets.
Problem 2 DSS: Using scikit-learn (10 pts)
(DSS rules apply.) One important part of data science is fluency with popular libraries. One library that is
useful for a variety of machine learning tasks (especially those not related to deep learning) is scikit-learn,
a.k.a., sklearn. Use sklearn to perform multiclass ridge regression classification, taking special care to make
sure the regularization parameter λ and the use of an offset are the same. Turn in your code, an explanation
of what functions and options you used to perform the multiclass classification, as well as the classification
error on the training and test sets.
Problem 3 SLT (5 pts)
(SLT rules apply.) UML, Ch. 2, Exercise 2. State how long you worked on the problem before looking at
the solution.
Problem 4 SLT (5 pts each)
(SLT rules apply.) UML, Ch. 2, Exercise 3.1-3.2. State how long you worked on the problem before looking
at the solution.
1The vector ek is the kth standard basis vector, taking zero everywhere except in the kth element, where it takes the value
one.
1
