Description

1. (Point Estimation) You are given a coin and a thumbtack and you put Beta priors Beta(100; 100) and Beta(1; 1) on the coin and thumbtack respectively. You perform the following experiment: toss both the thumbtack and the coin 100 times. To your surprise, you get 60 heads and 40 tails for both the coin and the thumbtack. Are the following two statements true or false?
_ The MLE estimate of both the coin and the thumbtack is the same but the MAP estimate is not.
_ The MAP estimate of the parameter θ (probability of landing heads) for the coin is greater than the MAP estimate of θ for the thumbtack.
Explain your answer mathematically. [5 Points]

2. Point Estimation

a. Given that p is the failure probability, what is the probability of k failures before a suitable “match” is found by your friend.
b. You have performed m independent experiments of this form (namely, asked m of your friends to go out on dates until they find a suitable match), recording k1,…….. km. Estimate the most likely value of p as a function of m and k1,…….. km.
[8 Points]

3. Naive Bayes is a linear classifier. True or False. Explain. [2 Points]

4. Consider learning a function X -> Y where Y is boolean, where X 〈𝑋𝑋1, 𝑋𝑋2〉, and where 𝑋𝑋1 is a boolean variable and 𝑋𝑋2 is continuous variable. State the parameters that must be estimated to define a Naïve Bayes classifier in this case. Give the formula for computing P(Y|X), in terms of these parameters and the feature values 𝑋𝑋1 and 𝑋𝑋2.
[5 Points]

5. CLASSIFICATION :
Imagine that you are given the following set of training examples. Each feature can take on one of three nominal values: a, b, or c.
F1 F2 F3 Category
a c a +
c a c +
a a c –
b c a –
c c b –

How would a Naive Bayes system classify the following test example? Be sure to show your work.
F1 = a, F2 = c , F3 = b
[10 Points]

6. Naïve Bayes
Classify whether a given person is a male or a female based on the measured features using naïve bayes classifier. The features include height, weight, and foot size.
[10 Points] Training Data for the classifier is given in the below table.

Person height (feet) weight (lbs) foot size(inches)
male 6 180 12
male 5.92 (5’11”) 190 11
male 5.58 (5’7″) 170 12
male 5.92 (5’11”) 165 10
female 5 100 6
female 5.5 (5’6″) 150 8
female 5.42 (5’5″) 130 7
female 5.75 (5’9″) 150 9

Below is a sample to be classified as male or female.
Person height (feet) weight (lbs) foot size(inches)
sample 6 130 8

7. Regularization separate terms in 2d logistic regression
[10 Points]

a. Consider the data in Figure where we fir model 𝑝𝑝(𝑦𝑦 = 1 | 𝑥𝑥, 𝑤𝑤) = σ(w0 + w1x1 + w2x2). Suppose we fit the model by maximum likelihood or we minimize
𝐽𝐽(𝑤𝑤) = −𝑙𝑙(𝑤𝑤, 𝐷𝐷𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡)

where 𝑙𝑙(𝑤𝑤, 𝐷𝐷𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡) is the log likelihood on the training set. Sketch a possible decision boundary corresponding to w. Is your answer (decision boundary) unique? How many classification errors does your method make on the training set?

b. Now suppose we regularize only the 𝑤𝑤0 parameter i.e. we minimize

𝐽𝐽(𝑤𝑤) = −𝑙𝑙(𝑤𝑤, 𝐷𝐷𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡) + 𝜆𝜆𝑤𝑤02

Suppose λ is a very large number, so we regularize 𝑤𝑤0 all the way to 0, but all other parameters are unregularized. Sketch a possible decision boundary. How many classification errors does your method make on training set?

c. Now suppose we regularize only the 𝑤𝑤1 parameter i.e. we minimize
𝐽𝐽(𝑤𝑤) = −𝑙𝑙(𝑤𝑤, 𝐷𝐷𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡) + 𝜆𝜆𝑤𝑤12
Sketch a possible decision boundary. How many classification errors does your method make on training set?

d. Now suppose we regularize only the 𝑤𝑤2 parameter i.e. we minimize
𝐽𝐽(𝑤𝑤) = −𝑙𝑙(𝑤𝑤, 𝐷𝐷𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡) + 𝜆𝜆𝑤𝑤22
Sketch a possible decision boundary. How many classification errors does your method make on training set?