Description

From Chapter 6 page 261 (Use R or Rstudio)
(a) (3 marks) Show that the ridge regression optimization problem in this setting (or the quantity in equation 6.5 in Chapter 6 in this setting) is 2(𝑦’ − (𝛽’ + 𝛽()𝑥’’)( + 𝜆(𝛽’( + 𝛽(().
(b) (5 marks) Show that in the setting (a), the ridge coefficient estimates satisfy 𝛽-’ = 𝛽-( .
(c) (3 marks) Show that the lasso regression optimization problem in this setting (or the quantity in equation 6.7 in Chapter 6 in this setting) is 2(𝑦’ − (𝛽’ + 𝛽()𝑥’’)( + 𝜆(|𝛽’| + |𝛽(|).
(d) (5 marks) Show that in the setting (c), the lasso coefficients 𝛽-’and 𝛽-’are not unique—in other words, there are many possible solutions to the optimization problem.

Q8. In this exercise, we will generate simulated data, and will then use this data to perform best model selection. Use the 𝑟𝑛𝑜𝑟𝑚() function to generate a predictor 𝑋 of length 𝑛 = 100 , as well as a noise vector 𝜖 of length n = 100 such that 𝜖 = 0.1 * 𝑟𝑛𝑜𝑟𝑚(𝑛)
(a) (1 mark) Generate (use set.seed(19)) a response vector 𝑌 of length 𝑛 = 100 according to the model
𝑌 = 𝛽. + 𝛽’𝑋 + 𝛽(𝑋( + 𝛽;𝑋; + 𝜖
where 𝛽., 𝛽’, 𝛽(, and 𝛽; are constants as 𝛽. = 1.0, 𝛽’ = −0.1, 𝛽( = 0.05, 𝛽; = 0.75
(b) Use the regsubsets() function to perform best subset selection in order to choose the best model containing the predictors 𝑋, 𝑋(, 𝑋;, … , 𝑋A using the measures 𝑪𝒑, 𝑩𝑰𝑪, 𝒂𝒅𝒋𝒖𝒔𝒕𝒆𝒅 𝑹𝟐
(i) (6 marks) Plot each measure against number of predictors on the same page using par(mfrow=c(2,2)).
(ii) (3 marks) Give the best model coefficients obtained from each
𝐶P, 𝐵𝐼𝐶, 𝑎𝑑𝑗𝑢𝑠𝑡𝑒𝑑 𝑅(.
Note:
1. You will need to use the data.frame() function to create a single data set containing both X and Y.
(c) Now fit a ridge regression model to the simulated data, again using 𝑋, 𝑋(, 𝑋;, … , 𝑋A as predictors.
(i) (2 marks) Plot the extracted coefficients as a function of log(λ) with a legend containing each curve colour and its predictor name at the top-right corner.
(ii) (4 marks) Plot the cross-validation (set.seed(20)) error as a function of log(λ) to find the optimal λ .
(iii) (1 mark) Give coefficient estimates for the optimal value of λ.
(d) Now fit a lasso model to the simulated data, again using 𝑋, 𝑋(, 𝑋;, … , 𝑋A as predictors.
(i) (2 marks) Plot the extracted coefficients as a function of log(λ) with a legend containing each curve colour and its predictor name at the top-right corner.
(ii) (4 marks) Plot the cross-validation (set.seed(21)) error as a function of log(λ) to find the optimal λ .
(iii) (1 mark) Give coefficient estimates for the optimal value of λ.
Note:
1. Use cv.glmnet() to do the cross-validation and use the default of 10-fold cross-validation.