Assignment Task

Question 1:

In maximum likelihood estimation, the estimator is obtained by maximizing the log-likelihood function. However, most of the log-likelihood has to be optimized by the Newton-Raphson algorithm. In this question, we will learn to program the Newton-Raphson algorithm for a univariate function. Consider the function f(θ) = 3θ 2−1 1+θ 3 , θ > 0. Use the Newton-Raphson method to find the maximizer of the function. Implement the algorithms in R and run your code to obtain the maximizer. (Attach your screenshots of the output in R). You can use the online symbolic differentiation calculator to obtain f 0 (θ) and f 00(θ).

Question 2:

In the following marketing set, we have 9 years with the sales in 10 million euro and the advertising expenditure in million euro.

a) Based on the 9 observation and perform a ridge regression. Program it with R. Output the ridge regression results at a few different values of λ.

b) In your ridge regression, when λ increaes, what do you observe from the values of the estimated coefficients. Does any of the estimated coefficients shrink to zero like the L1 LASSO regression? Describe the difference between the output of a ridge regression and the output of a lasso regression.

Question 3:

a. In this question, we will investigate the problem of mulitple testing. Consider the hypothesis testing of H0 : µ = 0, vs Ha : µ 6= 0. Under the null hypothesis, the Z test statistic is a standard normal random variable. We reject the null hypothesis when |Z| is greater than 1.96 at the significance level of 0.05. Write a R program to simulate 1000 Z test statistic from standard normal N(0, 1). 

b. If we perform hypothesis testing using the significance level of 0.05. Among the 1000 test statistics you generated, how many of them are rejected?

c. Apply the Bonferroni method to control the overal type I error rate to be 0.05. Based on your program, how many rejections do you obtain from your simulation?

d. As the Zs are generated from the null hypothesis, we consider these rejections are all false positive discoveries. Please use a short paragraph to summarize the problem we are facing when we perform multiple testings.

Question 4: Analyze the QSAR fish toxicity data set: 

a) Simulate 100 unimportant predictors and append those predictors to the date set. (Use rnom command to generate the values for the unimportant predictors. Make sure each unimportant predictor has the same number of values as the six original biological measurements in the data set. Perform variable selection on the one hundred and six predictors using the lasso package.

b) Use the cross-validation method, select the best value for the fraction s based on the plot of cross-validation error againt the fraction s. The fraction s measures the ratio of the L1 norm of the penalized estimate over the L1 norm of the regular penalized estimate.

c) Use the optimum s you select, perform the penalized regression and output the optimum model. Is the model successful in terms of weeding out all the unimportant predictors?

Question 5: Analyze the German data set:

a) Perform the logistic regression on the dataset. Build a predictive model using some of the predictors in the model. Please use 900 observations as the training set and use your model to predict the default status of the remaining 100 loans. Choose one of the regression coefficents and interpret the regression coefficient. What is the cutoff value of the probability do you use for your analysis? How many default ones are predicted to be non-default ones (number of false negative)? How many non-default ones are predicted to be default ones (number of false positive). Then you need to improve your model by adding more predictors or adding some higher-order terms or interaction terms. Please demonstrate that your new model has lesser errors than your first model in the 100 testing cases.

b) Please investigate how the sensitivity and specificity change with respect to the different cutoff value of probability. 

Question 6:

In logistic regression, we assume Y = (Y1, . . . , Yn) T are a collection of n binary observations. For each Yi , we observe Xi = (Xi1, Xi2, Xi3) T predictors. We assume

log pi/1-pi 1 − pi = X T i θ,

where θ = (θ1, . . . , θ3) T is the vector of regression coefficients.

a) Formulate the overall loglikelihood of the dataset l(Y ).

b) Derive the first derivative ∂l(Y )/∂θ2.

c) Derive the second derivative ∂ 2 l(Y )/∂θ2∂θ3.

d) Suppose ˆθ = (0.1, 0.2, 0.3)T, and we have a new observation Xn+1 = (3, 2, 4)T. Predict the probability p of success for this new observation.