Logistic regression

In this lab, we will see how to apply what we have learned about logistic regression.

1. Just to be sure to understand the math

In this section we will check if the maths related to logistic regression are properly understood. To this aim, we will fit a logistic regression model on simulated data (so that the true model is perfectly known) from scratch and using the dedicated builtin R function.

a) Generate some artificial data

Let’s first simulate some data

set.seed(1)##fix the random seed so that we all have the same results
n.obs <- 100

## Build some features
X1 <- cos(seq(0, 2 * pi, length = n.obs))
X2 <- 1:n.obs / n.obs

## A design matrix
X <- cbind(1, X1, X2)

## The true parameter beta
beta <- c(2, -2, 0.5)

## The true response
odds <- exp(X %*% beta)
ptheo <- odds / (1 + odds)

## The observed response (0, 1)
Y <- rbinom(n.obs, 1, ptheo)
table(Y)

Y
 0  1 
15 85 

b) Analysis “from scratch”

1. Try to understand every lines of the code below (which computes the negative log-likelihood for a logistic regression model)

nllik <- function(par, features, response){## nllik stands for Negative Log Likelihood
  ## par is the vector of parameters, i.e., beta_0, beta_1 and beta_3 here
  ## features is the data.frame storing features
  ## response is the observed response (binary response here)
  
  dsgn.mat <- cbind(1, features)
  log.odds <- dsgn.mat %*% par
  odds <- exp(log.odds)
  proba <- odds / (1 + odds)
  
  return(-sum(log1p(-proba) + response * log.odds))
}

## A typical call to the nllik function
nllik(c(1, 0, 0), features = cbind(X1, X2), response = Y)

[1] 46.32617

2. Learn how to use the nlm function to retrieve the maximum likelihood estimate

## Fill in this part

3. Tweak your previous line of code to get standard errors as well

## Fill in this part

4. Have a look at the function glm and use it to fit our logistic model.

## Fill in this part

c) Analysis using the dedicated R function

1. Read the help file of the glm function.
2. Fit the corresponding logistic regression model using the glm function.

## Fill in this part

4. Run the following code and comment

predict(fit)##fit is the output of the previous glm call, I'm here to explain each plot ;-)
predict(fit, type = "response")

5. Give the odds for each feature

## Fill in this part

2. Redoing the Heart Disease in South Africa analysis

We move a bit further by analyzing the Heart Disease in South Africa analysis of the lecture. The dataset is available from the here. Try to redo the analysis we did during the lecture.

3. Analysing the Default data set

In this section we will analyze the Default dataset (available from here). The main goal of this study is to predict whether an individual will default on his or her credit card payment, on the basis of annual income, monthly credit card balance and if the individual is a student or not.

1. Familiarize yourself with the data and produce some (useful!) exploratory plots and summary statistics
2. Fit a first logistic regression model and analyze your fitted model.
3. According to your fitted model, is a student less or more likely to default (for a fixed value of balance and income)? Is this in agreement with your investigations on question 1?
4. What is the probability of default for a student having a credit card balance of 650$ and an income of 17000$?