---
title: "Predictive Modelling -- Logistic Regression and Random Forests"
output: html_document
---

```{r setup, include=FALSE}
knitr::opts_chunk$set(echo = TRUE)
knitr::opts_chunk$set(cache=TRUE)
```

```{r echo=T, message=F}
library(ggplot2)
library(boot)
library(randomForest)
#setwd("/media/guerzhoy/Windows/STA303/projects/p2")
#setwd("/media/guerzhoy/Windows/STA303/lectures/lec11")
setwd("c:/STA303/lectures/lec11")
```

Let's read in the data and rename the columns and values to something more readable data (note: you didn't have to rename the values.)

```{r}
german <- read.csv("german.data", sep = " ", header=F)

attr_info <- "Attribute 1: (qualitative) 
AccountStatus
A11 : ... < 0 DM 
A12 : 0 <= ... < 200 DM 
A13 : ... >= 200 DM / salary assignments for at least 1 year 
A14 : no checking account 

Attribute 2: (numerical) 
Duration

Attribute 3: (qualitative) 
CreditHistory
A30 : no credits taken/ all credits paid back duly 
A31 : all credits at this bank paid back duly 
A32 : existing credits paid back duly till now 
A33 : delay in paying off in the past 
A34 : critical account/ other credits existing (not at this bank) 

Attribute 4: (qualitative) 
Purpose
A40 : car (new) 
A41 : car (used) 
A42 : furniture/equipment 
A43 : radio/television 
A44 : domestic appliances 
A45 : repairs 
A46 : education 
A48 : retraining 
A49 : business 
A410 : others 

Attribute 5: (numerical) 
CreditAmount

Attribute 6: (qualitative) 
Account
A61 : ... < 100 DM 
A62 : 100 <= ... < 500 DM 
A63 : 500 <= ... < 1000 DM 
A64 : .. >= 1000 DM 
A65 : unknown/ no savings account 

Attribute 7: (qualitative) 
EmploymentSince
A71 : unemployed 
A72 : ... < 1 year 
A73 : 1 <= ... < 4 years 
A74 : 4 <= ... < 7 years 
A75 : .. >= 7 years 

Attribute 8: (numerical) 
PrecentOfIncome

Attribute 9: (qualitative) 
PersonalStatus
A91 : male : divorced/separated 
A92 : female : divorced/separated/married 
A93 : male : single 
A94 : male : married/widowed 
A95 : female : single 

Attribute 10: (qualitative) 
OtherDebtors
A101 : none 
A102 : co-applicant 
A103 : guarantor 

Attribute 11: (numerical) 
ResidenceSince

Attribute 12: (qualitative) 
Property
A121 : real estate 
A122 : if not A121 : building society savings agreement/ life insurance 
A123 : if not A121/A122 : car or other, not in attribute 6 
A124 : unknown / no property 

Attribute 13: (numerical) 
Age

Attribute 14: (qualitative) 
OtherInstallPlans
A141 : bank 
A142 : stores 
A143 : none 

Attribute 15: (qualitative) 
Housing
A151 : rent 
A152 : own 
A153 : for free 

Attribute 16: (numerical) 
NumExistingCredits

Attribute 17: (qualitative) 
Job
A171 : unemployed/ unskilled - non-resident
A172 : unskilled - resident
A173 : skilled employee / official
A174 : management/ self-employed/highly qualified employee/ officer

Attribute 18: (numerical) 
NumMaintenance

Attribute 19: (qualitative) 
Telephone
A191 : none 
A192 : yes, registered under the customers name 

Attribute 20: (qualitative) 
ForeignWorker
A201 : yes 
A202 : no 

Attribute 21: (quantitative)
GoodCredit
1: 1
2: 0
"

attr_list <- as.list(as.vector(strsplit(attr_info, "Attribute ")[[1]])[2:22])

attr_names <- sapply(attr_list, function(attr_desc) strsplit(attr_desc, "\n")[[1]][2])
colnames(german) <- attr_names

for(i in 1:21){
  attr_desc <- attr_list[[i]]
  if(length(grep("qualitative",attr_desc))>0){
    lines_all <- as.vector(strsplit(attr_desc, "\n")[[1]])
    lines <- lines_all[3:(length(lines_all)-1)]
    str_vals <- sapply(lines, function(line) strsplit(line, " : ")[[1]][2])
    german[,i] <-as.numeric(german[,i])
    
    for (val_i in 1:length(lines)){
      german[,i] <- ifelse(german[,i]==val_i, str_vals[val_i], german[,i])
    }
  }
}
```

Here are the first three lines of `german` now:

```{r}
german[c(1, 2, 3),]
```

Now, let's set things up so that Good Credit is 1, and Bad Credit is 0.

```{r}
german[, "GoodCredit"] <- ifelse(german[, "GoodCredit"]==1, 1, 0)
```

## Logistic Regression Model

First, let's fit the full model (but with no variable transformations/interactions to see which variables seem to matter a lot

```{r}

summary(glm(GoodCredit~., family=binomial, data=german))
```

A quick and dirty way to fit the model is to use the variables which are significant.

Let's split the data into a training and a test set.

```{r}
set.seed(0)
idx <- sample(1000, 800)
train_set <- german[idx,]
test_set <- german[-idx,]
```


```{r}
cost_classification <- function(r, pi) mean(abs(r-pi) > 0.5)

model1 <- glm(GoodCredit~AccountStatus+Duration+CreditHistory+CreditAmount+Account+EmploymentSince+PrecentOfIncome+PersonalStatus+OtherDebtors+ForeignWorker, family=binomial, data=train_set)
cost_classification(train_set$GoodCredit, predict(model1, train_set, type="response"))
cost_classification(test_set$GoodCredit, predict(model1, test_set, type="response"))
```

It doesn't look like there is overfitting (in fact, the classification cost is a little smaller on the test set, just due to random variation). Let's see how much better we can do on the training set if we include all the variables.

```{r}
cost_classification(train_set$GoodCredit, predict(glm(GoodCredit~., family=binomial, data=train_set), train_set, type="response"))
```

So it seems like we can't do that much better.

Now, let's fit a Random Forest model.

```{r}
library(randomForest)
credit.rf <- randomForest(as.factor(train_set$GoodCredit)~as.factor(train_set$AccountStatus)+train_set$Duration+as.factor(train_set$CreditHistory)+train_set$CreditAmount+as.factor(train_set$Account)+as.factor(train_set$EmploymentSince)+train_set$PrecentOfIncome+as.factor(train_set$PersonalStatus)+as.factor(train_set$OtherDebtors)+as.factor(train_set$ForeignWorker), ntrees=10000)


```

Let's now evaluate the costs.

```{r}
train.pred.rf <- ifelse(as.character(predict(credit.rf, type="response"))=="1", 1, 0)
cost_classification(train_set$GoodCredit, train.pred.rf)

test.pred.rf <- ifelse(as.character(predict(credit.rf, new_data=test_set, type="response"))=="1", 1, 0)
cost_classification(test_set$GoodCredit, test.pred.rf)
```

In this case, we are not doing as well as logistic regression on the training set, and random forest overfit pretty badly. Random Forests generally do perform pretty well though!

Finally, let's try some transformations of the variables and see where that gets us.

```{r}
model2 <- glm(GoodCredit~AccountStatus+Duration+I(Duration^2)+I(log(1+Duration))+CreditHistory+CreditAmount+I(CreditAmount^2)+I(log(CreditAmount))+Account+EmploymentSince+PrecentOfIncome+I(PrecentOfIncome^2)+I(log(PrecentOfIncome))+PersonalStatus+OtherDebtors+ForeignWorker, family=binomial, data=train_set)
cost_classification(train_set$GoodCredit, predict(model2, train_set, type="response"))
cost_classification(test_set$GoodCredit, predict(model2, test_set, type="response"))
```

Looks like we're doing even better!
