1. Indirect

The TPR and TNR can be indirectly calculated by using the values in the confusion matrix.

# extract all modelId
model_id<-List_Model3 %>% filter(!is.na(`Weighted AUC.crossValidation`)) %>% pull(modelId)

However, the API was unable to obtain the confusion matrix.

(GetModel(project_ID, model_id[1]) %>% GetConfusionChart(source = DataPartition$VALIDATION))

(GetModel(project_ID, model_id[1]) %>% GetConfusionChart(source = DataPartition$CROSSVALIDATION))

(GetModel(project_ID, model_id[9]) %>% GetConfusionChart(source = DataPartition$VALIDATION))

(GetModel(project_ID, model_id[9]) %>% GetConfusionChart(source = DataPartition$CROSSVALIDATION))

## [1] "Error: (404) No confusion chart for validation"

## [1] "Error: (404) No confusion chart for validation"

## [1] "Error: (404) No confusion chart for validation"

## [1] "Error: (404) No confusion chart for validation"

2. Direct way

The TPR and TNR is directly printed on the Selection Summary under the ROC option in the DataRobot GUI.

The same can be accessed via the GetRocCurve function in the API.

# loop to get all TNR and TPR from ROC Curve option. https://stackoverflow.com/questions/29402528/append-data-frames-together-in-a-for-loop
TPR_TNR<-data.frame()

for (i in model_id){ 
selection_summary<-GetModel(project_ID, i) %>% GetRocCurve(source = DataPartition$CROSSVALIDATION)

temp<-selection_summary$rocPoints%>% 
  filter(f1Score==max(f1Score)) %>% select(truePositiveRate, trueNegativeRate) %>% 
  mutate(modelId= i)

TPR_TNR<-bind_rows(TPR_TNR, temp)
}

TPR_TNR %>% head()

## # A tibble: 6 x 3
##   truePositiveRate trueNegativeRate modelId                 
##              <dbl>            <dbl> <chr>                   
## 1            0.724            0.903 5f564d6cdf91a0559ffbfb8c
## 2            0.621            0.931 5f54de5d269d39249ea468a3
## 3            0.770            0.876 5f54de5d269d39249ea468a2
## 4            0.649            0.936 5f54de5d269d39249ea468a4
## 5            0.787            0.886 5f54de5d269d39249ea468a7
## 6            0.736            0.906 5f564d880a859d0d17d859c8

Updating main dataframe

TPR_TNR was joined with the main dataframe, List_Model3, to allow all metrics to be in a common dataframe.

List_Model3<-List_Model3 %>% left_join(TPR_TNR, by = "modelId") %>%
  # select columns of interest
  select(modelType, expandedModel, modelId, blueprintId, featurelistName, featurelistId, `Weighted AUC.crossValidation`, TPR=truePositiveRate, TNR=trueNegativeRate) %>% mutate(`Weighted AUC.crossValidation`= as.double(`Weighted AUC.crossValidation`))

## Warning in mask$eval_all_mutate(dots[[i]]): NAs introduced by coercion

LightGBT

Description of the deployed Light GBT

# modelId of Light GBT 
LightGBT_Id <-deploy_results %>% filter(str_detect(modelType, "Light")) %>% pull(modelId)
#blueprint
LightGBT_blueprint<-GetModelBlueprintDocumentation(project_ID, LightGBT_Id) %>% data.frame()

 (glue::glue ("Description of the deployed model:: \n{LightGBT_blueprint$description}"))

## Description of the deployed model:: 
## The Classifier uses the LightGBM implementation of Gradient Boosted Trees.
## LightGBM is a gradient boosting framework designed to be distributed and efficient with the following advantages:
## Gradient Boosting Machines (or Gradient Boosted Trees, depending on who you ask to explain the acronym ‘GBM’) are a cutting-edge algorithm for fitting extremely accurate predictive models.  GBMs have won a number of recent predictive modeling competitions and are considered among many Data Scientists to be the most versatile and useful predictive modeling algorithm.  GBMs require very little preprocessing, elegantly handle missing data, strike a good balance between bias and variance and are typically able to find complicated interaction terms, which makes them a useful “swiss army knife” of predictive models. GBMs are a generalization of Freund and Schapire’s adaboost algorithm (1995) to handle arbitrary loss functions.  They are very similar in concept to random forests, in that they fit individual decision trees to random re-samples of the input data, where each tree sees a boostrap sample of the rows of a the dataset and N arbitrarily chosen columns where N is a configural parameter of the model.  GBMs differ from random forests in a single major aspect: rather than fitting the trees in parallel, the GBM fits each successive tree to the residual errors from all the previous trees combined. This is advantageous, as the model focuses each iteration on the examples that are most difficult to predict (and therefore most useful to get correct). Due to their iterative nature, GBMs are almost guaranteed to overfit the training data, given enough iterations.  The 2 critial parameters of the algorithm; therefore, are the learning rate (or how fast the model fits the data) and the number of trees the model is allowed to fit.  It is critical to cross-validate one of these 2 parameters.
## Early stopping is used to determine the best number of trees where overfitting begins. In this manner GBMs are usually capable of squeezing every last bit of information out of the training set and producing the model with the highest possible accuracy.

Hyper-parameters which were tuned::

### tuned values
# list within a list 
para<-GetModel(project_ID, LightGBT_Id) %>% GetTuningParameters()

# option 1 to covert to df
para_df<-summary(para) 

# option 2 to convert to df2
#paraName<- list()
#value<-list()

#for(i in 1:18){
#  paraName_temp<-para[["tuningParameters"]][[i]][["parameterName"]] %>% data.frame()
#  paraName[[i]]<-paraName_temp
  
#  value_temp<-para[["tuningParameters"]][[i]][["currentValue"]] %>% data.frame()
#  value[[i]]<-value_temp
#}

#paraName<-bind_rows(paraName)

#value_df<-data.frame()
#for (i in 1:length(value)){
#  temp<- value[[i]] %>% data.frame()
#  value_df<-rbind(temp, value_df)
#}

#bind_cols(paraName, value_df)

### parameters description
para_describe<-LightGBT_blueprint %>% 
  # as num ID to para w no ID (e.g. parameters.type-> parameters.type.0)
  rename_with(.fn=~paste0(.x, ".0") , .cols=matches("parameter(.*)[^0-9]$")) %>%
  # select only parameters var
  select(starts_with("parameter")) %>% 
  # parameters.type.0-> type.0 
  rename_with(.fn=~str_replace(.x, "parameters.", "")) %>% 
  # type.0-> type_parameter0. 
  ##need [] to encapsulate full stop, list of permitted char
  rename_with(.fn=~str_replace(.x, "[.]", "_parameter")) %>% 
  # https://tidyr.tidyverse.org/articles/pivot.html#multiple-observations-per-row-1
  pivot_longer(cols=everything(), names_to = c(".value", "parameter"), names_sep = "_") %>% 
  # parameter10-> 10
  ## sep the alphabets not numbers, convert to int
  separate(parameter, into = c(NA, "para"), sep = "[a-z]+", convert = T) %>%
 # remove parthesis and abbreviation for name
   mutate(name= str_remove(name, "\\(.*\\)") %>% str_trim()) 

There is an extra tuned parameter in para_df which is the type of classification. This extra parameter was ignored.

anti_join(para_df, para_describe, by="name") 

## # A tibble: 1 x 4
##   name      current default constraint                                    
##   <chr>     <chr>   <chr>   <chr>                                         
## 1 objective binary  binary  select from: binary, multiclass, multiclassova

Description of the hyper-parameters and the tuned values

left_join(para_describe, para_df, by="name") %>% select(parameter=name, description, limits=constraint, default_value=default, tuned_values=current)  %>% DT::datatable(rownames = F, options = list(searchHighlight = TRUE, paging= T))