ROC-GLM for DataSHIELD

THE PACKAGE HAS MOVED TO THE dsBinVal REPOSITORY!

The package provides functionality to conduct and visualize ROC analysis
on decentralized data. The basis is the
DataSHIELD](https://www.datashield.org) infrastructure for
distributed computing. This package provides the calculation of the
ROC-GLM as well as
AUC confidence
intervals. In order to
calculate the ROC-GLM it is necessry to push models and predict them at
the servers. This is done automatically by the base package
dsPredictBase. Note
that DataSHIELD uses an option datashield.privacyLevel to indicate the
minimal amount of numbers required to be allowed to share an aggregated
value of these numbers. Instead of setting the option, we directly
retrieve the privacy level from the
DESCRIPTION
file each time a function calls for it. This options is set to 5 by
default.

Installation

At the moment, there is no CRAN version available. Install the
development version from GitHub:

remotes::install_github("difuture-lmu/dsROCGLM")

Register methods

It is necessary to register the assign and aggregate methods in the OPAL
administration. These methods are registered automatically when
publishing the package on OPAL (see
DESCRIPTION).

Note that the package needs to be installed at both locations, the
server and the analysts machine.

Usage

A more sophisticated example is available
here.

library(DSI)
#> Loading required package: progress
#> Loading required package: R6
library(DSOpal)
#> Loading required package: opalr
#> Loading required package: httr
library(dsBaseClient)
library(dsPredictBase)
library(dsROCGLM)

Log into DataSHIELD server

builder = newDSLoginBuilder()
surl     = "https://opal-demo.obiba.org/"
username = "administrator"
password = "password"
builder$append(
  server   = "ds1",
  url      = surl,
  user     = username,
  password = password
)
builder$append(
  server   = "ds2",
  url      = surl,
  user     = username,
  password = password
)
connections = datashield.login(logins = builder$build(), assign = TRUE)
#>
#> Logging into the collaborating servers

Assign iris and validation vector at DataSHIELD (just for testing)

datashield.assign(connections, "iris", quote(iris))
datashield.assign(connections, "y", quote(c(rep(1, 50), rep(0, 100))))

Load test model, push to DataSHIELD, and calculate predictions

# Model predicts if species of iris is setosa or not.
iris$y = ifelse(iris$Species == "setosa", 1, 0)
mod = glm(y ~ Sepal.Length, data = iris, family = binomial())
# Push the model to the DataSHIELD servers using `dsPredictBase`:
pushObject(connections, mod)
# Calculate scores and save at the servers using `dsPredictBase`:
pfun =  "predict(mod, newdata = D, type = 'response')"
predictModel(connections, mod, "pred", "iris", predict_fun = pfun)
datashield.symbols(connections)
#> $ds1
#> [1] "iris" "mod"  "pred" "y"
#>
#> $ds2
#> [1] "iris" "mod"  "pred" "y"

Calculate l2-sensitivity

# In order to securely calculate the ROC-GLM, we have to assess the
# l2-sensitivity to set the privacy parameters of differential
# privacy adequately:
l2s = dsL2Sens(connections, "iris", "pred")
l2s
#> [1] 0.1280699
# Due to the results presented in https://arxiv.org/abs/2203.10828, we set the privacy parameters to
# - epsilon = 0.2, delta = 0.1 if i      l2s <= 0.01
# - epsilon = 0.3, delta = 0.4 if 0.01 < l2s <= 0.03
# - epsilon = 0.5, delta = 0.3 if 0.03 < l2s <= 0.05
# - epsilon = 0.5, delta = 0.5 if 0.05 < l2s <= 0.07
# - epsilon = 0.5, delta = 0.5 if 0.07 < l2s BUT results may be not good!

Calculate ROC-GLM

roc_glm = dsROCGLM(connections, truth_name = "y", pred_name = "pred",
  dat_name = "iris", seed_object = "y")
#>
#> [2022-04-04 12:47:46] L2 sensitivity is: 0.1281
#> Warning in dsROCGLM(connections, truth_name = "y", pred_name = "pred", dat_name
#> = "iris", : l2-sensitivity may be too high for good results! Epsilon = 0.5 and
#> delta = 0.5 is used which may lead to bad results.
#>
#> [2022-04-04 12:47:47] Setting: epsilon = 0.5 and delta = 0.5
#>
#> [2022-04-04 12:47:47] Initializing ROC-GLM
#>
#> [2022-04-04 12:47:47] Host: Received scores of negative response
#> [2022-04-04 12:47:47] Receiving negative scores
#> [2022-04-04 12:47:49] Host: Pushing pooled scores
#> [2022-04-04 12:47:50] Server: Calculating placement values and parts for ROC-GLM
#> [2022-04-04 12:47:52] Server: Calculating probit regression to obtain ROC-GLM
#> [2022-04-04 12:47:53] Deviance of iter1=137.2431
#> [2022-04-04 12:47:54] Deviance of iter2=121.5994
#> [2022-04-04 12:47:56] Deviance of iter3=147.7237
#> [2022-04-04 12:47:57] Deviance of iter4=140.4008
#> [2022-04-04 12:47:58] Deviance of iter5=129.2244
#> [2022-04-04 12:48:00] Deviance of iter6=123.9979
#> [2022-04-04 12:48:01] Deviance of iter7=123.1971
#> [2022-04-04 12:48:02] Deviance of iter8=124.1615
#> [2022-04-04 12:48:04] Deviance of iter9=124.5356
#> [2022-04-04 12:48:05] Deviance of iter10=124.5503
#> [2022-04-04 12:48:06] Deviance of iter11=124.5504
#> [2022-04-04 12:48:08] Deviance of iter12=124.5504
#> [2022-04-04 12:48:08] Host: Finished calculating ROC-GLM
#> [2022-04-04 12:48:08] Host: Cleaning data on server
#> [2022-04-04 12:48:09] Host: Calculating AUC and CI
#> [2022-04-04 12:48:18] Finished!
roc_glm
#>
#> ROC-GLM after Pepe:
#>
#>  Binormal form: pnorm(2.51 + 1.55*qnorm(t))
#>
#>  AUC and 0.95 CI: [0.86----0.91----0.95]
plot(roc_glm)

Deploy information:

Build by root (Darwin) on 2022-04-04 12:48:23.

This readme is built automatically after each push to the repository.
Hence, it also is a test if the functionality of the package works also
on the DataSHIELD servers. We also test these functionality in
tests/testthat/test_on_active_server.R. The system information of the
local and remote servers are as followed:

• Local machine:
  • R version: R version 4.1.3 (2022-03-10)
  • Version of DataSHELD client packages:

• Remote DataSHIELD machines:
  • R version of ds1: R version 4.1.1 (2021-08-10)
  • R version of ds2: R version 4.1.1 (2021-08-10)
  • Version of server packages: