Disclaimer: MedCo is still an experimental software under development and should not, at this point, use real sensitive data.

Resources

• MedCo software repositories

• Other resources

• Deprecated repositories

  • (forked from )

Contact

For assistance with deploying MedCo or any other technical questions, send an email at or any of the contributors.

• (Privacy and Security Software Engineer, EPFL) -

• (Privacy and Security Software Engineer, EPFL) -

• (Privacy and Security Software Engineer, EPFL) -

License

MedCo is licensed under a End User Software License Agreement (‘EULA’) for non-commercial use. If you need more information, please contact us.

Profile test-local-3nodes

This test profile deploys 3 MedCo nodes on a single machine for test purposes. It can be used either on your local machine, or any other machine to which you have access. The version of the docker images used are the latest released versions. This profile is for example used for the MedCo public demo.

MedCo Stack Deployment

First step is to get the MedCo Deployment latest release.

Next step is to download the docker images:

The default configuration of the deployment is suitable if the stack is deployed on your local host, and if you do not need to modify the default passwords. If so, edit the file ~/medco-deployment/compose-profiles/test-local-3nodes/.env to reflect your configuration. For example:

MEDCO_NODE_URL should be the fully qualified domain name of the host, HTTP_SCHEME should be http or https. The other fields control the default passwords for the various services running. Note that setting the passwords that way works only on the first deployment. If the passwords need to be updated later, you should use the specific component way of modifying password.

Follow to set up the certificates needed for HTTPS. If you are deploying on another host than the local host without HTTPS take note of the following: .

Final step is to run the nodes, all three will run simultaneously:

Wait some time for the initialization of the containers to be done (up to the message: “i2b2-medco-srv… - Started x of y services (z services are lazy, passive or on-demand)”), this can take up to 10 minutes. For the subsequent runs, the startup will be faster. In order to stop the containers, hit Ctrl+C in the active window.

You can use the command docker-compose up -d instead to run MedCo in the background and thus not keeping the console captive. In that case use docker-compose stop to stop the containers.

Keycloak Configuration

Follow the instructions from to be able to use Glowing Bear.

Test the deployment

In order to test that the local test deployment of MedCo is working, access Glowing Bear in your web browser at http(s)://<domain name> and use the credentials previously configured during the . If you are new to Glowing Bear you can watch the video.

By default MedCo loads a specific test data, refer to for expected results to queries. To load a dataset, follow the guide . For reference, the database address (host) to use during loading is <domain name>:5432 and the databases i2b2medcosrv0, i2b2medcosrv1 and i2b2medcosrv2.

v0.2.1 - 15 Aug. 2019

• Implementation of additional query types (patient list, locally obfuscated count, global count, shuffled count)

• Implementation of OIDC-based authorization model

• Implementation of CLI client

• Timers improvements

• Upgrade of dependencies

• Various smaller improvements and bug fixes

v0.2.0 - 3rd May 2019

• Architecture revisit

• Replaced medco-i2b2-cell by medco-connector

• Upgrades (IRCT v1.4 to PICSURE v2.0, Onet suite v3, Keycloak, Nginx, PHP)

v0.1.1 - 23rd Jan. 2019

• Deployment for test purposes on several machines

• Enhancements of documentation and deployment infrastructure

• Nginx reverse proxy with HTTPS support

v0.1.0 - 1st Dec. 2018

First public release of MedCo, running with i2b2 v1.7, PIC-SURE/IRCT v1.4 and centralized OpenID Connect authentication. Deployment for development and test purpose on a single machine.

• Local Test Deployment

  • MedCo Stack Deployment

These pages explain how to deploy MedCo in different scenarios. Each deployment scenario corresponds to a deployment profile, as described below. All these instructions use the deployment scripts from the repository.

If you are new to MedCo…

… and want to try to deploy the system on a single machine to test it, you should should follow the guide.

… and want to create or join a MedCo network, you should follow the guide.

… and want to develop around MedCo, you should follow the guide.

Deployment Profiles

A deployment profile is composed of two things:

• a compose profile in ~/medco-deployment/compose-profiles/<profile name>/: docker-compose file and parameters like ports to expose, log level, etc.

• a configuration profile in ~/medco-deployment/configuration-profiles/<profile name>/: files mounted in the docker containers, containing the cryptographic keys, the certificates, etc.

Some profiles are provided by default, for development or testing purposes. Those should not be used in a production scenario with real data, as the private keys are set by default, thus not private. Other types of profiles must generated using the scripts in ~/medco-deployment/resources/profile-generation-scripts/<profile name>/.

The different profiles are the following:

• test-local-3nodes ()

  • for test on a single machine (used by the )

  • 3 nodes on any host

The database is pre-loaded with some encrypted test data using a key that is pre-generated from the combination of all the participating nodes’ public keys. For the test-network deployment profile this data will not be correctly encrypted, since the public key of each node is generated independently, and, as such, the data must be re-loaded.

Profile dev-local-3nodes

This deployment profile deploys 3 MedCo nodes on a single machine for development purposes. It is meant to be used only on your local machine, i.e. localhost. The tags of the docker images used are all dev, i.e. the ones built from the development version of the different source codes. They are available either through Docker Hub, or built locally.

MedCo Stack Deployment (except Glowing Bear)

First step is to clone the medco-deployment repository with the correct branch. This example gets the data in the home directory of the current user, but that can be changed.

Next step is to build the docker images:

Note that instead of building the dev docker images locally, it is possible to download them from Docker Hub by using docker-compose pull, but there is no guarantee on the current status of those images are they are automatically built.

Next step is to run the nodes. They will run simultaneously, and the logs of the running containers will maintain the console captive. No configuration changes are needed in this scenario before running the nodes. To run them:

Wait some time for the initialization of the containers to be done (up to the message: “i2b2-medco-srv… - Started x of y services (z services are lazy, passive or on-demand)”), this can take up to 10 minutes. For the subsequent runs, the startup will be faster.

Glowing Bear Deployment

First step is to clone the glowing-bear repository with the correct branch.

Glowing Bear is deployed separately for development, as we use its convenient live development server:

Note that the first run will take a significant time in order to build everything.

In order to stop the containers, simply hit Ctrl+C in all the active windows.

Keycloak Configuration

Follow the instructions from to be able to use Glowing Bear.

Test the deployment

In order to test that the development deployment of MedCo is working, access Glowing Bear in your web browser at http://localhost:4200 and use the credentials previously configured during the . If you are new to Glowing Bear you can watch the video.

By default MedCo loads a specific test data, refer to for expected results to queries. To load a dataset, follow the guide . For reference, the database address (host) to use during loading is localhost:5432 and the databases i2b2medcosrv0, i2b2medcosrv1 and i2b2medcosrv2.

Profile test-network

This test profile deploys an arbitrary set of MedCo nodes independently in different machines that together form a MedCo network. This deployment assumes each node is deployed in a single dedicated machine. All the machines have to be reachable between each other. Nodes should agree on a network name and individual indexes beforehand (to be assigned a unique ID). The node with index 0 is the central node, which is the only one running Glowing Bear, PICSURE and Keycloak.

The next set of steps must be executed individually by each node of the network.

Preliminaries

First step is to get the MedCo Deployment latest release at each node.

Generation of the Deployment Profile

Next the compose and configuration profiles must be generated using a script, executed in two steps.

• Step 1: each node generates its keys and certificates, and shares its public information with the other nodes

• Step 2: each node collects the public keys and certificates of the all the other nodes

For step 1, the network name should be common to all the nodes. <node DNS name> corresponds to the machine domain name where the node is being deployed. As mentioned before the different parties should have agreed beforehand on the members of the network, and assigned an index to each different node to construct its UID (starting from 0, to n-1, n being the total number of nodes). Remember that node 0 is the central node.

This script will generate the compose profile and part of the configuration profile, including a file srv<node index>-public.tar.gz. This file should be shared with the other nodes, and all of them need to place it in their configuration profile folder (~/medco-deployment/configuration-profiles/test-network-<network name>-node<node index>).

Once all nodes have shared their srv<node index>-public.tar.gz file with all other nodes, step 2 can be executed:

At this point, it is possible to edit the default configuration generated in ~/medco-deployment/configuration-profiles/test-network-<network name>-node<node index>/.env. This is needed if you want to modify the default passwords. When editing this file, be careful to change only the passwords and not the other values. Note that setting the passwords that way works only on the first deployment. If the passwords need to be updated later, you should use the specific component way of modifying password.

The deployment profile is now ready to be used.

MedCo Stack Deployment

Next step is to download the docker images and run the node. The process is different for the central node and for the other nodes. If you manage the central node run the following:

If you manage a node other than the central one (index > 0), run the following:

Wait some time for the initialization of the containers to be done, this can take up to 10 minutes. For the subsequent runs, the startup will be faster. You can use docker-compose -f docker-compose... stop to stop the containers.

Keycloak Configuration

Follow the instructions from and then you should be able to login in Glowing Bear.

Data Loading

Contrary to the other deployment profiles the default test data will not be working (the queries made will fail) since the data is not encrypted with the collective key that was generated (encryption key derived from all the nodes’ public keys). Run the MedCo loader (see ) to be able to test this deployment. For reference, the database address (host) to use during loading is <domain name>:5432 and the database i2b2medco.

Test the deployment

In order to test that the network deployment of MedCo is working, access Glowing Bear in your web browser at http://<node domain name> and use the credentials previously configured during the . If you are new to Glowing Bear you can watch the video.

Note that by default the certificates generated by the script are self-signed and thus, when using Glowing Bear, the browser will issue a security warning. To use your own valid certificates, see .

HTTPS is supported for the profiles test-local-3nodes and test-network.

Certificate

The certificates are held in the configuration profile folder (e.g, ~/medco-deployment/configuration-profiles/test-local-3nodes):

• certificate.key: private key

• certificate.crt: certificate of own node

• srv0-certificate.crt, srv1-certificate.crt, …: certificates of all nodes of the network

Enable HTTPS for the Test Local Deployment

To enable HTTPS for the profile test-local-3nodes, replace the files certificate.key and certificate.crt from the configuration profile folder with your own versions. Such a certificate can be obtained for example through .

Then edit the file .env from the compose profile, replace the http with https, and restart the deployment.

Configure HTTPS for the Test Network Deployment

For this profile, HTTPS is mandatory. The profile generation scripts generates and use default self-signed certificates for each node. Those are perfectly fine to be used, but because they are self-signed, an HTTPS warning will be displayed to users in their browser when accessing Glowing Bear. There are two ways of avoiding this warning:

• Configuring the browsers of your users to trust this certificate. This procedure is specific to the browsers and operating systems used at your site.

• Use a certificate obtained by an authority trusted by the browser you are using: see below.

If you wish to use a certificate from your own making, gather its key and the certificate itself. Note that using your own certificate is only needed on the central node (as it is the one hosting the web application Glowing Bear). In the configuration profile of the central node (~/medco-deployment/configuration-profiles/test-network-<network name>-node<node index>/) copy the certificate and its key in the respective files certificate.crt and certificate.key. Then duplicate the file certificate.crt in srv0-certificate.crt. Restart the deployment and the central node configuration is ready.

Now the other nodes need to get this certificate to trust it. Get and copy the srv0-certificate.crt file into each of the configuration profile directory of the other nodes, and restart all the deployments. The configuration of HTTPS is now ready.

This guide explains the deployment and configuration of MedCo instances.

• Specifications

• Deployment

Configuration

• Keycloak Configuration

Here follows some MedCo-specific instructions for the administration of Keycloak. For anything else, please refer to the .

Accessing the web administration interface

In the case of the development profile dev-local-3nodes (i.e. without reverse proxy), the address is http://localhost:8081/auth/admin

We recommend the following specifications for running MedCo:

• Network Bandwidth: >100 Mbps (ideal), >10 Mbps (minimum), symmetrical

• Ports Opening and IP Restrictions: see

Administration with PgAdmin

PgAdmin can be accessed through http://<node domain name>/pgadmin with username admin and password admin (by default). To access the test database just create a server with the name MedCo, the address postgresql, username postgres and password postgres (by default).

Prerequisites

• A MedCo network is up and running, with one or more functional Keycloak within the network.

The v0 loader expects an ontology, with mutation and clinical data in the MAF format. As the ontology data you must use ~/medco-deployment/resources/data/genomic/tcga_cbio/clinical_data.csv and ~/medco-deployment/resources/data/genomic/tcga_cbio/mutation_data.csv. For clinical data you can keep using the same two files or a subset of the data (e.g. 8_clinical_data.csv). More information about how to generate sample datafiles can be found below. After the following script is executed all the data is encrypted and ‘deterministically tagged’ in compliance with the MedCo data model.

Example

The following example allows to load data into a running MedCo development deployment (dev-local-3nodes), on the node 0. Adapt accordingly the docker-compose service being ran to load the two other nodes of this profile.

Explanation of the arguments:

Data Manipulation

Inside ~/medco-loader/data/scripts/ you can find a small python application to extract (or replicate) data out of the original tcga_cbio dataset. You can decide which patients you want to consider for you ‘new’ dataset or simply randomly pick a sample.

To check that it is working you can query for:

-> MedCo Gemomic Ontology -> Gene Name -> BRPF3

For the small dataset 8_xxxx you should obtain 3 matching subjects (one at each site).

The v1 loader expects an already existing i2b2 database (in .csv format) that will be converted in a way that is compliant with the MedCo data model. This involves encrypting and ‘deterministically tagging’ some of the data.

List of input (‘original’) files:

• all i2b2metadata files (e.g. i2b2.csv)

• dummy_to_patient.csv

• patient_dimension.csv

• visit_dimension.csv

• concept_dimension.csv

• modifier_dimension.csv

• observation_fact.csv

• table_access.csv

Dummy Generation

The provided example data set files come with dummy data pre-generated. Those data are random dummy entries whose purpose is to prevent frequency attacks. For more information on how this dummy generation is done please refer to ~/medco-loader/data/scripts/import-tool/report/report.pdf. In a future release, the generation will be done dynamically by the loader.

Example

The following example allows to load data into a running MedCo development deployment (dev-local-3nodes), on the node 0. Adapt accordingly the docker-compose service being ran to load the two other nodes of this profile.

Explanation of the arguments:

To check that it is working you can query for:

-> Diagnoses -> Neoplasm -> Benign neoplasm -> Benign neoplasm of breast

You should obtain 2 matching subjects.

• v0 (Genomic Data)

  • Example

The current version offers two different loading alternatives: (v0) loading of clinical and genomic data based on MAF datasets; and (v1) loading of generic i2b2 data. Currently these two loaders support each one dataset:

• v0: a genomic dataset (tcga_cbio publicly available in )

• v1: the .

Future releases of this software will allow for other arbitrary data sources, given that they follow a specific structure (e.g. BAM format).

Pre-Requisites

Download test data

From the medco-deployment folder, execute the resources/data/download.sh script to download the test datasets.

Containers

medco-connector

External Entities

Entities that need to connect to a machine running MedCo can be categorized as follow:

The default data loaded in MedCo is a small artificially generated dataset, appropriate to test a fresh deployment. The same data is replicated on all the nodes. Note that as of now it is encrypted using the test keys, i.e. the ones used for deployment profiles dev-local-3nodes and test-local-3nodes.

It contains 4 patients: 1 (real), 2 (real), 3 (real), 4 (dummy).

And 3 concepts: 1, 2, and 3.

The observation fact contains the following entries:

MedCo provides a client command-line interface (CLI) to interact with the medco-connector APIs.

Prerequisites

To use the CLI, you must first follow one of the . However, the version of the CLI documented here is the one shipped with the .

• System Architecture

  • Containers

If you are interested in developing around MedCo, the first thing you might want to do is to follow the guide to set up the development version of MedCo.