Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
We recommend the following specifications for running MedCo:
Network Bandwidth: >100 Mbps (ideal), >10 Mbps (minimum), symmetrical
Ports Opening and IP Restrictions: see Network Architecture
Hardware
CPU: 8 cores (ideal), 4 cores (minimum)
RAM: >16 GB (ideal), >8GB (minimum)
Storage: dependent on data loaded, >100GB
Software
OS: Any flavor of Linux, physical or virtualized (tested with Ubuntu 16.04, 18.04, Fedora 29)
Softwares: OpenSSL, Docker (tested with Docker 18.09.1) & Docker-Compose (tested with Docker-Compose 1.23.2), Git and Git-LFS
This guide explains the deployment and configuration of MedCo instances.
HTTPS is supported for the profiles test-local-3nodes and test-network.
The certificates are held in the configuration profile folder (e.g, ~/medco-deployment/configuration-profiles/test-local-3nodes):
certificate.key: private key
certificate.crt: certificate of own node
srv0-certificate.crt, srv1-certificate.crt, …: certificates of all nodes of the network
To enable HTTPS for the profile test-local-3nodes, replace the files certificate.key and certificate.crt from the configuration profile folder with your own versions. Such a certificate can be obtained for example through Let’s Encrypt.
Then edit the file .env from the compose profile, replace the http with https, and restart the deployment.
For this profile, HTTPS is mandatory. The profile generation scripts generates and use default self-signed certificates for each node. Those are perfectly fine to be used, but because they are self-signed, an HTTPS warning will be displayed to users in their browser when accessing Glowing Bear. There are two ways of avoiding this warning:
Configuring the browsers of your users to trust this certificate. This procedure is specific to the browsers and operating systems used at your site.
Use a certificate obtained by an authority trusted by the browser you are using: see below.
If you wish to use a certificate from your own making, gather its key and the certificate itself. Note that using your own certificate is only needed on the central node (as it is the one hosting the web application Glowing Bear). In the configuration profile of the central node (~/medco-deployment/configuration-profiles/test-network-<network name>-node<node index>/) copy the certificate and its key in the respective files certificate.crt and certificate.key. Then duplicate the file certificate.crt in srv0-certificate.crt. Restart the deployment and the central node configuration is ready.
Now the other nodes need to get this certificate to trust it. Get and copy the srv0-certificate.crt file into each of the configuration profile directory of the other nodes, and restart all the deployments. The configuration of HTTPS is now ready.
Implementation of additional query types (patient list, locally obfuscated count, global count, shuffled count)
Implementation of OIDC-based authorization model
Implementation of CLI client
Timers improvements
Upgrade of dependencies
Various smaller improvements and bug fixes
Architecture revisit
Replaced medco-i2b2-cell by medco-connector
Upgrades (IRCT v1.4 to PICSURE v2.0, Onet suite v3, Keycloak, Nginx, PHP)
Consolidation of deployment
Many smaller fixes and enhancements
Deployment for test purposes on several machines
Enhancements of documentation and deployment infrastructure
Nginx reverse proxy with HTTPS support
Keycloak update
First public release of MedCo, running with i2b2 v1.7, PIC-SURE/IRCT v1.4 and centralized OpenID Connect authentication. Deployment for development and test purpose on a single machine.
These pages explain how to deploy MedCo in different scenarios. Each deployment scenario corresponds to a deployment profile, as described below. All these instructions use the deployment scripts from the medco-deployment repository.
If you are new to MedCo…
… and want to try to deploy the system on a single machine to test it, you should should follow the Local Test Deployment guide.
… and want to create or join a MedCo network, you should follow the Network Test Deployment guide.
… and want to develop around MedCo, you should follow the Local Development Deployment guide.
A deployment profile is composed of two things:
a compose profile in ~/medco-deployment/compose-profiles/<profile name>/: docker-compose file and parameters like ports to expose, log level, etc.
a configuration profile in ~/medco-deployment/configuration-profiles/<profile name>/: files mounted in the docker containers, containing the cryptographic keys, the certificates, etc.
Some profiles are provided by default, for development or testing purposes. Those should not be used in a production scenario with real data, as the private keys are set by default, thus not private. Other types of profiles must generated using the scripts in ~/medco-deployment/resources/profile-generation-scripts/<profile name>/.
The different profiles are the following:
test-local-3nodes (Local Test Deployment)
for test on a single machine (used by the MedCo live demo)
3 nodes on any host
using the latest release of the source codes
no debug logging
profile pre-generated
test-network (Network Test Deployment)
for test on several different hosts
a single node on a host part of a MedCo network
using the latest release of the source codes
no debug logging
profile must be generated prior to use with the provided scripts
dev-local-3nodes (Local Development Deployment)
for software development
3 nodes on the local host
using development version of source codes
debug logging enabled
profile pre-generated
The database is pre-loaded with some encrypted test data using a key that is pre-generated from the combination of all the participating nodes’ public keys. For the test-network deployment profile this data will not be correctly encrypted, since the public key of each node is generated independently, and, as such, the data must be re-loaded.
Disclaimer: MedCo is still an experimental software under development and should not, at this point, use real sensitive data.
MedCo software repositories
Glowing Bear MedCo (forked from The Hyve)
Other resources
Deprecated repositories
For assistance with deploying MedCo or any other technical questions, send an email at medco@epfl.ch or any of the contributors.
Mickaël Misbach (Privacy and Security Software Engineer, EPFL) - mickael.misbach@epfl.ch
Francesco Marino (Privacy and Security Software Engineer, EPFL) - francesco.marino@epfl.ch
Joao Andre Sa (Privacy and Security Software Engineer, EPFL) - joao.gomesdesaesousa@epfl.ch
Jean Louis Raisaro (Data Protection Specialist, CHUV) - jean.raisaro@chuv.ch
Juan Troncoso-Pastoriza (Post-Doctoral Researcher, EPFL) - juan.troncoso-pastoriza@epfl.ch
Jean-Pierre Hubaux (Professor, EPFL) - jean-pierre.hubaux@epfl.ch
MedCo is licensed under a End User Software License Agreement (‘EULA’) for non-commercial use. If you need more information, please contact us.
Profile dev-local-3nodes
This deployment profile deploys 3 MedCo nodes on a single machine for development purposes. It is meant to be used only on your local machine, i.e. localhost. The tags of the docker images used are all dev, i.e. the ones built from the development version of the different source codes. They are available either through Docker Hub, or built locally.
First step is to clone the medco-deployment repository with the correct branch. This example gets the data in the home directory of the current user, but that can be changed.
$ cd ~
$ git clone -b dev https://github.com/ldsec/medco-deployment.gitNext step is to build the docker images:
$ cd ~/medco-deployment/compose-profiles/dev-local-3nodes
$ docker-compose buildNote that instead of building the dev docker images locally, it is possible to download them from Docker Hub by using docker-compose pull, but there is no guarantee on the current status of those images are they are automatically built.
Next step is to run the nodes. They will run simultaneously, and the logs of the running containers will maintain the console captive. No configuration changes are needed in this scenario before running the nodes. To run them:
$ docker-compose upWait some time for the initialization of the containers to be done (up to the message: “i2b2-medco-srv… - Started x of y services (z services are lazy, passive or on-demand)”), this can take up to 10 minutes. For the subsequent runs, the startup will be faster.
First step is to clone the glowing-bear repository with the correct branch.
$ cd ~
$ git clone -b dev https://github.com/ldsec/glowing-bear-medco.gitGlowing Bear is deployed separately for development, as we use its convenient live development server:
$ cd ~/glowing-bear-medco/deployment
$ docker-compose up dev-serverNote that the first run will take a significant time in order to build everything.
In order to stop the containers, simply hit Ctrl+C in all the active windows.
Follow the instructions from Keycloak Configuration to be able to use Glowing Bear.
In order to test that the development deployment of MedCo is working, access Glowing Bear in your web browser at http://localhost:4200 and use the credentials previously configured during the Keycloak Configuration. If you are new to Glowing Bear you can watch the Glowing Bear user interface walkthrough video.
By default MedCo loads a specific test data, refer to Description of the default test data for expected results to queries. To load a dataset, follow the guide Loading Data. For reference, the database address (host) to use during loading is localhost:5432 and the databases i2b2medcosrv0, i2b2medcosrv1 and i2b2medcosrv2.
Here follows some MedCo-specific instructions for the administration of Keycloak. For anything else, please refer to the Keycloak Server Administration Guide.
In the case of the development profile dev-local-3nodes (i.e. without reverse proxy), the address is http://localhost:8081/auth/admin. In the other cases (with the reverse proxy), the address is http://<node domain name>/auth/admin. The credentials are :
User keycloak
Password keycloak by default, or whatever else was configured at the initial deployment.
When deploying the test-local-3nodes profile without HTTPS on a machine other than localhost, the administration interface will refuse to load. To solve this, access pgAdmin (see The PostgreSQL database) and execute the following SQL on the keycloak database:
update REALM set ssl_required = 'NONE' where id = 'master';You need to restart the Keycloak docker container to enable the changes.
Import the provided realm configuration into Keycloak. This will create the MedCo client with the appropriate roles.
Go to the Import menu
Click on Select file and select the file keycloak-medco-realm.json that you will find in ~/medco-deployment/resources/configuration.
Select to import everything, and to Skip if resources already exist
In the Settings tab, fill Valid Redirect URIs according to the following table:
Deployment Profile
Valid Redirect URIs
test-local-3nodes
http(s)://<node domain name>/glowing-bear
test-network
https://<node domain name>/glowing-bear
dev-local-3nodes
http://localhost:4200
In the same tab, fill Web Origins with + and save.
Add a user
Go to the configuration panel Users, click on Add user.
Fill the Username field, toggle to ON the Email Verified button and click Save.
In the next window, click on Credentials, enter twice the user’s password, toggle to OFF the Temporary button if desired and click Reset Password.
Give query permissions to a user
Go to the configuration panel Users, search for the user you want to give authorization to and click on Edit.
Go to the Role Mappings tab, and select medco (or another client ID set up for the MedCo OIDC client) in the Client Roles.
Add the roles you wish to give the user, each of the roles maps to a query type.
Profile test-network
This test profile deploys an arbitrary set of MedCo nodes independently in different machines that together form a MedCo network. This deployment assumes each node is deployed in a single dedicated machine. All the machines have to be reachable between each other. Nodes should agree on a network name and individual indexes beforehand (to be assigned a unique ID). The node with index 0 is the central node, which is the only one running Glowing Bear, PICSURE and Keycloak.
The next set of steps must be executed individually by each node of the network.
First step is to get the MedCo Deployment latest release at each node.
$ cd ~
$ wget https://github.com/ldsec/medco-deployment/archive/v0.2.1-1.tar.gz
$ tar xvzf v0.2.1-1.tar.gz
$ mv medco-deployment-0.2.1-1 medco-deploymentNext the compose and configuration profiles must be generated using a script, executed in two steps.
Step 1: each node generates its keys and certificates, and shares its public information with the other nodes
Step 2: each node collects the public keys and certificates of the all the other nodes
For step 1, the network name should be common to all the nodes. <node DNS name> corresponds to the machine domain name where the node is being deployed. As mentioned before the different parties should have agreed beforehand on the members of the network, and assigned an index to each different node to construct its UID (starting from 0, to n-1, n being the total number of nodes). Remember that node 0 is the central node.
$ cd ~/medco-deployment/resources/profile-generation-scripts/test-network
$ bash step1.sh <network name> <node index> <node DNS name>This script will generate the compose profile and part of the configuration profile, including a file srv<node index>-public.tar.gz. This file should be shared with the other nodes, and all of them need to place it in their configuration profile folder (~/medco-deployment/configuration-profiles/test-network-<network name>-node<node index>).
Once all nodes have shared their srv<node index>-public.tar.gz file with all other nodes, step 2 can be executed:
$ bash step2.sh <network name> <node index>At this point, it is possible to edit the default configuration generated in ~/medco-deployment/configuration-profiles/test-network-<network name>-node<node index>/.env. This is needed if you want to modify the default passwords. When editing this file, be careful to change only the passwords and not the other values. Note that setting the passwords that way works only on the first deployment. If the passwords need to be updated later, you should use the specific component way of modifying password.
The deployment profile is now ready to be used.
Next step is to download the docker images and run the node. The process is different for the central node and for the other nodes. If you manage the central node run the following:
$ cd ~/medco-deployment/compose-profiles/test-network-<network name>-node0
$ docker-compose -f docker-compose.common.yml -f docker-compose.node.yml -f docker-compose.central.yml pull
$ docker-compose -f docker-compose.common.yml -f docker-compose.node.yml -f docker-compose.central.yml up -dIf you manage a node other than the central one (index > 0), run the following:
$ cd ~/medco-deployment/compose-profiles/test-network-<network name>-node<node index>
$ docker-compose -f docker-compose.common.yml -f docker-compose.node.yml pull
$ docker-compose -f docker-compose.common.yml -f docker-compose.node.yml up -dWait some time for the initialization of the containers to be done, this can take up to 10 minutes. For the subsequent runs, the startup will be faster. You can use docker-compose -f docker-compose... stop to stop the containers.
Follow the instructions from Keycloak Configuration and then you should be able to login in Glowing Bear.
Contrary to the other deployment profiles the default test data will not be working (the queries made will fail) since the data is not encrypted with the collective key that was generated (encryption key derived from all the nodes’ public keys). Run the MedCo loader (see Loading Data) to be able to test this deployment. For reference, the database address (host) to use during loading is <domain name>:5432 and the database i2b2medco.
In order to test that the network deployment of MedCo is working, access Glowing Bear in your web browser at http://<node domain name> and use the credentials previously configured during the Keycloak Configuration. If you are new to Glowing Bear you can watch the Glowing Bear user interface walkthrough video.
Note that by default the certificates generated by the script are self-signed and thus, when using Glowing Bear, the browser will issue a security warning. To use your own valid certificates, see HTTPS Configuration.
PgAdmin can be accessed through http://<node domain name>/pgadmin with username admin and password admin (by default). To access the test database just create a server with the name MedCo, the address postgresql, username postgres and password postgres (by default).
The v0 loader expects an ontology, with mutation and clinical data in the MAF format. As the ontology data you must use ~/medco-deployment/resources/data/genomic/tcga_cbio/clinical_data.csv and ~/medco-deployment/resources/data/genomic/tcga_cbio/mutation_data.csv. For clinical data you can keep using the same two files or a subset of the data (e.g. 8_clinical_data.csv). More information about how to generate sample datafiles can be found below. After the following script is executed all the data is encrypted and ‘deterministically tagged’ in compliance with the MedCo data model.
The following example allows to load data into a running MedCo development deployment (dev-local-3nodes), on the node 0. Adapt accordingly the docker-compose service being ran to load the two other nodes of this profile.
Explanation of the arguments:
Inside ~/medco-loader/data/scripts/ you can find a small python application to extract (or replicate) data out of the original tcga_cbio dataset. You can decide which patients you want to consider for you ‘new’ dataset or simply randomly pick a sample.
To check that it is working you can query for:
-> MedCo Gemomic Ontology -> Gene Name -> BRPF3
For the small dataset 8_xxxx you should obtain 3 matching subjects (one at each site).
The current version offers two different loading alternatives: (v0) loading of clinical and genomic data based on MAF datasets; and (v1) loading of generic i2b2 data. Currently these two loaders support each one dataset:
v0: a genomic dataset (tcga_cbio publicly available in )
v1: the .
Future releases of this software will allow for other arbitrary data sources, given that they follow a specific structure (e.g. BAM format).
Download test data
From the medco-deployment folder, execute the resources/data/download.sh script to download the test datasets.
$ cd ~/medco-deployment/compose-profiles/dev-local-3nodes
$ docker-compose -f docker-compose.tools.yml run medco-loader-srv0 v0 \
--ont_clinical /data/genomic/tcga_cbio/8_clinical_data.csv \
--sen /data/genomic/sensitive.txt \
--ont_genomic /data/genomic/tcga_cbio/8_mutation_data.csv \
--clinical /data/genomic/tcga_cbio/8_clinical_data.csv \
--genomic /data/genomic/tcga_cbio/8_mutation_data.csv \
--output /data/NAME:
medco-loader v0 - Load genomic data (e.g. tcga_bio dataset)
USAGE:
medco-loader v0 [command options] [arguments...]
OPTIONS:
--group value, -g value UnLynx group definition file
--entryPointIdx value, --entry value Index (relative to the group definition file) of the collective authority server to load the data
--sensitive value, --sen value File containing a list of sensitive concepts
--dbHost value, --dbH value Database hostname
--dbPort value, --dbP value Database port (default: 0)
--dbName value, --dbN value Database name
--dbUser value, --dbU value Database user
--dbPassword value, --dbPw value Database password
--ont_clinical value, --oc value Clinical ontology to load
--ont_genomic value, --og value Genomic ontology to load
--clinical value, --cl value Clinical file to load
--genomic value, --gen value Genomic file to load
--output value, -o value Output path to the .csv files$ cd ~/medco-deployment/resources/data
$ bash download.shIf you are interested in developing around MedCo, the first thing you might want to do is to follow the Local Development Deployment guide to set up the development version of MedCo.
MedCo provides a client command-line interface (CLI) to interact with the medco-connector APIs.
To use the CLI, you must first follow one of the deployment guides. However, the version of the CLI documented here is the one shipped with the Local Development Deployment.
To show the CLI manual, run:
cd medco-deployment/compose-profiles/dev-local-3nodes/
docker-compose -f docker-compose.tools.yml run medco-cli-client --user [USERNAME] --password [PASSWORD] --help
NAME:
medco-cli-client - Command-line query tool for MedCo.
USAGE:
medco-cli-client [global options] command [command options] [arguments...]
VERSION:
1.0.0
COMMANDS:
query, q Query the MedCo network
genomic-annotations-get-values, gval Get genomic annotations values
genomic-annotations-get-variants, gvar Get genomic annotations variants
help, h Shows a list of commands or help for one command
GLOBAL OPTIONS:
--user value, -u value OIDC user login
--password value, -p value OIDC password login
--token value, -t value OIDC token
--disableTLSCheck Disable check of TLS certificates
--help, -h show help
--version, -v print the versionYou can use this command to query the MedCo network.
docker-compose -f docker-compose.tools.yml run medco-cli-client --user test --password test query --help
NAME:
medco-cli-client query - Query the MedCo network
USAGE:
medco-cli-client query [command options] [patient_list|count_per_site|count_per_site_obfuscated|count_per_site_shuffled|count_per_site_shuffled_obfuscated|count_global|count_global_obfuscated] [query string]
OPTIONS:
--resultFile value, -r value Output file for the result CSV. Printed to stdout if omitted.This is the syntax of an example query using the pre-loaded default testa data.
docker-compose -f docker-compose.tools.yml run medco-cli-client --user test --password test query patient_list 1 AND 2 OR 3You will get something like that:
node_name,count,patient_list,DDTRequestTime,KSRequestTime,KSTimeCommunication,KSTimeExec,TaggingTimeCommunication,TaggingTimeExec,medco-connector-DDT,medco-connector-i2b2-PDO,medco-connector-i2b2-PSM,medco-connector-local-agg,medco-connector-local-patient-list-masking,medco-connector-overall,medco-connector-unlynx-key-switch-count,medco-connector-unlynx-key-switch-patient-list
0,8,[1 2 3 4 5 6 7 8],268,162,154,0,218,23,275,2561,16507,0,32,19619,50,179
1,0,[],212,61,43,0,179,12,232,2078,17993,0,1,20433,38,87
2,0,[],196,38,30,0,172,7,216,1574,18889,1,2,20779,30,53You can use this command to get the values of the genomic annotations that MedCo nodes make available for queries.
docker-compose -f docker-compose.tools.yml run medco-cli-client --user test --password test genomic-annotations-get-values --help
NAME:
medco-cli-client genomic-annotations-get-values - Get genomic annotations values
USAGE:
medco-cli-client genomic-annotations-get-values [command options] [-l limit] [annotation] [value]
OPTIONS:
--limit value, -l value Maximum number of returned values (default: 0)To do some tests, you may want to load some data first.
Then, for example, if you want to know which genomic annotations of type "protein_change" containing the string "g32" are available, you can run:
docker-compose -f docker-compose.tools.yml run medco-cli-client --user test --password test genomic-annotations-get-values protein_change g32You will get:
G325R
G32EYou can use this command to get the variant ID of a certain genomic annotation.
docker-compose -f docker-compose.tools.yml run medco-cli-client --user test --password test genomic-annotations-get-variants --help
NAME:
medco-cli-client genomic-annotations-get-variants - Get genomic annotations variants
USAGE:
medco-cli-client genomic-annotations-get-variants [command options] [-z zygosity] [-e] [annotation] [value]
DESCRIPTION:
zygosity can be either heterozygous, homozygous, unknown or a combination of the three separated by |
If omitted, the command will execute as if zygosity was equal to "heterozygous|homozygous|unknown|"
OPTIONS:
--zygosity value, -z value Variant zygosysty
--encrypted, -e Return encrypted variant id
To do some tests, you may want to load some data first.
Then, for example, if you want to know the variant ID of the genomic annotation "HTR5A" of type "hugo_gene_symbol" with zygosity "heterozygous" or "homozygous", you can run:
docker-compose -f docker-compose.tools.yml run medco-cli-client --user test --password test genomic-annotations-get-variants -z "heterozygous|homozygous" hugo_gene_symbol HTR5AYou will get:
-7039476204566471680
-7039476580443220992The default data loaded in MedCo is a small artificially generated dataset, appropriate to test a fresh deployment. The same data is replicated on all the nodes. Note that as of now it is encrypted using the test keys, i.e. the ones used for deployment profiles dev-local-3nodes and test-local-3nodes.
It contains 4 patients: 1 (real), 2 (real), 3 (real), 4 (dummy).
And 3 concepts: 1, 2, and 3.
The observation fact contains the following entries:
patient 1, concept 1
patient 2, concept 1
patient 2, concept 2
patient 3, concept 2
patient 3, concept 3
patient 4, concept 1
patient 4, concept 2
patient 4, concept 3
Example queries and expected results:
1 AND 2: 1 (patient 2)
2 AND 3: 1 (patient 3)
1 AND 2 AND 3: 0
(1 OR 2): 3 (patients 1, 2 and 3)
(1 OR 3) AND 2: 1 (patients 2 and 3)
Profile test-local-3nodes
This test profile deploys 3 MedCo nodes on a single machine for test purposes. It can be used either on your local machine, or any other machine to which you have access. The version of the docker images used are the latest released versions. This profile is for example used for the MedCo public demo.
First step is to get the MedCo Deployment latest release.
$ cd ~
$ wget https://github.com/ldsec/medco-deployment/archive/v0.2.1-1.tar.gz
$ tar xvzf v0.2.1-1.tar.gz
$ mv medco-deployment-0.2.1-1 medco-deploymentNext step is to download the docker images:
$ cd ~/medco-deployment/compose-profiles/test-local-3nodes
$ docker-compose pullThe default configuration of the deployment is suitable if the stack is deployed on your local host, and if you do not need to modify the default passwords. If so, edit the file ~/medco-deployment/compose-profiles/test-local-3nodes/.env to reflect your configuration. For example:
MEDCO_NODE_HOST=medco-demo.epfl.ch
HTTP_SCHEME=https
POSTGRES_PASSWORD=postgres1
PGADMIN_PASSWORD=admin
KEYCLOAK_PASSWORD=keycloak
I2B2_WILDFLY_PASSWORD=admin
I2B2_SERVICE_PASSWORD=pFjy3EjDVwLfT2rB9xkK
I2B2_USER_PASSWORD=demouserMEDCO_NODE_URL should be the fully qualified domain name of the host, HTTP_SCHEME should be http or https. The other fields control the default passwords for the various services running. Note that setting the passwords that way works only on the first deployment. If the passwords need to be updated later, you should use the specific component way of modifying password.
Follow HTTPS Configuration to set up the certificates needed for HTTPS. If you are deploying on another host than the local host without HTTPS take note of the following: Disabling HTTPS requirement for external connections.
Final step is to run the nodes, all three will run simultaneously:
$ docker-compose upWait some time for the initialization of the containers to be done (up to the message: “i2b2-medco-srv… - Started x of y services (z services are lazy, passive or on-demand)”), this can take up to 10 minutes. For the subsequent runs, the startup will be faster. In order to stop the containers, hit Ctrl+C in the active window.
You can use the command docker-compose up -d instead to run MedCo in the background and thus not keeping the console captive. In that case use docker-compose stop to stop the containers.
Follow the instructions from Keycloak Configuration to be able to use Glowing Bear.
In order to test that the local test deployment of MedCo is working, access Glowing Bear in your web browser at http(s)://<domain name> and use the credentials previously configured during the Keycloak Configuration. If you are new to Glowing Bear you can watch the Glowing Bear user interface walkthrough video.
By default MedCo loads a specific test data, refer to Description of the default test data for expected results to queries. To load a dataset, follow the guide Loading Data. For reference, the database address (host) to use during loading is <domain name>:5432 and the databases i2b2medcosrv0, i2b2medcosrv1 and i2b2medcosrv2.
This guide walks you through the process of configuring Keycloak as a Service Provider to one or more SwitchAAI identity provider(s), in order for MedCo to rely on SwitchAAI for user authentication.
A MedCo network is up and running, with one or more functional Keycloak within the network.
One or several identity provider(s) part of the SwitchAAI federation is/are chosen to be used as user source.
The institution at which the Keycloak of MedCo is deployed is ready to accept being registered as the home organization.
You have access to the SwitchAAI Resource Registry.
Note that right now the SwitchAAI WAYF (Where Are You From) mechanism is not supported (i.e. the web UI used to select with institution the user wishes to login). This means that you will need to register in Keycloak each identity provider you wish to support.
Note also that the process described in this guide will need to be repeated for each instance of Keycloak deployed, if there are more than one in the MedCo network.
The following instructions are to be executed on the administration UI of Keycloak, e.g. https://medco-demo.epfl.ch/auth/admin.
The behavior of Keycloak during the very login of users through the identity provider is highly customisable. We propose below an example of a working flow but this can be changed to fit your need.
Navigate to Authentication > Flows, select First Broker Login and make a Copy of it. Name it for example SwitchAAI-Test Demo IdP First Broker Login.
Change the list of executions to make it look like the following image.
In the Identity Providers menu, choose Add provider... > SAML v2.0
Specify an Alias. Note this will not be changeable later without redoing the whole process. Example: SwitchAAI-Test.
Specify a Display Name, which will be displayed to the user in the login page. Example: SwitchAAI-Test Demo IdP.
Specify the Single Sign-On Service URL of the identity provider you are linking with. Example: https://aai-demo-idp.switch.ch/idp/profile/SAML2/POST/SSO.
Specify the First Login Flow previously configured to use. Example: SwitchAAI-Test Demo IdP First Broker Login.
Toggle to ON the following buttons:
Enabled
Trust Email
HTTP-POST Binding Response
HTTP-POST Binding for AuthnRequest
Validate Signature
Specify the NameID Policy Format as Persistent.
Add the certificate(s) (PEM format, separated by commas if there are several of them) of the identity provider you are linking with in Validating X509 Certificates.
Save the changes.
We need to import a unique but intelligible username in Keycloak from the identity provider. For this we use the SwitchAAI mandatory attribute swissEduPersonUniqueID.
Open the Mappers tab and click Create.
Fill the field as:
Name: SwitchAAI Unique ID.
Mapper Type: Username Template Importer.
Template: ${ATTRIBUTE.swissEduPersonUniqueID}
Save the changes.
A certificate compliant with the SwitchAAI federation needs to be generated and configured. First follow this SwitchAAI guide to generate a self-signed certificate that meets their requirements. You will need from the Keycloak instance:
Its FQDN (fully-qualified domain name). Example: medco-demo.epfl.ch.
Its SAML entityID, that you can find out in the XML descriptor from the Export tab of the previously configured Keycloak identity provider. Example: https://medco-demo.epfl.ch/auth/realms/master.
Once you have generated the certificate, set it up in Keycloak:
Navigate to the settings page Realm Settings > Keys > Providers and select Add Keystore... > rsa.
Specify a name in Console Display Name. Example: rsa-switchaaitest.
Specify a Priority higher than any other RSA key. Example: 150.
In Private RSA Key and X509 Certificate fields, copy/paste the respective PEM parts of both the private key and the certificate that were previously generated.
The following instructions are to be executed in the AAI Resource Registry. As a result, a Keycloak instance will be registered as a service provider linked to a home organization in the SwitchAAI federation.
Click Add a Resource Description and fill the 7 categories of information according to the following instructions. Note that if some fields are not listed in this documentation, their value are not important for the registration of the Keycloak instance and can be set according to the explanations provided by the resource registry.
Entity ID: the same SAML entityID you used to generate the certificate. Example: https://medco-demo.epfl.ch/auth/realms/master.
Home Organization: the organization that hosts the Keycloak instance currently being registered. The responsible persons of the organization specified here will need to approve the registration. This will typically be the the institution where the MedCo node is deployed. For the purpose of our test we are using AAI Demo Home Organization (aai-demo-idp.switch.ch, AAI Test).
Home URL: the address of the MedCo node, at which the UI Glowing Bear can be accessed. Example: https://medco-demo.epfl.ch/.
SAML2 HTTP POST binding (x2): the URL at witch the SwitchAAI infrastructure will communicate with the Keycloak instance. You will find it in the configuration page of the configured identity provider in Keycloak under Redirect URI. Example: https://medco-demo.epfl.ch/auth/realms/master/broker/SwitchAAI-Test/endpoint
Copy/paste in this field the PEM part of the certificate that was previously generated. Note that in the example showed below the certificate has already been validated through a separate channel.
Put on Required at least the following attributes. Note that the release of attributes needs to have a justification.
E-mail (email). Example reason: Identify user for being able to assign them specific authorizations.
Unique ID (swissEduPersonUniqueID). Example reason: Get a unique ID of user.
Once submitted, the responsible persons from the home organization will need to approve the new resource and validate the fingerprint of the certificate submitted. This is a manual process that will most likely be done through email.
Once this is done, the setup should be functional, and the users will be able to select the configured identity provider to login. Don't forget that this covers only users' authentication, their authorization needs to be handled manually through Keycloak after they login at least once.
The v1 loader expects an already existing i2b2 database (in .csv format) that will be converted in a way that is compliant with the MedCo data model. This involves encrypting and ‘deterministically tagging’ some of the data.
List of input (‘original’) files:
all i2b2metadata files (e.g. i2b2.csv)
dummy_to_patient.csv
patient_dimension.csv
visit_dimension.csv
concept_dimension.csv
modifier_dimension.csv
observation_fact.csv
table_access.csv
The provided example data set files come with dummy data pre-generated. Those data are random dummy entries whose purpose is to prevent frequency attacks. For more information on how this dummy generation is done please refer to ~/medco-loader/data/scripts/import-tool/report/report.pdf. In a future release, the generation will be done dynamically by the loader.
The following example allows to load data into a running MedCo development deployment (dev-local-3nodes), on the node 0. Adapt accordingly the docker-compose service being ran to load the two other nodes of this profile.
$ cd ~/medco-deployment/compose-profiles/dev-local-3nodes
$ docker-compose -f docker-compose.tools.yml run medco-loader-srv0 v1 \
--sen /data/i2b2/sensitive.txt \
--files /data/i2b2/files.tomlExplanation of the arguments:
NAME:
medco-loader v1 - Convert existing i2b2 data model
USAGE:
medco-loader v1 [command options] [arguments...]
OPTIONS:
--group value, -g value UnLynx group definition file
--entryPointIdx value, --entry value Index (relative to the group definition file) of the collective authority server to load the data
--sensitive value, --sen value File containing a list of sensitive concepts
--dbHost value, --dbH value Database hostname
--dbPort value, --dbP value Database port (default: 0)
--dbName value, --dbN value Database name
--dbUser value, --dbU value Database user
--dbPassword value, --dbPw value Database password
--files value, -f value Configuration toml with the path of the all the necessary i2b2 files
--empty, -e Empty patient and visit dimension tables (y/n)To check that it is working you can query for:
-> Diagnoses -> Neoplasm -> Benign neoplasm -> Benign neoplasm of breast
You should obtain 2 matching subjects.
Entities that need to connect to a machine running MedCo can be categorized as follow:
System administrators: Persons administrating the MedCo node. Likely to remain inside the clinical site internal network.
End-users: Researchers using MedCo to access the shared. Likely to remain inside the clinical site internal network.
Other MedCo nodes: MedCo nodes belonging to other clinical sites of the network.
The following ports should be accessible by the listed entities, which makes IP address white-listing possible:
Port 22, 5432 (TCP): System Administrators
Port 80 (TCP): End-Users (HTTP automatic redirect to HTTPS (443))
Port 443 (TCP): System Administrators, End-Users, Other MedCo Nodes
Ports 2000-2001 (TCP): Other MedCo Nodes
Component orchestrating the MedCo query at the clinical site. Implements the resource-side of the PIC-SURE API. It communicates with medco-unlynx to execute the distributed cryptographic protocols.
The software executing the distributed cryptographic protocols, based on Unlynx.
The i2b2 stack (all the cells).
The query translation and broadcasting layer.
Nginx web server serving Glowing Bear and the javascript crypto module.
ETL tool to encrypt and load data into MedCo.
OpenID Connect identity provider, providing user management and their authentication to MedCo.
The SQL database used by all other services, contains all the data.
A web-based administration tool for the PostgreSQL database.
Web server and (HTTPS-enabled) reverse proxy.
PHP processor running with FPM (FastCGI Process Manager), used by Nginx. Executes the PHP code needed to serve the genomic annotations.
