Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Profile test-network
This test profile deploys an arbitrary set of MedCo nodes independently in different machines that together form a MedCo network. This deployment assumes each node is deployed in a single dedicated machine. All the machines have to be reachable between each other. Nodes should agree on a network name and individual indexes beforehand (to be assigned a unique ID). The node with index 0 is the central node, which is the only one running Glowing Bear, PICSURE and Keycloak.
The next set of steps must be executed individually by each node of the network.
First step is to get the MedCo Deployment latest release at each node.
Next the compose and configuration profiles must be generated using a script, executed in two steps.
Step 1: each node generates its keys and certificates, and shares its public information with the other nodes
Step 2: each node collects the public keys and certificates of the all the other nodes
For step 1, the network name should be common to all the nodes. <node DNS name> corresponds to the machine domain name where the node is being deployed. As mentioned before the different parties should have agreed beforehand on the members of the network, and assigned an index to each different node to construct its UID (starting from 0, to n-1, n being the total number of nodes). Remember that node 0 is the central node.
This script will generate the compose profile and part of the configuration profile, including a file srv<node index>-public.tar.gz. This file should be shared with the other nodes, and all of them need to place it in their configuration profile folder (~/medco-deployment/configuration-profiles/test-network-<network name>-node<node index>).
Once all nodes have shared their srv<node index>-public.tar.gz file with all other nodes, step 2 can be executed:
At this point, it is possible to edit the default configuration generated in ~/medco-deployment/configuration-profiles/test-network-<network name>-node<node index>/.env. This is needed if you want to modify the default passwords. When editing this file, be careful to change only the passwords and not the other values. Note that setting the passwords that way works only on the first deployment. If the passwords need to be updated later, you should use the specific component way of modifying password.
The deployment profile is now ready to be used.
Next step is to download the docker images and run the node. The process is different for the central node and for the other nodes. If you manage the central node run the following:
If you manage a node other than the central one (index > 0), run the following:
Wait some time for the initialization of the containers to be done, this can take up to 10 minutes. For the subsequent runs, the startup will be faster. You can use docker-compose -f docker-compose... stop to stop the containers.
Follow the instructions from Keycloak Configuration and then you should be able to login in Glowing Bear.
Contrary to the other deployment profiles the default test data will not be working (the queries made will fail) since the data is not encrypted with the collective key that was generated (encryption key derived from all the nodes’ public keys). Run the MedCo loader (see Loading Data) to be able to test this deployment. For reference, the database address (host) to use during loading is <domain name>:5432 and the database i2b2medco.
In order to test that the network deployment of MedCo is working, access Glowing Bear in your web browser at http://<node domain name> and use the credentials previously configured during the Keycloak Configuration. If you are new to Glowing Bear you can watch the Glowing Bear user interface walkthrough video.
Note that by default the certificates generated by the script are self-signed and thus, when using Glowing Bear, the browser will issue a security warning. To use your own valid certificates, see HTTPS Configuration.
We recommend the following specifications for running MedCo:
Network Bandwidth: >100 Mbps (ideal), >10 Mbps (minimum), symmetrical
Ports Opening and IP Restrictions: see Network Architecture
Hardware
CPU: 8 cores (ideal), 4 cores (minimum)
RAM: >16 GB (ideal), >8GB (minimum)
Storage: dependent on data loaded, >100GB
Software
OS: Any flavor of Linux, physical or virtualized (tested with Ubuntu 16.04, 18.04, Fedora 29)
Softwares: OpenSSL, Docker (tested with Docker 18.09.1) & Docker-Compose (tested with Docker-Compose 1.23.2), Git and Git-LFS
Profile test-local-3nodes
This test profile deploys 3 MedCo nodes on a single machine for test purposes. It can be used either on your local machine, or any other machine to which you have access. The version of the docker images used are the latest released versions. This profile is for example used for the .
First step is to get the MedCo Deployment latest release.
Next step is to download the docker images:
The default configuration of the deployment is suitable if the stack is deployed on your local host, and if you do not need to modify the default passwords. If so, edit the file ~/medco-deployment/compose-profiles/test-local-3nodes/.env to reflect your configuration. For example:
MEDCO_NODE_URL should be the fully qualified domain name of the host, HTTP_SCHEME should be http or https. The other fields control the default passwords for the various services running. Note that setting the passwords that way works only on the first deployment. If the passwords need to be updated later, you should use the specific component way of modifying password.
Final step is to run the nodes, all three will run simultaneously:
Wait some time for the initialization of the containers to be done (up to the message: “i2b2-medco-srv… - Started x of y services (z services are lazy, passive or on-demand)”), this can take up to 10 minutes. For the subsequent runs, the startup will be faster. In order to stop the containers, hit Ctrl+C in the active window.
You can use the command docker-compose up -d instead to run MedCo in the background and thus not keeping the console captive. In that case use docker-compose stop to stop the containers.
Follow to set up the certificates needed for HTTPS. If you are deploying on another host than the local host without HTTPS take note of the following: .
Follow the instructions from to be able to use Glowing Bear.
In order to test that the local test deployment of MedCo is working, access Glowing Bear in your web browser at http(s)://<domain name> and use the credentials previously configured during the . If you are new to Glowing Bear you can watch the video.
By default MedCo loads a specific test data, refer to for expected results to queries. To load a dataset, follow the guide . For reference, the database address (host) to use during loading is <domain name>:5432 and the databases i2b2medcosrv0, i2b2medcosrv1 and i2b2medcosrv2.
Here follows some MedCo-specific instructions for the administration of Keycloak. For anything else, please refer to the Keycloak Server Administration Guide.
In the case of the development profile dev-local-3nodes (i.e. without reverse proxy), the address is http://localhost:8081/auth/admin. In the other cases (with the reverse proxy), the address is http://<node domain name>/auth/admin. The credentials are :
User keycloak
Password keycloak by default, or whatever else was configured at the initial deployment.
When deploying the test-local-3nodes profile without HTTPS on a machine other than localhost, the administration interface will refuse to load. To solve this, access pgAdmin (see The PostgreSQL database) and execute the following SQL on the keycloak database:
You need to restart the Keycloak docker container to enable the changes.
Import the provided realm configuration into Keycloak. This will create the MedCo client with the appropriate roles.
Go to the Import menu
Click on Select file and select the file keycloak-medco-realm.json that you will find in ~/medco-deployment/resources/configuration.
Select to import everything, and to Skip if resources already exist
In the Settings tab, fill Valid Redirect URIs according to the following table:
Deployment Profile
Valid Redirect URIs
test-local-3nodes
http(s)://<node domain name>/glowing-bear
test-network
https://<node domain name>/glowing-bear
dev-local-3nodes
http://localhost:4200
In the same tab, fill Web Origins with + and save.
Add a user
Go to the configuration panel Users, click on Add user.
Fill the Username field, toggle to ON the Email Verified button and click Save.
In the next window, click on Credentials, enter twice the user’s password, toggle to OFF the Temporary button if desired and click Reset Password.
Give query permissions to a user
Go to the configuration panel Users, search for the user you want to give authorization to and click on Edit.
Go to the Role Mappings tab, and select medco (or another client ID set up for the MedCo OIDC client) in the Client Roles.
Add the roles you wish to give the user, each of the roles maps to a query type.
These pages explain how to deploy MedCo in different scenarios. Each deployment scenario corresponds to a deployment profile, as described below. All these instructions use the deployment scripts from the medco-deployment repository.
If you are new to MedCo…
… and want to try to deploy the system on a single machine to test it, you should should follow the Local Test Deployment guide.
… and want to create or join a MedCo network, you should follow the Network Test Deployment guide.
… and want to develop around MedCo, you should follow the Local Development Deployment guide.
A deployment profile is composed of two things:
a compose profile in ~/medco-deployment/compose-profiles/<profile name>/: docker-compose file and parameters like ports to expose, log level, etc.
a configuration profile in ~/medco-deployment/configuration-profiles/<profile name>/: files mounted in the docker containers, containing the cryptographic keys, the certificates, etc.
Some profiles are provided by default, for development or testing purposes. Those should not be used in a production scenario with real data, as the private keys are set by default, thus not private. Other types of profiles must generated using the scripts in ~/medco-deployment/resources/profile-generation-scripts/<profile name>/.
The different profiles are the following:
test-local-3nodes (Local Test Deployment)
for test on a single machine (used by the MedCo live demo)
3 nodes on any host
using the latest release of the source codes
no debug logging
profile pre-generated
test-network (Network Test Deployment)
for test on several different hosts
a single node on a host part of a MedCo network
using the latest release of the source codes
no debug logging
profile must be generated prior to use with the provided scripts
dev-local-3nodes (Local Development Deployment)
for software development
3 nodes on the local host
using development version of source codes
debug logging enabled
profile pre-generated
The database is pre-loaded with some encrypted test data using a key that is pre-generated from the combination of all the participating nodes’ public keys. For the test-network deployment profile this data will not be correctly encrypted, since the public key of each node is generated independently, and, as such, the data must be re-loaded.
This guide walks you through the process of configuring Keycloak as a Service Provider to one or more SwitchAAI identity provider(s), in order for MedCo to rely on SwitchAAI for user authentication.
A MedCo network is up and running, with one or more functional Keycloak within the network.
One or several identity provider(s) part of the SwitchAAI federation is/are chosen to be used as user source.
The institution at which the Keycloak of MedCo is deployed is ready to accept being registered as the home organization.
You have access to the .
Note that right now the SwitchAAI WAYF (Where Are You From) mechanism is not supported (i.e. the web UI used to select with institution the user wishes to login). This means that you will need to register in Keycloak each identity provider you wish to support.
Note also that the process described in this guide will need to be repeated for each instance of Keycloak deployed, if there are more than one in the MedCo network.
The following instructions are to be executed on the administration UI of Keycloak, e.g. https://medco-demo.epfl.ch/auth/admin.
The behavior of Keycloak during the very login of users through the identity provider is highly customisable. We propose below an example of a working flow but this can be changed to fit your need.
Navigate to Authentication > Flows, select First Broker Login and make a Copy of it. Name it for example SwitchAAI-Test Demo IdP First Broker Login.
Change the list of executions to make it look like the following image.
In the Identity Providers menu, choose Add provider... > SAML v2.0
Specify an Alias. Note this will not be changeable later without redoing the whole process. Example: SwitchAAI-Test.
Specify a Display Name, which will be displayed to the user in the login page. Example: SwitchAAI-Test Demo IdP.
Specify the Single Sign-On Service URL of the identity provider you are linking with. Example: https://aai-demo-idp.switch.ch/idp/profile/SAML2/POST/SSO.
Specify the First Login Flow previously configured to use. Example: SwitchAAI-Test Demo IdP First Broker Login.
Toggle to ON the following buttons:
Enabled
Trust Email
HTTP-POST Binding Response
HTTP-POST Binding for AuthnRequest
Validate Signature
Specify the NameID Policy Format as Persistent.
Add the certificate(s) (PEM format, separated by commas if there are several of them) of the identity provider you are linking with in Validating X509 Certificates.
Save the changes.
We need to import a unique but intelligible username in Keycloak from the identity provider. For this we use the SwitchAAI mandatory attribute swissEduPersonUniqueID.
Open the Mappers tab and click Create.
Fill the field as:
Name: SwitchAAI Unique ID.
Mapper Type: Username Template Importer.
Template: ${ATTRIBUTE.swissEduPersonUniqueID}
Save the changes.
Its FQDN (fully-qualified domain name). Example: medco-demo.epfl.ch.
Its SAML entityID, that you can find out in the XML descriptor from the Export tab of the previously configured Keycloak identity provider. Example: https://medco-demo.epfl.ch/auth/realms/master.
Once you have generated the certificate, set it up in Keycloak:
Navigate to the settings page Realm Settings > Keys > Providers and select Add Keystore... > rsa.
Specify a name in Console Display Name. Example: rsa-switchaaitest.
Specify a Priority higher than any other RSA key. Example: 150.
In Private RSA Key and X509 Certificate fields, copy/paste the respective PEM parts of both the private key and the certificate that were previously generated.
Click Add a Resource Description and fill the 7 categories of information according to the following instructions. Note that if some fields are not listed in this documentation, their value are not important for the registration of the Keycloak instance and can be set according to the explanations provided by the resource registry.
Entity ID: the same SAML entityID you used to generate the certificate. Example: https://medco-demo.epfl.ch/auth/realms/master.
Home Organization: the organization that hosts the Keycloak instance currently being registered. The responsible persons of the organization specified here will need to approve the registration. This will typically be the the institution where the MedCo node is deployed. For the purpose of our test we are using AAI Demo Home Organization (aai-demo-idp.switch.ch, AAI Test).
Home URL: the address of the MedCo node, at which the UI Glowing Bear can be accessed. Example: https://medco-demo.epfl.ch/.
SAML2 HTTP POST binding (x2): the URL at witch the SwitchAAI infrastructure will communicate with the Keycloak instance. You will find it in the configuration page of the configured identity provider in Keycloak under Redirect URI. Example: https://medco-demo.epfl.ch/auth/realms/master/broker/SwitchAAI-Test/endpoint
Copy/paste in this field the PEM part of the certificate that was previously generated. Note that in the example showed below the certificate has already been validated through a separate channel.
Put on Required at least the following attributes. Note that the release of attributes needs to have a justification.
E-mail (email). Example reason: Identify user for being able to assign them specific authorizations.
Unique ID (swissEduPersonUniqueID). Example reason: Get a unique ID of user.
Once submitted, the responsible persons from the home organization will need to approve the new resource and validate the fingerprint of the certificate submitted. This is a manual process that will most likely be done through email.
Once this is done, the setup should be functional, and the users will be able to select the configured identity provider to login. Don't forget that this covers only users' authentication, their authorization needs to be handled manually through Keycloak after they login at least once.
The v1 loader expects an already existing i2b2 database (in .csv format) that will be converted in a way that is compliant with the MedCo data model. This involves encrypting and ‘deterministically tagging’ some of the data.
List of input (‘original’) files:
all i2b2metadata files (e.g. i2b2.csv)
dummy_to_patient.csv
patient_dimension.csv
visit_dimension.csv
concept_dimension.csv
modifier_dimension.csv
observation_fact.csv
table_access.csv
The provided example data set files come with dummy data pre-generated. Those data are random dummy entries whose purpose is to prevent frequency attacks. For more information on how this dummy generation is done please refer to ~/medco-loader/data/scripts/import-tool/report/report.pdf. In a future release, the generation will be done dynamically by the loader.
The following example allows to load data into a running MedCo development deployment (dev-local-3nodes), on the node 0. Adapt accordingly the docker-compose service being ran to load the two other nodes of this profile.
Explanation of the arguments:
To check that it is working you can query for:
-> Diagnoses -> Neoplasm -> Benign neoplasm -> Benign neoplasm of breast
You should obtain 2 matching subjects.
A certificate compliant with the SwitchAAI federation needs to be generated and configured. First to generate a self-signed certificate that meets their requirements. You will need from the Keycloak instance:
The following instructions are to be executed in the . As a result, a Keycloak instance will be registered as a service provider linked to a home organization in the SwitchAAI federation.
MedCo provides a client command-line interface (CLI) to interact with the medco-connector APIs.
To use the CLI, you must first follow one of the deployment guides. However, the version of the CLI documented here is the one shipped with the Local Development Deployment.
To show the CLI manual, run:
For a start, you can use the credentials of the default user: username:test password:test
You will also need to specify the URL of the medco connector node you want to interact with by properly setting the MEDCO_CONNECTOR_URL variable in compose-profiles/docker-compose-definitions.yml
You can use this command to query the MedCo network.
This is the syntax of an example query using the pre-loaded default testa data.
You will get something like that:
Not that, in the queries, the OR operator has the highest priority, so 1 AND 2 OR 3 AND 2 is factorised as (1) AND (2 OR 3) AND (2).
You can use this command to get the values of the genomic annotations that MedCo nodes make available for queries.
To do some tests, you may want to load some data first.
Then, for example, if you want to know which genomic annotations of type "protein_change" containing the string "g32" are available, you can run:
You will get:
The matching is case-insensitive and it is not possible to use wildcards. At the moment only three types of genomic annotations are available: variant_name, protein_change and hugo_gene_symbol.
You can use this command to get the variant ID of a certain genomic annotation.
To do some tests, you may want to load some data first.
Then, for example, if you want to know the variant ID of the genomic annotation "HTR5A" of type "hugo_gene_symbol" with zygosity "heterozygous" or "homozygous", you can run:
You will get:
The matching is case-insensitive and it is not possible to use wildcards. If you request the ID of an annotation which is not available (e.g, in the previous, example, "HTR5") you will get an error message. At the moment only three types of genomic annotations are available: variant_name, protein_change and hugo_gene_symbol.
HTTPS is supported for the profiles test-local-3nodes and test-network.
The certificates are held in the configuration profile folder (e.g, ~/medco-deployment/configuration-profiles/test-local-3nodes):
certificate.key: private key
certificate.crt: certificate of own node
srv0-certificate.crt, srv1-certificate.crt, …: certificates of all nodes of the network
To enable HTTPS for the profile test-local-3nodes, replace the files certificate.key and certificate.crt from the configuration profile folder with your own versions. Such a certificate can be obtained for example through Let’s Encrypt.
Then edit the file .env from the compose profile, replace the http with https, and restart the deployment.
For this profile, HTTPS is mandatory. The profile generation scripts generates and use default self-signed certificates for each node. Those are perfectly fine to be used, but because they are self-signed, an HTTPS warning will be displayed to users in their browser when accessing Glowing Bear. There are two ways of avoiding this warning:
Configuring the browsers of your users to trust this certificate. This procedure is specific to the browsers and operating systems used at your site.
Use a certificate obtained by an authority trusted by the browser you are using: see below.
If you wish to use a certificate from your own making, gather its key and the certificate itself. Note that using your own certificate is only needed on the central node (as it is the one hosting the web application Glowing Bear). In the configuration profile of the central node (~/medco-deployment/configuration-profiles/test-network-<network name>-node<node index>/) copy the certificate and its key in the respective files certificate.crt and certificate.key. Then duplicate the file certificate.crt in srv0-certificate.crt. Restart the deployment and the central node configuration is ready.
Now the other nodes need to get this certificate to trust it. Get and copy the srv0-certificate.crt file into each of the configuration profile directory of the other nodes, and restart all the deployments. The configuration of HTTPS is now ready.
The v0 loader expects an ontology, with mutation and clinical data in the MAF format. As the ontology data you must use ~/medco-deployment/resources/data/genomic/tcga_cbio/clinical_data.csv and ~/medco-deployment/resources/data/genomic/tcga_cbio/mutation_data.csv. For clinical data you can keep using the same two files or a subset of the data (e.g. 8_clinical_data.csv). More information about how to generate sample datafiles can be found below. After the following script is executed all the data is encrypted and ‘deterministically tagged’ in compliance with the MedCo data model.
The following example allows to load data into a running MedCo development deployment (dev-local-3nodes), on the node 0. Adapt accordingly the docker-compose service being ran to load the two other nodes of this profile.
Explanation of the arguments:
Inside ~/medco-loader/data/scripts/ you can find a small python application to extract (or replicate) data out of the original tcga_cbio dataset. You can decide which patients you want to consider for you ‘new’ dataset or simply randomly pick a sample.
To check that it is working you can query for:
-> MedCo Gemomic Ontology -> Gene Name -> BRPF3
For the small dataset 8_xxxx you should obtain 3 matching subjects (one at each site).
Profile dev-local-3nodes
This deployment profile deploys 3 MedCo nodes on a single machine for development purposes. It is meant to be used only on your local machine, i.e. localhost. The tags of the docker images used are all dev, i.e. the ones built from the development version of the different source codes. They are available either through Docker Hub, or built locally.
First step is to clone the medco-deployment repository with the correct branch. This example gets the data in the home directory of the current user, but that can be changed.
Next step is to build the docker images:
Note that instead of building the dev docker images locally, it is possible to download them from Docker Hub by using docker-compose pull, but there is no guarantee on the current status of those images are they are automatically built.
Next step is to run the nodes. They will run simultaneously, and the logs of the running containers will maintain the console captive. No configuration changes are needed in this scenario before running the nodes. To run them:
Wait some time for the initialization of the containers to be done (up to the message: “i2b2-medco-srv… - Started x of y services (z services are lazy, passive or on-demand)”), this can take up to 10 minutes. For the subsequent runs, the startup will be faster.
First step is to clone the glowing-bear repository with the correct branch.
Glowing Bear is deployed separately for development, as we use its convenient live development server:
Note that the first run will take a significant time in order to build everything.
In order to stop the containers, simply hit Ctrl+C in all the active windows.
Follow the instructions from Keycloak Configuration to be able to use Glowing Bear.
In order to test that the development deployment of MedCo is working, access Glowing Bear in your web browser at http://localhost:4200 and use the credentials previously configured during the Keycloak Configuration. If you are new to Glowing Bear you can watch the Glowing Bear user interface walkthrough video.
By default MedCo loads a specific test data, refer to Description of the default test data for expected results to queries. To load a dataset, follow the guide Loading Data. For reference, the database address (host) to use during loading is localhost:5432 and the databases i2b2medcosrv0, i2b2medcosrv1 and i2b2medcosrv2.
The current version offers two different loading alternatives: (v0) loading of clinical and genomic data based on MAF datasets; and (v1) loading of generic i2b2 data. Currently these two loaders support each one dataset:
v0: a genomic dataset (tcga_cbio publicly available in cBioPortal)
v1: the i2b2 demodata.
Future releases of this software will allow for other arbitrary data sources, given that they follow a specific structure (e.g. BAM format).
Download test data
From the medco-deployment folder, execute the resources/data/download.sh script to download the test datasets.
Entities that need to connect to a machine running MedCo can be categorized as follow:
System administrators: Persons administrating the MedCo node. Likely to remain inside the clinical site internal network.
End-users: Researchers using MedCo to access the shared. Likely to remain inside the clinical site internal network.
Other MedCo nodes: MedCo nodes belonging to other clinical sites of the network.
The following ports should be accessible by the listed entities, which makes IP address white-listing possible:
Port 22, 5432 (TCP): System Administrators
Port 80 (TCP): End-Users (HTTP automatic redirect to HTTPS (443))
Port 443 (TCP): System Administrators, End-Users, Other MedCo Nodes
Ports 2000-2001 (TCP): Other MedCo Nodes
This guide explains the deployment and configuration of MedCo instances.
