Polites and Broadsea in a Nutshell

Polites is a Java based tool that enables the automation of complete OHDSI implementations for Broadsea based OHDSI deployments.

Broadsea is an amazing tool that enables the reliable deployment of a complete OHDSI implementation repleate with a demo/test Common Data Model (CDM) instance.

However, creating a production instance of a CDM and getting it connected to Broadsea is a lengthy meticulous process.

Polites provides a way to automate this process and includes tools that can be used for the maintenance of existing production deployments.

Creating an Instance of Broadsea

Creating an instance of Broadsea is very easy, very reliable, and requires very few dependencies.

To install Broadsea you will need an instance of Java (a correct version), an instance of Git (a correct version), and an instance of Docker (a correct version)

After the dependencies are installed, installing Docker is as simple as using Git to clone Broadsea from Github and checking out the version you want. After that Broadsea can be started by simply starting Docker from the Broadsea directory using docker-compose –profile default up -d. 

The entire process is shown below.

git config --global core.autocrlf false
git clone https://github.com/OHDSI/Broadsea
cd Broadsea
git checkout v3.0.0
cd Broadsea
docker-compose --profile default up -d

And just like that, we have an OHDSI stack up and running. We can now go and take a look at the demo database provided by Broadsea at http://127.0.0.1/atlas/

What Broadsea Does

Broadsea provides a turn-key solution with unparalleled ease of use and a standardized stable production deployment that includes an instance of a Common Data Model replete with test data in PostgreSql.

What’s Left to do Outside of Broadsea

For virtually every user of Broadsea, there will come a time when there is a need to point to an instance of the CDM other than the demo instance that is included in Broadsea.

This could be a development or test instance that uses a different data set. This could be an instance that exists in a different DBMS such as SQL Server, PostgreSql, Databricks, Redshift, etc., or this could be the CDM for your production data.

Setting up this other CDM instance can be a lengthy meticulous process that for the most part occurs outside of Broadsea.

This process has been automated by Polites.

An Example

As an example, lets look at the steps required to manually set up a CDM instance in SQL Server and get Broadsea to point to it.

Install Basic Software

  • Install Java (the correct version)
  • Install a Git client (a correct version)
  • Install Docker (a correct version)
  • Before we clone and check out Broadsea, remember that we need to use git config –global core.autocrlf false
  • Git clone Broadsea
  • Git checkout the version of Broadsea we want to use
  • Clear Docker of any previous version of Broadsea and possibly other application that may exist if this is not a new install of Docker
  • Add postgresql.conf:
    • shared_buffers = 32GB
    • work_mem = 32GB
    • effective_cache_size = 32GB
  • Modifications to docker-compose.yml
    • mem_limit: 32g
    • shm_size: 32g
    • Mapping to resources file: - C:/temp:/temp
    • Mapping to postgresql.conf: - ./postgresql.conf:/etc/postgresql/postgresql.conf

Set up for SQL Server

  • Find and Install Microsoft SQL Server (a correct version)
  • Download and install the JDBC Driver (the correct version)
  • Download, install, and configure the DLL for the JDBC driver (the correct version)
  • Enable TCP/IP
  • Enable Mixed Mode Authentication
  • Restart SQL Server

Databases, Users, Permissions, etc.

  • Create a SQL Server database for the CDM
  • Create a login for uid/pwd authentication
  • Grant privileges for login:
    • exec sp_addrolemember N’db_accessadmin’, N’
    • exec sp_addrolemember N’db_backupoperator’, N’
    • exec sp_addrolemember N’db_datareader’, N’
    • exec sp_addrolemember N’db_datawriter’, N’
    • exec sp_addrolemember N’db_ddladmin’, N’
    • exec sp_addrolemember N’db_owner’, N’
    • exec sp_addrolemember N’db_securityadmin’, N’
  • Create the tables for the CDM database using the DDL files from the CDM
  • Create a record in CDM_SOURCE
  • Create or find a suitable vocabulary download from Athena
  • Load the Athena files into the CDM Vocabulary tables
  • Import your data into the CDM data tables (PERSON, MEASUREMENT, OBSERVATION, etc.)
  • Create the indexes on the CDM using the DDL from the CDM
  • Create the constraints on the CDM using the DDL from the CDM
  • Create WEBAPI SOURCE and SOURCE_DAIMON records
  • Create the Achilles results database
  • Create the Achilles temp database
  • Grant appropriate privileges to appropriate users on the Achilles results and temp databases

Achilles

  • Run Achilles

Automation with Polites

All of the above can be achieved using Polites in a few easy steps. Not only is the Polites process much easier it is much more consistent, repeatable, and reliable.

  1. Download and run the YesPolitesInstaller-1.1.002.exe
  2. Run the SQL Server and SSM installerS provided by the YesPolitesInstaller found in c:\_YES_POLITES\databases\sqlserver
  3. Run the Docker installer provided by the YesPolitesInstaller found in C:\_YES_POLITES\tools\docker and other tools as needed (e.g. R)
  4. Install and configure Broadsea by running C:\_YES_POLITES\workspace\install-broadsea.bat
  5. Start Docker by running C:\_YES_POLITES\workspace\Broadsea\start.bat
  6. Download and run the Latest Release of Polites
  7. Restart Docker by running the stop.bat and start.bat files in C:\_YES_POLITES\workspace\Broadsea

Notes on Docker scripts

Polites includes a number of scripts in the C:\_YES_POLITES\workspace\Broadsea directory to make common tasks for Docker easer. These scripts include the following:

  • start.bat: This script will start Docker using the appropriate settings for Broadsea.
  • stop.bat: This script will stop Docker using the appropriate settings for Broadsea.
  • clean-docker.sh: This script will remove all images, containers, and volumes from Docker.
  • reset.bat: This script will keep the Postgres database used by Broadsea and remove all other images, containers, and volumes from Docker.
  • show-docker.bat: This script will show all of the images, containers, and volumes currently in Docker.
  • export-demo_db.bat: This script can be used to export the demo_cdm as csv files.

Notes on Convenience scripts

Polites also includes a number of convience scripts in C:\_YES_POLITES\tools\bat for common tasks including the following.

  • git-init.bat: This script sets the git user.email (parameter 1) and user.name (parameter 2)
  • commit: This script will commit all of your changes directly up to Github. The first parameter is used as the message.
  • jv.bat: This script allows you to change the current version of Java. Valid parameters are 8 and 11.
  • git-details, where-am-i, where-am-i-really: These are all variation on getting the details of where the current project is in Github (e.g. where it was cloned from, the current branch, the tag, the SHA, etc.

Notes on Other Polites Tools

Polites provides a number of other tools for common tasks including the following.

  • Burn Everything to the Ground: This Option deletes all of the items created by Polites
  • Create Dummy Location and Care Site Records: This options creates a “not known” record in location and care_site. This option is for data sets that do not have values for these tables. This option is required as these are required foreign keys.
  • Import and Export Options: The import and export options allow a CDM to be exported by one instance and imported into an other. This can be done for the entire CDM, just the data tables, or just the vocabulary tables.
  • Create Sequences for Primary Keys: This option creates sequences that can be used as primary keys when importing data from other sources such as FHIR.
  • Disable/Enable constraints: This option allows for constraints to be disabled and then re-enabled for activities such as bulk uploads.

Notes on Resources

Setting up an OHDSI stack requires multiple resources such as JDBC database drivers, vocabulary files, mapping files, export files, etc. Polites stores these files online and they are downloaded as part of the Polites build process. This eliminates the need to hunt for, find, download, and install these files (often finding the wrong version and needing to start over again).

All of the resources downloaded by Polites can be found in C:\temp\Polites.

Summary

  • Polites provides an easy way to create a CDM and connect it to the OHDSI stack implemented by Broadsea
  • This process is not only easier, but is also more reliable, consistent, and repeatable
  • This process ensures that the compatable versions of components are used
  • This process ensures stability by using code freezes and stable releases: Components are only versioned as part of planed development cycles, code freezes, and releases
  • Polites also provides tools for common tasks such as importing and exporting data as csv
  • Polites supports multiple DBMS implementations such as SQL Server, PostgreSql, Databricks, etc.
  • The ability to import and export data in a common csv format alows Polites to be used to migrate CDM instances from one DBMS to another
  • All configuration for Polites is in a small single properties file. This has several advantages including the ability to see the scope of all variables that are part of the build and to make all configureation changes in a single place. This also makes in incredibly easy to clone a system from one site to another. This ability is especailly useful for testing. Testing envrionments can be created and reproduced simply by sharing configuration files
  • Polites solves the problems that arise from needing to correctly identify, locate, and download resources such as JDBC drivers, vocabulary files, and other resources: Resources are posted to accessable URLs and downloaded automatically as part of the Polites build. This capability is configurable. For example, a project could define their own complete Athena vocabulary and use Polites to distribute it to multiple sites.

Polites: What’s in a Name

Polites was Odysseus’ best friend on the Odyssey. He was one of the first to enter Circe’s palace and he was the one to convince Odysseus to leave the island and go back out to the broad sea. Here’s George Romney’s painting from 1782 of Circe waving good by to Odysseus, Polites, and you as you sail out in to the Broadsea.