Helium Journey Sync Engine

Introduction

The Helium Journey Sync Engine is a middleware that performs multi-master synchronisation between a Helium app's database and a Journey app's database. The engine is a standalone, multi-threaded Java application that synchronises one Helium database with one Journey database. The engine is started by passing a command line argument - that is the path to a configuration file. Multiple instances of the engine can be run on the same server.

High Level Design

Figure 1 - High Level Design Overview

Helium	The sync engine connects to the Helium RESTful API as well as the app's PostgreSQL database.
Journey Apps	The sync engine connects to the Journey Apps RESTful API.
Configuration	The sync engine is configured through a single, JSON formatted configuration file. The format and configuration options are described further down this page.
Java App	The sync engine is packaged as a single Java SE jar file. It is based on Java 8 and JDK version 8u144 or later is recommended.
Working Directory	The workingDirectory is a mandatory configuration option. The sync engine maintains a reconciliation database, log files and error reports inside the working directory.
Recon DB	The reconciliation database is used to keep track of records received from Journey that cannot immediately be written to Helium. Journey uses a schema less database with very little validation for referential integrity whereas Helium uses a well defined schema with validation for referential integrity. Records inside the reconciliation database are processed from time to time to see if anything has changed that would allow these records to be written to Helium.
Log Files	The java app prints output to STDOUT and is typically run in such a way that the output is concatenated to a series of log files inside the working directory.
Error Reports	The sync engine dumps certain error reports that may be useful for debugging in specialised sub-directories of the working directory. These include details of errors received from the Journey RESTful API etc.

Basic Configuration

Introduction

The sync engine is configured through a single, JSON formatted configuration file. The total number of available configuration options for the sync engine is large enough that it can be challenging to keep track of which configuration options affect which parts of the sync engine. To keep things manageable, the configuration is divided into different categories. Each category is represented by a separate JSON object that forms part of the larger JSON object. More about this under Advanced Configuration.

Every Helium app and Journey app has a set of unique attributes that are required in order for the sync engine to perform its job. These attributes are referred to as mandatory configuration options and are extensively covered under this topic - Basic Configuration. All configuration options that do not specify default values are mandatory.

In addition to the mandatory configuration options, there are quite a number of optional configuration options that determine how the sync engine throttles certain steps in the sync processes, what the ideal batch sizes are for change-logs, how it deals with data model inconsistencies, how it deals with errors etc. Most optional configuration options have sensible default values, whilst other are only required when you need to enable certain features. When left out of the configuration, the default values will be used for these configuration options. We recommend that you do leave these options out of your configuration unless you need to change their values. This approach improves the readability of your configuration files and allows you to automatically leverage changes to default values from updates to the sync engine.

Optional configuration options that enable features, such as synchronization of enrollment fields, are documented under this topic - Basic Configuration. The remainder of optional configuration options are documented under the Advanced Configuration topic.

Sample Configuration

 {
    "workingDirectory": "/home/helium-sync-journey/apps/my-app",
    "helium": {
        "restAPICredentials": {
            "userName": "my-user-name",
            "password": "my-password"
        },
        "appId": "4898ffdd-7775-4799-9966-a088dba5697f",
        "dbServer": "localhost",
        "dbName": "helium",
        "dbSchema": "my_app",
        "dbCredentials": {
            "userName": "my-user-name",
            "password": "my-password"
        }
    },
    "journey": {
        "restAPICredentials": {
            "userName": "my-user-name",
            "password": "my-password"
        },
        "appId": "YzEyZDhhNmUtMzYwZi00NDFmLThk"
    },
    "heliumToJourney": {
    },
    "journeyToHelium": {        
    },
    "journeyToHeliumEnrollment": {
        "type": "user",
        "heliumObjectName": "mobileuser",
        "journeyObjectName": "mobileuser",
        "allowedAccessFieldName": "enrollment_allowed_access",
        "enrolledFieldName": "enrollment_enrolled",
        "journeyLauncherVersionFieldName": "enrollment_launcher_version",
        "deviceOsFieldName": "enrollment_device_os",
        "deviceModelFieldName": "enrollment_device_model",
        "lastConnectedFieldName": "enrollment_last_connected",
        "barcodeFieldName": "enrollment_barcode",
        "urlFieldName": "enrollment_url",
        "performReEnrollmentFieldName": "enrollment_re_enroll_flag",
        "reEnrollmentLastProcessedFieldName": "enrollment_re_enroll_tstamp"
    }
}

Sample Configuration Explained

Option	Description
workingDirectory	Working directory. The sync engine will use this directory for its recon/staging database, error reports etc.

helium	The Helium configuration
helium.restAPICredentials	Credentials for the Helium RESTful API
helium.appId	ID for the Helium App
helium.dbServer	Name for the Helium database server where the app's data is hosted
helium.dbName	Name for the Helium database where the app's data is hosted
helium.dbSchema	Name for the Postgres schema that contains the app's data
helium.dbCredentials	Credentials for the Helium database where the app's data is hosted

journey	The Journey configuration
journey.restAPICredentials	Credentials for the Journey RESTful API
journey.appId	ID for the Journey App

heliumToJourney	The Helium to Journey configuration. This part of the configuration is required if enableHeliumToJourney is set to "true". The JSON object may be left blank to use all default values.
journeyToHelium	The Journey to Helium configuration. This part of the configuration is required if enableJourneyToHelium is set to "true". The JSON object may be left blank to use all default values.

journeyToHeliumEnrollment	The Journey to Helium configuration for synchronizing enrollment data. When set, the sync engine will synchronize enrollment data from Journey to Helium. journeyToHelium is required when this part of the configuration is provided. Along with "type" at least one valid field mapping must be specified for synchronization of enrollment data to be activated.
journeyToHeliumEnrollment.type	The type of Journey to Helium enrollment configuration. Currently supported values are: user - This type integrates with the legacy Journey user management API where one user is limited to one device
journeyToHeliumEnrollment.heliumObjectName	Name for the Helium persistent object that represents the Journey user
journeyToHeliumEnrollment.journeyObjectName	Name for the Journey object that represents the Journey user
journeyToHeliumEnrollment.allowedAccessFieldName	Name for the field in the Helium persistent object that contains the flag that indicates if the user is allowed to access the Journey app. The field type may be boolean or string. For string fields the possible values are Yes and No.
journeyToHeliumEnrollment.enrolledFieldName	Name for the field in the Helium persistent object that contains the flag that indicates if the user is currently enrolled. The field type may be boolean or string. For string fields the possible values are Yes and No.
journeyToHeliumEnrollment.journeyLauncherVersionFieldName	Name for the field in the Helium persistent object that contains the version for the Journey Launcher last used by the user. The field type must be string.
journeyToHeliumEnrollment.deviceOsFieldName	Name for the field in the Helium persistent object that contains the description of the device operating system last used by the user. The field type must be string.
journeyToHeliumEnrollment.deviceModelFieldName	Name for the field in the Helium persistent object that contains the description of the device model last used by the user. The field type must be string.
journeyToHeliumEnrollment.lastConnectedFieldName	Name for the field in the Helium persistent object that contains the date and time when the user was last connected. The field type may be string, date or dateTime. For string fields the value will be an ISO 8601 date and time string in coordinated universal time (UTC).
journeyToHeliumEnrollment.barcodeFieldName	Name for the field in the Helium persistent object that contains the URL to the Journey enrollment barcode for the user. The field type must be string.
journeyToHeliumEnrollment.urlFieldName	Name for the field in the Helium persistent object that contains the URL to the Journey enrollment page for the user. The field type must be string.
journeyToHeliumEnrollment.performReEnrollmentFieldName	Name for the field in the Helium persistent object that contains the flag that indicates if re-enrollment must be triggered for the user. When specified, this field is read by the Helium-to-Journey sync process. Setting the flag to a value that equates to "true", will trigger a re-enrollment against the Journey user management API. Once the re-enrollment has been triggered, the flag will be set back to a value that equates to "false" and the reEnrollmentLastProcessedFieldName will be updated. Note that Helium-to-Journey synchronization must be enabled for this flag to be processed. The field type may be boolean or string. For string fields the possible values are yes, no, true, false, t and f. For string fields the values are not case sensitive.
journeyToHeliumEnrollment.reEnrollmentLastProcessedFieldName	Name for the field in the Helium persistent object that contains the date and time when the last re-enrollment request was processed for the user. The field type may be string, date or dateTime. For string fields the value will be an ISO 8601 date and time string in coordinated universal time (UTC).

Other Commonly Used Options

Option	Description
enableHeliumToJourney	Flag to indicate if Helium data should be synchronized to Journey. The default value is "true".
enableJourneyToHelium	Flag to indicate if Journey data should be synchronized to Helium. The default value is "true".
enabled	Flag to indicate if the sync engine in its entirety is enabled. The default value is "true".

helium.baseUrl	Base URL for Helium, i.e. https://helium.mezzanineware.com The default value is https://helium.mezzanineware.com
journey.baseUrl	Base URL for Journey, i.e. https://run.journeyapps.com The default is https://run.journeyapps.com
journeyToHeliumEnrollment.updateIntervalMins	Number of minutes to wait between runs of the Journey-to-Helium synchronization of enrollment data. The default value is 15. The minimum value is 5 and the maximum value is 4320.

Caveats

Records may be changed on either Helium or Journey at any point in time. When a record gets changed on both platforms at the same time, then the changes will collide and the outcome will be that either the new Helium or the new Journey version of the record will win the collision in its entirety. Meaning there will be no partial update of records where some fields from the Helium version and some fields from the Journey version are kept. The winner of the collision (Helium version or Journey version of the record) is not predictable.
For performance sake the sync engine updates BLOB values from Journey to Helium asynchronously. Each BLOB value has to be separately loaded from the Journey RESTful API, filtered against the Helium App's database schema and then queue for merging to the Helium App's database schema. If the sync engine gets unceremoniously killed, then it is possible that some BLOB values that were in queues in memory may not be updated in Helium and won't be retried when the engine gets started again. We will probably add reconciliation for this at some point in the future.
The current version of the sync engine loads the entire list of outstanding updates from Helium during every Helium to Journey synchronization attempt. This is due to the absence of a fat change log in Helium that would make efficient paging over change logs possible. The sync engine may run out of memory if the outstanding change log on Helium grows too large. This is unlikely to happen, unless one tries to re-sync an entire app that contains a significant volume of records. We strongly advise against the re-sync scenario, due to point (1) above. At some point in the future we will add fat change logs to Helium and revisit how changes are loaded from Helium to make the sync engine more robust.

Low Level Design

Journey to Helium Processing Pipeline

Data is synchronised from Journey to Helium through a series of steps that extract, transform and load data. One can think of this series of steps as being a pipeline.

The sync engine was designed to run every step in the pipeline in a separate thread. This design allows the sync engine to process multiple batches of Journey oplog entries concurrently, a feature that drastically improves throughput. The two steps in the pipeline that are typically the most expensive in terms of time are the steps that download the Journey oplog over HTTPS and that merges the changes to Helium. These steps are both disk and network bound, but from different systems. Executing these steps in parallel for different batches therefore has a significant impact on overall performance.

Throughput for the various steps/processes in the pipeline can be throttled by changing the sizes of the queues to which these steps/processes send their output. See Advanced Configuration below.

Figure 2 - Journey to Helium Processing Pipeline

Download Oplog	This step in the process downloads the Journey Oplog and places it on queue A for further processing. At the time of writing the Journey API only returns 400 records per request, but the sync engine can be configured to perform multiple requests and merge the results into single batches before placing them on the queue (See downloadOplogTargetSize). This step will keep on requesting oplog batches and placing them on the queue until a request to the Journey API returns less than downloadOplogPollingFloor records. At that point this step will wait for downloadOplogPollingDelaySecs before doing another request. The following configuration options affect this step in the process: oplogMinVersion, oplogFile, optimiseChangeLog, oplogSkipObjectTypes, downloadOplogMaxRetries, downloadOplogRetryDelayMins, downloadOplogTargetSize, downloadOplogPollingFloor, downloadOplogPollingDelaySecs and downloadedOplogQueueSize
Convert Oplog	This step in the process converts the Journey Oplog to Helium's batch change format and places it on queue B for further processing. The conversion is based on the Helium app's data model which is downloaded from the Helium Apps RESTful API. The following configuration options affect this step in the process: convertedOplogQueueSize
Merge Oplog	This step in the process merges the Journey Oplog to the Helium App's database schema. BLOB values are not merged to the database schema yet, seeing that the Journey Oplog API doesn't return the BLOB values (Only IDs for the BLOB values). The following configuration options affect this step in the process: mergedOplogQueueSize
Record Status	The sync engine "tracks" certain objects in order to perform reconciliation of references/foreign keys at a later stage. This step also records the last version number from the Journey oplog that were successfully merged to Helium. Once recorded the requests are placed back on the three D queues for processing of BLOB values, reconciliation of staged records and processing of enrollment data. These further steps are processed in parallel. The following configuration options affect this step in the process: recordedStatusQueueSize
Reconcile Records	References that would violate referential integrity constraints at the time of processing the Journey oplog are reconsidered as candidates for merging to the Helium App's database schema during reconciliation. During reconciliation the sync engine runs through all foreign key references that wasn't set during processing of the Merge Oplog stage in order to check if any of the non-existent objects that were referenced have since been created. It is however possible that the relationship on the referencing side has since been set by another actor in the system, hence the requirement to track and compare the oplog version and timestamp that resulted in the "eventual reference". This step in the process will only run after both Record Status has completed and reconIntervalMins minutes has passed since the last reconciliation. The following configuration options affect this step in the process: reconIntervalMins and reconCutOffDays
Filter BLOBs	This step in the process evaluates all BLOB value IDs in the oplog to determine if those versions of the BLOB values already exist in the Helium App's database schema. This step is performed against the Helium App's database. BLOB IDs that have been changed are placed on queue E for loading from the Journey RESTful API. The following configuration options affect this step in the process: filteredBlobsQueueSize
Download BLOBs	This step in the process downloads binary data for changed BLOB values from the Journey RESTful API. These values are then placed on queue F for merging to the Helium App's database schema. The following configuration options affect this step in the process: loadedBlobsQueueSize, loadBlobsMaxRetries and loadBlobsRetryDelayMins
Merge BLOBs	This step in the process merges the binary data for changed BLOB values to the Helium App's database schema.
Update Enrollment Data	This step in the process loads enrollment data from the Journey RESTful API, converts the data to Helium's batch change format and merges the changes to the Helium App's database schema. The journeyToHeliumEnrollment sub-section of the configuration affects this step in the process.

Helium to Journey Processing Pipeline

Data is synchronised from Helium to Journey through a series of steps that extract, transform and load data. One can think of this series of steps as being a pipeline.

The sync engine was designed to run every step in the pipeline in a separate thread. This design allows the sync engine to process multiple batches of change-log entries concurrently, a feature that drastically improves throughput. The two steps in the pipeline that are typically the most expensive in terms of time are the steps that download the Helium change-log from the Helium App's database schema and that merges the changes to Journey. These steps are both disk and network bound, but from different systems. Executing these steps in parallel for different batches therefore has a significant impact on overall performance.

Throughput for the various steps/processes in the pipeline can be throttled by changing the sizes of the queues to which these steps/processes send their output. See Advanced Configuration below.

Figure 3 - Helium to Journey Processing

Extract ChangeLog	This step in the process extracts the latest changes from the Helium App's database schema and places it on queue A for further processing. This step will keep on requesting batches and placing them on the queue until a request to the Helium App's database returns less than downloadChangesPollingFloor records, at which time this step will wait for downloadChangesPollingDelaySecs before doing another request. The following configuration options affect this step in the process: changeLogMinVersion, changeLogSkipTables, downloadChangesMaxRetries, downloadChangesRetryDelayMins, downloadChangesPollingFloor, downloadChangesPollingDelaySecs and downloadedChangesQueueSize
Split ChangeLog	This step in the process splits batches of changes retrieved from the Helium App's database schema into smaller batches for processing by the Journey API. At the time of writing the Journey batch API could only process 1000 records at a time. The mergeChangesMaxBatchSize configuration option determines the size of the batches that will be placed back on queue B for further processing. The following configuration options affect this step in the process: splitChangesQueueSize
Convert ChangeLog	This step in the process converts Helium changes to the format required by the Journey batch API and places it on queue C for further processing. The conversion is based on the Helium app's data model which is downloaded from the Helium Apps RESTful API. The following configuration options affect this step in the process: convertedChangesQueueSize
Merge ChangeLog	This step in the process merges Helium changes to the Journey batch API. Once merged the Helium changes are placed back on queue D for recording of the sync status. The following configuration options affect this step in the process: mergeChangesMaxRetries, mergeChangesRetryDelayMins, alwaysMergeBeforeDelete and mergedChangesQueueSize
Record Status	The sync engine tracks the last change log version number that was successfully merged to the Journey batch API. This step in the process updates the change log number in the helium-sync-status.json file inside of the working directory.
Filter Re-Enrollment Data	This step in the process filters Helium changes to extract records that are due for re-enrollment and places the IDs of the records on queue E for further processing. The filter is based on the Helium app's data model which is downloaded from the Helium Apps RESTful API. The following configuration options affect this step in the process: filteredReEnrollmentQueueSize
Process Re-Enrollment Data	This step in the process pushes the re-enrollment requests to the Journey RESTful API and updated the flag and associated timestamps in the Helium App's database schema.

Advanced Configuration

Main Configuration

The main configuration is the top-level JSON object in the configuration file.

Option	Description
workingDirectory	Working directory. The sync engine will use this directory for its recon/staging database, error reports etc.
helium	The Helium configuration.
journey	The Journey configuration.
enableHeliumToJourney	Flag to indicate if Helium data should be synchronized to Journey. The default value is "true".
heliumToJourney	The Helium to Journey configuration. This part of the configuration is required if enableHeliumToJourney is set to "true".
enableJourneyToHelium	Flag to indicate if Journey data should be synchronized to Helium. The default value is "true".
journeyToHelium	The Journey to Helium configuration. This part of the configuration is required if enableJourneyToHelium is set to "true".
journeyToHeliumEnrollment	The Journey to Helium configuration for synchronizing enrollment data. When set, the sync engine will synchronize enrollment data from Journey to Helium. journeyToHelium is required when this part of the configuration is provided.
enabled	Flag to indicate if the sync engine in its entirety is enabled. The default value is "true".
restartIntervalMins	Number of minutes to wait after an irrecoverable error before restarting the engine. The default value is 5. The minimum value is 1 and the maximum value is 1440.

Helium Configuration

This section in the configuration consists mostly of mandatory options that determine how the sync engine connects to Helium.

Option	Description
baseUrl	Base URL for Helium, i.e. https://helium.mezzanineware.com The default value is https://helium.mezzanineware.com
restAPICredentials	Credentials for the Helium RESTful API.
appId	ID for the Helium App.
clientId	ID for the Helium client. This will be recorded in transactions. The default value is d4ec6977-28ab-4caf-8872-edd39b471436.
dbServer	Name for the Helium database server where the app's data is hosted.
dbPort	Number for the Helium database port where the app's data is hosted. The default value is 5432.
dbName	Name for the Helium database where the app's data is hosted.
dbSchema	Name for the Postgres schema that contains the app's data.
dbCredentials	Credentials for the Helium database where the app's data is hosted.
allowParallelBatchRequests	Flag that indicates if the sync engine should allow multiple batch requests to be executed against the Helium app's database schema at the same time. The default is to process one batch request at a time, regardless if that is a read or write batch operation. Setting this flag to "true" may in some cases improve throughput, but may also decrease throughput depending on the design of the specific app's database schema and the types of changes that occur on either side of the sync engine.

Journey Configuration

This section in the configuration consists mostly of mandatory options that determine how the sync engine connects to Journey.

Option	Description
baseUrl	Base URL for Journey, i.e. https://run.journeyapps.com The default is https://run.journeyapps.com .
restAPICredentials	Credentials for the Journey RESTful API.
appId	ID for the Journey App.
allowParallelBatchRequests	Flag that indicates if the sync engine should allow multiple batch requests to be executed against the Journey app's API at the same time. The default is to process one batch request at a time, regardless if that is a read or write batch operation. Setting this flag to "true" may in some cases improve throughput, but may also decrease throughput depending on the design of the specific app's database schema and the types of changes that occur on either side of the sync engine.

Journey-to-Helium Configuration

This section of the configuration consists mostly of optional values that affect throttling, error handling etc. when synchronizing data from Journey to Helium.

Option	Description
oplogMinVersion	Minimum/first version of the Journey oplog that will be processed in the event that the working directory doesn't contain a journey-sync-status.json file. The default is to start from zero.
oplogFile	A dump file containing one journey oplog entry per line. This file will be replayed before the Journey RESTful API will be polled. The default is to not replay any files.
optimiseOplog	Setting this flag to "true" will result in the engine removing "create" and "update" entries from Journey oplog batches when later parts of the same oplog batches contain "delete" entries for the same records. Setting this flag to "true" will result in faster sync speeds from Journey to Helium, but may result in foreign key violations. The default value is "false".
oplogSkipObjectTypes	Object types to skip when encountered in the Journey oplog. The default is to not skip any object types.
oplogSkipFieldNames	Qualified names of fields to skip when encountered in the Journey oplog. The default is to not skip any fields. A qualified name consists of the Helium object name, a '.' and the Helium field/attribute name like "Patient.firstNames".
ignoreManyToManyRelationships	Setting this flag to "true" will result in all many-to-many relationships being silently ignored in both directions. When this flag is "false", the sync engine will throw exceptions when data models contain many-to-many relationships. The default value is "true".
implicitlyConvertValues	Setting this flag to "true" will result in the sync engine attempting some basic, implicit type conversions to overcome data model inconsistencies. When this flag is set to "false", the sync engine will throw exceptions when any data received in the Journey oplog does not match the declared property types for the Helium app. The default value is "true".
downloadOplogMaxRetries	The maximum number of times that the sync engine will attempt to download a part of the Journey oplog before exceptions will trigger a (temporary) shut down of the pipeline. The default value is 10.
downloadOplogRetryDelayMins	The number of minutes to wait after a failure to download a section of the Journey oplog before attempting the download again. The minimum allowed and default value is 1.
downloadOplogTargetSize	The target number of entries to process at once from the Journey oplog. The sync engine will keep on retrieving the maximum possible number of entries from the Journey oplog until this value is reached or until Journey returns a partial page, meaning there are no more entries available in the oplog. The default value is 1000. The minimum value is 50 and the maximum value is 5000.
downloadOplogPollingFloor	The minimum number of entries obtained from the Journey oplog that will trigger an immediate poll for another batch of entries. When a batch from the Journey oplog contains less records than this number, the sync engine will suspend polling for "downloadOplogPollingDelaySecs" seconds before another request will be made. The default value is 300. The minimum value is 10 and the maximum value is 1000.
downloadOplogPollingDelaySecs	The number of seconds that polling to the Journey oplog will be suspended for after a request to the oplog returns less than "downloadOplogPollingFloor" records. The default value is 30 seconds. The minimum value is 10 seconds.
downloadedOplogQueueSize	The maximum number of Journey oplog batches that can be placed in a queue for processing at any point in time. This queue contains the original Journey oplog as it was retrieved from Journey before being converted to the Helium format. The default value is 10. The minimum value is 1 and the maximum value is 50.
convertedOplogQueueSize	The maximum number of converted Journey oplog batches that can be placed in a queue for merging to Helium at any point in time. This queue contains the converted Journey oplog batches in Helium format. The default value is 1. The minimum value is 1 and the maximum value is 10. Converting entries from the Journey format to the Helium format is typically a very fast step in the process. Increasing the size of this queue is unlikely to have an impact on overall throughput.
mergedOplogQueueSize	The maximum number of merged Journey oplog batches that can be placed in a queue for recording the Journey oplog position, anomalies for later reconciliation etc. This queue contains the converted Journey oplog batches in Helium format. The default value is 5. The minimum value is 1 and the maximum value is 10.
recordedStatusQueueSize	The maximum number of requests that can be placed on any of the outbound queues from the step in the process that records the Journey oplog position, anomalies for later reconciliation etc. The default value is 10. The minimum value is 1 and the maximum value is 50.
filteredBlobsQueueSize	The maximum number of filtered BLOB updates that can be placed in a queue for loading from Journey at any point in time. This queue contains references to BLOB values that were verified as having been updated from the Journey app. This check is peformed by comparing BLOB IDs received from Journey as part of the oplog with meta data in the Helium app's database. The default value is 10. The minimum value is 1 and the maximum value is 50.
loadBlobsMaxRetries	The maximum number of times that the sync engine will attempt to download a Blob field update from Journey before exceptions will trigger a (temporary) shut down of the pipeline. The default value is 10.
loadBlobsRetryDelayMins	The number of minutes to wait after a failure to download a Blob update from Journey before attempting the download again. The minimum allowed and default value is 1.
loadedBlobsQueueSize	The maximum number of loaded BLOB updates that can be placed on a queue for merging to Helium at any point in time. This queue contains the binary data along with meta data for the BLOB values. The default value is 50. The minimum value is 1 and the maximum value is 500.
reconIntervalMins	The number of minutes to wait between reconciliation runs for records synced from Journey to Helium. The default value is 15. The minimum value is 5 and the maximum value is 4320.
reconCutOffDays	The number of days to wait before giving up on records that require reconciliation. The default value is 7. The minimum value is 1 and the maximum value is 60.

Helium-to-Journey Configuration

This section of the configuration consists mostly of optional values that affect throttling, error handling etc. when synchronizing data from Helium to Journey.

Option	Description
changeLogMinVersion	Minimum/first version of the Helium change-log that will be processed in the event that the working directory doesn't contain a helium-sync-status.json file. The default is to start from zero.
changeLogSkipNotTracked	Setting this flag to 'true' will result in all object types that are marked as 'not tracked' in the DSL being skipped during processing. The default value is 'true'.
changeLogSkipTables	Tables to skip when exporting the Helium change log. The default is to not skip any tables.
changeLogSkipFieldNames	Qualified names of fields to skip when encountered in the Helium change log. The default is to not skip any fields. A qualified name consists of the Helium object name, a '.' and the Helium field/attribute name like "Patient.firstNames".
downloadChangesMaxRetries	The maximum number of times that the sync engine will attempt to download changed Helium records before exceptions will trigger a (temporary) shut down of the pipeline. The default value is 10.
downloadChangesRetryDelayMins	The number of minutes to wait after a failure to download changed Helium records before attempting the download again. The minimum allowed and default value is 1.
downloadChangesPollingFloor	The minimum number of changes obtained from Helium that will trigger an immediate poll for another batch of changes. When a batch from Helium contains less records than this number, the sync engine will suspend polling "downloadChangesPollingDelaySecs" seconds when the request to Helium returns less than this number of records. The default value is 1000. The minimum value is 10 and the maximum value is 10000.
downloadChangesPollingDelaySecs	The number of seconds that polling to Helium will be suspended for after a request returns less than "downloadChangesPollingFloor" records. The default value is 30 seconds. The minimum value is 10 seconds.
downloadedChangesQueueSize	The maximum number of Helium change batches that can be placed in a queue for processing at any point in time. This queue contains the original Helium changes as it was retrieved from the Helium database before being converted to the Journey format. The default and minimum value is 1. The maximum value is 10.
splitChangesMaxBatchSize	The maximum number of Helium changes that should be sent to Journey as a single batch. The default and maximum value is 1000. The minimum value is 10.
splitChangesQueueSize	The maximum number of split Helium change batches that can be placed in a queue for merging to Journey at any point in time. This queue contains converted changes that can be sent to Journey. The default value is 2. The minimum value is 1 and the maximum value is 10.
convertedChangesQueueSize	The maximum number of converted Helium change batches that can be placed in a queue for merging to Journey at any point in time. This queue contains converted changes that can be sent to Journey. The default value is 2. The minimum value is 1 and the maximum value is 50.
mergeChangesMaxRetries	The maximum number of times that the sync engine will attempt to merge changed Helium records to Journey before exceptions will trigger a (temporary) shut down of the pipeline. The default value is 10.
mergeChangesRetryDelayMins	The number of minutes to wait after a failure to merge changed Helium records to Journey before attempting the merge again. The minimum allowed and default value is 1.
alwaysMergeBeforeDelete	Setting this flag to "true" will result in the engine sending create/merge operations to Journey before sending delete operations to Journey. This pollutes the Journey oplog somewhat, but avoids OBJECT_NOT_FOUND errors for delete operations. The default value is "false".
mergedChangesQueueSize	The maximum number of merged Helium change batches that can be placed in a queue for recording sync status at any point in time. The default value is 5. The minimum value is 1 and the maximum value is 10.
convertedReEnrollmentQueueSize	The maximum number of converted re-enrollment batches that can be placed in a queue for processing at any point in time. This queue contains batches of filtered object IDs for which re-enrollment needs to be triggered. The default value is 2. The minimum value is 1 and the maximum value is 10.

Journey-to-Helium Enrollment Configuration

This section of the configuration consists mostly of field name mappings for mobile user enrollment data.

Option	Description
type	The type of Journey to Helium enrollment configuration. Currently supported values are: user - This type integrates with the legacy Journey user management API where one user is limited to one device. session - This type integrates with the session based Journey user management API where one user has a potentially unlimited device sessions.
clientId	ID for the Helium client. This will be recorded in transactions. The default value is 7c27c0db-e813-4c2b-ac22-d388c6ba0241 This needs to be different from the clientId specified in the Helium Configuration if there is a need to sync updates to meta data fields back to Journey.
heliumObjectName	Name for the Helium persistent object that represents the Journey user.
updateIntervalMins	Number of minutes to wait between runs of the Journey-to-Helium synchronization of enrollment data. The default value is 15. The minimum value is 5 and the maximum value is 4320.
journeyObjectName	Name for the Journey object that represents the Journey user.
performReEnrollmentFieldName	Name for the field in the Helium persistent object that contains the flag that indicates if re-enrollment must be triggered for the user. When specified, this field is read by the Helium-to-Journey sync process. Setting the flag to a value that equates to "true", will trigger a re-enrollment against the Journey user management API. Once the re-enrollment has been triggered, the flag will be set back to a value that equates to "false" and the reEnrollmentLastProcessedFieldName will be updated. Note that Helium-to-Journey synchronization must be enabled for this flag to be processed. The field type may be boolean or string. For string fields the possible values are yes, no, true, false, t and f. For string fields the values are not case sensitive.
reEnrollmentLastProcessedFieldName	Name for the field in the Helium persistent object that contains the date and time when the last re-enrollment request was processed for the user. The field type may be string, date or dateTime. For string fields the value will be an ISO 8601 date and time string in coordinated universal time (UTC).
... Include optional field names based on user or session type ...

Include options from this section to map data model with Journey's legacy enrollment API

Journey To Helium Enrollment User Configuration
allowedAccessFieldName	Name for the field in the Helium persistent object that contains the flag that indicates if the user is allowed to access the Journey app. The field type may be boolean or string. For string fields the possible values are Yes and No.
enrolledFieldName	Name for the field in the Helium persistent object that contains the flag that indicates if the user is currently enrolled. The field type may be boolean or string. For string fields the possible values are Yes and No.
journeyLauncherVersionFieldName	Name for the field in the Helium persistent object that contains the version for the Journey Launcher last used by the user. The field type must be string.
deviceOsFieldName	Name for the field in the Helium persistent object that contains the description of the device operating system last used by the user. The field type must be string.
deviceModelFieldName	Name for the field in the Helium persistent object that contains the description of the device model last used by the user. The field type must be string.
lastConnectedFieldName	Name for the field in the Helium persistent object that contains the date and time when the user was last connected. The field type may be string, date or dateTime. For string fields the value will be an ISO 8601 date and time string in coordinated universal time (UTC).
barcodeFieldName	Name for the field in the Helium persistent object that contains the URL to the Journey enrollment barcode for the user. The field type must be string.
urlFieldName	Name for the field in the Helium persistent object that contains the URL to the Journey enrollment page for the user. The field type must be string.

Include options from this section to map data model with Journey's session based enrollment API

Journey To Helium Enrollment Type Session Configuration
lastConnectedAtFieldName	Name for the field in the Helium persistent object that contains the date and time when the user was last connected. The field type may be string, date or dateTime. For string fields the value will be an ISO 8601 date and time string in coordinated universal time (UTC).
enrolledFieldName	Name for the field in the Helium persistent object that contains the flag that indicates if the user is currently enrolled. The field type may be boolean or string. For string fields the possible values are Yes and No.
lockedFieldName	Name for the field in the Helium persistent object that contains the flag that indicates if a user is locked out of the Journey application. When a user is locked they are signed out if busy in a session and cannot log into the Journey application. The field type may be boolean or string. For string fields the possible values are Yes and No.
sessionObjectName	Name for the Helium persistent object that contains device session data for Journey users
sessionObjectFieldNames	Names of fields in Helium persistent object for Journey user session data - See Journey To Helium Enrollment Session Object Field Names Configuration
sessionContainerObjectName	Name for the Helium persistent object that contains device session container data for Journey users
sessionContainerObjectFieldNames	Names of fields in Helium persistent object for Journey user session container data - See Journey To Helium Enrollment Session Container Object Configuration

Journey To Helium Enrollment Session Object Field Names Configuration
stateFieldName	Name for the field in the Helium persistent object that contains string representation of a Sessions state. The field type must be string.
lastAuthenticatedAtFieldName	Name for the field in the Helium persistent object that contains the date and time when the user was last authenticated. The field type may be string, date or dateTime. For string fields the value will be an ISO 8601 date and time string in coordinated universal time (UTC).
createdAtFieldName	Name for the field in the Helium persistent object that contains the date and time when the user was created. The field type may be string, date or dateTime. For string fields the value will be an ISO 8601 date and time string in coordinated universal time (UTC).
signedOutAtFieldName	Name for the field in the Helium persistent object that contains the date and time when the user was last signed out. The field type may be string, date or dateTime. For string fields the value will be an ISO 8601 date and time string in coordinated universal time (UTC).
lastConnectedAtFieldName	Name for the field in the Helium persistent object that contains the date and time when the user was last connected. The field type may be string, date or dateTime. For string fields the value will be an ISO 8601 date and time string in coordinated universal time (UTC).
labelFieldname	Name for the field in the Helium persistent object that contains Journey Session Label. The field type must be string.
heliumObjectFkFieldName	Name of the id field in the Helium persistent object that contains the FK to the heliumObjectName table. The field type must be string.

Journey To Helium Enrollment Session Container Object Configuration
versionFieldName	Name for the field in the Helium persistent object that contains Journey container version for a device. An example value is '3.30.3'. The field type must be string.
bundleIdFieldName	Name for the field in the Helium persistent object that contains Journey application bundle id. An example value is xyz.appinstall.myapp. The field type must be string.
platformFieldName	Name for the field in the Helium persistent object that contains device platform name. An example value is IOS or Chrome. The field type must be string.
sessionObjectFkFieldName	Name for the field in the Helium persistent object that contains FK the to the session that this container belongs. The field type must be UUID.

Sample Configuration

This sample shows all available configuration options as documented above.

{
  "workingDirectory" : "/home/helium-sync-journey/apps/my-app",
  "helium" : {
    "baseUrl" : "https://helium.mezzanineware.com",
    "restAPICredentials" : {
      "userName" : "my-user-name",
      "password" : "my-password"
    },
    "appId" : "4898ffdd-7775-4799-9966-a088dba5697f",
    "clientId" : "d4ec6977-28ab-4caf-8872-edd39b471436",
    "dbServer" : "localhost",
    "dbPort" : 5432,
    "dbName" : "helium",
    "dbSchema" : "my_app",
    "dbCredentials" : {
      "userName" : "my-user-name",
      "password" : "my-password"
    }
  },
  "journey" : {
    "baseUrl" : "https://run.journeyapps.com",
    "restAPICredentials" : {
      "userName" : "my-user-name",
      "password" : "my-password"
    },
    "appId" : "YzEyZDhhNmUtMzYwZi00NDFmLThk"
  },
  "enableHeliumToJourney" : true,
  "heliumToJourney" : {
    "changeLogMinVersion" : 0,
	"changeLogSkipNotTracked" : true,
    "changeLogSkipTables" : [ ],
    "downloadChangesMaxRetries" : 10,
    "downloadChangesRetryDelayMins" : 1,
    "downloadChangesPollingFloor" : 1000,
    "downloadChangesPollingDelaySecs" : 30,
    "downloadedChangesQueueSize" : 1,
    "splitChangesMaxBatchSize" : 1000,
    "splitChangesQueueSize" : 2,
    "convertedChangesQueueSize" : 2,
    "mergeChangesMaxRetries" : 10,
    "mergeChangesRetryDelayMins" : 1,
    "alwaysMergeBeforeDelete" : true,
    "mergedChangesQueueSize" : 5,
    "filteredReEnrollmentQueueSize" : 2
  },
  "enableJourneyToHelium" : true,
  "journeyToHelium" : {
    "oplogMinVersion" : 0,
    "oplogFile" : null,
    "optimiseOplog" : false,
    "oplogSkipObjectTypes" : [ ],
    "ignoreManyToManyRelationships" : true,
    "implicitlyConvertValues" : true,
    "downloadOplogMaxRetries" : 10,
    "downloadOplogRetryDelayMins" : 1,
    "downloadOplogTargetSize" : 1000,
    "downloadOplogPollingFloor" : 300,
    "downloadOplogPollingDelaySecs" : 30,
    "downloadedOplogQueueSize" : 10,
    "convertedOplogQueueSize" : 1,
    "mergedOplogQueueSize" : 5,
    "recordedStatusQueueSize" : 10,
    "filteredBlobsQueueSize" : 10,
    "loadBlobsMaxRetries" : 10,
    "loadBlobsRetryDelayMins" : 1,
    "loadedBlobsQueueSize" : 50,
    "reconIntervalMins" : 15,
    "reconCutOffDays" : 7
  },
  "journeyToHeliumEnrollment" : {
    "type" : "user",
    "clientId" : "7c27c0db-e813-4c2b-ac22-d388c6ba0241",
    "heliumObjectName" : "mobileuser",
    "updateIntervalMins" : 15,
    "journeyObjectName" : "mobileuser",
    "allowedAccessFieldName" : "enrollment_allowed_access",
    "enrolledFieldName" : "enrollment_enrolled",
    "journeyLauncherVersionFieldName" : "enrollment_launcher_version",
    "deviceOsFieldName" : "enrollment_device_os",
    "deviceModelFieldName" : "enrollment_device_model",
    "lastConnectedFieldName" : "enrollment_last_connected",
    "barcodeFieldName" : "enrollment_barcode",
    "urlFieldName" : "enrollment_url",
    "performReEnrollmentFieldName" : "enrollment_re_enroll_flag",
    "reEnrollmentLastProcessedFieldName" : "enrollment_re_enroll_tstamp"
  },
  "enabled" : true,
  "restartIntervalMins" : 5
}

Sample Configuration using Session based enrollments

This sample shows available configuration options as for session based enrollment as documented above.

{
...
  "journeyToHeliumEnrollment" : {
    "type" : "session",
    "clientId" : "7c27c0db-e813-4c2b-ac22-d388c6ba0241",
    "heliumObjectName" : "mobileuser",
    "updateIntervalMins" : 15,
    "journeyObjectName" : "mobileuser",
    "lastConnectedAtFieldName": "last_connected_at",
    "enrolledFieldName": "enrolled",
    "lockedFieldName": "locked",
    "performReEnrollmentFieldName": "perform_re_enrollment",
    "reEnrollmentLastProcessedFieldName": "re_enrollment_last_processed",
    "sessionObjectName": "mobileusersession",
    "sessionObjectFieldNames": {
      "stateFieldName": "state",
      "lastAuthenticatedAtFieldName": "last_authenticated_at",
      "createdAtFieldName": "created_at_field",
      "signedOutAtFieldName": "signed_out_at",
      "lastConnectedAtFieldName": "last_connected_at",
      "labelFieldname": "label",
      "heliumObjectFkFieldName": "mobileusersession_mobileuser"
    },
    "sessionContainerObjectName": "mobileuser_session_container",
    "sessionContainerObjectFieldNames": {
      "versionFieldName": "version",
      "bundleIdFieldName": "bundle_id",
      "platformFieldName": "platform",
      "sessionObjectFkFieldName": "mobileusersessioncontainer_mobileusersession"
    }
  },
...
}

Helium Rapid Documentation

Helium Journey Sync Engine

Table of Contents

Introduction

High Level Design

Basic Configuration

Introduction

Sample Configuration

Sample Configuration Explained

Other Commonly Used Options

Caveats

Low Level Design

Journey to Helium Processing Pipeline

Helium to Journey Processing Pipeline

Advanced Configuration

Main Configuration

Helium Configuration

Journey Configuration

Journey-to-Helium Configuration

Helium-to-Journey Configuration

Journey-to-Helium Enrollment Configuration

Sample Configuration

Sample Configuration using Session based enrollments