Chapter 6. Advanced Configuration

Table of Contents

6.1. Multi-Tiered Synchronization
6.2. Bi-Directional Synchronization
6.3. Dead Triggers
6.4. Database Purging
6.5. Extension Points
6.5.1. IParameterFilter
6.5.2. IDataLoaderFilter
6.5.3. ITableColumnFilter
6.5.4. IBatchListener
6.5.5. IAcknowledgeEventListener
6.5.6. IReloadListener
6.5.7. IExtractorFilter
6.5.8. ISyncUrlExtension
6.5.9. INodeIdGenerator
6.5.10. ITriggerCreationListener
6.6. Encrypted Passwords
6.7. Secure Transport
6.7.1. Sym Launcher
6.7.2. Tomcat
6.7.3. Keystores
6.7.4. Generating Keys

6.1. Multi-Tiered Synchronization

At times, there may be scenarios where data needs to flow through multiple tiers of nodes that are organized in a tree-like network with each tier requiring a different subset of data. For example, you may have a system where the lowest tier may by a computer or device located in a store. Those devices may connect to a server located physically at that store. Then the store server may communicate with a corporate server or even a regional server.

The different tiers would be device, store, region and corporate. A tier is typically represented by a node group. Each node in the tier would belong to the node group representing that tier.

A node will always push and pull data to other node groups according to the node group link configuration. A node can only pull and push data to other nodes that are represented as rows in the node table in its database. Because of this, a tree-like hierarchy of nodes can be created by having only a subset of nodes belonging to the same node group represented at the different branches of the tree.

If auto registration is turned off, then this setup must occur manually by opening registration for the desired nodes at the desired parent node and by configuring each node's registration.url to be the parent node's URL. The parent node is always tracked by the setting of the parent's node_id in the created_at_node_id column of the new node. When a node registers and downloads its configuration it is always provided the configuration for all nodes that were created at the node that it is registering with.

Most tiers will require the SymmetricDS server software. It must be deployed to any node that is pulled from or pushed to. If the node only pushes and pulls from other nodes, then only the client software (no HTTP server) is required.

Nodes that service many clients may be deployed as a web farm for scalability purposes. In this scenario a single node may have multiple SymmetricDS server instances that are configured the same and are fronted by some type of stateless load balancer.

6.2. Bi-Directional Synchronization

SymmetricDS allows tables to be synchronized bi-directionally. Note that an outgoing synchronization does not process changes during an incoming synchronization on the same node unless the trigger was created with the sync_on_incoming_batch flag set. If the sync_on_incoming_batch flag is set, then update loops are prevented by a feature that is available in most database dialects. During an incoming synchronization the source node_id is put into a database session variable that is available to the database trigger. Data events are not generated if the target node_id on an outgoing synchronization is equal to the source node_id.

When synchronizing tables in both directions, the sync_column_level flag may be enabled. This indicates to the data loader that only columns that have changed should be updated. In order to accomplish this, both old and new data are transmitted from the source node to the target node. Because of the extra data overhead this feature should only be enabled if needed.

More complex conflict resolution strategies can be accomplished by using the IDataLoaderFilter extension point which has access to both old and new data.

6.3. Dead Triggers

Normally a Trigger is specified to capture data changes to a table and send them to a target Node Group. A dead Trigger is one that does not capture data changes. In other words, the sync_on_insert, sync_on_update, and sync_on_delete properties for the Trigger are all set to false. Because the Trigger is specified, it will be included in the initial load of data for target Nodes.

A dead Trigger might be used to load a read-only lookup table. It could be used to load a table that needs populated with example or default data. Another use is a recovery load of data for tables that have a single direction of synchronization. For example, a retail store records sales transaction that synchronize in one direction by trickling back to the central office. If the retail store needs to recover all the sales transactions, they can be sent are part of an initial load from the central office by setting up dead Triggers that "sync" in that direction.

The following SQL statement sets up a non-syncing dead Trigger that sends the sale_transaction table to the "store" Node Group from the "corp" Node Group during an initial load.

insert into SYM_TRIGGER 
  (source_table_name, source_node_group_id, target_node_group_id, channel_id, 
   sync_on_insert, sync_on_update, sync_on_delete, 
   initial_load_order, last_updated_by, last_updated_time, create_time)
values
  ('sale_transaction', 'corp', 'store', 'sale_transaction', 
   0, 0, 0, 
   105, 'demo', current_timestamp, current_timestamp);

6.4. Database Purging

6.5. Extension Points

SymmetricDS may be extended via a plug-in like architecture where extension point interfaces may be implemented by a custom class and registered with the synchronization engine. All supported extension points extend the IExtensionPoint interface. The currently available extension points are documented in the following sections.

When the synchronization engine starts up, a Spring post processor searches the Spring ApplicationContext for any registered classes which implement IExtensionPoint. An IExtensionPoint designates whether it should be auto registered or not. If the extension point is to be auto registered then the post processor registers the known interface with the appropriate service.

The INodeGroupExtensionPoint interface may be optionally implemented to designate that auto registered extension points should only be auto registered with specific node groups.

/**
 * Only apply this extension point to the 'root' node group.
 */
 public String[] getNodeGroupIdsToApplyTo() {
     return new String[] { "root" };
 }

SymmetricDS will look for Spring configured extensions in the application Classpath by importing any Spring XML configuration files found matching the following pattern: META-INF/services/symmetric-*-ext.xml.

6.5.1. IParameterFilter

Parameter values can be specified in code using a parameter filter. Note that there can be only one parameter filter per engine instance. The IParameterFilter replaces the depreciated IRuntimeConfig from prior releases.

public class MyParameterFilter 
    implements IParameterFilter, INodeGroupExtensionPoint {

    /**
     * Only apply this filter to stores
     */
    public String[] getNodeGroupIdsToApplyTo() {
        return new String[] { "store" };
    }

    public String filterParameter(String key, String value) {
        // look up a store number from an already existing properties file.
        if (key.equals(ParameterConstants.EXTERNAL_ID)) {
            return StoreProperties.getStoreProperties().
              getProperty(StoreProperties.STORE_NUMBER);
        } 
        return value;
    }

    public boolean isAutoRegister() {
        return true;
    }

}

6.5.2. IDataLoaderFilter

Data can be filtered as it is loaded into the target database or when it is extracted from the source database.

As data is loaded into the target database, a filter can change the data in a column or save it somewhere else. It can also specify by the return type of the function call that the data loader should continue on and load the data (by returning true) or ignore it (by returning false). One possible use of the filter might be to route credit card data to a secure database and blank it out as it loads into less-restricted reporting database.

An IDataLoaderContext is passed to each of the callback methods. A new context is created for each synchronization. The context provides methods to lookup column indexes by column name, get table meta data, and provides access to old data if the sync_column_level flag is enabled. The context also provides a means to share data during a synchronization between different rows of data that are committed in a database transaction and are in the same channel. It does so by providing a context cache which can be populated by the extension point.

Many times the IDataLoaderFilter will be combined with the IBatchListener. The XmlPublisherFilter (in the org.jumpmind.symmetric.ext package) is a good example of using the combination of the two extension points in order to create XML messages to be published to JMS.

A class implementing the IDataLoaderFilter interface is injected onto the DataLoaderService in order to receive callbacks when data is inserted, updated, or deleted.

public MyFilter implements IDataLoaderFilter {
                
    public boolean isAutoRegister() {
        return true;
    }
 
    public boolean filterInsert(IDataLoaderContext context,
        String[] columnValues) {
        return true;
    }
    
    public boolean filterUpdate(IDataLoaderContext context, 
        String[] columnValues, String[] keyValues) {
        return true;
    }
    
    public void filterDelete(IDataLoaderContext context, 
        String[] keyValues) {
        return true;
    }

}

The filter class is specified as a Spring-managed bean. A custom Spring XML file is specified as follows in a jar at META-INF/services/symmetric-myfilter-ext.xml.

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xmlns:context="http://www.springframework.org/schema/context"
    xsi:schemaLocation="http://www.springframework.org/schema/beans 
           http://www.springframework.org/schema/beans/spring-beans-2.5.xsd
           http://www.springframework.org/schema/context
           http://www.springframework.org/schema/context/spring-context-2.5.xsd">
 
    <bean id="myFilter" class="com.mydomain.MyFilter"/>
     
</beans>

6.5.3. ITableColumnFilter

Implement this extension point to filter out specific columns from use by the dataloader. Only one column filter may be added per target table.

6.5.4. IBatchListener

This extension point is called whenever a batch has completed loading but before the transaction has committed.

6.5.5. IAcknowledgeEventListener

Implement this extension point to receive callback events when a batch is acknowledged. The callback for this listener happens at the point of extraction.

6.5.6. IReloadListener

Implement this extension point to listen in and take action before or after a reload is requested for a Node. The callback for this listener happens at the point of extraction.

6.5.7. IExtractorFilter

This extension point is called after data has been extracted, but before it has been streamed. It has the ability to inspect each row of data to take some action and indicate, if necessary, that the row should not be streamed.

6.5.8. ISyncUrlExtension

This extension point is used to select an appropriate URL based on the URI provided in the sync_url column of sym_node.

To use this extension point configure the sync_url for a node with the protocol of ext://beanName. The beanName is the name you give the extension point in the extension xml file.

6.5.9. INodeIdGenerator

This extension point allows SymmetricDS users to implement their own algorithms for how node ids and passwords are generated or selected during the registration process. There may be only one node generator per SymmetricDS instance.

6.5.10. ITriggerCreationListener

Implement this extension point to get status callbacks during trigger creation.

6.6. Encrypted Passwords

The db.user and db.password properties will accept encrypted text, which protects against casual observation. The text is prefixed with "enc:" to indicate that it is encrypted. To encrypt text, use the following command:

sym -e secret

The text is encrypted by the cipher defined as alias "sym.secret" in the Java keystore. The keystore is specified by the "sym.keystore.file" system property, which defaults to security/keystore. If a cipher is not found, a default cipher using Triple DES with a random password is generated.

6.7. Secure Transport

By specifying the "https" protocol for a URL, SymmetricDS will communicate over Secure Sockets Layer (SSL) for an encrypted transport. The following properties need to be set with "https" in the URL:

my.url

This is the URL of the current node, so if you want to force other nodes to communicate over SSL with this node, you specify "https" in the URL.

registration.url

This is the URL where the node will connect for registration when it first starts up. To protect the registration with SSL, you specify "https" in the URL.

For incoming HTTPS connections, SymmetricDS depends on the webserver where it is deployed, so the webserver must be configured for HTTPS. As a standalone deployment, the "sym" launcher command provides options for enabling HTTPS support.

6.7.1. Sym Launcher

The "sym" launch command uses Jetty as an embedded web server. Using command line options, the web server can be told to listen for HTTP, HTTPS, or both.

sym --port 8080 --server

sym --secure-port 8443 --secure-server

sym --port 8080 --secure-port 8443 --mixed-server

6.7.2. Tomcat

If you deploy SymmetricDS to Apache Tomcat, it can be secured by editing the TOMCAT_HOME/conf/server.xml configuration file. There is already a line that can be uncommented and changed to the following:

<Connector port="8443" protocol="HTTP/1.1" SSLEnabled="true" 
  maxThreads="150" scheme="https" secure="true" 
  clientAuth="false" sslProtocol="TLS"
  keystoreFile="/symmetric-ds-1.x.x/security/keystore" />

6.7.3. Keystores

When SymmetricDS connects to a URL with HTTPS, Java checks the validity of the certificate using the built-in trusted keystore located at JRE_HOME/lib/security/cacerts. The "sym" launcher command overrides the trusted keystore to use its own trusted keystore instead, which is located at security/cacerts. This keystore contains the certificate aliased as "sym" for use in testing and easing deployments. The trusted keystore can be overridden by specifying the "javax.net.ssl.trustStore" system property.

When SymmetricDS is run as a secure server with the "sym" launcher, it accepts incoming requests using the key installed in the keystore located at security/keystore. The default key is provided for convenience of testing, but should be re-generated for security.

6.7.4. Generating Keys

To generate new keys and install a server certificate, use the following steps:

  1. Open a command prompt and navigate to the security subdirectory of your SymmetricDS installation.

  2. Delete the old key pair and certificate.

    keytool -keystore keystore -delete -alias sym

    keytool -keystore cacerts -delete -alias sym

    Enter keystore password:  changeit
  3. Generate a new key pair.

    keytool -keystore keystore -alias sym -genkey -keyalg RSA -validity 10950

    Enter keystore password:  changeit
    What is your first and last name?
      [Unknown]:  localhost
    What is the name of your organizational unit?
      [Unknown]:  SymmetricDS
    What is the name of your organization?
      [Unknown]:  JumpMind
    What is the name of your City or Locality?
      [Unknown]:
    What is the name of your State or Province?
      [Unknown]:
    What is the two-letter country code for this unit?
      [Unknown]:
    Is CN=localhost, OU=SymmetricDS, O=JumpMind, L=Unknown, ST=Unknown, C=Unknown
    correct?
      [no]:  yes
    
    Enter key password for <sym>
            (RETURN if same as keystore password):
  4. Export the certificate from the private keystore.

    keytool -keystore keystore -export -alias sym -rfc -file sym.cer

  5. Install the certificate in the trusted keystore.

    keytool -keystore cacerts -import -alias sym -file sym.cer