Chapter 5. Advanced Topics

Table of Contents

5.1. Advanced Synchronization
5.1.1. Bi-Directional Synchronization
5.1.2. Multi-Tiered Synchronization
5.1.2.1. Registration Redirect
5.2. JMS Publishing
5.3. Deployment Options
5.3.1. Web Archive (WAR)
5.3.2. Embedded
5.4. Standalone
5.5. Running SymmetricDS as a Service
5.5.1. Running as a Windows Service
5.5.2. Running as a *nix Service
5.6. Clustering
5.7. Encrypted Passwords
5.8. Secure Transport
5.8.1. Sym Launcher
5.8.2. Tomcat
5.8.3. Keystores
5.8.4. Generating Keys
5.9. Basic Authentication
5.10. Extension Points
5.10.1. IParameterFilter
5.10.2. IDatabaseWriterFilter
5.10.3. IDatabaseWriterErrorHandler
5.10.4. IDataLoaderFactory
5.10.5. IAcknowledgeEventListener
5.10.6. IReloadListener
5.10.7. ISyncUrlExtension
5.10.8. IColumnTransform
5.10.9. INodeIdCreator
5.10.10. ITriggerCreationListener
5.10.11. IBatchAlgorithm
5.10.12. IDataRouter
5.10.13. IHeartbeatListener
5.10.14. IOfflineClientListener
5.10.15. IOfflineServerListener
5.10.16. INodePasswordFilter
5.11. Synchronization to and from Android Devices

This chapter focuses on a variety of topics, including deployment options, jobs, clustering, encryptions, synchronization control, and configuration of SymmetricDS.

5.1. Advanced Synchronization

5.1.1. Bi-Directional Synchronization

SymmetricDS allows tables to be synchronized bi-directionally. Note that an outgoing synchronization does not process changes during an incoming synchronization on the same node unless the trigger was created with the sync_on_incoming_batch flag set. If the sync_on_incoming_batch flag is set, then update loops are prevented by a feature that is available in most database dialects. More specifically, during an incoming synchronization the source node_id is put into a database session variable that is available to the database trigger. Data events are not generated if the target node_id on an outgoing synchronization is equal to the source node_id.

By default, only the columns that changed will be updated in the target system.

Conflict resolution strategies can be configured for specific links and/or sets of tables.

5.1.2. Multi-Tiered Synchronization

As shown in Section 3.2, “Organizing Nodes”, there may be scenarios where data needs to flow through multiple tiers of nodes that are organized in a tree-like network with each tier requiring a different subset of data. For example, you may have a system where the lowest tier may be a computer or device located in a store. Those devices may connect to a server located physically at that store. Then the store server may communicate with a corporate server for example. In this case, the three tiers would be device, store, and corporate. Each tier is typically represented by a node group. Each node in the tier would belong to the node group representing that tier.

A node can only pull and push data to other nodes that are represented in the node's NODE table and in cases where that node's sync_enabled column is set to 1. Because of this, a tree-like hierarchy of nodes can be created by having only a subset of nodes belonging to the same node group represented at the different branches of the tree.

If auto registration is turned off, then this setup must occur manually by opening registration for the desired nodes at the desired parent node and by configuring each node's registration.url to be the parent node's URL. The parent node is always tracked by the setting of the parent's node_id in the created_at_node_id column of the new node. When a node registers and downloads its configuration it is always provided the configuration for nodes that might register with the node itself based on the Node Group Links defined in the parent node.

5.1.2.1. Registration Redirect

When deploying a multi-tiered system it may be advantageous to have only one registration server, even though the parent node of a registering node could be any of a number of nodes in the system. In SymmetricDS the parent node is always the node that a child registers with. The REGISTRATION_REDIRECT table allows a single node, usually the root server in the network, to redirect registering nodes to their true parents. It does so based on a mapping found in the table of the external id (registrant_external_id) to the parent's node id (registration_node_id).

For example, if it is desired to have a series of regional servers that workstations at retail stores get assigned to based on their external_id, the store number, then you might insert into REGISTRATION_REDIRECT the store number as the registrant_external_id and the node_id of the assigned region as the registration_node_id. When a workstation at the store registers, the root server sends an HTTP redirect to the sync_url of the node that matches the registration_node_id.

Important

Please see Section 4.6.3.2, “Initial Loads” for important details around initial loads and registration when using registration redirect.

5.2. JMS Publishing

With the proper configuration SymmetricDS can publish XML messages of captured data changes to JMS during routing or transactionally while data loading synchronized data into a target database. The following explains how to publish to JMS during synchronization to the target database.

The XmlPublisherDatabaseWriterFilter is a IDatabaseWriterFilter that may be configured to publish specific tables as an XML message to a JMS provider. See Section 5.10, “Extension Points” for information on how to configure an extension point. If the publish to JMS fails, the batch will be marked in error, the loaded data for the batch will be rolled back and the batch will be retried during the next synchronization run.

The following is an example extension point configuration that will publish four tables in XML with a root tag of 'sale'. Each XML message will be grouped by the batch and the column names identified by the groupByColumnNames property which have the same values.

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xmlns:context="http://www.springframework.org/schema/context"
    xsi:schemaLocation="http://www.springframework.org/schema/beans
           http://www.springframework.org/schema/beans/spring-beans-3.0.xsd
           http://www.springframework.org/schema/context
           http://www.springframework.org/schema/context/spring-context-3.0.xsd">

    <bean id="configuration-publishingFilter"
      class="org.jumpmind.symmetric.integrate.XmlPublisherDatabaseWriterFilter">
        <property name="xmlTagNameToUseForGroup" value="sale"/>
        <property name="tableNamesToPublishAsGroup">
            <list>
               <value>SALE_TX</value>
               <value>SALE_LINE_ITEM</value>
               <value>SALE_TAX</value>
               <value>SALE_TOTAL</value>
            </list>
        </property>
        <property name="groupByColumnNames">
            <list>
               <value>STORE_ID</value>
               <value>BUSINESS_DAY</value>
               <value>WORKSTATION_ID</value>
               <value>TRANSACTION_ID</value>
            </list>
        </property>
        <property name="publisher">
           <bean class="org.jumpmind.symmetric.integrate.SimpleJmsPublisher">
               <property name="jmsTemplate" ref="definedSpringJmsTemplate"/>
           </bean>
        </property>
    </bean>
</beans>

The publisher property on the XmlPublisherDatabaseWriterFilter takes an interface of type IPublisher. The implementation demonstrated here is an implementation that publishes to JMS using Spring's JMS template. Other implementations of IPublisher could easily publish the XML to other targets like an HTTP server, the file system or secure copy it to another server.

The above configuration will publish XML similar to the following:

<?xml version="1.0" encoding="UTF-8"?>
<sale xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  id="0012010-01-220031234" nodeid="00001" time="1264187704155">
  <row entity="SALE_TX" dml="I">
    <data key="STORE_ID">001</data>
    <data key="BUSINESS_DAY">2010-01-22</data>
    <data key="WORKSTATION_ID">003</data>
    <data key="TRANSACTION_ID">1234</data>
    <data key="CASHIER_ID">010110</data>
  </row>
  <row entity="SALE_LINE_ITEM" dml="I">
    <data key="STORE_ID">001</data>
    <data key="BUSINESS_DAY">2010-01-22</data>
    <data key="WORKSTATION_ID">003</data>
    <data key="TRANSACTION_ID">1234</data>
    <data key="SKU">9999999</data>
    <data key="PRICE">10.00</data>
    <data key="DESC" xsi:nil="true"/>
  </row>
  <row entity="SALE_LINE_ITEM" dml="I">
    <data key="STORE_ID">001</data>
    <data key="BUSINESS_DAY">2010-01-22</data>
    <data key="WORKSTATION_ID">003</data>
    <data key="TRANSACTION_ID">1234</data>
    <data key="SKU">9999999</data>
    <data key="PRICE">10.00</data>
    <data key="DESC" xsi:nil="true"/>
  </row>
  <row entity="SALE_TAX" dml="I">
    <data key="STORE_ID">001</data>
    <data key="BUSINESS_DAY">2010-01-22</data>
    <data key="WORKSTATION_ID">003</data>
    <data key="TRANSACTION_ID">1234</data>
    <data key="AMOUNT">1.33</data>
  </row>
  <row entity="SALE_TOTAL" dml="I">
    <data key="STORE_ID">001</data>
    <data key="BUSINESS_DAY">2010-01-22</data>
    <data key="WORKSTATION_ID">003</data>
    <data key="TRANSACTION_ID">1234</data>
    <data key="AMOUNT">21.33</data>
  </row>
</sale>
            

To publish JMS messages during routing the same pattern is valid, with the exception that the extension point would be the XmlPublisherDataRouter and the router would be configured by setting the router_type of a ROUTER to the Spring bean name of the registered extension point. Of course, the router would need to be linked through TRIGGER_ROUTERs to each TRIGGER table that needs published.

5.3. Deployment Options

An instance of SymmetricDS can be deployed in several ways:

  • Web application archive (WAR) deployed to an application server

    This option means packaging a WAR file and deploying to your favorite web server, like Apache Tomcat. It's a little more work, but you can configure the web server to do whatever you need. SymmetricDS can also be embedded in an existing web application, if desired.

  • Standalone service that embeds Jetty web server

    This option means running the sym command line, which launches the built-in Jetty web server. This is a simple option because it is already provided, but you lose the flexibility to configure the web server any further.

  • Embedded as a Java library in an application

    This option means you must write a wrapper Java program that runs SymmetricDS. You would probably use Jetty web server, which is also embeddable. You could bring up an embedded database like Derby or H2. You could configure the web server, database, or SymmetricDS to do whatever you needed, but it's also the most work of the three options discussed thus far.

The deployment model you choose depends on how much flexibility you need versus how easy you want it to be. Both Jetty and Tomcat are excellent, scalable web servers that compete with each other and have great performance. Most people choose either the Standalone or Web Archive with Tomcat 5.5 or 6. Deploying to Tomcat is a good middle-of-the-road decision that requires a little more work for more flexibility.

Next, we will go into a little more detail on the first three deployment options listed above.

5.3.1. Web Archive (WAR)

As a web application archive, a WAR is deployed to an application server, such as Tomcat, Jetty, or JBoss. The structure of the archive will have a web.xml file in the WEB-INF folder, an appropriately configured symmetric.properties file in the WEB-INF/classes folder, and the required JAR files in the WEB-INF/lib folder.

A war file can be generated using the standalone installation's symadmin utility and the create-war subcommand. The command requires the name of the war file to generate. It essentially packages up the web directory, the conf directory and includes an optional properties file. Note that if a properties file is included, it will be copied to WEB-INF/classes/symmetric.properties. This is the same location conf/symmetric.properties would have been copied to. The generated war distribution uses the same web.xml as the standalone deployment.

../bin/symadmin -p my-symmetric-ds.properties create-war /some/path/to/symmetric-ds.war

The web.base.servlet.path property in symmetric.properties can be set if the SymmetricServlet needs to coexist with other Servlets. By default, the value is blank. If you set it to, say, web.base.servlet.path=sync for example, registration.url would be http://server:port/sync.

5.3.2. Embedded

A Java application with the SymmetricDS Java Archive (JAR) library on its classpath can use the SymmetricWebServer to start the server.

import org.jumpmind.symmetric.SymmetricWebServer;

public class StartSymmetricEngine {

    public static void main(String[] args) throws Exception {

        SymmetricWebServer node = new SymmetricWebServer(
                                   "classpath://my-application.properties", "conf/web_dir");

        // this will create the database, sync triggers, start jobs running
        node.start(8080);

        // this will stop the node
        node.stop();
    }

}

This example starts the SymmetricDS server on port 8080. The configuration properties file, my-application.properties, is packaged in the application to provide properties that override the SymmetricDS default values. The second parameter to the constructor points to the web directory. The default location is ../web. In this example the web directory is located at conf/web_dir. The web.xml is expected to be found at conf/web_dir/WEB-INF/web.xml.

5.4. Standalone

The sym command line utility starts a standalone web server with SymmetricDS pre-deployed. The standalone server uses an embedded instance of the Jetty application server to handle web requests. The web server can be configured using command line options or the web server can be configured by changing properties in the conf/symmetric-server.properties file.

The following example starts the SymmetricDS server on port 8080 with the startup properties found in the root.properties file.

/symmetric/bin/sym --properties root.properties --port 8080 --server

Even though the port and properties settings can be passed in on the command line, the preferred configuration approach is to put each hosted node's properties file in the engines directory and to modify port settings and enable secure mode using the conf/symmetric-server.properties.

It is also suggested that SymmetricDS be configured to run as a service according to the instructions for your platform as documented in the following section.

5.5. Running SymmetricDS as a Service

SymmetricDS can be configured to start and run as a service in both Windows and *nix platforms.

5.5.1. Running as a Windows Service

SymmetricDS uses the Java Service Wrapper product from Tanuki Software to run in the background as a Windows system service. The Java Service Wrapper executable is named sym_service.exe so it can be easily identified from a list of running processes. To install the service, use the provided script:

bin\install_service.bat

The service configuration is found in conf/sym_service.conf. Edit this file if you want to change the default port number (8080), initial memory size (256 MB), log file size (10 MB), or other settings. When started, the server will look in the conf directory for the symmetric.properties file and the log4j.xml file. Logging for standard out, error, and application are written to the logs directory.

Most configuration changes do not require the service to be re-installed. To un-install the service, use the provided script:

bin\uninstall_service.bat

Use the net command to start and stop the service:

net start symmetricds
net stop symmetricds

5.5.2. Running as a *nix Service

SymmetricDS uses the 32 bit Java Service Wrapper product from Tanuki Software to run in the background as a Unix system service. The Java Service Wrapper executable is named sym_service so it can be easily identified from a list of running processes. The service configuration is found in conf/sym_service.conf. Edit this file if you want to change the initial memory size (256 MB), log file size (10 MB), or other settings.

If you happen to need a 64 bit wrapper, you'll need to download the Java Service Wrapper 64 bit version, then do the following:

  1. Unzip the download to some directory, say /tmp
  2. Change directory to the unzip location
  3. Run the following three commands:
    cp ./bin/wrapper /opt/symmetric-ds/bin/sym_service
    cp ./lib/wrapper.jar /opt/symmetric-ds/lib/wrapper.jar
    cp ./lib/libwrapper.so /opt/symmetric-ds/lib/libwrapper.so
                    
    where /opt/symmetric-ds should be replaced with wherever you have installed SymmetricDS.

An init script is provided to work with standard Unix run configuration levels. The sym_service.initd file follows the Linux Standard Base specification, which should work on many systems, including Fedora and Debian-based distributions. To install the script, copy it into the system init directory:

cp bin/sym_service.initd /etc/init.d/sym_service

Edit the init script to set the SYM_HOME variable to the directory where SymmetricDS is located. The init script calls the sym_service executable.

Enabling the service varies based on the version of Linux in use. Three possible approaches are listed below:

  1. Using chkconfig command:

    To enable the service to run automatically when the system is started:

    /sbin/chkconfig --add sym_service

    To disable the service from running automatically:

    /sbin/chkconfig --del sym_service

  2. Using install_initd command (CentOS Linux, Oracle Linux, SUSE Linux):

    On CentOS Linux, Oracle Linux, and SUSE Linux install the service by calling:

    /usr/lib/lsb/install_initd sym_service

    Remove the service by calling:

    /usr/lib/lsb/remove_initd sym_service

  3. Using sysv-rc-conf command (Ubuntu Linux):

    On Ubuntu Linux, you might need to use sysv-rc-conf instead of chkconfig. Try running sys-rc-conf as a super user (consider utilizing apt-get to install sysv-rc-conf if it is not present: sudo apt-get install sysv-rc-conf). Run sysv-rc-conf with the following command:

    sudo sysv-rc-conf 

    You should see a list of the scripts residing in your /etc/init.d folder. Use control-N to navigate through the list to locate sym_service, then activate the service for the desired run-levels (most likely 2-5).

Finally, you can use the service command to start, stop, and query the status of the service:

/sbin/service sym_service start
/sbin/service sym_service stop
/sbin/service sym_service status

Alternatively, call the init.d script directly:

/etc/init.d/sym_service start
/etc/init.d/sym_service stop
/etc/init.d/sym_service status
            

5.6. Clustering

A single SymmetricDS node may be clustered across a series of instances, creating a web farm. A node might be clustered to provide load balancing and failover, for example.

When clustered, a hardware load balancer is typically used to round robin client requests to the cluster. The load balancer should be configured for stateless connections. Also, the sync.url (discussed in Section 4.1, “Node Properties”) SymmetricDS property should be set to the URL of the load balancer.

If the cluster will be running any of the SymmetricDS jobs, then the cluster.lock.enabled property should be set to true. By setting this property to true, SymmetricDS will use a row in the LOCK table as a semaphore to make sure that only one instance at a time runs a job. When a lock is acquired, a row is updated in the lock table with the time of the lock and the server id of the locking job. The lock time is set back to null when the job is finished running. Another instance of SymmetricDS cannot aquire a lock until the locking instance (according to the server id) releases the lock. If an instance is terminated while the lock is still held, an instance with the same server id is allowed to reaquire the lock. If the locking instance remains down, the lock can be broken after a period of time, specified by the cluster.lock.timeout.ms property, has expired. Note that if the job is still running and the lock expires, two jobs could be running at the same time which could cause database deadlocks.

By default, the locking server id is the hostname of the server. If two clustered instances are running on the same server, then the cluster.server.id property may be set to indicate the name that the instance should use for its server id.

When deploying SymmetricDS to an application server like Tomcat or JBoss, no special session clustering needs to be configured for the application server.

5.7. Encrypted Passwords

The db.user and db.password properties will accept encrypted text, which protects against casual observation. The text is prefixed with enc: to indicate that it is encrypted. To encrypt text, use the following command:

symadmin -e {engine name} encrypt-text text-to-encrypt

or

symadmin -p {properties file} encrypt-text text-to-encrypt

The text is encrypted using a secret key named "sym.secret" that is retrieved from a keystore file. By default, the keystore is located in security/keystore. The location and filename of the keystore can be overridden by setting the "sym.keystore.file" system property. If the secret key is not found, the system will generate and install a secret key for use with Triple DES cipher.

Generate a new secret key for encryption using the keytool command that comes with the JRE. If there is an existing key in the keystore, first remove it:

keytool -keystore keystore -storepass changeit -storetype jceks \
   -alias sym.secret -delete

Then generate a secret key, specifying a cipher algorithm and key size. Commonly used algorithms that are supported include aes, blowfish, desede, and rc4.

keytool -keystore keystore -storepass changeit -storetype jceks \
   -alias sym.secret -genseckey -keyalg aes -keysize 128

If using an alternative provider, place the provider JAR file in the SymmetricDS lib folder. The provider class name should be installed in the JRE security properties or specified on the command line. To install in the JRE, edit the JRE lib/security/java.security file and set a security.provider.i property for the provider class name. Or, the provider can be specified on the command line instead. Both keytool and sym accept command line arguments for the provider class name. For example, using the Bouncy Castle provider, the command line options would look like:

keytool -keystore keystore -storepass changeit -storetype jceks \
   -alias sym.secret -genseckey -keyalg idea -keysize 56 \
   -providerClass org.bouncycastle.jce.provider.BouncyCastleProvider \
   -providerPath ..\lib\bcprov-ext.jar

symadmin -providerClass org.bouncycastle.jce.provider.BouncyCastleProvider -e secret

To customize the encryption, write a Java class that implements the ISecurityService or extends the default SecurityService, and place the class on the classpath in either lib or web/WEB-INF/lib folders. Then, in the symmetric.properties specify your class name for the security service.

security.service.class.name=org.jumpmind.security.SecurityService

Remember to specify your properties file when encrypting passwords, so it will use your custom ISecurityService.

symadmin -p symmetric.properties -e secret

5.8. Secure Transport

By specifying the "https" protocol for a URL, SymmetricDS will communicate over Secure Sockets Layer (SSL) for an encrypted transport. The following properties need to be set with "https" in the URL:

sync.url

This is the URL of the current node, so if you want to force other nodes to communicate over SSL with this node, you specify "https" in the URL.

registration.url

This is the URL where the node will connect for registration when it first starts up. To protect the registration with SSL, you specify "https" in the URL.

For incoming HTTPS connections, SymmetricDS depends on the webserver where it is deployed, so the webserver must be configured for HTTPS. As a standalone deployment, the "sym" launcher command provides options for enabling HTTPS support.

5.8.1. Sym Launcher

The "sym" launch command uses Jetty as an embedded web server. Using command line options, the web server can be told to listen for HTTP, HTTPS, or both.

sym --port 8080 --server

sym --secure-port 8443 --secure-server

sym --port 8080 --secure-port 8443 --mixed-server

5.8.2. Tomcat

If you deploy SymmetricDS to Apache Tomcat, it can be secured by editing the TOMCAT_HOME/conf/server.xml configuration file. There is already a line that can be uncommented and changed to the following:

<Connector port="8443" protocol="HTTP/1.1" SSLEnabled="true"
  maxThreads="150" scheme="https" secure="true"
  clientAuth="false" sslProtocol="TLS"
  keystoreFile="/symmetric-ds-1.x.x/security/keystore" />

5.8.3. Keystores

When SymmetricDS connects to a URL with HTTPS, Java checks the validity of the certificate using the built-in trusted keystore located at JRE_HOME/lib/security/cacerts. The "sym" launcher command overrides the trusted keystore to use its own trusted keystore instead, which is located at security/cacerts. This keystore contains the certificate aliased as "sym" for use in testing and easing deployments. The trusted keystore can be overridden by specifying the javax.net.ssl.trustStore system property.

When SymmetricDS is run as a secure server with the "sym" launcher, it accepts incoming requests using the key installed in the keystore located at security/keystore. The default key is provided for convenience of testing, but should be re-generated for security.

5.8.4. Generating Keys

To generate new keys and install a server certificate, use the following steps:

  1. Open a command prompt and navigate to the security subdirectory of your SymmetricDS installation on the server to which communication will be secured (typically the "root" or "central office" server).

  2. Delete the old key pair and certificate.

    keytool -keystore keystore -delete -alias sym

    keytool -keystore cacerts -delete -alias sym

    Enter keystore password:  changeit
  3. Generate a new key pair. Note that the first name/last name (the "CN") must match the fully qualified hostname the client will be using to communcate to the server.

    keytool -keystore keystore -alias sym -genkey -keyalg RSA -validity 10950

    Enter keystore password:  changeit
    What is your first and last name?
      [Unknown]:  localhost
    What is the name of your organizational unit?
      [Unknown]:  SymmetricDS
    What is the name of your organization?
      [Unknown]:  JumpMind
    What is the name of your City or Locality?
      [Unknown]:
    What is the name of your State or Province?
      [Unknown]:
    What is the two-letter country code for this unit?
      [Unknown]:
    Is CN=localhost, OU=SymmetricDS, O=JumpMind, L=Unknown, ST=Unknown, C=Unknown
    correct?
      [no]:  yes
    
    Enter key password for <sym>
            (RETURN if same as keystore password):
  4. Export the certificate from the private keystore.

    keytool -keystore keystore -export -alias sym -rfc -file sym.cer

  5. Install the certificate in the trusted keystore.

    keytool -keystore cacerts -import -alias sym -file sym.cer

  6. Copy the cacerts file that is generated by this process to the security directory of each client's SymmetricDS installation.

5.9. Basic Authentication

SymmetricDS supports basic authentication for client and server nodes.

To configure a client node to use basic authentication when communicating with a server node, specify the following startup parameters:

http.basic.auth.username

username for client node basic authentication. [ Default: ]

http.basic.auth.password

password for client node basic authentication. [ Default: ]

The SymmetricDS Standalone Web Server also supports Basic Authentication. It can be enabled by passing the following arguments to the startup program

--http-basic-auth-user

username for basic authentication [ Default: ]

--http-basic-auth-password

password for basic authentication [ Default: ]

If the server node is deployed to Tomcat or another application server as a WAR or EAR file, then basic authentication is setup with the standard configuration in the WEB.xml file.

5.10. Extension Points

SymmetricDS has a pluggable architecture that can be extended. A Java class that implements the appropriate extension point interface, can implement custom logic and change the behavior of SymmetricDS to suit special needs. All supported extension points extend the IExtensionPoint interface. The available extension points are documented in the following sections.

When SymmetricDS starts up, the ExtensionPointManager searches a Spring Framework context for classes that implement the IExtensionPoint interface, then creates and registers the class with the appropriate SymmetricDS component.

Extensions should be configured in the conf/symmetric-extensions.xml file as Spring beans. The jar file that contains the extension should be placed in the web/WEB-INF/lib directory.

If an extension point needs access to SymmetricDS services or needs to connect to the database it may implement the ISymmetricEngineAware interface in order to get a handle to the ISymmetricEngine.

The INodeGroupExtensionPoint interface may be optionally implemented to indicate that a registered extension point should only be registered with specific node groups.

/**
 * Only apply this extension point to the 'root' node group.
 */
 public String[] getNodeGroupIdsToApplyTo() {
     return new String[] { "root" };
 }

5.10.1. IParameterFilter

Parameter values can be specified in code using a parameter filter. Note that there can be only one parameter filter per engine instance. The IParameterFilter replaces the deprecated IRuntimeConfig from prior releases.

public class MyParameterFilter
    implements IParameterFilter, INodeGroupExtensionPoint {

    /**
     * Only apply this filter to stores
     */
    public String[] getNodeGroupIdsToApplyTo() {
        return new String[] { "store" };
    }

    public String filterParameter(String key, String value) {
        // look up a store number from an already existing properties file.
        if (key.equals(ParameterConstants.EXTERNAL_ID)) {
            return StoreProperties.getStoreProperties().
              getProperty(StoreProperties.STORE_NUMBER);
        }
        return value;
    }

    public boolean isAutoRegister() {
        return true;
    }

}

5.10.2. IDatabaseWriterFilter

Data can be filtered or manipulated before it is loaded into the target database. A filter can change the data in a column, save it somewhere else or do something else with the data entirely. It can also specify by the return value of the function call that the data loader should continue on and load the data (by returning true) or ignore it (by returning false). One possible use of the filter, for example, might be to route credit card data to a secure database and blank it out as it loads into a less-restricted reporting database.

A DataContext is passed to each of the callback methods. A new context is created for each synchronization. The context provides a mechanism to share data during the load of a batch between different rows of data that are committed in a single database transaction.

The filter also provides callback methods for the batch lifecycle. The DatabaseWriterFilterAdapter may be used if not all methods are required.

A class implementing the IDatabaseWriterFilter interface is injected onto the DataLoaderService in order to receive callbacks when data is inserted, updated, or deleted.

public class MyFilter extends DatabaseWriterFilterAdapter {

    @Override
    public boolean beforeWrite(DataContext context, Table table, CsvData data) {
        if (table.getName().equalsIgnoreCase("CREDIT_CARD_TENDER")
                && data.getDataEventType().equals(DataEventType.INSERT)) {
            String[] parsedData = data.getParsedData(CsvData.ROW_DATA);
            // blank out credit card number
            parsedData[table.getColumnIndex("CREDIT_CARD_NUMBER")] = null;
        }
        return true;
    }
}

The filter class should be specified in conf/symmetric-extensions.xml as follows.

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xmlns:context="http://www.springframework.org/schema/context"
    xsi:schemaLocation="http://www.springframework.org/schema/beans
           http://www.springframework.org/schema/beans/spring-beans-3.0.xsd
           http://www.springframework.org/schema/context
           http://www.springframework.org/schema/context/spring-context-3.0.xsd">

    <bean id="myFilter" class="com.mydomain.MyFilter"/>

</beans>

5.10.3. IDatabaseWriterErrorHandler

Implement this extension point to override how errors are handled. You can use this extension point to ignore rows that produce foreign key errors.

5.10.4. IDataLoaderFactory

Implement this extension point to provide a different implementation of the org.jumpmind.symmetric.io.data.IDataWriter that is used by the SymmetricDS data loader. Data loaders are configured for a channel. After this extension point is registered it can be activated for a CHANNEL by indicating the data loader name in the data_loader_type column.

SymmetricDS has two out of the box extensions of IDataLoaderFactory already implemented in its PostgresBulkDataLoaderFactory and OracleBulkDataLoaderFactory classes. These extension points implement bulk data loading capabilities for Oracle, Postgres and Greenplum dialects. See Appendix C. Database Notes for details.

Another possible use of this extension point is to route data to a NOSQL data sink.

5.10.5. IAcknowledgeEventListener

Implement this extension point to receive callback events when a batch is acknowledged. The callback for this listener happens at the point of extraction.

5.10.6. IReloadListener

Implement this extension point to listen in and take action before or after a reload is requested for a Node. The callback for this listener happens at the point of extraction.

5.10.7. ISyncUrlExtension

This extension point is used to select an appropriate URL based on the URI provided in the sync_url column of sym_node.

To use this extension point configure the sync_url for a node with the protocol of ext://beanName. The beanName is the name you give the extension point in the extension xml file.

5.10.8. IColumnTransform

This extension point allows custom column transformations to be created. There are a handful of out-of-the-box implementations. If any of these do not meet the column transformation needs of the application, then a custom transform can be created and registered. It can be activated by referencing the column transform's name transform_type column of TRANSFORM_COLUMN

5.10.9. INodeIdCreator

This extension point allows SymmetricDS users to implement their own algorithms for how node ids and passwords are generated or selected during the registration process. There may be only one node creator per SymmetricDS instance (Please note that the node creator extension has replaced the node generator extension).

5.10.10. ITriggerCreationListener

Implement this extension point to get status callbacks during trigger creation.

5.10.11. IBatchAlgorithm

Implement this extension point and set the name of the Spring bean on the batch_algorithm column of the Channel table to use. This extension point gives fine grained control over how a channel is batched.

5.10.12. IDataRouter

Implement this extension point and set the name of the Spring bean on the router_type column of the Router table to use. This extension point gives the ability to programmatically decide which nodes data should be routed to.

5.10.13. IHeartbeatListener

Implement this extension point to get callbacks during the heartbeat job.

5.10.14. IOfflineClientListener

Implement this extension point to get callbacks for offline events on client nodes.

5.10.15. IOfflineServerListener

Implement this extension point to get callbacks for offline events detected on a server node during monitoring of client nodes.

5.10.16. INodePasswordFilter

Implement this extension point to intercept the saving and rendering of the node password.

5.11. Synchronization to and from Android Devices

SymmetricDS now has its web-enabled, fault-tolerant, database synchronization software available on the Android mobile computing platform. The Android client follows all of the same concepts and brings to Android all of the same core SymmetricDS features as the full-featured, Java-based SymmetricDS client. The Android client is a little bit different in that it is not a stand-alone application, but is designed to be referenced as a library to run in-process with an Android application requiring synchronization for its SQLite database.

By using SymmetricDS, mobile application development is simplified, in that the mobile application developer can now focus solely on interacting with their local SQLite database. SymmetricDS takes care of capturing and moving data changes to and from a centralized database when the network is available

One of the main goals of the SymmetricDS 3 release was to support deployment to Android devices. SymmetricDS's overall footprint was reduced by eliminating a number of external dependencies in order to better deploy to Android. Also, the database access layer was abstracted so that the Android specific database access layer could be used. This allows SymmetricDS to be efficient in accessing the SQLite database on the Android device.

In order to convey how to use the SymmetricDS Android libraries, the example below will show how to integrate SymmetricDS into the NotePad sample application that comes with the Android ADK.

The NotePad sample application is a very simple task list application that persists “notes” to a SQLite database table called Notes. The Notes table will be configured to synchronize to a centralized version of the table. The Notes table looks as follows:

Column NameColumn Type
_ID(PK)INTEGER
TITLETEXT
NOTESTEXT
CREATEDINTEGER
MODIFIEDINTEGER

Table 5.1. Notes Table


Next, we will describe how to embed SymmetricDS in the NotePad application. Eclipse 3.7.2 and Android ADK 20.0.3 were used for this example.

To start,create the NotePad project. You do this by adding a new Android Sample Project. Select the NotePad project.

New Sample NotePad Project

Figure 5.1. New Sample NotePad Project


SymmetricDS for Android comes as a zip file of Java archives (jar files) that are required by the SymmetricDS client at runtime. This zip file ()symmetric-ds-3.4.7-android.zip) can be downloaded from the SymmetricDS.org website. The first step to using SymmetricDS in an Android application is to unzip the jar files into a location where the project will recognize them. The latest Android SDK and the Eclipse ADK requires that these jar files be put into a libs directory under the Android application project.

New Sample NotePad Project

Figure 5.2. New Sample NotePad Project


Unzip the symmetric-ds-x.x.x-android.zip file to the NotePad project directory. Refresh the NotePad project in Eclipse. You should end up with a libs directory that is automatically added to the Android Dependencies.

Jar Files Added to Libs

Figure 5.3. Jar Files Added to Libs


The Android version of the SymmetricDS engine is a Java class that can be instantiated directly or wired into an application via a provided Android service. Whether you are using the service or the engine directly you need to provide a few required startup parameters to the engine:

  • SQLiteOpenHelper – It is best (but not required) if the SQLiteOpenHelper is shared with the application that will be sharing the SQLite database. This core Android Java class provides software synchronization around the access to the database and minimizes locking errors.
  • registrationUrl – This is the URL of where the centralized SymmetricDS instance is hosted.
  • externalId – This is the identifier that can be used by the centralized SymmetricDS server to identify whether this instance should get data changes that happen on the server. It could be the serial number of the device, an account username, or some other business concept like store number.
  • nodeGroupId – This is the group id for the mobile device in the synchronization configuration. For example, if the nodeGroupId is 'handheld', then the SymmetricDS configuration might have a group called 'handheld' and a group called 'corp' where 'handheld' is configured to push and pull data from 'corp.'
  • properties – Optionally tweak the settings for SymmetricDS.

In order to integrate SymmetricDS into the NotePad application, the Android-specific SymmetricService will be used, and we need to tell the Android application this by adding the service to the AndroidManifest.xml file. Add the following snipped to the Manifest as the last entry under the <application> tag.

<service android:name="org.jumpmind.symmetric.android.SymmetricService" android:enabled="true" >           
    <intent-filter>
  		<action android:name="org.jumpmind.symmetric.android.SymmetricService" />
  	</intent-filter>
</service>
		

The other change required in the Manifest is to give the application permission to use the Internet. Add this as the first entry in the AndroidManifest.xml right before the <application> tag.

<uses-permission android:name="android.permission.INTERNET"></uses-permission> 
		

The only additional change needed is the call to start the service in the application. The service needs to be started manually because we need to give the application a chance to provide configuration information to the service.

In NotePadProvider.java add the following code snippet in the onCreate method.

NotePadProvider.java

Figure 5.4. NotePadProvider.java


		
final String HELPER_KEY = "NotePadHelperKey";

// Register the database helper, so it can be shared with the
SymmetricService
SQLiteOpenHelperRegistry.register(HELPER_KEY, mOpenHelper);

Intent intent = new Intent(getContext(), SymmetricService.class);

// Notify the service of the database helper key
intent.putExtra(SymmetricService.INTENTKEY_SQLITEOPENHELPER_REGISTRY_KEY,
HELPER_KEY);
intent.putExtra(SymmetricService.INTENTKEY_REGISTRATION_URL,
"http://10.0.2.2:31415/sync/server");
intent.putExtra(SymmetricService.INTENTKEY_EXTERNAL_ID,
"android-simulator");
intent.putExtra(SymmetricService.INTENTKEY_NODE_GROUP_ID, "client");
intent.putExtra(SymmetricService.INTENTKEY_START_IN_BACKGROUND,
true);

Properties properties = new Properties();
// initial load existing notes from the Client to the Server
properties.setProperty(ParameterConstants.AUTO_RELOAD_REVERSE_ENABLED,
"true");
intent.putExtra(SymmetricService.INTENTKEY_PROPERTIES, properties);

getContext().startService(intent);
			
		

This code snippet shows how the SQLiteOpenHelper is shared. The application's SQLiteOpenHelper is registered in a static registry provided by the SymmetricDS Android library. When the service is started, the key used to store the helper is passed to the service so that the service may pull the helper back out of the registry.

The various parameters needed by SymmetricDS are being set in the Intent which will be used by the SymmetricService to start the engine.

Most of the parameters will be familiar to SymmetricDS users. In this case a property is being set which will force an initial load of the existing Notes from the client to the server. This allows the user of the application to enter Notes for the first time offline or while the SymmetricDS engine is unregistered and still have them arrive at the centralized server once the SymmetricDS engine does get registered.

That's all you have to do to integrate SymmetricDS with your Android application.