The SymmetricDS Data Format is used to stream data from one node to another. The data format reader and writer are pluggable with an initial implementation using a format based on Comma Separated Values (CSV). Each line in the stream is a record with fields separated by commas. String fields are surrounded with double quotes. Double quotes and backslashes used in a string field are escaped with a backslash. Binary values are represented as a string with hex values in "\0xab" format. The absence of any value in the field indicates a null value. Extra spacing is ignored and lines starting with a hash are ignored.
The first field of each line gives the directive for the line. The following directives are used:
Identifies which node the data is coming from. Occurs once in CSV file.
Identifies the type of decoding the loader needs to use to decode binary data in the pay load. This varies depending on what database is the source of the data.
Identifies which channel a batch belongs to. The SymmetricDS data loader expects the channel to be specified before the batch.
Uniquely identifies a batch. Used to track whether a batch has been loaded before. A batch of -9999 is considered a virtual batch and will be loaded, but will not be recorded in incoming_batch.
The name of the schema that is being targeted.
The name of the catalog that is being targeted.
The name of the table that is being targeted.
Lists the column names that are used as the primary key for the table. Only needs to occur after the first occurrence of the table.
Lists all the column names (including key columns) of the table. Only needs to occur after the first occurrence of the table.
Insert into the table with the values that correspond with the columns.
Update the table using the old key values to set the new column values.
Represent all the old values of the data. This data can be used for conflict resolution.
Delete from the table using the old key values.
Optional notation that instructs the data loader to run the accompanying SQL statement.
Optional notation that instructs the data loader to run the accompanying BeanShell snippet.
Optional notation that instructs the data loader to run the accompanying DdlUtils XML table definition in order to create a database table.
An indicator that the batch has been transmitted and the data can be committed to the database.
nodeid, 1001 channel, pricing binary, BASE64 batch, 100 schema, catalog, table, item_selling_price keys, price_id columns, price_id, price, cost insert, 55, 0.65, 0.55 schema, catalog, table, item keys, item_id columns, item_id, price_id, name insert, 110000055, 55, "Soft Drink" delete, 110000001 schema, catalog, table, item_selling_price update, 55, 0.75, 0.65, 55 commit, 100
Example D.1. Data Format Stream