| Description | Problem: Large LOBs are brought entirely into memory when pulled from Event Data. This puts limitations on the size of LOB to be synchronized. DSW has a need to synchronize files and we would like to add the enhancement for streaming LOB directly from the database and into the transport. We have observed that a 2MB file will increase the heap by 4MB when a LOB is being synchronized. Solution: For backwards compatibility, we will make this feature configurable (lob.stream) and its default value will be false. When set to false, the existing approach to LOBs will be utilized. When set to true, the dB function fn_sym_gen_lob_key will be templated for the following java.sql.Types: BLOB: java.sql.Types.BLOB: java.sql.Types.BINARY: java.sql.Types.VARBINARY: java.sql.Types.LONGVARBINARY: CLOB: java.sql.Types.CLOB: Code in SQLTemplate: . . case Types.BLOB: case Types.BINARY: case Types.VARBINARY: case Types.LONGVARBINARY: // SQL-Server ntext binary type case -10: if() templateToUse = blobColumnTemplate; . . The fn_sym_gen_lob_key function will have the following sugnature: CREATE OR REPLACE FUNCTION fn_sym_blob2clob (table_in IN VARCHAR2, lob_in IN VARCHAR2) RETURN VARCHAR2 Where table_in is the name of the table and lob_in is the name of the column that holds the LOB. The return value is the key to be used in the delimited event data. e.g ${[table].[lobColumn].[pkColumn1..n=pkValue]} This key will be used by the org.jumpmind.symmetric.extract.csv.Util class. If lob.steam is true and this key is encountered while parsing the event data, then it will retrive the data from the table in the key. Example: Key that was found in the event data: ${corp_int_data.file_data.file_data_id=10} (table name, column name, pk value) SQL statement to be executed: select data_file from corp_int_data where file_data_id=10; The stream from ResultSet:getInputStream(1), where 1 is the column index form the resultSet, will be written to the transport stream in Symmetric. |
|---|