View Issue Details

IDProjectCategoryView StatusLast Update
0003341SymmetricDSImprovementpublic2018-02-14 15:08
ReportermmichalekAssigned Tommichalek 
Prioritynormal 
Status closedResolutionfixed 
Product Version3.8.33 
Target Version3.9.1Fixed in Version3.9.1 
Summary0003341: Improve PostgreSQL bulk loader to tolerate NULL bytes in incoming data
DescriptionThe PostgreSQL bulk loader will now filter out NULL bytes from string data before loading.

This prevents errors like the following when there are null bytes in the data. See the attached insert statement for a simple statement that will reproduce the issue using the demo data set.

2017-12-20 13:54:58,122 ERROR [store-003] [DataLoaderService] [store-003-dataloader-6] Failed to load batch 000-30 StackTraceKey.init [SqlException:2182076017]
org.jumpmind.db.sql.SqlException: ERROR: invalid byte sequence for encoding "UTF8": 0x00
Where: COPY item, line 1
     at org.jumpmind.db.sql.AbstractSqlTemplate.translate(AbstractSqlTemplate.java:300)
     at org.jumpmind.db.sql.AbstractSqlTemplate.translate(AbstractSqlTemplate.java:291)
     at org.jumpmind.symmetric.io.PostgresBulkDatabaseWriter.endCopy(PostgresBulkDatabaseWriter.java:184)
     at org.jumpmind.symmetric.io.PostgresBulkDatabaseWriter.end(PostgresBulkDatabaseWriter.java:200)
     at org.jumpmind.symmetric.io.data.writer.NestedDataWriter.end(NestedDataWriter.java:68)
     at org.jumpmind.symmetric.io.data.writer.NestedDataWriter.end(NestedDataWriter.java:68)
     at org.jumpmind.symmetric.io.data.writer.TransformWriter.end(TransformWriter.java:471)
     at org.jumpmind.symmetric.io.data.DataProcessor.forEachTableInBatch(DataProcessor.java:176)
     at org.jumpmind.symmetric.io.data.DataProcessor.process(DataProcessor.java:116)
     at org.jumpmind.symmetric.service.impl.DataLoaderService$LoadIntoDatabaseOnArrivalListener$1.call(DataLoaderService.java:988)
     at org.jumpmind.symmetric.service.impl.DataLoaderService$LoadIntoDatabaseOnArrivalListener$1.call(DataLoaderService.java:1)
     at java.util.concurrent.FutureTask.run(FutureTask.java:262)
     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
     at java.lang.Thread.run(Thread.java:745)
Caused by: org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding "UTF8": 0x00
Where: COPY item, line 1
     at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2455)
     at org.postgresql.core.v3.QueryExecutorImpl.processCopyResults(QueryExecutorImpl.java:1049)
     at org.postgresql.core.v3.QueryExecutorImpl.endCopy(QueryExecutorImpl.java:892)
     at org.postgresql.core.v3.CopyInImpl.endCopy(CopyInImpl.java:40)
     at org.jumpmind.symmetric.io.PostgresBulkDatabaseWriter.endCopy(PostgresBulkDatabaseWriter.java:178)
     ... 12 more
2017-12-20 13:54:58,128 ERROR [corp-000] [AcknowledgeService] [qtp1262353316-300] The outgoing batch 003-30 failed: ERROR: invalid byte sequence for encoding "UTF8": 0x00
Where: COPY item, line 1

Activities

mmichalek

2017-12-20 14:06

administrator  

insert-null-bytes.sql (111 bytes)

mmichalek

2017-12-20 14:19

administrator   ~0001089

There is a patch for 3.8.33 at: https://www.jumpmind.com/downloads/symmetricds/patches/3.8/patch-0003341.jar

Related Changesets

SymmetricDS: 3.8 70b80988

2017-12-20 14:09:23

mmichalek

Details Diff
0003341: Improve PostgreSQL bulk loader to tolerate NULL bytes in
incoming data

0003341
mod - symmetric-client/src/main/java/org/jumpmind/symmetric/io/PostgresBulkDatabaseWriter.java Diff File

SymmetricDS: 3.9 515e3eb0

2017-12-20 14:09:23

mmichalek

Details Diff
0003341: Improve PostgreSQL bulk loader to tolerate NULL bytes in
incoming data

0003341
mod - symmetric-client/src/main/java/org/jumpmind/symmetric/io/PostgresBulkDatabaseWriter.java Diff File

Issue History

Date Modified Username Field Change
2017-12-20 14:06 mmichalek New Issue
2017-12-20 14:06 mmichalek Status new => assigned
2017-12-20 14:06 mmichalek Assigned To => mmichalek
2017-12-20 14:06 mmichalek File Added: insert-null-bytes.sql
2017-12-20 14:17 mmichalek Status assigned => resolved
2017-12-20 14:17 mmichalek Resolution open => fixed
2017-12-20 14:17 mmichalek Fixed in Version => 3.9.1
2017-12-20 14:19 mmichalek Note Added: 0001089
2017-12-20 15:00 mmichalek Changeset attached => SymmetricDS 3.8 70b80988
2017-12-22 15:06 mmichalek Status resolved => closed
2018-02-14 15:08 mmichalek Changeset attached => SymmetricDS 3.9 515e3eb0