View Issue Details

IDProjectCategoryView StatusLast Update
0005175SymmetricDSBugpublic2022-01-17 13:39
Reporterpmarzullo Assigned Topmarzullo  
Prioritynormal 
Status closedResolutionfixed 
Product Version3.12.14 
Target Version3.12.15Fixed in Version3.12.15 
Summary0005175: JdbcBatchBulkDatabaseWriter only uses one record in the batches for deletes, and fails on record 10000
DescriptionBatch commit siize is set to 10000, and the writer only writes one record at a time during delete event type.
The issue is that the check in DefaultDatabaseWriter.requireNewStatement() checks the passed in applyChangesOnly against the lastApplyChangesOnly member variable to see if it changed, and if it did, returns an indicator that a new statement must be created.

The delete() function that doesn't really use the applyChangesOnly variable, and passes in the passed in useConflictDetection variable, and never sets the lastApplyChangesOnly member variable, leaving it false. And the useConflictDetection passed in variable for bulk is usually set to true, and so ends up always requiring a new statement.

The fix is to create a requireNewStatement() method in the JdbcBatchBulkDatabaseWriter class that passes in a false for applyChangesOnly variable when the eventType is a delete. Very surgical fix.

This fixes the batching of deletes to use more than one delete at a time (actually uses the db.jdbc.bulk.execute.batch.size parameter value), and succeeds on all 10000 records.
Steps To ReproduceSet up a channel that uses jdbc bulk for change data capture.
Insert 10000 records.
Allow 10000 records go to the target.
Delete 10000 records.
The deletes will delete one at a time, and will fail on the 10000th record.
Tagsdata writer

Activities

pmarzullo

2021-12-28 20:49

developer   ~0002042

Merged into 3.13
Patches created for 3.12 and 3.13.

Related Changesets

SymmetricDS: 3.12 a0a8db16

2021-12-28 20:32:50

pmarzullo

Details Diff
0005175: JdbcBatchBulkDatabaseWriter only uses one record in the batches
for deletes, and fails on record 10000
Affected Issues
0005175
mod - symmetric-client/src/main/java/org/jumpmind/symmetric/io/JdbcBatchBulkDatabaseWriter.java Diff File

SymmetricDS: 3.13 dc759cf4

2021-12-28 20:32:50

pmarzullo

Details Diff
0005175: JdbcBatchBulkDatabaseWriter only uses one record in the batches
for deletes, and fails on record 10000
# Conflicts:
# symmetric-client/src/main/java/org/jumpmind/symmetric/io/JdbcBatchBulkDatabaseWriter.java
Affected Issues
0005175
mod - symmetric-client/src/main/java/org/jumpmind/symmetric/io/JdbcBatchBulkDatabaseWriter.java Diff File

SymmetricDS: 3.14 97af6973

2021-12-28 20:32:50

pmarzullo

Details Diff
0005175: JdbcBatchBulkDatabaseWriter only uses one record in the batches
for deletes, and fails on record 10000
# Conflicts:
# symmetric-client/src/main/java/org/jumpmind/symmetric/io/JdbcBatchBulkDatabaseWriter.java
Affected Issues
0005175
mod - symmetric-client/src/main/java/org/jumpmind/symmetric/io/JdbcBatchBulkDatabaseWriter.java Diff File

Issue History

Date Modified Username Field Change
2021-12-28 19:58 pmarzullo New Issue
2021-12-28 19:58 pmarzullo Status new => assigned
2021-12-28 19:58 pmarzullo Assigned To => pmarzullo
2021-12-28 19:58 pmarzullo Tag Attached: data writer
2021-12-28 20:31 pmarzullo Product Version 3.12.13 => 3.12.14
2021-12-28 20:49 pmarzullo Status assigned => resolved
2021-12-28 20:49 pmarzullo Resolution open => fixed
2021-12-28 20:49 pmarzullo Fixed in Version => 3.12.15
2021-12-28 20:49 pmarzullo Note Added: 0002042
2021-12-28 21:00 pmarzullo Changeset attached => SymmetricDS 3.12 a0a8db16
2021-12-28 21:00 pmarzullo Changeset attached => SymmetricDS 3.13 dc759cf4
2021-12-28 21:00 pmarzullo Changeset attached => SymmetricDS 3.14 97af6973
2022-01-17 13:39 admin Status resolved => closed