View Issue Details

IDProjectCategoryView StatusLast Update
0004493SymmetricDSBugpublic2020-09-01 08:20
Reporterelong Assigned Toelong  
Prioritynormal 
Status closedResolutionfixed 
Product Version3.12.0 
Target Version3.12.3Fixed in Version3.12.3 
Summary0004493: Transforms start and end table for each row causing poor performance with bulk data loading
DescriptionThe transform writer incorrect detects a new table definition is being used for each row of a batch, so it signals the end and start of a table for each row. The bulk data loaders usually wait for the end of the table to perform bulk operations, so this causes poor performance. It would also hurt the performance somewhat of the default data loader for SQL Server when identity columns are used, because it will constantly turn on/off the identity insert.

The table change detection is case sensitive, so transforms using different cases will cause a difference. The detection also checks data types, sizes, and type mappings. While debugging with SQL-Server, it was observed that types are populated for one table object and not the other, causing a difference to be detected. I wasn't able to find the root cause in the code history, but this issue was first detected in 3.5.22 in 0001735 which was expected to fix it.
Tagsbulk

Relationships

related to 0001735 closedchenson Transforms write out the start of a table before every row which is inefficient and causes bulk loaders to become unusable 

Activities

There are no notes attached to this issue.

Related Changesets

SymmetricDS: 3.12 faf8d16d

2020-08-07 10:28:00

admin

Details Diff
0004493: Transforms start and end table for each row causing poor
performance with bulk data loading
Affected Issues
0004493
mod - symmetric-db/src/main/java/org/jumpmind/db/model/Column.java Diff File
mod - symmetric-db/src/main/java/org/jumpmind/db/model/Table.java Diff File
mod - symmetric-io/src/main/java/org/jumpmind/symmetric/io/data/writer/TransformWriter.java Diff File

SymmetricDS: 3.12 8ec66151

2020-08-07 10:28:57

admin

Details Diff
0004493: better debug logging Affected Issues
0004493
mod - symmetric-client/src/main/java/org/jumpmind/symmetric/io/MsSqlBulkDatabaseWriter.java Diff File

SymmetricDS: 3.12 98f70ba9

2020-08-12 14:39:49

admin

Details Diff
0004493: Transforms start and end table for each row causing poor
performance with bulk data loading
Affected Issues
0004493
mod - symmetric-client/src/main/java/org/jumpmind/symmetric/io/MsSqlBulkDatabaseWriter.java Diff File

SymmetricDS: 3.12 504574f2

2020-08-26 15:07:46

admin

Details Diff
0004493: Transforms start and end table for each row causing poor
performance with bulk data loading
Affected Issues
0004493
mod - symmetric-db/src/main/java/org/jumpmind/db/model/Table.java Diff File

Issue History

Date Modified Username Field Change
2020-08-07 09:02 elong New Issue
2020-08-07 09:02 elong Status new => assigned
2020-08-07 09:02 elong Assigned To => elong
2020-08-07 09:02 elong Tag Attached: bulk
2020-08-07 09:24 elong Relationship added related to 0001735
2020-08-07 10:29 elong Status assigned => resolved
2020-08-07 10:29 elong Resolution open => fixed
2020-08-07 10:29 elong Fixed in Version => 3.12.3
2020-08-07 11:00 admin Changeset attached => SymmetricDS 3.12 8ec66151
2020-08-07 11:00 admin Changeset attached => SymmetricDS 3.12 faf8d16d
2020-08-12 15:00 admin Changeset attached => SymmetricDS 3.12 98f70ba9
2020-08-26 16:00 admin Changeset attached => SymmetricDS 3.12 504574f2
2020-09-01 08:20 elong Status resolved => closed