View Issue Details

IDProjectCategoryView StatusLast Update
0002526SymmetricDSImprovementpublic2016-08-09 13:10
ReporterelongAssigned Toelong 
PriorityhighSeveritymajorReproducibilityN/A
Status closedResolutionfixed 
Product Version3.8.0 
Target Version3.8.0Fixed in Version3.8.0 
Summary0002526: Improve performance of data gap detection
DescriptionImprove performance of data gap detection that manages the sym_data_gap table. The current method queries sym_data_event for each sym_data_gap. For expiring old gaps, it also queries sym_data. Instead, use the output of the work done by the router service on sym_data_event to drive the gap detection and expiration.
TagsNo tags attached.

Activities

elong

2016-04-07 10:25

developer   ~0000779

New parameters:

routing.stale.dataid.gap.time.ms=1200000
routing.use.fast.gap.detector=true

Old parameters:

routing.stale.dataid.gap.time.ms=7200000

Related Changesets

SymmetricDS: 3.8 3a215dcd

2016-03-17 14:05:53

admin

Details Diff
0002526: Improve performance of data gap detection
0002526
mod - symmetric-core/src/main/java/org/jumpmind/symmetric/route/ChannelRouterContext.java Diff File
mod - symmetric-core/src/main/java/org/jumpmind/symmetric/route/DataGapDetector.java Diff File
mod - symmetric-core/src/main/java/org/jumpmind/symmetric/route/DataGapRouteReader.java Diff File
mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/impl/RouterService.java Diff File
mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/impl/RouterServiceSqlMap.java Diff File
mod - symmetric-core/src/main/resources/symmetric-default.properties Diff File

SymmetricDS: 3.8 c43b9090

2016-03-17 14:06:13

admin

Details Diff
0002526: Improve performance of data gap detection
0002526
add - symmetric-core/src/main/java/org/jumpmind/symmetric/route/DataGapFastDetector.java Diff File

SymmetricDS: 3.8 b58dcc73

2016-03-17 14:06:49

admin

Details Diff
0002526: Improve performance of data gap detection
0002526
mod - symmetric-client/src/main/java/org/jumpmind/symmetric/db/oracle/OracleSymmetricDialect.java Diff File
mod - symmetric-core/src/main/java/org/jumpmind/symmetric/db/AbstractSymmetricDialect.java Diff File
mod - symmetric-core/src/main/java/org/jumpmind/symmetric/db/ISymmetricDialect.java Diff File

SymmetricDS: 3.8 3ee7172f

2016-04-05 09:20:19

admin

Details Diff
0002526: Improve performance of data gap detection
0002526
mod - symmetric-core/src/main/java/org/jumpmind/symmetric/AbstractSymmetricEngine.java Diff File
mod - symmetric-core/src/main/java/org/jumpmind/symmetric/ISymmetricEngine.java Diff File
add - symmetric-core/src/main/java/org/jumpmind/symmetric/common/ContextConstants.java Diff File
mod - symmetric-core/src/main/java/org/jumpmind/symmetric/common/ParameterConstants.java Diff File
mod - symmetric-core/src/main/java/org/jumpmind/symmetric/common/TableConstants.java Diff File
mod - symmetric-core/src/main/java/org/jumpmind/symmetric/route/DataGapFastDetector.java Diff File
add - symmetric-core/src/main/java/org/jumpmind/symmetric/service/IContextService.java Diff File
add - symmetric-core/src/main/java/org/jumpmind/symmetric/service/impl/ContextService.java Diff File
add - symmetric-core/src/main/java/org/jumpmind/symmetric/service/impl/ContextServiceSqlMap.java Diff File
mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/impl/RouterService.java Diff File
mod - symmetric-core/src/main/resources/symmetric-schema.xml Diff File

SymmetricDS: 3.8 f1d07e40

2016-04-05 09:58:10

admin

Details Diff
0002526: Improve performance of data gap detection
Updated SQLite dialect to use context service

0002526
mod - symmetric-android/src/main/java/org/jumpmind/symmetric/android/AndroidSymmetricEngine.java Diff File
mod - symmetric-client/src/main/java/org/jumpmind/symmetric/ClientSymmetricEngine.java Diff File
mod - symmetric-client/src/main/java/org/jumpmind/symmetric/db/JdbcSymmetricDialectFactory.java Diff File
mod - symmetric-client/src/main/java/org/jumpmind/symmetric/db/sqlite/SqliteJdbcSymmetricDialect.java Diff File
mod - symmetric-core/src/main/java/org/jumpmind/symmetric/db/sqlite/SqliteSymmetricDialect.java Diff File
mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/IContextService.java Diff File
mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/impl/ContextService.java Diff File
mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/impl/ContextServiceSqlMap.java Diff File

SymmetricDS: 3.8 a2750110

2016-04-07 10:26:38

admin

Details Diff
0002526: Improve performance of data gap detection
0002526
mod - symmetric-core/src/main/resources/symmetric-default.properties Diff File

SymmetricDS: 3.8 7b82f2f1

2016-04-08 11:22:16

admin

Details Diff
0002526: Improve performance of data gap detection
0002526
mod - symmetric-core/src/test/java/org/jumpmind/symmetric/route/DataGapDetectorTest.java Diff File

SymmetricDS: 3.8 bc148280

2016-04-08 13:35:41

admin

Details Diff
0002526: Improve performance of data gap detection
0002526
mod - symmetric-core/src/main/java/org/jumpmind/symmetric/route/DataGapFastDetector.java Diff File
mod - symmetric-core/src/test/java/org/jumpmind/symmetric/route/DataGapDetectorTest.java Diff File

SymmetricDS: 3.8 96aeefa8

2016-04-15 13:03:50

admin

Details Diff
0002526: Improve performance of data gap detection
0002526
mod - symmetric-core/src/main/java/org/jumpmind/symmetric/route/DataGapFastDetector.java Diff File
mod - symmetric-core/src/test/java/org/jumpmind/symmetric/route/DataGapDetectorTest.java Diff File

SymmetricDS: 3.8 d0cd6323

2016-04-21 11:26:27

admin

Details Diff
0002526: Improve performance of data gap detection
More aggressive prevention of duplicate/overlapping gaps by checking
against existing gaps not just new gaps

0002526
mod - symmetric-core/src/main/java/org/jumpmind/symmetric/route/DataGapFastDetector.java Diff File
mod - symmetric-core/src/test/java/org/jumpmind/symmetric/route/DataGapDetectorTest.java Diff File

SymmetricDS: 3.8 e6a31c55

2016-04-22 12:25:12

admin

Details Diff
0002526: Improve performance of data gap detection
Delay expiration of gaps on a busy system.

0002526
mod - symmetric-core/src/main/java/org/jumpmind/symmetric/common/ContextConstants.java Diff File
mod - symmetric-core/src/main/java/org/jumpmind/symmetric/common/ParameterConstants.java Diff File
mod - symmetric-core/src/main/java/org/jumpmind/symmetric/route/DataGapFastDetector.java Diff File
mod - symmetric-core/src/main/resources/symmetric-default.properties Diff File
mod - symmetric-core/src/test/java/org/jumpmind/symmetric/route/DataGapDetectorTest.java Diff File

SymmetricDS: 3.7 9ceb7576

2016-04-28 09:29:02

admin

Details Diff
0002526: Improve performance of data gap detection
0002526
mod - symmetric-client/src/main/java/org/jumpmind/symmetric/db/oracle/OracleSymmetricDialect.java Diff File

SymmetricDS: 3.8 0f8f7e25

2016-05-23 12:04:34

admin

Details Diff
0002526: Improve performance of data gap detection
snapshot util should not insert last gap

0002526
mod - symmetric-client/src/main/java/org/jumpmind/symmetric/util/SnapshotUtil.java Diff File

SymmetricDS: 3.8 39e8457b

2016-05-24 11:00:36

admin

Details Diff
0002526: Improve performance of data gap detection
add routine that fixes overlapping gaps

0002526
mod - symmetric-core/src/main/java/org/jumpmind/symmetric/route/DataGapFastDetector.java Diff File
mod - symmetric-core/src/test/java/org/jumpmind/symmetric/route/DataGapDetectorTest.java Diff File

SymmetricDS: 3.8 da8519cf

2016-05-26 15:55:36

admin

Details Diff
0002526: Improve performance of data gap detection
changes from code review

0002526
mod - symmetric-core/src/main/java/org/jumpmind/symmetric/route/ConfigurationChangedDataRouter.java Diff File
mod - symmetric-core/src/main/java/org/jumpmind/symmetric/route/DataGapFastDetector.java Diff File
mod - symmetric-core/src/main/java/org/jumpmind/symmetric/route/SimpleRouterContext.java Diff File
mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/impl/RouterService.java Diff File

SymmetricDS: 3.8 a8f859c6

2016-07-19 13:10:14

admin

Details Diff
0002526: Improve performance of data gap detection
don't query for database time on each routing run

0002526
mod - symmetric-core/src/main/java/org/jumpmind/symmetric/model/DataGap.java Diff File
mod - symmetric-core/src/main/java/org/jumpmind/symmetric/route/DataGapFastDetector.java Diff File
mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/impl/DataService.java Diff File
mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/impl/DataServiceSqlMap.java Diff File
mod - symmetric-core/src/test/java/org/jumpmind/symmetric/route/DataGapDetectorTest.java Diff File

SymmetricDS: 3.8 db5c9ffd

2016-07-20 20:28:31

admin

Details Diff
0002526: Improve performance of data gap detection
avoid context service calls when running non-clustered

0002526
mod - symmetric-core/src/main/java/org/jumpmind/symmetric/route/DataGapDetector.java Diff File
mod - symmetric-core/src/main/java/org/jumpmind/symmetric/route/DataGapFastDetector.java Diff File
mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/impl/RouterService.java Diff File
mod - symmetric-core/src/main/resources/symmetric-default.properties Diff File
mod - symmetric-core/src/test/java/org/jumpmind/symmetric/route/DataGapDetectorTest.java Diff File

SymmetricDS: 3.8 21833404

2016-07-21 14:07:33

admin

Details Diff
0002526: Improve performance of data gap detection
on non-cluster system, keep gaps in memory between calls

0002526
mod - symmetric-core/src/main/java/org/jumpmind/symmetric/model/DataGap.java Diff File
mod - symmetric-core/src/main/java/org/jumpmind/symmetric/route/DataGapFastDetector.java Diff File
mod - symmetric-core/src/test/java/org/jumpmind/symmetric/route/DataGapDetectorTest.java Diff File

SymmetricDS: 3.8 1f8632bd

2016-07-21 16:56:21

admin

Details Diff
0002526: Improve performance of data gap detection
try to get unit tests working, which init engine differently

0002526
mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/impl/RouterService.java Diff File

SymmetricDS: 3.8 3c8e0dcf

2016-07-26 15:50:48

admin

Details Diff
0002526: Improve performance of data gap detection
fix problem with the process info

0002526
mod - symmetric-core/src/main/java/org/jumpmind/symmetric/route/DataGapFastDetector.java Diff File

SymmetricDS: 3.8 57c9666f

2016-07-29 10:37:01

admin

Details Diff
0002526: Improve performance of data gap detection
0002526
mod - symmetric-core/src/main/java/org/jumpmind/symmetric/common/ParameterConstants.java Diff File
mod - symmetric-core/src/main/java/org/jumpmind/symmetric/route/DataGapFastDetector.java Diff File
mod - symmetric-core/src/main/resources/symmetric-default.properties Diff File
mod - symmetric-core/src/test/java/org/jumpmind/symmetric/route/DataGapDetectorTest.java Diff File

SymmetricDS: 3.8 20497ce9

2016-08-02 08:29:50

admin

Details Diff
0002526: Improve performance of data gap detection,
separate processing of gaps from the insert/deletes to the database with
logging to show timings of each

0002526
mod - symmetric-core/src/main/java/org/jumpmind/symmetric/route/DataGapFastDetector.java Diff File
mod - symmetric-core/src/test/java/org/jumpmind/symmetric/route/DataGapDetectorTest.java Diff File

SymmetricDS: 3.8 2cad25cd

2016-08-02 10:46:46

admin

Details Diff
0002526: Improve performance of data gap detection,
add logging for timing of mapping the data ids into the gaps

0002526
mod - symmetric-core/src/main/java/org/jumpmind/symmetric/route/DataGapFastDetector.java Diff File

SymmetricDS: 3.8 c10a7caf

2016-08-04 16:02:30

admin

Details Diff
0002526: Improve performance of data gap detection,
use in-memory gap detection on non-clustered system when there is a
large number of changes to sym_data_gap

0002526
mod - symmetric-core/src/main/java/org/jumpmind/symmetric/common/ParameterConstants.java Diff File
mod - symmetric-core/src/main/java/org/jumpmind/symmetric/route/DataGapFastDetector.java Diff File
mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/IDataService.java Diff File
mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/impl/DataService.java Diff File
mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/impl/DataServiceSqlMap.java Diff File
mod - symmetric-core/src/main/resources/symmetric-default.properties Diff File
mod - symmetric-core/src/test/java/org/jumpmind/symmetric/route/DataGapDetectorTest.java Diff File

SymmetricDS: 3.8 ca476d13

2016-08-08 10:58:55

admin

Details Diff
0002526: Improve performance of data gap detection,
increase time we wait before checking if gaps are in sym_data on a busy
system

0002526
mod - symmetric-core/src/main/resources/symmetric-default.properties Diff File

Issue History

Date Modified Username Field Change
2016-03-17 13:43 elong New Issue
2016-03-17 13:43 elong Status new => assigned
2016-03-17 13:43 elong Assigned To => elong
2016-04-07 10:25 elong Note Added: 0000779
2016-04-15 13:04 elong Status assigned => resolved
2016-04-15 13:04 elong Fixed in Version => 3.8.0
2016-04-15 13:04 elong Resolution open => fixed
2016-04-28 10:01 admin Changeset attached => SymmetricDS 3.7 9ceb7576
2016-04-28 10:01 admin Changeset attached => SymmetricDS 3.8 e6a31c55
2016-04-28 10:01 admin Changeset attached => SymmetricDS 3.8 d0cd6323
2016-04-28 10:01 admin Changeset attached => SymmetricDS 3.8 96aeefa8
2016-04-28 10:01 admin Changeset attached => SymmetricDS 3.8 bc148280
2016-04-28 10:01 admin Changeset attached => SymmetricDS 3.8 7b82f2f1
2016-04-28 10:01 admin Changeset attached => SymmetricDS 3.8 a2750110
2016-04-28 10:01 admin Changeset attached => SymmetricDS 3.8 f1d07e40
2016-04-28 10:01 admin Changeset attached => SymmetricDS 3.8 3ee7172f
2016-04-28 10:01 admin Changeset attached => SymmetricDS 3.8 b58dcc73
2016-04-28 10:01 admin Changeset attached => SymmetricDS 3.8 c43b9090
2016-04-28 10:01 admin Changeset attached => SymmetricDS 3.8 3a215dcd
2016-05-23 13:00 admin Changeset attached => SymmetricDS 3.8 0f8f7e25
2016-05-24 12:00 admin Changeset attached => SymmetricDS 3.8 39e8457b
2016-05-27 13:00 admin Changeset attached => SymmetricDS 3.8 da8519cf
2016-07-21 16:00 admin Changeset attached => SymmetricDS 3.8 21833404
2016-07-21 16:00 admin Changeset attached => SymmetricDS 3.8 db5c9ffd
2016-07-21 16:00 admin Changeset attached => SymmetricDS 3.8 a8f859c6
2016-07-21 17:00 admin Changeset attached => SymmetricDS 3.8 1f8632bd
2016-07-26 22:00 admin Changeset attached => SymmetricDS 3.8 3c8e0dcf
2016-07-29 11:00 admin Changeset attached => SymmetricDS 3.8 57c9666f
2016-08-02 09:00 admin Changeset attached => SymmetricDS 3.8 20497ce9
2016-08-02 11:00 admin Changeset attached => SymmetricDS 3.8 2cad25cd
2016-08-04 17:00 admin Changeset attached => SymmetricDS 3.8 c10a7caf
2016-08-08 11:00 admin Changeset attached => SymmetricDS 3.8 ca476d13
2016-08-09 13:10 admin Status resolved => closed