0002526: Improve performance of data gap detection - SymmetricDS

ID	Project	Category	View Status	Date Submitted	Last Update

0002526	SymmetricDS	Improvement	public	2016-03-17 17:43	2016-08-09 17:10

Reporter	elong	Assigned To	elong
Priority	high
Status	closed	Resolution	fixed
Product Version	3.8.0
Target Version	3.8.0	Fixed in Version	3.8.0

Summary	0002526: Improve performance of data gap detection
Description	Improve performance of data gap detection that manages the sym_data_gap table. The current method queries sym_data_event for each sym_data_gap. For expiring old gaps, it also queries sym_data. Instead, use the output of the work done by the router service on sym_data_event to drive the gap detection and expiration.
Tags	No tags attached.

SymmetricDS: 3.8 3a215dcd 2016-03-17 14:05:53 admin Details Diff	0002526: Improve performance of data gap detection	Affected Issues 0002526
	mod - symmetric-core/src/main/java/org/jumpmind/symmetric/route/ChannelRouterContext.java	Diff File
	mod - symmetric-core/src/main/java/org/jumpmind/symmetric/route/DataGapDetector.java	Diff File
	mod - symmetric-core/src/main/java/org/jumpmind/symmetric/route/DataGapRouteReader.java	Diff File
	mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/impl/RouterService.java	Diff File
	mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/impl/RouterServiceSqlMap.java	Diff File
	mod - symmetric-core/src/main/resources/symmetric-default.properties	Diff File
SymmetricDS: 3.8 c43b9090 2016-03-17 14:06:13 admin Details Diff	0002526: Improve performance of data gap detection	Affected Issues 0002526
	add - symmetric-core/src/main/java/org/jumpmind/symmetric/route/DataGapFastDetector.java	Diff File
SymmetricDS: 3.8 b58dcc73 2016-03-17 14:06:49 admin Details Diff	0002526: Improve performance of data gap detection	Affected Issues 0002526
	mod - symmetric-client/src/main/java/org/jumpmind/symmetric/db/oracle/OracleSymmetricDialect.java	Diff File
	mod - symmetric-core/src/main/java/org/jumpmind/symmetric/db/AbstractSymmetricDialect.java	Diff File
	mod - symmetric-core/src/main/java/org/jumpmind/symmetric/db/ISymmetricDialect.java	Diff File
SymmetricDS: 3.8 3ee7172f 2016-04-05 09:20:19 admin Details Diff	0002526: Improve performance of data gap detection	Affected Issues 0002526
	mod - symmetric-core/src/main/java/org/jumpmind/symmetric/AbstractSymmetricEngine.java	Diff File
	mod - symmetric-core/src/main/java/org/jumpmind/symmetric/ISymmetricEngine.java	Diff File
	add - symmetric-core/src/main/java/org/jumpmind/symmetric/common/ContextConstants.java	Diff File
	mod - symmetric-core/src/main/java/org/jumpmind/symmetric/common/ParameterConstants.java	Diff File
	mod - symmetric-core/src/main/java/org/jumpmind/symmetric/common/TableConstants.java	Diff File
	mod - symmetric-core/src/main/java/org/jumpmind/symmetric/route/DataGapFastDetector.java	Diff File
	add - symmetric-core/src/main/java/org/jumpmind/symmetric/service/IContextService.java	Diff File
	add - symmetric-core/src/main/java/org/jumpmind/symmetric/service/impl/ContextService.java	Diff File
	add - symmetric-core/src/main/java/org/jumpmind/symmetric/service/impl/ContextServiceSqlMap.java	Diff File
	mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/impl/RouterService.java	Diff File
	mod - symmetric-core/src/main/resources/symmetric-schema.xml	Diff File
SymmetricDS: 3.8 f1d07e40 2016-04-05 09:58:10 admin Details Diff	0002526: Improve performance of data gap detection Updated SQLite dialect to use context service	Affected Issues 0002526
	mod - symmetric-android/src/main/java/org/jumpmind/symmetric/android/AndroidSymmetricEngine.java	Diff File
	mod - symmetric-client/src/main/java/org/jumpmind/symmetric/ClientSymmetricEngine.java	Diff File
	mod - symmetric-client/src/main/java/org/jumpmind/symmetric/db/JdbcSymmetricDialectFactory.java	Diff File
	mod - symmetric-client/src/main/java/org/jumpmind/symmetric/db/sqlite/SqliteJdbcSymmetricDialect.java	Diff File
	mod - symmetric-core/src/main/java/org/jumpmind/symmetric/db/sqlite/SqliteSymmetricDialect.java	Diff File
	mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/IContextService.java	Diff File
	mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/impl/ContextService.java	Diff File
	mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/impl/ContextServiceSqlMap.java	Diff File
SymmetricDS: 3.8 a2750110 2016-04-07 10:26:38 admin Details Diff	0002526: Improve performance of data gap detection	Affected Issues 0002526
	mod - symmetric-core/src/main/resources/symmetric-default.properties	Diff File
SymmetricDS: 3.8 7b82f2f1 2016-04-08 11:22:16 admin Details Diff	0002526: Improve performance of data gap detection	Affected Issues 0002526
	mod - symmetric-core/src/test/java/org/jumpmind/symmetric/route/DataGapDetectorTest.java	Diff File
SymmetricDS: 3.8 bc148280 2016-04-08 13:35:41 admin Details Diff	0002526: Improve performance of data gap detection	Affected Issues 0002526
	mod - symmetric-core/src/main/java/org/jumpmind/symmetric/route/DataGapFastDetector.java	Diff File
	mod - symmetric-core/src/test/java/org/jumpmind/symmetric/route/DataGapDetectorTest.java	Diff File
SymmetricDS: 3.8 96aeefa8 2016-04-15 13:03:50 admin Details Diff	0002526: Improve performance of data gap detection	Affected Issues 0002526
	mod - symmetric-core/src/main/java/org/jumpmind/symmetric/route/DataGapFastDetector.java	Diff File
	mod - symmetric-core/src/test/java/org/jumpmind/symmetric/route/DataGapDetectorTest.java	Diff File
SymmetricDS: 3.8 d0cd6323 2016-04-21 11:26:27 admin Details Diff	0002526: Improve performance of data gap detection More aggressive prevention of duplicate/overlapping gaps by checking against existing gaps not just new gaps	Affected Issues 0002526
	mod - symmetric-core/src/main/java/org/jumpmind/symmetric/route/DataGapFastDetector.java	Diff File
	mod - symmetric-core/src/test/java/org/jumpmind/symmetric/route/DataGapDetectorTest.java	Diff File
SymmetricDS: 3.8 e6a31c55 2016-04-22 12:25:12 admin Details Diff	0002526: Improve performance of data gap detection Delay expiration of gaps on a busy system.	Affected Issues 0002526
	mod - symmetric-core/src/main/java/org/jumpmind/symmetric/common/ContextConstants.java	Diff File
	mod - symmetric-core/src/main/java/org/jumpmind/symmetric/common/ParameterConstants.java	Diff File
	mod - symmetric-core/src/main/java/org/jumpmind/symmetric/route/DataGapFastDetector.java	Diff File
	mod - symmetric-core/src/main/resources/symmetric-default.properties	Diff File
	mod - symmetric-core/src/test/java/org/jumpmind/symmetric/route/DataGapDetectorTest.java	Diff File
SymmetricDS: 3.7 9ceb7576 2016-04-28 09:29:02 admin Details Diff	0002526: Improve performance of data gap detection	Affected Issues 0002526
	mod - symmetric-client/src/main/java/org/jumpmind/symmetric/db/oracle/OracleSymmetricDialect.java	Diff File
SymmetricDS: 3.8 0f8f7e25 2016-05-23 12:04:34 admin Details Diff	0002526: Improve performance of data gap detection snapshot util should not insert last gap	Affected Issues 0002526
	mod - symmetric-client/src/main/java/org/jumpmind/symmetric/util/SnapshotUtil.java	Diff File
SymmetricDS: 3.8 39e8457b 2016-05-24 11:00:36 admin Details Diff	0002526: Improve performance of data gap detection add routine that fixes overlapping gaps	Affected Issues 0002526
	mod - symmetric-core/src/main/java/org/jumpmind/symmetric/route/DataGapFastDetector.java	Diff File
	mod - symmetric-core/src/test/java/org/jumpmind/symmetric/route/DataGapDetectorTest.java	Diff File
SymmetricDS: 3.8 da8519cf 2016-05-26 15:55:36 admin Details Diff	0002526: Improve performance of data gap detection changes from code review	Affected Issues 0002526
	mod - symmetric-core/src/main/java/org/jumpmind/symmetric/route/ConfigurationChangedDataRouter.java	Diff File
	mod - symmetric-core/src/main/java/org/jumpmind/symmetric/route/DataGapFastDetector.java	Diff File
	mod - symmetric-core/src/main/java/org/jumpmind/symmetric/route/SimpleRouterContext.java	Diff File
	mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/impl/RouterService.java	Diff File
SymmetricDS: 3.8 a8f859c6 2016-07-19 13:10:14 admin Details Diff	0002526: Improve performance of data gap detection don't query for database time on each routing run	Affected Issues 0002526
	mod - symmetric-core/src/main/java/org/jumpmind/symmetric/model/DataGap.java	Diff File
	mod - symmetric-core/src/main/java/org/jumpmind/symmetric/route/DataGapFastDetector.java	Diff File
	mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/impl/DataService.java	Diff File
	mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/impl/DataServiceSqlMap.java	Diff File
	mod - symmetric-core/src/test/java/org/jumpmind/symmetric/route/DataGapDetectorTest.java	Diff File
SymmetricDS: 3.8 db5c9ffd 2016-07-20 20:28:31 admin Details Diff	0002526: Improve performance of data gap detection avoid context service calls when running non-clustered	Affected Issues 0002526
	mod - symmetric-core/src/main/java/org/jumpmind/symmetric/route/DataGapDetector.java	Diff File
	mod - symmetric-core/src/main/java/org/jumpmind/symmetric/route/DataGapFastDetector.java	Diff File
	mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/impl/RouterService.java	Diff File
	mod - symmetric-core/src/main/resources/symmetric-default.properties	Diff File
	mod - symmetric-core/src/test/java/org/jumpmind/symmetric/route/DataGapDetectorTest.java	Diff File
SymmetricDS: 3.8 21833404 2016-07-21 14:07:33 admin Details Diff	0002526: Improve performance of data gap detection on non-cluster system, keep gaps in memory between calls	Affected Issues 0002526
	mod - symmetric-core/src/main/java/org/jumpmind/symmetric/model/DataGap.java	Diff File
	mod - symmetric-core/src/main/java/org/jumpmind/symmetric/route/DataGapFastDetector.java	Diff File
	mod - symmetric-core/src/test/java/org/jumpmind/symmetric/route/DataGapDetectorTest.java	Diff File
SymmetricDS: 3.8 1f8632bd 2016-07-21 16:56:21 admin Details Diff	0002526: Improve performance of data gap detection try to get unit tests working, which init engine differently	Affected Issues 0002526
	mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/impl/RouterService.java	Diff File
SymmetricDS: 3.8 3c8e0dcf 2016-07-26 15:50:48 admin Details Diff	0002526: Improve performance of data gap detection fix problem with the process info	Affected Issues 0002526
	mod - symmetric-core/src/main/java/org/jumpmind/symmetric/route/DataGapFastDetector.java	Diff File
SymmetricDS: 3.8 57c9666f 2016-07-29 10:37:01 admin Details Diff	0002526: Improve performance of data gap detection	Affected Issues 0002526
	mod - symmetric-core/src/main/java/org/jumpmind/symmetric/common/ParameterConstants.java	Diff File
	mod - symmetric-core/src/main/java/org/jumpmind/symmetric/route/DataGapFastDetector.java	Diff File
	mod - symmetric-core/src/main/resources/symmetric-default.properties	Diff File
	mod - symmetric-core/src/test/java/org/jumpmind/symmetric/route/DataGapDetectorTest.java	Diff File
SymmetricDS: 3.8 20497ce9 2016-08-02 08:29:50 admin Details Diff	0002526: Improve performance of data gap detection, separate processing of gaps from the insert/deletes to the database with logging to show timings of each	Affected Issues 0002526
	mod - symmetric-core/src/main/java/org/jumpmind/symmetric/route/DataGapFastDetector.java	Diff File
	mod - symmetric-core/src/test/java/org/jumpmind/symmetric/route/DataGapDetectorTest.java	Diff File
SymmetricDS: 3.8 2cad25cd 2016-08-02 10:46:46 admin Details Diff	0002526: Improve performance of data gap detection, add logging for timing of mapping the data ids into the gaps	Affected Issues 0002526
	mod - symmetric-core/src/main/java/org/jumpmind/symmetric/route/DataGapFastDetector.java	Diff File
SymmetricDS: 3.8 c10a7caf 2016-08-04 16:02:30 admin Details Diff	0002526: Improve performance of data gap detection, use in-memory gap detection on non-clustered system when there is a large number of changes to sym_data_gap	Affected Issues 0002526
	mod - symmetric-core/src/main/java/org/jumpmind/symmetric/common/ParameterConstants.java	Diff File
	mod - symmetric-core/src/main/java/org/jumpmind/symmetric/route/DataGapFastDetector.java	Diff File
	mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/IDataService.java	Diff File
	mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/impl/DataService.java	Diff File
	mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/impl/DataServiceSqlMap.java	Diff File
	mod - symmetric-core/src/main/resources/symmetric-default.properties	Diff File
	mod - symmetric-core/src/test/java/org/jumpmind/symmetric/route/DataGapDetectorTest.java	Diff File
SymmetricDS: 3.8 ca476d13 2016-08-08 10:58:55 admin Details Diff	0002526: Improve performance of data gap detection, increase time we wait before checking if gaps are in sym_data on a busy system	Affected Issues 0002526
	mod - symmetric-core/src/main/resources/symmetric-default.properties	Diff File

Date Modified	Username	Field	Change
2016-03-17 17:43	elong	New Issue
2016-03-17 17:43	elong	Status	new => assigned
2016-03-17 17:43	elong	Assigned To	=> elong
2016-04-07 14:25	elong	Note Added: 0000779
2016-04-15 17:04	elong	Status	assigned => resolved
2016-04-15 17:04	elong	Fixed in Version	=> 3.8.0
2016-04-15 17:04	elong	Resolution	open => fixed
2016-04-28 14:01	admin	Changeset attached	=> SymmetricDS 3.7 9ceb7576
2016-04-28 14:01	admin	Changeset attached	=> SymmetricDS 3.8 e6a31c55
2016-04-28 14:01	admin	Changeset attached	=> SymmetricDS 3.8 d0cd6323
2016-04-28 14:01	admin	Changeset attached	=> SymmetricDS 3.8 96aeefa8
2016-04-28 14:01	admin	Changeset attached	=> SymmetricDS 3.8 bc148280
2016-04-28 14:01	admin	Changeset attached	=> SymmetricDS 3.8 7b82f2f1
2016-04-28 14:01	admin	Changeset attached	=> SymmetricDS 3.8 a2750110
2016-04-28 14:01	admin	Changeset attached	=> SymmetricDS 3.8 f1d07e40
2016-04-28 14:01	admin	Changeset attached	=> SymmetricDS 3.8 3ee7172f
2016-04-28 14:01	admin	Changeset attached	=> SymmetricDS 3.8 b58dcc73
2016-04-28 14:01	admin	Changeset attached	=> SymmetricDS 3.8 c43b9090
2016-04-28 14:01	admin	Changeset attached	=> SymmetricDS 3.8 3a215dcd
2016-05-23 17:00	admin	Changeset attached	=> SymmetricDS 3.8 0f8f7e25
2016-05-24 16:00	admin	Changeset attached	=> SymmetricDS 3.8 39e8457b
2016-05-27 17:00	admin	Changeset attached	=> SymmetricDS 3.8 da8519cf
2016-07-21 20:00	admin	Changeset attached	=> SymmetricDS 3.8 21833404
2016-07-21 20:00	admin	Changeset attached	=> SymmetricDS 3.8 db5c9ffd
2016-07-21 20:00	admin	Changeset attached	=> SymmetricDS 3.8 a8f859c6
2016-07-21 21:00	admin	Changeset attached	=> SymmetricDS 3.8 1f8632bd
2016-07-27 02:00	admin	Changeset attached	=> SymmetricDS 3.8 3c8e0dcf
2016-07-29 15:00	admin	Changeset attached	=> SymmetricDS 3.8 57c9666f
2016-08-02 13:00	admin	Changeset attached	=> SymmetricDS 3.8 20497ce9
2016-08-02 15:00	admin	Changeset attached	=> SymmetricDS 3.8 2cad25cd
2016-08-04 21:00	admin	Changeset attached	=> SymmetricDS 3.8 c10a7caf
2016-08-08 15:00	admin	Changeset attached	=> SymmetricDS 3.8 ca476d13
2016-08-09 17:10	admin	Status	resolved => closed

View Issue Details

Activities

Related Changesets

Issue History

elong 2016-04-07 14:25 developer ~0000779	New parameters: routing.stale.dataid.gap.time.ms=1200000 routing.use.fast.gap.detector=true Old parameters: routing.stale.dataid.gap.time.ms=7200000