View Issue Details
| ID | Project | Category | View Status | Date Submitted | Last Update |
|---|---|---|---|---|---|
| 0005827 | SymmetricDS | Improvement | public | 2023-05-09 18:26 | 2023-10-03 23:00 |
| Reporter | elong | Assigned To | elong | ||
| Priority | normal | ||||
| Status | closed | Resolution | fixed | ||
| Product Version | 3.15.0 | ||||
| Target Version | 3.15.0 | Fixed in Version | 3.15.0 | ||
| Summary | 0005827: Track expired data gaps and repair any data missed by routing | ||||
| Description | Improve handling of data gaps by tracking expired data gaps and repairing any data that arrives late, no matter how late it arrives. Router service keeps track of data IDs that are missing, called data gaps, and continues to query them, waiting for data to arrive. Eventually they become so old that they are considered expired and they are removed, to keep routing running fast. The problem is that some systems have transactions open for hours and even days before committing, and this data is purged. Keep track of expired data gaps with a flag is_expired on sym_data_gap. The router service will only use non-expired gaps. The purge service will watch for data to arrive in expired data gaps and be responsible for purging expired data gaps that are too old (based on a parameter setting). The purge can continue to use its fast range-based purge of sym_data, but it must step around any expired data gaps. (It would not be sufficient to check for expired data first and then blindly purge by ranges because they creates a race condition.) A new routine will query for data in the expired data gaps, and send the rows as reload events by their primary key values. For any deletes, it should query if the current row exists to determine if a reload or delete should be sent. The stranded data routine should be modified to also step around expired data gaps. Any data is does find should also be repaired, as it is likely data from an expired data gap that was purged. Track statistics for purging of expired data and stranded data. | ||||
| Tags | purge | ||||
|
SymmetricDS: 3.15 005e631e 2023-05-09 18:26:44 Details Diff |
0005827: Track expired data gaps and repair any data missed by routing |
Affected Issues 0005827 |
|
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/AbstractSymmetricEngine.java | Diff File | ||
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/common/ParameterConstants.java | Diff File | ||
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/route/DataGapFastDetector.java | Diff File | ||
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/IDataService.java | Diff File | ||
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/impl/DataService.java | Diff File | ||
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/impl/DataServiceSqlMap.java | Diff File | ||
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/impl/PurgeService.java | Diff File | ||
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/impl/PurgeServiceSqlMap.java | Diff File | ||
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/impl/RouterService.java | Diff File | ||
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/impl/StatisticService.java | Diff File | ||
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/impl/StatisticServiceSqlMap.java | Diff File | ||
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/statistic/HostStats.java | Diff File | ||
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/statistic/IStatisticManager.java | Diff File | ||
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/statistic/StatisticManager.java | Diff File | ||
| mod - symmetric-core/src/main/resources/symmetric-default.properties | Diff File | ||
| mod - symmetric-core/src/main/resources/symmetric-schema.xml | Diff File | ||
| mod - symmetric-core/src/test/java/org/jumpmind/symmetric/statistic/MockStatisticManager.java | Diff File | ||
| mod - symmetric-db/src/main/java/org/jumpmind/db/sql/DmlStatement.java | Diff File | ||
| Date Modified | Username | Field | Change |
|---|---|---|---|
| 2023-05-09 18:26 | elong | New Issue | |
| 2023-05-09 18:26 | elong | Status | new => assigned |
| 2023-05-09 18:26 | elong | Assigned To | => elong |
| 2023-05-09 18:26 | elong | Tag Attached: purge | |
| 2023-05-10 12:10 | elong | Status | assigned => resolved |
| 2023-05-10 12:10 | elong | Resolution | open => fixed |
| 2023-05-10 12:10 | elong | Fixed in Version | => 3.15.0 |
| 2023-10-01 23:46 | admin | Status | resolved => closed |
| 2023-10-03 23:00 | admin | Changeset attached | => SymmetricDS 3.15 005e631e |