View Issue Details

IDProjectCategoryView StatusLast Update
0003828SymmetricDS ProImprovementpublic2019-03-01 12:57
Reportergilles.riandAssigned Toelong 
Prioritynormal 
Status closedResolutionduplicate 
Product Version3.7.36 
Target VersionFixed in Version 
Summary0003828: Fast Purge Job
DescriptionHello

i'm using SymDS in a multi asterenvironnement wit a lot of store and a Master Node who make the Route for evey nodes.
I think that i can be possible to do the ougoing Purge job faster

1) calculate the older batch not OK...
      SELECT MIN(CREATE_TIME) AS OLDEST_BATCH
       FROM SYM_OUTGOING_BATCH
       WHERE STATUS!='OK'
      ORDER BY 1 DESC

After delete everything (in SYM_DATA) before this time or 10 minutes before .... probably in multiple statements...
I think it will be faster than this kind of queries

delete from sym_data where data_id between 3394307 and 3394407 and create_time < {ts '2018-12-10 14:33:37.003'}
and data_id in (select e.data_id from sym_data_event e where e.data_id between 3394307 and 3394407) and data_id not in
(select e.data_id from sym_data_event e where e.data_id between 3394307 and 3394407 and (e.data_id is null or e.batch_id in (select batch_id from sym_outgoing_batch where status != 'OK')))

What do you think about it ?





TagsNo tags attached.

Relationships

related to 0003670 closedelong SymmetricDS Improve purge performance of sym_data and sym_data_event 
related to 0003784 closedelong SymmetricDS More aggressive purge settings 

Activities

elong

2019-03-01 12:57

developer   ~0001397

It's a great idea. We implemented it in latest 3.8 and 3.9 and can be enabled with job.purge.first.pass=true parameter. It's been working so well, that it will be enabled by default in 3.10.

Issue History

Date Modified Username Field Change
2018-12-13 09:38 gilles.riand New Issue
2019-03-01 12:54 elong Relationship added related to 0003670
2019-03-01 12:55 elong Relationship added related to 0003784
2019-03-01 12:57 elong Assigned To => elong
2019-03-01 12:57 elong Status new => closed
2019-03-01 12:57 elong Resolution open => duplicate
2019-03-01 12:57 elong Note Added: 0001397