View Issue Details

IDProjectCategoryView StatusLast Update
0003829SymmetricDSBugpublic2019-08-26 12:45
Reportersagar5490 Assigned To 
Priorityhigh 
Status closedResolutionopen 
Product Version3.9.1 
Summary0003829: About Duplication Key Error come occasionally
DescriptionDear Team,

We are very thank full providing tool like this to set cross platform replication between sql to postgresql.
We are using symmetricds 3.9.1 community edition from past 1 year.It's working fine.
But from some day we are facing below error occasionally

ERROR [master-1] [AcknowledgeService] [master-1-push-default-6] The outgoing batch 001-307363 failed: ERROR: duplicate key value violates unique constraint "idx_temp_id"
  Detail: Key (tempid)=(1633313) already exists.

where my tempid is unique on both sql and postgresq db server.We don't know how it get duplicate id.

We analysis's log , probably this error come after below query(long run) execute.

 INFO [master-1] [DataGapRouteReader] [master-1-router-reader-3112] Reading data to route for channel 'default' has been processing for 2093 seconds. The following stats have been gathered: datdaRereadCount=0, dataCount=1
 INFO [master-1] [RouterService] [master-1-job-18] Routing for channel 'default' has been processing for 2093 seconds. The following stats have been gathered: totalDataRoutedCount=1, totalDataEventCount=1, startDataId=85593568, endDataId=85595867, dataReadCount=2000, peekAheadFillCount=1, transactions=[58234924], dataGaps=[{ startId: 85593568, endId: 135593567, createTime: "2018-12-13 11:59:32.293" }]
 INFO [master-1] [RouterService] [master-1-job-18] Done routing for channel 'default' which took 2123 seconds
INFO [master-1] [RouterService] [master-1-job-18] Routing default, data.enqueue.time.ms=9064, data.events.insert.count=406919, data.events.insert.time.ms=9124, data.read.total.time.ms=19944, data.routed.count=406919, data.router.time.ms=142, total.time.ms=2123426
[master-1] [RouterService] [master-1-job-18] Routed 406919 data events in 2123426 ms
 INFO [master-1] [DataGapFastDetector] [master-1-job-20] Full gap analysis is running
 INFO [master-1] [DataGapFastDetector] [master-1-job-20] Querying data in gaps from database took 94 ms
 INFO [master-1] [DataGapFastDetector] [master-1-job-20] Full gap analysis is done after 125 ms
INFO [master-1] [RouterService] [master-1-job-20] Routed 805 data events in 516 ms


Probably this error come when Querying data in gaps from database took 94 ms and Full gap analysis is done after 125 ms
if Querying data in gaps from database took 0 ms adn ] Full gap analysis is done after 0 ms then
may this error not come

please help us to recover from this issue otherwise
we need to manually delete duplicate id from target database to recover



TagsNo tags attached.

Activities

sagar5490

2018-12-27 05:57

reporter   ~0001369

Is any update on this issue

elong

2019-01-14 14:50

developer   ~0001377

If SQL-Server is the source, can you verify that the data is being captured correctly? There is a report that an update is captured in "sym_data" table with the correct new row in "row_data" field but an incorrect old row in "old_data" field when a multi-row update is captured. (https://www.symmetricds.org/issues/view.php?id=3847) The "sym_outgoing_batch" table has a "failed_data_id" field that you can use to query the "sym_data" table. See if the primary key fields in "row_data" are different from the "old_data" when this happens, which would be a clue it's issue 3847.

Is there only one instance of the master-1 node? If you tried to cluster two instances of the master-1 node but forgot to enable the cluster.lock.enabled=true parameter, then routing could cause batch errors like that.

elong

2019-04-25 18:17

developer   ~0001443

Any update on this? When you get this error, is it an insert or update statement that gets the duplicate key error? Are you 1-direction or bi-directional replication? In version 3.10, it detects the duplicate key error and fixes the data automatically now, so it might be worth trying. Still, the duplicate key error implies you got out of sync somehow.

admin

2019-08-26 12:45

administrator   ~0001575

Auto closing all issues waiting for feedback after 4 months. We don't have enough information to take any action.

Issue History

Date Modified Username Field Change
2018-12-14 08:50 sagar5490 New Issue
2018-12-27 05:57 sagar5490 Note Added: 0001369
2019-01-14 14:50 elong Note Added: 0001377
2019-04-25 18:17 elong Status new => feedback
2019-04-25 18:17 elong Note Added: 0001443
2019-08-26 12:45 admin Note Added: 0001575
2019-08-26 12:45 admin Status feedback => closed