View Issue Details

IDProjectCategoryView StatusLast Update
0004424SymmetricDSImprovementpublic2022-12-08 19:00
Reporterpmarzullo Assigned Topmarzullo  
Prioritynormal 
Status closedResolutionfixed 
Product Version3.10.0 
Target Version3.14.4Fixed in Version3.14.4 
Summary0004424: Log monitor fails to insert because of duplicate key when logs are shared across instances of a node
DescriptionIn a clustered environment, when SymmetricDS is deployed to a servlet container, and the servlet containers write to the same log file (log files are on a shared drive), the execution of the log monitor on each instance of the cluster sees the same error in the log, causing duplicate key errors because the create time of the log entry is part of the primary key.

Two choices that I can think of:

1. Create a new parameter that makes the Log Monitor in cluster mode.
2. Change the insert of sym_monitor_event to a update/insert. Possible loss of real monitor entries because of exact same time of create time.

-------------

The monitor job runs on both nodes. The logs are shared. Both nodes can see the same log entry and tries to insert when the query returns no log entry. The query to get the unresolved events is run outside of the monitor lock.

Either the cluster lock should wrap all of the code, or the insertMonitorEvent should be changed to saveMonitorEvent.
Steps To Reproduce1. Set up a clustered environment.
2. Configure the log4j2 log files to be common across the clustered nodes.
3. Make sure to configure a log monitor.
4. Cause an error in the log.
Duplicate key errors should occur when trying to save the log monitor entries.
Tagslogging, monitoring

Activities

pmarzullo

2021-03-12 22:05

developer   ~0001896

Can I change the log monitor type to use cluster locking?

pmarzullo

2022-12-08 18:07

developer   ~0002236

Change the MonitorService.updateMonitor() method to call saveMonitorEvent()
instead of insertMonitorEvent() every time so that the possibility of having primary key
constraints is removed. saveMonitorEvent() calls updateMonitorEvent() followed by insertMonitorEvent()
(only if the update returns 0 rows updated). This change will affect every insert of
monitor events to change into a update/insert.

Make the log monitor a Cluster lock monitor.
This will affect the log monitor by locking out the other nodes while the monitor event
is inserted/updated in the database. This will make sure that the record is not inserted
on another node while the monitor event is being inserted into the database.

Related Changesets

SymmetricDS: 3.14 e84f0a48

2022-12-08 18:06:08

pmarzullo

Details Diff
0004424: Log monitor fails to insert because of duplicate key when logs
are shared across instances of a node
Affected Issues
0004424
mod - symmetric-core/src/main/java/org/jumpmind/symmetric/monitor/MonitorTypeLog.java Diff File
mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/impl/MonitorService.java Diff File

Issue History

Date Modified Username Field Change
2020-06-11 13:01 pmarzullo New Issue
2020-06-11 13:01 pmarzullo Status new => assigned
2020-06-11 13:01 pmarzullo Assigned To => pmarzullo
2020-06-11 13:03 pmarzullo Description Updated View Revisions
2020-06-24 20:32 elong Project SymmetricDS Pro => SymmetricDS
2020-06-24 20:33 elong Product Version 3.11.9 => 3.10.0
2020-06-24 20:33 elong Target Version 3.11.10 => 3.10.13
2020-07-06 13:05 pmarzullo Tag Attached: logging
2020-07-06 13:05 pmarzullo Tag Attached: monitoring
2020-11-09 13:43 admin Target Version 3.10.13 => 3.12.6
2020-11-09 13:44 admin Target Version 3.12.6 => 3.10.14
2021-03-12 22:05 pmarzullo Note Added: 0001896
2021-11-02 20:57 elong Target Version 3.10.14 => 3.12.13
2021-12-11 17:11 admin Target Version 3.12.13 => 3.12.14
2021-12-20 13:55 admin Target Version 3.12.14 => 3.12.15
2022-01-13 13:34 elong Target Version 3.12.15 => 3.14.0
2022-07-25 16:52 elong Target Version 3.14.0 => 3.14.1
2022-08-24 18:56 elong Target Version 3.14.1 => 3.14.2
2022-10-03 14:38 elong Target Version 3.14.2 =>
2022-12-08 18:02 pmarzullo Target Version => 3.14.4
2022-12-08 18:02 pmarzullo Steps to Reproduce Updated View Revisions
2022-12-08 18:07 pmarzullo Status assigned => resolved
2022-12-08 18:07 pmarzullo Resolution open => fixed
2022-12-08 18:07 pmarzullo Fixed in Version => 3.14.4
2022-12-08 18:07 pmarzullo Note Added: 0002236
2022-12-08 19:00 pmarzullo Changeset attached => SymmetricDS 3.14 e84f0a48
2023-01-16 20:59 admin Status resolved => closed