View Issue Details

IDProjectCategoryView StatusLast Update
0003974SymmetricDSBugpublic2019-06-25 14:06
ReporterdiltonnAssigned Toelong 
Priorityurgent 
Status closedResolutionfixed 
Product Version3.10.2 
Target Version3.10.3Fixed in Version3.10.3 
Summary0003974: Too many threads, crashing server
DescriptionWe have a multi-tenant system running around 20 replication engines.
Replication happens both ways, 'cloud-to-client' and 'client-to-cloud'.

We upgraded from 3.9.16 to 3.10.2 and our server crashed a couple times before we realized it was due to SymmetricDS.

First we noticed a huge increase in memory usage, we upgraded the server memory which made it stay up for longer.

Later we noticed the number of threads open by the Symmetric service was too high and it was continuously increasing.

We went back to v3.9.16 which won't go over 400 threads, while v3.10.2 got to more than 11K threads at some point.

No changes were made on the settings (at least not knowingly), so I guess the issue is related to the new version.

AWS Ubuntu Server
SymmetricDS v3.10.2 running with "sym_service"
PostgreSQL 10
Java 1.8.0_212
Tagsdata sync

Relationships

related to 0003873 closedelong Keep alive for slow acknowledgement 

Activities

elong

2019-05-21 16:39

developer   ~0001500

I'm guessing 0003873 is the cause of the additional threads. It was an attempt to keep the connection alive when the acknowledgement comes in. In retrospect, using an additional thread for each ack was not a good idea. Let me see if I can preserve the feature and get rid of the extra threads.

elong

2019-05-22 13:38

developer   ~0001501

Attaching patch file for 3.10.2 that removes additional threads from acknowledgement. Feel free to test and give feedback. Sending fix to integration testing.

patch-0003974.jar (3,379 bytes)

Related Changesets

SymmetricDS: 3.10 29416197

2019-05-22 13:52:26

admin

Details Diff
0003974: Too many threads, crashing server
0003974
mod - symmetric-server/src/main/java/org/jumpmind/symmetric/web/AckUriHandler.java Diff File

Issue History

Date Modified Username Field Change
2019-05-21 16:02 diltonn New Issue
2019-05-21 16:39 elong Assigned To => elong
2019-05-21 16:39 elong Status new => confirmed
2019-05-21 16:39 elong Note Added: 0001500
2019-05-22 13:38 elong File Added: patch-0003974.jar
2019-05-22 13:38 elong Note Added: 0001501
2019-05-22 13:47 elong Relationship added related to 0003873
2019-05-22 13:53 elong Priority normal => urgent
2019-05-22 13:53 elong Target Version => 3.10.3
2019-05-22 14:00 admin Changeset attached => SymmetricDS 3.10 29416197
2019-05-22 14:07 elong Tag Attached: data sync
2019-05-29 12:57 elong Status confirmed => resolved
2019-05-29 12:57 elong Resolution open => fixed
2019-05-29 12:57 elong Fixed in Version => 3.10.3
2019-06-25 14:06 admin Status resolved => closed