View Issue Details

IDProjectCategoryView StatusLast Update
0004546SymmetricDSBugpublic2020-09-23 20:34
Reporterelong Assigned Toelong  
Prioritynormal 
Status closedResolutionfixed 
Product Version3.11.11 
Target Version3.11.12Fixed in Version3.11.12 
Summary0004546: Data Extractor gets "java.lang.IllegalStateException: There is no content to read" from staging
DescriptionAfter optimizing concurrency in 0004472, a race condition was introduced where staging manager can remove a staged resource before it is push or pulled. When purging resources based on batches (stream.to.file.purge.on.ttl.enabled=false), the staging manager gets a list of batches from the database at the beginning of its run. As it looks through staged resources, if the resource does not have a corresponding batch from the list, it will remove the resource. The problem is that resources can be created while the job is in the middle of running, and those batches aren't on the list. To fix, we'll make a note of the start time of the job, and resources must have a last modified time before the job start time to be eligible for purging.

2020-09-11 17:00:00,007 INFO [corp-000] [StagingManager] [corp-000-job-18] Cleaning staging...
2020-09-11 17:00:00,050 INFO [corp-000] [StagingManager] [corp-000-job-18] Purged 224 staging files, freed 187 KB of disk space.
2020-09-11 17:00:00,050 INFO [corp-000] [StagingManager] [corp-000-job-18] Finished cleaning staging in 0 seconds.
2020-09-11 17:02:59,541 ERROR [corp-000] [DataExtractorService] [corp-000-push-default-5] Failed to extract batch 003-333923 StackTraceKey.init [IllegalStateException:427582167] java.lang.IllegalStateException: There is no content to read. Memory buffer was empty and /export/home/postgres/symmetric-ds/tmp/corp-000/outgoing/common/923/0000333923.create was not found.
        at org.jumpmind.symmetric.io.stage.StagedResource.getReader(StagedResource.java:246)
        at org.jumpmind.symmetric.service.impl.DataExtractorService.transferFromStaging(DataExtractorService.java:1481)
        at org.jumpmind.symmetric.service.impl.DataExtractorService.sendOutgoingBatch(DataExtractorService.java:1440)
        at org.jumpmind.symmetric.service.impl.DataExtractorService.extract(DataExtractorService.java:789)
        at org.jumpmind.symmetric.service.impl.DataExtractorService.extract(DataExtractorService.java:577)
        at org.jumpmind.symmetric.service.impl.PushService.pushToNode(PushService.java:206)
        at org.jumpmind.symmetric.service.impl.PushService.execute(PushService.java:167)
        at org.jumpmind.symmetric.service.impl.NodeCommunicationService$1.run(NodeCommunicationService.java:519)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Steps To ReproduceAdd break point in debugger in BatchStagingManager to help with timing
Run Stage Management job
Change data and observe a new batch that is push/pulled
Allow Stage Management job to continue, and it will immediately delete the new batch's stage file
Tagsstaging

Relationships

related to 0004545 closedelong Data Extractor gets "java.lang.IllegalStateException: There is no content to read" from staging 
related to 0004472 closedpmarzullo Staging Purge stops extracts from continuing on when running on the same instance 
related to 0004547 closedelong Data Extractor gets "java.lang.IllegalStateException: There is no content to read" from staging 

Activities

There are no notes attached to this issue.

Issue History

Date Modified Username Field Change
2020-09-14 19:57 elong New Issue
2020-09-14 19:57 elong Status new => assigned
2020-09-14 19:57 elong Assigned To => elong
2020-09-14 19:57 elong Tag Attached: staging
2020-09-14 19:57 elong Issue generated from: 0004545
2020-09-14 19:57 elong Relationship added related to 0004545
2020-09-14 19:58 elong Relationship added related to 0004472
2020-09-14 19:58 elong Status assigned => resolved
2020-09-14 19:58 elong Resolution open => fixed
2020-09-14 19:58 elong Fixed in Version => 3.11.12
2020-09-14 19:59 elong Issue cloned: 0004547
2020-09-14 19:59 elong Relationship added related to 0004547
2020-09-23 20:34 admin Status resolved => closed