View Issue Details
| ID | Project | Category | View Status | Date Submitted | Last Update |
|---|---|---|---|---|---|
| 0002959 | SymmetricDS | Bug | public | 2017-01-12 09:08 | 2017-01-19 19:56 |
| Reporter | gatokino | Assigned To | chenson | ||
| Priority | high | ||||
| Status | closed | Resolution | fixed | ||
| Product Version | 3.8.11 | ||||
| Target Version | 3.8.14 | Fixed in Version | 3.8.14 | ||
| Summary | 0002959: BatchStagingManager deleting incoming files from staging that are not yet processed | ||||
| Description | We have a situation during the inital load with a lot of slow loading incoming batches. For that reason in the temporary directory we have aprox 250 files for incoming batches. BatchStaginManager is called and is deleting files that are present in temp directory for incoming batches but not yet present in the sym_incoming_batch sql table. Because of that we have later the following error: ***** ERROR [org.jumpmind.symmetric.service.impl.DataLoaderService] (client-pull-default-4) Failed while parsing batch: java.lang.IllegalStateException: There is no content to read. Memory buffer was empty and D:\********\incoming\00000\0000008322.ready was not found. at org.jumpmind.symmetric.io.stage.StagedResource.getReader(StagedResource.java:184) at org.jumpmind.symmetric.io.data.reader.ProtocolDataReader.open(ProtocolDataReader.java:121) at org.jumpmind.symmetric.io.data.DataProcessor.process(DataProcessor.java:86) at org.jumpmind.symmetric.service.impl.DataLoaderService$LoadIntoDatabaseOnArrivalListener$1.call(DataLoaderService.java:938) at org.jumpmind.symmetric.service.impl.DataLoaderService$LoadIntoDatabaseOnArrivalListener$1.call(DataLoaderService.java:913) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) As all other files are deleted the rest of the incoming batches, not yet processed, are redownloaded and retried ... | ||||
| Additional Information | Workaround was to set a high time to live for staging files and to use that value in the if that is deciding to erase: line 59 -> (recordIncomingBatchesEnabled && !incomingBatches.contains(batchId) && resourceIsOld) Also we have added a the line 44 -> (!resource.isInUse()) condition in a hope that this will fix the problem. | ||||
| Tags | No tags attached. | ||||
|
|
This is definitely a bug. We'll fix it in 3.8.14. Thank you for reporting it! |
|
|
I checked in a fix that I am testing. The approach is to not purge if the largest existing incoming batch for the specific node is smaller than the batch being evaluated for purging. |
|
SymmetricDS: 3.8 250a7da6 2017-01-12 14:04:21 Details Diff |
0002959: BatchStagingManager deleting incoming files from staging that are not yet processed |
Affected Issues 0002959 |
|
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/io/stage/BatchStagingManager.java | Diff File | ||
|
SymmetricDS: 3.8 92f32684 2017-01-12 14:06:04 Details Diff |
0002959: BatchStagingManager deleting incoming files from staging that are not yet processed |
Affected Issues 0002959 |
|
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/io/stage/BatchStagingManager.java | Diff File | ||
| Date Modified | Username | Field | Change |
|---|---|---|---|
| 2017-01-12 09:08 | gatokino | New Issue | |
| 2017-01-12 14:44 | chenson | Assigned To | => chenson |
| 2017-01-12 14:44 | chenson | Status | new => assigned |
| 2017-01-12 14:45 | chenson | Note Added: 0000949 | |
| 2017-01-12 19:04 | chenson | Status | assigned => resolved |
| 2017-01-12 19:04 | chenson | Fixed in Version | => 3.8.14 |
| 2017-01-12 19:04 | chenson | Resolution | open => fixed |
| 2017-01-12 19:07 | chenson | Note Added: 0000950 | |
| 2017-01-12 20:00 | chenson | Changeset attached | => SymmetricDS 3.8 92f32684 |
| 2017-01-12 20:00 | chenson | Changeset attached | => SymmetricDS 3.8 250a7da6 |
| 2017-01-19 19:56 | chenson | Status | resolved => closed |