View Revisions: Issue #2777

Summary 0002777: Initial load waits for extraction that never happens
Revision 2016-09-09 09:33 by elong
Description The initial load extractor was enabled in 3.8, which uses a background thread to extract batches. The foreground thread checks if a batch is ready to send. If its staging file is missing, it reschedules the batch for extraction. This happens if the user deletes staging files or the initial load takes a long time and the staging manager deletes the files. If one of the first batches has been sent and OK'ed, the extractor will not re-extract batches and gets into an infinite loop

Extracting batches for request 4. Starting at batch 96. Ending at batch 106
Batch {} is not ready for delivery. It is currently scheduled for extraction
Batch 1-106 is marked as ready but it has been deleted. Rescheduling it for extraction
Revision 2016-09-08 12:18 by elong
Description The initial load extractor was enabled in 3.8, which uses a background thread to extract batches. The foreground thread checks if a batch is ready to send. If its staging file is missing, it reschedules the batch for extraction. This happens if the user deletes staging files or the initial load takes a long time and the staging manager deletes the files. There is race condition between the foreground and background thread that causes the batch status to set to RQ for extraction request but the sym_extract_request is set to OK, so extraction will never happen. The user sees this in the log continuously:

Batch {} is not ready for delivery. It is currently scheduled for extraction