View Issue Details

IDProjectCategoryView StatusLast Update
0003445SymmetricDSBugpublic2018-03-12 08:26
ReportermmichalekAssigned Tommichalek 
Prioritynormal 
Status closedResolutionfixed 
Product Version3.8.30 
Target Version3.9.5Fixed in Version3.9.5 
Summary0003445: Batch extract locks aren't released when "java.lang.IllegalStateException: Had trouble renaming file." occurs.
DescriptionWhen a problem with staging occurs and an extracted batch can't be renamed from .create to .DONE, the associated in memory lock is not released. Subsequent attempts to extact the batch will hang indefinitely.

2018-02-22 13:33:43,735 ERROR [corp-000] [DataExtractorService] [qtp85241109-15] Failed to extract batch 001-55
java.lang.RuntimeException: java.lang.IllegalStateException: Had trouble renaming file. The current name is /var/folders/lz/t_xt3p4d1sg8n79lkqmffmww0000gn/T/corp-000/outgoing/common/0000000055.create and the desired state was DONE
    at org.jumpmind.symmetric.service.impl.DataExtractorService.extract(DataExtractorService.java:684)
    at org.jumpmind.symmetric.service.impl.DataExtractorService.extract(DataExtractorService.java:543)
    at org.jumpmind.symmetric.web.PullUriHandler.pull(PullUriHandler.java:130)
    at org.jumpmind.symmetric.web.PullUriHandler.handleWithCompression(PullUriHandler.java:100)
    at org.jumpmind.symmetric.web.AbstractCompressionUriHandler.handle(AbstractCompressionUriHandler.java:84)
    at org.jumpmind.symmetric.web.SymmetricServlet.service(SymmetricServlet.java:114)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
    at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669)
    at org.jumpmind.symmetric.web.HttpMethodFilter.doFilter(HttpMethodFilter.java:62)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
    at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
    at org.eclipse.jetty.server.Server.handle(Server.java:499)
    at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
    at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
    at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalStateException: Had trouble renaming file. The current name is /var/folders/lz/t_xt3p4d1sg8n79lkqmffmww0000gn/T/corp-000/outgoing/common/0000000055.create and the desired state was DONE
    at org.jumpmind.symmetric.io.stage.StagedResource.setState(StagedResource.java:170)
    at org.jumpmind.symmetric.service.impl.DataExtractorService.extractOutgoingBatch(DataExtractorService.java:939)
    at org.jumpmind.symmetric.service.impl.DataExtractorService.extractBatch(DataExtractorService.java:803)
    at org.jumpmind.symmetric.service.impl.DataExtractorService$1.call(DataExtractorService.java:624)
    at org.jumpmind.symmetric.service.impl.DataExtractorService$1.call(DataExtractorService.java:1)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    ... 1 more


This leads to lots of threads that look like this:

sun.misc.Unsafe.park()
java.util.concurrent.locks.LockSupport.park():186
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt():834
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly():994
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly():1303
java.util.concurrent.Semaphore.acquire():317
org.jumpmind.symmetric.service.impl.DataExtractorService$BatchLock.acquire():2397
org.jumpmind.symmetric.service.impl.DataExtractorService.acquireLock():997
org.jumpmind.symmetric.service.impl.DataExtractorService.extractOutgoingBatch():896
org.jumpmind.symmetric.service.impl.DataExtractorService.extractBatch():803
org.jumpmind.symmetric.service.impl.DataExtractorService$1.call():624
org.jumpmind.symmetric.service.impl.DataExtractorService$1.call():1
java.util.concurrent.FutureTask.run():262
java.util.concurrent.ThreadPoolExecutor.runWorker():1145
java.util.concurrent.ThreadPoolExecutor$Worker.run():615
java.lang.Thread.run():745

Relationships

related to 0003446 closedmmichalek Batch extract locks aren't released when "java.lang.IllegalStateException: Had trouble renaming file." occurs. 

Activities

There are no notes attached to this issue.

Related Changesets

SymmetricDS: 3.8 4c63eb22

2018-02-22 14:16:51

mmichalek

Details Diff
0003445: Batch extract locks aren't released when
"java.lang.IllegalStateException: Had trouble renaming file." occurs.

0003445
mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/impl/DataExtractorService.java Diff File

SymmetricDS: 3.9 79669d6e

2018-02-22 14:18:42

mmichalek

Details Diff
0003445: Batch extract locks aren't released when
"java.lang.IllegalStateException: Had trouble renaming file." occurs.

0003445
mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/impl/DataExtractorService.java Diff File

Issue History

Date Modified Username Field Change
2018-02-22 14:13 mmichalek New Issue
2018-02-22 14:13 mmichalek Status new => assigned
2018-02-22 14:13 mmichalek Assigned To => mmichalek
2018-02-22 14:13 mmichalek Summary Batch extract locks aren't released when java.lang.IllegalStateException: Had trouble renaming file. occurs. => Batch extract locks aren't released when "java.lang.IllegalStateException: Had trouble renaming file." occurs.
2018-02-22 14:19 mmichalek Status assigned => resolved
2018-02-22 14:19 mmichalek Resolution open => fixed
2018-02-22 14:19 mmichalek Fixed in Version => 3.9.5
2018-02-22 15:00 mmichalek Changeset attached => SymmetricDS 3.8 4c63eb22
2018-02-22 15:00 mmichalek Changeset attached => SymmetricDS 3.9 79669d6e
2018-02-22 16:48 mmichalek Relationship added related to 0003446
2018-03-12 08:26 chenson Status resolved => closed