Project

General

Profile

Bug #60159

Deadlock during indexing

Added by Alexey Bobyr almost 4 years ago. Updated about 3 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Alexey Bobyr
Category:
Scalix SIS
Target version:
Start date:
12/03/2015
Due date:
% Done:

0%

Estimated time:
Operation System:
--

Description

2015-11-25 23:04:27,850 ERROR [QueueManager] [UserIndexer.getReferenceWriter:127] org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: NativeFSLock@/var/opt/scalix/ml/indexes/07f00000f3dfe135-9.13.681.46/ref/write.lock
2015-11-25 23:04:27,850 ERROR [QueueManager] [BatchUpdater.performReferenceDeletes:232] User 07f00000f3dfe135-9.13.681.46: Failed to delete document(s) for user 07f00000f3dfe135-9.13.681.46! Reason: com.scalix.index.api.IndexerException: Trouble unlocking /var/opt/scalix/ml/indexes/07f00000f3dfe135-9.13.681.46/ref
2015-11-25 23:04:28,851 ERROR [QueueManager] [UserIndexer.getContentWriter:99] org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: NativeFSLock@/var/opt/scalix/ml/indexes/07f00000f3dfe135-9.13.681.46/write.lock
2015-11-25 23:04:28,852 ERROR [QueueManager] [BatchUpdater.processMods:158] Failed to delete content documents for user 07f00000f3dfe135-9.13.681.46
com.scalix.index.api.IndexerException: Trouble unlocking /var/opt/scalix/ml/indexes/07f00000f3dfe135-9.13.681.46
        at com.scalix.index.manager.UserIndexer.getContentWriter(UserIndexer.java:109)
        at com.scalix.index.manager.UserIndexer.deleteDocumentsFromContent(UserIndexer.java:295)
        at com.scalix.index.manager.BatchUpdater.performContentDeletes(BatchUpdater.java:337)
        at com.scalix.index.manager.BatchUpdater.processMods(BatchUpdater.java:154)
        at com.scalix.index.manager.QueueManager.processQueue(QueueManager.java:129)
        at com.scalix.index.manager.QueueManager.run(QueueManager.java:78)
        at java.lang.Thread.run(Thread.java:745)

2015-11-25 21:06:37,875  INFO [QueueManager] [BatchUpdater.processMods:182] User 0df00000f3dfe135-9.13.681.46: added 1 content, added 0 refs, deleted 0 content, deleted 0 refs in 5 ms
2015-11-25 21:06:39,360 ERROR [QueueManager] [UserIndexer.getReferenceWriter:127] org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: NativeFSLock@/var/opt/scalix/ml/indexes/07f00000f3dfe135-9.13.681.46/ref/write.lock
2015-11-25 21:06:39,361  WARN [QueueManager] [BatchUpdater.performReferenceAdds:269] User 07f00000f3dfe135-9.13.681.46: Failed to add reference for AddRefInfo{IndexInfo{userID='07f00000f3dfe135-9.13.681.46'}, indexID='902ba40-531efd3f-56566925-5a87eb', dRef='001342694a32
3f1f', parentDRef='0010cf348b9c104c', flagsList='unseen,unflagged,unanswered,undeleted,undraft,unlabel1,unlabel2,unlabel3,unlabel4,unlabel5,unlabel6,unlabel7,unlabel8,unjunk,unnonjunk,unforwarded'}
com.scalix.index.api.IndexerException: Trouble unlocking /var/opt/scalix/ml/indexes/07f00000f3dfe135-9.13.681.46/ref
        at com.scalix.index.manager.UserIndexer.getReferenceWriter(UserIndexer.java:137)
        at com.scalix.index.manager.UserIndexer.addDocumentToReferenceIndex(UserIndexer.java:260)
        at com.scalix.index.manager.BatchUpdater.performReferenceAdds(BatchUpdater.java:260)
        at com.scalix.index.manager.BatchUpdater.processMods(BatchUpdater.java:126)
        at com.scalix.index.manager.QueueManager.processQueue(QueueManager.java:129)
        at com.scalix.index.manager.QueueManager.run(QueueManager.java:78)
        at java.lang.Thread.run(Thread.java:745)
2015-11-25 21:06:40,362 ERROR [QueueManager] [UserIndexer.getContentWriter:99] org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: NativeFSLock@/var/opt/scalix/ml/indexes/07f00000f3dfe135-9.13.681.46/write.lock
2015-11-25 21:06:40,362 ERROR [QueueManager] [BatchUpdater.processMods:170] Failed to add content document(s) for user 07f00000f3dfe135-9.13.681.46
com.scalix.index.api.IndexerException: Trouble unlocking /var/opt/scalix/ml/indexes/07f00000f3dfe135-9.13.681.46
        at com.scalix.index.manager.UserIndexer.getContentWriter(UserIndexer.java:109)
        at com.scalix.index.manager.UserIndexer.addDocumentToContentIndex(UserIndexer.java:233)
        at com.scalix.index.manager.BatchUpdater.performContentAdds(BatchUpdater.java:377)
        at com.scalix.index.manager.BatchUpdater.processMods(BatchUpdater.java:166)
        at com.scalix.index.manager.QueueManager.processQueue(QueueManager.java:129)
        at com.scalix.index.manager.QueueManager.run(QueueManager.java:78)
        at java.lang.Thread.run(Thread.java:745)

015-11-25 12:49:17,853  INFO [QueueManager] [BatchUpdater.processMods:182] User 07f00000f3dfe135-9.13.681.46: added 0 content, added 0 refs, deleted 0 content, deleted 0 refs in 1 ms
2015-11-25 12:49:18,854 ERROR [QueueManager] [UserIndexer.getContentWriter:99] org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: NativeFSLock@/var/opt/scalix/ml/indexes/07f00000f3dfe135-9.13.681.46/write.lock
2015-11-25 12:49:18,855 ERROR [QueueManager] [UserIndexer.commitChanges:383] com.scalix.index.api.IndexerException: Trouble unlocking /var/opt/scalix/ml/indexes/07f00000f3dfe135-9.13.681.46
2015-11-25 12:49:19,855 ERROR [QueueManager] [UserIndexer.getReferenceWriter:127] org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: NativeFSLock@/var/opt/scalix/ml/indexes/07f00000f3dfe135-9.13.681.46/ref/write.lock
2015-11-25 12:49:19,856 ERROR [QueueManager] [UserIndexer.commitChanges:390] com.scalix.index.api.IndexerException: Trouble unlocking /var/opt/scalix/ml/indexes/07f00000f3dfe135-9.13.681.46/ref

History

#1

Updated by Alexey Bobyr almost 4 years ago

  • Status changed from Answered to Closed

Mostly it was just a warnings and they may be omitted.
increased log level for this messages to 'trace' just to not confuse rest users.
Also ensured that 'write.lock' file is removed when indexer is closed.
Closing userIndexer if it do nothing more that 30 min. when it needed it will be a new instance.

Alexey Bobyr wrote:

[...]

[...]

[...]

#2

Updated by Alex I about 3 years ago

  • Subject changed from Lock obtain timed out while indexing to Deadlock during indexing

Also available in: Atom PDF