Feature #60491
implement ThreadPoolExecutor to decrease queue time blocking
100%
Description
SIS is working now :
1. starts separate thread with a queue.
2. on http requests it tries to to get queue and put actions for user into that queue.
3. with some delay queue processed.
Despite that processing queue is async manipulating queue is not its synchronized. Its needed even request executes in different thread and
to prevent queue corruption/data loss it must be synchronized.
As a result when processing tasks in queue for 10 users it locks entire queue. ...
It has a side effect any other requests to put some action into queue can fails because processing of tasks in queue took for e.q. 1 min for 1 user and
and all requests during it was canceled by timeout from client side.
As a result lost actions for indexer.
Suggesting implement ThreadPoolExecutorService.
default max threads count is : 100.
to change this option add new config item in sis.properties.
index.thread.pool.workers.size
With this ThreadPoolExecutorService behavior will be the following:
1. logic in controllers .... , queues the same.
2. when time comes or we have enough amount of tasks we push from our queue to ThreadPoolExecutorService items (item contains many actions for a user index)
In this case time spend on locking our queue is minimal and we are not wait while batch process process all actions for user index.
They will process in separate thread, if thread finished his job it can take another one item in queue, if there are no available threads item will wait when
available threads will come.
it can prevent this kind of errors
WARNING Indexer (Indexer ) 06.13.16 04:58:47
[OM.DMON 3008] The Indexer cannot connect to SIS at this time. Details:
Failed to connect to SIS.
Reason: 503 (Service Unavailable)
Is SIS currently unavailable?
Current message details:
User name: xxx xxx /xxxxx
IndexId: 8215ba40-527935b1-575e75a1-d2aa63
Attempted action: Add item 0035200f5bcdd612 to folder 000316cea9effb6d
despite that http max connections is set to accept 30k connections it failed by timeout awaitng when the SIS queue unlocks.
History
Updated by Alexey Bobyr over 3 years ago
Updated by Alexey Bobyr some stats ~ 50 users receives at same time emails per second logs looks like this
2016-06-13 04:44:40,407 INFO [pool-1-thread-15] [BatchUpdater.processUserMods:128] User 0f220000274b1375-301.65.861.291: added 7 content, added 7 refs, deleted 7 content, deleted 7 refs in 989 ms 2016-06-13 04:44:40,508 INFO [pool-1-thread-23] [BatchUpdater.processUserMods:128] User 01520000274b1375-301.65.861.291: added 5 content, added 5 refs, deleted 5 content, deleted 5 refs in 1001 ms 2016-06-13 04:44:40,564 INFO [pool-1-thread-2] [BatchUpdater.processUserMods:128] User 0b420000274b1375-301.65.861.291: added 7 content, added 7 refs, deleted 7 content, d eleted 7 refs in 1183 ms 2016-06-13 04:44:40,620 INFO [pool-1-thread-100] [BatchUpdater.processUserMods:128] User 00320000274b1375-301.65.861.291: added 9 content, added 9 refs, deleted 9 content, deleted 9 refs in 1192 ms 2016-06-13 04:44:46,490 INFO [http-nio-8080-exec-7] [UserIndexer.<init>:37] Index for user 0a220000274b1375-301.65.861.291 initialized 2016-06-13 04:44:54,073 INFO [pool-1-thread-37] [BatchUpdater.processUserMods:128] User 0c420000274b1375-301.65.861.291: added 5 content, added 5 refs, deleted 5 content, deleted 5 refs in 28 ms 2016-06-13 04:44:54,075 INFO [pool-1-thread-66] [BatchUpdater.processUserMods:128] User 0b220000274b1375-301.65.861.291: added 4 content, added 4 refs, deleted 4 content, deleted 4 refs in 25 ms 2016-06-13 04:44:54,099 INFO [pool-1-thread-54] [BatchUpdater.processUserMods:128] User 07320000274b1375-301.65.861.291: added 5 content, added 5 refs, deleted 5 content, deleted 5 refs in 55 ms 2016-06-13 04:44:54,149 INFO [pool-1-thread-50] [BatchUpdater.processUserMods:128] User 07420000274b1375-301.65.861.291: added 7 content, added 7 refs, deleted 7 content, deleted 7 refs in 101 ms 2016-06-13 04:44:54,159 INFO [pool-1-thread-52] [BatchUpdater.processUserMods:128] User 00520000274b1375-301.65.861.291: added 5 content, added 5 refs, deleted 5 content, deleted 5 refs in 61 ms 2016-06-13 04:44:54,218 INFO [pool-1-thread-68] [BatchUpdater.processUserMods:128] User 06520000274b1375-301.65.861.291: added 6 content, added 6 refs, deleted 6 content, deleted 6 refs in 171 ms 2016-06-13 04:44:54,231 INFO [pool-1-thread-79] [BatchUpdater.processUserMods:128] User 08520000274b1375-301.65.861.291: added 7 content, added 7 refs, deleted 7 content, deleted 7 refs in 148 ms 2016-06-13 04:44:54,272 INFO [pool-1-thread-45] [BatchUpdater.processUserMods:128] User 04520000274b1375-301.65.861.291: added 8 content, added 8 refs, deleted 8 content, deleted 8 refs in 101 ms
For some users works reindex.
for users to manipulating with index files take a long time
deleted 9 refs in 1284 ms 2016-06-13 04:44:11,371 INFO [pool-1-thread-12] [BatchUpdater.processUserMods:128] User 03520000274b1375-301.65.861.291: added 9 content, added 9 refs, deleted 9 content, deleted 9 refs in 1256 ms 2016-06-13 04:44:11,393 INFO [pool-1-thread-100] [BatchUpdater.processUserMods:128] User 05320000274b1375-301.65.861.291: added 8 content, added 8 refs, deleted 8 content, deleted 8 refs in 1702 ms 2016-06-13 04:44:11,396 INFO [pool-1-thread-9] [BatchUpdater.processUserMods:128] User 03320000274b1375-301.65.861.291: added 6 content, added 6 refs, deleted 6 content, deleted 6 refs in 1472 ms
...
After implementing ThreadPoolExecutorService requests finishes more faster .... have not seen SIS unavailable for during concurrency tests.
Updated by Alexey Bobyr over 3 years ago
- Tracker changed from Feature internal to Feature
- Target version changed from 12.7 to Scalix 12.6
Updated by Alex I about 3 years ago
Updated by Alex I - Subject changed from enhancement implement ThreadPoolExecutor to decrease blocking queue time. to implement ThreadPoolExecutor to decrease queue time blocking