Project

General

Profile

Feature #60491

implement ThreadPoolExecutor to decrease queue time blocking

Added by Alexey Bobyr about 3 years ago. Updated about 3 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Alexey Bobyr
Category:
Scalix SIS
Target version:
Start date:
06/13/2016
Due date:
06/13/2016
% Done:

100%

Estimated time:
Operation System:
--

Description

SIS is working now :
1. starts separate thread with a queue.
2. on http requests it tries to to get queue and put actions for user into that queue.
3. with some delay queue processed.

Despite that processing queue is async manipulating queue is not its synchronized. Its needed even request executes in different thread and
to prevent queue corruption/data loss it must be synchronized.

As a result when processing tasks in queue for 10 users it locks entire queue. ...
It has a side effect any other requests to put some action into queue can fails because processing of tasks in queue took for e.q. 1 min for 1 user and
and all requests during it was canceled by timeout from client side.

As a result lost actions for indexer.

Suggesting implement ThreadPoolExecutorService.
default max threads count is : 100.
to change this option add new config item in sis.properties.
index.thread.pool.workers.size

With this ThreadPoolExecutorService behavior will be the following:
1. logic in controllers .... , queues the same.
2. when time comes or we have enough amount of tasks we push from our queue to ThreadPoolExecutorService items (item contains many actions for a user index)

In this case time spend on locking our queue is minimal and we are not wait while batch process process all actions for user index.
They will process in separate thread, if thread finished his job it can take another one item in queue, if there are no available threads item will wait when
available threads will come.

it can prevent this kind of errors

WARNING                        Indexer       (Indexer       ) 06.13.16 04:58:47
[OM.DMON 3008] The Indexer cannot connect to SIS at this time. Details:
Failed to connect to SIS.
Reason: 503 (Service Unavailable)
Is SIS currently unavailable?
Current message details:
         User name: xxx xxx /xxxxx 
         IndexId: 8215ba40-527935b1-575e75a1-d2aa63 
         Attempted action: Add item 0035200f5bcdd612 to folder 000316cea9effb6d

despite that http max connections is set to accept 30k connections it failed by timeout awaitng when the SIS queue unlocks.

History

#1

Updated by Alexey Bobyr about 3 years ago

some stats ~ 50 users receives at same time emails per second logs looks like this

2016-06-13 04:44:40,407  INFO [pool-1-thread-15] [BatchUpdater.processUserMods:128] User 0f220000274b1375-301.65.861.291: added 7 content, added 7 refs, deleted 7 content, 
deleted 7 refs in 989 ms
2016-06-13 04:44:40,508  INFO [pool-1-thread-23] [BatchUpdater.processUserMods:128] User 01520000274b1375-301.65.861.291: added 5 content, added 5 refs, deleted 5 content, 
deleted 5 refs in 1001 ms
2016-06-13 04:44:40,564  INFO [pool-1-thread-2] [BatchUpdater.processUserMods:128] User 0b420000274b1375-301.65.861.291: added 7 content, added 7 refs, deleted 7 content, d
eleted 7 refs in 1183 ms
2016-06-13 04:44:40,620  INFO [pool-1-thread-100] [BatchUpdater.processUserMods:128] User 00320000274b1375-301.65.861.291: added 9 content, added 9 refs, deleted 9 content,
 deleted 9 refs in 1192 ms
2016-06-13 04:44:46,490  INFO [http-nio-8080-exec-7] [UserIndexer.<init>:37] Index for user 0a220000274b1375-301.65.861.291 initialized
2016-06-13 04:44:54,073  INFO [pool-1-thread-37] [BatchUpdater.processUserMods:128] User 0c420000274b1375-301.65.861.291: added 5 content, added 5 refs, deleted 5 content, 
deleted 5 refs in 28 ms
2016-06-13 04:44:54,075  INFO [pool-1-thread-66] [BatchUpdater.processUserMods:128] User 0b220000274b1375-301.65.861.291: added 4 content, added 4 refs, deleted 4 content, 
deleted 4 refs in 25 ms
2016-06-13 04:44:54,099  INFO [pool-1-thread-54] [BatchUpdater.processUserMods:128] User 07320000274b1375-301.65.861.291: added 5 content, added 5 refs, deleted 5 content, 
deleted 5 refs in 55 ms
2016-06-13 04:44:54,149  INFO [pool-1-thread-50] [BatchUpdater.processUserMods:128] User 07420000274b1375-301.65.861.291: added 7 content, added 7 refs, deleted 7 content, 
deleted 7 refs in 101 ms
2016-06-13 04:44:54,159  INFO [pool-1-thread-52] [BatchUpdater.processUserMods:128] User 00520000274b1375-301.65.861.291: added 5 content, added 5 refs, deleted 5 content, 
deleted 5 refs in 61 ms
2016-06-13 04:44:54,218  INFO [pool-1-thread-68] [BatchUpdater.processUserMods:128] User 06520000274b1375-301.65.861.291: added 6 content, added 6 refs, deleted 6 content, 
deleted 6 refs in 171 ms
2016-06-13 04:44:54,231  INFO [pool-1-thread-79] [BatchUpdater.processUserMods:128] User 08520000274b1375-301.65.861.291: added 7 content, added 7 refs, deleted 7 content, deleted 7 refs in 148 ms
2016-06-13 04:44:54,272  INFO [pool-1-thread-45] [BatchUpdater.processUserMods:128] User 04520000274b1375-301.65.861.291: added 8 content, added 8 refs, deleted 8 content, deleted 8 refs in 101 ms

For some users works reindex.

for users to manipulating with index files take a long time

deleted 9 refs in 1284 ms
2016-06-13 04:44:11,371  INFO [pool-1-thread-12] [BatchUpdater.processUserMods:128] User 03520000274b1375-301.65.861.291: added 9 content, added 9 refs, deleted 9 content, deleted 9 refs in 1256 ms
2016-06-13 04:44:11,393  INFO [pool-1-thread-100] [BatchUpdater.processUserMods:128] User 05320000274b1375-301.65.861.291: added 8 content, added 8 refs, deleted 8 content, deleted 8 refs in 1702 ms
2016-06-13 04:44:11,396  INFO [pool-1-thread-9] [BatchUpdater.processUserMods:128] User 03320000274b1375-301.65.861.291: added 6 content, added 6 refs, deleted 6 content, deleted 6 refs in 1472 ms

...

After implementing ThreadPoolExecutorService requests finishes more faster .... have not seen SIS unavailable for during concurrency tests.

#3

Updated by Alexey Bobyr about 3 years ago

  • Tracker changed from Feature internal to Feature
  • Target version changed from 12.7 to Scalix 12.6
#4

Updated by Alex I about 3 years ago

  • Subject changed from enhancement implement ThreadPoolExecutor to decrease blocking queue time. to implement ThreadPoolExecutor to decrease queue time blocking

Also available in: Atom PDF