Project

General

Profile

Bug #50014

issue with Scalix indexer.

Added by Volodymyr Tomash over 4 years ago. Updated over 4 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Alex I
Category:
Scalix SIS
Target version:
Start date:
10/21/2015
Due date:
10/21/2015
% Done:

100%

Estimated time:
Operation System:
--

Description

issue with Scalix indexer.
/var/opt/scalix/ss/s/tomcat/logs/scalix-sis-indexer.log

2013-12-04 04:08:44,543 WARN [main] [BatchUpdater.processMods:84] Failed to get reference modifier for user 0166d100c1c23a84-5.441.202.591
2013-12-04 04:08:44,543 WARN [main] [BatchUpdater.processMods:84] Failed to get reference modifier for user 03b8c100c1c23a84-5.441.202.591
2013-12-04 04:08:44,544 WARN [main] [BatchUpdater.processMods:84] Failed to get reference modifier for user 0f78c100c1c23a84-5.441.202.591
2013-12-04 04:08:44,545 WARN [main] [BatchUpdater.processMods:84] Failed to get reference modifier for user 06b4c000c1c23a84-5.441.202.591
2013-12-04 04:08:44,545 WARN [main] [BatchUpdater.processMods:84] Failed to get reference modifier for user 0fe9c100c1c23a84-5.441.202.591
2013-12-04 04:08:44,546 INFO [main] [InternalIndexerServlet.destroy:396] Indexer Queue cleared and stopped
2013-12-04 04:08:44,548 INFO [main] [SISConfig.unload:167] SIS unloaded

2013-12-04 04:10:13,958 INFO [main] [SISConfig.init:61] Starting Scalix Search and Index Service (SIS) version 12.0.1.14335
2013-12-04 04:10:14,018 INFO [main] [SISConfig.loadConfigFromFile:175] SIS Indexer configuration loaded from file /var/opt/scalix/ss/sis/sis.properties
2013-12-04 04:10:14,038 INFO [main] [SISConfig.init:104] Index root is /var/opt/scalix/ss/indexes
2013-12-04 04:10:14,120 INFO [main] [SISConfig.load:47] SIS initialized and ready for indexing
2013-12-04 04:10:14,226 INFO [main] [InternalIndexerServlet.init:110] ical4j properties set up
2013-12-04 04:10:14,227 INFO [main] [InternalIndexerServlet.init:111] ====> SIS indexer is starting up, standby <====

/var/opt/scalix/ss/s/logs/fatal:

ERROR Indexer (Indexer ) Wed Dec 4 04:09:04 2013
[OM.DMON 3007] The SIS has been unable to process the Indexer's request. Details:
SIS could not handle this request.
Reason: 500 (Internal Server Error)
Indexer Service has discarded the message:
User name: Sandra Forstner /sxs-MM
IndexId: 590cac3-48a32c1c-5273eca9-54800f1
Attempted action: Add item 00587d879126e161 to folder 0053e68872fbd6be
Pid of logging process: 20951

found forum threads
http://www.scalix.com/forums/viewtopic.php?f=2&t=13623:
...
As you have noticed, the upgrade to 11.4.4 does indeed expose some new error messages from the SIS indexer. However, as some of you suspected, they are not fatal and can be safely ignored. The underlying cause is that not all mail message headers comply fully with the relevant RFC specs and are rejected by the SIS. This situation has always existed, it is just that logging in 11.4.4 is more sensitive which is why you have started to notice them.

There is also a bug fix in progress to clean up the message headers to remove the problem at source.

Please watch this thread for details on when the fix will be ready.
...

This issue appears not for all users after updgrade from scalix 11.4.6 to scalix 12.0.3


Files

omqdmsg.tar omqdmsg.tar 40 KB Volodymyr Tomash, 10/21/2015 07:38 AM
omq_one_month_later.tar omq_one_month_later.tar 40 KB Volodymyr Tomash, 10/21/2015 07:39 AM
unix.out-50015 unix.out-50015 514 KB Volodymyr Tomash, 10/21/2015 07:40 AM

History

#1

Updated by Volodymyr Tomash over 4 years ago

Hello all.
We have issues with one of our Scalix customer's server.
After updating to Scalix 12.0.3 on the server, it had troubles with sisindexer.
We have changed some Tomcat server settings and the problem partly was solved, but now there is another problem - the Tomcat should be restarted every 24 hours.
That's part of the log catalina.out:
...
Jan 9, 2014 6:30:03 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/webmail] is still processing a request that has yet to finish. This is very likely to create a memory leak. You can control the time allowed for requests to finish by using the unloadDelay attribute of the standard Context implementation.
...
Jan 9, 2014 6:30:15 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/sis] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@480143e8]) and a value of type [org.apache.lucene.index.SegmentTermEnum] (value [org.apache.lucene.index.SegmentTermEnum@4b115946]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak.
...
I set maxThreads option for Tomcat to 750 (which is quite a large value), but it didn't helped a lot.
As we found out with Alexey Bobyr reason for this is most likely in mod_proxy - it caches queries and tries to recover them after Tomcat restarting.
But I have a suspicion that the garbage collector can't handle. It is possible that the Scalix server has considerable memory leak,
and how to check it?
This is a very important issue, for a long time we have constant problems with the client's server.

#2

Updated by Alexey Bobyr over 4 years ago

ok

#3

Updated by Alex I over 4 years ago

It's not memory leak in server, because sis indexer is java application...
maybe it has sense to replace url in sis-related configs to not use modproxy_ajp and go directly to tomcat url ?

#4

Updated by Alex I over 4 years ago

as an option we could try to use apache mod_jk instead of mod_ajp(mod_proxy).

Also available in: Atom PDF