Project

General

Profile

Bug #50038

Device connections from apache saturate tomcat leading to 503 unavailability

Added by Just User about 5 years ago. Updated almost 5 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Alexey Bobyr
Category:
Scalix Wireless
Target version:
Start date:
10/21/2015
Due date:
% Done:

0%

Estimated time:
Operation System:
--

Description

Stefan Feicke 2015-06-25 07:16:30 EDT
Scalix 12.5
centOS 7 64 bit
apache version httpd-2.4.6-31.el7.centos.x86_64

Active-Sync is running on a dedicated server with own platform

Current behaviour: after fresh reboot the devices connect successfully to EAS
after a certain time the devices will receive a 503 error

logs:

[Thu Jun 25 08:30:03.238479 2015] [proxy_ajp:error] [pid 7817] (70007)The timeout specified has expired: [client 88.130.223.161:49164] AH00878: read response failed from (null) ()
[Thu Jun 25 08:30:03.919964 2015] [proxy_ajp:error] [pid 7980] (70007)The timeout specified has expired: AH01030: ajp_ilink_receive() can't receive header
[Thu Jun 25 08:30:03.920025 2015] [proxy_ajp:error] [pid 7980] [client 80.187.106.14:31400] AH00992: ajp_read_header: ajp_ilink_receive failed
[Thu Jun 25 08:30:03.920031 2015] [proxy_ajp:error] [pid 7980] (70007)The timeout specified has expired: [client 80.187.106.14:31400] AH00878: read response failed from (null) ()
[Thu Jun 25 08:30:09.583098 2015] [proxy_ajp:error] [pid 7972] (70007)The timeout specified has expired: AH01030: ajp_ilink_receive() can't receive header

History

#1

Updated by Volodymyr Tomash about 5 years ago

any tomcat tuning documentation for environments with a larger number of EAS-devices would be helpful.

Chris Barret had a tomcat tuning presentation for 11.5 on apache 2.2 So our question is:

is there a tuning document available for a larger number of active sync devices?

#2

Updated by Alexey Bobyr about 5 years ago

Please try to use this tool
https://github.com/scalix/general-utilities/tree/master/live_connection_monitor

it will how many opened connections currently on the server, how many of them in close_wait state (httpd or java)

The cause of close_wait connections is when client drop request for e.q. poor internet connection (not stable internet connection). In logs you can see 'Broken pipe'. Also it can be related to apache ajp connector. I have seen that on Centos 5 x86 server close_wait connections is less rather than in Centos 5 x64.

For now there are several workarounds.
1. increase max connections at httpd side and tomcat 6 to handle more.
2. use Apache MPM Worker instead of default Apache Prefork http://www.garron.me/en/blog/apache2-mpm-worker-prefork-php.html
Please note that Apache MPM Worker is not working with php so if you have site there or something worker won't work with mod_php you will have to remove it.

For Centos 5,6 to switch to MPM Worker please do following:
a.Edit: /etc/sysconfig/httpd

b.Uncomment:
HTTPD=/usr/sbin/httpd.worker
c. Restart

There are no special documentation of tuning tomcat .
You can use this presentation for e.g.
http://www.slideshare.net/Paganel/tomcatx-performancetuning

We know about that problem and working on it.
We have builds with tomcat 7 where performance is better if compare with tomcat 6

if nothing of those methods that I mentioned the only one is to restart httpd and tomcat by cron. for e.g. restart every day at 6 or 7 o'clock in the morning.

How many users/devices do you have on the AS server? it will be helpful information for me

#3

Updated by Alexey Bobyr about 5 years ago

For one customer helped the following settings:
1. httpd worker
2. httpd.conf
- KeepAlive Off --> KeepAlive On
- MaxKeepAliveRequests 100 --> MaxKeepAliveRequests 300

if httpd worker enabled encrease max clients
<IfModule worker.c>
StartServers 10
MaxClients 4000
MinSpareThreads 15
MaxSpareThreads 35
ThreadsPerChild 100
MaxRequestsPerChild 0
</IfModule>
if prefork.c is used please increase values for prefork.c

3. for each ajp connectiors one extra option - keepalive=on for e.g.
ProxyPass /api ajp://[Yor server mame]:8009/api retry=1 ttl=5 ping=5 keepalive=On
files are in folder - /etc/opt/scalix-tomcat/connector/ajp
4. /var/opt/scalix/??/tomcat/conf/server.xml
<Connector port="8080"
....
maxThreads="4050"
acceptCount="350"
maxConnections="-1"

the same for Connector port="8009"

Also it's better to update scalix to the latest version - 12.5.1 has been released

Please contact with scalix support for further information

#4

Updated by Volodymyr Tomash about 5 years ago

  • Subject changed from https://bugzilla.scalix.com/show_bug.cgi?id=50038 to Device connections from apache saturate tomcat leading to 503 unavailability
#5

Updated by Alexey Bobyr almost 5 years ago

  • Status changed from Answered to Closed

Please update to 12.5.2

Just User wrote:

Stefan Feicke 2015-06-25 07:16:30 EDT
Scalix 12.5
centOS 7 64 bit
apache version httpd-2.4.6-31.el7.centos.x86_64

Active-Sync is running on a dedicated server with own platform

Current behaviour: after fresh reboot the devices connect successfully to EAS
after a certain time the devices will receive a 503 error

logs:

[Thu Jun 25 08:30:03.238479 2015] [proxy_ajp:error] [pid 7817] (70007)The timeout specified has expired: [client 88.130.223.161:49164] AH00878: read response failed from (null) ()
[Thu Jun 25 08:30:03.919964 2015] [proxy_ajp:error] [pid 7980] (70007)The timeout specified has expired: AH01030: ajp_ilink_receive() can't receive header
[Thu Jun 25 08:30:03.920025 2015] [proxy_ajp:error] [pid 7980] [client 80.187.106.14:31400] AH00992: ajp_read_header: ajp_ilink_receive failed
[Thu Jun 25 08:30:03.920031 2015] [proxy_ajp:error] [pid 7980] (70007)The timeout specified has expired: [client 80.187.106.14:31400] AH00878: read response failed from (null) ()
[Thu Jun 25 08:30:09.583098 2015] [proxy_ajp:error] [pid 7972] (70007)The timeout specified has expired: AH01030: ajp_ilink_receive() can't receive header

Also available in: Atom PDF