Apache segmentation fault

faris · Unread post by **faris** » Mon Sep 22, 2008 9:35 am

What are people's httpd.conf like for Maxclients etc?

I wonder if reducing them might help?

We are on 100 for a 4Gb machine. Out of the box I think it was set to 30 or something, and that was way too low.

If this issue is somehow memory related I'm wondering if adjusting this setting might help?

Faris.

Unread post by **scott** » Mon Sep 22, 2008 10:16 am

That is an excellent point, Im using prefork with:

MaxClients 256
MaxRequestsPerChild 4000

What does everyone else who is experiencing this have those settings at?

Kalimari · Unread post by **Kalimari** » Mon Sep 22, 2008 10:46 am

RHEL 4 Box (with one recent occurrence of segmentation fault):

MaxClients 256
MaxRequestsPerChild 4000

biggles · Unread post by **biggles** » Mon Sep 22, 2008 11:47 am

MaxClients 10
MaxRequestsPerChild 4000

on both my servers...

faris · Unread post by **faris** » Mon Sep 22, 2008 12:11 pm

How do you choose whether to use prefork MPM or Worker MPM?

Also I found some very old posts about segfaults in apache with mod_sec:

http://blog.arithm.com/2008/02/09/how-t ... _security/
(talks about using the wrong header - can't see this as being useful but I include it just in case)

And in another very old post about Apache segfaults (not really mod_security) was this comment:

"The problem is the threaded MPM of apache. Using worker mpm it works fine ... seems to be a non-thread-safe library somewhere ..."

..and I'm wondering if somehow any of this has anything to do with anything. Remember these are very old threads and I don't think they relate to the current problems. I just got to thinking about memory and stuff and also about possible workarounds.

Sorry if all this is dumb and useless.

Faris.

Unread post by **scott** » Mon Sep 22, 2008 3:24 pm

This is preliminary, but on my system I had been using mod_mem_cache, with these settings:

<IfModule mod_mem_cache.c>
CacheEnable mem /
MCacheSize 65536
MCacheMaxObjectCount 400
MCacheMinObjectSize 64
MCacheMaxObjectSize 4096
</IfModule>

Disabling this completely seems to have resolved the segfault problem. I'd be interested looking at the httpd.conf files for anyone experiencing this that is not using a VPS. I suspect that the VPS users are falling victim of something else entirely.

aus-city · Unread post by **aus-city** » Tue Sep 23, 2008 7:31 am

Scott,

From /etc/httpd/conf/httpd.conf

Also I don't have a <IfModule mod_mem_cache.c> I am not using it so won't help me

Dedicated server NOT VDS

Do you want the whole httpd.conf file emailed to support?

Here is the MPM extracts:

# prefork MPM
# StartServers: number of server processes to start
# MinSpareServers: minimum number of server processes which are kept spare
# MaxSpareServers: maximum number of server processes which are kept spare
# ServerLimit: maximum value for MaxClients for the lifetime of the server
# MaxClients: maximum number of server processes allowed to start
# MaxRequestsPerChild: maximum number of requests a server process serves
<IfModule prefork.c>
StartServers 8
MinSpareServers 5
MaxSpareServers 20
ServerLimit 256
MaxClients 256
MaxRequestsPerChild 4000
</IfModule>

# worker MPM
# StartServers: initial number of server processes to start
# MaxClients: maximum number of simultaneous client connections
# MinSpareThreads: minimum number of worker threads which are kept spare
# MaxSpareThreads: maximum number of worker threads which are kept spare
# ThreadsPerChild: constant number of worker threads in each server process
# MaxRequestsPerChild: maximum number of requests a server process serves
<IfModule worker.c>
StartServers 2
MaxClients 150
MinSpareThreads 25
MaxSpareThreads 75
ThreadsPerChild 25
MaxRequestsPerChild 0
</IfModule>

I can tell you its definitely traffic or load related. My servers main domain has a new forum and mod_sec is going down about every 8 to 12 hours

Unread post by **scott** » Tue Sep 23, 2008 8:20 am

try commenting out the mod_mem_cache module completely.

Other than that, have you made any changes to httpd.conf?

aus-city · Unread post by **aus-city** » Tue Sep 23, 2008 8:26 am

Now my F9 internal server with ASL is doing it. Its got very small traffic..

I just ran an asl -s -f and httpd failed to stop and start, I had to killall httpd and then it ran okay.

Here is its httpd.conf:

# prefork MPM
# StartServers: number of server processes to start
# MinSpareServers: minimum number of server processes which are kept spare
# MaxSpareServers: maximum number of server processes which are kept spare
# ServerLimit: maximum value for MaxClients for the lifetime of the server
# MaxClients: maximum number of server processes allowed to start
# MaxRequestsPerChild: maximum number of requests a server process serves
<IfModule prefork.c>
StartServers 8
MinSpareServers 5
MaxSpareServers 20
ServerLimit 256
MaxClients 256
MaxRequestsPerChild 4000
</IfModule>

# worker MPM
# StartServers: initial number of server processes to start
# MaxClients: maximum number of simultaneous client connections
# MinSpareThreads: minimum number of worker threads which are kept spare
# MaxSpareThreads: maximum number of worker threads which are kept spare
# ThreadsPerChild: constant number of worker threads in each server process
# MaxRequestsPerChild: maximum number of requests a server process serves
<IfModule worker.c>
StartServers 2
MaxClients 150
MinSpareThreads 25
MaxSpareThreads 75
ThreadsPerChild 25
MaxRequestsPerChild 0
</IfModule>

aus-city · Unread post by **aus-city** » Tue Sep 23, 2008 8:28 am

Hi Scott,

On both these servers there is no module like you had put, but I find this:

LoadModule mem_cache_module modules/mod_mem_cache.so

Is that it?

I have commented it out and just restarted httpd.

LoadModule cache_module modules/mod_cache.so
LoadModule suexec_module modules/mod_suexec.so
LoadModule disk_cache_module modules/mod_disk_cache.so
LoadModule file_cache_module modules/mod_file_cache.so
#LoadModule mem_cache_module modules/mod_mem_cache.so

No its a stock Fedora httpd.conf completely..

Here is a dump of loaded modules now...

Loaded Modules core prefork http_core mod_so mod_auth_basic mod_auth_digest mod_authn_file mod_authn_alias mod_authn_anon mod_authn_dbm mod_authn_default mod_authz_host mod_authz_user mod_authz_owner mod_authz_groupfile mod_authz_dbm mod_authz_default util_ldap mod_authnz_ldap mod_include mod_log_config mod_logio mod_env mod_ext_filter mod_mime_magic mod_expires mod_deflate mod_headers mod_usertrack mod_setenvif mod_mime mod_dav mod_status mod_autoindex mod_info mod_dav_fs mod_vhost_alias mod_negotiation mod_dir mod_actions mod_speling mod_userdir mod_alias mod_rewrite mod_proxy mod_proxy_balancer mod_proxy_ftp mod_proxy_http mod_proxy_connect mod_cache mod_suexec mod_disk_cache mod_file_cache mod_cgi mod_security2 mod_unique_id mod_bw mod_fcgid mod_evasive20 mod_perl mod_php5 mod_proxy_ajp mod_python mod_ssl

Lets let it cook over night, if its still running in the morning I will be a happy camper...

Scott,

While I have your attention for these:
<IfModule prefork.c>
StartServers 8
MinSpareServers 5
MaxSpareServers 20
ServerLimit 256
MaxClients 256
MaxRequestsPerChild 4000
</IfModule>
and
<IfModule worker.c>
StartServers 2
MaxClients 150
MinSpareThreads 25
MaxSpareThreads 75
ThreadsPerChild 25
MaxRequestsPerChild 0
</IfModule>

What would acceptable limits be for a dual core 3.0 GHz with 4GB ram running i686?

Unread post by **scott** » Tue Sep 23, 2008 10:04 am

None of the maxclients, keepalives, etc had any effect on my test box. The culprit in my case was all memcache.

faris · Unread post by **faris** » Tue Sep 23, 2008 12:57 pm

Scott knows already, but in case it helps/doesn't help anybody else, I'm afraid that disabling mod_mem_cache didn't help on the two Virtuozzo VPSes we have that are exhibiting this problem.

In both cases the segfaults come by the score after a restart of the VPS, then, as memory usage increases, and after several apache restarts over the course of a few days to stop the segfault avalanche, things sabilise and I don't see them anymore, with the possible exception of after new rules are released/asl -u. I still get rule processing failures though.

The issue seems to be triggered by a lack of memory in our case because that's the only thing that changes following a reboot.

I was very certain that disabling mod_mem_cache, which is obviously memory-related, would solve the problem. As you can imagine I was somewhat crushed when removing it didn't work, especially when my initial few tests seem to indicate that it had worked.

Mind you, there is probably much more to this than meets the eye.

Faris.

p.s. remember, these are VPSes -- not physical servers. I'm crossing my fingers that the removal of mod_mem_cache will help those with physical servers just as it is doing in Scott's tests.

aus-city · Unread post by **aus-city** » Tue Sep 23, 2008 6:11 pm

Scott,

Its been 10 hours since I commented out:

#LoadModule mem_cache_module modules/mod_mem_cache.so

and not a single hickup

The last 3 days with the traffic I been seeing every 2 to 4 hours is the longest it held up so I am quite sure this is the issue.

Does this help track down what in mod_security is causing this? I also assume by disabling the mod_mem_cache its putting more load on the CPU?

I assume for VDS it is because it has to be disabled on the actual server?

Anyway happy its patched for now. If I see anything contrary I will let you know.

Unread post by **scott** » Tue Sep 23, 2008 7:53 pm

That is really good to hear, if you could try turning it on again and see if starts doing it again. That would help confirm that it really is the problem on your box.

I'm pretty sure it is, but this is going to help rule it out. I switched it on and off over a few hours on the system here and it was consistent (on segfault, off no segfault). Although in some cases it took over 1/2 an hour to segfault once I turned it on again. So give it at least that if you dont start getting them right away.

faris · Unread post by **faris** » Thu Sep 25, 2008 6:21 am

I'm afraid I had a spate of segfaults followed by continuing rule processing failures late last night and this morning.

Even though I was still getting them at boot, I was confident that the removal of mod_mem_cache and fiddling with the maxclients settings had massively improved things for me.

Unfortunately I was wrong.

Ah well. Back to square one.

I hope aus-city's system is still stable?

Faris.