Page 7 of 11
Posted: Sun Nov 23, 2008 5:34 pm
by faris
No, dedicated servers too.
To summarise my experience on several VPSes running VZ4*:
For reasons unknown, unlike on a dedicated server, all availablr RAM does not get grabbed instantly and instead you'll see memory usage increase over time.
With all rules enabled, after a reboot you'll get lots of segfaults, one after the other. Restarting apache a few times a day helps and eventually as morer memory get grabbed and held onto by the VPS, the segfaults reduce. This even applies to a VPS with 4Gb of memory allocated.
With the domain-blacklist rules removed, no segfaults happen at all. None. At least not until recently, when I'm seeing one or two a week happen. I think this because the other rulesets have increased in size, and essentially it is the size/amount of work that mod_sec/apache has to deal with that is causing the issue.
Unfortunately no httpd-debuginfo rpm seems to exist for Centos 4 or 5 so I can't try installing that, which solved hostingguy's or was it aus-city's problem.
I've tried several versions of apache and it makes no difference.
Like I say, the amount of RAM allocated to the VPS makes no difference either.
Most of the segfaults I personally see happen when people use webmail but it isn't just webmail.
Removing modules, e.g. mod_memcache has no effect.
Faris.
Definitely an apache bug
Posted: Sun Nov 23, 2008 6:08 pm
by mikeshinn
Which apache modules are loaded on the boxes where you see this happen?
Posted: Sun Nov 23, 2008 7:12 pm
by faris
This is my list:
Code: Select all
LoadModule access_module modules/mod_access.so
LoadModule auth_module modules/mod_auth.so
LoadModule auth_anon_module modules/mod_auth_anon.so
LoadModule auth_dbm_module modules/mod_auth_dbm.so
LoadModule auth_digest_module modules/mod_auth_digest.so
LoadModule ldap_module modules/mod_ldap.so
LoadModule auth_ldap_module modules/mod_auth_ldap.so
LoadModule include_module modules/mod_include.so
LoadModule log_config_module modules/mod_log_config.so
LoadModule env_module modules/mod_env.so
#LoadModule mime_magic_module modules/mod_mime_magic.so
LoadModule cern_meta_module modules/mod_cern_meta.so
LoadModule expires_module modules/mod_expires.so
LoadModule deflate_module modules/mod_deflate.so
LoadModule headers_module modules/mod_headers.so
LoadModule usertrack_module modules/mod_usertrack.so
LoadModule setenvif_module modules/mod_setenvif.so
LoadModule mime_module modules/mod_mime.so
#LoadModule dav_module modules/mod_dav.so
LoadModule status_module modules/mod_status.so
LoadModule autoindex_module modules/mod_autoindex.so
LoadModule asis_module modules/mod_asis.so
LoadModule info_module modules/mod_info.so
#LoadModule dav_fs_module modules/mod_dav_fs.so
LoadModule vhost_alias_module modules/mod_vhost_alias.so
LoadModule negotiation_module modules/mod_negotiation.so
LoadModule dir_module modules/mod_dir.so
LoadModule imap_module modules/mod_imap.so
LoadModule actions_module modules/mod_actions.so
LoadModule speling_module modules/mod_speling.so
LoadModule userdir_module modules/mod_userdir.so
LoadModule alias_module modules/mod_alias.so
LoadModule rewrite_module modules/mod_rewrite.so
LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_ftp_module modules/mod_proxy_ftp.so
LoadModule proxy_http_module modules/mod_proxy_http.so
LoadModule proxy_connect_module modules/mod_proxy_connect.so
LoadModule cache_module modules/mod_cache.so
LoadModule suexec_module modules/mod_suexec.so
LoadModule disk_cache_module modules/mod_disk_cache.so
LoadModule file_cache_module modules/mod_file_cache.so
#LoadModule mem_cache_module modules/mod_mem_cache.so
LoadModule cgi_module modules/mod_cgi.so
LoadModule logio_module /usr/lib/httpd/modules/mod_logio.so
#LoadModule jk_module /usr/lib/httpd/modules/mod_jk.so
#LoadFile /usr/lib/libxml2.so
#LoadModule unique_id_module modules/mod_unique_id.so
#LoadModule security2_module modules/mod_security2.so
LoadModule frontpage_module /usr/lib/httpd/modules/mod_frontpage.so
(note - mod_security and mod_unique get loaded elsewhere)
Try disabling these modules
Posted: Sun Nov 23, 2008 8:01 pm
by mikeshinn
You're still using mod_cache which could be part of the problem, mom_mem_cache is a sub-module for mod_cache. So try turning mod_cache off as well, and make sure you turn off these two modules as well (they shouldnt work without mod_cache):
LoadModule disk_cache_module modules/mod_disk_cache.so
LoadModule file_cache_module modules/mod_file_cache.so
Also mod_file_cache is still experimental so I wouldnt want to run it without the time to debug it and report any issues to the apache guys - mod_disk_cache is not experimental.
Also, unrelated to your issue, if you dont need any of those other modules I recommend you turn them off. For example the proxy modules are generally not needed unless you are proxying something, and can allow someone to use your machine as proxy without your permission (although there are ASL rules to prevent this).
LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_ftp_module modules/mod_proxy_ftp.so
LoadModule proxy_http_module modules/mod_proxy_http.so
LoadModule proxy_connect_module modules/mod_proxy_connect.so
Posted: Mon Nov 24, 2008 8:40 pm
by faris
Interesting. OK, I'll have a go later on.
All that stuff was the default config (apart from the stuff I commented out, like mod_dav) in the C4 install (centosplus repo enabled).
Faris.
Posted: Tue Nov 25, 2008 9:34 am
by faris
Unfortunately removing the suggested modules had no effect.
With the full ruleset enabled I was able to generate segfaults almost on demand by simply logging in to webmail.
It quite happily coped with an ab -n 1000 (run from a separate machine, but pointed to a .php page) though, with no segfaults.
Faris.
Posted: Tue Nov 25, 2008 12:02 pm
by scott
neat, so you've got a reproducible crash with just webmail now? Id love to see a core dump from that if you can get one. If anything you might have found a more reliable way of duplicating than the ab bruteforce.
Posted: Tue Nov 25, 2008 1:28 pm
by faris
Yeah but you need the http-debuginfo thing to give you any useful info, don't you, and I just can't find one for httpd-2.0.63-2.el4 Centos 4.
Can you point me to some source code?
Faris.
Posted: Tue Dec 02, 2008 8:36 am
by faris
Did we lose some posts? I could have sworn I posted another message in this thread about things getting worse.
However, I'm no longer absolutely convinced it is mod_sec that's causing this particular spate of segfaults for me.
I think it is all horde. Especially since they started getting much much worse when I updates Plesk's base packages (inc horde updates) this weekend.
Here's the kind of thing I'm seeing:
The first one looks like a clear attempt at an exploit?
Code: Select all
[Mon Dec 01 13:42:31 2008] [error] [client xx.xx.xx.xx] File does not exist: /home/httpd/vhosts/domain.co.uk/httpdocs/robots.txt
[Mon Dec 01 13:50:51 2008] [error] [client xx.xx.xx.xx] PHP Fatal error: Call to undefined function C\xcc\xcb\xb7ion_\x18\xe8\xcb\xb7X\x98\xcc\xb7\x18() in /usr/share/psa-horde/lib/Horde.php on line 1228, referer: http://webmail.domain.com(etc)
*** glibc detected *** corrupted double-linked list: 0xbfb0d378 ***
[Mon Dec 01 13:50:52 2008] [notice] child pid 25854 exit signal Aborted (6)
But then things look more general:
Here's a missing class:
Code: Select all
[Tue Dec 02 10:55:13 2008] [error] [client xx.xxx.xx.xxx] PHP Fatal error: Class 'IMP_Folder' not found in /usr/share/psa-horde/imp/lib/IMP.php on line 572, referer: http://webmail.domain.co.uk/horde/imp/mailbox.php?mailbox=INBOX
[Tue Dec 02 10:55:13 2008] [notice] child pid 20080 exit signal Segmentation fault (11)
*** glibc detected *** free(): invalid next size (normal): 0xbfa01fa0 ***
Here's an undefined function:
Code: Select all
[Mon Dec 01 21:15:11 2008] [error] [client yy.yy.yy.yy] ModSecurity: [file "/etc/httpd/modsecurity.d/10_asl_rules.conf"] [line "1910"] [id "340684"] [rev "4"] [msg "Remote File Injection attempt in ARGS"] [severity "CRITICAL"] Warning$
[Mon Dec 01 21:15:11 2008] [error] [client yy.yy.yy.yy] ModSecurity: [file "/etc/httpd/modsecurity.d/10_asl_rules.conf"] [line "1912"] [id "340685"] [rev "4"] [msg "Remote File Injection attempt in ARGS"] [severity "CRITICAL"] Warning$
[Mon Dec 01 21:15:12 2008] [error] [client yy.yy.yy.yy] PHP Fatal error: Call to undefined function p() in /usr/share/psa-horde/lib/Horde.php on line 1228, referer: http://webmail.domain.co.uk/imp/compose.php?uniq=50c463mupqos
[Mon Dec 01 21:15:12 2008] [notice] child pid 21786 exit signal Segmentation fault (11)
*** glibc detected *** free(): invalid next size (normal): 0xbf9dfbb8 ***
[Mon Dec 01 21:28:04 2008] [notice] child pid 26370 exit signal Aborted (6)
Here's a memory exhaustion:
Code: Select all
[Mon Dec 01 19:37:44 2008] [error] [client xx.xx.xx.xx] PHP Fatal error: Allowed memory size of 50331648 bytes exhausted (tried to allocate 4500771 bytes) in /usr/share/psa-horde/imp/lib/MIME/Contents.php on line 173, referer: http://w
*** glibc detected *** free(): invalid next size (fast): 0xbfafafd8 ***
[Mon Dec 01 20:05:57 2008] [notice] child pid 28106 exit signal Aborted (6)
FATAL: emalloc(): Unable to allocate -1211328479 bytes
*** glibc detected *** corrupted double-linked list: 0xbfb0aff0 ***
[Mon Dec 01 20:08:38 2008] [notice] child pid 17817 exit signal Aborted (6)
Posted: Tue Dec 02, 2008 11:35 am
by scott
Wow, good work. Ive seen a handful of those glibc faults come up with horde in the past too. This was on CentOS 4 with really big message attachments (over 8MB or so), are you seeing the same thing? Or is this on general messages?
Posted: Tue Dec 02, 2008 4:09 pm
by faris
I'm guessing they are large attachments. The examples are all coming from two particular webmail users (domains). One is a printer/publisher, the other more general. Both use webmail A LOT.
This is under C4.
The VPS has 4Gigs or ram available to it though, currently using about 1.4Gb.
php is set as follows:
memory_limit = 48M (why? I know I did this. But why? I can't remember!)
post_max_size = 8M
upload_max_filesize = 2M
Incidentally, we have not been comparing some of mod_sec's settings, have we?
I just checked the config and I see I'm back to:
MODSEC_RESPONSEBODYLIMIT="2621440"
MODSEC_REQMEMLIMIT="131072"
I had previously increased both of those but they are back to default again. I guess after I did a yum update on ASL today?
Faris.
Faris.
Posted: Wed Dec 03, 2008 11:47 am
by scott
What I saw in my cores for that were PHP timing out after 300 seconds, and then I'd get the glibc heap error. It was also always on horde, grabbing a big email attachment (I think). It could also have been someone coming over a slow connection.
Posted: Wed Dec 03, 2008 12:23 pm
by faris
That makes sense.
I know having the full domain-blacklist rules will still cause the "normal" segfaults and rule processing failures (can we have an opt-out on them in the asl config please? All the other rulesets seem to have a similar option?) so I won't put them back.
But I'll put the malware blacklist back in full and see what happens.
Faris.
Posted: Thu Dec 04, 2008 5:54 pm
by faris
Well, bad news basically.
With all rules re-enabled, simply logging in to webmail (with no messages) generated a spate of segfaults.
Code: Select all
[Thu Dec 04 21:52:16 2008] [notice] child pid 13590 exit signal Segmentation fault (11)
[Thu Dec 04 21:52:16 2008] [notice] child pid 1782 exit signal Segmentation fault (11)
*** glibc detected *** free(): invalid pointer: 0xbfd35bb8 ***
[Thu Dec 04 21:52:17 2008] [notice] child pid 24047 exit signal Segmentation fault (11)
[Thu Dec 04 21:52:17 2008] [notice] child pid 3197 exit signal Aborted (6)
This is with all the mod_memcache stuff (and proxy stuff) disabled in httpd.conf
So I'm back to removing the domain-blacklist rules.
Faris.
Posted: Thu Dec 04, 2008 6:16 pm
by faris
This is interesting:
http://blog.modsecurity.org/2008/08/transformation.html
It talks about transformation cacheing a problem and that it should be disabled. It was mentioned in the mod_sec mailing list when someone asked about segfaults.
Does the ASL config enable this at all? Apparently it is disabled by default in 2.5.6 and later. I can't see any explicit enable in the config, but I thought I'd ask anyway.
Faris.