Apache bus error with mod_security, httpd-debuginfo missing?

[ikkk]

Its built in already if your using that version, just follow the instructions at https://www.atomicorp.com/wiki/index.php/Apache

[premierhosting]

ikkk, yes, following those instructions says to install httpd-debuginfo

[ikkk]

Ignore that bit it works fine following the rest of that guide.

[premierhosting]

OK - then I should be g2g. Of course installing the latest version from atomic-testing may have fixed the core dump issue anyway. No core dumps since enabling.

[jas8522]

I've been encountering this bus error for months now - I think since early this year. It only occurs as described with http authentication and is always 'resolved' by a restart of apache. The backtrace always reveals memset() as the final stack entry, yet the servers rarely have more than 50% of their memory in use.

For quite some time I thought this was a problem with file descriptors hitting their limit and simply being reported as a memory allocation error incorrectly (or that this was using a memory mapped file). Is it possible this is still the case? Keep in mind that I did recompile Apache with considerably higher file descriptors as the limit (65536) and the problem remained.

A few days ago I updated Apache to 2.2.21 from the atomic-testing repo on both servers where this problem was occurring since it was suggested earlier in the thread that the update might resolve the problem. As of this afternoon the segfaults began to re-appear while running 2.2.21. The top of the stack trace looks pretty much identical to those reported earlier in this thread with the indication that the problem is in apr.

#0 0xb7be1697 in memset () from /lib/libc.so.6
#1 0xb7eaf7ab in apr_password_validate () from /usr/lib/libaprutil-1.so.0

What version of apr is 2.2.21 in atomic-testing built against? Does changing my system installed version of apr have any effect on this bug? (My assumption thus far has been that the version of apr installed only affects applications built using apr on that server).

Since there are already stack traces here for this issue, is there anything more I can provide to help resolve this problem? Someone mentioned there being an existing report for this in the apache bugtracker - is this confirmed, and if so, where is it?

Thanks for the help

Jordan

[ikkk]

jas, are you running asl itself, if you (temporarily) removed the asl_anti_malware rules, do you still get the issue, this seems to fix it for me on the serves i was seeing he issue on (openvz based setups)

[jas8522]

ikkk: yeah, I saw your earlier posts in the thread indicating that disabling that rule works around the issue. I'll certainly do that if the actual problem cannot be resolved, however I'd like to see whatever is actually causing the problem fixed. That's why I'm trying to find out what the status of getting to the bottom of the actual problem is.

From what I read here, many had believed this issue to be repaired as of httpd 2.2.17 thanks to an updated version of apr. My objective for posting was to say that it's not actually fixed as of 2.2.21 and hopefully find out who (apache bugtracker?) needs to get a bug report, and what should be included in it, in order to get this fixed.

[ikkk]

I havent tried 2.2.21 yet, when i went to try it mod_ssl wasnt working so was unable to test as my guinea pig relies on ssl.

I will try to switch to this in the next few days to see if we still get the same issues with this version as well.

Are you running in an openvz enviroment or such, or just a pure server? (trying to see if any other similarities)

[jas8522]

mod_ssl seems to be working fine with 2.2.21 now, otherwise I'd be getting quite a few complaints right about now!

I'm running on Virtuozzo, so yes OpenVZ also.

[ikkk]

Interesting wonder if everyone else that had issues is in an OpenVZ/Virtuozza enviroment.

Are you seeing any increase in any of the user beancounters when the issues arises (im pretty sure i wasnt)

[breun]

The one system where I saw this issue was indeed a Virtuozzo server.

[jas8522]

Interesting wonder if everyone else that had issues is in an OpenVZ/Virtuozza enviroment.

Are you seeing any increase in any of the user beancounters when the issues arises (im pretty sure i wasnt)

No, the last time this occurred while I was monitoring user beancounters, there was no increase at the same time. That was also my first thought - I had hit memory (or other) limits on the container - but that doesn't appear to be the case. The main box where this occurs has 8GB dedicated memory (with privvmpages set to 10GB since the container makes up the entire server) and memory usage normally hovers around 4-4.5GB, rarely exceeding 5GB.

[jas8522]

Just started going nuts with core dumps again:

Code: Select all

[New Thread 32510]
[Thread debugging using libthread_db enabled]
Core was generated by `/usr/sbin/httpd'.
Program terminated with signal 7, Bus error.
#0  0xb7bb8697 in memset () from /lib/libc.so.6

Thread 1 (Thread 0xb783ca60 (LWP 32510)):
#0  0xb7bb8697 in memset () from /lib/libc.so.6
No symbol table info available.
#1  0xb7e877ab in apr_password_validate (passwd=0xae7171b6 "PASSWORD_REMOVED", hash=0xae70e590 "$1$EO5cFpTj$n4oms0bQ1b15QnYiTMxSA/")
    at crypto/apr_md5.c:710
        buffer = {keysched = '\000' <repeats 127 times>, sb0 = '\000' <repeats 32767 times>,
          sb1 = '\000' <repeats 32767 times>,
          sb2 = '\000' <repeats 27084 times>"\201, \f\273\267", '\000' <repeats 32 times>"\370, \324ɷx\001\000\000\367\005տp\324ɷ\000\000\000\000p\321ɷ\000\000\000\000@\321ɷ\006\000\273\267@\321ɷ\201\f\273\267", '\000' <repeats 16 times>, "i\001\000\000p\321ɷ\003\000\000\000\000\000\000\000\360\324ɷ\350\003\000\000C\006տp\324ɷ\240\000Ƿp\321ɷ\000\000\000\000@\321ɷ\322\003\000\000@\321ɷ\017\000\000\000\001", '\000' <repeats 15 times>"\331, \003\000\000p\321ɷ\003\000\000\000\000\000\000\000\001\000\000\000\364\277ɷ\350\034g\256 \366l\256d\006տ\334(\273\267 \366l\256\310\033g\256$\001\000\000\000\000\000\000\000\000\000\000G\000\000\000\201\f\273\267\020\001m\256@\321ɷ\000\000\000\000(\001\000\000\000\000\000\000\370\000ү\030\366l\256\364\277ɷ@\321ɷ\000\003\000\000x\006տ\267\204\365\266\310\033g\256\"\343Ʒ\246\064\273\267\344h\000\267\322\003\000\000\205\364\340\372\340\003i\256"...,
          sb3 = "\000\003<\004d\375\000d\370\000\003<\004d\376\000d\377\000\003<\004d\000\001d\377\000\003<\004d\001\001d\377\000\003<\004d\002\001d\377\000\003<\004d\003\001d\377\000\003<\004d\004\001d\005\001\003<\004d\006\001d\005\001\003<\004d\a\001d\005\001\003<\004d\b\001d\005\001\003<\004d\t\001d\005\001\003<\004d\n\001d\005\001\003<\004d\v\001d\005\001\003<\004d\f\001d\r\001\003<\004d\016\001d\r\001\003<\004d\017\001d\r\001\003<\004d\020\001d\r\001\003<\004d\021\001d\r\001\003<\004d\022\001d\r\001\003<\004d\023\001d\r\001\003<\004d\024\001d\r\001\003<\004d\025\001d\026\001\003<\004d\027\001d\026\001"...,
          crypt_3_buf = "\000\000\000\000\030\234տ \342̷X\234", current_salt = "տ", current_saltbits = 0,
          direction = -1076519992, initialized = -1211307727}
        sample = "\350sη\320qq\256\234+\313\267\061\377̷\254\001\000\000\000\000\000\000W\274տ\350sη\320qq\256\000\377̷؛տ\350\061\315\267\320qq\256\320qq\256\000\377̷\350sη\220\345p\256\b\206t\256\370\233տ\355\376̷ȅt\256\320qq\256\000\377̷rC\363\267\311\376̷(\372\366\267\b\234տ=L\363\267\320qq\256\224\337", <incomplete sequence \357\267>
        crypt_pw = <value optimized out>

No increase in any of the user_beancounters metrics.

[jas8522]

I renamed 10_asl_antimalware.conf to 10_asl_antimalware.disabled and it certainly delayed this problem for a bit longer than it would take before, but it still returns.

Is there any chance that this is a file descriptor limit problem? It could explain why more rules cause the problem to trigger faster. That said, I've recompiled every srpm that Parallel's KB article tells me to do in order to up the FD limit and the problem always returns. Perhaps mod_security also needs to be recompiled for a higher descriptor limit? In other words, maybe there's some additional package that needs to be recompiled to up the limit than what Parallels says to do in their article.

[faris]

I don't think it is an FD limit issue.

Also VZ here, and have also had the same problems.

If you reduce the size of the malware blacklist and domain blacklist you'll find the problem will also go away (or at least not happen very often). [ note -- you have to copy your reduced files to /var/asl/rules/modsec or some similar place as they get copied from there into /etc/httpd/modsecurity.d every so often, overwriting any modifications you might make]

Interestingly, since ASL 3.x seems to have improved the situation in some mysterious way. I've not seen a single one since I updated (but I still reduce the malware/domain blacklists).

It appears to therefore be the amount of data in memory that triggers the "bug" - the more there is, the more likely it is to trigger.

In the past I had been looking for some way to use a dnsbl instead of those files, thus removing them from memory, but unfortunately it isn't possible since mod_sec basically has to lookup every single domain in both lists against an url in the request, and this can't be done using adnsbl.

I thought the bug had been narrowed down to APR, anyway? I can't update it on my systems (all Centos 4)