Apache bus error with mod_security, httpd-debuginfo missing?

[mikeshinn]

No I dont recall what it was, sorry. I found the bug and it was already fixed upstream in the main Apache tree which we already used, and those customers I was working with just upgraded to our RPMs which already had the fix so I havent bothered to follow it. I've seen other folks run into the same bug in July, and they too just upgraded to our rpms so I would guess the bug is still there. The bug in APR, that much I remember off the top of my head. I'll see if I can find the backtraces we have at least.

[mikeshinn]

backtrace from 2.2.3:

Core was generated by `/usr/sbin/httpd'.

Program terminated with signal 11, Segmentation fault.

#0 0x5092ca4c in memcpy () from /lib/libc.so.6



Thread 1 (Thread 19940):

#0 0x5092ca4c in memcpy () from /lib/libc.so.6

No symbol table info available.

#1 0x50bf55e0 in apr_brigade_flatten () from /usr/lib/libaprutil-1.so.0

No symbol table info available.

#2 0x503919e3 in ?? () from /etc/httpd/modules/mod_security2.so

No symbol table info available.

#3 0x5039247c in ?? () from /etc/httpd/modules/mod_security2.so

No symbol table info available.

#4 0x139030f0 in ap_pass_brigade (next=0x7d0, bb=0x17fb6f98) at /usr/src/debug/httpd-2.2.3/server/util_filter.c:526

e = 0x17fb3b10

#5 0x138ea299 in end_output_stream (r=0x17f65590) at /usr/src/debug/httpd-2.2.3/server/protocol.c:1114

c = 0x17f17298

bb = 0x17fb6f98

b = <value optimized out>

#6 0x13907398 in ap_process_request (r=0x17f65590) at /usr/src/debug/httpd-2.2.3/modules/http/http_request.c:268

access_status = 0

#7 0x1390428f in ap_process_http_connection (c=0x17f17298) at /usr/src/debug/httpd-2.2.3/modules/http/http_core.c:184

r = 0x17f65590

csd = 0x0

#8 0x138ff92d in ap_run_process_connection (c=0x17f17298) at /usr/src/debug/httpd-2.2.3/server/connection.c:43

n = 2

rv = <value optimized out>

#9 0x138ffa2c in ap_process_connection (c=0x17f17298, csd=0x17f17100) at /usr/src/debug/httpd-2.2.3/server/connection.c:178

rc = 0

#10 0x1390bdd4 in child_main (child_num_arg=<value optimized out>) at /usr/src/debug/httpd-2.2.3/server/mpm/prefork/prefork.c:640

current_conn = 0x17f17298

csd = 0x17f17100

ptrans = 0x17f170c0

allocator = 0x17f15030

status = <value optimized out>

i = 2

lr = <value optimized out>

pollset = 0x17f151f8

sbh = 0x17f151f0

bucket_alloc = 0x17f63548

last_poll_idx = 1

#11 0x1390c0e1 in make_child (s=0x13931cf8, slot=32) at /usr/src/debug/httpd-2.2.3/server/mpm/prefork/prefork.c:736

pid = 0

#12 0x1390cae3 in ap_mpm_run (_pconf=0x1392fe50, plog=0x1395df08, s=0x13931cf8)

at /usr/src/debug/httpd-2.2.3/server/mpm/prefork/prefork.c:871

status = 11

pid = {pid = -1, in = 0x1395df08, out = 0x13929fe8, err = 0x2}

child_slot = <value optimized out>

exitwhy = 6

processed_status = <value optimized out>

index = <value optimized out>

remaining_children_to_start = 0

rv = <value optimized out>

#13 0x138e3157 in main (argc=328392400, argv=0x16a79090) at /usr/src/debug/httpd-2.2.3/server/main.c:717

c = 0 '\000'

configtestonly = 0

confname = 0x1390ef89 "conf/httpd.conf"

def_server_root = 0x1390ef99 "/etc/httpd"

temp_error_log = 0x0

error = <value optimized out>

process = 0x1392ded0

server_conf = <value optimized out>

pglobal = 0x1392de48

pconf = 0x1392fe50

plog = 0x1395df08

ptemp = 0x13961f18

pcommands = 0x13931e58

opt = 0x13931ef8

rv = 0

optarg = 0x13929e20 " \236\222\023 \236\222\023\001"

And heres the feedback from one customer after upgrading to 2.2.17:

"I have been running the apache 2.2.17 builds in the atomic-testing channel since you informed me of them and so far I have not had a single segmentation fault."

And that was back in February when they upgraded, and I just checked with them and they still havent had a segfault. So, short answer, upgrade Apache there be bugs in the older versions.

[ikkk]

Ok we re-installed http-2.2.17 from atomic-testing with the asl_antimalware disabled still, and everything was still 100% fine.

So this morning readded in that ruleset and managed to recreate the results

[Mon Aug 08 09:55:07 2011] [notice] Apache/2.2.17 (Unix) DAV/2 mod_ssl/2.2.17 OpenSSL/0.9.8e-fips-rhel5 Apache configured -- resuming normal operations
[Mon Aug 08 10:34:21 2011] [notice] child pid 24365 exit signal Bus error (7), possible coredump in /tmp/apache2-gdb-dump

Below is the backtrace, and it doesnt mentioned mod_security at all, so im guessing its something else but im not good at reading these things .

(I have altered the users username/password for security reasons in this)

# cat backtrace.core.24365.log
Core was generated by `/usr/sbin/httpd'.
Program terminated with signal 7, Bus error.
#0 0xb7be1697 in memset () from /lib/libc.so.6

Thread 1 (Thread 24365):
#0 0xb7be1697 in memset () from /lib/libc.so.6
No symbol table info available.
#1 0xb7eaf7ab in apr_password_validate () from /usr/lib/libaprutil-1.so.0
No symbol table info available.
#2 0xb7f21bda in check_password (r=0xb5b4ce10, user=0xb5b13c10 "USER", password=0xb5b13bff "PASS") at /builddir/build/BUILD/httpd-2.2.17/modules/aaa/mod_authn_file.c:107
conf = <value optimized out>
f = 0xb5baf4c8
l = "USER:$1$XXXXb$ygf7Bxs9jf6i2M6bnAp85.\000\000\000\333\000\000\000\001\000\000\000\020\016\000\000\023\000\000\000\062\060\061\061-08-08 10:33:33\000Q{\247\267\264\274\252\267\201\000\000\000\000\000\000\000\370\213\227\277\006s\237\267\000\000\000\000\201\000\000\000\002\000\000\000\240h\252\267\000\000\000\000\000\000\000\000è\277\240h\252\267p\216\227\277l5ɵ\000\000\000\000\225l\237\267@.\t\267\001\000\000\000\004\000\000\000\264,\b\267\200\203\252\267l5ɵH\214\227\277\305qÉ·\216\245÷\305qÉ·\364OÌ·\000\340\n\000\230\346\f\000 \214\227\277\021â½·\000\000\000\000D\214\227\277cl\275\267\000\000\000\000\344\355\035\267@aÌ·\000\300\273\265\364OÌ·hÙ®\265\230\346\f\000Ð\227\277\006\220\275\267\310\361зg\277\246\267g\277\246\267h\214\227\277и\317\267d\365\372\266\260\377\027\266\006\220\275\267\344\355\035\267`\365\372\266\000\000\000\000@aÌ·\b讵\370\327\f\000\342r"...
status = <value optimized out>
file_password = 0xb5baf530 "$1$XXXXXb$ygf7Bxs9jf6i2M6bnAp85."
#3 0xb7f25c68 in authenticate_basic_user (r=0xb5b4ce10) at /builddir/build/BUILD/httpd-2.2.17/modules/aaa/mod_auth_basic.c:230
provider = 0xb7f23e60
sent_user = 0xb5b13c10 "USER"
current_auth = <value optimized out>
res = <value optimized out>
auth_result = 3048290432
current_provider = 0x0
#4 0xb7f6638d in ap_run_check_user_id (r=0xb5b4ce10) at /builddir/build/BUILD/httpd-2.2.17/server/request.c:77
n = 1
rv = <value optimized out>
#5 0xb7f67767 in ap_process_request_internal (r=0xb5b4ce10) at /builddir/build/BUILD/httpd-2.2.17/server/request.c:230
file_req = 0
access_status = 0
#6 0xb7f7b78b in ap_process_request (r=0xb5b4ce10) at /builddir/build/BUILD/httpd-2.2.17/modules/http/http_request.c:280
access_status = 0
#7 0xb7f783ef in ap_process_http_connection (c=0xb5af1528) at /builddir/build/BUILD/httpd-2.2.17/modules/http/http_core.c:190
r = 0xb5b4ce10
csd = 0x0
#8 0xb7f7397d in ap_run_process_connection (c=0xb5af1528) at /builddir/build/BUILD/httpd-2.2.17/server/connection.c:43
n = 1
rv = <value optimized out>
#9 0xb7f73a7c in ap_process_connection (c=0xb5af1528, csd=0xb5af1390) at /builddir/build/BUILD/httpd-2.2.17/server/connection.c:190
rc = 0
#10 0xb7f804e6 in child_main (child_num_arg=<value optimized out>) at /builddir/build/BUILD/httpd-2.2.17/server/mpm/prefork/prefork.c:662
current_conn = 0xb5af1528
csd = 0xb5af1390
ptrans = 0xb5af1350
allocator = 0xb5aef2c0
status = <value optimized out>
i = 2
lr = <value optimized out>
pollset = 0xb5aef3e0
sbh = 0xb5aef3d8
bucket_alloc = 0xb5af56b8
last_poll_idx = 1
#11 0xb7f80811 in make_child (s=0xb8ca34f8, slot=11) at /builddir/build/BUILD/httpd-2.2.17/server/mpm/prefork/prefork.c:763
pid = 0
#12 0xb7f81213 in ap_mpm_run (_pconf=0xb8ca1600, plog=0xb8ccf6b8, s=0xb8ca34f8) at /builddir/build/BUILD/httpd-2.2.17/server/mpm/prefork/prefork.c:898
status = 0
pid = {pid = -1, in = 0xb7f94a28, out = 0xbf97af18, err = 0xb7f6b819}
child_slot = <value optimized out>
exitwhy = APR_PROC_EXIT
processed_status = <value optimized out>
index = <value optimized out>
remaining_children_to_start = 0
rv = <value optimized out>
#13 0xb7f567ce in main (argc=-1194723712, argv=0xbf97b074) at /builddir/build/BUILD/httpd-2.2.17/server/main.c:739
c = 0 '\000'
configtestonly = 0
confname = 0xb7f838e0 "conf/httpd.conf"
def_server_root = 0xb7f838f0 "/etc/httpd"
temp_error_log = 0x0
error = <value optimized out>
process = 0xb8c9f680
server_conf = <value optimized out>
pglobal = 0xb8c9f5f8
pconf = 0xb8ca1600
plog = 0xb8ccf6b8
ptemp = 0xb8cd36c8
pcommands = 0xb8ca3608
opt = 0xb8ca36a8
rv = 0
optarg = 0xb7a81316 "OPENSSL_ia32cap"

Any pointers of where to look as to what this is would be very appreciated, this is an openvz machine, and none of the failcounters are increasing when this is triggered

[breun]

Looks like another APR bug to me ("apr_password_validate () from /usr/lib/libaprutil-1.so.0"), but Mike might be able to tell you more.

[ikkk]

The weird thing is though like mentioned, with the asl_antimalware rules disable there is no issues.

[breun]

That doesn't mean the problem is with that ruleset. Like Mike wrote in this thread: "However, each time we ran a backtrace it turned out to be something else entirely that was the cause. Not the rules and not modsec."

[ikkk]

Totally agree, as i mentioned no mention of mod_security in that backtrace, interested to see what Mike thinks!

[mikeshinn]

In your case, thats an APR bug. Try upgrading to 2.2.17+ from the atomic-testing repo (I think Scott has pushed 2.2.19 now, but 2.2.17 is fine), if the problem goes away, then its confirmed that its a bug in apaches APR.

[breun]

I believe ikkk said he's already running 2.2.17 and got the core dump with that version.

[ikkk]

Mike - yes this is using the latest from atomic-testing this morning - doesnt seem to be any newer one there

[root@server /]# yum update httpd --enablerepo=atomic-testing
Loaded plugins: fastestmirror, posttransaction
Loading mirror speeds from cached hostfile
* atomic: www7.atomicorp.com
* atomic-testing: www6.atomicorp.com
* base: centos.mirror.transip.nl
* centosplus: mirror.bytemark.co.uk
* contrib: centos.mirror.transip.nl
* extras: mirror.nl.leaseweb.net
* updates: mirror.bytemark.co.uk
atomic | 1.9 kB 00:00
atomic-testing | 1.9 kB 00:00
base | 1.1 kB 00:00
centosplus | 1.9 kB 00:00
contrib | 951 B 00:00
extras | 2.1 kB 00:00
updates | 1.9 kB 00:00
Setting up Update Process
No Packages marked for Update
[root@server /]# rpm -qa | grep httpd
httpd-2.2.17-1.el5.art
httpd-tools-2.2.17-1.el5.art
httpd-devel-2.2.17-1.el5.art

[mikeshinn]

OK, I think Scott is still preparing to push 2.2.19 then. In the mean time, dont be shy about opening a bug report with redhat/centos on this too.

[ikkk]

Ill wait for 2.2.19 to give that a try first, Spoke to scott quickly and hes having some issues with itk on it, but im sure he will sort that soon - ill check regularly.

[scott]

We could cut ITK completely, is it really necessary now that mod_ruid2 is available?

[ikkk]

As you know im using mod_ruid2 - works very well!

Do have some issues due to mod_security and it - but they can generally be worked around as reported on another forum topic (issue is that the audit logs are owned by apache, and when a security rule is hit when under mod_ruid2 its owned by user and cant write to it on the standard setup)

Do also however see some issues with it not being able to access the DBM files msa/ip msa/global which i havent fully been able to recreate to look into in much detail, but most of these i find are triggered by a rule previously so once the rule/exception etc is fixed this doesnt seem to become a problem.

So we have no intentions ourselves of using mpm_itk while mod_ruid2 is as functional as it is with very little changes needed to the system.

[premierhosting]

The httpd-debuginfo... is this the way to get it?

Code: Select all

yum install httpd-debuginfo --enablerepo=atomic-testing

Cos' that ain't doing it for me. I do have httpd 2.2.17 installed off atomic-testing.

Been segfaulting and want to trace some core dumps. I'm pretty sure it's an obfuscation engine doing it, but...