httpd dead but subsys locked - only websrvmng starts Apache

General Discussion of atomic repo and development projects.

Ask for help here with anything else not covered by other forums.
breun
Long Time Forum Regular
Long Time Forum Regular
Posts: 2813
Joined: Sat Aug 20, 2005 9:30 am
Location: The Netherlands

httpd dead but subsys locked - only websrvmng starts Apache

Unread post by breun »

Apache suddenly stopped working on a fully updated CentOS 5.6 32-bit Virtuozzo VPS with Plesk 8.6.0 and ASL 2.2.11. Trying to start Apache starts one httpd process using nearly 100% CPU. Quering 'service httpd status' afterwards shows 'httpd dead but subsys locked' and Apache isn't serving pages. No amount of killing httpd processes, deleting /var/run/httpd and /var/lock/subsys/httpd and trying to restart Apache will get it back up and /var/log/httpd/error_log isn't showing anything obvious.

The only way I've found to get Apache to serve pages again is to manually kill the /usr/sbin/httpd process and run '/usr/local/psa/admin/bin/websrvmng -a -v'. I have no idea what magic Plesk's websrvmng is applying that I'm not, but after that Apache is serving pages again. The output of 'ps aux | grep apache' now shows Apache processes like '/usr/sbin/httpd -k graceful', while they were plain '/usr/sbin/httpd' before.

I tried downgrading apr (update installed yesterday), but that didn't make a difference.

Does anyone have any idea what is going on here or how I can find out for myself?
Lemonbit Internet Dedicated Server Management
faris
Long Time Forum Regular
Long Time Forum Regular
Posts: 2321
Joined: Thu Dec 09, 2004 11:19 am

Re: httpd dead but subsys locked - only websrvmng starts Apa

Unread post by faris »

I happened to update some Centos 4 and 5 systems yesterday, including APR.

No issues so far, but in my usual pre-upgrade research regarding potential Centos 5 upgrade issues I found a forum where a number of Virtuozzo customers were complaining of very high loads and apache taking ages to serve pages.

Once person suggested it might be the iscsi initiator (which is newly updated along with mkinitrd) and therefore disableing iscsi and iscsid which load pointlessly might be an option.

I also had issues with OOM during the Centos 5 upgrade process, causing yum to get killed initially, then to just unkillably and sort of invisibly hang (there was no appropriate process listed by ps to kill). Rebooting, then doing the yum update again, then yum-complete-transactions (which had nothing to do but cleaned up the warning caused by the hung yum update) resolved all problems.

OK, so none of this sounds like your particular problem, but I wonder if there's a small possibility that it might be an OOM issue that websrvmngr is somehow getting around by restarting apache in a different way?

I'm also crossing my fingers that this APR update, which has a rewritten matching engine, might solve the mod_sec segfault issues we've been having in Centos 4. So far no segfaults, even immediately after the reboot, which was something that used to happen almost repeatably before.

I'm not confident enough to re-enable the big domain blacklists etc yet though. I'll monitor it for a while first. Load is partcularly low at the moment as it is a weekend.
--------------------------------
<advert>
If you want to rent a UK-based VPS that comes with friendly advice and support from a fellow ART fan, please get in touch.
</advert>
faris
Long Time Forum Regular
Long Time Forum Regular
Posts: 2321
Joined: Thu Dec 09, 2004 11:19 am

Re: httpd dead but subsys locked - only websrvmng starts Apa

Unread post by faris »

p.s. for some reason the only place the OOM issues were visible were in dmesg - nothing in the Virtuozzo logs. I may have been looking in the wrong places though - it was late at night and I'm coming down with a cold so my brain is not 100%.
--------------------------------
<advert>
If you want to rent a UK-based VPS that comes with friendly advice and support from a fellow ART fan, please get in touch.
</advert>
breun
Long Time Forum Regular
Long Time Forum Regular
Posts: 2813
Joined: Sat Aug 20, 2005 9:30 am
Location: The Netherlands

Re: httpd dead but subsys locked - only websrvmng starts Apa

Unread post by breun »

We have discovered that after disabling mod_security Apache does stop and start normally. Why Plesk's websrvmng is able to start Apache with mod_security enabled and Apache's own init script isn't is a mystery to us, but we will open a ticket with ASL support.
Lemonbit Internet Dedicated Server Management
faris
Long Time Forum Regular
Long Time Forum Regular
Posts: 2321
Joined: Thu Dec 09, 2004 11:19 am

Re: httpd dead but subsys locked - only websrvmng starts Apa

Unread post by faris »

Interesting! Definitely check for oom conditions IMHO.
--------------------------------
<advert>
If you want to rent a UK-based VPS that comes with friendly advice and support from a fellow ART fan, please get in touch.
</advert>
breun
Long Time Forum Regular
Long Time Forum Regular
Posts: 2813
Joined: Sat Aug 20, 2005 9:30 am
Location: The Netherlands

Re: httpd dead but subsys locked - only websrvmng starts Apa

Unread post by breun »

dmesg doesn't return anything and there are no mentions of memory problems in /var/log/messages either. We don't have access to the Virtuozzo host in this case.
Lemonbit Internet Dedicated Server Management
User avatar
mikeshinn
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin
Posts: 4149
Joined: Thu Feb 07, 2008 7:49 pm
Location: Chantilly, VA

Re: httpd dead but subsys locked - only websrvmng starts Apa

Unread post by mikeshinn »

So the regular init script will start apache, but not the plesk websrvmng? That sounds like a bug in websrvmng, what does Parallels have to say?
breun
Long Time Forum Regular
Long Time Forum Regular
Posts: 2813
Joined: Sat Aug 20, 2005 9:30 am
Location: The Netherlands

Re: httpd dead but subsys locked - only websrvmng starts Apa

Unread post by breun »

mikeshinn wrote:So the regular init script will start apache, but not the plesk websrvmng?
No, it's the other way around.
Lemonbit Internet Dedicated Server Management
User avatar
mikeshinn
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin
Posts: 4149
Joined: Thu Feb 07, 2008 7:49 pm
Location: Chantilly, VA

Re: httpd dead but subsys locked - only websrvmng starts Apa

Unread post by mikeshinn »

So what is the output of this command:

/etc/init.d/httpd restart

And are there any errors in apaches error log?

Also, graceful means something has and/or is trying to start apache with the graceful option, which means apache will wait until all requests are finished before restarting, and if something is stuck, etc. apache will just wait.
breun
Long Time Forum Regular
Long Time Forum Regular
Posts: 2813
Joined: Sat Aug 20, 2005 9:30 am
Location: The Netherlands

Re: httpd dead but subsys locked - only websrvmng starts Apa

Unread post by breun »

You're asking the same questions ASL support is asking. :)

It seemingly restarts Apache ok, but the result is the problem I described above: one httpd process eating a lot of CPU and not serving pages.
Last edited by breun on Mon May 16, 2011 1:46 pm, edited 1 time in total.
Lemonbit Internet Dedicated Server Management
User avatar
mikeshinn
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin
Posts: 4149
Joined: Thu Feb 07, 2008 7:49 pm
Location: Chantilly, VA

Re: httpd dead but subsys locked - only websrvmng starts Apa

Unread post by mikeshinn »

I'm also crossing my fingers that this APR update, which has a rewritten matching engine, might solve the mod_sec segfault issues we've been having in Centos 4. So far no segfaults, even immediately after the reboot, which was something that used to happen almost repeatably before.
There was absolutely a bug in APR in Centos 4 that caused segfaults. In C5 there may still be, it was only fixed upstream in 2.2.17 of Apache and may not have been backported in C5. The up to date apache and apr are in the atomic repo that includes the apr fixes.
User avatar
mikeshinn
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin
Posts: 4149
Joined: Thu Feb 07, 2008 7:49 pm
Location: Chantilly, VA

Re: httpd dead but subsys locked - only websrvmng starts Apa

Unread post by mikeshinn »

It seemingly restarts Apache ok, but the result is the problem I described above: one httpd process eating a lot of CPU and not serving pages.
Can you post a little more detail here? Like the output of the start, and perhaps a trace of the stuck apache process?

And it sounds like if you have a hung apache process that something else is going on. Nothing has changed in modsecurity, and I've never seen this happen so I dont think that modsecurity is the cause. I looked at your case, and you have gettimeday loop which isnt in modsecurity, so if thats where the stuck apache thread or parent is then its definitely not in modsec. Something is waiting on your system and that code isnt in modsec.

So, the big question is, was this system ever working correctly? And if it was, what changed? If it wasnt ever working, do you have a similar system you can examine as well?
breun
Long Time Forum Regular
Long Time Forum Regular
Posts: 2813
Joined: Sat Aug 20, 2005 9:30 am
Location: The Netherlands

Re: httpd dead but subsys locked - only websrvmng starts Apa

Unread post by breun »

mikeshinn wrote:Can you post a little more detail here? Like the output of the start, and perhaps a trace of the stuck apache process?
This is what httpd is doing in a loop according to strace (of course, the specific numbers vary):
read(103, "", 12828) = 0
gettimeofday({1305548222, 894721}, NULL) = 0
gettimeofday({1305548222, 894783}, NULL) = 0
gettimeofday({1305548222, 894845}, NULL) = 0
gettimeofday({1305548222, 894909}, NULL) = 0
poll([{fd=103, events=POLLIN}], 1, 3000) = 1 ([{fd=103, revents=POLLHUP}])
mikeshinn wrote:And it sounds like if you have a hung apache process that something else is going on. Nothing has changed in modsecurity, and I've never seen this happen so I dont think that modsecurity is the cause. I looked at your case, and you have gettimeday loop which isnt in modsecurity, so if thats where the stuck apache thread or parent is then its definitely not in modsec. Something is waiting on your system and that code isnt in modsec.
Maybe it's not the root cause, but disabling mod_security does solve the problem and allows us to normally stop and start Apache through its init script.
mikeshinn wrote:So, the big question is, was this system ever working correctly?
Yes, it was fine until last Sunday.
mikeshinn wrote:And if it was, what changed?
The latest updates were installed through yum the day before, which were filesystem, apr and python-numeric. I thought the apr update might be the culprit, but I downgraded that package and that didn't solve the problem.
mikeshinn wrote:If it wasnt ever working, do you have a similar system you can examine as well?
It was working before, but we don't have access to a similar system.
Lemonbit Internet Dedicated Server Management
User avatar
mikeshinn
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin
Posts: 4149
Joined: Thu Feb 07, 2008 7:49 pm
Location: Chantilly, VA

Re: httpd dead but subsys locked - only websrvmng starts Apa

Unread post by mikeshinn »

read(103, "", 12828) = 0
gettimeofday({1305548222, 894721}, NULL) = 0
gettimeofday({1305548222, 894783}, NULL) = 0
gettimeofday({1305548222, 894845}, NULL) = 0
gettimeofday({1305548222, 894909}, NULL) = 0
poll([{fd=103, events=POLLIN}], 1, 3000) = 1 ([{fd=103, revents=POLLHUP}])
So that call is not modsecurity, thats apache waiting for something. If you like I can log into the system and see whats going on. Is this a virtual machine like a VPS btw?
breun
Long Time Forum Regular
Long Time Forum Regular
Posts: 2813
Joined: Sat Aug 20, 2005 9:30 am
Location: The Netherlands

Re: httpd dead but subsys locked - only websrvmng starts Apa

Unread post by breun »

mikeshinn wrote:Is this a virtual machine like a VPS btw?
Yes, it's a Virtuozzo VPS. See post 1, line 1. :)
Lemonbit Internet Dedicated Server Management
Post Reply