httpd dead but subsys locked - only websrvmng starts Apache

General Discussion of atomic repo and development projects.

Ask for help here with anything else not covered by other forums.
paulie
Forum User
Forum User
Posts: 76
Joined: Tue Apr 20, 2010 2:49 am

Re: httpd dead but subsys locked - only websrvmng starts Apa

Unread post by paulie »

Hi Breun,

Whats the output of uname -a (not that its reliable thanks to ksplice!), sounds like you're a victim of these (probably related) issues :

http://forum.parallels.com/showthread.php?t=110074 (Kernel 85.3 introduced your bug)
http://forum.parallels.com/showthread.php?t=109969

Paul
breun
Long Time Forum Regular
Long Time Forum Regular
Posts: 2813
Joined: Sat Aug 20, 2005 9:30 am
Location: The Netherlands

Re: httpd dead but subsys locked - only websrvmng starts Apa

Unread post by breun »

Code: Select all

# uname -vipor
2.6.9-023stab052.4-enterprise #1 SMP Tue May 11 19:40:55 MSD 2010 i686 i386 GNU/Linux
Hm, those issues are about time being off and high load. We actually see neither of those things on this system, but thanks for trying to help.
Lemonbit Internet Dedicated Server Management
User avatar
mikeshinn
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin
Posts: 4149
Joined: Thu Feb 07, 2008 7:49 pm
Location: Chantilly, VA

Re: httpd dead but subsys locked - only websrvmng starts Apa

Unread post by mikeshinn »

You mentioned you only installed a couple of rpms, did anything chance with virtuzzo? As nothing change with modsecurity, and you suddenly have a change in apache behavior (and you installed/uninstalled apr), what else changed, if anything?
breun
Long Time Forum Regular
Long Time Forum Regular
Posts: 2813
Joined: Sat Aug 20, 2005 9:30 am
Location: The Netherlands

Re: httpd dead but subsys locked - only websrvmng starts Apa

Unread post by breun »

mikeshinn wrote:You mentioned you only installed a couple of rpms, did anything chance with virtuzzo?
We have no idea, as we don't have access to the Virtuozzo host node.
mikeshinn wrote:As nothing change with modsecurity, and you suddenly have a change in apache behavior (and you installed/uninstalled apr), what else changed, if anything?
That's what we've been trying to find out as well, so far we came up with nothing.
Lemonbit Internet Dedicated Server Management
scott
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin
Posts: 8355
Joined: Wed Dec 31, 1969 8:00 pm
Location: earth
Contact:

Re: httpd dead but subsys locked - only websrvmng starts Apa

Unread post by scott »

Keep in mind packages can change from the master side without you being able to detect it on the guest side.
faris
Long Time Forum Regular
Long Time Forum Regular
Posts: 2321
Joined: Thu Dec 09, 2004 11:19 am

Re: httpd dead but subsys locked - only websrvmng starts Apa

Unread post by faris »

The only changes that are likely to have happened are that the host node will have been upgraded too. Other than the kernel (we are running the latest), I can't see how anything will have changed in the Container.

Our nodes were just upgraded (from 5.5 to 5.6) and we've seen no problems other than the OOM during the yum updates on the Containers themselves that I mentioned before. We had previously updated the kernel via Ksplice and again not problems.

Breun - I've double-checked. There was nothing in the Container's /var/log/messages. Nothing in the container's logs either. But on the hardware node's dmesg (and /var/log/messages) the OOMs can be clearly seen.

Again I know this may be totally unrelated, but it might be worth asking your VPS provider if they would be willing to take a look at /var/log/messages on the host node just in case this is actually the cause of the problem. Having a process unkillably and invisibly hang due to OOM is obviously not what's supposed to happen, so there's a bug somewhere in Virtuozzo.

Faris.
--------------------------------
<advert>
If you want to rent a UK-based VPS that comes with friendly advice and support from a fellow ART fan, please get in touch.
</advert>
User avatar
mikeshinn
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin
Posts: 4149
Joined: Thu Feb 07, 2008 7:49 pm
Location: Chantilly, VA

Re: httpd dead but subsys locked - only websrvmng starts Apa

Unread post by mikeshinn »

I just personally logged into the box, and I can't reproduce this. So either I totally misunderstand the problem, or whatever was wrong with the host node or this VPS has been fixed by someone else. Apache starts/restarts/graceful/etc. with and without modsecurity with no issues, serves websites without issues, no load problems, no delays, etc. I won't post the details (they are in the case), but I can't find anything wrong with this system.

/etc/init.d/httpd restart
/etc/init.d/httpd graceful

Both work fine with modsecurity enabled. So you can rule out modsecurity. Whatever was wrong with this system had/has nothing to do with modsecurity, maybe something was broken in the host node, dunno, but its not modsecurity as apache is working just fine with it now and I didnt change anything (except to change the modsec config from .conf.disabled to .conf).

Or was restarting apache via the init scripts not the issue?
breun
Long Time Forum Regular
Long Time Forum Regular
Posts: 2813
Joined: Sat Aug 20, 2005 9:30 am
Location: The Netherlands

Re: httpd dead but subsys locked - only websrvmng starts Apa

Unread post by breun »

Yes, that was the issue. I just checked and indeed now it just works. I'm stumped...
Lemonbit Internet Dedicated Server Management
User avatar
mikeshinn
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin
Posts: 4149
Joined: Thu Feb 07, 2008 7:49 pm
Location: Chantilly, VA

Re: httpd dead but subsys locked - only websrvmng starts Apa

Unread post by mikeshinn »

Er, what was the issue? I wish I could say I fixed something, but I didn't change anything. :-)
breun
Long Time Forum Regular
Long Time Forum Regular
Posts: 2813
Joined: Sat Aug 20, 2005 9:30 am
Location: The Netherlands

Re: httpd dead but subsys locked - only websrvmng starts Apa

Unread post by breun »

Stopping and starting Apache with mod_security enabled resulted in a hanging httpd process. That was the issue. But it works fine now. And no one changed anything, so that got me stumped.
Lemonbit Internet Dedicated Server Management
User avatar
mikeshinn
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin
Posts: 4149
Joined: Thu Feb 07, 2008 7:49 pm
Location: Chantilly, VA

Re: httpd dead but subsys locked - only websrvmng starts Apa

Unread post by mikeshinn »

Stopping and starting Apache with mod_security enabled resulted in a hanging httpd process. That was the issue. But it works fine now. And no one changed anything, so that got me stumped.
Yeah me too. Best guesses right now, something changed in the host node, or maybe this was just apache being slow for some reason and the cause was transient (DNS, i/o load, iscsi issues, etc. etc.)
breun
Long Time Forum Regular
Long Time Forum Regular
Posts: 2813
Joined: Sat Aug 20, 2005 9:30 am
Location: The Netherlands

Re: httpd dead but subsys locked - only websrvmng starts Apa

Unread post by breun »

This problem reappeared, on the same machine. I'll go and re-open the support case.
Lemonbit Internet Dedicated Server Management
User avatar
mikeshinn
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin
Posts: 4149
Joined: Thu Feb 07, 2008 7:49 pm
Location: Chantilly, VA

Re: httpd dead but subsys locked - only websrvmng starts Apa

Unread post by mikeshinn »

This problem reappeared, on the same machine. I'll go and re-open the support case.
As you recall, the last time we looked at this on this system of yours we could not reproduce this issue, it was completely transient, which means its not a problem with modsecurity or even apache, its something else with the system. I personally logged in, and nothing was wrong with the system at that time. So if this is happening again, with the same system, then its not modsecurity thats the cause, something else changed on your system or with the host node, etc. and thats the cause.

As I recall you were using Virtuozzo and Plesk in that case, which are *huge* variables, particularly the VPS which has enormous effects on the entire environment and absolutely can cause bizarre behavior like this. And in the previous case, the problem just went away on its own, we didnt and you didnt change anything. Which means its something else entirely that caused this for you, such as the OS, virtualization technology, control panel, host node, etc. So, have you also opened a case with your control panel, virtualization, host node and OS vendors?

In short, its highly likely the same thing is going to happen again: we wont be able to reproduce this on the same system because its the VPS, OS or control panel software itself thats causing this issue for you. So I highly recommend you open a case with your OS, control panel and virtualization vendors. In all likelihood, as with the previous case you opened, which was also not caused by either modsecurity, or apache, this is likely to the be the same thing here.

In that case it was some artifact of the OS or virtual environment, and its probably the same thing here, which is not something we can fix for you. :-(
breun
Long Time Forum Regular
Long Time Forum Regular
Posts: 2813
Joined: Sat Aug 20, 2005 9:30 am
Location: The Netherlands

Re: httpd dead but subsys locked - only websrvmng starts Apa

Unread post by breun »

We were able to reproduce this issue for hours and hours on end before you went to take a look. And it has been a few hours now again where we've been able to reproduce this issue. I agree that's is probably something else locally to this server, since all of our other servers are doing fine with the same mod_security setup. Sadly I'm also pretty sure that everyone else (OS vendor, etc.) will point to mod_security, since everything is working fine when we disable it. Plus we're running ART's httpd because of segfaults we got with the CentOS package, which is also not helping when trying to get support from others. I'm happy to believe that mod_security is not the root cause, but as long as I don't know what is there's not much I can do but disable mod_security. Any ideas on what we can do to find the root cause here?
Lemonbit Internet Dedicated Server Management
User avatar
mikeshinn
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin
Posts: 4149
Joined: Thu Feb 07, 2008 7:49 pm
Location: Chantilly, VA

Re: httpd dead but subsys locked - only websrvmng starts Apa

Unread post by mikeshinn »

I agree that's is probably something else locally to this server, since all of our other servers are doing fine with the same mod_security setup.
So lets eliminate variables, whats different between those systems and this one system you have that isnt behaving correctly?
Post Reply