httpd dead but subsys locked - only websrvmng starts Apache
Re: httpd dead but subsys locked - only websrvmng starts Apa
Hi Breun,
Whats the output of uname -a (not that its reliable thanks to ksplice!), sounds like you're a victim of these (probably related) issues :
http://forum.parallels.com/showthread.php?t=110074 (Kernel 85.3 introduced your bug)
http://forum.parallels.com/showthread.php?t=109969
Paul
Whats the output of uname -a (not that its reliable thanks to ksplice!), sounds like you're a victim of these (probably related) issues :
http://forum.parallels.com/showthread.php?t=110074 (Kernel 85.3 introduced your bug)
http://forum.parallels.com/showthread.php?t=109969
Paul
-
- Long Time Forum Regular
- Posts: 2813
- Joined: Sat Aug 20, 2005 9:30 am
- Location: The Netherlands
Re: httpd dead but subsys locked - only websrvmng starts Apa
Code: Select all
# uname -vipor
2.6.9-023stab052.4-enterprise #1 SMP Tue May 11 19:40:55 MSD 2010 i686 i386 GNU/Linux
Lemonbit Internet Dedicated Server Management
- mikeshinn
- Atomicorp Staff - Site Admin
- Posts: 4152
- Joined: Thu Feb 07, 2008 7:49 pm
- Location: Chantilly, VA
Re: httpd dead but subsys locked - only websrvmng starts Apa
You mentioned you only installed a couple of rpms, did anything chance with virtuzzo? As nothing change with modsecurity, and you suddenly have a change in apache behavior (and you installed/uninstalled apr), what else changed, if anything?
Michael Shinn
Atomicorp - Security For Everyone
Atomicorp - Security For Everyone
-
- Long Time Forum Regular
- Posts: 2813
- Joined: Sat Aug 20, 2005 9:30 am
- Location: The Netherlands
Re: httpd dead but subsys locked - only websrvmng starts Apa
We have no idea, as we don't have access to the Virtuozzo host node.mikeshinn wrote:You mentioned you only installed a couple of rpms, did anything chance with virtuzzo?
That's what we've been trying to find out as well, so far we came up with nothing.mikeshinn wrote:As nothing change with modsecurity, and you suddenly have a change in apache behavior (and you installed/uninstalled apr), what else changed, if anything?
Lemonbit Internet Dedicated Server Management
-
- Atomicorp Staff - Site Admin
- Posts: 8355
- Joined: Wed Dec 31, 1969 8:00 pm
- Location: earth
- Contact:
Re: httpd dead but subsys locked - only websrvmng starts Apa
Keep in mind packages can change from the master side without you being able to detect it on the guest side.
Re: httpd dead but subsys locked - only websrvmng starts Apa
The only changes that are likely to have happened are that the host node will have been upgraded too. Other than the kernel (we are running the latest), I can't see how anything will have changed in the Container.
Our nodes were just upgraded (from 5.5 to 5.6) and we've seen no problems other than the OOM during the yum updates on the Containers themselves that I mentioned before. We had previously updated the kernel via Ksplice and again not problems.
Breun - I've double-checked. There was nothing in the Container's /var/log/messages. Nothing in the container's logs either. But on the hardware node's dmesg (and /var/log/messages) the OOMs can be clearly seen.
Again I know this may be totally unrelated, but it might be worth asking your VPS provider if they would be willing to take a look at /var/log/messages on the host node just in case this is actually the cause of the problem. Having a process unkillably and invisibly hang due to OOM is obviously not what's supposed to happen, so there's a bug somewhere in Virtuozzo.
Faris.
Our nodes were just upgraded (from 5.5 to 5.6) and we've seen no problems other than the OOM during the yum updates on the Containers themselves that I mentioned before. We had previously updated the kernel via Ksplice and again not problems.
Breun - I've double-checked. There was nothing in the Container's /var/log/messages. Nothing in the container's logs either. But on the hardware node's dmesg (and /var/log/messages) the OOMs can be clearly seen.
Again I know this may be totally unrelated, but it might be worth asking your VPS provider if they would be willing to take a look at /var/log/messages on the host node just in case this is actually the cause of the problem. Having a process unkillably and invisibly hang due to OOM is obviously not what's supposed to happen, so there's a bug somewhere in Virtuozzo.
Faris.
--------------------------------
<advert>
If you want to rent a UK-based VPS that comes with friendly advice and support from a fellow ART fan, please get in touch.
</advert>
<advert>
If you want to rent a UK-based VPS that comes with friendly advice and support from a fellow ART fan, please get in touch.
</advert>
- mikeshinn
- Atomicorp Staff - Site Admin
- Posts: 4152
- Joined: Thu Feb 07, 2008 7:49 pm
- Location: Chantilly, VA
Re: httpd dead but subsys locked - only websrvmng starts Apa
I just personally logged into the box, and I can't reproduce this. So either I totally misunderstand the problem, or whatever was wrong with the host node or this VPS has been fixed by someone else. Apache starts/restarts/graceful/etc. with and without modsecurity with no issues, serves websites without issues, no load problems, no delays, etc. I won't post the details (they are in the case), but I can't find anything wrong with this system.
/etc/init.d/httpd restart
/etc/init.d/httpd graceful
Both work fine with modsecurity enabled. So you can rule out modsecurity. Whatever was wrong with this system had/has nothing to do with modsecurity, maybe something was broken in the host node, dunno, but its not modsecurity as apache is working just fine with it now and I didnt change anything (except to change the modsec config from .conf.disabled to .conf).
Or was restarting apache via the init scripts not the issue?
/etc/init.d/httpd restart
/etc/init.d/httpd graceful
Both work fine with modsecurity enabled. So you can rule out modsecurity. Whatever was wrong with this system had/has nothing to do with modsecurity, maybe something was broken in the host node, dunno, but its not modsecurity as apache is working just fine with it now and I didnt change anything (except to change the modsec config from .conf.disabled to .conf).
Or was restarting apache via the init scripts not the issue?
Michael Shinn
Atomicorp - Security For Everyone
Atomicorp - Security For Everyone
-
- Long Time Forum Regular
- Posts: 2813
- Joined: Sat Aug 20, 2005 9:30 am
- Location: The Netherlands
Re: httpd dead but subsys locked - only websrvmng starts Apa
Yes, that was the issue. I just checked and indeed now it just works. I'm stumped...
Lemonbit Internet Dedicated Server Management
- mikeshinn
- Atomicorp Staff - Site Admin
- Posts: 4152
- Joined: Thu Feb 07, 2008 7:49 pm
- Location: Chantilly, VA
Re: httpd dead but subsys locked - only websrvmng starts Apa
Er, what was the issue? I wish I could say I fixed something, but I didn't change anything.
Michael Shinn
Atomicorp - Security For Everyone
Atomicorp - Security For Everyone
-
- Long Time Forum Regular
- Posts: 2813
- Joined: Sat Aug 20, 2005 9:30 am
- Location: The Netherlands
Re: httpd dead but subsys locked - only websrvmng starts Apa
Stopping and starting Apache with mod_security enabled resulted in a hanging httpd process. That was the issue. But it works fine now. And no one changed anything, so that got me stumped.
Lemonbit Internet Dedicated Server Management
- mikeshinn
- Atomicorp Staff - Site Admin
- Posts: 4152
- Joined: Thu Feb 07, 2008 7:49 pm
- Location: Chantilly, VA
Re: httpd dead but subsys locked - only websrvmng starts Apa
Yeah me too. Best guesses right now, something changed in the host node, or maybe this was just apache being slow for some reason and the cause was transient (DNS, i/o load, iscsi issues, etc. etc.)Stopping and starting Apache with mod_security enabled resulted in a hanging httpd process. That was the issue. But it works fine now. And no one changed anything, so that got me stumped.
Michael Shinn
Atomicorp - Security For Everyone
Atomicorp - Security For Everyone
-
- Long Time Forum Regular
- Posts: 2813
- Joined: Sat Aug 20, 2005 9:30 am
- Location: The Netherlands
Re: httpd dead but subsys locked - only websrvmng starts Apa
This problem reappeared, on the same machine. I'll go and re-open the support case.
Lemonbit Internet Dedicated Server Management
- mikeshinn
- Atomicorp Staff - Site Admin
- Posts: 4152
- Joined: Thu Feb 07, 2008 7:49 pm
- Location: Chantilly, VA
Re: httpd dead but subsys locked - only websrvmng starts Apa
As you recall, the last time we looked at this on this system of yours we could not reproduce this issue, it was completely transient, which means its not a problem with modsecurity or even apache, its something else with the system. I personally logged in, and nothing was wrong with the system at that time. So if this is happening again, with the same system, then its not modsecurity thats the cause, something else changed on your system or with the host node, etc. and thats the cause.This problem reappeared, on the same machine. I'll go and re-open the support case.
As I recall you were using Virtuozzo and Plesk in that case, which are *huge* variables, particularly the VPS which has enormous effects on the entire environment and absolutely can cause bizarre behavior like this. And in the previous case, the problem just went away on its own, we didnt and you didnt change anything. Which means its something else entirely that caused this for you, such as the OS, virtualization technology, control panel, host node, etc. So, have you also opened a case with your control panel, virtualization, host node and OS vendors?
In short, its highly likely the same thing is going to happen again: we wont be able to reproduce this on the same system because its the VPS, OS or control panel software itself thats causing this issue for you. So I highly recommend you open a case with your OS, control panel and virtualization vendors. In all likelihood, as with the previous case you opened, which was also not caused by either modsecurity, or apache, this is likely to the be the same thing here.
In that case it was some artifact of the OS or virtual environment, and its probably the same thing here, which is not something we can fix for you.
Michael Shinn
Atomicorp - Security For Everyone
Atomicorp - Security For Everyone
-
- Long Time Forum Regular
- Posts: 2813
- Joined: Sat Aug 20, 2005 9:30 am
- Location: The Netherlands
Re: httpd dead but subsys locked - only websrvmng starts Apa
We were able to reproduce this issue for hours and hours on end before you went to take a look. And it has been a few hours now again where we've been able to reproduce this issue. I agree that's is probably something else locally to this server, since all of our other servers are doing fine with the same mod_security setup. Sadly I'm also pretty sure that everyone else (OS vendor, etc.) will point to mod_security, since everything is working fine when we disable it. Plus we're running ART's httpd because of segfaults we got with the CentOS package, which is also not helping when trying to get support from others. I'm happy to believe that mod_security is not the root cause, but as long as I don't know what is there's not much I can do but disable mod_security. Any ideas on what we can do to find the root cause here?
Lemonbit Internet Dedicated Server Management
- mikeshinn
- Atomicorp Staff - Site Admin
- Posts: 4152
- Joined: Thu Feb 07, 2008 7:49 pm
- Location: Chantilly, VA
Re: httpd dead but subsys locked - only websrvmng starts Apa
So lets eliminate variables, whats different between those systems and this one system you have that isnt behaving correctly?I agree that's is probably something else locally to this server, since all of our other servers are doing fine with the same mod_security setup.
Michael Shinn
Atomicorp - Security For Everyone
Atomicorp - Security For Everyone