Atomicorp

Hi Breun,

Whats the output of uname -a (not that its reliable thanks to ksplice!), sounds like you're a victim of these (probably related) issues :

http://forum.parallels.com/showthread.php?t=110074 (Kernel 85.3 introduced your bug)
http://forum.parallels.com/showthread.php?t=109969

Paul

Hm, those issues are about time being off and high load. We actually see neither of those things on this system, but thanks for trying to help.

You mentioned you only installed a couple of rpms, did anything chance with virtuzzo? As nothing change with modsecurity, and you suddenly have a change in apache behavior (and you installed/uninstalled apr), what else changed, if anything?

mikeshinn wrote:You mentioned you only installed a couple of rpms, did anything chance with virtuzzo?

We have no idea, as we don't have access to the Virtuozzo host node.

mikeshinn wrote:As nothing change with modsecurity, and you suddenly have a change in apache behavior (and you installed/uninstalled apr), what else changed, if anything?

That's what we've been trying to find out as well, so far we came up with nothing.

Keep in mind packages can change from the master side without you being able to detect it on the guest side.

The only changes that are likely to have happened are that the host node will have been upgraded too. Other than the kernel (we are running the latest), I can't see how anything will have changed in the Container.

Our nodes were just upgraded (from 5.5 to 5.6) and we've seen no problems other than the OOM during the yum updates on the Containers themselves that I mentioned before. We had previously updated the kernel via Ksplice and again not problems.

Breun - I've double-checked. There was nothing in the Container's /var/log/messages. Nothing in the container's logs either. But on the hardware node's dmesg (and /var/log/messages) the OOMs can be clearly seen.

Again I know this may be totally unrelated, but it might be worth asking your VPS provider if they would be willing to take a look at /var/log/messages on the host node just in case this is actually the cause of the problem. Having a process unkillably and invisibly hang due to OOM is obviously not what's supposed to happen, so there's a bug somewhere in Virtuozzo.

Faris.

I just personally logged into the box, and I can't reproduce this. So either I totally misunderstand the problem, or whatever was wrong with the host node or this VPS has been fixed by someone else. Apache starts/restarts/graceful/etc. with and without modsecurity with no issues, serves websites without issues, no load problems, no delays, etc. I won't post the details (they are in the case), but I can't find anything wrong with this system.

/etc/init.d/httpd restart
/etc/init.d/httpd graceful

Both work fine with modsecurity enabled. So you can rule out modsecurity. Whatever was wrong with this system had/has nothing to do with modsecurity, maybe something was broken in the host node, dunno, but its not modsecurity as apache is working just fine with it now and I didnt change anything (except to change the modsec config from .conf.disabled to .conf).

Or was restarting apache via the init scripts not the issue?

Yes, that was the issue. I just checked and indeed now it just works. I'm stumped...

Er, what was the issue? I wish I could say I fixed something, but I didn't change anything.

Stopping and starting Apache with mod_security enabled resulted in a hanging httpd process. That was the issue. But it works fine now. And no one changed anything, so that got me stumped.

Stopping and starting Apache with mod_security enabled resulted in a hanging httpd process. That was the issue. But it works fine now. And no one changed anything, so that got me stumped.

Yeah me too. Best guesses right now, something changed in the host node, or maybe this was just apache being slow for some reason and the cause was transient (DNS, i/o load, iscsi issues, etc. etc.)

This problem reappeared, on the same machine. I'll go and re-open the support case.

This problem reappeared, on the same machine. I'll go and re-open the support case.

As you recall, the last time we looked at this on this system of yours we could not reproduce this issue, it was completely transient, which means its not a problem with modsecurity or even apache, its something else with the system. I personally logged in, and nothing was wrong with the system at that time. So if this is happening again, with the same system, then its not modsecurity thats the cause, something else changed on your system or with the host node, etc. and thats the cause.

As I recall you were using Virtuozzo and Plesk in that case, which are *huge* variables, particularly the VPS which has enormous effects on the entire environment and absolutely can cause bizarre behavior like this. And in the previous case, the problem just went away on its own, we didnt and you didnt change anything. Which means its something else entirely that caused this for you, such as the OS, virtualization technology, control panel, host node, etc. So, have you also opened a case with your control panel, virtualization, host node and OS vendors?

In short, its highly likely the same thing is going to happen again: we wont be able to reproduce this on the same system because its the VPS, OS or control panel software itself thats causing this issue for you. So I highly recommend you open a case with your OS, control panel and virtualization vendors. In all likelihood, as with the previous case you opened, which was also not caused by either modsecurity, or apache, this is likely to the be the same thing here.

In that case it was some artifact of the OS or virtual environment, and its probably the same thing here, which is not something we can fix for you.

We were able to reproduce this issue for hours and hours on end before you went to take a look. And it has been a few hours now again where we've been able to reproduce this issue. I agree that's is probably something else locally to this server, since all of our other servers are doing fine with the same mod_security setup. Sadly I'm also pretty sure that everyone else (OS vendor, etc.) will point to mod_security, since everything is working fine when we disable it. Plus we're running ART's httpd because of segfaults we got with the CentOS package, which is also not helping when trying to get support from others. I'm happy to believe that mod_security is not the root cause, but as long as I don't know what is there's not much I can do but disable mod_security. Any ideas on what we can do to find the root cause here?

I agree that's is probably something else locally to this server, since all of our other servers are doing fine with the same mod_security setup.

So lets eliminate variables, whats different between those systems and this one system you have that isnt behaving correctly?