Atomicorp

Hi,

I'm hoping someone can point me in the right direction here as I'm a bit confused and at a dead end!

My Plesk server hosts around 100 domains and is secured with ASL.

Everything works perfectly fine 95% of the time, when all of a sudden the memory usage of one of the apache threads grows massive and uses GBs of swap.

Once it uses that amount, the thrashing occurs, spiking the load and bringing the server to a crawl.
That thread does eventually die if it is left alone (i.e. if I'm not at a terminal to notice it and restart apache on my own).

I've made my httpd.conf server config very conservative to test this and I think I've ruled out over stretching the server's capabilities (Pentium(R) Dual-Core CPU E6500 @ 2.93GHz, 4GB Ram) and all I can think of now is a script on the server has a memory leak.


My question(s)

• Does that seem a correct guess at this stage?
• How can I go about identifying either which vhost is causing the spike or (even better) which script is?
• Is there anything else I should be looking at here?

Thanks in advance!

Ive absolutely seen mod_rewrite do that on a system. Either if you have a huge number of rules, or if you end up with a loop.

Quite probable, alot of the sites use that - to varying extents.

It woul make sense that just one rewrite rule, enacted inrequently, would cause this intermittant issue.

My problem is how to narrow down the search to at least a specific vhost?

I dont know any shortcuts there unfortunately, you'll have to look through them all.

That's what I was afraid of!

Is there no way to know what each of the apache threads are up to?

e.g. As soon as I notice a spike on the load, run a command to profile the process?

Would listing the open files of the process help on this occasion? Hoping I see a path to an .htaccess file?

theres mod_top, but I dont know how the license works on that these days.

Great, thanks - got it installed.

I may now also use something like an online sitemap generator to follow links on the most recent websites - working in reverse - to try to reproduce the spike.


That's not something I'd like to immediately sit down and do (better things to be doing right now) - but if anyone else has any tips or ideas, please let me know!


Thanks

you could also try to track it down with pmap, strace and valgrind.

Thanks.

So far, it's not happened again (the day before it happened 6 or 7 times!).

If I find the cause, I'll get back with my solution!

a very stupid and time exhausting way with no guarantee (since other might surf the causing site parallel to you surfing another) is to surf each site you host and use almost all features the site brings. one after each other. maybe one of them will peak your apache. if you host sites with visitors from one country than one night session would be needed for that. otherwise this idea is pretty useless.

have you monitored your memory in some way than you could try to compare the time with access logs to pinpoint it.
because if you have 6 or 7 peaks than there should be one host that stands out from the rest.

EDIT: A lot of CMS admin areas are very memory exhausting. Like Wordpress for example which is a real nightmare.
Maybe someone edited a lot on the day you had 6-7 peaks.

Yes, I agree - that's why I thought of a sitemap generator - to do a basic crawl of all pages to try to provoke it.
Of course, it would only be front-end - and not go deeper into the admin.

I could look at the logs, yes - but the general logs (such as /var/log/messages) don't give up any information.

I'd have to go through each vhost again to check them indivdually.


The peaks are exhaustive - going 2GB+ into swap for just one thread.
I'm sure there's some error or other configuration issue to blame rather than a CMS.


Next time it happens, I'm now equipped with tools to pinpoint it.
I'd expect if it was a customer doing something, they'd have been in touch already saying "my website breaks when I...".



A rogue mod_rewrite rule does still seem to fit the bill here!


Thanks!

I think I've FINALLY found the culprit!

Just for completeness, I thought I'd share the solution.


I was looking through my client list on Plesk and I noticed one of my client's disk usage was extremely high (5GB instead of the usual few hundred MB).

I looked further and it was their error logs causing the bulk - the most recent at 4GB!


Looking into those, I saw a line repeating over and over again: The IP address of the requesting computer? It resolved to an MSN crawl bot!


This is part of a Joomla distribution - v1.5.14 - which, on searching, seemed to be causing problems for other users too.


The core problem is creating PDFs of articles on the Joomla site.


I upgraded their site to the latest version of 1.5 - 1.5.22 and disabled PDF generation.


Time will tell if this was the culprit - but I'm sure it was.

great detective work there, if this is the culprit we might want to add this to the FAQ

I'm sure it is.

I tried to view the log file with VIM and it choked on the size of it.

I viewed the last 50000 lines and it was over a 30 second period with this error.

I can now understand why it was intermittent and at strange times - users may never have clicked the PDF icon, but a crawl bot will go for whatever it finds.

Glad to hear you got it solved!

On the mod_top install, anyone else having trouble verifying that it's "working"?

I've gone through the steps and am running mod_top, but even when accessing tons of php pages on the system I'm not seeing anything in my mod_top console. Ideas? Probably should break this off to another thread.