Server gets overloaded

[scott]

Check out the data from clamdtop too. It will show you exactly what clamd is working on at the time

[biggles]

I have re-enabled dazuko so I can study it when the next crash occurs.

[biggles]

Hmmmm, it may not be a race condition, it might be a pipelining issue with so many files backed up for scanning. Do you know if clamd was busy when this occured? And I believe you said you have a backup script that runs around the time of the first spike in processes on your system, was dazuko watching directories that were being backed up?

I have tried to run the backup and check the process with clamdtop, but I don't see any increased activity. I donät think it's the backup causing it. It's much to random in time for that...

[mikeshinn]

Do you have anything excluded for dazuko?

[biggles]

Yes,

Code: Select all

/var/spool/qscan/
/var/spamtmp
/var/spool/qscan/tmp/
/root/tmp
/var/tmp/clamd

[biggles]

I just had another clamd crash. I now describe it as a clamd crash, because I cought it in time and only clamd was hung. Unfortunatly I was unable to run clamdtop, it only waited at "Connecting to: /tmp/clamd.socket". Running top didn't show any clamd processess consuming much resources. ps showed quite a lot of clamd processess. Restarting clamd seem to fix the problem:

Code: Select all

/etc/init.d/clamd restart
Stopping Clam AntiVirus Daemon:                            [FAILED]
Starting Clam AntiVirus Daemon: Bytecode: Security mode set to "TrustSigned".
                                                           [  OK  ]

The last line from the /var/log/clamav/clamd.log (about to hours before the event "To many processess" was triggered)

Code: Select all

Tue Jun 21 03:01:23 2011 -> SelfCheck: Database modification detected. Forcing reload.
Tue Jun 21 03:01:23 2011 -> Stopping and restarting Clamuko.
Tue Jun 21 03:01:23 2011 -> Clamuko stopped.
Tue Jun 21 03:01:24 2011 -> Reading databases from /var/clamav
Tue Jun 21 03:01:39 2011 -> Database correctly reloaded (2545764 signatures)
Tue Jun 21 03:01:39 2011 -> Stopping and restarting Clamuko.
Tue Jun 21 03:01:39 2011 -> ERROR: Can't unregister with Dazuko
Tue Jun 21 03:01:39 2011 -> Clamuko stopped.

To me it looks like it's right after freshclam is telling the database to reload. The last freshclam started started at Tue Jun 21 03:01:17 2011.

[scott]

Ah ok heres an experiment to try next, lets turn off freshclam via cron updates with:

mv /etc/cron.hourly/freshclam /root/

and run it as a daemon instead with:
freshclam -d

This defaults to checking for updates every 2 hours. If you want to increase this you can go as high as 50 times a day by setting the Checks token in /etc/freshclam.conf

[biggles]

I'm all set up!

Let the testing begin!

[mikeshinn]

Yes,

Code:
/var/spool/qscan/
/var/spamtmp
/var/spool/qscan/tmp/
/root/tmp
/var/tmp/clamd

That does not look complete, what directories are you watching? For example, if you are watching your web hosts directories (and you are using Plesk) then you need to make sure you followed the instructions here:

https://www.atomicorp.com/wiki/index.php/Anti_virus

Which describes the need to exclude certain plesk directories such as these:

/var/www/vhosts/www.example.com/statistics/
/var/www/vhosts/www.example.com/conf/
/var/www/vhosts/www.example.com/pd/

Also make sure you aren't watching your entire filesystem. Theres no need to do that, just the areas where untrusted users can write (/home, /var/www, /tmp, etc.), otherwise you are just wasting cycles.

We recommend you only watch directories like:

/var/www/
/home
/var/tmp
/tmp

And definitely make sure you are excluding the plesk conf, statistics and pd directories (users cant touch these, and they are HUGE I/O bottlenecks on clamd when apache restarts).

[biggles]

I'm watching (dazuko-include):

Code: Select all

/home
/var/tmp
/usr/local/psa/tmp
/tmp

and have been doing so since dazuko was introduced. This issue started just a few weeks ago, so I'm guessing a clamd update or dazuko kmodule update changed something...

[mikeshinn]

Make sure there arent any Plesk vhost directories buried in /home, back in the day there was a symlink and sometimes thats where apache lived and /var/www was the symlink.

[mikeshinn]

/var/tmp
/usr/local/psa/tmp
/tmp

Also, with your temp dirs, check to see if you have any applications that scan with clamav and use them to temporarily copy the files. That can definitely create some interesting loops with the kernel module.

[biggles]

Make sure there arent any Plesk vhost directories buried in /home, back in the day there was a symlink and sometimes thats where apache lived and /var/www was the symlink.

Just my user home directory. And atomic. And one backup user, but that's one isn't used.

[biggles]

/var/tmp
/usr/local/psa/tmp
/tmp

Also, with your temp dirs, check to see if you have any applications that scan with clamav and use them to temporarily copy the files. That can definitely create some interesting loops with the kernel module.

The only thing that seems to be double scanning is spamassassin/qmail-scanner. I really haven't been able to get spamasassin to use another directory for scanning, even though you provided excellent instructions (http://atomicorp.com/forums/viewtopic.p ... sin+dazuko). WP/php is probably using /tmp for uploads, but that's how it's suppose to be, isn't it?

[biggles]

Another crash. This one I wasn't able to stop before logginfg in was inpossible so a restart was required. I'll have to remove dazuko scanning until a solution is found...

Last lines from clamd.log:

Code: Select all

Wed Jun 22 17:09:54 2011 -> SelfCheck: Database modification detected. Forcing reload.
Wed Jun 22 17:09:54 2011 -> Stopping and restarting Clamuko.
Wed Jun 22 17:09:54 2011 -> Clamuko stopped.
Wed Jun 22 17:09:54 2011 -> Reading databases from /var/clamav
Wed Jun 22 17:10:09 2011 -> Database correctly reloaded (2569880 signatures)
Wed Jun 22 17:10:09 2011 -> Stopping and restarting Clamuko.
Wed Jun 22 23:35:50 2011 -> +++ Started at Wed Jun 22 23:35:50 2011

Last lines from freshclam.log:

Code: Select all

Received signal: wake up
ClamAV update process started at Wed Jun 22 17:09:47 2011
main.cvd is up to date (version: 53, sigs: 846214, f-level: 53, builder: sven)
daily.cld is up to date (version: 13227, sigs: 129691, f-level: 60, builder: guitar)
Downloading safebrowsing-30292.cdiff [100%]
Downloading safebrowsing-30293.cdiff [100%]
safebrowsing.cld updated (version: 30293, sigs: 772817, f-level: 60, builder: google)
bytecode.cld is up to date (version: 143, sigs: 40, f-level: 60, builder: edwin)
Database updated (1748762 signatures) from db.se.clamav.net (IP: 192.121.13.5)
Clamd successfully notified about the update.
--------------------------------------
Received signal: wake up
ClamAV update process started at Wed Jun 22 19:10:14 2011
main.cvd is up to date (version: 53, sigs: 846214, f-level: 53, builder: sven)
Downloading daily-13228.cdiff [100%]
daily.cld updated (version: 13228, sigs: 130688, f-level: 60, builder: ccordes)
Downloading safebrowsing-30294.cdiff [100%]
Downloading safebrowsing-30295.cdiff [100%]
safebrowsing.cld updated (version: 30295, sigs: 773574, f-level: 60, builder: google)
bytecode.cld is up to date (version: 143, sigs: 40, f-level: 60, builder: edwin)
Database updated (1750516 signatures) from db.se.clamav.net (IP: 192.121.13.5)
Clamd successfully notified about the update.
--------------------------------------
Update process interrupted