temporarily disable or reschedule ossec rootcheck

[ghazlewood]

I have a server with a failing drive which seems to be causing problems when the ossec rootcheck process kicks in. Load goes up and services start becoming inaccessible after about an hour.

I put this down to some of the checks that rootcheck is running and under normal circumstances would be quite happy for it to be running but in this case I'd like to temporarily disable or reschedule the check to run overnight.

Is it possible to disable the rootcheck scan, if so how? I don't want to tinker with ossec directly without an idea of whether ASL will override the config.

Can the rootcheck be scheduled to happen at a particular time? It always seems to kick in at midday at the moment.

I have the drive scheduled for replacement on Sunday morning to solve the real problem but was wondering what can be done in the meantime.

Many thanks

George

[ghazlewood]

Drive has checked out ok, and RAID is all running correctly according to DC engineer.

Still getting high load and unresponsive services directly related to rootcheck, am going to open a ticket I think as this needs proper investigation.

[mikeshinn]

That sounds like the I/O bus may just be overloaded (rootcheck is doing reads). What kind of RAID is this? (RAID 5, RAID 1, RAID 10) And what kind of I/O does the RAID normally experience?

Also, what happens when you disable rootcheck? Please make sure you have edited this file:

/var/asl/data/templates/template-ossec-server.conf

Go to the <rootcheck> section and change it in its entirety to this:

<rootcheck>
<disabled>yes</disabled>
</rootcheck>

Then run:

asl -s -f

Keep in mind though that if you are seeing I/O issues with rootcheck, thats telling you that something else is happening with the system. rootcheck is just doing reads, nothing fancy. So if the system can't handle those reads, I'd be worried that the I/O is already overloaded, and rootcheck is just the last straw. For example, if the RAID is slow (maybe its a software raid, or one of those "hardware" raids that really uses the systems CPU and not any onboard hardware).

Can you tell us a little more about the type of RAID, its configuration, and how its computing either the xor or how its doing the mirror copies? (In dedicated processors, or does it use the systems processor? Most RAIDs use the later.)

[ghazlewood]

That sounds like the I/O bus may just be overloaded (rootcheck is doing reads). What kind of RAID is this? (RAID 5, RAID 1, RAID 10) And what kind of I/O does the RAID normally experience?

Thanks for the detailed reply, I'll not clog up the forum with too much unnecessary detail - I've added lspci details to the ticket but this is a machine running RAID 1 with a MegaRAID Fusion card which I believe is a SAS card with an LSI controller chip. I'm not sure if it is hardware or software based but this isn't a new machine, it was provisioned in 2009 and has been running ASL that entire time.

Also, what happens when you disable rootcheck? Please make sure you have edited this file:

/var/asl/data/templates/template-ossec-server.conf

Go to the <rootcheck> section and change it in its entirety to this:

<rootcheck>
<disabled>yes</disabled>
</rootcheck>

Then run:

asl -s -f

Have applied that change now, thanks for explaining. Will keep an eye on it but would much rather be running rootcheck obviously.

Keep in mind though that if you are seeing I/O issues with rootcheck, thats telling you that something else is happening with the system. rootcheck is just doing reads, nothing fancy. So if the system can't handle those reads, I'd be worried that the I/O is already overloaded, and rootcheck is just the last straw. For example, if the RAID is slow (maybe its a software raid, or one of those "hardware" raids that really uses the systems CPU and not any onboard hardware).

Can you tell us a little more about the type of RAID, its configuration, and how its computing either the xor or how its doing the mirror copies? (In dedicated processors, or does it use the systems processor? Most RAIDs use the later.)

I *think* it is hardware RAID but I'd have to check with the hardware guys to be sure.

To be honest I'm not sure that this hasn't been a problem for a while but it's only recently that it's started affecting service and I've looked into it properly and discovered that it was rootcheck causing problems. I agree that it isn't a problem with rootcheck per se but just indicates a deeper issue. Thanks for helping to troubleshoot.

Cheers

George

[mikeshinn]

I'm wondering if this might be a bus issue, looking at the kernel data you provided it looks like you are running a 32bit kernel, so the system is going to have a hard low limit on I/O and memory compared to a 64bit system. You may even see that if you have 64bit CPUs, they may run sub-optimally when they switch to 32bit mode, and possibly even your RAID controller may be constrained. For example, many modern RAID controls will have buses that can utilize 64bit, and a 32bit kernel and drivers wont be able to use the bus completely (and sometimes not even correctly) and the drivers may not work with it well either (32bit might have been an afterthought if its a newer driver). The list goes on, but you might be running into a limit in the OS itself because its 32bit.

Can't say for sure thats what is causing this, but I can say that the 32bit system is definitely going to make your system slower, and will severely limit the amount of memory an application and the kernel can use on the system.

Did you intend to run a 32bit kernel and drivers on the system?

[ghazlewood]

Just wanted to check, I've disabled rootcheck using the process described above several times now. Is it not permanent?