psmon reports ossec-hids fails to restart (one solution)

[faris]

I was bombarded by emails from psmon telling me ossec-hids was failing to restart this morning. It started on one server and soon after all but one was doing the same thing. The one that didn't go nuts was one where I had updated ossec-hids to .37 a couple of days ago.

Looking at /var/ossec/logs/ossec.log on most of the servers, ossec-hids was complaining about something in exclusion_rules.xml but on one it was moaning about something to do with some dovecote rules.

One solution when you get this sort of thing is to run aum -u followed by asl -s -f and if that sorts it, great. But not in my case.

The other day I learned from Mike that exclusion_rules.xml is generated from /etc/asl/rules which in turn contains modifications you might make to rules via the ASL GUI.

On most of the systems that were going nuts, the solution for me was to update ossec-hids to the latest version then remove the customised rules starting with a G (for global, I think?) from /etc/asl/rules then doing an asl -s -f.

Removing those custom rules may not be necessary for you, so just try updating ossec-hids first and then asl -s -f to see if it sorts it out.

Indeed, on one of the servers having this problem there were no custom rules and all that was required was to update ossec-hids and then asl -s -f. ossec-hids was then happy as a radio active clam.

I don't know what triggered this problem overnight, but if it hits you, don't panic (I've done all your panicking for you) and just try to trouble-shoot it slowly: update ossec-hids > aum -u > asl -s -f > if fixed then great, otherwise > look at removing customised rules in /etc/asl/rules and asl -s -f afterwards.

service ossec-hids status will tell you how ossec-hids is doing, and /var/ossec/logs/ossec.log will give you a good idea about what's going on. Ignore errors about geoIP stuff.

I hope this helps someone at some stage! I know similar things have been discussed in other threads but I thought I'd post my particular experience here.

[BruceLee]

thanks faris. read your post too late but just wanted to confirm your experience.
yum update > aum -u > asl -s -f and two restarts of ossec-hids did it for me. I have no custom rules.

[faris]

And if it happens again (as it did to all my systems after the daily ASL cron ran at 3am) and you see stuff about duplicate decoders, especially apache-errorlog-ip, in /var/ossec/logs/ossec.log like this:

Code: Select all

2014/01/23 07:02:33 ossec-testrule: INFO: Reading decoder file etc/decoder.xml.
2014/01/23 07:02:33 ossec-testrule: INFO: Reading decoder file etc/decoders.d/01-asl-decoder.xml.
2014/01/23 07:02:33 ossec-testrule: INFO: Reading decoder file etc/decoders.d/10-asl-drupal-decoder.xml.
2014/01/23 07:02:33 ossec-testrule: INFO: Reading decoder file etc/decoders.d/50-asl-aix-ipsec-decoder.xml.
2014/01/23 07:02:33 ossec-testrule: INFO: Reading decoder file etc/decoders.d/50-asl-apache-decoder.xml.
2014/01/23 07:02:33 ossec-analysisd(2102): ERROR: Duplicated decoder with prematch: 'apache-errorlog-ip'.
2014/01/23 07:02:33 ossec-analysisd(2105): ERROR: Error loading decoder options.
2014/01/23 07:02:33 ossec-analysisd(2106): ERROR: Error adding decoder plugin.
2014/01/23 07:02:33 ossec-testrule(1202): ERROR: Configuration error at 'etc/decoders.d/50-asl-apache-decoder.xml'. Exiting.

then try the following to resolve it:

Code: Select all

aum -uf

I think there's been a change in where some decoders are located, and unfortunately a configuration directive in ossec-server.conf pointing to the old location wasn't removed, resulting in decoders being loaded twice and ossec having a bit of a moan.

aum -uf sorts this out by re-downloading all the rules and regenerating ossec-server.conf with the directive removed and also a few other adjustments as well, I think.

(I also upgraded asl to the version released yesterday, just to be on the safe side, and of course ran "asl -s -f" afterwards as well)

[DarkF@der]

I have the same experience.

[faris]

Essentially lit appears that

etc/decoder.xml (the original file ?)

and the contents of

/etc/decoders.d/ (new location for multiple files containing what was in decoders.xml ?)

were both being loaded in ossec-server.conf

I did try commenting out with <!-- --> the decoder.xml line in ossec-servers.conf but that wasn't enough -- it then complained about some rules or something like that. So there was more than just that one line that needed to be changed. aum -uf did the trick though.

Now I need to get back to sleep. I don't function well on 4 hours sleep.

We do need a way of preventing this from happening again though please Mike/Scott.

[scott]

Make sure you are using ASL 3.2.16. Decoder.xml was retired and older versions of ASL will not handle that condition gracefully.

[faris]

3.2.16 came out a day or so ago, right? I will have had 3.2.15 or whatever version it was before .16

I don't feel comfortable allowing software to auto-update when I'm not there to see if it goes wrong.
And unless it resolves a very dangerous security hole, I also prefer to leave things a week or so before updating most software packages, just in case any show-stoppers appear.

So I have ASL set to only update the rules automatically.
This has never caused problems before. In fact I have it set this way to prevent problems like this.

There's a very neat mechanism in ASL that rolls back mod_sec rules if the new set won't load for some reason. I really like that feature.

So I guess we need something similar for ossec in ASL 4.x please?

But more importantly, please please please can you not issue any rules via the auto-updater that might stop ossec from starting until that happens? If I was using a version of ASL or ossec that was months old, and maybe had been warned via the forum/mailing list/whatever that I needed to update or bad things would happen if I didn't update soon, then I wouldn't mind so much.

[hostingg]

you might want to look at the docs then, because they clearly say:

Important Notice: Some rule and signature updates may not work without ASL updates, so if you set this to "rules only" be sure to regularly check your system for any software updates for ASL to be fully protected and to ensure compatibility.

http://www.atomicorp.com/wiki/index.php ... PDATE_TYPE

[mikeshinn]

Rule updates have always been like this. And there definitely are times where they absolutely depend on ASL updates, theres no way to decouple them. If you just do rule updates, you may occasionally have problems where the rules simply do not work with the version of software you have installed.

To that end, yes we will be adding in lock outs for all rule types, so if you havent updated and a rule update requires a specific version of something the update will not install. This happens for modsec, but not for the other types.

[faris]

If you just do rule updates, you may occasionally have problems where the rules simply do not work with the version of software you have installed.

Yes, absolutely, but in this particular case the situation was avoidable. It would have been useful to issue a warning. If you could do that for future changes that might be along the same lines then it would be great.

[biggles]

+1

[Imaging]

Agreed, would be helpful.

+1

[hostingg]

there is a warning in the documentation:

Important Notice: Some rule and signature updates may not work without ASL updates, so if you set this to "rules only" be sure to regularly check your system for any software updates for ASL to be fully protected and to ensure compatibility.

[faris]

there is a warning in the documentation:

Um, well, yes, but even if there was no specific warning, nobody should expect things to work 100% if they don't keep their installations up to date. That would be just be unreasonable.

But that's not really the issue here. Those of us who were affected by the particular problem are just trying to minimise the possibility of it happening again. We feel it might have been avoided with a bit more communication. But Mike has indicated that he's taken that on board and equally importantly ASL 4.x will have a mechanism to that will help protect against problems like this. So, as usual, ASL just gets better and better.

(Mike - was this change in ASL/rule layout anything to do with the problem you spent hours helping me solve the other week? It would be quite hilarious if it was -- "OSSEC II: Revenge of the HIDS").