Ossec wont restart after update

[hostingguy]

Upgraded ossec-hids today, and now the servers fail to start ossec

# grep 'Jan 06' /var/log/yum.log

Jan 06 09:31:14 Updated: asl-php-common-5.4.23-21.el5.art.x86_64
Jan 06 09:31:37 Updated: nss-3.15.3-4.el5_10.x86_64
Jan 06 09:31:39 Updated: asl-php-pdo-5.4.23-21.el5.art.x86_64
Jan 06 09:31:39 Updated: asl-php-mysqlnd-5.4.23-21.el5.art.x86_64
Jan 06 09:31:40 Updated: nss-tools-3.15.3-4.el5_10.x86_64
Jan 06 09:31:40 Updated: asl-php-process-5.4.23-21.el5.art.x86_64
Jan 06 09:31:41 Updated: asl-php-gd-5.4.23-21.el5.art.x86_64
Jan 06 09:31:42 Updated: mod_security-2.7.7-17.el5.art.x86_64
Jan 06 09:31:42 Updated: asl-php-cli-5.4.23-21.el5.art.x86_64
Jan 06 09:31:44 Updated: asl-php-5.4.23-21.el5.art.x86_64
Jan 06 09:31:46 Updated: nss-devel-3.15.3-4.el5_10.x86_64
Jan 06 09:31:51 Updated: tzdata-java-2013i-1.el5.x86_64
Jan 06 09:32:02 Updated: tzdata-2013i-1.el5.x86_64
Jan 06 09:32:03 Updated: lynis-1.3.8-7.el5.art.noarch
Jan 06 09:32:05 Updated: nss-3.15.3-4.el5_10.i386
Jan 06 15:07:34 Updated: ossec-hids-2.7.1-36.el5.art.x86_64
Jan 06 15:07:51 Updated: ossec-hids-server-2.7.1-36.el5.art.x86_64
Jan 06 15:07:51 Installed: ossec-hids-mysql-2.7.1-36.el5.art.x86_64
Jan 06 15:07:55 Updated: 1:asl-3.2.15-32.el5.art.x86_64
Jan 06 15:07:56 Updated: 1:asl-waf-module-3.2.15-32.el5.art.x86_64
Jan 06 15:08:00 Updated: 1:asl-web-3.2.15-32.el5.art.x86_64
Jan 06 15:29:45 Updated: ossec-hids-2.7.1-37.el5.art.x86_64
Jan 06 15:29:57 Updated: ossec-hids-server-2.7.1-37.el5.art.x86_64
Jan 06 15:29:57 Updated: ossec-hids-mysql-2.7.1-37.el5.art.x86_64

Ossec is just trying to constantly restart itself and fails.

# service ossec-hids status
ossec-monitord not running...
ossec-logcollector not running...
ossec-remoted not running...
ossec-syscheckd not running...
ossec-analysisd not running...
ossec-maild not running...
ossec-execd not running...
ossec-dbd not running...

tail -n20 /var/ossec/logs/ossec.log

2014/01/06 15:55:55 ossec-analysisd: Duplicate rule ID:20101
2014/01/06 15:55:55 ossec-testrule(1220): ERROR: Error loading the rules: 'exclusion_rules.xml'.
2014/01/06 15:57:05 ossec-testrule: INFO: Reading decoder file etc/decoder.xml.
2014/01/06 15:57:05 ossec-testrule: INFO: Reading decoder file etc/decoders.d/01-asl-decoder.xml.
2014/01/06 15:57:05 ossec-testrule: INFO: Reading decoder file etc/decoders.d/10-asl-drupal-decoder.xml.
2014/01/06 15:57:05 ossec-testrule: INFO: Reading decoder file etc/decoders.d/50-asl-waf-decoder.xml.
2014/01/06 15:57:05 ossec-testrule: INFO: Reading decoder file etc/decoders.d/75-asl-deltaadmin-decoder.xml.
2014/01/06 15:57:05 ossec-testrule: INFO: Reading decoder file etc/decoders.d/75-asl-exim-decoder.xml.
2014/01/06 15:57:06 ossec-analysisd: Duplicate rule ID:20101
2014/01/06 15:57:06 ossec-testrule(1220): ERROR: Error loading the rules: 'exclusion_rules.xml'.
2014/01/06 15:58:16 ossec-testrule: INFO: Reading decoder file etc/decoder.xml.
2014/01/06 15:58:16 ossec-testrule: INFO: Reading decoder file etc/decoders.d/01-asl-decoder.xml.
2014/01/06 15:58:16 ossec-testrule: INFO: Reading decoder file etc/decoders.d/10-asl-drupal-decoder.xml.
2014/01/06 15:58:16 ossec-testrule: INFO: Reading decoder file etc/decoders.d/50-asl-waf-decoder.xml.
2014/01/06 15:58:16 ossec-testrule: INFO: Reading decoder file etc/decoders.d/75-asl-deltaadmin-decoder.xml.
2014/01/06 15:58:16 ossec-testrule: INFO: Reading decoder file etc/decoders.d/75-asl-exim-decoder.xml.
2014/01/06 15:58:17 ossec-analysisd: Duplicate rule ID:20101
2014/01/06 15:58:17 ossec-testrule(1220): ERROR: Error loading the rules: 'exclusion_rules.xml'.

Downgrading doesnt resolve the issue either.

=============================================================================================================================================================================================================================================
Package Arch Version Repository Size
=============================================================================================================================================================================================================================================
Downgrading:
ossec-hids x86_64 2.7.1-36.el5.art asl-3.0 37 k
ossec-hids-mysql x86_64 2.7.1-36.el5.art asl-3.0 78 k
ossec-hids-server x86_64 2.7.1-36.el5.art asl-3.0 1.8 M

Running aum -uf doesnt help, nor does asl -f -s
If I remove or empty the file exclusion_rules.xml it fails to start due to a file format error. Once I do asl -f -s it reconstructs the original file and gets the same original error.

Any suggestions on how to fix?

[skiper43]

Seems to be similar issue here, just started this evening. Email every minute with:

Command executed: /sbin/service ossec-hids restart
Exit value: 1
Signal number: 0
Dumped core?: 0

Shutting down ossec-hids: [ OK ]
Starting ossec-hids: [FAILED]

and in /var/ossec/logs/ossec.log:

2014/01/06 21:33:42 ossec-testrule: INFO: Reading decoder file etc/decoder.xml.
2014/01/06 21:33:42 ossec-testrule: INFO: Reading decoder file etc/decoders.d/01-asl-decoder.xml.
2014/01/06 21:33:42 ossec-testrule: INFO: Reading decoder file etc/decoders.d/10-asl-drupal-decoder.xml.
2014/01/06 21:33:42 ossec-testrule: INFO: Reading decoder file etc/decoders.d/50-asl-waf-decoder.xml.
2014/01/06 21:33:42 ossec-testrule: INFO: Reading decoder file etc/decoders.d/75-asl-deltaadmin-decoder.xml.
2014/01/06 21:33:42 ossec-testrule: INFO: Reading decoder file etc/decoders.d/75-asl-exim-decoder.xml.
2014/01/06 21:33:43 ossec-analysisd: Duplicate rule ID:393602
2014/01/06 21:33:43 ossec-testrule(1220): ERROR: Error loading the rules: 'exclusion_rules.xml'.
2014/01/06 21:34:53 ossec-testrule: INFO: Reading decoder file etc/decoder.xml.
2014/01/06 21:34:53 ossec-testrule: INFO: Reading decoder file etc/decoders.d/01-asl-decoder.xml.
2014/01/06 21:34:53 ossec-testrule: INFO: Reading decoder file etc/decoders.d/10-asl-drupal-decoder.xml.
2014/01/06 21:34:53 ossec-testrule: INFO: Reading decoder file etc/decoders.d/50-asl-waf-decoder.xml.
2014/01/06 21:34:53 ossec-testrule: INFO: Reading decoder file etc/decoders.d/75-asl-deltaadmin-decoder.xml.
2014/01/06 21:34:53 ossec-testrule: INFO: Reading decoder file etc/decoders.d/75-asl-exim-decoder.xml.
2014/01/06 21:34:53 ossec-analysisd: Duplicate rule ID:393602
2014/01/06 21:34:53 ossec-testrule(1220): ERROR: Error loading the rules: 'exclusion_rules.xml'.

Thanks

[chrismcb]

Just had the same issue on three boxes.

The rule listed as a "duplicate", for me, is an override in `/etc/asl/rules`

Code: Select all

2014/01/07 04:23:17 ossec-analysisd: Duplicate rule ID:3901

Clearing this file out, running asl -s -f and trying again, brings up another, different rule

Code: Select all

2014/01/07 04:16:24 ossec-analysisd: Duplicate rule ID:70800

Stopping psmon for the time-being until a fix is found

[hostingguy]

i tried that too, but psmon restarted itself apparently and keeps spamming me with emails...

[ghazlewood]

Also experiencing this on two servers, error in ossec.log is:

Code: Select all

2014/01/07 09:17:33 ossec-testrule: INFO: Reading decoder file etc/decoder.xml.
2014/01/07 09:17:33 ossec-testrule: INFO: Reading decoder file etc/decoders.d/01-asl-decoder.xml.
2014/01/07 09:17:33 ossec-testrule: INFO: Reading decoder file etc/decoders.d/10-asl-drupal-decoder.xml.
2014/01/07 09:17:33 ossec-testrule: INFO: Reading decoder file etc/decoders.d/50-asl-waf-decoder.xml.
2014/01/07 09:17:33 ossec-testrule: INFO: Reading decoder file etc/decoders.d/75-asl-deltaadmin-decoder.xml.
2014/01/07 09:17:33 ossec-testrule: INFO: Reading decoder file etc/decoders.d/75-asl-exim-decoder.xml.
2014/01/07 09:17:34 ossec-analysisd: Duplicate rule ID:71001
2014/01/07 09:17:34 ossec-testrule(1220): ERROR: Error loading the rules: 'exclusion_rules.xml'.

exclusion_rules.xml contains:

Code: Select all

<group name="local,syslog,modsecurity,">
  <rule id="999999" level="0">
    <description>List of rules to be ignored.</description>
  </rule>
        </group>

<group name="modsecurity,">

        <rule id="71001" level="9">
                <if_sid>60118, 60121</if_sid>
                <match>id "341245"</match>
                <description>Custom event for rule id 341245</description>
        </rule>

        <rule id="71002" level="9">
                <if_sid>60118, 60121</if_sid>
                <match>id "340148"</match>
                <description>Custom event for rule id 340148</description>
        </rule>

</group>

So as far as I can see not a duplicated rule as such in this file, I assume it means across all the rulesets.

[chrismcb]

i tried that too, but psmon restarted itself apparently and keeps spamming me with emails...

Simply comment out the cron task temporarily at:

Code: Select all

/etc/cron.d/psmon

Then stop psmon:

Code: Select all

service psmon stop

[chrismcb]

Has anyone submitted this as a case via the support portal?
Don't want to open multiples if it's already being addressed.

[mikeshinn]

That can happen if you have an incomplete upgrade, for example if your system picked up an older version of ossec or didnt run the full upgrade. Run these commands:

yum clean all

aum -uf

asl -s -f

[chrismcb]

Hi Mike,

I have already ran these commands and the issue persists.

The upgrade happened automatically for me, it wasn't a manual request.

ossec-hids at the version below:

Code: Select all

# rpm -qa | grep ossec
ossec-hids-server-2.7.1-37.el6.art.x86_64
ossec-hids-2.7.1-37.el6.art.x86_64
ossec-hids-mysql-2.7.1-37.el6.art.x86_64

Anything else you can suggest?

[joburke]

Mike:

Tried, that. ossec still won't start

it gives me
2014/01/07 09:25:19 ossec-testrule: INFO: Reading decoder file etc/decoder.xml.
2014/01/07 09:25:19 rules_list: Category '1' not found. Invalid 'category'.
2014/01/07 09:25:43 ossec-testrule: INFO: Reading decoder file etc/decoder.xml.
2014/01/07 09:25:43 rules_list: Category '1' not found. Invalid 'category'.

[mikeshinn]

Can you post the output of the upgrade commands:

aum -u

asl -s -f

Then restart ossec, so we can see if any errors are occurring:

service ossec-hids restart

grep -i error /var/ossec/logs/ossec.log

And also this, so we can see whats installed on your system:

rpm -qa | egrep "ossec-hids|^asl"

[chrismcb]

Hi Mike,

As requested:

Code: Select all

# aum -u
Checking for updates..
  ASL version is current: 3.2.15-33.el6.art                [OK]
    Installing malware detection module: successful        [OK]
  Updating ASL Kernel: 3.2.53-58                           [OK]
  APPINV rules are current: 201308071122                   [OK]
  CLAMAV rules are current: 201401061420                   [OK]
  GEOMAP rules are current: 201401070856                   [OK]
  MODSEC rules are current: 201401070908                   [OK]
  OSSEC rules are current: 201312301336                    [OK]

Code: Select all

# asl -uf
Checking for updates..
  Upgrading ASL Components
    Updating ASL Core: successful                          [OK]
    Installing malware detection module: successful        [OK]
  Updating ASL Kernel: 3.2.53-58                           [OK]
  Updating APPINV to 201308071122: updated                 [OK]
  Updating CLAMAV to 201401061420: updated                 [OK]
  Updating GEOMAP to 201401070856: updated                 [OK]
  Updating MODSEC to 201401070908: updated                 [OK]
    Updating Anti-Spam Protection: updated                 [OK]
    Updating Attack Protection: updated                    [OK]
    Updating Dataloss Protection: updated                  [OK]
    Updating Malware Protection: updated                   [OK]
    Updating Rootkit Protection: updated                   [OK]
    Updating Shell Protection: updated                     [OK]
  Updating OSSEC to 201312301336: updated                  [OK]
    Updating Self Healing modules: updated                 [OK]
    Updating Brute Force Protection: updated               [OK]
    Updating Rootkit Protection: updated                   [OK]

Output of asl -s -f attached

Code: Select all

#grep -i error /var/ossec/logs/ossec.log
2014/01/07 14:16:30 ossec-testrule(1220): ERROR: Error loading the rules: 'exclusion_rules.xml'.
2014/01/07 14:17:23 ossec-testrule(1220): ERROR: Error loading the rules: 'exclusion_rules.xml'.
2014/01/07 14:17:43 ossec-testrule(1220): ERROR: Error loading the rules: 'exclusion_rules.xml'.
2014/01/07 14:18:09 ossec-testrule(1220): ERROR: Error loading the rules: 'exclusion_rules.xml'.
2014/01/07 14:39:03 ossec-testrule(1220): ERROR: Error loading the rules: 'exclusion_rules.xml'.
2014/01/07 14:39:40 ossec-testrule(1220): ERROR: Error loading the rules: 'exclusion_rules.xml'.
2014/01/07 14:40:07 ossec-testrule(1220): ERROR: Error loading the rules: 'exclusion_rules.xml'.
2014/01/07 14:41:17 ossec-testrule(1220): ERROR: Error loading the rules: 'exclusion_rules.xml'.

*above only a recent snippet - same for hundreds of lines

Code: Select all

# rpm -qa | egrep "ossec-hids|^asl"
asl-php-process-5.4.23-21.el6.art.x86_64
asl-3.2.15-33.el6.art.x86_64
asl-php-common-5.4.23-21.el6.art.x86_64
asl-php-pdo-5.4.23-21.el6.art.x86_64
asl-php-5.4.23-21.el6.art.x86_64
asl-php-mysqlnd-5.4.23-21.el6.art.x86_64
asl-php-gd-5.4.23-21.el6.art.x86_64
ossec-hids-server-2.7.1-37.el6.art.x86_64
asl-waf-module-3.2.15-33.el6.art.x86_64
asl-stream-client-1.0-4.el6.art.x86_64
asl-php-pecl-apc-3.1.13-4.el6.art.x86_64
asl-php-cli-5.4.23-21.el6.art.x86_64
ossec-hids-2.7.1-37.el6.art.x86_64
ossec-hids-mysql-2.7.1-37.el6.art.x86_64
asl-web-3.2.15-33.el6.art.x86_64

[zeki]

i have the same issue, too on my server.

[hostingguy]

case opened.

[chrismcb]

case opened.

Sorry, should have said, I opened a case earlier... It has just been closed as Mike has been replying on this thread.