Duplicate counter error after upgrading to 3.6.0

Support/Development for OSSEC
titleistfour
Forum User
Forum User
Posts: 7
Joined: Tue Mar 15, 2022 12:05 pm

Duplicate counter error after upgrading to 3.6.0

Unread post by titleistfour »

Hello,

We recently updated an older OSSEC server (2.9.0) to version 3.6.0. We followed the upgrade guidelines for backing up all the OSSEC files and restoring them on the new server. The existing agents appeared to communicate just fine and are working. However, when we add a new agent on the new server, we immediately get a duplicate counter error. It's easy to resolve that issue, but going forward we don't want to have to do that for every new agent we add.

These are for a Linux server (Oracle Linux 8) and Linux agents (various OL6, 7 and 8).

What is going on and is there a solution to this?

Thanks,
J
User avatar
cponton
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin
Posts: 61
Joined: Fri Oct 09, 2020 9:41 am

Re: Duplicate counter error after upgrading to 3.6.0

Unread post by cponton »

I think you may be saying that you have duplicate agent ids? If so you can remove them:

Step 1: Use the manage agents service to remove the agent from the Manager. Provide the ID of the agent you want to remove:



[root@atomic-manager ~]# /var/ossec/bin/manage_agents


****************************************
* OSSEC HIDS v4.0.0 Agent manager. *
* The following options are available: *
****************************************
(A)dd an agent (A).
(E)xtract key for an agent (E).
(L)ist already added agents (L).
(R)emove an agent (R).
(Q)uit.
Choose your action: A,E,L,R or Q: R

Available agents:
ID: 001, Name: DEFAULT_LOCAL_AGENT, IP: 127.0.0.1
ID: 002, Name: linux-agent, IP: any
ID: 003, Name: windows-agent, IP: any

Provide the ID of the agent to be removed (or '\q' to quit):


Step 2: Remove the line with the agent id in /var/ossec/etc/client.keys:

[root@atomic-manager ~]# vi /var/ossec/etc/client.keys

001 DEFAULT_LOCAL_AGENT 127.0.0.1 46760643808e75da0f8208993905458fc36d6297bff9eff475eeb5ea25677033
002 linux-agent any dbd752e9b9c9d3511c8fd11734e2b48efba7087e8b0cf1e76daf96b4d6456d5d
003 windows-agent any aa87a8e874c8e83e5c4afa993c40607d1bae21e884b1a44b66367e61b4e2d0cc
move the cursor to the line of the agent that was removed. Type: dd to remove the agent. Then hit esc : x

so that it saves the changes and exits the file.

Step 3: Restart OSSEC
titleistfour
Forum User
Forum User
Posts: 7
Joined: Tue Mar 15, 2022 12:05 pm

Re: Duplicate counter error after upgrading to 3.6.0

Unread post by titleistfour »

Hi,

Actually, no that is not what I mean. What we see are these types of errors now when adding new agents.

Code: Select all

2022/03/17 08:29:37 ossec-remoted: WARN: Duplicate error:  global: 1, local: 3154, saved global: 1, saved local:3155
2022/03/17 08:29:37 ossec-remoted(1407): ERROR: Duplicated counter for 'xxxxxxxxxxxxxxxxxxxx'.
2022/03/17 08:29:37 ossec-remoted: WARN: Duplicate error:  global: 1, local: 2944, saved global: 1, saved local:2945
2022/03/17 08:29:37 ossec-remoted(1407): ERROR: Duplicated counter for 'xxxxxxxxxxxxxxxxxxxx'.
2022/03/17 08:29:38 ossec-remoted: WARN: Duplicate error:  global: 1, local: 3149, saved global: 1, saved local:3150
2022/03/17 08:29:38 ossec-remoted(1407): ERROR: Duplicated counter for 'xxxxxxxxxxxxxxxxxxxx'.
2022/03/17 08:29:38 ossec-remoted: WARN: Duplicate error:  global: 1, local: 2965, saved global: 1, saved local:2966
2022/03/17 08:29:38 ossec-remoted(1407): ERROR: Duplicated counter for 'xxxxxxxxxxxxxxxxxxxx'.
2022/03/17 08:29:39 ossec-remoted: WARN: Duplicate error:  global: 1, local: 3415, saved global: 1, saved local:3416
2022/03/17 08:29:39 ossec-remoted(1407): ERROR: Duplicated counter for 'xxxxxxxxxxxxxxxxxxxx'.
2022/03/17 08:29:41 ossec-remoted: WARN: Duplicate error:  global: 1, local: 2957, saved global: 1, saved local:2958
2022/03/17 08:29:41 ossec-remoted(1407): ERROR: Duplicated counter for 'xxxxxxxxxxxxxxxxxxxx'.
In the past I've been able to follow the instructions here to resolve this, but it doesn't seem to be working for new agents.
https://www.ossec.net/docs/faq/unexpect ... ate-errors

Not really sure what to do at this point. We are adding a lot of new agents and these errors are showing up in the logs constantly.

Thanks,
J
User avatar
cponton
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin
Posts: 61
Joined: Fri Oct 09, 2020 9:41 am

Re: Duplicate counter error after upgrading to 3.6.0

Unread post by cponton »

I see now.

Can you tell me what the setting is for remoted.verify_msg_id in /var/ossec/etc/internal_options.conf
titleistfour
Forum User
Forum User
Posts: 7
Joined: Tue Mar 15, 2022 12:05 pm

Re: Duplicate counter error after upgrading to 3.6.0

Unread post by titleistfour »

On both of our OSSEC servers, that option is set to

Code: Select all

remoted.verify_msg_id=1
On our clients, it appears to be set the same. I checked and our old servers had the same option enabled.

J
User avatar
cponton
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin
Posts: 61
Joined: Fri Oct 09, 2020 9:41 am

Re: Duplicate counter error after upgrading to 3.6.0

Unread post by cponton »

Thank you.

You can disable the counter by changing that remoted.verify_msg_id=1 option to a 0 or you can delete the agent counters

Check your counters on both sides. The agent side should be higher that the HUB side
# Agent side
# cat /var/ossec/queue/rids/sender_counter
0:4243:

# Manager side. Replace myagent for your actual agent's name
# cat /var/ossec/queue/rids/$(grep -E "[[:digit:]]+ myagent" /var/ossec/etc/client.keys | cut -d' ' -f1)
0:4097:

I believe you have already been using this option but you can delete the counters on the agent side:
# /var/ossec/bin/ossec-control stop
# find /var/ossec/queue/rids -type f -not -name sender_counter | xargs rm
# /var/ossec/bin/ossec-control start
titleistfour
Forum User
Forum User
Posts: 7
Joined: Tue Mar 15, 2022 12:05 pm

Re: Duplicate counter error after upgrading to 3.6.0

Unread post by titleistfour »

Thank you for the response. But if these are brand new agents, why is this necessary?

I really don't want to disable this security feature.

J
titleistfour
Forum User
Forum User
Posts: 7
Joined: Tue Mar 15, 2022 12:05 pm

Re: Duplicate counter error after upgrading to 3.6.0

Unread post by titleistfour »

I just manually added a new client on the OSSEC server, imported the key on the client, and a few minutes later, duplicate counter now shows in the log.

So could our new server be setup incorrectly or the migration was done wrong somehow? I don't really understand the underlying issue on why all the new agents are showing duplicate counter. Shouldn't it generate a new counter for each new agent?

J
scott
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin
Posts: 8355
Joined: Wed Dec 31, 1969 8:00 pm
Location: earth
Contact:

Re: Duplicate counter error after upgrading to 3.6.0

Unread post by scott »

That also happen if a key is pinned to an IP address, or if you've got NAT involved and multiple agents are coming from the same source IP
titleistfour
Forum User
Forum User
Posts: 7
Joined: Tue Mar 15, 2022 12:05 pm

Re: Duplicate counter error after upgrading to 3.6.0

Unread post by titleistfour »

Thanks. We definitely don't have any NAT going on, this is all on a local LAN. But, what do you mean by "if a key is pinned to an IP address"? I thought every client key was associated to a hostname and IP. That's not default behavior?

J
titleistfour
Forum User
Forum User
Posts: 7
Joined: Tue Mar 15, 2022 12:05 pm

Re: Duplicate counter error after upgrading to 3.6.0

Unread post by titleistfour »

I've seen some posts about remove everything in the queue/rids folder on both the server and client. Tried that, and restarted agent and server.
Immediately get duplicate counters again. So something is not right somewhere that is causing this. Would appreciate some advice and where else to look for a solution.

J
Post Reply