store | blogs | forums | twitter | facebook | wiki | downloads | support portal
Atomic Secure Linux
It is currently Fri Dec 14, 2018 9:19 pm

» Feed - Atomicorp

All times are UTC - 5 hours [ DST ]




Post new topic Reply to topic  [ 4 posts ] 
Author Message
 Post subject: Limit ossec queue diff retention
Unread postPosted: Sun Jan 15, 2017 8:49 am 
Offline
Long Time Forum Regular
Long Time Forum Regular

Joined: Thu Dec 09, 2004 11:19 am
Posts: 2321
I recently found well over 1.5 million files (and growing) in a specific /var/ossec/queue/diff/local directory, which was resulting in backups taking a very long time on one of my systems.

I would therefore like to suggest some sort of limit on the number of files that are stored in this way by ossec.
and/or add a specific exclusion by default to ignore /etc/magicspam/db/


Here are the details:

In Virtuozzo 4.x, which I use, an acronis-based backup system is the default backup type. This is very sensitive to file numbers, and doesn't care whether a directory including lots of files is excluded or not in the backup - it still affects backup times.

I discovered this when backing up one of my virtual environments - the backup time went from 1.5 hours to nearly twelve hours in a matter of two months, even though the amount of data had barely increased.

After some investigation, I found that /var/ossec/queue/diff/local/etc/magicspam/db contained over 1.5 million versions of a single tiny file.
As far as I can tell, /etc/magicspam/db contains some kind of database that gets updated extremely frequently, hence the millions of versions that file that ossec was keeping.

Obviously such a file does not belong in /etc/, which is supposed to be where configurations are stored, but that's where it is.

Now I realise not many people will use a combination of Virtuozzo 4.x (now EOL), Plesk, Magicspam, ASL and an acronis-based backup.

But plenty of people will be using Plesk with MagicSpam and ASL under Virtuozzo, and in those circumstances the number of inodes that can be used by the virtual environment is very likely to be limited. More limited than with a physical server at any rate.

Now I'm sure there must be SOME sort of limit on these diff files. I can't see ossec saving diffs for all eternity. That would be madness. But whatever this limit is, it certainly isn't in terms of numbers. So as things stand, the number of diff files could easily get to the point where a virtual environment hit an inode limit.

I realise that you can manually exclude a directory from ossec's monitoring systems. That's what I have done to resolve the problem.
But unless you regularly check for directories containing millions of files, you probably wouldn't know you had a problem brewing until the system stopped working.

So I think it would be useful to add something to ASL 5 (or 6) that limits the number of diffs that are stored, or at least to warn the user that huge numbers were being stored for a particular file.

Having said that, we are talking about a relatively unique case here - MagicSpam storing something that changes extremely frequently in a place where it should not really be stored.

So maybe a simpler option would be just to add an exclude for that specific directory /etc/magicspam/db in the default ASL ossec installation?

_________________
--------------------------------
<advert>
If you want to rent a UK-based VPS that comes with friendly advice and support from a fellow ART fan, please get in touch.
</advert>


Top
 Profile  
Reply with quote  
 Post subject: Re: Limit ossec queue diff retention
Unread postPosted: Fri Feb 03, 2017 5:41 pm 
Offline
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin
User avatar

Joined: Thu Feb 07, 2008 7:49 pm
Posts: 4066
Location: Chantilly, VA
Thanks for the feature requests.

Quote:
I would therefore like to suggest some sort of limit on the number of files that are stored in this way by ossec.
and/or add a specific exclusion by default to ignore /etc/magicspam/db/


Added in a few weeks as an exclusion.

Also, this setting:

HIDS_CLEAN_DIFF="60"

Sets the number of days to keep diffs. Is that what you were thinking of?

_________________
Michael Shinn
Atomicorp - Security For Everyone


Top
 Profile  
Reply with quote  
 Post subject: Re: Limit ossec queue diff retention
Unread postPosted: Fri Feb 03, 2017 10:02 pm 
Offline
Long Time Forum Regular
Long Time Forum Regular

Joined: Thu Dec 09, 2004 11:19 am
Posts: 2321
Thanks Mike,

I was thinking more in terms of number of files than in terms of number of days.

For reasons unknown, ASL is able to use the realtime option for file change activity, even though I'm not on the ASL kernel.

So it was saving copies of the file in question a huge number of times per day. In the end, over 1.5 million copies (and I guess that's in 60 days).

So how about a file number limit? Or at least an alert when over 1000 copies of a file are stored?

_________________
--------------------------------
<advert>
If you want to rent a UK-based VPS that comes with friendly advice and support from a fellow ART fan, please get in touch.
</advert>


Top
 Profile  
Reply with quote  
 Post subject: Re: Limit ossec queue diff retention
Unread postPosted: Fri Feb 10, 2017 9:39 am 
Offline
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin
User avatar

Joined: Thu Feb 07, 2008 7:49 pm
Posts: 4066
Location: Chantilly, VA
Quote:
For reasons unknown, ASL is able to use the realtime option for file change activity, even though I'm not on the ASL kernel.


If a newer kernel supports the method ASL will use it. I've open a feature request, thanks for the idea!

_________________
Michael Shinn
Atomicorp - Security For Everyone


Top
 Profile  
Reply with quote  
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 4 posts ] 

» Feed - Atomicorp

All times are UTC - 5 hours [ DST ]


Who is online

Users browsing this forum: No registered users and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
cron
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group