Some basic help please...

[chris74]

Hi, I just signed up for a subscription for my server and I would like to set up a basic cron job to download the rules every day. From what I can tell, the rules are only protected by an htaccess login so I can use wget to download them.

The first problem I can see is that the filename of the latest ruleset changes each time with a different "timestamp". At a glance, the latest one appears to be modsec-201306110809.tar.gz - that doesn't make it very easy to get hold of the latest rules. How can I automate that? Why don't they just name the latest archive "modsec-latest" - so anyone subscribing to the rules without their ASL product can get hold of them easily? That would be common sense right?

After all, these are called "real time" rules - so they need to make it easy for me to automate the process of downloading them. Unfortunately It doesn't seem to me like they have put much effort into that. It seems I can have automation only if I pay for ASL - which I don't need or want.

Has anyone developed a script to do this? I noticed on this forum there was talk of an asl-lite script, but this doesn't appear to be available for download any longer.

It seems very strange to me that I need to ask these questions. You'd think when paying for a "real time" regularly updated file download, that this would be simple to automate. I'm not a programmer but I can write basic scripts to automate tasks - although I'm really not sure how I would go about identifying the latest modsec file to download from that directory without checking manually first.

[scott]

Sure, if you look in the subscription/ directory, there is a file called VERSION, that lists the versions of each ruleset (waf, hids, clam, etc). The filenames are <ruleset name>-<version>.tar.gz

We had used the "-latest" naming convention in the past, but had to get rid of it because people wouldn't be able to tell what version they had downloaded. It was a lot of overhead for us to figure out what they were really using (Us: What version are you running, Them: latest!, Us: Which latest?, Them: From yesterday! Us: What time yesterday? Them: 3pm Us: Eastern standard time?). Needless to say that conversation took a while, and thus the VERSION file was born.

[chris74]

Ah - ok, that's very useful thank you for taking the time to respond.

The file contains...

MODSEC_VERSION=201306111500

So I presume I'd need to use sed or grep to identify this and turn it into a filename ? This is a little complicated for me unfortunately. I could probably workout eventually how to output the digits after MODSEC_VERSION=, but I think it would require some programming to make that a "variable", or whatever you call it - and insert it as a filename into a wget command.

I would guess that you already did this in your asl-lite product - could you possibly provide the details? If part of that product contains a download script that performs the above - would it be possible to obtain that script - instead of having to install the whole outdated an unsupported asl-lite application?

[scott]

Yeah basic shell 101, if you source a file from your script, it treats elements separated by = as an internal variable:

example:
source /path/to/VERSION

Now all those = separated lines are $variables. So say you have FOO=bar

echo $FOO

would return: bar

echo "Test example-$FOO" would return:

Test example-bar

or
wget http://domain.com/path/to/something/somefile-$FOO

[chris74]

Thanks for your efforts. I think I understand.

So would this work?

Code: Select all

wget --user=username --password=password http://www.atomicorp.com/channels/rules/subscription/VERSION

source VERSION

wget --user=username --password=password http://www.atomicorp.com/channels/rules/subscription/$MODSEC_VERSION.tar.gz

[scott]

pretty close, each rule set in there is called <something>-<version>. You'll see (at the moment!) modsec-<version> and modsec-spam-<version>. The former is everything, the -spam one is just the anti-spam rules. Very soon we will have more subgroups targeted at rulesets just for specific applications, and the VERSION file will start to list those as separate variables.

[chris74]

"Pretty close" means that it will work or it wont?

Obviously I'd have to add a few other things to it, such as deleting the version file each time and unpacking the archive etc but essentially that would work just to download the correct archive right?

[scott]

The logic is spot on, you just missed the filename:

You'll see (at the moment!) modsec-<version> and modsec-spam-<version>.

[chris74]

Oh Yeah DOH! missed that.

Thanks for your help.

[JamesOakley]

You'll see (at the moment!) modsec-<version> and modsec-spam-<version>. The former is everything, the -spam one is just the anti-spam rules.

Can I just clarify? Are the modsec-spam-vvv files a subset of the modsec-vvv files? To ask another way: If I am downloading modsec-vvv.tar.gz, do I ever need to download modsec-spam-vvv.tar.gz as well?

[mikeshinn]

do I ever need to download modsec-spam-vvv.tar.gz as well?

No. They are just a subset. If you are ever not sure, you can just download the file and extract it to compare.

[jimlongo]

James, excuse me for butting into your thread.
I've been trying to get this shell script working.

I can connect and download the file with the raw shell command

Code: Select all

wget --user="username" --password="xxxxxxx" "http://www.atomicorp.com/channels/rules/subscription/VERSION"

However when I try that in a script I get an error "Unsupported Scheme"

Code: Select all

WGET='wget --user="username" --password="xxxxxxx" "http://www.atomicorp.com/channels/rules/subscription/VERSION"'
`$WGET`

OR alternatively

Code: Select all

WGET=' --user="username" --password="xxxxxxx" "http://www.atomicorp.com/channels/rules/subscription/VERSION"'
wget $WGET

Have you had success with this?

UPDATE SINCE LAST NIGHT:

I was able to accomplish this by

Code: Select all

URL_BASE="http://updates.atomicorp.com/channels/rules/subscription"
WGET_OPTS='--user=username --password=xxxxxxxxxxxxxx'

$wget ${WGET_OPTS} "${URL_BASE}/VERSION"

[mikeshinn]

updates.atomicorp.com is the correct server to download updates from. www.atomicorp.com is not correct.