Atomicorp

Got a hungry bot on our server this morning. It is spread across a wide range of networks, so blocking it by IP would be at least impractical and at most ineffective.

Assuming the people who run it will keep using the same third level domain for all rDNS addresses where this bot comes from, is there a way to block any request whose rDNS is 007AC9.net?

PS: Although the link above shows this bot using an unique UA, it hit my server using a generic UA ("Mozilla/5.0 (Windows; U; Windows NT 6.1; de-DE) AppleWebKit/534.17 (KHTML, like Gecko) Chrome/10.0.649.0 Safari/534.17").

Yes, you can do this with this ruleset:

https://www.atomicorp.com/wiki/index.ph ... AIN_BLOCKS

mikeshinn wrote:Yes, you can do this with this ruleset:

https://www.atomicorp.com/wiki/index.ph ... AIN_BLOCKS

Thanks Mike. I placed "007ac9.net" in the file. Will it satisfy the filter to effectively block, for example, crawl07.lp.007ac9.net (91.121.79.180)? AFAIU it should work:

Yes, that will block everything from that domain.

Just got a very draining bot coming from bzq-82-80-249-168.dcenter.bezeqint.net.

I added dcenter.bezeqint.net to the MODSEC_01_DOMAIN_BLOCKS list (I was able to get rid of 007AC9.net this way). But dcenter.bezeqint.net didnt work for bzq-82-80-249-168.dcenter.bezeqint.net. Why?

I can't block the entire bezeqint.net netblock because they are also an ISP for legit customers.

Thanks

Also, I couldnt block the evil (1, 2, 3)

coming from dozens of different IPs to scrape a magento site, which threw it for an endless loop.

The offending netblocks were

172.255.0.0/16 NOBIS-TECHNOLOGY-GROUP-15
23.80.0.0/14 NOBIS-TECHNOLOGY-GROUP-17
23.104.0.0/13 NOBIS-TECHNOLOGY-GROUP-18

and an example IP 23.81.239.84.

Adding "as15003.net" to the custom-domain-blocks file didn't do it.

Am I missing something or is was not supposed to work for these IPs?

The forward and reverse records have to match for the domain blocking rules to work, and in this case they dont:

[mshinn@kungfu ~]$ nslookup 23.81.239.84.rdns.as15003.net
;; Got SERVFAIL reply from 8.8.8.8, trying next server
;; Got SERVFAIL reply from 8.8.8.8, trying next server
Server: 8.8.4.4
Address: 8.8.4.4#53

** server can't find 23.81.239.84.rdns.as15003.net: SERVFAIL

[mshinn@kungfu ~]$ nslookup 23.81.239.84
Server: 8.8.8.8
Address: 8.8.8.8#53

Non-authoritative answer:
84.239.81.23.in-addr.arpa name = 23.81.239.84.rdns.as15003.net.

We could add in a capability to make the lookups non-verified (the PTR doesnt have to match the A) - or both (you decide how verified it needs to be). This would only work on ASL systems so if thats something you'd like we can add it into the FRs and see about rolling it out next week.

If those network ranges are something you really want to block I supposed you could blacklist them, but I gather you may not want to do that. But let me know what approach you prefer. We may be able to create an RBL for this kind of thing as well, which would make it more dynamic. We'd have to brainstorm a little about how to do that intelligently for this use case.

mikeshinn wrote:We could add in a capability to make the lookups non-verified (the PTR doesnt have to match the A) - or both (you decide how verified it needs to be). This would only work on ASL systems so if thats something you'd like we can add it into the FRs and see about rolling it out next week.

That would be great. Please add it to the FR list.

mikeshinn wrote: If those network ranges are something you really want to block I supposed you could blacklist them, but I gather you may not want to do that.

Correct. Plus I'd have to stay on top of it every time they add a new netblock.

mikeshinn wrote:We may be able to create an RBL for this kind of thing as well, which would make it more dynamic. We'd have to brainstorm a little about how to do that intelligently for this use case.

I'd be glad to assist with placing these bad agents in an RBL.

Thanks

Can you share your access logs with me, I'll see what we might be able to do on the RBL side as well. Since we can create as many RBLs as you can imagine, I'm thinking we might create some RBLs for things like "impolite bots" similar to the spammer RBLs and others we already have.

mikeshinn wrote:Can you share your access logs with me, I'll see what we might be able to do on the RBL side as well. Since we can create as many RBLs as you can imagine, I'm thinking we might create some RBLs for things like "impolite bots" similar to the spammer RBLs and others we already have.

PM me an email address I'll give it access to papertrail so you can browse/filter/search the logs.