Page 1 of 1

Search engines crawlers being blocked

Posted: Wed Jun 11, 2014 5:42 am
by dutchnet
in the recent events screen the search engine crawlers like google and msn bots are being blocked on all our ASL servers.

some of the IP addresses being blocked:

66.249.65.139
66.249.67.133
66.249.67.95

from the config file /etc/asl/config

MODSEC_00_WHITELIST="no"
MODSEC_00_SEARCHENGINE="yes"
MODSEC_00_AUTOWHITELIST_SEARCHENGINE="yes"
MODSEC_00_ANTIEVASION="yes"
MODSEC_00_STRICT="no"
MODSEC_00_RBL="yes"


from /etc/resolv.conf

# Generated by NetworkManager
search dutch-net.com
nameserver 127.0.0.1
nameserver 80.84.230.153
nameserver 80.84.230.151
options single-request

Re: Search engines crawlers being blocked

Posted: Thu Jun 12, 2014 9:44 am
by scott
Are you using nginx as a reverse proxy or ipv6 by any chance?