WP xmlrpc attack

faris · Unread post by **faris** » Thu Jul 31, 2014 8:11 am

I'm experiencing some interesting behaviour on a couple of systems, where ASL's active response appears to be inadvertently amplifying the effect of a certain type of attack on a particular system (though this is not ASL's fault, btw).

What appears to be a botnet (or at least an uncountable numbers of different IPs) are sending POSTs to Wordpress's xmlrpc.php. e.g:

Code: Select all

197.202.32.133 - - [31/Jul/2014:12:43:41 +0100] "POST /xmlrpc.php HTTP/1.1" 403 1234 "-" "Mozilla/4.0 (compatible; Win32; WinHttp.WinHttpRequest.5)"
79.178.9.178 - - [31/Jul/2014:12:43:44 +0100] "POST /xmlrpc.php HTTP/1.1" 403 1234 "-" "Mozilla/4.0 (compatible; Win32; WinHttp.WinHttpRequest.5)"
2.50.55.13 - - [31/Jul/2014:12:43:45 +0100] "POST /xmlrpc.php HTTP/1.1" 403 1234 "-" "-"
220.255.2.122 - - [31/Jul/2014:12:43:48 +0100] "POST /xmlrpc.php HTTP/1.1" 403 1234 "-" "-"

I'm assuming this is something to do with the Wordpress pingback DDoS issue, although none of the sites that are being attacked are running 3.5 which was the directly vulnerable version of Wordpress if I recall correctly.

I'm seeing this on two systems - one has ASL, one does not.

On the system NOT running ASL, this attack causes load levels to increase to 15 while the attack is in progress.

I can reduce the load back to normal and prevent further issues on the site being attacked by adding something like this to the site's .htaccess :

Code: Select all

RewriteRule ^xmlrpc\.php$ "http\:\/\/0\.0\.0\.0\/" [R=301,L]

I assume that basically since we're not using php to process anything, and all apache has to do is send the 301 redirect, load instantly reduces. Note that you could alternatively use a different type of redirect. This is just the one I used. You could redirect to a 403 or something.

Obviously this rule prevents legitimate use of xmlrpc but I don't particularly care as none of the sites use it in any way.

On the system protected by ASL, things are different.

mod_sec intercepts and blocks the POSTs and returns a 403. The rule being triggered is
392301: Request Containing Content, but Missing Content-Type header

This is fine and great and exactly what we want.

However, things go wrong as a result of the Active Response.

Because of the nature of the attack, large numbers of IPs are added to the firewall. Periodically, these IPs get deleted after the shun time expires:

Code: Select all

/bin/sh /var/ossec/active-response/bin/asl-shun.pl delete [stuff]

When this happens, load skyrockets and sometimes stays that way for quite a while. 15 min load averages can easily hit 25 or more.

The main issue appears to be memory - during the shun deletes, lots and lots of memory appears to being consumed, and I can't figure out why this might be so.

On one occasion, when I made a configuration change to ASL in the middle of one of these attacks, load hit 500+ while ASL tried to restart as it was trying to delete scores of entries at a time and the system (VMware) kept killing the shun delete processes due to oom conditions.

I suppose I could avoid the load issue by turning active response off for this particular rule. Problem is then solved, I guess, and since mod_sec is returning a 403 everything is happening exactly as I'd want it and all is well.

Nevertheless, I would like to figure out if there might be a better solution. This apparently huge memory use during the shun deletes bothers me. The system in question has a whopping 16Gb of RAM and plenty of processing power - this is not a lightweight VPS.

Has anyone else encountered something similar?

prupert · Unread post by **prupert** » Thu Jul 31, 2014 8:59 am

Yes, we have encountered the same on a couple of machines.

The exact same thing happened:
- distributed attacks on xmlrpc.php
- mod_security blocks requests with rule 392301
- active responses initiated: stacking up of ASL shun processes, causing the load to spike

Disabling 'active responses' for rule 392301 does mitigate the load impact.

It would be nice if the process of shunning an IP address was made more efficiently. The HIDS dealing with such attacks should not cause a denial of service.

Unread post by **mikeshinn** » Thu Jul 31, 2014 9:50 am

On a hunch, whats the output of this command?

grep directories /var/ossec/etc/ossec-server.conf

prupert · Unread post by **prupert** » Thu Jul 31, 2014 11:59 am

I don't think that has anything to do with this issue, but here you go:

Code: Select all

# grep directories /var/ossec/etc/ossec-server.conf
    <directories check_all="yes">/etc,/usr/bin,/usr/sbin</directories>
    <directories check_all="yes">/bin,/sbin</directories>
    <!-- Files/directories to ignore -->

faris · Unread post by **faris** » Thu Jul 31, 2014 12:27 pm

On the ASL system affected:

Code: Select all

# grep directories /var/ossec/etc/ossec-server.conf
        <directories realtime="yes" check_all="yes" report_changes="yes"       >                                                                                               /etc</directories>
        <directories realtime="yes" check_all="yes" report_changes="yes"       >                                                                                               /var/ossec</directories>
        <directories realtime="yes" check_all="yes" report_changes="yes"       >                                                                                               /bin</directories>
        <directories realtime="yes" check_all="yes" report_changes="yes"       >                                                                                               /lib</directories>
        <directories realtime="yes" check_all="yes" report_changes="yes"       >                                                                                               /lib64</directories>
        <directories realtime="yes" check_all="yes" report_changes="yes"       >                                                                                               /opt</directories>
        <directories realtime="yes" check_all="yes" report_changes="yes"       >                                                                                               /sbin</directories>
        <directories realtime="yes" check_all="yes" report_changes="yes"       >                                                                                               /usr/bin</directories>
        <directories realtime="yes" check_all="yes" report_changes="yes"       >                                                                                               /usr/lib</directories>
        <directories realtime="yes" check_all="yes" report_changes="yes"       >                                                                                               /usr/lib64</directories>
        <directories realtime="yes" check_all="yes" report_changes="yes"       >                                                                                               /usr/local/bin</directories>
        <directories realtime="yes" check_all="yes" report_changes="yes"       >                                                                                               /usr/local/lib</directories>
        <directories realtime="yes" check_all="yes" report_changes="yes"       >                                                                                               /usr/local/sbin</directories>

Unread post by **mikeshinn** » Thu Jul 31, 2014 2:02 pm

Thank you. Drop this:

<directories realtime="yes" check_all="yes" report_changes="yes" > /var/ossec</directories>

Let me know how that effects your system. I think you might be he hitting some I/O contention.

faris · Unread post by **faris** » Thu Jul 31, 2014 4:44 pm

ok. Thanks. Done and restarted ossec-hids.

I'll keep you updated. These attacks happen relatively frequently.

DarkF@der · Unread post by **DarkF@der** » Thu Jul 31, 2014 4:52 pm

How to drop it?

load average: 875.42, 547.08, 256.42

Code: Select all

Rule	Level	Count	  
392301	6	10600	Atomicorp.com WAF Rules: Request Containing Content, but Missing Content-Type header

Code: Select all

10648 root      20   0     0    0    0 D  1.3  0.0   0:00.07 [ar-tracking.sh]
 1980 named     20   0  228m  30m 2692 S  1.1  0.2   1:12.41 /usr/sbin/named -u named -c /etc/named.conf -u named -n 2 -t /var/named/chroot
 7166 root      20   0     0    0    0 D  1.1  0.0   0:00.09 [mysql]
 7609 root      20   0  125m 3116 2308 R  1.1  0.0   0:00.08 /usr/bin/mysql -h 127.0.0.1 -u tortix -px xxxxxx tortix -e UPDATE aslw_blocklist set is_blocked=0 where src
 8281 root      20   0  9276 1368 1096 S  1.1  0.0   0:00.06 /bin/sh /var/ossec/active-response/bin/ar-tracking.sh delete - 88.229.109.176 1406847263.10240695 392301 /v
 8883 root      20   0  9276 1120  956 R  1.1  0.0   0:00.06 /bin/sh /var/ossec/active-response/bin/ar-tracking.sh delete - 39.46.48.53 1406847265.10241411 392301 /var/
 9012 root      20   0 52340 1816 1300 R  0.9  0.0   0:00.05 /usr/bin/mysql -h 127.0.0.1 -u tortix -pqgZPKBLT tortix -e UPDATE aslw_blocklist set is_blocked=0 where src
21571 root      20   0 13604 1384  752 S  0.9  0.0   0:02.48 /var/ossec/bin/ossec-execd
 1257 root      20   0  9544 1624 1140 S  0.8  0.0   0:00.43 /bin/sh /var/asl/lib/modules/report
 7612 root      20   0  9284 1416 1152 S  0.8  0.0   0:00.07 /bin/sh /var/ossec/active-response/bin/asl-shun.pl add - 46.50.59.44 1406847679.10434959 392301 /var/log/ht
 7704 root      20   0  9280 1368 1096 S  0.8  0.0   0:00.06 /bin/sh /var/ossec/active-response/bin/ar-tracking.sh delete - 177.65.3.39 1406847262.10239985 392301 /var/
 8091 root      20   0  9276 1120  948 S  0.8  0.0   0:00.04 /bin/sh /var/ossec/active-response/bin/asl-shun.pl delete - 88.229.109.176 1406847263.10240695 392301 /var/
 9410 root      20   0  9276 1116  952 R  0.8  0.0   0:00.04 /bin/sh /var/ossec/active-response/bin/ar-tracking.sh delete - 179.180.168.9 1406847265.10242121 392301 /va
 9628 root      20   0  9280 1252 1080 S  0.8  0.0   0:00.04 /bin/sh /var/ossec/active-response/bin/host-deny.sh add - 222.154.182.230 1406847683.10435669 392301 /var/l
 9639 root      20   0  9276 1252 1072 S  0.8  0.0   0:00.04 /bin/sh /var/ossec/active-response/bin/asl-shun.pl add - 222.154.182.230 1406847683.10435669 392301 /var/lo
10853 root      20   0     0    0    0 D  0.8  0.0   0:00.04 [chmod]
21168 root      20   0  9284 1420 1152 S  0.8  0.0   0:00.85 /bin/sh /var/ossec/active-response/bin/asl-shun.pl add - 80.99.252.128 1406847504.10346022 392301 /var/log/
 1124 root      20   0  9280 1348 1140 S  0.6  0.0   0:00.11 /bin/sh /var/ossec/active-response/bin/host-deny.sh add - 141.136.164.205 1406847667.10416933 392301 /var/l
 1146 root      20   0  9284 1416 1152 S  0.6  0.0   0:00.12 /bin/sh /var/ossec/active-response/bin/asl-shun.pl add - 141.136.164.205 1406847667.10416933 392301 /var/lo
 1160 root      20   0  9280 1348 1140 S  0.6  0.0   0:00.11 /bin/sh /var/ossec/active-response/bin/host-deny.sh add - 212.93.100.41 1406847667.10418320 392301 /var/log
 1176 root      20   0  9284 1416 1152 S  0.6  0.0   0:00.12 /bin/sh /var/ossec/active-response/bin/asl-shun.pl add - 212.93.100.41 1406847667.10418320 392301 /var/log/
 1227 root      20   0  9280 1348 1140 S  0.6  0.0   0:00.11 /bin/sh /var/ossec/active-response/bin/host-deny.sh add - 93.39.226.112 1406847667.10419748 392301 /var/log
 1240 root      20   0  237m 2644  980 S  0.6  0.0   0:59.08 /sbin/rsyslogd -i /var/run/syslogd.pid -c 5
 1319 root      20   0  9280 1416 1152 S  0.6  0.0   0:00.12 /bin/sh /var/ossec/active-response/bin/asl-shun.pl add - 209.197.166.240 1406847667.10421174 392301 /var/lo
 1343 root      20   0  9280 1348 1140 S  0.6  0.0   0:00.11 /bin/sh /var/ossec/active-response/bin/host-deny.sh add - 122.54.86.114 1406847667.10421892 392301 /var/log
 1364 root      20   0  9280 1348 1140 S  0.6  0.0   0:00.11 /bin/sh /var/ossec/active-response/bin/host-deny.sh add - 81.214.158.43 1406847667.10422606 392301 /var/log
 1428 root      20   0  9280 1348 1140 S  0.6  0.0   0:00.11 /bin/sh /var/ossec/active-response/bin/host-deny.sh add - 107.194.94.28 1406847667.10424032 392301 /var/log
 1481 root      20   0  9280 1416 1152 S  0.6  0.0   0:00.12 /bin/sh /var/ossec/active-response/bin/asl-shun.pl add - 197.160.84.30 1406847667.10426086 392301 /var/log/
 1552 root      20   0  9280 1416 1152 S  0.6  0.0   0:00.12 /bin/sh /var/ossec/active-response/bin/asl-shun.pl add - 187.135.57.160 1406847667.10430863 392301 /var/log
 3719 root      20   0  9280 1416 1152 S  0.6  0.0   0:00.36 /bin/sh /var/ossec/active-response/bin/asl-shun.pl add - 109.146.203.91 1406847604.10398916 392301 /var/log
 3746 root      20   0  9280 1348 1140 S  0.6  0.0   0:00.35 /bin/sh /var/ossec/active-response/bin/host-deny.sh add - 105.227.169.102 1406847604.10399632 392301 /var/l

prupert · Unread post by **prupert** » Thu Jul 31, 2014 5:15 pm

mikeshinn wrote:Thank you. Drop this:

<directories realtime="yes" check_all="yes" report_changes="yes" > /var/ossec</directories>

Let me know how that effects your system. I think you might be he hitting some I/O contention.

All servers we have seen this issue on do not have this directory being watched according to /var/ossec/etc/ossec-server.conf.

Unread post by **mikeshinn** » Thu Jul 31, 2014 6:29 pm

Thats odd, that directory is set by default. Would it be possible to log into one of the effected machines?

faris · Unread post by **faris** » Fri Aug 01, 2014 7:25 am

DarkF@der wrote:How to drop it?

Disable Active Response in GUI for rule 392301
Or, if only one site is being attacked, disable Active Response for that rule for only that site.

Be aware that it is possible that changing the setting might cause a large load during reconfiguration. It would be better to do it when the system is not overloaded. One way to do that might be to firewall port 80 temporarily to reduce load to normal, change Active Response on the rule, then un-block 80 again.

prupert · Unread post by **prupert** » Mon Aug 04, 2014 6:07 am

mikeshinn wrote:Thats odd, that directory is set by default. Would it be possible to log into one of the effected machines?

I have just opened support case no. 36946.

Unread post by **mikeshinn** » Mon Aug 04, 2014 11:01 am

Yeah, its definitely there on all 3 boxes you gave me access to (hostnames dropped):

grep /var/ossec *conf | grep directories

etc]# grep /var/ossec *conf | grep directories
ossec.conf: <directories realtime="yes" check_all="yes" report_changes="yes" >/var/ossec</directories>
etc]# grep /var/ossec *conf | grep directories
ossec.conf: <directories realtime="yes" check_all="yes" report_changes="yes" >/var/ossec</directories>
etc]# grep /var/ossec *conf | grep directories
ossec.conf: <directories realtime="yes" check_all="yes" report_changes="yes" >/var/ossec</directories>

Go ahead and remove those from your file integrity checks.

chrismcb · Unread post by **chrismcb** » Mon Aug 04, 2014 12:54 pm

Is removing this line the solution here?

Or a stop-gap measure until something more robust is changed with ASL?

I'm guessing the configuration was there for a reason in the first place?

Unread post by **mikeshinn** » Mon Aug 04, 2014 3:23 pm

Its been removed in 4.0.6 which will be released in a week. We are using a different method for ossec self watching now that uses less i/o.

Atomicorp

WP xmlrpc attack

WP xmlrpc attack

Re: WP xmlrpc attack

Re: WP xmlrpc attack

Re: WP xmlrpc attack

Re: WP xmlrpc attack

Re: WP xmlrpc attack

Re: WP xmlrpc attack

Re: WP xmlrpc attack

Re: WP xmlrpc attack

Re: WP xmlrpc attack

Re: WP xmlrpc attack

Re: WP xmlrpc attack

Re: WP xmlrpc attack

Re: WP xmlrpc attack

Re: WP xmlrpc attack