UNDER ATTACK

KrazyBob · Unread post by **KrazyBob** » Mon Aug 11, 2014 8:45 pm

I am under attack on a VPS. xmlrpc.php is being called so fast that I don't know how to find the hole. I actually renamed all of the xmlrpc.php to xmlrpc.php.dont-run yet TOP says they are running. What can I do?

Code: Select all

entered into VE 107
-bash-3.2 clss01 # tail -f /var/log/httpd/access_log  
46.27.163.199 - - [11/Aug/2014:17:17:41 -0700] "POST /xmlrpc.php HTTP/1.1" 500 3 "-" "-"
78.90.107.107 - - [11/Aug/2014:17:17:41 -0700] "POST /xmlrpc.php HTTP/1.1" 500 3 "-" "-"
85.250.184.98 - - [11/Aug/2014:17:17:45 -0700] "POST /xmlrpc.php HTTP/1.1" 500 254 "-" "-"
216.224.173.163 - - [11/Aug/2014:17:17:41 -0700] "GET / HTTP/1.1" 302 302 "-" "Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 4.0)"
95.92.14.103 - - [11/Aug/2014:17:17:38 -0700] "POST /xmlrpc.php HTTP/1.1" 500 3 "-" "-"
132.195.84.100 - - [11/Aug/2014:17:17:39 -0700] "POST /xmlrpc.php HTTP/1.1" 500 3 "-" "-"
109.64.57.113 - - [11/Aug/2014:17:17:43 -0700] "POST /xmlrpc.php HTTP/1.1" 500 3 "-" "-"
66.81.212.244 - - [11/Aug/2014:17:17:41 -0700] "GET /My%20Forum/phpBB3/styles/prosilver/template/styleswitcher.js HTTP/1.1" 200 3041 "http://www.foxcollectors.com/My%20Forum/phpBB3/index.php" "Mozilla/4.0 (compatible; MSIE 5.5; MSN 2.5; Windows 98)"
66.81.212.244 - - [11/Aug/2014:17:18:02 -0700] "GET /My%20Forum/phpBB3/styles/prosilver/template/forum_fn.js HTTP/1.1" 200 9142 "http://www.foxcollectors.com/My%20Forum/phpBB3/index.php" "Mozilla/4.0 (compatible; MSIE 5.5; MSN 2.5; Windows 98)"
74.86.158.107 - - [11/Aug/2014:17:18:01 -0700] "HEAD / HTTP/1.1" 200 - "-" "Mozilla/5.0+(compatible; UptimeRobot/2.0; http://www.uptimerobot.com/)"
83.25.200.173 - - [11/Aug/2014:17:42:25 -0700] "POST /xmlrpc.php HTTP/1.1" 500 254 "-" "-"
212.25.119.193 - - [11/Aug/2014:17:42:25 -0700] "POST /xmlrpc.php HTTP/1.1" 500 254 "-" "-"
178.148.242.107 - - [11/Aug/2014:17:42:26 -0700] "POST /xmlrpc.php HTTP/1.1" 500 254 "-" "-"
219.92.23.39 - - [11/Aug/2014:17:42:26 -0700] "POST /xmlrpc.php HTTP/1.1" 500 254 "-" "-"
79.180.143.71 - - [11/Aug/2014:17:42:26 -0700] "POST /xmlrpc.php HTTP/1.1" 500 254 "-" "-"
27.55.216.173 - - [11/Aug/2014:17:42:26 -0700] "POST /xmlrpc.php HTTP/1.1" 500 254 "-" "-"
31.209.198.112 - - [11/Aug/2014:17:42:27 -0700] "POST /xmlrpc.php HTTP/1.1" 500 254 "-" "-"
203.128.81.158 - - [11/Aug/2014:17:42:27 -0700] "POST /xmlrpc.php HTTP/1.1" 500 254 "-" "-"
145.255.73.154 - - [11/Aug/2014:17:42:27 -0700] "POST /xmlrpc.php HTTP/1.1" 500 254 "-" "-"
123.49.19.158 - - [11/Aug/2014:17:42:27 -0700] "POST /xmlrpc.php HTTP/1.1" 500 254 "-" "Mozilla/4.0 (compatible; Win32; WinHttp.WinHttpRequest.5)"
198.143.189.226 - - [11/Aug/2014:17:42:27 -0700] "GET /gallery/login.php?referer=login.php?referer=login.php?referer=login.php?referer=login.php?referer=thumbnails.php?album=11&page=1&sort=da HTTP/1.1" 200 183 "-" "Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup-crawler.com/)"
46.185.152.36 - - [11/Aug/2014:17:42:27 -0700] "POST /xmlrpc.php HTTP/1.1" 500 254 "-" "-"
105.227.169.102 - - [11/Aug/2014:17:42:27 -0700] "POST /xmlrpc.php HTTP/1.1" 500 254 "-" "Mozilla/4.0 (compatible; Win32; WinHttp.WinHttpRequest.5)"
67.247.172.24 - - [11/Aug/2014:17:42:27 -0700] "POST /xmlrpc.php HTTP/1.1" 500 254 "-" "-"
24.113.140.97 - - [11/Aug/2014:17:42:27 -0700] "POST /xmlrpc.php HTTP/1.1" 500 254 "-" "-"
39.48.185.1 - - [11/Aug/2014:17:42:27 -0700] "POST /xmlrpc.php HTTP/1.1" 500 254 "-" "-"
85.57.59.84 - - [11/Aug/2014:17:42:27 -0700] "POST /xmlrpc.php HTTP/1.1" 500 254 "-" "Mozilla/4.0 (compatible; Win32; WinHttp.WinHttpRequest.5)"
66.244.102.44 - - [11/Aug/2014:17:42:28 -0700] "GET / HTTP/1.1" 301 1 "-" "Mozilla/5.0 (iPad; CPU OS 7_1_2 like Mac OS X) AppleWebKit/537.51.2 (KHTML, like Gecko) Version/7.0 Mobile/11D257 Safari/9537.53"
46.149.119.105 - - [11/Aug/2014:17:42:28 -0700] "POST /xmlrpc.php HTTP/1.1" 500 254 "-" "-"
201.221.97.82 - - [11/Aug/2014:17:42:28 -0700] "POST /xmlrpc.php HTTP/1.1" 500 254 "-" "-"
201.83.62.218 - - [11/Aug/2014:17:42:28 -0700] "POST /xmlrpc.php HTTP/1.1" 500 254 "-" "-"
46.113.2.51 - - [11/Aug/2014:17:42:28 -0700] "POST /xmlrpc.php HTTP/1.1" 500 254 "-" "-"
180.191.239.111 - - [11/Aug/2014:17:42:28 -0700] "POST /xmlrpc.php HTTP/1.1" 500 254 "-" "-"
100.43.81.134 - - [11/Aug/2014:17:42:28 -0700] "GET /catalog/index.php?osCsid=2b2b8et24f6ofd09l31gl0gbt0 HTTP/1.1" 200 37 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)"
179.52.25.143 - - [11/Aug/2014:17:42:28 -0700] "POST /xmlrpc.php HTTP/1.0" 500 254 "-" "-"
37.48.87.44 - - [11/Aug/2014:17:42:28 -0700] "POST /xmlrpc.php HTTP/1.1" 500 254 "-" "-"

KrazyBob · Unread post by **KrazyBob** » Mon Aug 11, 2014 8:47 pm

One more thing: it launches hundreds of /usr/sbin/httpd at once and the server load goes straight to 200~!

faris · Unread post by **faris** » Tue Aug 12, 2014 5:52 am

Put this in .htaccess in /httpdocs (i.e. root of site under attack)

Code: Select all

RewriteRule ^xmlrpc\.php$ "http\:\/\/0\.0\.0\.0\/" [R=301,L]

This will almost instantly reduce the attack to nothing and prevent reoccurrences.

faris · Unread post by **faris** » Tue Aug 12, 2014 6:10 am

The Rewrite rule above will prevent the attacks from causing problems. But lets look at what's actually happening.

It is normally a botnet behind this type of attack, so you'll see loads and loads of IPs trying to access xmlrpc.php

With xmlrpc.php in place, apache has to process the request, which involves running some php code, and with quite a few of them going on at once, load can get high. Even if you limit maxclients in httpd.conf you can still end up with high load, or apache can reject requests once it hits maxclients and you end up with websites being unavailable.

With xmlrpc.php removed (or renamed), you'd expect load to return to normal, but it isn't in your case. I can't say for sure why this is, but it may be to do with Wordpress doings its thing -- running plenty of php code to either return a wordpress-generated 404 or find a page that's at least similar to whatever was requested and return a 200.

All I can say for sure is that by adding the Rewrite rule the problem goes away -- in this situation all apache has to do is send back a 301 redirect. Most of the load is then on the connecting attacker (if it follows the redirect).

All this is mostly guesswork - I could be very wrong.

KrazyBob · Unread post by **KrazyBob** » Tue Aug 12, 2014 6:32 am

ALL WordPress sites are under attack at once. I located them by using

Code: Select all


cd /var/www/vhosts

find . -name "xmlrpc.php"

Of course this printed to screen the sites running WordPress. One by one I renamed xmlrpc.php to something different. As you noted this did not stop the problem and made clear that something was calling httpd first and then attempting to POST.

What I did next was I added the httpd stop and start commands toi buttons in my ssh client. I then allowed the load to jump up to about 10 and then I ran

Code: Select all

netstat -nap | sort

THEN I stopped httpd. This allowed me to capture a large set of IP's being used for the attack. Using IPTables I blocked them and stopped the problem.

Reading further about this well known problem I was surprised to read about WordPress complaining that they fixed this issue in WP 3.5+ (we have an auto-installer 3.9.1) but for lesser versions they simply told the Admin to turn it off manually. Most Admin's just install WordPress and haven't got a clue. I wrote the same message on their forum and not a single reply.

I wonder if the Atomic Security has a rule for this.

faris · Unread post by **faris** » Tue Aug 12, 2014 6:44 am

https://www.atomicorp.com/forum/viewtop ... f=3&t=7757

Unread post by **mikeshinn** » Tue Aug 12, 2014 5:41 pm

One more thing: it launches hundreds of /usr/sbin/httpd at once and the server load goes straight to 200~!

If thats happening, then you'll want to check your Apache settings, they sound like they are too high for your system:

Examples:

StartServers 8
MinSpareServers 5
MaxSpareServers 20
ServerLimit 256
MaxClients 256
MaxRequestsPerChild 4000

StartServers 4
MaxClients 300
MinSpareThreads 25
MaxSpareThreads 75
ThreadsPerChild 25
MaxRequestsPerChild 0

KrazyBob · Unread post by **KrazyBob** » Tue Aug 12, 2014 6:48 pm

Thanks for jumping in, Mike. Here are my current settings and I may have unintentionally inflated the number when I said "hundreds". I used an application that "tunes up" httpd.conf:

Code: Select all

<IfModule prefork.c>
StartServers      2
MinSpareServers   2
MaxSpareServers   5
ServerLimit      200
MaxClients       200
MaxRequestsPerChild  4000
</IfModule>

<IfModule worker.c>
StartServers         2
MaxClients         150
MinSpareThreads     25
MaxSpareThreads     75
ThreadsPerChild     25
MaxRequestsPerChild  0
</IfModule>

I may need to try your settings. This server is a dual processor / dual core AMD 275 with 8GB of RAM. It sits on a VPS as do all of my servers. This allows easy back-ups to a centralized server(s), as well as the ability to bounce or migrate the server in the event of a failure. I may need to look at adjusting the VPS parameters ti increase the number of inodes. At the moment it says that I am only using 11% of the allocated inodes. As you know there are many interactive settings on a VPS that will allow the load to run up. But what I saw was hundreds of httpd calls combined with I am assuming is an equal number of calls to POST xmlrpc.php

Atomicorp

UNDER ATTACK

UNDER ATTACK

Re: UNDER ATTACK

Re: UNDER ATTACK

Re: UNDER ATTACK

Re: UNDER ATTACK

Re: UNDER ATTACK

Re: UNDER ATTACK

Re: UNDER ATTACK