Plesk attacks

[faris]

All our Plesk installations appear to be under some form of attack. We are seeing unusually high instances of
/usr/bin/sw-engine-cgi [...stuff] running, which is causing relatively high load. It is also causing our Nagios installation to report an unusually high number of processes.

My guess is that this is being caused by a brute-force password-guessing attack, although I don't know for certain. I guess it could be an attempt to get some info via the heartbleed vulnerability.

Unfortunately ASL isn't doing anything about it. This may be because there's nothing in any logs to show what's really going on, so nothing for ASL to look for.

Is there anything we can do other than block port 8443, which would not be a good thing to do!

All I see in /usr/local/psa/admin/logs/httpsd_access_log is many scores of these (no IP):

Code: Select all

- - - [11/Aug/2014:01:17:08 +0100] "POST /login_up.php3 HTTP/1.1" 200 5149 "-" "-"

(there are also a few returning code 500)


var/log/sw-cp-server/error_log has a lot of this type of thing:

Code: Select all

2014-08-11 01:07:02: (mod_fastcgi.c.2873) backend is overloaded, we disable it for a 2 seconds and send the request to another backend instead: reconnects: 5 load: 187

[faris]

OK. This is interesting. The logs above, with no IP, are from an 10.4.4 installation.
On an 11.5 installation the offending IP is shown:

Code: Select all

62.193.229.100 - - [11/Aug/2014:00:55:15 +0100] "POST /login_up.php3 HTTP/1.1" 200 5027 "-" "-" "-"'/login_up.php3' '' '/usr/local/psa/admin/htdocs'
62.193.229.100 - - [11/Aug/2014:00:55:15 +0100] "POST /login_up.php3 HTTP/1.1" 200 5027 "-" "-" "-"'/login_up.php3' '' '/usr/local/psa/admin/htdocs'
62.193.229.100 - - [11/Aug/2014:00:55:15 +0100] "POST /login_up.php3 HTTP/1.1" 200 5027 "-" "-" "-"'/login_up.php3' '' '/usr/local/psa/admin/htdocs'
62.193.229.100 - - [11/Aug/2014:00:55:15 +0100] "POST /login_up.php3 HTTP/1.1" 200 5027 "-" "-" "-"'/login_up.php3' '' '/usr/local/psa/admin/htdocs'
62.193.229.100 - - [11/Aug/2014:00:55:15 +0100] "POST /login_up.php3 HTTP/1.1" 200 5027 "-" "-" "-"'/login_up.php3' '' '/usr/local/psa/admin/htdocs'
62.193.229.100 - - [11/Aug/2014:00:55:15 +0100] "POST /login_up.php3 HTTP/1.1" 200 5027 "-" "-" "-"'/login_up.php3' '' '/usr/local/psa/admin/htdocs'
62.193.229.100 - - [11/Aug/2014:00:55:16 +0100] "POST /login_up.php3 HTTP/1.1" 200 5027 "-" "-" "-"'/login_up.php3' '' '/usr/local/psa/admin/htdocs'
62.193.229.100 - - [11/Aug/2014:00:55:16 +0100] "POST /login_up.php3 HTTP/1.1" 200 5027 "-" "-" "-"'/login_up.php3' '' '/usr/local/psa/admin/htdocs'
62.193.229.100 - - [11/Aug/2014:00:55:16 +0100] "POST /login_up.php3 HTTP/1.1" 200 5027 "-" "-" "-"'/login_up.php3' '' '/usr/local/psa/admin/htdocs'
62.193.229.100 - - [11/Aug/2014:00:55:16 +0100] "POST /login_up.php3 HTTP/1.1" 200 5027 "-" "-" "-"'/login_up.php3' '' '/usr/local/psa/admin/htdocs'

(I checked, and this IP was connecting to all our Plesk installations on all IPs, all at the same time. I've blocked it via our edge firewall)

Can someone with a Plesk 10.4.4 installation please check to see if IPs are being logged in your
/usr/local/psa/admin/logs/httpsd_access_log

These Plesk installations are running under Virtuozzo. I'm just wondering if 10.4.4 may act differently under Virtuozzo compared to normal, while 11.5 is happy and still logs IPs?

*** I note, however, that ASL didn't take any action even on the 11.5 installation where the IPs were logged.
Is there something special that needs to be enabled?

[hostingg]

did you configure asls web firewall to protect the plesk port 8443?

[faris]

Hmm...I didn't think the WAF had much to do with this type of attack. I thought the WAF is there to do mod_sec-type stuff - block "bad" or dangerous requests and detect specific types of request and so forth, and also to add clamav scanning to uploads and so on.

I thought that reading logfiles is a job for ossec.

Unfortunately there's no "login failed" for it to act on. Just loads of these connection entries.

What we need is for the WAF to somehow read the entire conversation, including the response, and to act if it detects failed logins in the response. Can it do this? Or maybe the question should be does it already do this?

[mikeshinn]

Good question. So yes, the WAF is used to detect login failures.

It does this by reading the actual output from the web application. This lets us support really anything, and allows ASL to both immediately and accurate detect login failures and to not make mistakes.

This is based on experience with this kind of setup. Using a simpler method like say counting the number of attempts and assuming its brute force with generate false positives, and wont work well for slower attacks or cases where lots of users are coming from the same IP.

Looking at the output from the web aplication does scale well because you can detect the difference between say a bunch of users all behind the same IP and an actual login failure. We can even do neat little tricks with cookies to pick out the exact user from that IP and just block that user, or even just lock out a users account as opposed to an IP.

And you since you know its an actual login failure, you can then apply logic to the number of failures to block all kinds of combinations of attacks, from really slow (which counting requests alone is guaranteed to generate false positives), to really fast and everything in between.

Plus, as you noticed, Plesk doesnt log the IP in older versions of Plesk anyway, so unless you are using 12 it wouldnt matter anyway.

[faris]

And there are specific rules for Plesk login failures already? Can you point me to the definition files please?

One other thing -- In ASL 3.x, enabling the WAF on 8443 made it impossible to install SSL certificates, and caused a few other issues in Plesk. Is this dealt with in ASL 4.x?

And does it now work on all IP addresses and not just one (which made it ineffective in 3.x on a system with more than one IP)?

[mikeshinn]

377313 logs plesk login failures, and then ASL takes it from there.

[faris]

Yeah, well it still doesn't cover all IPs on the server, and on one server (but not another!) it still kills the installation of SSL certificates.

This kind of makes it useless for anyone with more than one IP

And enabling/disabling it also seems to add a duplicate "allow all related/established" entry to the firewall.

We need to do something about this.

Unfortunately I skipped the last minor ASL update. We're still on 4.0.5-15. I was waiting for this week's larger one with the less disk intensive ossec file change scanning, so I'm guessing there's no point opening a case until that's out and has been installed?

[mikeshinn]

Yeah, well it still doesn't cover all IPs on the server, and on one server (but not another!) it still kills the installation of SSL certificates.

Plesk 12 is still using @s incorrectly in URLs on your box? Any chance I can log in to see what its doing this time? I wasnt able to cause it to do that on the 12 boxes we have here, perhaps I'm missing something you're doing or maybe we have a different build?

[faris]

Nah, I'm on 10.4.4

Surely it must be possible to take account of this odd use of @ on ASL's built-in Plesk 8443 proxy?

But irrespective of that, the real problem is covering just one IP. That's got to be easy to fix?
0.0.0.0 instead of a specific IP in the nat prerouting rule? It was on the todo list ages ago.

[scott]

Not exactly, its a reserved character in the HTTP protocol. It deliniates authentication in the URI.

[faris]

Well, if it isn't easy then given that it works with 12.x I'm not going to moan.

But please can we get the IP thing sorted?