Unmanagable Container

Community support forums for Virtualization products, such as Xen(TM), VMWare(TM), Virtuozzo(TM), KVM, Qemu, lguest, openvz and others! There is no such thing as a bad question here as long as it pertains to a virtualization technology or product. Newbies feel free to get help getting started or asking questions that may be obvious.
KrazyBob
Forum Regular
Forum Regular
Posts: 310
Joined: Mon Mar 19, 2007 3:47 pm

Unmanagable Container

Unread post by KrazyBob »

As I move along with my migrations to Plesk 12 I keep running into errors elsewhere. Right now I am showing a high CPU on particularly one container. I have searched Google,; I have search the Plesk KB. Nothing that pertains. This is Plesk 9.3 on CentOS 5.x -- being migrated to Centos 6.x and Plesk 12.
7:09pm, up 822 days, 20:23, 3 users, load average: 11.17, 12.34, 13.69
VENum 3, procs 388: R 12, S 376, D 0, Z 0, T 0, X 0
CPU [CRIT]: VEs 100%, VE0 0%, user 94%, sys 6%, idle 0%, lat(ms) 1236/8
Mem [ OK ]: total 3930MB, free 814MB/0MB (low/high), lat(ms) 1/0
ZONE0 (DMA): size 15MB, act 0MB, inact 0MB, free 9MB (0/0/0)
ZONE1 (Normal): size 4079MB, act 2377MB, inact 439MB, free 805MB (7/15/23)
Mem lat (ms): A0 0, K0 0, U0 0, K1 1, U1 0
Slab pages: 208MB/208MB (ino 81MB, de 43MB, bh 3MB, pb 29MB)
Swap [ OK ]: tot 8189MB, free 7850MB, in 0.000MB/s, out 0.000MB/s
Net [ OK ]: tot: in 0.014MB/s 208pkt/s, out 0.510MB/s 378pkt/s
lo: in 0.000MB/s 0pkt/s, out 0.000MB/s 0pkt/s
eth0: in 0.014MB/s 208pkt/s, out 0.510MB/s 378pkt/s
eth1: in 0.000MB/s 0pkt/s, out 0.000MB/s 0pkt/s
Disks [ OK ]: in 0.020MB/s, out 0.990MB/s

VEID ST %VM %KM PROC CPU SOCK FCNT MLAT IP
1 OK 2.4/26 0.1/1.6 0/36/256 0.0/1.2 56/1512 0 0 xx.xx.xx
107 OK 56/153 0.6/10 0/129/10062 2.3/67 93/20124 0 26 xx.xx.xx
108 !! 42/60 1.0/10 12/130/10062 96/32 143/20124 0 1236 xx.xx.xx
The container load runs ~10. I have run vzsplit -n2 on a dual AMD 248 that has run very efficiently for 7 years and I cannot find the culprit. After running security and rootkit it sometimes tells me that:

Code: Select all

cat: error while loading shared libraries: libc.so.6: cannot create shared object descriptor: Cannot allocate memory
Error: Invalid --screen-indent value given: 4   Display line: display --to SCREEN+LOG --type PLAIN --screen-indent 4 --result OK --color GREEN NAME /bin/chown
/usr/local/psa/admin/sbin/modules//watchdog/rkhunter: fork: Cannot allocate memory
Others times it shows as an Internal Server Error 500. Running cat/proc/loadavg or w is an instant segfault. Eventually nothing works.

But the other container on the same hardware node has no issues suggesting not bad memory or CPU issue.
KrazyBob
Forum Regular
Forum Regular
Posts: 310
Joined: Mon Mar 19, 2007 3:47 pm

Re: Unmanagable Container

Unread post by KrazyBob »

I have determined that this container has been hacked. I found one after the other after another of cron entries for .X11... I killed them all but the container is hacked.

http://kb.sp.parallels.com/en/116999 details using:

Code: Select all

vzctl reinstall <VZID>
but experiments in the past weren't so successful. This is a Plesk 9.3 site and EOL so I have no safety umbrella. I have a backup from 11/06 but it will contain the hacks.

Any words of wisdom? I am trying to do for myself and have Googled my fingers sore.
faris
Long Time Forum Regular
Long Time Forum Regular
Posts: 2321
Joined: Thu Dec 09, 2004 11:19 am

Re: Unmanagable Container

Unread post by faris »

I'd run a clamdscan on it to see if it can find some nasty stuff. It might even allow you to identify the site that was/is responsible.

rkhunter may also help you locate other bits of malware.
Maybe also Maldetect? https://www.rfxn.com/projects/linux-malware-detect/

Add apache to /etc/cron.deny

The binaries in the container are not necessarily compromised (if you are lucky). It could be that something like the malware kit that was used most often by the people who exploited the Plesk vulnerability from a couple of years or so ago has been installed. That's relatively easy to remove, I think? Parallels even released a script that did so automatically at any rate.

Note that the vzctl reinstall command is only used for Standard templates, not EZ Templates, according to http://kb.sp.parallels.com/en/1012 (I didn't know that!)

So maybe concentrate on finding the malware.

I found these .. not sure if they are relevant:
http://security.stackexchange.com/quest ... ing-hacked

http://forum.sp.parallels.com/threads/s ... sk.290730/
[ ** see http://forum.sp.parallels.com/threads/s ... ost-706610 and afterwards in particular ** ]

Then when you think you have found the most likely culprits, try a Migration to Plesk 12 on a fresh container?
I would reset all the Plesk/ftp/email/system passwords as well.

And basically suspect Wordpress/Joomla etc sites in particular.
--------------------------------
<advert>
If you want to rent a UK-based VPS that comes with friendly advice and support from a fellow ART fan, please get in touch.
</advert>
KrazyBob
Forum Regular
Forum Regular
Posts: 310
Joined: Mon Mar 19, 2007 3:47 pm

Re: Unmanagable Container

Unread post by KrazyBob »

I've definitely been hacked. http://kb.sp.parallels.com/en/116999. I am glad that you told me that EZ-Templates can 't be reinstalled. I am trying to finish up migrations to new dual Intel XEON servers and this stuff is in my way. I found /tmp/.X11 in multiple cron's so I don't think that any site has been hacked. But I don't know how they got in. The kernel is current and I am just moving to Plesk 12 and prefer a clean install on new HDD's. The stuff that I get into. Rkhunter was useless. It found nothing. I had to search all of my internal KB articles and tried everything I've learned the hard way. I save all of the answers you guys help me with.
faris
Long Time Forum Regular
Long Time Forum Regular
Posts: 2321
Joined: Thu Dec 09, 2004 11:19 am

Re: Unmanagable Container

Unread post by faris »

The KB I referred to is the only place I've ever seen where it states that "reinstall" can only be used for Standard templates. It may not be correct.
--------------------------------
<advert>
If you want to rent a UK-based VPS that comes with friendly advice and support from a fellow ART fan, please get in touch.
</advert>
KrazyBob
Forum Regular
Forum Regular
Posts: 310
Joined: Mon Mar 19, 2007 3:47 pm

Re: Unmanagable Container

Unread post by KrazyBob »

I will check deeper into it. Right now I have a CRON restarting the container once every 15 minutes. I am not a bash programmer but I wish that I could write a script that would detect segfault and restart the container.
Post Reply