SSD is being killed by ASL

Customer support forums for Atomic Protector (formerly Atomic Secured Linux). There is no such thing as a bad question here as long as it pertains to using Atomic Protector. Newbies feel free to get help getting started or asking questions that may be obvious. Regular users are asked to be gentle. :-)
jbourque
Forum User
Forum User
Posts: 38
Joined: Thu Jul 15, 2010 9:42 am

SSD is being killed by ASL

Unread post by jbourque »

Starting sshd: /etc/init.d/sshd: line 128: 3318 Killed $SSHD $OPTIONS
[FAILED]


I have emails every minute where SSHD is trying to restart. I had a number of entries in the event logs for

Rules
60038 Process Monitor: Failed to spawn service
61027 Denied a RWX mprotect event. An application just attmpted to use the mprotect function to bypass memory protection functions in the kernel.
61028 Denied an untrusted non system library binary from hooking an application.

I disabled all these rules and still can't get it to start

Joe
User avatar
hostingg
Forum User
Forum User
Posts: 63
Joined: Mon Mar 18, 2013 6:26 pm
Location: Earth

Re: SSD is being killed by ASL

Unread post by hostingg »

did you read those articles? they explain what to do
If everything was easy, then the world wouldn't need engineers.
User avatar
mikeshinn
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin
Posts: 4152
Joined: Thu Feb 07, 2008 7:49 pm
Location: Chantilly, VA

Re: SSD is being killed by ASL

Unread post by mikeshinn »

If SSHD is triggering this rule:

https://www.atomicorp.com/wiki/index.php/HIDS_61027

Then its either been replaced by a backdoored version, or someone has horribly misconfigured it so that its trying to do something very dangerous on your system. Either way, SSH never ever does this otherwise, and does not need to do this. This will only happen if your system has been either compromised, or someone has done something very very wrong to sshd. Either way, its bad.

Whats the exact event log message on your system for 60127, for example you'll see something like this:

May 5 09:24:02 host kernel: grsec: From 1.2.3.4: denied RWX mprotect of /lib64/ld-2.12.so by /usr/sbin/sshd[sshd:3705] uid/euid:0/0 gid/egid:0/0, parent /usr/sbin/sshd[sshd:3642] uid/euid:0/0 gid/egid:0/0

Please post the log message so we can see whats happening on your system. Also, thankfully disabling that rule will not disable that protection, it just tells ASL to not alert you that your system is trying to be compromised and ASL is preventing the compromise of your system.
jbourque
Forum User
Forum User
Posts: 38
Joined: Thu Jul 15, 2010 9:42 am

Re: SSD is being killed by ASL

Unread post by jbourque »

This is what I see in the event log

srv01 kernel: grsec: denied RWX mprotect of /usr/sbin/sshd by /usr/sbin/sshd[sshd:19653] uid/euid:0/0 gid/egid:0/0, parent /etc/rc.d/init.d/sshd[sshd:19639] uid/euid:0/0 gid/egid:0/0

This was the log for 61028

srv01 kernel: grsec: denied exec of usermode helper binary /usr/libexec/abrt-hook-ccpp located outside of /sbin and system library paths
User avatar
mikeshinn
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin
Posts: 4152
Joined: Thu Feb 07, 2008 7:49 pm
Location: Chantilly, VA

Re: SSD is being killed by ASL

Unread post by mikeshinn »

SSHD shouldnt be doing that and definitely doesnt need to do that. ASL is definitely protecting you from something bad. Either someone has replaced sshd with a backdoored version or someone has seriously misconfigured sshd on your system. Either way, do not allow this. Your system is either compromised, or is about to be compromised.

The first thing I would do is check the file integrity watches in ASL to see when that file was changed. If this just started to happen, then you know it was very recent.

If the files integrity is valid, that is its not been replaced, then someone modified the execstack settings on sshd to allow it to this this dangerous operation it doesnt need to do. You'll need to remove that, however you really need to confirm that sshd hasnt been replaced before you do that. I definitely have seen backdoored versions of sshd do this.
jbourque
Forum User
Forum User
Posts: 38
Joined: Thu Jul 15, 2010 9:42 am

Re: SSD is being killed by ASL

Unread post by jbourque »

hmm ok so not sure of an appropriate approach to fix this at this point :(
jbourque
Forum User
Forum User
Posts: 38
Joined: Thu Jul 15, 2010 9:42 am

Re: SSD is being killed by ASL

Unread post by jbourque »

I haven't changed SSHD in a couple years the only thing I did was change the port and this problem just started this past week so something is wrong.

Joe
User avatar
mikeshinn
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin
Posts: 4152
Joined: Thu Feb 07, 2008 7:49 pm
Location: Chantilly, VA

Re: SSD is being killed by ASL

Unread post by mikeshinn »

Step 1: confirm it hasnt been replaced by someone else

Step 2: if someone set RWX mprotect on sshd this can happen as well, but dont assume it was that. Thats a weird thing for someone to do, but people do it all the time on other things like PHP thinking they need to. So its not impossible, but very strange for someone to do that. Definitely start with step 1, I've seen this happen with backdoored versions of SSH (probably because the bad guys thought they needed to do this as well)
jbourque
Forum User
Forum User
Posts: 38
Joined: Thu Jul 15, 2010 9:42 am

Re: SSD is being killed by ASL

Unread post by jbourque »

I am the only person that monitors and updates this server and I haven't changed anything recently. Looks like I will have to make a trip to the DataCenter :(
User avatar
mikeshinn
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin
Posts: 4152
Joined: Thu Feb 07, 2008 7:49 pm
Location: Chantilly, VA

Re: SSD is being killed by ASL

Unread post by mikeshinn »

Looks like you have KVM access, so before you do that check the file integrity reports from ASL to see if it reported any changes to SSHD. Also, check to see who else has logged into the system, perhaps the bad guys stole credentials to the system and logged in as root. The real time file integrity checks will have a record of any changes to /usr/sbin/sshd provided the defaults were left in place for the file integrity system.
jbourque
Forum User
Forum User
Posts: 38
Joined: Thu Jul 15, 2010 9:42 am

Re: SSD is being killed by ASL

Unread post by jbourque »

NO KVM access and I can't SSH into it. I just checked the file integrity and don't see anything pertaining to SSHD. I am logged into my WHM interface
User avatar
mikeshinn
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin
Posts: 4152
Joined: Thu Feb 07, 2008 7:49 pm
Location: Chantilly, VA

Re: SSD is being killed by ASL

Unread post by mikeshinn »

What do you see in the ASL file integrity reports inside the ASL web console?
gevensen
Forum User
Forum User
Posts: 14
Joined: Mon Jan 19, 2015 8:12 am
Location: United States

Re: SSD is being killed by ASL

Unread post by gevensen »

I was having a different problem with ssh and uninstalled and reinstalled via yum and it solved my issues

https://www.atomicorp.com/forum/viewtop ... f=3&t=7915
jbourque
Forum User
Forum User
Posts: 38
Joined: Thu Jul 15, 2010 9:42 am

Re: SSD is being killed by ASL

Unread post by jbourque »

So I removed SSH and reinstalled it via Cpanel and still having the same issue which does not make sense?
User avatar
mikeshinn
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin
Posts: 4152
Joined: Thu Feb 07, 2008 7:49 pm
Location: Chantilly, VA

Re: SSD is being killed by ASL

Unread post by mikeshinn »

sshd is still trying to smash your stack? If so, then thats not the real sshd, someones modified it or replaced. The real sshd doesnt do that.
Post Reply