SSD is being killed by ASL
SSD is being killed by ASL
Starting sshd: /etc/init.d/sshd: line 128: 3318 Killed $SSHD $OPTIONS
[FAILED]
I have emails every minute where SSHD is trying to restart. I had a number of entries in the event logs for
Rules
60038 Process Monitor: Failed to spawn service
61027 Denied a RWX mprotect event. An application just attmpted to use the mprotect function to bypass memory protection functions in the kernel.
61028 Denied an untrusted non system library binary from hooking an application.
I disabled all these rules and still can't get it to start
Joe
[FAILED]
I have emails every minute where SSHD is trying to restart. I had a number of entries in the event logs for
Rules
60038 Process Monitor: Failed to spawn service
61027 Denied a RWX mprotect event. An application just attmpted to use the mprotect function to bypass memory protection functions in the kernel.
61028 Denied an untrusted non system library binary from hooking an application.
I disabled all these rules and still can't get it to start
Joe
Re: SSD is being killed by ASL
did you read those articles? they explain what to do
If everything was easy, then the world wouldn't need engineers.
- mikeshinn
- Atomicorp Staff - Site Admin
- Posts: 4152
- Joined: Thu Feb 07, 2008 7:49 pm
- Location: Chantilly, VA
Re: SSD is being killed by ASL
If SSHD is triggering this rule:
https://www.atomicorp.com/wiki/index.php/HIDS_61027
Then its either been replaced by a backdoored version, or someone has horribly misconfigured it so that its trying to do something very dangerous on your system. Either way, SSH never ever does this otherwise, and does not need to do this. This will only happen if your system has been either compromised, or someone has done something very very wrong to sshd. Either way, its bad.
Whats the exact event log message on your system for 60127, for example you'll see something like this:
May 5 09:24:02 host kernel: grsec: From 1.2.3.4: denied RWX mprotect of /lib64/ld-2.12.so by /usr/sbin/sshd[sshd:3705] uid/euid:0/0 gid/egid:0/0, parent /usr/sbin/sshd[sshd:3642] uid/euid:0/0 gid/egid:0/0
Please post the log message so we can see whats happening on your system. Also, thankfully disabling that rule will not disable that protection, it just tells ASL to not alert you that your system is trying to be compromised and ASL is preventing the compromise of your system.
https://www.atomicorp.com/wiki/index.php/HIDS_61027
Then its either been replaced by a backdoored version, or someone has horribly misconfigured it so that its trying to do something very dangerous on your system. Either way, SSH never ever does this otherwise, and does not need to do this. This will only happen if your system has been either compromised, or someone has done something very very wrong to sshd. Either way, its bad.
Whats the exact event log message on your system for 60127, for example you'll see something like this:
May 5 09:24:02 host kernel: grsec: From 1.2.3.4: denied RWX mprotect of /lib64/ld-2.12.so by /usr/sbin/sshd[sshd:3705] uid/euid:0/0 gid/egid:0/0, parent /usr/sbin/sshd[sshd:3642] uid/euid:0/0 gid/egid:0/0
Please post the log message so we can see whats happening on your system. Also, thankfully disabling that rule will not disable that protection, it just tells ASL to not alert you that your system is trying to be compromised and ASL is preventing the compromise of your system.
Michael Shinn
Atomicorp - Security For Everyone
Atomicorp - Security For Everyone
Re: SSD is being killed by ASL
This is what I see in the event log
srv01 kernel: grsec: denied RWX mprotect of /usr/sbin/sshd by /usr/sbin/sshd[sshd:19653] uid/euid:0/0 gid/egid:0/0, parent /etc/rc.d/init.d/sshd[sshd:19639] uid/euid:0/0 gid/egid:0/0
This was the log for 61028
srv01 kernel: grsec: denied exec of usermode helper binary /usr/libexec/abrt-hook-ccpp located outside of /sbin and system library paths
srv01 kernel: grsec: denied RWX mprotect of /usr/sbin/sshd by /usr/sbin/sshd[sshd:19653] uid/euid:0/0 gid/egid:0/0, parent /etc/rc.d/init.d/sshd[sshd:19639] uid/euid:0/0 gid/egid:0/0
This was the log for 61028
srv01 kernel: grsec: denied exec of usermode helper binary /usr/libexec/abrt-hook-ccpp located outside of /sbin and system library paths
- mikeshinn
- Atomicorp Staff - Site Admin
- Posts: 4152
- Joined: Thu Feb 07, 2008 7:49 pm
- Location: Chantilly, VA
Re: SSD is being killed by ASL
SSHD shouldnt be doing that and definitely doesnt need to do that. ASL is definitely protecting you from something bad. Either someone has replaced sshd with a backdoored version or someone has seriously misconfigured sshd on your system. Either way, do not allow this. Your system is either compromised, or is about to be compromised.
The first thing I would do is check the file integrity watches in ASL to see when that file was changed. If this just started to happen, then you know it was very recent.
If the files integrity is valid, that is its not been replaced, then someone modified the execstack settings on sshd to allow it to this this dangerous operation it doesnt need to do. You'll need to remove that, however you really need to confirm that sshd hasnt been replaced before you do that. I definitely have seen backdoored versions of sshd do this.
The first thing I would do is check the file integrity watches in ASL to see when that file was changed. If this just started to happen, then you know it was very recent.
If the files integrity is valid, that is its not been replaced, then someone modified the execstack settings on sshd to allow it to this this dangerous operation it doesnt need to do. You'll need to remove that, however you really need to confirm that sshd hasnt been replaced before you do that. I definitely have seen backdoored versions of sshd do this.
Michael Shinn
Atomicorp - Security For Everyone
Atomicorp - Security For Everyone
Re: SSD is being killed by ASL
hmm ok so not sure of an appropriate approach to fix this at this point
Re: SSD is being killed by ASL
I haven't changed SSHD in a couple years the only thing I did was change the port and this problem just started this past week so something is wrong.
Joe
Joe
- mikeshinn
- Atomicorp Staff - Site Admin
- Posts: 4152
- Joined: Thu Feb 07, 2008 7:49 pm
- Location: Chantilly, VA
Re: SSD is being killed by ASL
Step 1: confirm it hasnt been replaced by someone else
Step 2: if someone set RWX mprotect on sshd this can happen as well, but dont assume it was that. Thats a weird thing for someone to do, but people do it all the time on other things like PHP thinking they need to. So its not impossible, but very strange for someone to do that. Definitely start with step 1, I've seen this happen with backdoored versions of SSH (probably because the bad guys thought they needed to do this as well)
Step 2: if someone set RWX mprotect on sshd this can happen as well, but dont assume it was that. Thats a weird thing for someone to do, but people do it all the time on other things like PHP thinking they need to. So its not impossible, but very strange for someone to do that. Definitely start with step 1, I've seen this happen with backdoored versions of SSH (probably because the bad guys thought they needed to do this as well)
Michael Shinn
Atomicorp - Security For Everyone
Atomicorp - Security For Everyone
Re: SSD is being killed by ASL
I am the only person that monitors and updates this server and I haven't changed anything recently. Looks like I will have to make a trip to the DataCenter
- mikeshinn
- Atomicorp Staff - Site Admin
- Posts: 4152
- Joined: Thu Feb 07, 2008 7:49 pm
- Location: Chantilly, VA
Re: SSD is being killed by ASL
Looks like you have KVM access, so before you do that check the file integrity reports from ASL to see if it reported any changes to SSHD. Also, check to see who else has logged into the system, perhaps the bad guys stole credentials to the system and logged in as root. The real time file integrity checks will have a record of any changes to /usr/sbin/sshd provided the defaults were left in place for the file integrity system.
Michael Shinn
Atomicorp - Security For Everyone
Atomicorp - Security For Everyone
Re: SSD is being killed by ASL
NO KVM access and I can't SSH into it. I just checked the file integrity and don't see anything pertaining to SSHD. I am logged into my WHM interface
- mikeshinn
- Atomicorp Staff - Site Admin
- Posts: 4152
- Joined: Thu Feb 07, 2008 7:49 pm
- Location: Chantilly, VA
Re: SSD is being killed by ASL
What do you see in the ASL file integrity reports inside the ASL web console?
Michael Shinn
Atomicorp - Security For Everyone
Atomicorp - Security For Everyone
Re: SSD is being killed by ASL
I was having a different problem with ssh and uninstalled and reinstalled via yum and it solved my issues
https://www.atomicorp.com/forum/viewtop ... f=3&t=7915
https://www.atomicorp.com/forum/viewtop ... f=3&t=7915
Re: SSD is being killed by ASL
So I removed SSH and reinstalled it via Cpanel and still having the same issue which does not make sense?
- mikeshinn
- Atomicorp Staff - Site Admin
- Posts: 4152
- Joined: Thu Feb 07, 2008 7:49 pm
- Location: Chantilly, VA
Re: SSD is being killed by ASL
sshd is still trying to smash your stack? If so, then thats not the real sshd, someones modified it or replaced. The real sshd doesnt do that.
Michael Shinn
Atomicorp - Security For Everyone
Atomicorp - Security For Everyone