Page 1 of 1
ASL new install - Hyper V problem - hv_kvp_daemon
Posted: Tue Oct 15, 2013 8:56 am
by bristaruk
Hey,
I've just gone through a complete server build and update of a Centos 6, PHP 5.4 system running on a Hyper-V controlled virtual machine. Everything was working perfectly until I installed Atomicorp Secured Linux along with its ASL kernel.
The problem that's occurred is that the hv_kvp_daemon is now maxed out at near 100% CPU load (depending on other processes running) taking all available CPU resources.
Is this a problem that anyone has come across before, or know the best way to solve?
Re: ASL new install - Hyper V problem - hv_kvp_daemon
Posted: Tue Oct 15, 2013 9:42 am
by scott
As far as I know, you're the only person actually using Hyper V. Its not a platform we support, but Id certainly love to know more about your configuration. The fact that it booted at all is confirmation of some (untested) changes we made in the 3.2 branch.
The officially supported virtualization are:
https://www.atomicorp.com/wiki/index.ph ... tion_Notes
Re: ASL new install - Hyper V problem - hv_kvp_daemon
Posted: Tue Oct 15, 2013 10:03 am
by bristaruk
Well all works perfectly (so far), and ASL 3.2.14-31.el6.art installed quite easily on a Centos 6.4 VM also running Plesk 11.5, bar this one small problem with a process that I can easily stop...
I just need to stop it manually after each boot unless I never allow it to start but not sure what issues it may cause as it's used to pass config information to/from the host and guest.
Day one of your 30 day trial looks ok so far, just need to see what issues not running hv_kvp_daemon may cause...
Re: ASL new install - Hyper V problem - hv_kvp_daemon
Posted: Tue Oct 15, 2013 12:35 pm
by mikeshinn
Is it that this daemon wont start, or it wont stop?
Re: ASL new install - Hyper V problem - hv_kvp_daemon
Posted: Tue Oct 15, 2013 1:11 pm
by bristaruk
mikeshinn wrote:Is it that this daemon wont start, or it wont stop?
when ASL starts running the daemon begins taking up 100% of CPU resources unless you stop it (am only running 2 cores on this Virtual Machine)
Re: ASL new install - Hyper V problem - hv_kvp_daemon
Posted: Tue Oct 15, 2013 2:03 pm
by mikeshinn
Wow, if you run an strace on this daemon whats it doing?
And is anything logged, perhaps its generating some error condition in the system logs or its logs?
Re: ASL new install - Hyper V problem - hv_kvp_daemon
Posted: Tue Oct 15, 2013 4:00 pm
by bristaruk
mikeshinn wrote:Wow, if you run an strace on this daemon whats it doing?
And is anything logged, perhaps its generating some error condition in the system logs or its logs?
can't see any errors in the logs but
strace hv_kvp_daemon gives this:
Code: Select all
# strace hv_kvp_daemon
execve("/usr/sbin/hv_kvp_daemon", ["hv_kvp_daemon"], [/* 24 vars */]) = 0
brk(0) = 0x2827110
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2d1d1b98000
access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=54588, ...}) = 0
mmap(NULL, 54588, PROT_READ, MAP_PRIVATE, 3, 0) = 0x2d1d1b8a000
close(3) = 0
open("/lib64/libc.so.6", O_RDONLY) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\360\355A^<\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=1922152, ...}) = 0
mmap(0x3c5e400000, 3745960, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x2d1d15e8000
mprotect(0x2d1d1772000, 2093056, PROT_NONE) = 0
mmap(0x2d1d1971000, 20480, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x189000) = 0x2d1d1971000
mmap(0x2d1d1976000, 18600, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x2d1d1976000
close(3) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2d1d1b89000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2d1d1b88000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2d1d1b87000
arch_prctl(ARCH_SET_FS, 0x2d1d1b88700) = 0
mprotect(0x2d1d1971000, 16384, PROT_READ) = 0
mprotect(0x2d1d1b9a000, 4096, PROT_READ) = 0
munmap(0x2d1d1b8a000, 54588) = 0
clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x2d1d1b889d0) = 53511
exit_group(0)
I'm still only learning with linux but looks to me like I'm missing etc/ld.so.preload can anyone spot anything else?
Re: ASL new install - Hyper V problem - hv_kvp_daemon
Posted: Tue Oct 15, 2013 6:34 pm
by bristaruk
just found a new problem in that the server has been behaving all day and now ossec-analysisd is now taking up 75% of CPU and the asl webclient will no longer load probably as the CPU is too busy..
This is on an empty server with no load.
Re: ASL new install - Hyper V problem - hv_kvp_daemon
Posted: Wed Oct 16, 2013 12:53 pm
by mikeshinn
So looking at that strace, it looks like the daemon needs to use the insecure mprotect() call. You can enable it for the daemon by following this FAQ:
https://www.atomicorp.com/wiki/index.ph ... _denied.29
As for ossec-analysisd, if its got a high sustained load then that means its doing actual work in the form of log analysis. If load is really sustained over time, then it means its getting fed a LOT of logs. What this look like on your system:
grep location /var/ossec/etc/ossec.conf
Thats a list of the log files its monitoring (ignore the "local" entries from that grep, they do something else). Are any of those log files being inundated with messages? For example, a sustained stream of errors from a mail functioning application? If you arent sure, you can always send us the logs and we can tell if thats the case here.