Centrally logging more than syslog

[faris]

I've lost the ability to use one of the most important tools in my admin arsenal - multitail - so now I'm looking for an alternative solution.

I need some way to view mail, apache etc logs from multiple systems in one place.

I'm aware that rsyslog (and indeed syslog) can pipe to a remote location and in the past I've actually had this working reasonably well.

But what about logs that don't get written via syslog? I'm thinking of debug logs, such as /var/qscan/qmail-queue.log as an example.

I'd also like to pipe Plesk logs, such as the admin access_log and error_logs but I don't think those go via syslog?

Does anyone have any hints or tips on some cool utilities that can help me with all this? At one time I did look at logly but in this case I'm really thinking in terms of things that run locally as opposed to on someone else's systems.

[faris]

Actually there's more to central logging than meets the eye.

For example /var/log/httpd/error_log is not created via syslog.
You need to change ErrorLog in httpd.conf to syslog:local1 and then edit /etc/rsyslog.conf to add local1.* -/var/log/httpd/error_log

But more than that, if the central logging server is an ASL-protected system then ossec is going to read the logs and alert for error conditions from the remote servers. It will also shun IPs that aren't actually doing anything bad on the logging server. I don't think this is a problem, but it is worth keeping in mind.

In addition, if you have the SpamDyke control panel for Plesk installed, you obviously get data from all remote servers logged in the maillog and so it can see all of them. Quite handy actually!

I did find a perl script that can be used to sort of convert any application to log to syslog, but I'm not happy with it just at the moment.

I also think I'm going to have to configure the central syslog server to optionally log all non-local stuff to different logs. I know it is easy with rsyslog but I just need to figure it out.

All very interesting!

[faris]

Just to add to my commentary....

To get mysql to use syslog you have to comment out log-file line completely and just add

syslog

in its place, under [safe_mysql].

mysql is hard-wired to use daemon.err as the facility.priority, so you'll need to add

daemon.err -/var/log/mysqld.log to get the log entries to go to where they were going previously.

NOTE: This is only supported in 5.1.20 onwards as far as I can tell. It isn't supported in earlier versions.


Regarding splitting up received syslog messages on the central logging server into separate logfiles, this is apparently easy to do in rsyslog: http://www.rsyslog.com/article60/

Further info to follow!

[faris]

Wow. What fun this is!

OK, on the central logging server, to automatically create separate logfiles for each remote server sending info to it, add this to rsyslog.conf BELOW the entries in the GLOBAL DIRECTIVES section and ABOVE the RULES section in /etc/rsyslog.conf

Code: Select all

$template DynFile,"/var/log/remotersyslog/%HOSTNAME%/%programname%.log"
if $fromhost-ip != '127.0.0.1' then -?DynFile
:fromhost-ip, !contains, "127.0.0.1" ~

Some explanation is required, and there are some other ways you might want to do this:
Using %programname% in line 1 splits the logs into interesting bits, such as spamdyke.log spamd.log, exnitd.log and so on. This is how I want it as it gives me more flexibility.

But if you want a more manageable set of logs, change that to %syslogfacility-text% instead. You'll then end up with pretty much the same number of logs you have now, although there will be a few additional ones where things you didn't realise were logging are actually logging but which rsyslog isn't set up to look at. local6.log, for example, contains logs from clamd! Yay!

Line 2 limits the "separate" logging to everything OTHER than 127.0.0.1 = localhost = the central logging server. In this way you don't end up with duplicate logs for the central logging server, one set in /var/log/messages/normal stuff and one set in /var/log/remotersyslog/hostname/stuff.log

This may not be what you want and it is important to understand it. If it isn't what you want just change line 2 to be

Code: Select all

*.*  -?DynFile

The Third line is potentially dangerous. It is stripping out all syslog entries that do not originate from localhost = the central logging server for ALL subsequent rules. ALL OF THEM. Got it? OK. Took me ages to figure that one out

[faris]

And finally....

Now that I've had a fun day for the first time in ages, is someone going to tell me that using syslog for apache and mysql is a bad idea....? I hope not!


Faris.

[scott]

I think it depends on the volume and the utilty, access_log I probably wouldnt bother w/ syslog. error_log is a maybe. BTW you can also use the ossec daemon as a syslog server

[faris]

Right...I'm only looking at error_log. Basically I want one place to see what's happening, mainly with email, but also FTP logs, qmail-scanner logs and apache error logs because they can indicate a problem very easily.

(Also sorry I've not updated one of my cases. Not been having a good time. All well now)