Yum Update - Why logs went back in time and server crashed.

[damon]

Something very strange happened after I ran Yum Update today on a Centos 5 1&1 server with ASL.

I ran the update this afternoon, here are the results (aprox 3:40pm Central time)

Code: Select all

Jul 30 15:42:59 Updated: freetype-2.2.1-21.el5_3.x86_64
Jul 30 15:43:00 Updated: audit-libs-1.7.7-6.el5_3.3.x86_64
Jul 30 15:43:00 Updated: nspr-4.7.4-1.el5_3.1.x86_64
Jul 30 15:43:11 Updated: python-2.4.3-24.el5_3.6.x86_64
Jul 30 15:43:11 Updated: 30:bind-libs-9.3.4-10.P1.el5_3.3.x86_64
Jul 30 15:43:13 Updated: mysql-libs-5.0.83-1.el5.art.x86_64
Jul 30 15:43:14 Updated: nspr-4.7.4-1.el5_3.1.i386
Jul 30 15:43:15 Updated: nss-3.12.3.99.3-1.el5.centos.2.x86_64
Jul 30 15:43:15 Updated: freetype-2.2.1-21.el5_3.i386
Jul 30 15:43:16 Updated: pango-1.14.9-5.el5.centos.x86_64
Jul 30 15:43:17 Updated: libtiff-3.8.2-7.el5_3.4.x86_64
Jul 30 15:43:18 Updated: 1:cups-libs-1.3.7-8.el5_3.6.x86_64
Jul 30 15:43:19 Updated: file-4.17-15.el5_3.1.x86_64
Jul 30 15:43:20 Updated: apr-util-1.2.7-7.el5_3.1.x86_64
Jul 30 15:43:23 Updated: httpd-2.2.3-22.el5.centos.2.x86_64
Jul 30 15:43:26 Updated: samba-common-3.0.33-3.7.el5_3.1.x86_64
Jul 30 15:43:35 Updated: mysql-5.0.83-1.el5.art.x86_64
Jul 30 15:43:40 Updated: libwmf-0.2.8.4-10.2.x86_64
Jul 30 15:43:41 Updated: libicu-3.6-5.11.4.x86_64
Jul 30 15:43:45 Updated: mysql-server-5.0.83-1.el5.art.x86_64
Jul 30 15:43:53 Updated: samba-3.0.33-3.7.el5_3.1.x86_64
Jul 30 15:43:53 Updated: 1:mod_ssl-2.2.3-22.el5.centos.2.x86_64
Jul 30 15:43:58 Updated: 1:cups-1.3.7-8.el5_3.6.x86_64
Jul 30 15:44:00 Updated: 30:bind-9.3.4-10.P1.el5_3.3.x86_64
Jul 30 15:44:00 Updated: 30:bind-utils-9.3.4-10.P1.el5_3.3.x86_64
Jul 30 15:44:01 Updated: audit-libs-python-1.7.7-6.el5_3.3.x86_64
Jul 30 15:44:02 Updated: ipsec-tools-0.6.5-13.el5_3.1.x86_64
Jul 30 15:44:03 Updated: audit-1.7.7-6.el5_3.3.x86_64
Jul 30 15:44:05 Updated: acpid-1.0.4-7.el5_3.1.x86_64
Jul 30 15:44:07 Updated: kexec-tools-1.102pre-56.el5_3.2.x86_64
Jul 30 15:44:09 Updated: 1:autofs-5.0.1-0.rc2.102.el5_3.1.x86_64
Jul 30 15:44:10 Updated: crash-4.0-7.2.3.el5.centos.1.x86_64
Jul 30 15:44:11 Updated: clamav-db-0.95.2-1.el5.art.x86_64
Jul 30 15:44:14 Updated: clamav-0.95.2-1.el5.art.x86_64
Jul 30 15:44:14 Updated: nash-5.1.19.6-44.1.x86_64
Jul 30 15:44:15 Updated: mkinitrd-5.1.19.6-44.1.x86_64
Jul 30 15:44:16 Updated: clamd-0.95.2-1.el5.art.x86_64
Jul 30 15:44:17 Updated: sos-1.7-9.16.el5_3.5.noarch
Jul 30 15:44:21 Updated: tzdata-2009i-2.el5.noarch
Jul 30 15:44:23 Updated: ntp-4.2.2p1-9.el5.centos.2.x86_64
Jul 30 15:44:24 Updated: nss-3.12.3.99.3-1.el5.centos.2.i386
Jul 30 15:44:25 Updated: libtiff-3.8.2-7.el5_3.4.i386
Jul 30 15:44:31 Updated: xulrunner-1.9.0.12-1.el5.x86_64
Jul 30 15:44:32 Updated: 1:cups-libs-1.3.7-8.el5_3.6.i386
Jul 30 15:44:34 Updated: pango-1.14.9-5.el5.centos.i386
Jul 30 15:44:35 Updated: libwmf-0.2.8.4-10.2.i386
Jul 30 15:44:36 Updated: audit-libs-1.7.7-6.el5_3.3.i386
Jul 30 15:44:37 Updated: nss-tools-3.12.3.99.3-1.el5.centos.2.x86_64
Jul 30 15:44:37 Updated: mkinitrd-5.1.19.6-44.1.i386

Then I ran top and checked that all services and websites were functioning normally, everything appeared normal.

Then later during a late night email check I couldn't connect to mail or any websites on the server, couldn't connect via SSH either, or startup the Plesk control panel. So rebooted the server and it came up.

Looking at the log files in /var/log/ I find that the following logs all are now dated 2/23/2006.

dmesg - acpid - denyhosts - mysqld.log - btmp - lastlog - wtmp - secure

and inside the /var/log/httpd/ folder the error and ssl_error_log files are also now dated 2/23/2006

This is true of log files inside the statistics/logs for each domain as well.

Here are the last 25 lines of /var/log/httpd/error_log

Code: Select all

[Thu Jul 30 16:20:47 2009] [warn] RSA server certificate CommonName (CN) `plesk' does NOT match server name!?
[Thu Jul 30 16:20:47 2009] [warn] RSA server certificate CommonName (CN) `plesk' does NOT match server name!?
[Thu Jul 30 16:20:47 2009] [warn] RSA server certificate CommonName (CN) `plesk' does NOT match server name!?
[Thu Jul 30 16:20:47 2009] [warn] Init: SSL server IP/port conflict: atmail.webmail:443 (/etc/httpd/conf.d/zzz_atmail_vhost.conf:48) vs. horde.webmail:443 (/etc/httpd/conf.d/zzz_horde_vhost.conf:41)
[Thu Jul 30 16:20:47 2009] [warn] Init: SSL server IP/port conflict: default-1.2.3.4:443 (/etc/httpd/conf.d/zz010_psa_httpd.conf:104) vs. horde.webmail:443 (/etc/httpd/conf.d/zzz_horde_vhost.conf:41)
[Thu Jul 30 16:20:47 2009] [warn] Init: You should not use name-based virtual hosts in conjunction with SSL!!
[Thu Jul 30 16:20:48 2009] [notice] Apache/2.2.3 (Red Hat) configured -- resuming normal operations
[Thu Jul 30 18:28:29 2009] [error] [client 68.0.61.201] File does not exist: /usr/share/psa-horde/imp/themes/wps_sober/graphics/addressbook_browse.png, referer: http://webmail.example.com/imp/compose.php?actionID=reply&index=490&identity=0&array_index=0&thismailbox=INBOX&uniq=1248996474906
[Thu Jul 30 18:33:29 2009] [error] [client 68.0.61.201] File does not exist: /usr/share/psa-horde/favicon.ico
[Thu Feb 23 12:36:50 2006] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Thu Feb 23 12:36:50 2006] [warn] RSA server certificate CommonName (CN) `plesk' does NOT match server name!?
[Thu Feb 23 12:36:50 2006] [warn] RSA server certificate CommonName (CN) `plesk' does NOT match server name!?
[Thu Feb 23 12:36:50 2006] [warn] RSA server certificate CommonName (CN) `plesk' does NOT match server name!?
[Thu Feb 23 12:36:50 2006] [warn] Init: SSL server IP/port conflict: atmail.webmail:443 (/etc/httpd/conf.d/zzz_atmail_vhost.conf:48) vs. horde.webmail:443 (/etc/httpd/conf.d/zzz_horde_vhost.conf:41)
[Thu Feb 23 12:36:50 2006] [warn] Init: SSL server IP/port conflict: default-1.2.3.4:443 (/etc/httpd/conf.d/zz010_psa_httpd.conf:104) vs. horde.webmail:443 (/etc/httpd/conf.d/zzz_horde_vhost.conf:41)
[Thu Feb 23 12:36:50 2006] [warn] Init: You should not use name-based virtual hosts in conjunction with SSL!!
[Thu Feb 23 12:36:50 2006] [notice] Digest: generating secret for digest authentication ...
[Thu Feb 23 12:36:50 2006] [notice] Digest: done
[Thu Feb 23 12:36:52 2006] [notice] mod_python: Creating 4 session mutexes based on 256 max processes and 0 max threads.
[Thu Feb 23 12:36:52 2006] [warn] RSA server certificate CommonName (CN) `plesk' does NOT match server name!?
[Thu Feb 23 12:36:52 2006] [warn] RSA server certificate CommonName (CN) `plesk' does NOT match server name!?
[Thu Feb 23 12:36:52 2006] [warn] RSA server certificate CommonName (CN) `plesk' does NOT match server name!?
[Thu Feb 23 12:36:52 2006] [warn] Init: SSL server IP/port conflict: atmail.webmail:443 (/etc/httpd/conf.d/zzz_atmail_vhost.conf:48) vs. horde.webmail:443 (/etc/httpd/conf.d/zzz_horde_vhost.conf:41)
[Thu Feb 23 12:36:52 2006] [warn] Init: SSL server IP/port conflict: default-1.2.3.4:443 (/etc/httpd/conf.d/zz010_psa_httpd.conf:104) vs. horde.webmail:443 (/etc/httpd/conf.d/zzz_horde_vhost.conf:41)
[Thu Feb 23 12:36:52 2006] [warn] Init: You should not use name-based virtual hosts in conjunction with SSL!!
[Thu Feb 23 12:36:52 2006] [notice] Apache/2.2.3 (Red Hat) configured -- resuming normal operations

and just for a sample the last lines of the /var/log/clamav/clamd.log

Code: Select all

Thu Jul 30 18:12:39 2009 -> SelfCheck: Database status OK.
Thu Jul 30 18:22:39 2009 -> SelfCheck: Database status OK.
Thu Jul 30 18:32:39 2009 -> SelfCheck: Database status OK.
Thu Jul 30 18:42:39 2009 -> SelfCheck: Database status OK.
Thu Jul 30 18:52:39 2009 -> SelfCheck: Database status OK.
Thu Jul 30 19:02:40 2009 -> SelfCheck: Database status OK.
Thu Feb 23 12:36:24 2006 -> +++ Started at Thu Feb 23 12:36:24 2006
Thu Feb 23 12:36:24 2006 -> clamd daemon 0.95.2 (OS: linux-gnu, ARCH: x86_64, CPU: x86_64)
Thu Feb 23 12:36:24 2006 -> Running as user clamav (UID 112, GID 102)
Thu Feb 23 12:36:24 2006 -> Log file size limit disabled.
Thu Feb 23 12:36:24 2006 -> Reading databases from /var/clamav
Thu Feb 23 12:36:24 2006 -> Not loading PUA signatures.
Thu Feb 23 12:36:31 2006 -> Loaded 1658720 signatures.
Thu Feb 23 12:36:31 2006 -> TCP: Bound to address 127.0.0.1 on port 3310
Thu Feb 23 12:36:31 2006 -> TCP: Setting connection queue length to 30
Thu Feb 23 12:36:31 2006 -> LOCAL: Removing stale socket file /tmp/clamd.socket
Thu Feb 23 12:36:31 2006 -> LOCAL: Unix socket file /tmp/clamd.socket
Thu Feb 23 12:36:31 2006 -> LOCAL: Setting connection queue length to 30
Thu Feb 23 12:36:31 2006 -> Limits: Global size limit set to 104857600 bytes.
Thu Feb 23 12:36:31 2006 -> Limits: File size limit set to 26214400 bytes.
Thu Feb 23 12:36:31 2006 -> Limits: Recursion level limit set to 16.
Thu Feb 23 12:36:31 2006 -> Limits: Files limit set to 10000.
Thu Feb 23 12:36:31 2006 -> Archive support enabled.
Thu Feb 23 12:36:31 2006 -> Algorithmic detection enabled.
Thu Feb 23 12:36:31 2006 -> Portable Executable support enabled.
Thu Feb 23 12:36:31 2006 -> ELF support enabled.
Thu Feb 23 12:36:31 2006 -> Detection of broken executables enabled.
Thu Feb 23 12:36:31 2006 -> Mail files support enabled.
Thu Feb 23 12:36:31 2006 -> OLE2 support enabled.
Thu Feb 23 12:36:31 2006 -> PDF support enabled.
Thu Feb 23 12:36:31 2006 -> HTML support enabled.
Thu Feb 23 12:36:31 2006 -> Self checking every 600 seconds.
Thu Feb 23 12:46:31 2006 -> No stats for Database check - forcing reload
Thu Feb 23 12:46:31 2006 -> Reading databases from /var/clamav
Thu Feb 23 12:46:36 2006 -> Database correctly reloaded (1658720 signatures)
Thu Feb 23 12:56:36 2006 -> SelfCheck: Database status OK.

I watched the log files and it was almost an hour after the server reboot before new entries (with the correct date/time) began appearing in the various logs.

What happened? I'd like to understand what went wrong, it seems like every time I do a general Yum Update instead of a specific package I have trouble.

Thanks for any pointers,

Damon

[mneese]

were you aware that you were updating to Plesk 9.2.2 which was available yesterday? I did the update and things have been really smooth.
Did you have a backup of the server?

[faris]

You are on EL5 and you updated python.

For some unknown reason this wanted to bring with it a xen kernel on our machines, and a PAE one on another machine (well, it was kernel-xen ... maybe headers. Or support files. Don't know. Did annoy me. Didn't upgrade it until I had more time to think and look)

Do a uname -a just in case?

Faris.

[damon]

were you aware that you were updating to Plesk 9.2.2 which was available yesterday? I did the update and things have been really smooth.
Did you have a backup of the server?

I'm still on 9.2.1, didn't notice till you mentioned it that there is 9.2.2 available.

Are the atomic repos ok for 9.2.2? Sounds like a small bug fix.

Maybe it will fix my ongoing problem. Ever since the problem above Spamassassin shows running, but it's not checking email, getting a lot of spam, mostly from forwards that go to spamassassin protected mailboxes.

I did take a full backup using the plesk panel backup off to the ftp space before I did the updates.

Damon

[damon]

You are on EL5 and you updated python.
For some unknown reason this wanted to bring with it a xen kernel on our machines, and a PAE one on another machine (well, it was kernel-xen ... maybe headers. Or support files. Don't know. Did annoy me. Didn't upgrade it until I had more time to think and look)
Do a uname -a just in case?
Faris.

I've got kernel in the yum excludes.

Code: Select all

exclude=kernel* sendmail bind-chroot caching-nameserver

Code: Select all

uname -a
Linux example.com 2.6.27.7-9.art.x86_64 #1 SMP Fri Dec 19 11:42:37 EST 2008 x86_64 x86_64 x86_64 GNU/Linux

Wondering if I should do the Plesk update to 9.2.2

[mneese]

I have updated and things went very smoothly. Looking at your file updates again...everything updated except the psa updates for 9.2.2 (if that's the complete list). I didn't look that closely the first time around, so just assumed that was the 9.2.2 update file list.

So, maybe you updated just before the psa files were available, but the supporting cast was available. I would say you have half of the updates already done, might as well go ahead. I did mine with yum update and no problems at all, in fact I think things are much more stable.

[damon]

I have updated and things went very smoothly.

I went ahead and did the 9.2.2 update via the Plesk Control Panel updater and it went well. Fixed the previous problem with Spamassassin not checking emails event though it was running.

Still don't understand why the logs had entries from 2006 after the first yum updates.

[scott]

maybe the bios battery is dead?