VZMigrate Fails Virtuozzo 3

[KrazyBob]

I am trying to move a large container. The hardware node is being upgraded to Virtuozzo 4 and the failing drives to to larger and faster enterprise drives. At the end of a 24 hour collection and transfer it failed.

Code: Select all

Can't undump: Invalid argument
VE start failedconntrack: cannot find NAT helper, some module is not loaded
conntrack: cannot find helper, some module is not loaded
rst_restore_net: -22

VE is unmounted
VE start failed

Can't move/copy VE#303 -> VE#303, [], [] : can't send command to destination side : Broken pipe
Resuming VE#303 ...
done
You have new mail in /var/spool/mail/root

The files however are still on the destination.

How can I manually compete the process on the destination node and move forward?

[faris]

When you re-migrate the same Container, the rsync will be very quick because the majority of the files are already there. So don't necessarily look too hard at trying to manually start the Container in case it screws things up in other ways.

The main issue is just fixing the reason why the Container failed to start following the migration.

From the error, it is just a matter of an iptables module not being loaded.
There's going to be a log with, hopefully, exactly which module is missing.

But alternatively, maybe compare which modules are enabled within the Container on the old node with what's enabled in the main config on the new node. There's got to be a mismatch.

What I expect to happen once you fix this is that when you migrate again it will take minutes not hours.

This is all assuming I'm on the right track about what's wrong, of course!

[KrazyBob]

Great help from you as usual Faris. Where can I find the log?

In the mean time I am doing a normal vzbackup that I can then restore to the new node.

When you re-migrate the same Container, the rsync will be very quick because the majority of the files are already there. So don't necessarily look too hard at trying to manually start the Container in case it screws things up in other ways.

The main issue is just fixing the reason why the Container failed to start following the migration.

From the error, it is just a matter of an iptables module not being loaded.
There's going to be a log with, hopefully, exactly which module is missing.

But alternatively, maybe compare which modules are enabled within the Container on the old node with what's enabled in the main config on the new node. There's got to be a mismatch.

What I expect to happen once you fix this is that when you migrate again it will take minutes not hours.

This is all assuming I'm on the right track about what's wrong, of course!

[faris]

Good question about the log. There may be something in /var/log/messages on the new node.

If not, when you restore from the new backup, I would expect the same error on start. In that case, however, you might get more detail on startup than you would for the migration.

Now I'm thinking aloud here. Remember the limitations of my knowledge and how easily I get confused please. BUT:

/etc/sysconfig/iptables-config (on a Virtuozzo Hardware Node) defines which iptables modules are loaded on the hardware node itself. If a particular module isn't loaded on the hardware node itself then it can't be loaded in a Container no matter what.

WARNING: On Centos 6 and probably other flavours, unless /etc/sysconfig/iptables exists (even with no content), then /etc/sysconfig/iptables-config will not get looked at (iptables won't be loaded) and this means none of the iptables modules that you may have specified in that file will actually get loaded on the HN. In turn, this will cause problems with any Containers that make use of any significant sort of firewall because the modules will not be available.

/etc/sysconfig/vz file on a Virtuozzo Hardware Node defines....what? The modules that are enabled by default in a Container, I think. This should be the same as, or a subset of, the modules defined in /etc/sysconfig/iptables-config

Modules defined in /etc/sysconfig/iptables-config but not /etc/sysconfig/vz can be loaded on a per-container basis via the Container's individual configuration file.

My advice: make sure /etc/sysconfig/iptables-config and /etc/sysconfig/vz on the new node have the same (or more) modules as on the old node AND make sure those modules are actually loading by using lsmod | grep 'name of module'. If they are not, check that iptables itself is loading, if necessary by creating a blank /etc/sysconfg/iptables file

Here's a link to the KB that deals with how to use APF in a Container, in case it is useful in terms of which minimum set of modules are essential: http://kb.parallels.com/en/875

[KrazyBob]

Well, after two days of waiting I was told that the product is EOL and Parallels won't help. They had no problem taking my money to upgrade to Virtuozzo 4 and Plesk 11 but now that I am moving things around trying to actually utilize the new releases they tell me "The product is EOL and we won't help you."

When will Parallels get with it? They seem to do more to piss customers off than to actually resolve the problems. They need to understand the individual and unique problems.

Yes, the product is EOL. I've spent nearly 6 years of my life now living in and out of a cancer hospital. That IS my problem and I'll have to deal with it.

@Faris:It would be nice knowing where the log file is so that I can try to solve this. The error refers to iptables but I run a hardware firewall and don't run iptables.

[KrazyBob]

Let me try a different approach here instead of complaining. vzmigrate failed but left the files on the destination. As I recall one can simply create a directory and something else to make the migrated container active.

Ring a bell?

[faris]

What's EOL? VZ3 yes, but not 4.

This could be a language issue in 1st line support. I don't see this as a VZ3 issue. It is a support issue in VZ4.7 which is definitely not EOL. I'd complain to your sales rep.

The answer I'd expect once you get higher up the food chain support-wise would be similar to what I've been saying but with more specifics and most probably a "fixed it - it just needed ABC XYZ" once they had logged in.

Of course, as I mentioned before, this is all assuming I'm on the right track about what's wrong.

I really don't know which log it would be in. I'd be looking in all of them in order to find it, but I'd expect it in /var/log/messages and /var/log/vzctl.log but it could be elsewhere. But go look at the iptables configurations first. It doesn't matter that you have a hardware firewall - from the sounds of things something in the Container is trying to load an iptables module, even if it never gets used for anything.

And if your setup is anything like mine, when I had VZ3 under Centos 5, I didn't need to do anything special on the HN for a Container to use iptables. In contrast, VZ4 under Centos 6 required me to do stuff as outlined in my other post to get any Container using iptables to work.

To force start it, I think you just need to rename the ct303.conf.failed or whatever it might be to ct303.conf? Please don't assume I'm right, OK? I've never done this. And since the config is still live on the old HN, god knows what will happen with routing and stuff. I don't like this at all. I'm not at all comfortable with it.

[KrazyBob]

The server in question is VZ 3. I cannot get it to VZ 4 until i clear a server, put in new HDD's, and the format for VZ 4. This is where the failure occurred. VZ 3 migration to another VZ 3 node failed. I don't want to bitch and whine. I need a solution.

[faris]

Hmm..OK, how about doing an lsmod on both HNs, compare output, and load anything that looks conntrac/nat/rst related that's on the source but not the destination

However, I'm now less sure my original diagnosis was correct - have you recently moved other containers to this HN, but without errors?

Also, post on parallels forums. More people there who know about this sort of thing.

[KrazyBob]

I tried using vzbackup on Virtuozzo three to do this, which has always worded in the past. But now I am getting an unusual error message:

Code: Select all

[root@hw009 lock]# vzrestore -d 192.168.220.105 -e 304
Starting restore VE 304(2013-01-22T145600-0800@hw001.XXXXX.net) on node 192.168.220.105...
Doing exec - /usr/share/vzbackup-3.0.0-17.swsoft/vzrestore1 -t 2013-01-22T145600-0800@hw001.XXXXX.net   304 -.
Created /vz/private/304-tmpu15094
Restoring VE... 
Turn quota on
   -... 
Extract from file - error.
/usr/share/vzbackup-3.0.0-17.swsoft/vzrestore1: line 386: 15097 File size limit exceededtar xf "${name}" ${opt} --fast-links -C "${BKPTMPDIR}"
Turn quota off
Cleaning up...
Error - connection to root@192.168.220.105 is lost.
Failed to restore 192.168.220.105 VE 304 tag 2013-01-22T145600-0800@hw001.XXXXX.net.
You have new mail in /var/spool/mail/root

I see the file size limiter exceeded but I don't remember what this referred to.

[KrazyBob]

I was truly hoping someone knew the answer to this. I only have so many server and I need to get this to VZ4. But I have to first get it off of a server with HDD's throwing off RAID errors.

Any help is greatly appreciated.