Migrate an OpenStack instance across compute nodes¶
Method 1: using migrate/live-migrate commands¶
Case A: boot volume on ephemeral disk¶
STOPPED/SHUTDOWN/PAUSED instances: migration
Migration for ephemeral instances with status stopped/suspended/paused:
$ nova migrate --poll 679511f2-4df1-44cc-883d-b3a37402e976
ACTIVE instances: migration with instances reboot
If the instance is in Active status and it is build on Ephemeral storage, the follows commands allows the migration with instances reboot:
$ nova migrate 27924a4a-b48e-4e3a-9786-955284d21e88
If you want it is possible add the poll option to report the server migration progress until it completes.
Alternatively you can use the Openstack command for the migration:
$ openstack server migrate 27924a4a-b48e-4e3a-9786-955284d21e88
After the migration command it will be necessary confirm the resize:
$ nova resize-confirm 27924a4a-b48e-4e3a-9786-955284d21e88
ACTIVE instances: live migration
If the instance is built on Ephemeral storage, live migration is possible with the following command:
$ nova live-migration --block-migrate 27924a4a-b48e-4e3a-9786-955284d21e88
OR
$ openstack server migrate --live-migration --block-migration 27924a4a-b48e-4e3a-9786-955284d21e88
You may optionally specify the destination host by adding its name as the last parameter:
$ nova live-migration --block-migrate 27924a4a-b48e-4e3a-9786-955284d21e88 pa1-r1-s04
OR
$ openstack server migrate --live-migration --block-migration --host pa1-r1-s04 27924a4a-b48e-4e3a-9786-955284d21e88
After the migration, make sure to check if the istance needs a resize confirmation.
Case B: boot volume on non-ephemeral (Ceph) disk¶
STOPPED/SHUTDOWN/PAUSED instances: migration
Migration for non-ephemeral instances with status stopped/suspended/paused:
$ openstack server migrate --wait 679511f2-4df1-44cc-883d-b3a37402e976
ACTIVE instances: migration with instances reboot
If the instance is in Active status and it is built on Ceph volume, the follows commands allows the migration with instances reboot:
$ openstack server migrate fcc624c7-24b7-4c11-b6cf-13cdae84d640
Also in this case, after the migration command it will be necessary confirm the resize:
$ nova resize-confirm fcc624c7-24b7-4c11-b6cf-13cdae84d640
ACTIVE instances: live migration
If the instance is built on Ceph volume, execute:
$ nova live-migration fcc624c7-24b7-4c11-b6cf-13cdae84d640
OR
$ openstack server migrate --live-migration 27924a4a-b48e-4e3a-9786-955284d21e88
If you want to have control over the destination host to move the VM to, append the hostname to the above command, or use:
$ openstack server migrate --live-migration --block-migration --host ct1-r3-s15 fcc624c7-24b7-4c11-b6cf-13cdae84d640
If the previous command returns an error, you can try without –block-migration option:
$ openstack server migrate --live-migration --host ct1-r3-s15 fcc624c7-24b7-4c11-b6cf-13cdae84d640
General case¶
If the instance is built on Ceph volume, execute:
$ nova live-migration <id vm>
If you want to migrate all instances that run on a specific compute node:
$ nova host-servers-migrate <id compute>
Method 2: manual procedure¶
This post describes the procedure to manually migrate an instance from an OpenStack compute node to a destination host, in case the live migration option is unavailable.
These instructions have been extracted and adapted to our MAAS+Juju Cloud platform from this page .
Identify instance¶
Let’s first identify which is the compute ($HOST_ORIG) which hosts the VM we want to migrate (which we call $MYVM). On our OpenStack client let’s do:
openstack server show $MYVM
The output of the command will be something like (I only pasted the relevant parameters):
Field |
Value |
OS-EXT-SRV-ATTR:host |
ct1-r1-s09 |
OS-EXT-SRV-ATTR:hypervisor_hostname |
ct1-r1-s09.maas |
os-extended-volumes:volumes_attached |
[{u’id’: u’9d0ee9fc-e0ac-4b6e-b408-19d3356394c9’}] |
addresses |
default=192.168.250.79, 90.147.166.88 |
id |
f5a95044-a6c1-4ee9-a4f5-e8cfbc4c1c07 |
security_groups |
[{u’name’: u’default’}, {u’name’: u’allow22fromDir’}] |
Take note of the origin host name ($ORIG_HOST) ans hypervisor_hostname (which in our MAAS environment will be in the form ${ORIG_HOST}.maas) and of the VM’s ID, IP addresses and attached volumes and security groups, as we will need them later.
Let’s stop the VM:
openstack server stop $MYVM
Also detach any volume:
nova volume-detach $VM_UUID $VOLUME_UUID
Identify origin and destination hosts¶
Using MAAS and Juju identify the Juju ID and IP Addresses of the origin and destintation host (respectively $ORIG_HOST_ID, $ORIG_HOST_IP, $DEST_HOST_ID, $DEST_HOST_IP).
Copy instance to the destination compute node¶
On the hypervisor the instance is usually located in a folder called /var/lib/nova/instances/$MYVM_ID. We need to copy this directory to the destination host.
We need to do some operations since we cannot do scp as root to another host without installing a SSH key pair. Let’s do it from our Juju client. Copy the private key to the destination host and the public key to the origin host’s /root/.ssh/authorized_keys2:
juju scp .ssh/$MYPRIV_KEY $DEST_HOST_ID:.ssh/.
juju ssh $ORIG_HOST_ID sudo cp /home/ubuntu/.ssh/authorized_keys /root/.ssh/authorized_keys2
In this case we use the SSH keypair we injected to the hosts via Juju.
Login to that hypervisor via SSH. Navigate to the folder /var/lib/nova-compute/instances/$MYVM_ID.
The instance is booted from an image based root disk, named disk. qemu in our case diffs the root disk from the image the VM was created from. Therefore the new hypervisor also needs that backing image. Find out which file is the backing image:
qemu-img info disk # disk is the filename of the instance root disk
Example output:
image: disk
file format: qcow2
virtual size: 32G (34359738368 bytes)
disk size: 1.3G
cluster_size: 65536
backing file: /var/lib/nova-compute/instances/_base/d00[...]61
Format specific information:
compat: 1.1
lazy refcounts: false
The file /var/lib/nova-compute/instances/_base/d00…61 is the backing disk.
Check the target hypervisor for the existence of that image. If it is not there, copy that file to the target hypervisor first, from the Juju client:
juju ssh $DEST_HOST_ID sudo rsync -Pav -e \"ssh -i /home/ubuntu/.ssh/$MYPRIV_KEY\" ubuntu@$ORIG_HOST_IP:/var/lib/nova/instances/_base/d00[...]61 /var/lib/nova/instances/_base/d00[...]61
Now copy the instance folder to the new hypervisor:
juju ssh $DEST_HOST_ID sudo rsync -Pav -e \"ssh -i /home/ubuntu/.ssh/$MYPRIV_KEY\" ubuntu@$ORIG_HOST_IP:/var/lib/nova/instances/$MYVM_ID /var/lib/nova/instances/
Check that ownership and permissions are preserved.
Update Nova DB¶
Log in to your database server. In our case that is a MySQL Percona cluster:
juju ssh percona-cluster
sudo su -
Start up a MySQL command prompt in the nova database:
mysql -u root -p$PASSWORD nova
Execute the following command to update the nova database with the new hypervisor for this VM:
update instances set node='${DEST_HOST}.maas', host=$DEST_HOST where uuid='$MYVM_ID';
Use the nova show command to see if the new hypervisor is set. If so, start the VM:
nova start $MYVM
Attach any volumes that were detached earlier:
nova volume-attach $MYVM_ID $VOLUME
Detach and re-attach network ports¶
Unlike what reported in the instructions we needed to detach and re-attach the VM network interface to enable networking:
nova interface-list $MYVM_ID
Copy the Port ID ($MYVM_PORT_ID) and Net ID ($MYVM_NET_ID) fields corresponding to $MYVM_IP, then:
nova interface-detach $MYVM_ID $MYVM_PORT_ID
nova interface-attach --net-id $MYVM_NET_ID --fixed-ip $MYVM_FIXED_IP $MYVM_ID
Now re-associate the public IP to the VM.
Finally do:
nova reboot $MYVM_ID
Some useful commands¶
List all the migrations in the region:
nova migration-list
List all the migrations for a specified instance (useful for obtaining migration ids):
nova --os-compute-api-version 2.23 --os-endpoint-override http://90.147.165.93:8774/v2.1 server-migration-list 4274d25a-533b-4377-842e-9ec16a49d825
In Openstack Queens, –os-compute-api-version can be any number from 2.23 (fist compute-api-version that supports the command) to 2.60 ( max compute-api-version supported in Queens). We also need to override the compute endpoint URL since the compute endpoint (for the CT region in this case) registered in keystone is v2.0, which do not support compute-api microversions such as 2.23.
Show details of a particular migration for a specified instance (server-migration-show <server-id> <migration-id>):
nova --os-compute-api-version 2.23 --os-endpoint-override http://90.147.165.93:8774/v2.1 server-migration-show 4274d25a-533b-4377-842e-9ec16a49d825 8185
Abort a live migration (state running or preparing):
nova --os-compute-api-version 2.23 --os-endpoint-override http://90.147.165.93:8774/v2.1 live-migration-abort 4274d25a-533b-4377-842e-9ec16a49d825 8185
We can also abort a live migration in a queued state since Openstack Rocky (not in Queens)
