Skip to content

[KVM] Skip volumes on shared storage when live migrating VM with local volumes (root / datadisk) - #6509

Closed
GabrielBrascher wants to merge 2 commits into
apache:mainfrom
CLDIN:virDomainMigrate3-libvirt
Closed

[KVM] Skip volumes on shared storage when live migrating VM with local volumes (root / datadisk)#6509
GabrielBrascher wants to merge 2 commits into
apache:mainfrom
CLDIN:virDomainMigrate3-libvirt

Conversation

@GabrielBrascher

@GabrielBrascher GabrielBrascher commented Jun 27, 2022

Copy link
Copy Markdown
Member

Description

The new release Libvirt-Java v0.5.3 now supports virDomainMigrate3[1],[2], a powerful and flexible domain migrate command in Libvirt. This command allows pointing just the Root volume to be migrated, saving a lot of compute resources, network bandwidth, and time.

Likely, data disks hold TBs of data, while Root volumes hundreds of GBs. This allows saving crucial time, bandwidth, and processing.

It can easily save several hours to offload a KVM machine depending on the amount of VMs to be migrated and data disks attached to the respective VMs.

[1] Libvirt-Java
[2] commit that added virtDomainMigrate3 into the Libvirt-Java
Fore reference, PR #6472 added libvirt-java v0.5.3 into CloudStack.

Types of changes

  • Breaking change (fix or feature that would cause existing functionality to change)
  • New feature (non-breaking change which adds functionality)
  • Bug fix (non-breaking change which fixes an issue)
  • Enhancement (improves an existing feature and functionality)
  • Cleanup (Code refactoring and cleanup, that may add test cases)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

  • Major
  • Minor

Screenshots (if appropriate):

How Has This Been Tested?

Tested migration of VM with local Root (50GB) + data-disk (100GB)

  1. As an example, this is a scenario without this PR:
~# virsh domjobinfo i-2-345-VM
Job type:         Unbounded   
Operation:        Outgoing migration
Time elapsed:     17063        ms
Data processed:   2.938 GiB
Data remaining:   147.065 GiB
Data total:       150.004 GiB
File processed:   2.938 MiB
File remaining:   147.065 GiB
File total:       150.004 GiB

Data total 150.004 GiB

  1. When testing the same migration but with the changes of the PR:
~# virsh domjobinfo i-2-345-VM
Job type:         Unbounded   
Operation:        Outgoing migration
Time elapsed:     10300        ms
Data processed:   952.000 MiB
Data remaining:   49.070 GiB
Data total:       50.000 GiB
File processed:   952.000 MiB
File remaining:   49.070 GiB
File total:       50.000 GiB

Data total 50.000 GiB

@GabrielBrascher GabrielBrascher added this to the 4.17.1.0 milestone Jun 27, 2022
@GabrielBrascher GabrielBrascher self-assigned this Jun 27, 2022
DaanHoogland
DaanHoogland approved these changes Jun 27, 2022
View reviewed changes

@DaanHoogland DaanHoogland left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clgtm

@DaanHoogland

DaanHoogland commented Jun 27, 2022

Copy link
Copy Markdown
Contributor

@blueorangutan package

@sureshanaparti sureshanaparti changed the title [KVM] Skip migraiton of datadisks when migrating VMs with local Root volume [KVM] Skip migration of datadisks when migrating VMs with local Root volume Jun 27, 2022
@blueorangutan

blueorangutan commented Jun 27, 2022

Copy link
Copy Markdown

@DaanHoogland a Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

@blueorangutan

blueorangutan commented Jun 27, 2022

Copy link
Copy Markdown

Packaging result: ✔️ el7 ✖️ el8 ✔️ debian ✔️ suse15. SL-JID 3654

RodrigoDLopez
RodrigoDLopez reviewed Jun 27, 2022
View reviewed changes

@RodrigoDLopez RodrigoDLopez left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clgtm

@DaanHoogland

DaanHoogland commented Jun 28, 2022

Copy link
Copy Markdown
Contributor

@GabrielBrascher can you have a look on the conflicts

@github-actions

github-actions Bot commented Jun 28, 2022

Copy link
Copy Markdown

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

@GabrielBrascher
GabrielBrascher force-pushed the virDomainMigrate3-libvirt branch from a4162b1 to ed0d27a Compare June 30, 2022 11:22
@GabrielBrascher
GabrielBrascher changed the base branch from main to 4.17 June 30, 2022 11:36
@GabrielBrascher
GabrielBrascher changed the base branch from 4.17 to main June 30, 2022 11:37
@GabrielBrascher

GabrielBrascher commented Jun 30, 2022

Copy link
Copy Markdown
Member Author

@DaanHoogland I was going to fix the branch conflict but then realise that it happens on main but not on 4.17.

Do you guys feel like this would be fine to be added in 4.17.1? @harikrishna-patnala @rohityadavcloud @DaanHoogland @wido @nvazquez others
Or should I target it to the main?

It could be placed as bugfix (since cloudstack migrates disks that are not desired due to Libvirt-Java 0.5.2 limitations).
Could also be argued as an enhancement PR as migrations were working (but taking waaay too long).

@GabrielBrascher

GabrielBrascher commented Jun 30, 2022 •
edited
Loading

Copy link
Copy Markdown
Member Author

I see some work recently done that allows multiple volumes on local storage. I will double-check these new PRs and see potential risks as I've considered 1 local volume (root) to be mapped in the new Libvirt migration flow.

@DaanHoogland

DaanHoogland commented Jul 4, 2022

Copy link
Copy Markdown
Contributor

I see some work recently done that allows multiple volumes on local storage. I will double-check these new PRs and see potential risks as I've considered 1 local volume (root) to be mapped in the new Libvirt migration flow.

so should we move this to draft, @GabrielBrascher ?

@GabrielBrascher
GabrielBrascher marked this pull request as draft July 5, 2022 12:20
@GabrielBrascher GabrielBrascher changed the title [KVM] Skip migration of datadisks when migrating VMs with local Root volume [KVM] Skip volumes on shared storage when live migrating VM with local volumes (root / datadisk) Jul 12, 2022
@GabrielBrascher
GabrielBrascher force-pushed the virDomainMigrate3-libvirt branch from ed0d27a to 4bf10c8 Compare July 12, 2022 21:11
@GabrielBrascher
Add virDomainMigrate3 into Libvirt migration.
80c3007 
This implementation uses virDomainMigrate3 in order to point which disk device should be migrated. Thus, avoid mapping shared volumes to libvirt when copying disks.

This allows to save crucial time (can be even hours depending on all data-disks), bandwidth, and processing.
@GabrielBrascher
GabrielBrascher force-pushed the virDomainMigrate3-libvirt branch from 4bf10c8 to 80c3007 Compare July 12, 2022 21:48
@acs-robot

acs-robot commented Jul 13, 2022

Copy link
Copy Markdown

Found UI changes, kicking a new UI QA build
@blueorangutan ui

@blueorangutan

blueorangutan commented Jul 13, 2022

Copy link
Copy Markdown

@acs-robot a Jenkins job has been kicked to build UI QA env. I'll keep you posted as I make progress.

@blueorangutan

blueorangutan commented Jul 13, 2022

Copy link
Copy Markdown

UI build: ✔️
Live QA URL: http://qa.cloudstack.cloud:8080/client/pr/6509 (SL-JID-1949)

@GabrielBrascher
GabrielBrascher force-pushed the virDomainMigrate3-libvirt branch from 271ac8a to a23c716 Compare July 13, 2022 07:55
@acs-robot

acs-robot commented Jul 13, 2022

Copy link
Copy Markdown

Found UI changes, kicking a new UI QA build
@blueorangutan ui

@blueorangutan

blueorangutan commented Jul 13, 2022

Copy link
Copy Markdown

@acs-robot a Jenkins job has been kicked to build UI QA env. I'll keep you posted as I make progress.

@blueorangutan

blueorangutan commented Jul 13, 2022

Copy link
Copy Markdown

UI build: ✔️
Live QA URL: http://qa.cloudstack.cloud:8080/client/pr/6509 (SL-JID-1950)

@sureshanaparti

sureshanaparti commented Jul 13, 2022

Copy link
Copy Markdown
Contributor

@blueorangutan package

@blueorangutan

blueorangutan commented Jul 13, 2022

Copy link
Copy Markdown

@sureshanaparti a Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

@blueorangutan

blueorangutan commented Jul 13, 2022

Copy link
Copy Markdown

Packaging result: ✖️ el7 ✖️ el8 ✖️ debian ✖️ suse15. SL-JID 3760

@sureshanaparti

sureshanaparti commented Jul 13, 2022

Copy link
Copy Markdown
Contributor

@GabrielBrascher Can you the fix these audit errors.

[ERROR] /jenkins/workspace/acs-centos7-pkg-builder/dist/rpmbuild/BUILD/cloudstack-4.18.0.0-SNAPSHOT/plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/wrapper/LibvirtMigrateCommandWrapper.java:27:8: Unused import - java.util.Collections. [UnusedImports]
[ERROR] /jenkins/workspace/acs-centos7-pkg-builder/dist/rpmbuild/BUILD/cloudstack-4.18.0.0-SNAPSHOT/plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/wrapper/LibvirtMigrateCommandWrapper.java:38:8: Unused import - java.util.stream.Collectors. [UnusedImports]

@GabrielBrascher
Manage multiple local disk devices labels for VM live-migration.
55fd42d 
Signed-off-by: gabriel <gabriel@apache.org>
@GabrielBrascher
GabrielBrascher force-pushed the virDomainMigrate3-libvirt branch from a23c716 to 55fd42d Compare July 14, 2022 06:41
@acs-robot

acs-robot commented Jul 14, 2022

Copy link
Copy Markdown

Found UI changes, kicking a new UI QA build
@blueorangutan ui

@blueorangutan

blueorangutan commented Jul 14, 2022

Copy link
Copy Markdown

@acs-robot a Jenkins job has been kicked to build UI QA env. I'll keep you posted as I make progress.

@blueorangutan

blueorangutan commented Jul 14, 2022

Copy link
Copy Markdown

UI build: ✔️
Live QA URL: http://qa.cloudstack.cloud:8080/client/pr/6509 (SL-JID-1958)

@sonarqubecloud

sonarqubecloud Bot commented Jul 14, 2022

Copy link
Copy Markdown

SonarCloud Quality Gate failed.    Quality Gate failed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 1 Code Smell

18.8% 18.8% Coverage
0.0% 0.0% Duplication

@github-actions

github-actions Bot commented Jul 27, 2022

Copy link
Copy Markdown

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

@shwstppr

shwstppr commented Aug 3, 2022

Copy link
Copy Markdown
Contributor

Hi @GabrielBrascher, should we consider this for 4.17.1.0? Currently, it has conflicts, can you please check

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@DaanHoogland DaanHoogland DaanHoogland approved these changes
+1 more reviewer
@RodrigoDLopez RodrigoDLopez RodrigoDLopez approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

@harikrishna-patnala harikrishna-patnala

@GabrielBrascher GabrielBrascher

Labels

component:kvm component:local-storage status:has-conflicts type:enhancement

Projects

None yet

Milestone

4.17.1.0

Development

Successfully merging this pull request may close these issues.

8 participants

@GabrielBrascher @DaanHoogland @blueorangutan @acs-robot @sureshanaparti @shwstppr @RodrigoDLopez @harikrishna-patnala