Fix backup dates by JoaoJandre · Pull Request #6473 · apache/cloudstack

JoaoJandre · 2022-06-17T15:35:57Z

Description

Currently, the backup date in Veeam is collected as a String and saved as VARCHAR(255) in the database, which can lead to errors and inconsistencies.

We are proposing with this PR to persist the backup dates as a DATETIME object, which will enable us to properly manipulate it, regardless of the date format used in the environment. The UI was also changed to display the complete data format.

When upgrading the DB schema, we delete all backup entries in the DB; these backups are synchronized with Veeam backups by ACS afterwards. In the end, no Veeam backups should be lost. Only dummy backups will be permanently deleted, and since they are not relevant in production, no harm will be caused.

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

Screenshots (if appropriate):

Storage->Backups

Instances->VM->Backups

How Has This Been Tested?

This was tested in a local lab, by upgrading the DB schema and then checking if ACS would correctly synchronize with Veeam. After that, the UI was checked to see if everything was as expected.
New backups were created and checked. The same tests were made using the dummy provider.

GutoVeronezi · 2022-06-21T16:31:17Z

@blueorangutan package

blueorangutan · 2022-06-21T16:32:02Z

@GutoVeronezi a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2022-06-21T16:32:03Z

@GutoVeronezi a Jenkins job has been kicked to build packages. It will be bundled with
SystemVM template(s). I'll keep you posted as I make progress.

acs-robot · 2022-06-21T16:32:15Z

Found UI changes, kicking a new UI QA build
@blueorangutan ui

blueorangutan · 2022-06-21T16:33:03Z

@acs-robot a Jenkins job has been kicked to build UI QA env. I'll keep you posted as I make progress.

blueorangutan · 2022-06-21T16:33:03Z

@acs-robot a Jenkins job has been kicked to build UI QA env. I'll keep you posted as I make progress.

blueorangutan · 2022-06-21T16:47:58Z

UI build: ✖️
(SL-JID-1799)

blueorangutan · 2022-06-21T16:48:33Z

UI build: ✖️
(SL-JID-3)

blueorangutan · 2022-06-21T17:33:50Z

Packaging result: ✖️ el7 ✖️ el8 ✖️ debian ✖️ suse15. SL-JID 1

blueorangutan · 2022-06-21T17:43:36Z

Packaging result: ✖️ el7 ✖️ el8 ✖️ debian ✖️ suse15. SL-JID 3617

acs-robot · 2022-06-22T11:47:01Z

Found UI changes, kicking a new UI QA build
@blueorangutan ui

blueorangutan · 2022-06-22T11:48:03Z

@acs-robot a Jenkins job has been kicked to build UI QA env. I'll keep you posted as I make progress.

blueorangutan · 2022-06-22T11:48:03Z

@acs-robot a Jenkins job has been kicked to build UI QA env. I'll keep you posted as I make progress.

blueorangutan · 2022-06-22T11:54:08Z

UI build: ✖️
(SL-JID-1810)

Co-authored-by: dahn <daan.hoogland@gmail.com>

acs-robot · 2022-06-24T13:41:39Z

Found UI changes, kicking a new UI QA build
@blueorangutan ui

blueorangutan · 2022-06-24T13:42:02Z

@acs-robot a Jenkins job has been kicked to build UI QA env. I'll keep you posted as I make progress.

blueorangutan · 2022-06-24T13:48:04Z

UI build: ✖️
(SL-JID-1837)

sonarqubecloud · 2022-06-24T14:33:23Z

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

0.0% Coverage
0.0% Duplication

github-actions · 2022-06-27T04:43:48Z

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

blueorangutan · 2023-05-25T14:40:15Z

[LL]Trillian test result (tid-6537)
Environment: vmware-70u3 (x2), Advanced Networking with Mgmt server r8
Total time taken: 23895 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr6473-t6537-vmware-70u3.zip
Smoke tests completed. 86 look OK, 0 have errors, 24 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
all_test_safe_shutdown	`Skipped`	---	test_safe_shutdown.py
all_test_metrics_api	`Skipped`	---	test_metrics_api.py
all_test_outofbandmanagement	`Skipped`	---	test_outofbandmanagement.py
all_test_outofbandmanagement_nestedplugin	`Skipped`	---	test_outofbandmanagement_nestedplugin.py
all_test_routers_iptables_default_policy	`Skipped`	---	test_routers_iptables_default_policy.py
all_test_secondary_storage	`Skipped`	---	test_secondary_storage.py
all_test_service_offerings	`Skipped`	---	test_service_offerings.py
all_test_storage_policy	`Skipped`	---	test_storage_policy.py
all_test_templates	`Skipped`	---	test_templates.py
all_test_update_security_group	`Skipped`	---	test_update_security_group.py
all_test_usage_events	`Skipped`	---	test_usage_events.py
all_test_vm_autoscaling	`Skipped`	---	test_vm_autoscaling.py
all_test_vm_deployment_planner	`Skipped`	---	test_vm_deployment_planner.py
all_test_vm_life_cycle	`Skipped`	---	test_vm_life_cycle.py
all_test_vm_lifecycle_unmanage_import	`Skipped`	---	test_vm_lifecycle_unmanage_import.py
all_test_vm_snapshot_kvm	`Skipped`	---	test_vm_snapshot_kvm.py
all_test_vm_snapshots	`Skipped`	---	test_vm_snapshots.py
all_test_volumes	`Skipped`	---	test_volumes.py
all_test_vpc_ipv6	`Skipped`	---	test_vpc_ipv6.py
all_test_vpc_redundant	`Skipped`	---	test_vpc_redundant.py
all_test_vpc_router_nics	`Skipped`	---	test_vpc_router_nics.py
all_test_vpc_vpn	`Skipped`	---	test_vpc_vpn.py
all_test_host_maintenance	`Skipped`	---	test_host_maintenance.py
all_test_hostha_kvm	`Skipped`	---	test_hostha_kvm.py

JoaoJandre · 2023-07-13T13:36:06Z

@DaanHoogland could we run the CI again for this PR as well?

DaanHoogland · 2023-07-13T14:44:51Z

@blueorangutan package

blueorangutan · 2023-07-13T14:46:03Z

@DaanHoogland a [SF] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2023-07-13T15:40:14Z

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 6475

DaanHoogland · 2023-07-14T06:54:24Z

@blueorangutan test rocky8 vmware-67u3

blueorangutan · 2023-07-14T06:56:03Z

@DaanHoogland a [SF] Trillian-Jenkins test job (rocky8 mgmt + vmware-67u3) has been kicked to run smoke tests

harikrishna-patnala · 2023-07-14T09:45:13Z

changed the label to 4.19 as the upgrade file changes are related to 4.19 version

JoaoJandre · 2023-08-09T17:31:20Z

@DaanHoogland I think BO ended up not running the tests on this one, can we retry?

JoaoJandre · 2023-08-09T17:31:27Z

@blueorangutan package

blueorangutan · 2023-08-09T17:32:03Z

@JoaoJandre a [SF] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2023-08-09T18:39:54Z

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 6723

DaanHoogland · 2023-08-10T06:53:23Z

@blueorangutan test rocky8 vmware-67u3

blueorangutan · 2023-08-10T06:54:03Z

@DaanHoogland a [SF] Trillian-Jenkins test job (rocky8 mgmt + vmware-67u3) has been kicked to run smoke tests

blueorangutan · 2023-08-11T06:29:44Z

[SF] Trillian test result (tid-7358)
Environment: vmware-67u3 (x2), Advanced Networking with Mgmt server r8
Total time taken: 83324 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr6473-t7358-vmware-67u3.zip
Smoke tests completed. 86 look OK, 27 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_02_list_cpvm_vm	`Failure`	0.04	test_ssvm.py
test_04_cpvm_internals	`Failure`	0.04	test_ssvm.py
ContextSuite context=TestHostControlState>:setup	`Error`	4.50	test_host_control_state.py
test_01_add_delete_kubernetes_supported_version	`Error`	0.07	test_kubernetes_supported_versions.py
ContextSuite context=TestListIdsParams>:setup	`Error`	0.00	test_list_ids_parameter.py
ContextSuite context=TestRouterDns>:setup	`Error`	0.00	test_router_dns.py
ContextSuite context=TestLoadBalance>:setup	`Error`	0.00	test_loadbalance.py
ContextSuite context=TestRouterDnsService>:setup	`Error`	0.00	test_router_dnsservice.py
ContextSuite context=TestRouterServices>:setup	`Error`	0.00	test_routers.py
ContextSuite context=TestMetrics>:setup	`Error`	0.00	test_metrics_api.py
ContextSuite context=TestIsolatedNetworks>:setup	`Error`	0.00	test_routers_network_ops.py
ContextSuite context=TestRedundantIsolateNetworks>:setup	`Error`	0.00	test_routers_network_ops.py
ContextSuite context=TestRouterIpTablesPolicies>:setup	`Error`	0.00	test_routers_iptables_default_policy.py
ContextSuite context=TestVPCIpTablesPolicies>:setup	`Error`	0.00	test_routers_iptables_default_policy.py
test_01_sys_vm_start	`Failure`	60.21	test_secondary_storage.py
ContextSuite context=TestServiceOfferings>:setup	`Error`	1516.18	test_service_offerings.py
ContextSuite context=TestVMWareStoragePolicies>:setup	`Error`	0.00	test_storage_policy.py
test_02_create_template_with_checksum_sha1	`Error`	65.58	test_templates.py
test_03_create_template_with_checksum_sha256	`Error`	65.59	test_templates.py
test_04_create_template_with_checksum_md5	`Error`	65.58	test_templates.py
test_05_create_template_with_no_checksum	`Error`	65.59	test_templates.py
ContextSuite context=TestTemplates>:setup	`Error`	6.76	test_templates.py
ContextSuite context=TestVmAutoScaling>:setup	`Error`	0.00	test_vm_autoscaling.py
test_01_deploy_vm_on_specific_host	`Error`	1.32	test_vm_deployment_planner.py
test_02_deploy_vm_on_specific_cluster	`Error`	1.27	test_vm_deployment_planner.py
test_03_deploy_vm_on_specific_pod	`Error`	1.31	test_vm_deployment_planner.py
test_04_deploy_vm_on_host_override_pod_and_cluster	`Error`	1.34	test_vm_deployment_planner.py
test_05_deploy_vm_on_cluster_override_pod	`Error`	1.32	test_vm_deployment_planner.py
ContextSuite context=TestDeployVM>:setup	`Error`	0.00	test_vm_life_cycle.py
test_01_offline_migrate_VM_and_root_volume	`Error`	1.33	test_vm_life_cycle.py
test_02_offline_migrate_VM_with_two_data_disks	`Error`	1.30	test_vm_life_cycle.py
test_03_live_migrate_VM_with_two_data_disks	`Error`	1.29	test_vm_life_cycle.py
test_04_migrate_detached_volume	`Error`	1.32	test_vm_life_cycle.py
ContextSuite context=TestVAppsVM>:setup	`Error`	1518.65	test_vm_life_cycle.py
ContextSuite context=TestVMLifeCycle>:setup	`Error`	1522.76	test_vm_life_cycle.py
test_01_unmanage_vm_cycle	`Error`	4.95	test_vm_lifecycle_unmanage_import.py
ContextSuite context=TestUnmanageVM>:teardown	`Error`	4.97	test_vm_lifecycle_unmanage_import.py
ContextSuite context=TestVMSchedule>:setup	`Error`	0.00	test_vm_schedule.py
test_change_service_offering_for_vm_with_snapshots	`Error`	1.35	test_vm_snapshots.py
ContextSuite context=TestVmSnapshot>:setup	`Error`	10.57	test_vm_snapshots.py
ContextSuite context=TestCreateVolume>:setup	`Error`	0.00	test_volumes.py
ContextSuite context=TestVolumes>:setup	`Error`	1516.36	test_volumes.py
ContextSuite context=TestIpv6Vpc>:setup	`Error`	0.00	test_vpc_ipv6.py
ContextSuite context=TestVPCRedundancy>:setup	`Error`	0.00	test_vpc_redundant.py
ContextSuite context=TestVPCNics>:setup	`Error`	0.00	test_vpc_router_nics.py
ContextSuite context=TestRVPCSite2SiteVpn>:setup	`Error`	0.00	test_vpc_vpn.py
ContextSuite context=TestVPCSite2SiteVPNMultipleOptions>:setup	`Error`	0.00	test_vpc_vpn.py
ContextSuite context=TestVpcRemoteAccessVpn>:setup	`Error`	0.00	test_vpc_vpn.py
ContextSuite context=TestVpcSite2SiteVpn>:setup	`Error`	0.00	test_vpc_vpn.py
test_02_cancel_host_maintenace_with_migration_jobs	`Error`	1.63	test_host_maintenance.py
test_03_cancel_host_maintenace_with_migration_jobs_failure	`Error`	1.67	test_host_maintenance.py

JoaoJandre · 2023-08-11T14:30:11Z

I looked at some of these errors and none of them seemed related to this PR, could we rerun the tests?

DaanHoogland · 2023-08-11T14:40:46Z

@blueorangutan test rocky8 vmware-67u3

blueorangutan · 2023-08-11T14:42:03Z

@DaanHoogland a [SF] Trillian-Jenkins test job (rocky8 mgmt + vmware-67u3) has been kicked to run smoke tests

blueorangutan · 2023-08-12T04:04:04Z

[SF] Trillian test result (tid-7378)
Environment: vmware-67u3 (x2), Advanced Networking with Mgmt server r8
Total time taken: 72244 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr6473-t7378-vmware-67u3.zip
Smoke tests completed. 109 look OK, 4 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_02_trigger_shutdown	`Failure`	337.13	test_safe_shutdown.py
test_list_vms_metrics_admin	`Error`	3614.90	test_metrics_api.py
test_list_vms_metrics_history	`Error`	5.67	test_metrics_api.py
test_list_volumes_metrics_history	`Error`	3623.34	test_metrics_api.py
test_01_deploy_vm_on_specific_host	`Error`	18.76	test_vm_deployment_planner.py
test_02_deploy_vm_on_specific_cluster	`Error`	3603.64	test_vm_deployment_planner.py
test_03_deploy_vm_on_specific_pod	`Error`	4.45	test_vm_deployment_planner.py
test_04_deploy_vm_on_host_override_pod_and_cluster	`Error`	4.43	test_vm_deployment_planner.py
test_05_deploy_vm_on_cluster_override_pod	`Error`	2.33	test_vm_deployment_planner.py
test_09_expunge_vm	`Failure`	426.69	test_vm_life_cycle.py

blueorangutan · 2023-08-12T07:27:37Z

[SF] Trillian test result (tid-7386)
Environment: vmware-67u3 (x2), Advanced Networking with Mgmt server r8
Total time taken: 58837 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr6473-t7386-vmware-67u3.zip
Smoke tests completed. 112 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_01_invalid_upgrade_kubernetes_cluster	`Failure`	3610.45	test_kubernetes_clusters.py
test_02_upgrade_kubernetes_cluster	`Failure`	3619.28	test_kubernetes_clusters.py
test_03_deploy_and_scale_kubernetes_cluster	`Failure`	0.05	test_kubernetes_clusters.py
test_04_autoscale_kubernetes_cluster	`Failure`	0.06	test_kubernetes_clusters.py
test_05_basic_lifecycle_kubernetes_cluster	`Failure`	0.06	test_kubernetes_clusters.py
test_06_delete_kubernetes_cluster	`Failure`	0.08	test_kubernetes_clusters.py
test_07_deploy_kubernetes_ha_cluster	`Failure`	0.08	test_kubernetes_clusters.py
test_08_upgrade_kubernetes_ha_cluster	`Failure`	0.05	test_kubernetes_clusters.py
test_09_delete_kubernetes_ha_cluster	`Failure`	0.06	test_kubernetes_clusters.py
test_10_vpc_tier_kubernetes_cluster	`Failure`	72.65	test_kubernetes_clusters.py
test_11_test_unmanaged_cluster_lifecycle	`Error`	1.32	test_kubernetes_clusters.py
ContextSuite context=TestKubernetesCluster>:teardown	`Error`	68.26	test_kubernetes_clusters.py

JoaoJandre · 2023-08-16T18:06:31Z

@blueorangutan package

blueorangutan · 2023-08-16T18:08:03Z

@JoaoJandre a [SF] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

DaanHoogland · 2023-08-17T07:29:31Z

@blueorangutan test rocky8 vmware-67u3

blueorangutan · 2023-08-17T07:30:03Z

@DaanHoogland a [SF] Trillian-Jenkins test job (rocky8 mgmt + vmware-67u3) has been kicked to run smoke tests

blueorangutan · 2023-08-17T21:27:22Z

[SF] Trillian test result (tid-7456)
Environment: vmware-67u3 (x2), Advanced Networking with Mgmt server r8
Total time taken: 48802 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr6473-t7456-vmware-67u3.zip
Smoke tests completed. 112 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_02_upgrade_kubernetes_cluster	`Failure`	588.15	test_kubernetes_clusters.py
test_04_autoscale_kubernetes_cluster	`Failure`	71.02	test_kubernetes_clusters.py

Fix backup dates

21a12aa

boring-cyborg Bot added component:api component:backup component:UI labels Jun 17, 2022

GutoVeronezi closed this Jun 21, 2022

GutoVeronezi reopened this Jun 21, 2022

GutoVeronezi reviewed Jun 21, 2022

View reviewed changes

Comment thread engine/schema/src/main/resources/META-INF/db/schema-41710to41800.sql Outdated

Address review

e0fd674

DaanHoogland reviewed Jun 23, 2022

View reviewed changes

Comment thread plugins/backup/veeam/src/main/java/org/apache/cloudstack/backup/veeam/VeeamClient.java Outdated

Address reviews

dc3555e

Co-authored-by: dahn <daan.hoogland@gmail.com>

github-actions Bot added the status:has-conflicts label Jun 27, 2022

harikrishna-patnala modified the milestones: 4.18.1.0, 4.19.0.0 Jul 14, 2023

DaanHoogland added the has-3rdparty-components label Aug 18, 2023

DaanHoogland merged commit 1065e90 into apache:main Aug 18, 2023

sudo87 mentioned this pull request Aug 7, 2026

Improve date/timestamp handling performance in GenericDaoBase and DateUtil #13809

Open

14 tasks

Conversation

JoaoJandre commented Jun 17, 2022

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

Uh oh!

GutoVeronezi commented Jun 21, 2022

Uh oh!

blueorangutan commented Jun 21, 2022

Uh oh!

blueorangutan commented Jun 21, 2022

Uh oh!

acs-robot commented Jun 21, 2022

Uh oh!

blueorangutan commented Jun 21, 2022

Uh oh!

blueorangutan commented Jun 21, 2022

Uh oh!

Uh oh!

blueorangutan commented Jun 21, 2022

Uh oh!

blueorangutan commented Jun 21, 2022

Uh oh!

blueorangutan commented Jun 21, 2022

Uh oh!

blueorangutan commented Jun 21, 2022

Uh oh!

acs-robot commented Jun 22, 2022

Uh oh!

blueorangutan commented Jun 22, 2022

Uh oh!

blueorangutan commented Jun 22, 2022

Uh oh!

blueorangutan commented Jun 22, 2022

Uh oh!

Uh oh!

acs-robot commented Jun 24, 2022

Uh oh!

blueorangutan commented Jun 24, 2022

Uh oh!

blueorangutan commented Jun 24, 2022

Uh oh!

sonarqubecloud Bot commented Jun 24, 2022

Uh oh!

github-actions Bot commented Jun 27, 2022

Uh oh!

blueorangutan commented May 25, 2023

Uh oh!

JoaoJandre commented Jul 13, 2023

Uh oh!

DaanHoogland commented Jul 13, 2023

Uh oh!

blueorangutan commented Jul 13, 2023

Uh oh!

blueorangutan commented Jul 13, 2023

Uh oh!

DaanHoogland commented Jul 14, 2023

Uh oh!

blueorangutan commented Jul 14, 2023

Uh oh!

harikrishna-patnala commented Jul 14, 2023

Uh oh!

JoaoJandre commented Aug 9, 2023

Uh oh!

JoaoJandre commented Aug 9, 2023

Uh oh!

blueorangutan commented Aug 9, 2023

Uh oh!

blueorangutan commented Aug 9, 2023

Uh oh!

DaanHoogland commented Aug 10, 2023

Uh oh!

blueorangutan commented Aug 10, 2023

Uh oh!

blueorangutan commented Aug 11, 2023

Uh oh!