[jmsperu]

Summary

Adds four optional, zone-scoped features to NAS backup operations on KVM, all disabled by default:

• Compression (-c): Uses qcow2 internal compression (qemu-img convert -c) to reduce backup size
• LUKS Encryption (-e): Encrypts backup files at rest using LUKS via qemu-img convert --object secret
• Bandwidth Throttle (-b): Limits backup I/O — virsh blockjob --bandwidth for running VMs, qemu-img convert -r + ionice for stopped VMs
• Integrity Check (--verify): Runs qemu-img check on each backup file after creation

Configuration Keys (Zone scope)

Setting	Type	Default	Description
nas.backup.compression.enabled	Boolean	false	Enable qcow2 compression for backup files
nas.backup.encryption.enabled	Boolean	false	Enable LUKS encryption for backup files
nas.backup.encryption.passphrase	String (Secure)	""	Passphrase for LUKS encryption
nas.backup.bandwidth.limit.mbps	Integer	0	Bandwidth limit in MiB/s (0 = unlimited)
nas.backup.integrity.check	Boolean	false	Run qemu-img check after backup

Architecture

1. NASBackupProvider reads zone-scoped ConfigKeys and populates a details map on TakeBackupCommand
2. TakeBackupCommand carries the details map from management server to KVM agent
3. LibvirtTakeBackupCommandWrapper extracts the details and translates them to nasbackup.sh CLI flags
4. nasbackup.sh implements the actual compression, encryption, throttling, and verification logic

Files Changed

• scripts/vm/hypervisor/kvm/nasbackup.sh — new -c, -b, -e, --verify flags with encrypt_backup() and verify_backup() functions
• core/.../TakeBackupCommand.java — added details map (HashMap) with getter/setter/addDetail
• plugins/backup/nas/.../NASBackupProvider.java — 5 new ConfigKeys, populate command details in takeBackup()
• plugins/hypervisors/kvm/.../LibvirtTakeBackupCommandWrapper.java — extract details, build dynamic CLI args, temp passphrase file lifecycle

Notes

• All existing functionality (quiesce, cleanup, RBD support, stats) is preserved unchanged
• Encryption passphrase is written to a temp file on the KVM host and deleted after backup completes
• The passphrase ConfigKey uses the "Secure" category so it is not exposed in API responses
• Combines and supersedes PRs nasbackup.sh: add optional backup compression via -c flag #12844, nasbackup.sh: add bandwidth throttle via -b flag #12846, nasbackup.sh: add LUKS encryption for backup files via -e flag #12848, nasbackup.sh: verify backup integrity with qemu-img check #12845

Test plan

• Verify backup works with all four features disabled (default) — no behavioral change
• Enable nas.backup.compression.enabled at zone scope, take backup, verify qcow2 files are compressed
• Enable nas.backup.bandwidth.limit.mbps (e.g. 50), take backup of running VM, verify virsh blockjob bandwidth is applied
• Enable nas.backup.bandwidth.limit.mbps, take backup of stopped VM, verify qemu-img -r rate limit is applied
• Enable nas.backup.encryption.enabled with passphrase, take backup, verify files are LUKS encrypted (qemu-img info shows encryption)
• Enable nas.backup.integrity.check, take backup, verify qemu-img check runs and passes
• Test with multiple features enabled simultaneously (compression + integrity check)
• Verify restore still works for backups created with compression/encryption
• Test with RBD storage pools — verify bandwidth throttle applies correctly

[codecov]

Codecov Report

❌ Patch coverage is 73.91304% with 48 lines in your changes missing coverage. Please review.
✅ Project coverage is 18.28%. Comparing base (30ed411) to head (34967c0).

Files with missing lines	Patch %	Lines
...ce/wrapper/LibvirtRestoreBackupCommandWrapper.java	71.87%	13 Missing and 5 partials ⚠️
.../kvm/resource/wrapper/NasBackupPassphraseFile.java	62.50%	6 Missing and 3 partials ⚠️
...ource/wrapper/LibvirtTakeBackupCommandWrapper.java	78.94%	6 Missing and 2 partials ⚠️
...rg/apache/cloudstack/backup/TakeBackupCommand.java	40.00%	6 Missing ⚠️
...rg/apache/cloudstack/backup/NASBackupProvider.java	90.47%	0 Missing and 4 partials ⚠️
...apache/cloudstack/backup/RestoreBackupCommand.java	50.00%	3 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               4.22   #12898      +/-   ##
============================================
+ Coverage     18.26%   18.28%   +0.01%     
- Complexity    16608    16643      +35     
============================================
  Files          5939     5940       +1     
  Lines        537697   537856     +159     
  Branches      65922    65943      +21     
============================================
+ Hits          98210    98324     +114     
- Misses       428299   428333      +34     
- Partials      11188    11199      +11     

Flag	Coverage Δ
uitests	4.08% <ø> (ø)
unittests	19.36% <73.91%> (+0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[Copilot]

Pull request overview

Adds optional, zone-scoped enhancements for KVM NAS backups (compression, LUKS encryption, bandwidth throttling, and post-backup integrity verification) by plumbing config from management server → TakeBackupCommand details → KVM agent wrapper → nasbackup.sh flags.

Changes:

• Add new CLI flags and implementation in nasbackup.sh for compression (-c), encryption (-e), bandwidth throttling (-b), and verification (--verify).
• Extend TakeBackupCommand with a details map to carry optional settings to the agent.
• Add zone-scoped NAS backup ConfigKeys and populate command details; update KVM wrapper to translate details into script args and manage a temporary passphrase file.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 10 comments.

File	Description
scripts/vm/hypervisor/kvm/nasbackup.sh	Implements compression/encryption/throttle/verify logic and argument parsing for NAS backup operations.
core/src/main/java/org/apache/cloudstack/backup/TakeBackupCommand.java	Adds a details map to carry optional backup feature settings from management to agent.
plugins/backup/nas/src/main/java/org/apache/cloudstack/backup/NASBackupProvider.java	Introduces zone-scoped ConfigKeys and passes enabled settings into TakeBackupCommand details.
plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/wrapper/LibvirtTakeBackupCommandWrapper.java	Builds dynamic nasbackup.sh command args from TakeBackupCommand details and writes an encryption passphrase temp file.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[sureshanaparti]

@blueorangutan package

[blueorangutan]

@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

[blueorangutan]

Packaging result [SF]: ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 17323

[sureshanaparti]

@jmsperu can you check/fix the build failure.

[jmsperu]

@sureshanaparti Fixed. The test failures were caused by overrideConfigValue() in NASBackupProviderTest only setting _value on ConfigKey, but the zone-scoped configs (valueIn(zoneId)) fall back to _defaultValue when s_depot is null in the test context. All 5 config values (compression, bandwidth, encryption, encryption passphrase, integrity check) were returning null instead of the test values.

The fix also sets _defaultValue via ReflectionTestUtils so valueIn() correctly resolves test values.

Also addressed in the previous commit: replaced 2>&1 | tee -a with >> logFile 2>&1 in nasbackup.sh to prevent tee from masking non-zero exit codes, and added return 1 after cleanup on virsh backup job failure.

Could you please retrigger the build? @blueorangutan package

[blueorangutan]

@jmsperu a [SL] Jenkins job has been kicked to build packages. It will be bundled with` SystemVM template(s). I'll keep you posted as I make progress.

[blueorangutan]

Packaging result [SF]: ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 17346

[Copilot]

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.

Comments suppressed due to low confidence (1)

scripts/vm/hypervisor/kvm/nasbackup.sh:321

• mount_operation runs under set -e, but mount is executed as a standalone command and then $? is checked. With set -e, a mount failure will exit the script immediately and bypass the custom error message/handling. Consider rewriting this as a conditional if ! mount ...; then ...; fi so failures are handled deterministically while still logging stderr.

  mount -t ${NAS_TYPE} ${NAS_ADDRESS} ${mount_point} $([[ ! -z "${MOUNT_OPTS}" ]] && echo -o ${MOUNT_OPTS}) >> "$logFile" 2>&1
  if [ $? -eq 0 ]; then
      log -ne "Successfully mounted ${NAS_TYPE} store"
  else
      echo "Failed to mount ${NAS_TYPE} store"

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[jmsperu]

Synced with 4.22 after #13877 and #14006 landed (merge 2a9bbbb). Both sets of settings are kept and the enhancement flags now live in the new runBackupScript.

One behavioural point for review: compression and LUKS encryption rewrite each backup file with qemu-img convert, which would flatten an incremental through its backing chain (or leave an encrypted child whose parent needs its own secret). So when nas.backup.compression.enabled or nas.backup.encryption.enabled is on for a zone, decideChain returns a full backup, and nasbackup.sh rejects -M incremental combined with -c/-e as a backstop (test: decideChainReturnsLegacyFullWhenCompressionOrEncryptionEnabled). Making both features chain-aware can follow separately if there's interest.

Local results (JDK 17): NASBackupProviderTest 28/28, Restore wrapper 22/22, Take wrapper 5/5.

[DaanHoogland]

@abh1sar can you have another look?

[jmsperu]

@abh1sar thanks for the thorough review. All seven inline points are answered in their threads; the fixes are in f82d33f (management server), 1f596ed (nasbackup.sh) and d32f570 (restore wrapper). Local results: NASBackupProviderTest 29/29, LibvirtRestoreBackupCommandWrapperTest 24/24, LibvirtTakeBackupCommandWrapperTest 5/5, checkstyle clean. I also ran the script changes through a PATH-shimmed harness with real qemu-img, comparing old and new, for compression + encryption, -r with and without support, ionice gating, and qemu-img check exit codes 0, 2 and 3. One item is still open: confirming on a real host that blockjob --bandwidth applies to a push-mode backup job (see that thread).

On your question about rotating the passphrase: yes, today a restore only gets the zone's current nas.backup.encryption.passphrase, so changing it makes backups taken with the old one unrestorable until the old value is set back. You were right to raise it. For now the setting's description says so explicitly (f82d33f), and a restore that cannot open the image fails rather than copying it. Proper rotation would mean keeping the passphrase per backup (for example, a reference stored in backup details that keeps old passphrases readable). I would rather do that as a follow-up than grow this PR. Would that work for you?

[github-actions]

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

[jmsperu]

Rebased on the current 4.22 (1d2eec1). The branch history had several merges of 4.22, so it is now one commit on top of 4.22; the message lists the commits it replaces. The only upstream change that conflicted was "Nasbackup quiesce fixes" (#14305):

• TakeBackupCommand and NASBackupProvider: kept both sides, the enhancement details and validation from this PR plus the new quiesce timeout setting and the cleanup failure alert.
• nasbackup.sh: kept both option sets (compression, bandwidth, encryption, verify and --quiesce-timeout) in the variables, usage text and argument parsing, and took upstream's exit 1 after a failed backup job.
• LibvirtTakeBackupCommandWrapperTest was added on both sides: upstream's file stays as is and this PR's enhancement flag tests now live in LibvirtTakeBackupCommandWrapperEnhancementFlagsTest.

core, NAS plugin and KVM plugin build with checkstyle; NASBackupProviderTest 31/31, Libvirt take/restore backup wrapper tests 32/32; bash -n clean. Ready for review.

[jmsperu]

A note on the rebase: the branch had picked up several merges of 4.22, so I rebased it as a single commit on top of 4.22 (its message lists the 13 commits it replaces) rather than replaying each one. Commit hashes quoted in earlier review threads are therefore no longer on the branch. If reviewers would rather see the separate commits again, let me know and I will restore them; otherwise I will keep it as one commit, since the PR would be squash-merged anyway.

[Damans227]

If reviewers would rather see the separate commits again, let me know and I will restore them

@jmsperu one commit is fine. checked the rebased commit against abhisars seven points and they look fixed except the running vm throttle test on a real host

[github-actions]

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

[jmsperu]

Rebased on 4.22 (90ef9b9) as 3586cf2, still one commit. Only the two test files conflicted; both now carry the upstream tests and this PR's tests. The upstream restore position test needed this PR's qemu-img info mock because the encryption check fails closed when the image cannot be read. nasbackup.sh had no upstream changes.

[github-actions]

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

[jmsperu]

Rebased on 4.22 now that #14194 (LINSTOR incremental nasbackup) has landed; the conflicts were all in that change's footprint.

How the overlap was resolved:

• NASBackupProvider.decideChain: the compression/encryption check runs first (either enabled in the zone means a legacy full backup), then the LINSTOR content-based check, then the checkpoint-capable check. So a LINSTOR VM in a zone with compression or encryption on takes a full backup like any other VM, and the content-based pull path is never asked to compress or encrypt.
• NASBackupProviderTest and LibvirtTakeBackupCommandWrapper keep both sets of imports and tests.
• nasbackup.sh backup_stopped_vm: the qemu-img convert line carries this PR's -c / -r options and ionice wrapper, with the upstream exit 1 on failure.

Three small fixes that came out of the rebase:

1. encrypt_backup and verify_backup failures used return 1, but the dispatcher at the bottom of the script does not check the function's return value, so the script could end with status 0 after a failed encryption or integrity check. Both now exit 1 like every other failure path.
2. The pull-mode (content-based) path copies through libvirt's NBD export with qemu-img create and rebase, which have no rate option, and the push-mode blockjob throttle does not reach it either. When a bandwidth limit is configured the script now logs a warning that the pull-mode backup runs unthrottled rather than silently ignoring the setting.
3. The LINSTOR restore tests from Linstor add support for incremental nasbackup #14194 look up replaceBlockDeviceWithBackup by reflection with the pre-change signature and let the qemu-img info probe fail; they now pass the key file parameter and see a plain qcow2 header like the other restore tests.

Tests (offline, -pl api,core,plugins/backup/nas,plugins/hypervisors/kvm): NASBackupProviderTest 36/36, LibvirtRestoreBackupCommandWrapperTest 28/28, LibvirtTakeBackupCommandWrapperTest 18/18, LibvirtTakeBackupCommandWrapperEnhancementFlagsTest 5/5.

https://claude.ai/code/session_01EUU5skTaj95LfpkSbrWhPN