Bulk delete of snapshots fails #2930
Description
Is there an existing issue for this?
- There is no existing issue for this bug
Is this happening on an up to date version of Incus?
- This is happening on a supported version of Incus
Incus system details
config:
cluster.https_address: '[fd42:97fe:2948:99e1:1266:6aff:fede:d114]:8443'
core.https_address: :8443
storage.backups_volume: local/backups
storage.images_volume: local/images
api_extensions:
- storage_zfs_remove_snapshots
- container_host_shutdown_timeout
- container_stop_priority
- container_syscall_filtering
- auth_pki
- container_last_used_at
- etag
- patch
- usb_devices
- https_allowed_credentials
- image_compression_algorithm
- directory_manipulation
- container_cpu_time
- storage_zfs_use_refquota
- storage_lvm_mount_options
- network
- profile_usedby
- container_push
- container_exec_recording
- certificate_update
- container_exec_signal_handling
- gpu_devices
- container_image_properties
- migration_progress
- id_map
- network_firewall_filtering
- network_routes
- storage
- file_delete
- file_append
- network_dhcp_expiry
- storage_lvm_vg_rename
- storage_lvm_thinpool_rename
- network_vlan
- image_create_aliases
- container_stateless_copy
- container_only_migration
- storage_zfs_clone_copy
- unix_device_rename
- storage_lvm_use_thinpool
- storage_rsync_bwlimit
- network_vxlan_interface
- storage_btrfs_mount_options
- entity_description
- image_force_refresh
- storage_lvm_lv_resizing
- id_map_base
- file_symlinks
- container_push_target
- network_vlan_physical
- storage_images_delete
- container_edit_metadata
- container_snapshot_stateful_migration
- storage_driver_ceph
- storage_ceph_user_name
- resource_limits
- storage_volatile_initial_source
- storage_ceph_force_osd_reuse
- storage_block_filesystem_btrfs
- resources
- kernel_limits
- storage_api_volume_rename
- network_sriov
- console
- restrict_dev_incus
- migration_pre_copy
- infiniband
- dev_incus_events
- proxy
- network_dhcp_gateway
- file_get_symlink
- network_leases
- unix_device_hotplug
- storage_api_local_volume_handling
- operation_description
- clustering
- event_lifecycle
- storage_api_remote_volume_handling
- nvidia_runtime
- container_mount_propagation
- container_backup
- dev_incus_images
- container_local_cross_pool_handling
- proxy_unix
- proxy_udp
- clustering_join
- proxy_tcp_udp_multi_port_handling
- network_state
- proxy_unix_dac_properties
- container_protection_delete
- unix_priv_drop
- pprof_http
- proxy_haproxy_protocol
- network_hwaddr
- proxy_nat
- network_nat_order
- container_full
- backup_compression
- nvidia_runtime_config
- storage_api_volume_snapshots
- storage_unmapped
- projects
- network_vxlan_ttl
- container_incremental_copy
- usb_optional_vendorid
- snapshot_scheduling
- snapshot_schedule_aliases
- container_copy_project
- clustering_server_address
- clustering_image_replication
- container_protection_shift
- snapshot_expiry
- container_backup_override_pool
- snapshot_expiry_creation
- network_leases_location
- resources_cpu_socket
- resources_gpu
- resources_numa
- kernel_features
- id_map_current
- event_location
- storage_api_remote_volume_snapshots
- network_nat_address
- container_nic_routes
- cluster_internal_copy
- seccomp_notify
- lxc_features
- container_nic_ipvlan
- network_vlan_sriov
- storage_cephfs
- container_nic_ipfilter
- resources_v2
- container_exec_user_group_cwd
- container_syscall_intercept
- container_disk_shift
- storage_shifted
- resources_infiniband
- daemon_storage
- instances
- image_types
- resources_disk_sata
- clustering_roles
- images_expiry
- resources_network_firmware
- backup_compression_algorithm
- ceph_data_pool_name
- container_syscall_intercept_mount
- compression_squashfs
- container_raw_mount
- container_nic_routed
- container_syscall_intercept_mount_fuse
- container_disk_ceph
- virtual-machines
- image_profiles
- clustering_architecture
- resources_disk_id
- storage_lvm_stripes
- vm_boot_priority
- unix_hotplug_devices
- api_filtering
- instance_nic_network
- clustering_sizing
- firewall_driver
- projects_limits
- container_syscall_intercept_hugetlbfs
- limits_hugepages
- container_nic_routed_gateway
- projects_restrictions
- custom_volume_snapshot_expiry
- volume_snapshot_scheduling
- trust_ca_certificates
- snapshot_disk_usage
- clustering_edit_roles
- container_nic_routed_host_address
- container_nic_ipvlan_gateway
- resources_usb_pci
- resources_cpu_threads_numa
- resources_cpu_core_die
- api_os
- container_nic_routed_host_table
- container_nic_ipvlan_host_table
- container_nic_ipvlan_mode
- resources_system
- images_push_relay
- network_dns_search
- container_nic_routed_limits
- instance_nic_bridged_vlan
- network_state_bond_bridge
- usedby_consistency
- custom_block_volumes
- clustering_failure_domains
- resources_gpu_mdev
- console_vga_type
- projects_limits_disk
- network_type_macvlan
- network_type_sriov
- container_syscall_intercept_bpf_devices
- network_type_ovn
- projects_networks
- projects_networks_restricted_uplinks
- custom_volume_backup
- backup_override_name
- storage_rsync_compression
- network_type_physical
- network_ovn_external_subnets
- network_ovn_nat
- network_ovn_external_routes_remove
- tpm_device_type
- storage_zfs_clone_copy_rebase
- gpu_mdev
- resources_pci_iommu
- resources_network_usb
- resources_disk_address
- network_physical_ovn_ingress_mode
- network_ovn_dhcp
- network_physical_routes_anycast
- projects_limits_instances
- network_state_vlan
- instance_nic_bridged_port_isolation
- instance_bulk_state_change
- network_gvrp
- instance_pool_move
- gpu_sriov
- pci_device_type
- storage_volume_state
- network_acl
- migration_stateful
- disk_state_quota
- storage_ceph_features
- projects_compression
- projects_images_remote_cache_expiry
- certificate_project
- network_ovn_acl
- projects_images_auto_update
- projects_restricted_cluster_target
- images_default_architecture
- network_ovn_acl_defaults
- gpu_mig
- project_usage
- network_bridge_acl
- warnings
- projects_restricted_backups_and_snapshots
- clustering_join_token
- clustering_description
- server_trusted_proxy
- clustering_update_cert
- storage_api_project
- server_instance_driver_operational
- server_supported_storage_drivers
- event_lifecycle_requestor_address
- resources_gpu_usb
- clustering_evacuation
- network_ovn_nat_address
- network_bgp
- network_forward
- custom_volume_refresh
- network_counters_errors_dropped
- metrics
- image_source_project
- clustering_config
- network_peer
- linux_sysctl
- network_dns
- ovn_nic_acceleration
- certificate_self_renewal
- instance_project_move
- storage_volume_project_move
- cloud_init
- network_dns_nat
- database_leader
- instance_all_projects
- clustering_groups
- ceph_rbd_du
- instance_get_full
- qemu_metrics
- gpu_mig_uuid
- event_project
- clustering_evacuation_live
- instance_allow_inconsistent_copy
- network_state_ovn
- storage_volume_api_filtering
- image_restrictions
- storage_zfs_export
- network_dns_records
- storage_zfs_reserve_space
- network_acl_log
- storage_zfs_blocksize
- metrics_cpu_seconds
- instance_snapshot_never
- certificate_token
- instance_nic_routed_neighbor_probe
- event_hub
- agent_nic_config
- projects_restricted_intercept
- metrics_authentication
- images_target_project
- images_all_projects
- cluster_migration_inconsistent_copy
- cluster_ovn_chassis
- container_syscall_intercept_sched_setscheduler
- storage_lvm_thinpool_metadata_size
- storage_volume_state_total
- instance_file_head
- instances_nic_host_name
- image_copy_profile
- container_syscall_intercept_sysinfo
- clustering_evacuation_mode
- resources_pci_vpd
- qemu_raw_conf
- storage_cephfs_fscache
- network_load_balancer
- vsock_api
- instance_ready_state
- network_bgp_holdtime
- storage_volumes_all_projects
- metrics_memory_oom_total
- storage_buckets
- storage_buckets_create_credentials
- metrics_cpu_effective_total
- projects_networks_restricted_access
- storage_buckets_local
- loki
- acme
- internal_metrics
- cluster_join_token_expiry
- remote_token_expiry
- init_preseed
- storage_volumes_created_at
- cpu_hotplug
- projects_networks_zones
- network_txqueuelen
- cluster_member_state
- instances_placement_scriptlet
- storage_pool_source_wipe
- zfs_block_mode
- instance_generation_id
- disk_io_cache
- amd_sev
- storage_pool_loop_resize
- migration_vm_live
- ovn_nic_nesting
- oidc
- network_ovn_l3only
- ovn_nic_acceleration_vdpa
- cluster_healing
- instances_state_total
- auth_user
- security_csm
- instances_rebuild
- numa_cpu_placement
- custom_volume_iso
- network_allocations
- zfs_delegate
- storage_api_remote_volume_snapshot_copy
- operations_get_query_all_projects
- metadata_configuration
- syslog_socket
- event_lifecycle_name_and_project
- instances_nic_limits_priority
- disk_initial_volume_configuration
- operation_wait
- image_restriction_privileged
- cluster_internal_custom_volume_copy
- disk_io_bus
- storage_cephfs_create_missing
- instance_move_config
- ovn_ssl_config
- certificate_description
- disk_io_bus_virtio_blk
- loki_config_instance
- instance_create_start
- clustering_evacuation_stop_options
- boot_host_shutdown_action
- agent_config_drive
- network_state_ovn_lr
- image_template_permissions
- storage_bucket_backup
- storage_lvm_cluster
- shared_custom_block_volumes
- auth_tls_jwt
- oidc_claim
- device_usb_serial
- numa_cpu_balanced
- image_restriction_nesting
- network_integrations
- instance_memory_swap_bytes
- network_bridge_external_create
- network_zones_all_projects
- storage_zfs_vdev
- container_migration_stateful
- profiles_all_projects
- instances_scriptlet_get_instances
- instances_scriptlet_get_cluster_members
- instances_scriptlet_get_project
- network_acl_stateless
- instance_state_started_at
- networks_all_projects
- network_acls_all_projects
- storage_buckets_all_projects
- resources_load
- instance_access
- project_access
- projects_force_delete
- resources_cpu_flags
- disk_io_bus_cache_filesystem
- instance_oci
- clustering_groups_config
- instances_lxcfs_per_instance
- clustering_groups_vm_cpu_definition
- disk_volume_subpath
- projects_limits_disk_pool
- network_ovn_isolated
- qemu_raw_qmp
- network_load_balancer_health_check
- oidc_scopes
- network_integrations_peer_name
- qemu_scriptlet
- instance_auto_restart
- storage_lvm_metadatasize
- ovn_nic_promiscuous
- ovn_nic_ip_address_none
- instances_state_os_info
- network_load_balancer_state
- instance_nic_macvlan_mode
- storage_lvm_cluster_create
- network_ovn_external_interfaces
- instances_scriptlet_get_instances_count
- cluster_rebalance
- custom_volume_refresh_exclude_older_snapshots
- storage_initial_owner
- storage_live_migration
- instance_console_screenshot
- image_import_alias
- authorization_scriptlet
- console_force
- network_ovn_state_addresses
- network_bridge_acl_devices
- instance_debug_memory
- init_preseed_storage_volumes
- init_preseed_profile_project
- instance_nic_routed_host_address
- instance_smbios11
- api_filtering_extended
- acme_dns01
- security_iommu
- network_ipv4_dhcp_routes
- network_state_ovn_ls
- network_dns_nameservers
- acme_http01_port
- network_ovn_ipv4_dhcp_expiry
- instance_state_cpu_time
- network_io_bus
- disk_io_bus_usb
- storage_driver_linstor
- instance_oci_entrypoint
- network_address_set
- server_logging
- network_forward_snat
- memory_hotplug
- instance_nic_routed_host_tables
- instance_publish_split
- init_preseed_certificates
- custom_volume_sftp
- network_ovn_external_nic_address
- network_physical_gateway_hwaddr
- backup_s3_upload
- snapshot_manual_expiry
- resources_cpu_address_sizes
- disk_attached
- limits_memory_hotplug
- disk_wwn
- server_logging_webhook
- storage_driver_truenas
- container_disk_tmpfs
- instance_limits_oom
- backup_override_config
- network_ovn_tunnels
- init_preseed_cluster_groups
- usb_attached
- backup_iso
- instance_systemd_credentials
- cluster_group_usedby
- bpf_token_delegation
- file_storage_volume
- network_hwaddr_pattern
- storage_volume_full
- storage_bucket_full
- device_pci_firmware
- resources_serial
- ovn_nic_limits
- storage_lvmcluster_qcow2
- oidc_allowed_subnets
- file_delete_force
- nic_sriov_select_ext
- network_zones_dns_contact
- nic_attached_connected
api_status: stable
api_version: "1.0"
auth: trusted
public: false
auth_methods:
- tls
- oidc
auth_user_name: e74b9c17ada8480d8985fc21ef665dd3c205b7ad17687ee12da784545dae6cd8
auth_user_method: tls
environment:
addresses:
- 10.10.21.111:8443
- '[fd42:97fe:2948:99e1:1266:6aff:fede:d114]:8443'
- 10.10.22.3:8443
- '[fd42:f076:f84b:b2b6:1266:6aff:fe20:4481]:8443'
- 10.50.50.1:8443
- '[fd42:5a62:ea55:4c65::1]:8443'
- '[fd42:fa55:bbd2:ca78:1266:6aff:fe6e:5a65]:8443'
architectures:
- x86_64
- i686
driver: qemu | lxc
driver_version: 10.2.0 | 6.0.5
firewall: nftables
kernel: Linux
kernel_architecture: x86_64
kernel_features:
idmapped_mounts: "true"
netnsid_getifaddrs: "true"
seccomp_listener: "true"
seccomp_listener_continue: "true"
uevent_injection: "true"
unpriv_binfmt: "true"
unpriv_fscaps: "true"
kernel_version: 6.18.9-zabbly+
lxc_features:
cgroup2: "true"
core_scheduling: "true"
devpts_fd: "true"
idmapped_mounts_v2: "true"
mount_injection_file: "true"
network_gateway_device_route: "true"
network_ipvlan: "true"
network_l2proxy: "true"
network_phys_macvlan_mtu: "true"
network_veth_router: "true"
pidfd: "true"
seccomp_allow_deny_syntax: "true"
seccomp_notify: "true"
seccomp_proxy_send_notify_fd: "true"
os_name: IncusOS
os_version: "202602110253"
project: default
server: incus
server_clustered: true
server_event_mode: full-mesh
server_name: incusos-1
server_pid: 1314
server_version: "6.21"
storage: lvmcluster | zfs
storage_version: 2.03.31(2) (2025-02-27) / 1.02.205 (2025-02-27) / 4.50.0 | 2.4.0-1
storage_supported_drivers:
- name: lvm
version: 2.03.31(2) (2025-02-27) / 1.02.205 (2025-02-27) / 4.50.0
remote: false
- name: lvmcluster
version: 2.03.31(2) (2025-02-27) / 1.02.205 (2025-02-27) / 4.50.0
remote: true
- name: btrfs
version: "6.14"
remote: false
- name: dir
version: "1"
remote: false
- name: zfs
version: 2.4.0-1
remote: false
Instance details
architecture: x86_64
config:
image.architecture: amd64
image.description: Debian trixie amd64 (20260209_06:20)
image.os: Debian
image.release: trixie
image.serial: "20260209_06:20"
image.type: disk-kvm.img
image.variant: default
volatile.base_image: 7b9a082ad2aaea28d92b4113c650b5370755bf20c1aa61bdc7a7bf233e59ea4a
volatile.cloud-init.instance-id: 9a30c80e-acd3-47d3-be39-833c3759e40c
volatile.eth0.host_name: tap3d1ddf38
volatile.eth0.hwaddr: 10:66:6a:24:70:08
volatile.last_state.power: RUNNING
volatile.uuid: 487380f3-b4f8-46bc-a3fe-65a54f579d26
volatile.uuid.generation: 487380f3-b4f8-46bc-a3fe-65a54f579d26
volatile.vm.definition: pc-q35-10.2
volatile.vm.rtc_adjustment: "-13"
volatile.vm.rtc_offset: "-1"
volatile.vsock_id: "3476836486"
devices: {}
ephemeral: false
profiles:
- default
stateful: false
description: ""
Instance log
incus monitor --pretty
DEBUG [2026-02-12T09:40:15Z] [incusos-1] Handling API request ip="10.10.21.4:60806" method=DELETE protocol=tls url="/1.0/instances/renewing-manatee/snapshots/snap2?project=default" username=e74b9c17ada8480d8985fc21ef665dd3c205b7ad17687ee12da784545dae6cd8
DEBUG [2026-02-12T09:40:15Z] [incusos-1] Handling API request ip="10.10.21.4:60806" method=DELETE protocol=tls url="/1.0/instances/renewing-manatee/snapshots/snap1?project=default" username=e74b9c17ada8480d8985fc21ef665dd3c205b7ad17687ee12da784545dae6cd8
DEBUG [2026-02-12T09:40:15Z] [incusos-1] Handling API request ip="10.10.21.4:60806" method=DELETE protocol=tls url="/1.0/instances/renewing-manatee/snapshots/snap0?project=default" username=e74b9c17ada8480d8985fc21ef665dd3c205b7ad17687ee12da784545dae6cd8
DEBUG [2026-02-12T09:40:15Z] [incusos-1] Matched trusted cert fingerprint=e74b9c17ada8480d8985fc21ef665dd3c205b7ad17687ee12da784545dae6cd8 subject="CN=stephen@hoth,O=Linux Containers"
DEBUG [2026-02-12T09:40:15Z] [incusos-1] Matched trusted cert fingerprint=e74b9c17ada8480d8985fc21ef665dd3c205b7ad17687ee12da784545dae6cd8 subject="CN=stephen@hoth,O=Linux Containers"
DEBUG [2026-02-12T09:40:15Z] [incusos-1] Sending heartbeat request address="[fd42:97fe:2948:99e1:1266:6aff:fe2f:bf13]:8443"
INFO [2026-02-12T09:40:15Z] [incusos-1] Deleting instance snapshot created="2026-02-12 09:38:14.362499474 +0000 UTC" ephemeral=false instance=renewing-manatee/snap1 instanceType=virtual-machine project=default used="0001-01-01 00:00:00 +0000 UTC"
DEBUG [2026-02-12T09:40:15Z] [incusos-1] New operation class=task description="Deleting snapshot" operation=729369ed-d0ef-44bc-b7b4-c3b2802fdc37 project=default
DEBUG [2026-02-12T09:40:15Z] [incusos-1] Started operation class=task description="Deleting snapshot" operation=729369ed-d0ef-44bc-b7b4-c3b2802fdc37 project=default
DEBUG [2026-02-12T09:40:15Z] [incusos-1] Instance operation lock created action=delete instance=renewing-manatee/snap1 project=default reusable=false
INFO [2026-02-12T09:40:15Z] ID: 729369ed-d0ef-44bc-b7b4-c3b2802fdc37, Class: task, Description: Deleting snapshot CreatedAt="2026-02-12 09:40:15.078913042 +0000 UTC" Err= Location=incusos-1 MayCancel=false Metadata="map[]" Resources="map[instances:[/1.0/instances/renewing-manatee] instances_snapshots:[/1.0/instances/renewing-manatee/snapshots/snap1]]" Status=Pending StatusCode=Pending UpdatedAt="2026-02-12 09:40:15.078913042 +0000 UTC"
INFO [2026-02-12T09:40:15Z] ID: 729369ed-d0ef-44bc-b7b4-c3b2802fdc37, Class: task, Description: Deleting snapshot CreatedAt="2026-02-12 09:40:15.078913042 +0000 UTC" Err= Location=incusos-1 MayCancel=false Metadata="map[]" Resources="map[instances:[/1.0/instances/renewing-manatee] instances_snapshots:[/1.0/instances/renewing-manatee/snapshots/snap1]]" Status=Running StatusCode=Running UpdatedAt="2026-02-12 09:40:15.078913042 +0000 UTC"
DEBUG [2026-02-12T09:40:15Z] [incusos-1] Started operation class=task description="Deleting snapshot" operation=5c28e82b-dfdc-4246-9784-b520146520b9 project=default
DEBUG [2026-02-12T09:40:15Z] [incusos-1] Successful heartbeat remote="[fd42:97fe:2948:99e1:1266:6aff:fe2f:bf13]:8443"
DEBUG [2026-02-12T09:40:15Z] [incusos-1] New operation class=task description="Deleting snapshot" operation=5c28e82b-dfdc-4246-9784-b520146520b9 project=default
INFO [2026-02-12T09:40:15Z] ID: 5c28e82b-dfdc-4246-9784-b520146520b9, Class: task, Description: Deleting snapshot CreatedAt="2026-02-12 09:40:15.082954908 +0000 UTC" Err= Location=incusos-1 MayCancel=false Metadata="map[]" Resources="map[instances:[/1.0/instances/renewing-manatee] instances_snapshots:[/1.0/instances/renewing-manatee/snapshots/snap0]]" Status=Pending StatusCode=Pending UpdatedAt="2026-02-12 09:40:15.082954908 +0000 UTC"
INFO [2026-02-12T09:40:15Z] ID: 5c28e82b-dfdc-4246-9784-b520146520b9, Class: task, Description: Deleting snapshot CreatedAt="2026-02-12 09:40:15.082954908 +0000 UTC" Err= Location=incusos-1 MayCancel=false Metadata="map[]" Resources="map[instances:[/1.0/instances/renewing-manatee] instances_snapshots:[/1.0/instances/renewing-manatee/snapshots/snap0]]" Status=Running StatusCode=Running UpdatedAt="2026-02-12 09:40:15.082954908 +0000 UTC"
DEBUG [2026-02-12T09:40:15Z] [incusos-1] Instance operation lock created action=delete instance=renewing-manatee/snap0 project=default reusable=false
INFO [2026-02-12T09:40:15Z] [incusos-1] Deleting instance snapshot created="2026-02-12 09:38:08.858755833 +0000 UTC" ephemeral=false instance=renewing-manatee/snap0 instanceType=virtual-machine project=default used="0001-01-01 00:00:00 +0000 UTC"
INFO [2026-02-12T09:40:15Z] [incusos-1] Deleting instance snapshot created="2026-02-12 09:38:22.963964966 +0000 UTC" ephemeral=false instance=renewing-manatee/snap2 instanceType=virtual-machine project=default used="0001-01-01 00:00:00 +0000 UTC"
DEBUG [2026-02-12T09:40:15Z] [incusos-1] Instance operation lock created action=delete instance=renewing-manatee/snap2 project=default reusable=false
DEBUG [2026-02-12T09:40:15Z] [incusos-1] Started operation class=task description="Deleting snapshot" operation=e9c3b988-16b3-4106-a4c7-7f1d7e27677a project=default
DEBUG [2026-02-12T09:40:15Z] [incusos-1] New operation class=task description="Deleting snapshot" operation=e9c3b988-16b3-4106-a4c7-7f1d7e27677a project=default
INFO [2026-02-12T09:40:15Z] ID: e9c3b988-16b3-4106-a4c7-7f1d7e27677a, Class: task, Description: Deleting snapshot CreatedAt="2026-02-12 09:40:15.074774989 +0000 UTC" Err= Location=incusos-1 MayCancel=false Metadata="map[]" Resources="map[instances:[/1.0/instances/renewing-manatee] instances_snapshots:[/1.0/instances/renewing-manatee/snapshots/snap2]]" Status=Running StatusCode=Running UpdatedAt="2026-02-12 09:40:15.074774989 +0000 UTC"
INFO [2026-02-12T09:40:15Z] ID: e9c3b988-16b3-4106-a4c7-7f1d7e27677a, Class: task, Description: Deleting snapshot CreatedAt="2026-02-12 09:40:15.074774989 +0000 UTC" Err= Location=incusos-1 MayCancel=false Metadata="map[]" Resources="map[instances:[/1.0/instances/renewing-manatee] instances_snapshots:[/1.0/instances/renewing-manatee/snapshots/snap2]]" Status=Pending StatusCode=Pending UpdatedAt="2026-02-12 09:40:15.074774989 +0000 UTC"
DEBUG [2026-02-12T09:40:15Z] [incusos-1] DeleteInstanceSnapshot started driver=lvmcluster instance=renewing-manatee/snap0 pool=shared project=default
DEBUG [2026-02-12T09:40:15Z] [incusos-1] Deleting instance snapshot volume driver=lvmcluster instance=renewing-manatee/snap0 pool=shared project=default snapshotName=snap0 volName=renewing-manatee
DEBUG [2026-02-12T09:40:15Z] [incusos-1] DeleteInstanceSnapshot started driver=lvmcluster instance=renewing-manatee/snap2 pool=shared project=default
DEBUG [2026-02-12T09:40:15Z] [incusos-1] Deleting instance snapshot volume driver=lvmcluster instance=renewing-manatee/snap2 pool=shared project=default snapshotName=snap2 volName=renewing-manatee
DEBUG [2026-02-12T09:40:15Z] [incusos-1] Deleting instance snapshot volume driver=lvmcluster instance=renewing-manatee/snap1 pool=shared project=default snapshotName=snap1 volName=renewing-manatee
DEBUG [2026-02-12T09:40:15Z] [incusos-1] DeleteInstanceSnapshot started driver=lvmcluster instance=renewing-manatee/snap1 pool=shared project=default
DEBUG [2026-02-12T09:40:15Z] [incusos-1] QCOW2 blockdev chain: blockdev="[incus_root incus_root_overlay0 incus_root_overlay1 incus_root_overlay2]" instance=renewing-manatee instanceType=virtual-machine project=default snapshotIndex=1
DEBUG [2026-02-12T09:40:15Z] [incusos-1] QCOW2 blockdev chain: blockdev="[incus_root incus_root_overlay0 incus_root_overlay1 incus_root_overlay2]" instance=renewing-manatee instanceType=virtual-machine project=default snapshotIndex=2
DEBUG [2026-02-12T09:40:15Z] [incusos-1] DeleteInstanceSnapshot finished driver=lvmcluster instance=renewing-manatee/snap2 pool=shared project=default
DEBUG [2026-02-12T09:40:15Z] [incusos-1] Instance operation lock finished action=delete err="<nil>" instance=renewing-manatee/snap2 project=default reusable=false
INFO [2026-02-12T09:40:15Z] ID: e9c3b988-16b3-4106-a4c7-7f1d7e27677a, Class: task, Description: Deleting snapshot CreatedAt="2026-02-12 09:40:15.074774989 +0000 UTC" Err="GenericError: Need a root block node" Location=incusos-1 MayCancel=false Metadata="map[]" Resources="map[instances:[/1.0/instances/renewing-manatee] instances_snapshots:[/1.0/instances/renewing-manatee/snapshots/snap2]]" Status=Failure StatusCode=Failure UpdatedAt="2026-02-12 09:40:15.074774989 +0000 UTC"
DEBUG [2026-02-12T09:40:15Z] [incusos-1] Failure for operation class=task description="Deleting snapshot" err="GenericError: Need a root block node" operation=e9c3b988-16b3-4106-a4c7-7f1d7e27677a project=default
DEBUG [2026-02-12T09:40:15Z] [incusos-1] QCOW2 blockdev chain: blockdev="[incus_root incus_root_overlay0 incus_root_overlay1 incus_root_overlay2]" instance=renewing-manatee instanceType=virtual-machine project=default snapshotIndex=0
DEBUG [2026-02-12T09:40:15Z] [incusos-1] Failure for operation class=task description="Deleting snapshot" err="GenericError: Need a root block node" operation=5c28e82b-dfdc-4246-9784-b520146520b9 project=default
DEBUG [2026-02-12T09:40:15Z] [incusos-1] DeleteInstanceSnapshot finished driver=lvmcluster instance=renewing-manatee/snap0 pool=shared project=default
INFO [2026-02-12T09:40:15Z] ID: 5c28e82b-dfdc-4246-9784-b520146520b9, Class: task, Description: Deleting snapshot CreatedAt="2026-02-12 09:40:15.082954908 +0000 UTC" Err="GenericError: Need a root block node" Location=incusos-1 MayCancel=false Metadata="map[]" Resources="map[instances:[/1.0/instances/renewing-manatee] instances_snapshots:[/1.0/instances/renewing-manatee/snapshots/snap0]]" Status=Failure StatusCode=Failure UpdatedAt="2026-02-12 09:40:15.082954908 +0000 UTC"
DEBUG [2026-02-12T09:40:15Z] [incusos-1] Instance operation lock finished action=delete err="<nil>" instance=renewing-manatee/snap0 project=default reusable=false
DEBUG [2026-02-12T09:40:16Z] [incusos-1] Unmounted logical volume snapshot driver=lvmcluster path=/var/lib/incus/storage-pools/shared/virtual-machines-snapshots/renewing-manatee/snap1 pool=shared
DEBUG [2026-02-12T09:40:16Z] [incusos-1] Mounted logical volume dev=/dev/dm-10 driver=lvmcluster options=discard path=/var/lib/incus/storage-pools/shared/virtual-machines/renewing-manatee.incustmp pool=shared volName=renewing-manatee
DEBUG [2026-02-12T09:40:16Z] [incusos-1] Skipping unmount as in use driver=lvmcluster pool=shared refCount=1 volName=renewing-manatee
DEBUG [2026-02-12T09:40:17Z] [incusos-1] Sending heartbeat request address="[fd42:97fe:2948:99e1:1266:6aff:fef2:69a6]:8443"
DEBUG [2026-02-12T09:40:17Z] [incusos-1] Logical volume removed dev=vg0/virtual-machines_renewing--manatee-snap2.block driver=lvmcluster pool=shared
DEBUG [2026-02-12T09:40:17Z] [incusos-1] Successful heartbeat remote="[fd42:97fe:2948:99e1:1266:6aff:fef2:69a6]:8443"
DEBUG [2026-02-12T09:40:17Z] [incusos-1] Logical volume renamed dev=vg0/virtual-machines_renewing--manatee-snap1.block driver=lvmcluster new_dev=vg0/virtual-machines_renewing--manatee-snap2.block pool=shared
DEBUG [2026-02-12T09:40:17Z] [incusos-1] DeleteInstanceSnapshot finished driver=lvmcluster instance=renewing-manatee/snap1 pool=shared project=default
INFO [2026-02-12T09:40:17Z] [incusos-1] Deleted instance snapshot created="2026-02-12 09:38:14.362499474 +0000 UTC" ephemeral=false instance=renewing-manatee/snap1 instanceType=virtual-machine project=default used="0001-01-01 00:00:00 +0000 UTC"
INFO [2026-02-12T09:40:17Z] Action: instance-snapshot-deleted, Source: /1.0/instances/renewing-manatee/snapshots/snap1, Requestor: tls/e74b9c17ada8480d8985fc21ef665dd3c205b7ad17687ee12da784545dae6cd8 (10.10.21.4)
DEBUG [2026-02-12T09:40:17Z] [incusos-1] Rebalancing member roles in heartbeat local="[fd42:97fe:2948:99e1:1266:6aff:fede:d114]:8443"
DEBUG [2026-02-12T09:40:17Z] [incusos-1] Handling API request ip="10.10.21.110:60612" method=GET protocol=tls url="/1.0/instances/snap1?project=default&recursion=1" username=a6ae926a7fac6b70ea6bfc0e30f1fb8b40b3b10d0d3e2224a2db5616b320cae4
DEBUG [2026-02-12T09:40:17Z] [incusos-1] Matched trusted cert fingerprint=a6ae926a7fac6b70ea6bfc0e30f1fb8b40b3b10d0d3e2224a2db5616b320cae4 subject="CN=root@bcd998c7-8c48-4ca7-8ff9-9444a09c4776,O=Linux Containers"
DEBUG [2026-02-12T09:40:17Z] [incusos-1] UpdateInstanceBackupFile started driver=lvmcluster instance=renewing-manatee pool=shared project=default
DEBUG [2026-02-12T09:40:17Z] [incusos-1] CacheInstanceSnapshots started driver=lvmcluster instance=renewing-manatee pool=shared project=default
DEBUG [2026-02-12T09:40:17Z] [incusos-1] CacheInstanceSnapshots finished driver=lvmcluster instance=renewing-manatee pool=shared project=default
DEBUG [2026-02-12T09:40:17Z] [incusos-1] Completed heartbeat round duration=2.648986942s
DEBUG [2026-02-12T09:40:17Z] [incusos-1] Success for operation class=task description="Deleting snapshot" operation=729369ed-d0ef-44bc-b7b4-c3b2802fdc37 project=default
DEBUG [2026-02-12T09:40:17Z] [incusos-1] Skipping unmount as in use driver=lvmcluster pool=shared refCount=1 volName=renewing-manatee
INFO [2026-02-12T09:40:17Z] ID: 729369ed-d0ef-44bc-b7b4-c3b2802fdc37, Class: task, Description: Deleting snapshot CreatedAt="2026-02-12 09:40:15.078913042 +0000 UTC" Err= Location=incusos-1 MayCancel=false Metadata="map[]" Resources="map[instances:[/1.0/instances/renewing-manatee] instances_snapshots:[/1.0/instances/renewing-manatee/snapshots/snap1]]" Status=Success StatusCode=Success UpdatedAt="2026-02-12 09:40:15.078913042 +0000 UTC"
DEBUG [2026-02-12T09:40:17Z] [incusos-1] Instance operation lock finished action=delete err="<nil>" instance=renewing-manatee/snap1 project=default reusable=false
DEBUG [2026-02-12T09:40:17Z] [incusos-1] UpdateInstanceBackupFile finished driver=lvmcluster instance=renewing-manatee pool=shared project=default
DEBUG [2026-02-12T09:40:17Z] [incusos-1] Handling API request ip="10.10.21.4:60806" method=GET protocol=tls url="/1.0/instances/renewing-manatee?project=default&recursion=2" username=e74b9c17ada8480d8985fc21ef665dd3c205b7ad17687ee12da784545dae6cd8
DEBUG [2026-02-12T09:40:17Z] [incusos-1] Matched trusted cert fingerprint=e74b9c17ada8480d8985fc21ef665dd3c205b7ad17687ee12da784545dae6cd8 subject="CN=stephen@hoth,O=Linux Containers"
DEBUG [2026-02-12T09:40:17Z] [incusos-1] CacheInstanceSnapshots started driver=lvmcluster instance=renewing-manatee pool=shared project=default
DEBUG [2026-02-12T09:40:17Z] [incusos-1] Connecting to a VM agent over a VM socket
DEBUG [2026-02-12T09:40:17Z] [incusos-1] CacheInstanceSnapshots finished driver=lvmcluster instance=renewing-manatee pool=shared project=default
Current behavior
Bulk deletion of snapshots on lvmcluster fails:
From incus monitor --pretty
INFO [2026-02-12T09:40:15Z] ID: e9c3b988-16b3-4106-a4c7-7f1d7e27677a, Class: task, Description: Deleting snapshot CreatedAt="2026-02-12 09:40:15.074774989 +0000 UTC" Err="GenericError: Need a root block node" Location=incusos-1 MayCancel=false Metadata="map[]" Resources="map[instances:[/1.0/instances/renewing-manatee] instances_snapshots:[/1.0/instances/renewing-manatee/snapshots/snap2]]" Status=Failure StatusCode=Failure UpdatedAt="2026-02-12 09:40:15.074774989 +0000 UTC"
DEBUG [2026-02-12T09:40:15Z] [incusos-1] Failure for operation class=task description="Deleting snapshot" err="GenericError: Need a root block node" operation=e9c3b988-16b3-4106-a4c7-7f1d7e27677a project=default
DEBUG [2026-02-12T09:40:15Z] [incusos-1] QCOW2 blockdev chain: blockdev="[incus_root incus_root_overlay0 incus_root_overlay1 incus_root_overlay2]" instance=renewing-manatee instanceType=virtual-machine project=default snapshotIndex=0
DEBUG [2026-02-12T09:40:15Z] [incusos-1] Failure for operation class=task description="Deleting snapshot" err="GenericError: Need a root block node" operation=5c28e82b-dfdc-4246-9784-b520146520b9 project=default
DEBUG [2026-02-12T09:40:15Z] [incusos-1] DeleteInstanceSnapshot finished driver=lvmcluster instance=renewing-manatee/snap0 pool=shared project=default
INFO [2026-02-12T09:40:15Z] ID: 5c28e82b-dfdc-4246-9784-b520146520b9, Class: task, Description: Deleting snapshot CreatedAt="2026-02-12 09:40:15.082954908 +0000 UTC" Err="GenericError: Need a root block node" Location=incusos-1 MayCancel=false Metadata="map[]" Resources="map[instances:[/1.0/instances/renewing-manatee] instances_snapshots:[/1.0/instances/renewing-manatee/snapshots/snap0]]" Status=Failure StatusCode=Failure UpdatedAt="2026-02-12 09:40:15.082954908 +0000 UTC"
Only one of three snapshots where deleted.
Attempting to bulk delete the remaining two snapshots results in the same scenario.
One snapshot is deleted and one remains due to the same error.
Suspect some kind of locking is allowing only one operation at a time.
I haven't been able to reproduce this on zfs so I assume its only lvmcluster.
Expected behavior
I would expect to be able to bulk delete snapshots.
Steps to reproduce
- Create an instance on an lvmcluster storage pool.
- Create three snapshots.
- Attempt to bulk delete them from the UI.
Metadata
Metadata
Assignees
Labels
No labels