Warning/crash on suspend in VM with Linux 6.16+

[marmarek]

Qubes OS release

Qubes OS 4.2, Qubes OS 4.3

Brief summary

With Linux 6.16-rc6+, sys-whonix crashes on suspend. sys-firewall shows a kernel warning.

Steps to reproduce

1. Install Linux 6.16 for VM
2. Suspend the system

Expected behavior

No crash

Actual behavior

sys-whonix crash:

[2025-08-20 20:08:59] [  302.971448] Kernel panic - not syncing: kernel: panic_on_warn set ...
[2025-08-20 20:08:59] [  302.971474] CPU: 1 UID: 0 PID: 81 Comm: xenwatch Not tainted 6.16.1-1.qubes.1.fc41.x86_64 #1 PREEMPT(full) 
[2025-08-20 20:08:59] [  302.971501] Call Trace:
[2025-08-20 20:08:59] [  302.971510]  <TASK>
[2025-08-20 20:08:59] [  302.971519]  dump_stack_lvl+0x5d/0x80
[2025-08-20 20:08:59] [  302.971533]  panic+0x154/0x30e
[2025-08-20 20:08:59] [  302.971548]  ? pm_restrict_gfp_mask+0x3c/0x50
[2025-08-20 20:08:59] [  302.971564]  check_panic_on_warn.cold+0xf/0x1e
[2025-08-20 20:08:59] [  302.971581]  __warn.cold+0x9f/0xf3
[2025-08-20 20:08:59] [  302.971594]  ? pm_restrict_gfp_mask+0x3c/0x50
[2025-08-20 20:08:59] [  302.971611]  report_bug+0xff/0x140
[2025-08-20 20:08:59] [  302.971627]  ? pm_restrict_gfp_mask+0x3c/0x50
[2025-08-20 20:08:59] [  302.971643]  handle_bug+0x19e/0x260
[2025-08-20 20:08:59] [  302.971658]  exc_invalid_op+0x17/0x70
[2025-08-20 20:08:59] [  302.971670]  asm_exc_invalid_op+0x1a/0x20
[2025-08-20 20:08:59] [  302.971684] RIP: 0010:pm_restrict_gfp_mask+0x3c/0x50
[2025-08-20 20:08:59] [  302.971700] Code: c1 ff ff 84 c0 74 23 8b 05 01 0a c9 02 85 c0 75 25 8b 05 b7 f7 00 02 89 05 f1 09 c9 02 24 3f 89 05 a9 f7 00 02 c3 cc cc cc cc <0f> 0b 8b 05 dc 09 c9 02 85 c0 74 db 0f 0b eb d7 0f 1f 40 00 90 90
[2025-08-20 20:08:59] [  302.971749] RSP: 0018:ffffc90000f43e18 EFLAGS: 00010246
[2025-08-20 20:08:59] [  302.971766] RAX: 0000000000000000 RBX: 0000000000000003 RCX: 0000000000000018
[2025-08-20 20:08:59] [  302.971791] RDX: 0000000000000000 RSI: ffff888005239908 RDI: ffffffff822742c0
[2025-08-20 20:08:59] [  302.971814] RBP: 0000000000000001 R08: ffffffff82272c48 R09: ffff888004c01e20
[2025-08-20 20:08:59] [  302.971836] R10: 7355203a6567616e R11: 6565726620676e69 R12: 0000004689f3cbd0
[2025-08-20 20:08:59] [  302.971858] R13: ffffc90000f43e64 R14: ffffffff80ff0be0 R15: 0000000000000000
[2025-08-20 20:08:59] [  302.971880]  ? __pfx_xenwatch_thread+0x10/0x10
[2025-08-20 20:08:59] [  302.971899]  ? pm_restrict_gfp_mask+0x15/0x50
[2025-08-20 20:08:59] [  302.971918]  dpm_suspend_start+0x51/0x60
[2025-08-20 20:08:59] [  302.971932]  do_suspend.cold+0x6a/0x1b8
[2025-08-20 20:08:59] [  302.971946]  shutdown_handler+0x14c/0x180
[2025-08-20 20:08:59] [  302.971961]  xenwatch_thread+0x92/0x1c0
[2025-08-20 20:08:59] [  302.971975]  ? __pfx_autoremove_wake_function+0x10/0x10
[2025-08-20 20:08:59] [  302.971993]  kthread+0xfc/0x240
[2025-08-20 20:08:59] [  302.972007]  ? __pfx_kthread+0x10/0x10
[2025-08-20 20:08:59] [  302.972020]  ret_from_fork+0x155/0x180
[2025-08-20 20:08:59] [  302.972033]  ? __pfx_kthread+0x10/0x10
[2025-08-20 20:08:59] [  302.972047]  ret_from_fork_asm+0x1a/0x30
[2025-08-20 20:08:59] [  302.972061]  </TASK>
[2025-08-20 20:08:59] [  302.977314] Kernel Offset: disabled

sys-firewall warning:

[2025-08-20 19:25:35] [  332.375610] Freezing user space processes
[2025-08-20 19:25:55] [  352.379897] Freezing user space processes failed after 20.004 seconds (1 tasks refusing to freeze, wq_busy=0):
[2025-08-20 19:25:55] [  352.379945] task:xl              state:D stack:0     pid:466   tgid:466   ppid:1      task_flags:0x400040 flags:0x00004006
[2025-08-20 19:25:55] [  352.379978] Call Trace:
[2025-08-20 19:25:55] [  352.379988]  <TASK>
[2025-08-20 19:25:55] [  352.380010]  __schedule+0x2f3/0x780
[2025-08-20 19:25:55] [  352.380029]  schedule+0x27/0x80
[2025-08-20 19:25:55] [  352.380043]  schedule_preempt_disabled+0x15/0x30
[2025-08-20 19:25:55] [  352.380061]  __mutex_lock.constprop.0+0x49f/0x880
[2025-08-20 19:25:55] [  352.380080]  unregister_xenbus_watch+0x216/0x230
[2025-08-20 19:25:55] [  352.380099]  xenbus_write_watch+0xb9/0x220
[2025-08-20 19:25:55] [  352.380114]  xenbus_file_write+0x131/0x1b0
[2025-08-20 19:25:55] [  352.380128]  vfs_writev+0x26c/0x3d0
[2025-08-20 19:25:55] [  352.380152]  ? do_writev+0xeb/0x110
[2025-08-20 19:25:55] [  352.380165]  do_writev+0xeb/0x110
[2025-08-20 19:25:55] [  352.380179]  do_syscall_64+0x84/0x2c0
[2025-08-20 19:25:55] [  352.380194]  ? do_syscall_64+0x200/0x2c0
[2025-08-20 19:25:55] [  352.380208]  ? generic_handle_irq+0x3f/0x60
[2025-08-20 19:25:55] [  352.380223]  ? syscall_exit_work+0x108/0x140
[2025-08-20 19:25:55] [  352.380239]  ? do_syscall_64+0x200/0x2c0
[2025-08-20 19:25:55] [  352.380254]  ? __irq_exit_rcu+0x4c/0xe0
[2025-08-20 19:25:55] [  352.380276]  entry_SYSCALL_64_after_hwframe+0x76/0x7e
[2025-08-20 19:25:55] [  352.380293] RIP: 0033:0x79b618138642
[2025-08-20 19:25:55] [  352.380307] RSP: 002b:00007fff9a192fc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000014
[2025-08-20 19:25:55] [  352.380331] RAX: ffffffffffffffda RBX: 00000000024fd490 RCX: 000079b618138642
[2025-08-20 19:25:55] [  352.380359] RDX: 0000000000000003 RSI: 00007fff9a193120 RDI: 0000000000000014
[2025-08-20 19:25:55] [  352.380382] RBP: 00007fff9a193000 R08: 0000000000000000 R09: 0000000000000000
[2025-08-20 19:25:55] [  352.380404] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000014
[2025-08-20 19:25:55] [  352.380426] R13: 00007fff9a193120 R14: 0000000000000003 R15: 0000000000000000
[2025-08-20 19:25:55] [  352.380450]  </TASK>
[2025-08-20 19:25:55] [  352.380759] OOM killer enabled.
[2025-08-20 19:25:55] [  352.380772] Restarting tasks: Starting
[2025-08-20 19:25:55] [  352.381581] Restarting tasks: Done
[2025-08-20 19:25:55] [  352.381604] xen:manage: do_suspend: freeze processes failed -16

Additional information

See other comments in QubesOS/qubes-linux-kernel#1157
Upstream report: https://lore.kernel.org/xen-devel/aKiBJeqsYx_4Top5@mail-itl/

[marmarek]

bisect says:

12ffc3b1513ebc1f11ae77d053948504a94a68a6 is the first bad commit
commit 12ffc3b1513ebc1f11ae77d053948504a94a68a6
Author: Mario Limonciello <mario.limonciello@amd.com>
Date:   Fri Jun 13 16:43:44 2025 -0500

    PM: Restrict swap use to later in the suspend sequence
    
    Currently swap is restricted before drivers have had a chance to do
    their prepare() PM callbacks. Restricting swap this early means that if
    a driver needs to evict some content from memory into sawp in it's
    prepare callback, it won't be able to.
    
    On AMD dGPUs this can lead to failed suspends under memory pressure
    situations as all VRAM must be evicted to system memory or swap.
    
    Move the swap restriction to right after all devices have had a chance
    to do the prepare() callback.  If there is any problem with the sequence,
    restore swap in the appropriate dpm resume callbacks or error handling
    paths.
    
    Closes: https://github.com/ROCm/ROCK-Kernel-Driver/issues/174
    Closes: https://gitlab.freedesktop.org/drm/amd/-/issues/2362
    Signed-off-by: Mario Limonciello <mario.limonciello@amd.com>
    Tested-by: Nat Wittstock <nat@fardog.io>
    Tested-by: Lucian Langa <lucilanga@7pot.org>
    Link: https://patch.msgid.link/20250613214413.4127087-1-superm1@kernel.org
    Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>

 drivers/base/power/main.c | 5 ++++-
 include/linux/suspend.h   | 5 +++++
 kernel/kexec_core.c       | 1 +
 kernel/power/hibernate.c  | 3 ---
 kernel/power/power.h      | 5 -----
 kernel/power/suspend.c    | 3 +--
 6 files changed, 11 insertions(+), 11 deletions(-)