nohang-desktop stopped working?

[dim-geo]

Hi, I use cachy-os kernel patches & zram and I have noticed that nohang-desktop is not protecting from OOM anymore.
kswapd0 is triggering oom-killer.

How can I debug this further? Can I change some limits in nohang-desktop to force nohang to intervene earlier?

Logs:

Dec 23 12:11:20 gentoo nohang-desktop[81157]: PSI:   5.76 | MemAvail:  9578 M, 40.8 % | SwapFree:  528 M,   9.0 % | MemUsedZram:   479 M,  2.0 % | dMem:   -36 M/s
Dec 23 12:12:21 gentoo nohang-desktop[81157]: PSI:   2.82 | MemAvail:  9880 M, 42.1 % | SwapFree:  951 M,  16.2 % | MemUsedZram:   480 M,  2.0 % | dMem:    12 M/s
...
Dec 23 12:13:22 gentoo nohang-desktop[81157]: PSI:   7.95 | MemAvail:  9149 M, 39.0 % | SwapFree:  654 M,  11.1 % | MemUsedZram:   479 M,  2.0 % | dMem:   -17 M/s
Dec 23 12:13:33 gentoo kernel: kswapd0 invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=0
Dec 23 12:13:33 gentoo kernel: CPU: 0 UID: 0 PID: 74 Comm: kswapd0 Tainted: G S      W  OE      6.12.6-cachyos-6.12.6-x86_64 #1
Dec 23 12:13:33 gentoo kernel: Tainted: [S]=CPU_OUT_OF_SPEC, [W]=WARN, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
Dec 23 12:13:33 gentoo kernel: Hardware name:
Dec 23 12:13:33 gentoo kernel: Sched_ext: flash (enabled+all), task: runnable_at=-1ms
Dec 23 12:13:33 gentoo kernel: Call Trace:
Dec 23 12:13:33 gentoo kernel:  <TASK>
Dec 23 12:13:33 gentoo kernel:  dump_stack_lvl+0x56/0x80
Dec 23 12:13:33 gentoo kernel:  dump_header+0x50/0x36d
Dec 23 12:13:33 gentoo kernel:  ? bpf_task_storage_get+0xec/0x100
Dec 23 12:13:33 gentoo kernel:  ? ktime_get_mono_fast_ns+0x30/0xb0
Dec 23 12:13:33 gentoo kernel:  ? ___ratelimit+0x99/0xf0
Dec 23 12:13:33 gentoo kernel:  oom_kill_process.cold+0x8/0x8a
Dec 23 12:13:33 gentoo kernel:  out_of_memory+0x102/0x410
Dec 23 12:13:33 gentoo kernel:  balance_pgdat+0x778/0xb30
Dec 23 12:13:33 gentoo kernel:  kswapd+0x1cd/0x350
Dec 23 12:13:33 gentoo kernel:  ? complete+0x80/0x80
Dec 23 12:13:33 gentoo kernel:  ? balance_pgdat+0xb30/0xb30
Dec 23 12:13:33 gentoo kernel:  kthread+0xcb/0x100
Dec 23 12:13:33 gentoo kernel:  ? kthread_park+0x80/0x80
Dec 23 12:13:33 gentoo kernel:  ret_from_fork+0x2d/0x50
Dec 23 12:13:33 gentoo kernel:  ? kthread_park+0x80/0x80
Dec 23 12:13:33 gentoo kernel:  ret_from_fork_asm+0x11/0x20
Dec 23 12:13:33 gentoo kernel:  </TASK>
Dec 23 12:13:33 gentoo kernel: Mem-Info:
Dec 23 12:13:33 gentoo kernel: active_anon:1399496 inactive_anon:1800837 isolated_anon:0
                                   active_file:364556 inactive_file:538027 isolated_file:59
                                   unevictable:7488 dirty:1723 writeback:0
                                   slab_reclaimable:65189 slab_unreclaimable:82292
                                   mapped:134366 shmem:5761 pagetables:22881
                                   sec_pagetables:5215 bounce:0
                                   kernel_misc_reclaimable:0
                                   free:1303866 free_pcp:1856 free_cma:0
Dec 23 12:13:33 gentoo kernel: Node 0 active_anon:5597984kB inactive_anon:7203348kB active_file:1458224kB inactive_file:2152108kB unevictable:29952kB isolated(anon):0kB isolated(file):236kB mapped:537464kB dirty:6892kB writeback:0kB shmem:23044kB shmem_thp:0kB shmem_pmdmapped:0kB anon_thp:9814016kB writeback_tmp:0kB kernel_stack:23036kB pagetables:91524kB sec_pagetables:20860kB all_unreclaimable? no
Dec 23 12:13:33 gentoo kernel: DMA free:1024kB boost:0kB min:40kB low:52kB high:64kB reserved_highatomic:0KB active_anon:0kB inactive_anon:10240kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15984kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Dec 23 12:13:33 gentoo kernel: lowmem_reserve[]: 0 2683 23474 0 0
Dec 23 12:13:33 gentoo kernel: DMA32 free:86388kB boost:0kB min:7672kB low:10400kB high:13128kB reserved_highatomic:0KB active_anon:184136kB inactive_anon:2449644kB active_file:1124kB inactive_file:868kB unevictable:0kB writepending:0kB present:2813388kB managed:2747852kB mlocked:0kB bounce:0kB free_pcp:2696kB local_pcp:648kB free_cma:0kB
Dec 23 12:13:33 gentoo kernel: lowmem_reserve[]: 0 0 20790 0 0
Dec 23 12:13:33 gentoo kernel: Normal free:5128052kB boost:0kB min:59864kB low:81148kB high:102432kB reserved_highatomic:2048KB active_anon:5413384kB inactive_anon:4743680kB active_file:1456068kB inactive_file:2151516kB unevictable:29936kB writepending:6660kB present:21772288kB managed:21289640kB mlocked:29628kB bounce:0kB free_pcp:4720kB local_pcp:1316kB free_cma:0kB
Dec 23 12:13:33 gentoo kernel: lowmem_reserve[]: 0 0 0 0 0
Dec 23 12:13:33 gentoo kernel: DMA: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 1*1024kB (U) 0*2048kB 0*4096kB = 1024kB
Dec 23 12:13:33 gentoo kernel: DMA32: 454*4kB (UME) 331*8kB (UME) 224*16kB (UME) 162*32kB (UME) 109*64kB (UME) 69*128kB (UME) 53*256kB (UME) 42*512kB (UME) 22*1024kB (UM) 0*2048kB 0*4096kB = 86640kB
Dec 23 12:13:33 gentoo kernel: Normal: 22142*4kB (UE) 38717*8kB (UE) 68941*16kB (UMEH) 28856*32kB (UMEH) 10626*64kB (UME) 4913*128kB (UMEH) 2383*256kB (UME) 981*512kB (UMEH) 275*1024kB (UMEH) 0*2048kB 0*4096kB = 5127600kB
Dec 23 12:13:33 gentoo kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Dec 23 12:13:33 gentoo kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Dec 23 12:13:33 gentoo kernel: 913829 total pagecache pages
Dec 23 12:13:33 gentoo kernel: 2066 pages in swap cache
Dec 23 12:13:33 gentoo kernel: Free swap  = 324kB
Dec 23 12:13:33 gentoo kernel: Total swap = 6013180kB
Dec 23 12:13:33 gentoo kernel: 6150415 pages RAM
Dec 23 12:13:33 gentoo kernel: 0 pages HighMem/MovableOnly
Dec 23 12:13:33 gentoo kernel: 137202 pages reserved
Dec 23 12:13:33 gentoo kernel: 0 pages cma reserved
Dec 23 12:13:33 gentoo kernel: 0 pages hwpoisoned
Dec 23 12:13:33 gentoo kernel: Tasks state (memory values in pages):
Dec 23 12:13:33 gentoo kernel: [  pid  ]   uid  tgid total_vm      rss rss_anon rss_file rss_shmem pgtables_bytes swapents oom_score_adj name
Dec 23 12:13:33 gentoo kernel: [   2898]     0  2898    45425    11955        0    11955         0   409600      480          -250 systemd-journal
Dec 23 12:13:33 gentoo kernel: [   2934]     0  2934     8644     1248        0     1248         0    94208     1184         -1000 systemd-udevd
Dec 23 12:13:33 gentoo kernel: [   4630]   129  4630     4705     1374        0     1374         0    81920      352             0 systemd-network
Dec 23 12:13:33 gentoo kernel: [   4760]   101  4760     2416      736        0      736         0    65536      608          -900 dbus-daemon
Dec 23 12:13:33 gentoo kernel: [   4761]     0  4761     2001      839        0      839         0    57344      128             0 bash
Dec 23 12:13:33 gentoo kernel: [   4768]     0  4768     2001      454        0      454         0    53248      160             0 bash
Dec 23 12:13:33 gentoo kernel: [   4773]     0  4773    18967     2582      480     2102         0    73728        0             0 ntpd
Dec 23 12:13:33 gentoo kernel: [   4780]   126  4780     5056      613        0      613         0    53248       64             0 rtkit-daemon
Dec 23 12:13:33 gentoo kernel: [   4781]     0  4781   103335      492        0      492         0   102400      128             0 scx_loader
Dec 23 12:13:33 gentoo kernel: [   4783]     0  4783     2121      990        0      990         0    53248      384             0 smartd
Dec 23 12:13:33 gentoo kernel: [   4787]     0  4787     4607     1504       64     1440         0    77824      832             0 snmpd
Dec 23 12:13:33 gentoo kernel: [   4801]     0  4801     3655     1227        0     1227         0    73728      384             0 systemd-logind
Dec 23 12:13:33 gentoo kernel: [   4802]     0  4802    10435      313        0      313         0    69632      192             0 sensord
Dec 23 12:13:33 gentoo kernel: [   4804]     0  4804     3452     1121        0     1121         0    73728      256             0 systemd-machine
Dec 23 12:13:33 gentoo kernel: [   4808]     0  4808      591      363        0      363         0    49152        0             0 lowlatency
Dec 23 12:13:33 gentoo kernel: [   4810]     0  4810    41452     1215       96     1119         0    73728      544          -999 ananicy-cpp
Dec 23 12:13:33 gentoo kernel: [   4824]     0  4824      618      259        0      259         0    45056        0             0 netatopd
Dec 23 12:13:33 gentoo kernel: [   4831]    84  4831     9029      988       32      956         0    69632      160             0 upsd
Dec 23 12:13:33 gentoo kernel: [   4852]     0  4852    54739      538       27      511         0    94208     2272             0 scx_flash
Dec 23 12:13:33 gentoo kernel: [   4856]     0  4856    88836     4069        0     4069         0   200704     6880             0 python3
Dec 23 12:13:33 gentoo kernel: [   4881]     0  4881    36929     3777        0     3777         0   147456      544             0 sddm
Dec 23 12:13:33 gentoo kernel: [   4884]     0  4884     2346      652        0      652         0    57344      160             0 upsmon
Dec 23 12:13:33 gentoo kernel: [   4976]    84  4976     2942     1425        8     1417         0    61440      256             0 upsmon
Dec 23 12:13:33 gentoo kernel: [   5117] 65534  5117     1424      328        1      327         0    53248       96             0 dnsmasq
Dec 23 12:13:33 gentoo kernel: [   5118]     0  5118     1424      296       33      263         0    53248       32             0 dnsmasq
Dec 23 12:13:33 gentoo kernel: [   5149] 65534  5149     1424      313        1      312         0    57344       64             0 dnsmasq
Dec 23 12:13:33 gentoo kernel: [   5150]     0  5150     1424      129        1      128         0    57344       64             0 dnsmasq
Dec 23 12:13:33 gentoo kernel: [   5153]     0  5153    15066      698        0      698         0   155648     6624             0 python3
Dec 23 12:13:33 gentoo kernel: [   5345]     0  5345    17298     1801        0     1801         0   135168      608             0 nmbd
Dec 23 12:13:33 gentoo kernel: [   5347]   122  5347    95334     1533        0     1533         0   102400      672             0 polkitd
Dec 23 12:13:33 gentoo kernel: [   5350]     0  5350    18475     1894        0     1883        11   143360      640             0 winbindd
Dec 23 12:13:33 gentoo kernel: [   5365]     0  5365    18521     1284        0     1273        11   139264      704             0 wb[GENTOO]
Dec 23 12:13:33 gentoo kernel: [   5366]     0  5366    19556     1901        0     1871        30   151552      672             0 smbd
Dec 23 12:13:33 gentoo kernel: [   5370]     0  5370    19088      503        2      501         0   122880      608             0 smbd-notifyd
Dec 23 12:13:33 gentoo kernel: [   5371]     0  5371    19088      418        0      418         0   118784      672             0 smbd-cleanupd
Dec 23 12:13:33 gentoo kernel: [   5383]     0  5383    18475      648       30      618         0   122880      576             0 wb-idmap
Dec 23 12:13:33 gentoo kernel: [   5567]    84  5567     1236      384        1      383         0    49152       32             0 blazer_ser
Dec 23 12:13:33 gentoo kernel: [   6142]     0  6142    77246     1263        0     1263         0   110592      672             0 upowerd
Dec 23 12:13:33 gentoo kernel: [   6190]     0  6190   116354     1794        0     1794         0   139264      928             0 udisksd
Dec 23 12:13:33 gentoo kernel: [   6202]     0  6202    77027     1281        0     1281         0   102400      160             0 power-profiles-
Dec 23 12:13:33 gentoo kernel: [   8644]     0  8644   426846     2949        0     2949         0   380928     7584             0 libvirtd
Dec 23 12:13:33 gentoo kernel: [   8981]     0  8981    17277     2498        0     2498         0   126976      448          -900 virtlogd
Dec 23 12:13:33 gentoo kernel: [  10361]     0 10361   283303    19651     2357    11985      5309   667648     8555             0 X
Dec 23 12:13:33 gentoo kernel: [  10391]  1000 10391  1893251  1068987  1053385    15597         5 10465280    11993             0 qemu-system-x86
Dec 23 12:13:33 gentoo kernel: [  10501]     0 10501    19753     1255        0     1255         0   155648      704             0 smbd[192.168.12
Dec 23 12:13:33 gentoo kernel: [  10692]     0 10692    18973     3639        0     3639         0   143360      512             0 sddm-helper
Dec 23 12:13:33 gentoo kernel: [  10698]  1000 10698     4363     1423        0     1423         0    73728      768           100 systemd
Dec 23 12:13:33 gentoo kernel: [  10700]  1000 10700     4185      499        6      493         0    65536      320           100 (sd-pam)
Dec 23 12:13:33 gentoo kernel: [  10716]  1000 10716    56769     5837     2765     2458       614   225280     3115           200 pipewire
Dec 23 12:13:33 gentoo kernel: [  10717]  1000 10717   313067      165       20      145         0   147456     2240           200 syncthing
Dec 23 12:13:33 gentoo kernel: [  10718]  1000 10718   150037     3310       27     3283         0   282624     2391           200 wireplumber
Dec 23 12:13:33 gentoo kernel: [  10720]  1000 10720    44355     1448        0     1421        27   122880     3220           200 pipewire-pulse
Dec 23 12:13:33 gentoo kernel: [  10730]  1000 10730   282600    11868       60    11808         0   589824     3680             0 kwalletd6
Dec 23 12:13:33 gentoo kernel: [  10735]  1000 10735     2003      899      128      771         0    53248      480           200 dbus-daemon
Dec 23 12:13:33 gentoo kernel: [  10742]  1000 10742    53146     4349        0     4349         0   229376     1120             0 startplasma-x11
Dec 23 12:13:33 gentoo kernel: [  10749]  1000 10749   330348     6593     3805     2788         0   282624     5481           200 syncthing
Dec 23 12:13:33 gentoo kernel: [  10839]  1000 10839   269925    11049        0    11049         0   524288     3296           200 kglobalacceld
Dec 23 12:13:33 gentoo kernel: [  10846]  1000 10846   269067     1625        0     1625         0   196608      576           200 xdg-desktop-por
Dec 23 12:13:33 gentoo kernel: [  10853]  1000 10853   132849     1241        0     1241         0   135168      192           200 xdg-document-po
Dec 23 12:13:33 gentoo kernel: [  10860]  1000 10860    76908     1214        0     1214         0   102400      128           200 xdg-permission-
Dec 23 12:13:33 gentoo kernel: [  10870]  1000 10870      597      450        0      450         0    49152        0           200 fusermount3
Dec 23 12:13:33 gentoo kernel: [  10879]  1000 10879   281566    11830        0    11830         0   585728     3680           200 ksmserver
Dec 23 12:13:33 gentoo kernel: [  10881]  1000 10881   450453    13672       68    13604         0   802816     8864           200 kded6
Dec 23 12:13:33 gentoo kernel: [  10882]  1000 10882   547759    26390     8172    18218         0  1097728    10592           200 kwin_x11
Dec 23 12:13:33 gentoo kernel: [  10940]  1000 10940   824364    77152    46116    31036         0  2265088    72640           200 plasmashell
Dec 23 12:13:33 gentoo kernel: [  10955]  1000 10955    57434     1027        0     1027         0    86016      256           200 dconf-service
Dec 23 12:13:33 gentoo kernel: [  10987]  1000 10987   294306    11522        0    11522         0   540672     3360           200 kactivitymanage
Dec 23 12:13:33 gentoo kernel: [  10989]  1000 10989   262478    10200       38    10162         0   466944     2400           200 gmenudbusmenupr
Dec 23 12:13:33 gentoo kernel: [  10990]  1000 10990   281625    11660       44    11616         0   589824     3648           200 kaccess
Dec 23 12:13:33 gentoo kernel: [  10991]  1000 10991   281318    11948        0    11948         0   589824     3392           200 polkit-kde-auth
Dec 23 12:13:33 gentoo kernel: [  10992]  1000 10992   335590    11941        0    11941         0   606208     3264           200 org_kde_powerde
Dec 23 12:13:33 gentoo kernel: [  10993]  1000 10993   282106    11759        0    11759         0   577536     3776           200 xdg-desktop-por
Dec 23 12:13:33 gentoo kernel: [  10994]  1000 10994   262311    10159        0    10159         0   471040     2400           200 xembedsniproxy
Dec 23 12:13:33 gentoo kernel: [  11053]  1000 11053    41652     4677      608     4069         0   184320      960           200 ksystemstats
Dec 23 12:13:33 gentoo kernel: [  11102]  1000 11102     1865      910        0      910         0    61440       96           200 xsettingsd
Dec 23 12:13:33 gentoo kernel: [  11105]  1000 11105    55177     3615        0     3615         0   155648      640           200 kscreen_backend
Dec 23 12:13:33 gentoo kernel: [  11200]  1000 11200   266412     9388        5     9383         0   495616     2592           200 xdg-desktop-por
Dec 23 12:13:33 gentoo kernel: [  11202]  1000 11202   430159    14951     3775    11145        31   798720     5312           200 openrgb
Dec 23 12:13:33 gentoo kernel: [  11216]  1000 11216   282399    11856        0    11856         0   593920     3392           200 kclockd
Dec 23 12:13:33 gentoo kernel: [  11217]  1000 11217   309603    14917       88    14829         0   634880     5248           200 kdeconnectd
Dec 23 12:13:33 gentoo kernel: [  11244]  1000 11244    77058     1252        0     1252         0   110592      128           200 agent
Dec 23 12:13:33 gentoo kernel: [  11249]  1000 11249   375370    12313       34    12279         0   745472     5886           200 kalendarac
Dec 23 12:13:33 gentoo kernel: [  11299]  1000 11299    77700     1179        0     1179         0   110592      192           200 gvfsd
Dec 23 12:13:33 gentoo kernel: [  11324]  1000 11324    94606     1207        0     1207         0   102400      160           200 at-spi-bus-laun
Dec 23 12:13:33 gentoo kernel: [  11333]  1000 11333     1656      709        0      709         0    53248       96           200 dbus-daemon
Dec 23 12:13:33 gentoo kernel: [  11335]  1000 11335    58124     1403        0     1403         0    86016      192           200 at-spi2-registr
Dec 23 12:13:33 gentoo kernel: [  11548]     0 11548    88836     1563       23     1540         0   196608     6816             0 python3
Dec 23 12:13:33 gentoo kernel: [  11631]  1000 11631   663007    33148     4295    28853         0   962560    12800           200 looking-glass-c
Dec 23 12:13:33 gentoo kernel: [  12134]  1000 12134   281776    11896       66    11830         0   589824     3680           200 baloorunner
Dec 23 12:13:33 gentoo kernel: [  12792]  1000 12792   368691    25035     1848    17897      5290   786432     7449           200 konsole
Dec 23 12:13:33 gentoo kernel: [  12820]  1000 12820    33301     3269     1729     1540         0   167936     5664           200 python2
Dec 23 12:13:33 gentoo kernel: [  17378]  1000 17378  5210480    44311     6430    37881         0  3469312    89572           200 firefox-bin
Dec 23 12:13:33 gentoo kernel: [  17463]  1000 17463    52527     4224        0     4206        18   217088     2144           200 Socket Process
Dec 23 12:13:33 gentoo kernel: [  17506]  1000 17506   633751    16618      156    16419        43   946176    11086           167 Privileged Cont
Dec 23 12:13:33 gentoo kernel: [  17565]  1000 17565   649542    17652      597    17055         0  1150976    29432           167 Isolated Web Co
Dec 23 12:13:33 gentoo kernel: [  17572]  1000 17572   649165    17264      307    16950         7  1114112    20992           167 Isolated Web Co
Dec 23 12:13:33 gentoo kernel: [  17685]  1000 17685  4992737    24022     3241    20781         0  2707456    61220           100 WebExtensions
Dec 23 12:13:33 gentoo kernel: [  17716]  1000 17716    86461     5394        0     5394         0   389120     4384           200 Utility Process
Dec 23 12:13:33 gentoo kernel: [  18183]  1000 18183   636513    16997      454    16543         0   868352    12192           167 Isolated Web Co
Dec 23 12:13:33 gentoo kernel: [  18187]  1000 18187   635109    16764      309    16455         0   839680    12224           167 Isolated Web Co
Dec 23 12:13:33 gentoo kernel: [  18822]  1000 18822   633958    17381     1238    16143         0   884736     9291           167 Isolated Web Co
Dec 23 12:13:33 gentoo kernel: [  19265]  1000 19265   637918    16578      126    16439        13   937984    15904           167 Isolated Web Co
Dec 23 12:13:33 gentoo kernel: [  19976]   130 19976     5914     2234      192     2042         0    90112     1102             0 systemd-resolve
Dec 23 12:13:33 gentoo kernel: [  20391]  1000 20391     1935      802       32      770         0    53248       64             0 sh
Dec 23 12:13:33 gentoo kernel: [  20394]  1000 20394 68773504    10777     2573     8196         8   974848    19264             0 jackett
Dec 23 12:13:33 gentoo kernel: [  21269]  1000 21269   651007    21277     2833    18444         0  1105920    26381           100 Isolated Web Co
Dec 23 12:13:33 gentoo kernel: [  35655]  1000 35655   755983    27376     9785    17591         0  2920448    64486           167 Isolated Web Co
Dec 23 12:13:33 gentoo kernel: [  37314]  1000 37314   624910    15046      293    14718        35   770048     9014           167 Isolated Web Co
Dec 23 12:13:33 gentoo kernel: [  37415]  1000 37415   101712     6086       46     6015        25   471040     2921           200 RDD Process
Dec 23 12:13:33 gentoo kernel: [  37663]  1000 37663   664142    17055      372    16675         8  1347584    20205           167 Isolated Web Co
Dec 23 12:13:33 gentoo kernel: [  38274]  1000 38274   658276    19628      814    18802        12  1220608    30798           167 Isolated Web Co
Dec 23 12:13:33 gentoo kernel: [  38849]  1000 38849   670188    20306     1951    18314        41  1490944    23378           167 Isolated Web Co
Dec 23 12:13:33 gentoo kernel: [  38991]  1000 38991   656068    18542      441    18101         0  1204224    16354           167 Isolated Web Co
Dec 23 12:13:33 gentoo kernel: [  39893]  1000 39893   642394    19106      352    18732        22   995328    16110           167 Isolated Web Co
Dec 23 12:13:33 gentoo kernel: [  44248]  1000 44248   648367    18586      393    18193         0  1105920    14700           167 Isolated Web Co
Dec 23 12:13:33 gentoo kernel: [  58670]  1000 58670   655492    29989     5470    24519         0  1232896    14050           167 Isolated Web Co
Dec 23 12:13:33 gentoo kernel: [  80156]  1000 80156   327963    20244      174    20070         0   811008     8320           200 yakuake
Dec 23 12:13:33 gentoo kernel: [  80971]  1000 80971     2750      919       16      903         0    69632      864           200 bash
Dec 23 12:13:33 gentoo kernel: [  81001]  1000 81001     2749      912        0      912         0    65536      864           200 bash
Dec 23 12:13:33 gentoo kernel: [  81024]  1000 81024     2066      758        0      758         0    61440       96           200 su
Dec 23 12:13:33 gentoo kernel: [  81041]     0 81041     3342      903        0      903         0    69632     1472           200 bash
Dec 23 12:13:33 gentoo kernel: [  81157]     0 81157     6449     4910     3248     1662         0    98304        0             0 python3
Dec 23 12:13:33 gentoo kernel: [  81176]     0 81176   151825     3901      504     3397         0  1253376   145184           200 emerge
Dec 23 12:13:33 gentoo kernel: [  81223]  1000 81223     2749      922        0      922         0    61440      864           200 bash
Dec 23 12:13:33 gentoo kernel: [  81293]  1000 81293     2066      767        0      767         0    57344       96           200 su
Dec 23 12:13:33 gentoo kernel: [  81303]     0 81303     3342      874        1      873         0    69632     1472           200 bash
Dec 23 12:13:33 gentoo kernel: [  81419]     0 81419   214637     6294       64     6230         0   987136      192           200 journalctl
Dec 23 12:13:33 gentoo kernel: [  81965]  1000 81965   771265    39571    11373    28198         0  3452928   106239           100 Isolated Web Co
Dec 23 12:13:33 gentoo kernel: [  82206]  1000 82206   625590    22442      744    21698         0   774144     7743           100 Isolated Web Co
Dec 23 12:13:33 gentoo kernel: [  84742]  1000 84742   610500    11462       75    11371        16   479232     3296           233 Web Content
Dec 23 12:13:33 gentoo kernel: [  85142]  1000 85142   610500    11836      398    11417        21   487424     3040           233 Web Content
Dec 23 12:13:33 gentoo kernel: [  90419]     0 90419      646      376        0      376         0    45056       32             0 inotifywait
Dec 23 12:13:33 gentoo kernel: [  94172]     0 94172     4550     1535        0     1535         0    81920     1536           200 python3.12
Dec 23 12:13:33 gentoo kernel: [  94173]     0 94173     4551     1545        0     1545         0    81920     1536           200 python3.12
Dec 23 12:13:33 gentoo kernel: [  94176]   250 94176      604      374        0      374         0    45056       32           200 sandbox
Dec 23 12:13:33 gentoo kernel: [  94177]   250 94177     3210      875        0      875         0    69632     1216           200 bash
Dec 23 12:13:33 gentoo kernel: [  94193]   250 94193     3215      611        0      611         0    65536     1248           200 bash
Dec 23 12:13:33 gentoo kernel: [  94219]   250 94219    21554      895        2      893         0   225280    19936           200 ninja
Dec 23 12:13:33 gentoo kernel: [ 107345]   250 107345     3679      505        0      505         0    69632      480           200 x86_64-pc-linux
Dec 23 12:13:33 gentoo kernel: [ 107400]   250 107400     2266      564        0      564         0    57344      256           200 x86_64-pc-linux
Dec 23 12:13:33 gentoo kernel: [ 107401]   250 107401  1119335   933149   930026     3123         0  8974336   159155           200 cc1plus
Dec 23 12:13:33 gentoo kernel: [ 107402]   250 107402     2549      541        0      541         0    61440      192           200 as
Dec 23 12:13:33 gentoo kernel: [ 108109]  1000 108109     2878      867        0      867         0    61440      992           200 bash
Dec 23 12:13:33 gentoo kernel: [ 108500]  1000 108500   610500    13961       97    13864         0   483328     3264           233 Web Content
Dec 23 12:13:33 gentoo kernel: [ 108859]   250 108859     3514     2365        0     2365         0    77824      352           200 x86_64-pc-linux
Dec 23 12:13:33 gentoo kernel: [ 108877]   250 108877     2266      780       32      748         0    61440      256           200 x86_64-pc-linux
Dec 23 12:13:33 gentoo kernel: [ 108878]   250 108878   550147   382065   375082     6983         0  4345856   142272           200 cc1plus
Dec 23 12:13:33 gentoo kernel: [ 108879]   250 108879     3473     1854      960      894         0    73728      160           200 as
Dec 23 12:13:33 gentoo kernel: [ 108958]   250 108958     3514     2320        4     2316         0    69632      352           200 x86_64-pc-linux
Dec 23 12:13:33 gentoo kernel: [ 108973]   250 108973     2266      775       32      743         0    61440      256           200 x86_64-pc-linux
Dec 23 12:13:33 gentoo kernel: [ 108974]   250 108974   488514   399255   393285     5970         0  3862528    60896           200 cc1plus
Dec 23 12:13:33 gentoo kernel: [ 108976]   250 108976     2549      862        0      862         0    65536      192           200 as
Dec 23 12:13:33 gentoo kernel: [ 109030]   250 109030     3552     2417        2     2415         0    77824      352           200 x86_64-pc-linux
Dec 23 12:13:33 gentoo kernel: [ 109061]   250 109061     2266      721        0      721         0    65536      256           200 x86_64-pc-linux
Dec 23 12:13:33 gentoo kernel: [ 109062]   250 109062   361225   314781   310048     4733         0  2760704     6112           200 cc1plus
Dec 23 12:13:33 gentoo kernel: [ 109063]   250 109063     2549      936        0      936         0    57344      192           200 as
Dec 23 12:13:33 gentoo kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/user.slice/user-1000.slice/user@1000.service/app.slice/app-dummy@631f2652c7e14536a180d9bc90083198.service,task=cc1plus,pid=107401,uid=250
Dec 23 12:13:33 gentoo kernel: Out of memory: Killed process 107401 (cc1plus) total-vm:4477340kB, anon-rss:3720104kB, file-rss:12492kB, shmem-rss:0kB, UID:250 pgtables:8764kB oom_score_adj:200
Dec 23 12:13:33 gentoo systemd[1]: user@1000.service: A process of this unit has been killed by the OOM killer.

Config:

## This is the configuration file of the nohang daemon.

## The configuration includes the following sections:
##  0. Check kernel messages for OOM events
##  1. Common zram settings
##  2. Common PSI settings
##  3. Poll rate
##  4. Warnings and notifications
##  5. Soft (SIGTERM) threshold
##  6. Hard (SIGKILL) threshold
##  7. Customize victim selection: adjusting badness of processes
##  8. Customize soft corrective actions
##  9. Misc settings
## 10. Verbosity, debug, logging

## WARNING!
##  - Lines starting with #, tabs and whitespace characters are comments.
##  - Lines starting with @ contain optional parameters that may be repeated.
##  - All values are case sensitive.
##  - nohang doesn't forbid you to shoot yourself in the foot. Be careful!
##  - Restart the daemon after editing the file to apply the new settings.
##  - You can find the file with default values here: /usr/share/nohang/nohang.conf

## To find config keys descriptions see man(8) nohang

###############################################################################

##  0. Check kernel messages for OOM events

#   @check_kmsg
##  Type: boolean
##  Comment/uncomment to disable/enable checking kmsg for OOM events

#   @debug_kmsg
##  Type: boolean
##  Comment/uncomment to disable/enable debug checking kmsg

###############################################################################

    1. Common zram settings

    Key: zram_checking_enabled
    Description:
    Type: boolean
    Valid values: True | False
    Default value: False

zram_checking_enabled = True

###############################################################################

    2. Common PSI settings

    Key: psi_checking_enabled
    Description:
    Type: boolean
    Valid values: True | False
    Default value: True

psi_checking_enabled = True

    Key: psi_path
    Description:
    Type: string
    Valid values: any string
    Default value: /proc/pressure/memory

psi_path = /proc/pressure/memory

    Key: psi_metrics
    Description:
    Type: string
    Valid values: some_avg10, some_avg60, some_avg300,
                  full_avg10, full_avg60, full_avg300
    Default value: full_avg10

psi_metrics = full_avg10

    Key: psi_excess_duration
    Description:
    Type: float
    Valid values: >= 0
    Default value: 30

psi_excess_duration = 30

    Key: psi_post_action_delay
    Description:
    Type: float
    Valid values: >= 10
    Default value: 15

psi_post_action_delay = 15

###############################################################################

    3. Poll rate

    Key: fill_rate_mem
    Description:
    Type: float
    Valid values: >= 100
    Default value: 6000

fill_rate_mem  = 6000

    Key: fill_rate_swap
    Description:
    Type: float
    Valid values: >= 100
    Default value: 2000

fill_rate_swap = 2000

    Key: fill_rate_zram
    Description:
    Type: float
    Valid values: >= 100
    Default value: 4000

fill_rate_zram = 4000

    Key: max_sleep
    Description:
    Type: float
    Valid values: >= 0.01 and >= min_sleep
    Default value: 3

max_sleep = 3

    Key: min_sleep
    Description:
    Type: float
    Valid values: >= 0.01 and <= max_sleep
    Default value: 0.1

min_sleep = 0.1

###############################################################################

    4. Warnings and notifications

    4.1. GUI notifications after corrective actions

    Key: post_action_gui_notifications
    Description:
    Type: boolean
    Valid values: True | False
    Default value: True

post_action_gui_notifications = True

    Key: hide_corrective_action_type
    Description:
    Type: boolean
    Valid values: True | False
    Default value: False

hide_corrective_action_type = False

    4.2. Low memory warnings

    Key: low_memory_warnings_enabled
    Description:
    Type: boolean
    Valid values: True | False
    Default value: True

low_memory_warnings_enabled = True

    Key: warning_exe
    Description:
    Type: string
    Valid values: any string
    Default value: (empty string)

warning_exe =

    Key: warning_threshold_min_mem
    Description:
    Type: float (with % or M)
    Valid values: from the range [0; 100] %
    Default value: 20 %

warning_threshold_min_mem  = 20 %

    Key: warning_threshold_min_swap
    Description:
    Type: float (with % or M)
    Valid values: [0; 100] % or >= 0 M
    Default value: 20 %

warning_threshold_min_swap = 25 %

    Key: warning_threshold_max_zram
    Description:
    Type: float (with % or M)
    Valid values: from the range [0; 100] %
    Default value: 45 %

warning_threshold_max_zram = 45 %

    Key: warning_threshold_max_psi
    Description:
    Type: float
    Valid values: from the range [0; 100]
    Default value: 10

warning_threshold_max_psi  = 10

    Key: min_post_warning_delay
    Description:
    Type: float
    Valid values: >= 1
    Default value: 60

min_post_warning_delay = 60

    Key: env_cache_time
    Description:
    Type: float
    Valid values: >= 0
    Default value: 300

env_cache_time = 300

###############################################################################

    5. Soft threshold (thresholds for sending the SIGTERM signal or
                       implementing other soft corrective action)

    Key: soft_threshold_min_mem
    Description:
    Type: float (with % or M)
    Valid values: from the range [0; 50] %
    Default value: 5 %

soft_threshold_min_mem = 5 %

    Key: soft_threshold_min_swap
    Description:
    Type: float (with % or M)
    Valid values: [0; 100] % or >= 0 M
    Default value: 10 %

soft_threshold_min_swap = 10 %

    Key: soft_threshold_max_zram
    Description:
    Type: float (with % or M)
    Valid values: from the range [10; 90] %
    Default value: 55 %

soft_threshold_max_zram = 55 %

    Key: soft_threshold_max_psi
    Description:
    Type: float
    Valid values: from the range [5; 100]
    Default value: 40

soft_threshold_max_psi  = 40

###############################################################################

    6. Hard threshold (thresholds for sending the SIGKILL signal)

    Key: hard_threshold_min_mem
    Description:
    Type: float (with % or M)
    Valid values: from the range [0; 50] %
    Default value: 2 %

hard_threshold_min_mem = 2 %

    Key: hard_threshold_min_swap
    Description:
    Type: float (with % or M)
    Valid values: [0; 100] % or >= 0 M
    Default value: 4 %

hard_threshold_min_swap = 4 %

    Key: hard_threshold_max_zram
    Description:
    Type: float (with % or M)
    Valid values: from the range [10; 90] %
    Default value: 60 %

hard_threshold_max_zram = 60 %

    Key: hard_threshold_max_psi
    Description:
    Type: float
    Valid values: from the range [5; 100]
    Default value: 90

hard_threshold_max_psi = 90

###############################################################################

    7. Customize victim selection: adjusting badness of processes

    7.1. Ignore positive oom_score_adj

    Key: ignore_positive_oom_score_adj
    Description:
    Type: boolean
    Valid values: True | False
    Default value: False

ignore_positive_oom_score_adj = False

    7.2.1. Matching process names with RE patterns change their badness

    Syntax:

    @BADNESS_ADJ_RE_NAME  badness_adj  ///  RE_pattern

    New badness value will be += badness_adj

    It is possible to compare multiple patterns
    with different badness_adj values.

    Example:
    @BADNESS_ADJ_RE_NAME -500 /// ^sshd$

    Prefer terminating Firefox tabs instead of terminating the entire browser.
    (In Chromium and Electron-based apps child processes get oom_score_adj=300
    by default.)
@BADNESS_ADJ_RE_NAME   200  ///  ^(Web Content|Privileged Cont|file:// Content)$

@BADNESS_ADJ_RE_NAME  -200  ///  ^(dnf|yum|packagekitd)$


    7.2.2. Matching CGroup_v1-line with RE patterns

    @BADNESS_ADJ_RE_CGROUP_V1 -50 ///  ^/system\.slice/

    @BADNESS_ADJ_RE_CGROUP_V1  50 ///  /foo\.service$

    @BADNESS_ADJ_RE_CGROUP_V1 -50 ///  ^/user\.slice/

    7.2.3. Matching CGroup_v2-line with RE patterns

    @BADNESS_ADJ_RE_CGROUP_V2  100 /// ^/workload

    7.2.4. Matching eUIDs with RE patterns

    @BADNESS_ADJ_RE_UID -100 /// ^0$

    7.2.5. Matching /proc/[pid]/exe realpath with RE patterns

    Example:
    @BADNESS_ADJ_RE_REALPATH  20  ///  ^/usr/bin/foo$

    Protect X.
@BADNESS_ADJ_RE_REALPATH -200  ///  ^(/usr/libexec/Xorg|/usr/lib/xorg/Xorg|/usr/lib/Xorg|/usr/bin/X|/usr/bin/Xorg|/usr/bin/Xwayland|/usr/bin/weston|/usr/bin/sway)$

    Protect GNOME.
@BADNESS_ADJ_RE_REALPATH -200  ///  ^(/usr/bin/gnome-shell|/usr/bin/metacity|/usr/bin/mutter|/usr/lib/gnome-session/gnome-session-binary|/usr/libexec/gnome-session-binary|/usr/libexec/gnome-session-ctl)$

    Protect KDE Plasma.
@BADNESS_ADJ_RE_REALPATH -200  ///  ^(/usr/bin/plasma-desktop|/usr/bin/plasmashell|/usr/bin/plasma_session|/usr/bin/kwin|/usr/bin/kwin_x11|/usr/bin/kwin_wayland)$
@BADNESS_ADJ_RE_REALPATH -200  ///  ^(/usr/bin/startplasma-wayland|/usr/lib/x86_64-linux-gnu/libexec/startplasma-waylandsession|/usr/bin/ksmserver)$

    Protect Cinnamon.
@BADNESS_ADJ_RE_REALPATH -200  ///  ^(/usr/bin/cinnamon|/usr/bin/muffin|/usr/bin/cinnamon-session|/usr/bin/cinnamon-launcher)$

    Protect Xfce.
@BADNESS_ADJ_RE_REALPATH -200  ///  ^(/usr/bin/xfwm4|/usr/bin/xfce4-session|/usr/bin/xfce4-panel|/usr/bin/xfdesktop)$

    Protect Mate.
@BADNESS_ADJ_RE_REALPATH -200  ///  ^(/usr/bin/marco|/usr/bin/mate-session|/usr/bin/caja|/usr/bin/mate-panel)$

    Protect LXQt.
@BADNESS_ADJ_RE_REALPATH -200  ///  ^(/usr/bin/lxqt-panel|/usr/bin/pcmanfm-qt|/usr/bin/lxqt-session)$

    Protect Budgie Desktop.
@BADNESS_ADJ_RE_REALPATH -200  ///  ^(/usr/bin/budgie-wm|/usr/bin/budgie-panel)$

    Protect other.
@BADNESS_ADJ_RE_REALPATH -200  ///  ^(/usr/bin/compiz|/usr/bin/openbox|/usr/bin/fluxbox|/usr/bin/awesome|/usr/bin/icewm|/usr/bin/enlightenment|/usr/bin/gala|/usr/bin/wingpanel|/usr/bin/i3)$
@BADNESS_ADJ_RE_REALPATH -200  ///  ^(/usr/bin/teams|/usr/bin/carla|/opt/teams/teams)$

    Protect display managers.
@BADNESS_ADJ_RE_REALPATH -200  ///  ^(/usr/sbin/gdm|/usr/sbin/gdm3|/usr/sbin/sddm|/usr/bin/sddm|/usr/lib/x86_64-linux-gnu/sddm/sddm-helper|/usr/bin/slim|/usr/sbin/lightdm|/usr/libexec/gdm-session-worker|/usr/libexec/gdm-wayland-session|/usr/lib/gdm3/gdm-wayland-session|/usr/lib/gdm3/gdm-session-worker)$
@BADNESS_ADJ_RE_REALPATH -200  ///  ^/usr/lib/gdm3/

    Protect systemd-logind.
@BADNESS_ADJ_RE_REALPATH -200  ///  ^(/lib/systemd/systemd-logind|/usr/lib/systemd/systemd-logind)$

    Protect `systemd --user`.
@BADNESS_ADJ_RE_REALPATH -200  ///  ^(/lib/systemd/systemd|/usr/lib/systemd/systemd)$

    Protect dbus.
@BADNESS_ADJ_RE_REALPATH -200  ///  ^(/usr/bin/dbus-daemon|/usr/bin/dbus-run-session|/usr/bin/dbus-broker-launcher|/usr/bin/dbus-broker)$

    Protect package managers and distro installers.
@BADNESS_ADJ_RE_REALPATH -200  ///  ^(/usr/bin/calamares|/usr/bin/dpkg|/usr/bin/pacman|/usr/bin/yay|/usr/bin/pamac|/usr/bin/pamac-daemon|/usr/bin/pamac-manager|/usr/bin/emerge)$

    Prefer stress.
    @BADNESS_ADJ_RE_REALPATH  900  ///  ^(/usr/bin/stress|/usr/bin/stress-ng)$


    7.2.6. Matching /proc/[pid]/cwd realpath with RE patterns

    @BADNESS_ADJ_RE_CWD  200  ///  ^/home/

    7.2.7. Matching cmdlines with RE patterns
    WARNING: using this option can greatly slow down the search for a victim
    in conditions of heavily swapping.

    Prefer Chromium tabs and Electron-based apps
    @BADNESS_ADJ_RE_CMDLINE  200 /// --type=renderer

    Prefer Firefox tabs (Web Content and WebExtensions)
    @BADNESS_ADJ_RE_CMDLINE  100 /// -appomni

    @BADNESS_ADJ_RE_CMDLINE -200 /// ^/usr/lib/virtualbox

    7.2.8. Matching environ with RE patterns
    WARNING: using this option can greatly slow down the search for a victim
    in conditions of heavily swapping.

    @BADNESS_ADJ_RE_ENVIRON 100 /// USER=user


    Note that you can control badness also via systemd units via
    OOMScoreAdjust, see
    www.freedesktop.org/software/systemd/man/systemd.exec.html#OOMScoreAdjust=

###############################################################################

    8. Customize soft corrective actions

    Run the command instead of sending a signal with at soft corrective action
    if the victim's name or cgroup matches the regular expression.

    Syntax:
    KEY                         REGEXP      SEPARATOR     COMMAND

    @SOFT_ACTION_RE_NAME       ^foo$             /// kill -USR1  $PID
    @SOFT_ACTION_RE_CGROUP_V1  ^/system\.slice/  /// systemctl restart $SERVICE
    @SOFT_ACTION_RE_CGROUP_V2  /foo\.service$    /// systemctl restart $SERVICE

    $PID will be replaced by process PID.
    $NAME will be replaced by process name.
    $SERVICE will be replaced by .service if it exists (overwise it will be
                                                        relpaced by empty line)

###############################################################################

    9. Misc settings

    Key: max_soft_exit_time
    Description:
    Type: float
    Valid values: >= 0.1
    Default value: 10

max_soft_exit_time = 10

    Key: post_kill_exe
    Description:
    Type: string
    Valid values: any string
    Default value: (empty string)

post_kill_exe =

    Key: min_badness
    Description:
    Type: integer
    Valid values: >= 1
    Default value: 1

min_badness = 1

    Key: post_soft_action_delay
    Description:
    Type: float
    Valid values: >= 0.1
    Default value: 3

post_soft_action_delay = 3

    Key: post_zombie_delay
    Description:
    Type: float
    Valid values: >= 0
    Default value: 0.1

post_zombie_delay = 0.1

    Key: victim_cache_time
    Description:
    Type: float
    Valid values: >= 0
    Default value: 10

victim_cache_time = 10

    Key: exe_timeout
    Description:
    Type: float
    Valid values: >= 0.1
    Default value: 20

exe_timeout = 20

###############################################################################

   10. Verbosity, debug, logging

    Key: print_config_at_startup
    Description:
    Type: boolean
    Valid values: True | False
    Default value: False

print_config_at_startup = False

    Key: print_mem_check_results
    Description:
    Type: boolean
    Valid values: True | False
    Default value: False

print_mem_check_results = True

    Key: min_mem_report_interval
    Description:
    Type: float
    Valid values: >= 0
    Default value: 60

min_mem_report_interval = 60

    Key: print_proc_table
    Description:
    Type: boolean
    Valid values: True | False
    Default value: False

print_proc_table = True

    Key: extra_table_info
    Description:
    WARNING: using "cmdline" or "environ" keys can greatly slow down
    the search for a victim in conditions of heavy swapping.
    Type: string
    Valid values: None, cgroup_v1, cgroup_v2, realpath,
                  cwd, cmdline, environ
    Default value: None

extra_table_info = None

    Key: print_victim_status
    Description:
    Type: boolean
    Valid values: True | False
    Default value: True

print_victim_status = True

    Key: print_victim_cmdline
    Description:
    Type: boolean
    Valid values: True | False
    Default value: False

print_victim_cmdline = False

    Key: max_victim_ancestry_depth
    Description:
    Type: integer
    Valid values: >= 1
    Default value: 3

max_victim_ancestry_depth = 3

    Key: print_statistics
    Description:
    Type: boolean
    Valid values: True | False
    Default value: True

print_statistics = True

    Key: debug_psi
    Description:
    Type: boolean
    Valid values: True | False
    Default value: False

debug_psi = False

    Key: debug_gui_notifications
    Description:
    Type: boolean
    Valid values: True | False
    Default value: False

debug_gui_notifications = True

    Key: debug_sleep
    Description:
    Type: boolean
    Valid values: True | False
    Default value: False

debug_sleep = False

    Key: debug_threading
    Description:
    Type: boolean
    Valid values: True | False
    Default value: False

debug_threading = False

    Key: separate_log
    Description:
    Type: boolean
    Valid values: True | False
    Default value: False

separate_log = False

###############################################################################

    Use cases, feature requests and any questions are welcome:
    https://github.com/hakavlad/nohang/issues

[hakavlad]

nohang-desktop stopped working?

I can't say that nohang has stopped working. I can assume that kernel's OOM killer triggered prematurely. We need to find out why.

[hakavlad]

Dec 23 12:13:22 gentoo nohang-desktop[81157]: PSI: 7.95 | MemAvail: 9149 M, 39.0 % | SwapFree: 654 M, 11.1 %

then

Dec 23 12:13:33 gentoo kernel: kswapd0 invoked oom-killer:

[hakavlad]

How can I debug this further?

Maybe try

min_mem_report_interval = 0

to check (and log) memory more often.

[hakavlad]

Is MGLRU enabled?

With MGLRU enabled, the killer may be more aggressive.

[hakavlad]

I reproduced this with MGLRU: with nohang disabled, killer killed with a lot of SwapfFree even with min_ttl_ms=0.

[dim-geo]

MGLRU was enabled. Disabled MGLRU with set_mglru 0 and run again nohang --memload. OOM killer was triggered again before nohang. (I already set min_mem_report_interval = 1)

[dim-geo]

Warnings are triggered but no action is taken.

Dec 23 16:45:35 gentoo nohang-desktop[25555]: Executing Command-63 ['sudo', '-u', 'none', 'env', 'DISPLAY=:0', 'DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus', 'notify-send', '--icon=dialog-warning', '--app-name=noha>
Dec 23 16:45:35 gentoo nohang-desktop[25555]: Command-63 execution completed in 0.068s; exit status: 0
Dec 23 16:45:35 gentoo nohang-desktop[25555]: PSI:  14.73 | MemAvail:  3395 M, 14.5 % | SwapFree:   507 M,   2.2 % | MemUsedZram:   429 M,  1.8 % | dMem:  -659 M/s
Dec 23 16:45:35 gentoo nohang-desktop[25555]: Warning threshold exceeded
Dec 23 16:45:35 gentoo nohang-desktop[25555]: Memory info, MiB:
Dec 23 16:45:35 gentoo nohang-desktop[25555]:   total=23489, used=19697, free=120, available=3395, shared=28, buffers=7, cache=3666,
Dec 23 16:45:35 gentoo nohang-desktop[25555]:   swap_total=23489, swap_used=22983, swap_free=506
Dec 23 16:45:35 gentoo nohang-desktop[25555]: Memory pressure (system-wide):
Dec 23 16:45:35 gentoo nohang-desktop[25555]:   some avg10=15.11 avg60=7.58 avg300=3.46
Dec 23 16:45:35 gentoo nohang-desktop[25555]:   full avg10=14.73 avg60=7.28 avg300=3.30
Dec 23 16:45:35 gentoo nohang-desktop[25555]: Found env in 0 ms (cached)
Dec 23 16:45:35 gentoo nohang-desktop[25555]:   Title:    ['Low memory']
Dec 23 16:45:35 gentoo nohang-desktop[25555]:   Body:     ['Save your unsaved data!\nClose unused apps!']
Dec 23 16:45:35 gentoo nohang-desktop[25555]:   Env list: [('none', 'DISPLAY=:0', 'DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus')]
Dec 23 16:45:35 gentoo sudo[28055]:     root : PWD=/ ; USER=none ; COMMAND=/usr/sbin/env DISPLAY=:0 DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus notify-send --icon=dialog-warning --app-name=nohang 'Low memory' 'Save>
Dec 23 16:45:35 gentoo sudo[28055]: pam_unix(sudo:session): session opened for user none(uid=1000) by (uid=0)
Dec 23 16:45:35 gentoo sudo[28055]: pam_unix(sudo:session): session closed for user none
Dec 23 16:45:35 gentoo kernel: python3 invoked oom-killer: gfp_mask=0x140dca(GFP_HIGHUSER_MOVABLE|__GFP_COMP|__GFP_ZERO), order=0, oom_score_adj=200
Dec 23 16:45:36 gentoo kernel: CPU: 3 UID: 1000 PID: 27562 Comm: python3 Tainted: G S      W  OE      6.12.6-cachyos-6.12.6-x86_64 #1
Dec 23 16:45:36 gentoo kernel: Tainted: [S]=CPU_OUT_OF_SPEC, [W]=WARN, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
Dec 23 16:45:36 gentoo kernel: Hardware name: Gigabyte Technology Co., Ltd. H97M-D3H/H97M-D3H, BIOS F7 08/03/2015
Dec 23 16:45:36 gentoo kernel: Sched_ext: flash (enabled+all), task: runnable_at=+0ms
Dec 23 16:45:36 gentoo kernel: Call Trace:
Dec 23 16:45:36 gentoo kernel:  <TASK>
Dec 23 16:45:36 gentoo kernel:  dump_stack_lvl+0x56/0x80
Dec 23 16:45:36 gentoo kernel:  dump_header+0x50/0x36d
Dec 23 16:45:36 gentoo kernel:  ? ___ratelimit+0x99/0xf0
Dec 23 16:45:36 gentoo kernel:  oom_kill_process.cold+0x8/0x8a
Dec 23 16:45:36 gentoo kernel:  out_of_memory+0x102/0x410
Dec 23 16:45:36 gentoo kernel:  __alloc_pages_slowpath.constprop.2+0x82f/0xce0
Dec 23 16:45:36 gentoo kernel:  __alloc_pages_noprof+0x2ec/0x300
Dec 23 16:45:36 gentoo kernel:  __folio_alloc_noprof+0x10/0x30
Dec 23 16:45:36 gentoo kernel:  do_anonymous_page+0x2ef/0x7b0
Dec 23 16:45:36 gentoo kernel:  ? __pte_offset_map+0x17/0x170
Dec 23 16:45:36 gentoo kernel:  __handle_mm_fault+0xa24/0xaa0
Dec 23 16:45:36 gentoo kernel:  handle_mm_fault+0xd9/0x2d0
Dec 23 16:45:36 gentoo kernel:  do_user_addr_fault+0x214/0x850
Dec 23 16:45:36 gentoo kernel:  exc_page_fault+0x6f/0x160
Dec 23 16:45:36 gentoo kernel:  asm_exc_page_fault+0x22/0x30
Dec 23 16:45:36 gentoo kernel: RIP: 0033:0x74bd9b2d1682
Dec 23 16:45:36 gentoo kernel: Code: 4c 89 ee 49 39 d0 49 89 48 60 0f 95 c2 48 29 d8 48 83 c7 10 0f b6 d2 48 83 c8 01 48 c1 e2 02 48 09 da 48 83 ca 01 48 89 57 f8 <48> 89 41 08 48 89 3c 24 e8 21 c7 ff ff 48 8b 3c 24 e9 a6 f7 >
Dec 23 16:45:36 gentoo kernel: RSP: 002b:00007fff3d848220 EFLAGS: 00010206
Dec 23 16:45:36 gentoo kernel: RAX: 000000000000c971 RBX: 000000000000a010 RCX: 000060a931eff690
Dec 23 16:45:36 gentoo kernel: RDX: 000000000000a011 RSI: 000000000000a001 RDI: 000060a931ef5690
Dec 23 16:45:36 gentoo kernel: RBP: ffffffffffffffb0 R08: 000074bd9b3f6ac0 R09: 0000000000000004
Dec 23 16:45:36 gentoo kernel: R10: 0000000001ffffff R11: 0000000000000002 R12: 0000000000000002
Dec 23 16:45:36 gentoo kernel: R13: 000000000000a001 R14: 0000000000000000 R15: 000074bd9b3f6b20
Dec 23 16:45:36 gentoo kernel:  </TASK>

[hakavlad]

You can log each memory check:

min_mem_report_interval = 0

You can set more aggressive settings, increase the frequency of memory checks:

fill_rate_mem = 8000
fill_rate_swap = 4000
min_sleep = 0.02

[hakavlad]

Question: is it nohang missing the state when it needs to react, or is it the kernel killing the process prematurely, before the state at which nohang starts to react.

[dim-geo]

I see that nohang understands that there is a problem ongoing, but it doesn't react fast enough. OOM kills it first.
Even with these settings.

fill_rate_mem = 100
fill_rate_swap = 100
min_sleep = 0.02

[hakavlad]

fill_rate_mem = 100
fill_rate_swap = 100

It's too low! It can react in time if the memory fills up at a speed of up to 100 MiB/s, but not more (but checks beetween min_sleep and max_sleep).

[dim-geo]

Increased the fill_rate, but no change :(

[hakavlad]

run again nohang --memload. OOM killer was triggered

This shouldn't have happened. This command does not cause an OOM, it terminates itself if no one interferes with it.

# nohang --memload
Enter the numbers 381268 to confirm that you are not a robot: 381268
--------------------------------------------------------------------
Warning! The process will consume memory until 40 MiB of memory
(MemAvailable + SwapFree) remain free, and it will be terminated via SIGUSR1
at the end. This may cause the system to freeze and processes to terminate.
Do you want to continue? [No/Yes] Yes
Memory consumption has started!
MemAvailable: 34 MiB, SwapFree: 5 MiB                
Self terminating by SIGUSR1
User defined signal 1

[hakavlad]

• Try after swapoff
• What is your fs? (ZFS uses Adaptive Replacement Cache)
• What happens when other kernel versions used

[dim-geo]

Tried without any swap, same results. I don't use zfs, ext4 and btrfs. I use 6.12.5 now. It seems that oomkiller is triggered earlier than usual. Maybe cachy patches have something to do with it. Στις Τρί, 24 Δεκ 2024, 13:05 ο χρήστης Alexey Avramov < ***@***.***> έγραψε:

…

- Try after swapoff - What is your fs? (ZFS uses Adaptive Replacement Cache) - What happens when other kernel versions used — Reply to this email directly, view it on GitHub <#142 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABNOHTITMG42DCT7R7GVR7L2HE5YPAVCNFSM6AAAAABUCZ5AM6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKNRQHE4TMMBUGI> . You are receiving this because you authored the thread.Message ID: ***@***.***>