Segfault after module setup and nginx reload

[eliteaz]

Bug Overview

systemd-coredump[2348953]: [🡕] Process 2192085 (nginx) of user 0 dumped core.
                           
                           Stack trace of thread 2192085:
                           #0  0x00007feba2d6b39e __memcmp_evex_movbe (libc.so.6 + 0x16b39e)
                           #1  0x00007feb9f986c95 _ZN9nginx_sys6string81_$LT$impl$u20$core..cmp..PartialEq$u20$for$u20$nginx_sys..bindings..ngx_str_t$GT$2eq17hb0fae718d2ca9>
                           #2  0x00007feb9f9490d9 _ZN10nginx_acme4conf14cmd_add_issuer17h2e46cd5a8b82c47aE (ngx_http_acme_module.so + 0x6d0d9)
                           #3  0x0000560ab4d1833c ngx_conf_parse (nginx + 0x4933c)
                           #4  0x0000560ab4d59006 ngx_http_block.lto_priv.0 (nginx + 0x8a006)
                           #5  0x0000560ab4d1833c ngx_conf_parse (nginx + 0x4933c)
                           #6  0x0000560ab4d191a8 ngx_init_cycle (nginx + 0x4a1a8)
                           #7  0x0000560ab4d2fb73 ngx_master_process_cycle (nginx + 0x60b73)
                           #8  0x0000560ab4d03839 main (nginx + 0x34839)
                           #9  0x00007feba2c2a610 __libc_start_call_main (libc.so.6 + 0x2a610)
                           #10 0x00007feba2c2a6c0 __libc_start_main@@GLIBC_2.34 (libc.so.6 + 0x2a6c0)
                           #11 0x0000560ab4d03885 _start (nginx + 0x34885)
                           ELF object binary architecture: AMD x86-64
systemd[1]: nginx.service: Main process exited, code=dumped, status=11/SEGV
systemd[1]: nginx.service: Failed with result 'core-dump'.
systemd[1]: nginx.service: Consumed 7min 3.248s CPU time, 351.2M memory peak.

Expected Behavior

Do not segfault when I do systemctl reload nginx after initially module setup

Steps to Reproduce the Bug

yum install nginx-module-acme

load_module modules/ngx_http_acme_module.so;

directive acme_issuer under http{}

    acme_issuer zerossl {
        uri https://acme.zerossl.com/v2/DV90;
        state_path /var/cache/nginx/acme-zerossl;
        accept_terms_of_service;
        external_account_key 'KEY' /etc/nginx/zerossl-eab.key;
}

Define acme_certificate under server{}

    acme_certificate zerossl;

    ssl_certificate $acme_certificate;
    ssl_certificate_key $acme_certificate_key;

#systemctl reload nginx

Environment Details

[root@ops conf.d]# yum list installed | grep -e nginx -e glibc
glibc.x86_64                         2.34-256.el9               @baseos         
glibc-common.x86_64                  2.34-256.el9               @baseos         
glibc-gconv-extra.x86_64             2.34-256.el9               @baseos         
glibc-langpack-en.x86_64             2.34-256.el9               @baseos         
nginx.x86_64                         2:1.29.5-1.el9.ngx         @nginx          
nginx-module-acme.x86_64             1:1.29.5+0.3.1-1.el9.ngx   @nginx          

and OS ver.

[root@ops conf.d]# cat /proc/version 
Linux version 5.14.0-615.el9.x86_64 (mockbuild@x86-05.stream.rdu2.redhat.com) (gcc (GCC) 11.5.0 20240719 (Red Hat 11.5.0-11), GNU ld version 2.35.2-67.el9) #1 SMP PREEMPT_DYNAMIC Fri Sep 12 23:10:41 UTC 2025

Additional Context

after same setup and nginx restart it works properly, but reload fails

[bavshin-f5]

I'm afraid I cannot reproduce this.
The backtrace points here,

nginx-acme/src/conf/issuer.rs

Line 310 in cd2c4b7

if let Some(oissuer) = oamcf.issuer(&self.name) {

where we look through the list of configured acme issuers from the previous configuration. This data is guaranteed to be valid while we are loading the configuration, unless something else corrupted the memory of the previous configuration cycle.

Can you post the full configuration required to reproduce this, collected with nginx -T? Feel free to sanitize any sensitive values.

I'd also appreciate if you could open one of the cores with coredumpctl gdb and run the following commands:

info registers
x/s $rcx
x/s $rsi

[eliteaz]

thanks for info @bavshin-f5
nginx -t or nginx -T shows valid configs without any mistakes
paste coredumpctl gdb output:

[root@ops conf.d]# coredumpctl gdb
           PID: 1042432 (nginx)
           UID: 0 (root)
           GID: 0 (root)
        Signal: 11 (SEGV)
     Timestamp: Tue 2026-03-03 07:28:33 UTC (12min ago)
  Command Line: $'nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf'
    Executable: /usr/sbin/nginx
 Control Group: /system.slice/nginx.service
          Unit: nginx.service
         Slice: system.slice
       Boot ID: cc2c5d79a15048a28886bea1b651b093
    Machine ID: 148235bb51e74e00b16fe536d4fc81b7
      Hostname: ops
       Storage: /var/lib/systemd/coredump/core.nginx.0.cc2c5d79a15048a28886bea1b651b093.1042432.1772522913000000.zst (present)
  Size on Disk: 354.3K
       Message: Process 1042432 (nginx) of user 0 dumped core.
                
                Stack trace of thread 1042432:
                #0  0x00007fcf6375a559 __memcmp_avx2_movbe (libc.so.6 + 0x15a559)
                #1  0x00007fcf64106c95 _ZN9nginx_sys6string81_$LT$impl$u20$core..cmp..PartialEq$u20$for$u20$nginx_sys..bindings..ngx_str_t$GT$2eq17hb0fae718d2ca9eebE (ngx_http_acme_module.so + 0xaac95)
                #2  0x00007fcf640c90d9 _ZN10nginx_acme4conf14cmd_add_issuer17h2e46cd5a8b82c47aE (ngx_http_acme_module.so + 0x6d0d9)
                #3  0x000055ef7e26a33c ngx_conf_parse (nginx + 0x4933c)
                #4  0x000055ef7e26c23e ngx_conf_include (nginx + 0x4b23e)
                #5  0x000055ef7e26a33c ngx_conf_parse (nginx + 0x4933c)
                #6  0x000055ef7e2ab006 ngx_http_block.lto_priv.0 (nginx + 0x8a006)
                #7  0x000055ef7e26a33c ngx_conf_parse (nginx + 0x4933c)
                #8  0x000055ef7e26b1a8 ngx_init_cycle (nginx + 0x4a1a8)
                #9  0x000055ef7e281b73 ngx_master_process_cycle (nginx + 0x60b73)
                #10 0x000055ef7e255839 main (nginx + 0x34839)
                #11 0x00007fcf6362a610 __libc_start_call_main (libc.so.6 + 0x2a610)
                #12 0x00007fcf6362a6c0 __libc_start_main@@GLIBC_2.34 (libc.so.6 + 0x2a6c0)
                #13 0x000055ef7e255885 _start (nginx + 0x34885)
                ELF object binary architecture: AMD x86-64

GNU gdb (CentOS Stream) 16.3-2.el9
Copyright (C) 2024 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/sbin/nginx...

This GDB supports auto-downloading debuginfo from the following URLs:
  <https://debuginfod.centos.org/>
Enable debuginfod for this session? (y or [n]) info registers
Please answer y or [n].
Enable debuginfod for this session? (y or [n]) 
Debuginfod has been disabled.
To make this setting permanent, add 'set debuginfod enabled off' to .gdbinit.
Reading symbols from .gnu_debugdata for /usr/sbin/nginx...
(No debugging symbols found in .gnu_debugdata for /usr/sbin/nginx)

warning: Can't open file /dev/zero (deleted) during file-backed mapping note processing
[New LWP 1042432]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `nginx: master process /usr/sbin/nginx -c'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007fcf6375a559 in __memcmp_avx2_movbe () from /lib64/libc.so.6
Missing rpms, try: dnf --enablerepo='*debug*' install nginx-debuginfo-1.29.5-1.el9.ngx.x86_64 libxcrypt-debuginfo-4.4.18-3.el9.x86_64 pcre2-debuginfo-10.40-6.el9.x86_64 openssl-libs-debuginfo-3.5.5-1.el9.x86_64 zlib-debuginfo-1.2.11-41.el9.x86_64 glibc-debuginfo-2.34-256.el9.x86_64 sssd-client-debuginfo-2.9.8-1.el9.x86_64 nginx-module-acme-debuginfo-1.29.5+0.3.1-1.el9.ngx.x86_64 libgcc-debuginfo-11.5.0-14.el9.x86_64
(gdb) info registers
rax            0x453               1107
rbx            0x8396bb3fe04498    37038952923808920
rcx            0x55ef7f948452      94487125984338
rdx            0x7                 7
rsi            0x55ef7f948452      94487125984338
rdi            0x1                 1
rbp            0x0                 0x0
rsp            0x7fff04281248      0x7fff04281248
r8             0x0                 0
r9             0x10                16
r10            0x10                16
r11            0x1                 1
r12            0x55ef7f948452      94487125984338
r13            0x55ef7f846b38      94487124929336
r14            0x0                 0
r15            0x7                 7
rip            0x7fcf6375a559      0x7fcf6375a559 <__memcmp_avx2_movbe+761>
eflags         0x10283             [ CF SF IF RF ]
cs             0x33                51
ss             0x2b                43
ds             0x0                 0
es             0x0                 0
fs             0x0                 0
gs             0x0                 0
fs_base        0x7fcf6418e740      140528714311488
gs_base        0x0                 0
(gdb) x/s $rcx
0x55ef7f948452:	"zerossl"
(gdb) x/s $rsi
0x55ef7f948452:	"zerossl"
(gdb) quit

[eliteaz]

more detailed dump with enabled debuginfod for session

[root@ops nginx]# coredumpctl gdb
           PID: 1477907 (nginx)
           UID: 0 (root)
           GID: 0 (root)
        Signal: 11 (SEGV)
     Timestamp: Fri 2026-03-06 08:18:02 UTC (13s ago)
  Command Line: $'nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf'
    Executable: /usr/sbin/nginx
 Control Group: /system.slice/nginx.service
          Unit: nginx.service
         Slice: system.slice
       Boot ID: cc2c5d79a15048a28886bea1b651b093
    Machine ID: 148235bb51e74e00b16fe536d4fc81b7
      Hostname: ops
       Storage: /var/lib/systemd/coredump/core.nginx.0.cc2c5d79a15048a28886bea1b651b093.1477907.1772785082000000.zst (present)
  Size on Disk: 307.1K
       Message: Process 1477907 (nginx) of user 0 dumped core.
                
                Stack trace of thread 1477907:
                #0  0x00007f0eb215a559 __memcmp_avx2_movbe (libc.so.6 + 0x15a559)
                #1  0x00007f0eb2ab6c95 _ZN9nginx_sys6string81_$LT$impl$u20$core..cmp..PartialEq$u20$for$u20$nginx_sys..bindings..ngx_str_t$GT$2eq17hb0fae718d2ca9eebE (ngx_http_acme_module.so + 0xaac95)
                #2  0x00007f0eb2a790d9 _ZN10nginx_acme4conf14cmd_add_issuer17h2e46cd5a8b82c47aE (ngx_http_acme_module.so + 0x6d0d9)
                #3  0x000055e63702a33c ngx_conf_parse (nginx + 0x4933c)
                #4  0x000055e63702c23e ngx_conf_include (nginx + 0x4b23e)
                #5  0x000055e63702a33c ngx_conf_parse (nginx + 0x4933c)
                #6  0x000055e63706b006 ngx_http_block.lto_priv.0 (nginx + 0x8a006)
                #7  0x000055e63702a33c ngx_conf_parse (nginx + 0x4933c)
                #8  0x000055e63702b1a8 ngx_init_cycle (nginx + 0x4a1a8)
                #9  0x000055e637041b73 ngx_master_process_cycle (nginx + 0x60b73)
                #10 0x000055e637015839 main (nginx + 0x34839)
                #11 0x00007f0eb202a610 __libc_start_call_main (libc.so.6 + 0x2a610)
                #12 0x00007f0eb202a6c0 __libc_start_main@@GLIBC_2.34 (libc.so.6 + 0x2a6c0)
                #13 0x000055e637015885 _start (nginx + 0x34885)
                ELF object binary architecture: AMD x86-64

GNU gdb (CentOS Stream) 16.3-2.el9
Copyright (C) 2024 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/sbin/nginx...

This GDB supports auto-downloading debuginfo from the following URLs:
  <https://debuginfod.centos.org/>
Enable debuginfod for this session? (y or [n]) y
Debuginfod has been enabled.
To make this setting permanent, add 'set debuginfod enabled on' to .gdbinit.
Reading symbols from .gnu_debugdata for /usr/sbin/nginx...
(No debugging symbols found in .gnu_debugdata for /usr/sbin/nginx)

warning: Can't open file /dev/zero (deleted) during file-backed mapping note processing
[New LWP 1477907]
Downloading 471.91 K separate debug info for /lib64/libcrypt.so.2
Downloading 61.51 K separate debug info for /root/.cache/debuginfod_client/6ce4e5eb200e61d07398af52f8bcb316cf8466e0/debuginfo                                                                                      
Downloading 1.47 M separate debug info for /lib64/libpcre2-8.so.0                                                                                                                                                  
Downloading 72.58 K separate debug info for /root/.cache/debuginfod_client/481282542036ac05da5dbbe987bb1feae7f7eb4a/debuginfo                                                                                      
Downloading 3.48 M separate debug info for /lib64/libssl.so.3                                                                                                                                                      
Downloading 689.45 K separate debug info for /root/.cache/debuginfod_client/454c2bdb384e576cbb12284caf750ed5e94179f9/debuginfo                                                                                     
Downloading 13.90 M separate debug info for /lib64/libcrypto.so.3                                                                                                                                                  
Downloading 208.80 K separate debug info for /lib64/libz.so.1                                                                                                                                                      
Downloading 6.50 M separate debug info for /lib64/libc.so.6                                                                                                                                                        
Downloading 177.23 K separate debug info for /lib64/libnss_sss.so.2                                                                                                                                                
Downloading 470.37 K separate debug info for /root/.cache/debuginfod_client/f1c2870bdf989bbed69c33483a5eb2e8173e7c7d/debuginfo                                                                                     
Downloading 375.47 K separate debug info for /lib64/libgcc_s.so.1                                                                                                                                                  
--Type <RET> for more, q to quit, c to continue without paging--c                                                                                                                                                  
Downloading 112.69 M separate debug info for /root/.cache/debuginfod_client/a29b0cf0634decd576d636cf5b9b6412afeb4fac/debuginfo
Downloading separate debug info for system-supplied DSO at 0x7ffeca1cb000                                                                                                                                          
[Thread debugging using libthread_db enabled]                                                                                                                                                                      
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `nginx: master process /usr/sbin/nginx -c'.
Program terminated with signal SIGSEGV, Segmentation fault.
Download failed: Invalid argument.  Continuing without source file ../sysdeps/x86_64/multiarch/../sysdeps/x86_64/multiarch/memcmp-avx2-movbe.S.
#0  __memcmp_avx2_movbe () at ../sysdeps/x86_64/multiarch/memcmp-avx2-movbe.S:413

warning: 413	../sysdeps/x86_64/multiarch/memcmp-avx2-movbe.S: No such file or directory
Missing rpms, try: dnf --enablerepo='*debug*' install nginx-debuginfo-1.29.5-1.el9.ngx.x86_64 nginx-module-acme-debuginfo-1.29.5+0.3.1-1.el9.ngx.x86_64
(gdb) x/s $rcx
0x55e639055452:	"zerossl"
(gdb) x/s $rsi
0x55e639055452:	"zerossl"
(gdb) quit