:farmer: test_secure_publisher_subscriber tests failing consistently in fastrtp for linux debug and linux aarch64 debug

[miguelgonrod]

Generated by Generative AI

No response

Operating System:

Ubuntu Resolute

ROS version or commit hash:

Rolling

RMW implementation (if applicable):

No response

RMW Configuration (if applicable):

No response

Client library (if applicable):

No response

'ros2 doctor --report' output

No response

Steps to reproduce issue

1. Go to https://ci.ros2.org/job/nightly_linux_debug or https://ci.ros2.org/job/nightly_linux-aarch64_debug
2. Build a job
3. See tests failing

Expected behavior

To build without test regressions

Actual behavior

Building with lots of test regressions

Additional information

Reference Build

• nightly_linux_debug#3834
• https://ci.ros2.org/job/nightly_linux-aarch64_debug/3567

Failing tests

projectroot.test_secure_publisher_subscriber__Empty__rmw_fastrtps_cpp__secure_comm_0
projectroot.test_secure_publisher_subscriber__Empty__rmw_fastrtps_cpp__secure_comm_1
projectroot.test_secure_publisher_subscriber__Empty__rmw_fastrtps_cpp__secure_comm_2
projectroot.test_secure_publisher_subscriber__Empty__rmw_fastrtps_cpp__secure_comm_3
projectroot.test_secure_publisher_subscriber__Empty__rmw_fastrtps_cpp__secure_not_connecting_0
projectroot.test_secure_publisher_subscriber__Empty__rmw_fastrtps_cpp__secure_not_connecting_1
projectroot.test_secure_publisher_subscriber__UnboundedSequences__rmw_fastrtps_cpp__secure_comm_0
projectroot.test_secure_publisher_subscriber__UnboundedSequences__rmw_fastrtps_cpp__secure_comm_1
projectroot.test_secure_publisher_subscriber__UnboundedSequences__rmw_fastrtps_cpp__secure_comm_2
projectroot.test_secure_publisher_subscriber__UnboundedSequences__rmw_fastrtps_cpp__secure_comm_3
projectroot.test_secure_publisher_subscriber__UnboundedSequences__rmw_fastrtps_cpp__secure_not_connecting_0
projectroot.test_secure_publisher_subscriber__UnboundedSequences__rmw_fastrtps_cpp__secure_not_connecting_1
test_security.TestSecurePublisherSubscriberAfterShutdown.test_processes_finished_gracefully
test_security.TestSecurePublisherSubscriberAfterShutdown.test_processes_finished_gracefully
test_security.TestSecurePublisherSubscriberAfterShutdown.test_processes_finished_gracefully
test_security.TestSecurePublisherSubscriberAfterShutdown.test_processes_finished_gracefully
test_security.TestSecurePublisherSubscriberAfterShutdown.test_processes_finished_gracefully
test_security.TestSecurePublisherSubscriberAfterShutdown.test_processes_finished_gracefully
test_security.TestSecurePublisherSubscriberAfterShutdown.test_processes_finished_gracefully
test_security.TestSecurePublisherSubscriberAfterShutdown.test_processes_finished_gracefully
test_security.TestSecurePublisherSubscriberAfterShutdown.test_processes_finished_gracefully
test_security.TestSecurePublisherSubscriberAfterShutdown.test_processes_finished_gracefully
test_security.TestSecurePublisherSubscriberAfterShutdown.test_processes_finished_gracefully
test_security.TestSecurePublisherSubscriberAfterShutdown.test_processes_finished_gracefully

Log Output

[test_subscriber-2] [INFO] [1778476613.354867111] [rcl]: Found security directory: /home/jenkins-agent/workspace/nightly_linux_debug/ws/build/test_security/test/test_security_files/enclaves/subscriber
[test_publisher-1] /usr/include/c++/15/bits/shared_ptr_base.h:1344: _Tp* std::__shared_ptr_deref(_Tp*) [with _Tp = eprosima::fastdds::rtps::security::Handle]: Assertion '__p != nullptr' failed.
[test_subscriber-2] /usr/include/c++/15/bits/shared_ptr_base.h:1344: _Tp* std::__shared_ptr_deref(_Tp*) [with _Tp = eprosima::fastdds::rtps::security::Handle]: Assertion '__p != nullptr' failed.
[ERROR] [test_subscriber-2]: process has died [pid 371526, exit code -6, cmd '/home/jenkins-agent/workspace/nightly_linux_debug/ws/build/test_security/test_secure_subscriber_cpp Empty false /'].
ok

----------------------------------------------------------------------
Ran 1 test in 0.277s

OK
[INFO] [test_publisher-1]: sending signal 'SIGINT' to process[test_publisher-1]
[ERROR] [test_publisher-1]: process has died [pid 371525, exit code -6, cmd '/home/jenkins-agent/workspace/nightly_linux_debug/ws/build/test_security/test_secure_publisher_cpp Empty /'].
test_processes_finished_gracefully (test_security.TestSecurePublisherSubscriberAfterShutdown.test_processes_finished_gracefully)
Test that both executables finished gracefully. ... FAIL

======================================================================
FAIL: test_processes_finished_gracefully (test_security.TestSecurePublisherSubscriberAfterShutdown.test_processes_finished_gracefully)
Test that both executables finished gracefully.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jenkins-agent/workspace/nightly_linux_debug/ws/build/test_security/test_secure_publisher_subscriber__Empty__rmw_fastrtps_cpp__secure_comm_0_Debug.py", line 86, in test_processes_finished_gracefully
    launch_testing.asserts.assertExitCodes(
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        proc_info,
        ^^^^^^^^^^
        [launch_testing.asserts.EXIT_OK],
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        'test_subscriber'
        ^^^^^^^^^^^^^^^^^
    )
    ^
  File "/home/jenkins-agent/workspace/nightly_linux_debug/ws/install/launch_testing/lib/python3.14/site-packages/launch_testing/asserts/assert_exit_codes.py", line 62, in assertExitCodes
    assert info.returncode in allowable_exit_codes, 'Proc {} exited with code {}'.format(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: Proc test_subscriber-2 exited with code -6

Flakyness 15 days

job_name	last_fail	first_fail	build_count	failure_count	failure_percentage
nightly_linux_debug	2026-05-11	2026-04-29	11	10	90.91
nightly_linux-aarch64_debug	2026-05-11	2026-04-29	12	10	83.33

I didn't found major changes that could have caused this regression. Began to happen in https://ci.ros2.org/job/nightly_linux_debug/3822

CC: @Crola1702

[fujitatomoya]

@MiguelCompany eProsima/Fast-DDS#6371 could be related to this?
EDP matching path calls into security_manager().remove_reader() / remove_writer() and then into the cached listener->on_*_matched(...).
the new mutex narrows the listener-call window, which can let a matching callback fire after the security manager has begun releasing its shared_ptr, which may be producing the null deref the libstdc++ debug header catches.

i am not sure about this, but see possibility here. what do you think?

[MiguelCompany]

eProsima/Fast-DDS#6371 could be related to this?

I agree that this might have surfaced due to that PR (especially taking into account that it was merged the day before the first failure reported here), but the root cause is not related to those changes.

We will investigate.