Bug: _shutdown_startup_interface lead to tons of swss log and fill the space

[wenjwang-nv]

Is it platform specific

generic

Importance or Severity

Critical

Description of the bug

In PR #23090, to make the test be compatible with multi asic platform, test_lldp_entry_table_after_all_batched_flap toggles L3 uplink ports one by one rather than toggle them all at once. Each toggle triggers BGP reconvergence; orchagent rewrites the ECMP routes and records every update into swss.rec. The cumulative 2.5 GB fills the 4 GB /var/log partition in our scenario, rsyslogd can no longer write the loganalyzer end marker, and the teardown times out.

Steps to Reproduce

this issue can happen when /var/log is half filled, and run the test can reproduce the issue

Actual Behavior and Expected Behavior

test failed at teardown because no end marker is found, because it is not be written to syslog because of capacity

Relevant log output

Failed: Processes "['analyze_logs--']" failed with exit code "1"
Exception:
run module command failed, Ansible Results =>
failed = True
changed = True
rc = 1
cmd = ['python', '/tmp/loganalyzer.py', '--action', 'add_end_marker', '--run_id', 'test_lldp_entry_table_after_all_batched_flap.2026-08-24-23:03:25']
start = 2026-08-25 02:24:32.227070
end = 2026-08-25 02:26:45.395070
delta = 0:02:13.168000
msg = non-zero return code
invocation = {'module_args': {'_raw_params': 'python /tmp/loganalyzer.py --action add_end_marker --run_id test_lldp_entry_table_after_all_batched_flap.2026-08-24-23:03:25', '_uses_shell': False, 'expand_argument_vars': True, 'stdin_add_newline': True, 'strip_empty_ends': True, 'cmd': None, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}
exception = (traceback unavailable)
_ansible_no_log = False
stdout =
stderr =
/tmp/loganalyzer.py:18: SyntaxWarning: invalid escape sequence '\ '
  -m /home/hrachya/projects/loganalyzer/match.file.1.log,/home/hrachya/projects/loganalyzer/match.file.2.log \    # noqa: E501 W605
Traceback (most recent call last):
  File "/tmp/loganalyzer.py", line 898, in 
    main(sys.argv[1:])
    ~~~~^^^^^^^^^^^^^^
  File "/tmp/loganalyzer.py", line 879, in main
    analyzer.place_marker(
    ~~~~~~~~~~~~~~~~~~~~~^
        log_file_list, analyzer.create_end_marker(), wait_for_marker=True)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/loganalyzer.py", line 280, in place_marker
    raise RuntimeError(
        "cannot find marker {} in /var/log/syslog".format(marker))
RuntimeError: cannot find marker end-LogAnalyzer-test_lldp_entry_table_after_all_batched_flap.2026-08-24-23:03:25 in /var/log/syslog
Traceback:
Traceback (most recent call last):
  File "/root/mars/workspace/sonic-mgmt/tests/common/helpers/parallel.py", line 98, in run
    Process.run(self)
  File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/root/mars/workspace/sonic-mgmt/tests/common/helpers/parallel.py", line 334, in wrapper
    target(*args, **kwargs)
  File "/root/mars/workspace/sonic-mgmt/tests/common/plugins/loganalyzer/__init__.py", line 111, in analyze_logs
    analyzer_summary = dut_analyzer.analyze(markers[node.hostname], fail_test, store_la_logs=store_la_logs)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/mars/workspace/sonic-mgmt/tests/common/plugins/loganalyzer/loganalyzer.py", line 466, in analyze
    self._add_end_marker(marker)
  File "/root/mars/workspace/sonic-mgmt/tests/common/plugins/loganalyzer/loganalyzer.py", line 212, in _add_end_marker
    self.ansible_host.command(cmd)
  File "/root/mars/workspace/sonic-mgmt/tests/common/devices/multi_asic.py", line 405, in _run_on_asics_wrapper
    return self._run_on_asics(attr, *module_args, **complex_args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/mars/workspace/sonic-mgmt/tests/common/devices/multi_asic.py", line 162, in _run_on_asics
    return getattr(self.sonichost, multi_asic_attr)(*module_args, **complex_args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/mars/workspace/sonic-mgmt/tests/common/devices/base.py", line 163, in _run_wrapper
    return self._run(module_name, *module_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/mars/workspace/sonic-mgmt/tests/common/devices/base.py", line 248, in _run
    raise RunAnsibleModuleFail("run module {} failed".format(module_name), hostname_res)
tests.common.errors.RunAnsibleModuleFail: run module command failed, Ansible Results =>
failed = True
changed = True
rc = 1
cmd = ['python', '/tmp/loganalyzer.py', '--action', 'add_end_marker', '--run_id', 'test_lldp_entry_table_after_all_batched_flap.2026-08-24-23:03:25']
start = 2026-08-25 02:24:32.227070
end = 2026-08-25 02:26:45.395070
delta = 0:02:13.168000
msg = non-zero return code
invocation = {'module_args': {'_raw_params': 'python /tmp/loganalyzer.py --action add_end_marker --run_id test_lldp_entry_table_after_all_batched_flap.2026-08-24-23:03:25', '_uses_shell': False, 'expand_argument_vars': True, 'stdin_add_newline': True, 'strip_empty_ends': True, 'cmd': None, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}
exception = (traceback unavailable)
_ansible_no_log = False
stdout =
stderr =
/tmp/loganalyzer.py:18: SyntaxWarning: invalid escape sequence '\ '
  -m /home/hrachya/projects/loganalyzer/match.file.1.log,/home/hrachya/projects/loganalyzer/match.file.2.log \    # noqa: E501 W605
Traceback (most recent call last):
  File "/tmp/loganalyzer.py", line 898, in 
    main(sys.argv[1:])
    ~~~~^^^^^^^^^^^^^^
  File "/tmp/loganalyzer.py", line 879, in main
    analyzer.place_marker(
    ~~~~~~~~~~~~~~~~~~~~~^
        log_file_list, analyzer.create_end_marker(), wait_for_marker=True)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/loganalyzer.py", line 280, in place_marker
    raise RuntimeError(
        "cannot find marker {} in /var/log/syslog".format(marker))
RuntimeError: cannot find marker end-LogAnalyzer-test_lldp_entry_table_after_all_batched_flap.2026-08-24-23:03:25 in /var/log/syslog

Output of show version

Attach files (if any)

No response