[Bug]: Batch mode does not work with 3008.0/1 ?

[tazaki]

What happened?

I am trying to run the salt command in batch mode.

[root@saltmaster ~]# salt '*' test.ping
minion1:
    True
saltmaster:
    True

[root@saltmaster ~]# salt -b 1 '*' test.ping

Executing run on ['minion1']

[ERROR   ] An un-handled exception was caught by Salt's global exception handler:
SaltClientError: Some exception handling minion payload
Traceback (most recent call last):
  File "/opt/saltstack/salt/lib/python3.14/site-packages/salt/client/__init__.py", line 539, in run_job
    pub_data = self.pub(
        tgt,
    ...<7 lines>...
        **kwargs,
    )
  File "/opt/saltstack/salt/lib/python3.14/site-packages/salt/client/__init__.py", line 2149, in pub
    raise PublishError(error)
salt.exceptions.PublishError: Some exception handling minion payload

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/salt", line 12, in <module>
    sys.exit(salt_main())
             ~~~~~~~~~^^
  File "/opt/saltstack/salt/lib/python3.14/site-packages/salt/scripts.py", line 586, in salt_main
    client.run()
    ~~~~~~~~~~^^
  File "/opt/saltstack/salt/lib/python3.14/site-packages/salt/cli/salt.py", line 50, in run
    self._run()
    ~~~~~~~~~^^
  File "/opt/saltstack/salt/lib/python3.14/site-packages/salt/cli/salt.py", line 76, in _run
    self._run_batch()
    ~~~~~~~~~~~~~~~^^
  File "/opt/saltstack/salt/lib/python3.14/site-packages/salt/cli/salt.py", line 322, in _run_batch
    for res, job_retcode in batch.run():
                            ~~~~~~~~~^^
  File "/opt/saltstack/salt/lib/python3.14/site-packages/salt/cli/batch.py", line 194, in run
    new_returns, timed_out = self._poll_iterators(
                             ~~~~~~~~~~~~~~~~~~~~^
        iters, minion_tracker, raw_mode, raw_by_minion
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/opt/saltstack/salt/lib/python3.14/site-packages/salt/cli/batch.py", line 317, in _poll_iterators
    part = next(queue)
  File "/opt/saltstack/salt/lib/python3.14/site-packages/salt/client/__init__.py", line 1117, in cmd_iter_no_block
    pub_data = self.run_job(
        tgt,
    ...<7 lines>...
        **kwargs,
    )
  File "/opt/saltstack/salt/lib/python3.14/site-packages/salt/client/__init__.py", line 561, in run_job
    raise SaltClientError(general_exception)
salt.exceptions.SaltClientError: Some exception handling minion payload
Traceback (most recent call last):
  File "/opt/saltstack/salt/lib/python3.14/site-packages/salt/client/__init__.py", line 539, in run_job
    pub_data = self.pub(
        tgt,
    ...<7 lines>...
        **kwargs,
    )
  File "/opt/saltstack/salt/lib/python3.14/site-packages/salt/client/__init__.py", line 2149, in pub
    raise PublishError(error)
salt.exceptions.PublishError: Some exception handling minion payload

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/salt", line 12, in <module>
    sys.exit(salt_main())
             ~~~~~~~~~^^
  File "/opt/saltstack/salt/lib/python3.14/site-packages/salt/scripts.py", line 586, in salt_main
    client.run()
    ~~~~~~~~~~^^
  File "/opt/saltstack/salt/lib/python3.14/site-packages/salt/cli/salt.py", line 50, in run
    self._run()
    ~~~~~~~~~^^
  File "/opt/saltstack/salt/lib/python3.14/site-packages/salt/cli/salt.py", line 76, in _run
    self._run_batch()
    ~~~~~~~~~~~~~~~^^
  File "/opt/saltstack/salt/lib/python3.14/site-packages/salt/cli/salt.py", line 322, in _run_batch
    for res, job_retcode in batch.run():
                            ~~~~~~~~~^^
  File "/opt/saltstack/salt/lib/python3.14/site-packages/salt/cli/batch.py", line 194, in run
    new_returns, timed_out = self._poll_iterators(
                             ~~~~~~~~~~~~~~~~~~~~^
        iters, minion_tracker, raw_mode, raw_by_minion
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/opt/saltstack/salt/lib/python3.14/site-packages/salt/cli/batch.py", line 317, in _poll_iterators
    part = next(queue)
  File "/opt/saltstack/salt/lib/python3.14/site-packages/salt/client/__init__.py", line 1117, in cmd_iter_no_block
    pub_data = self.run_job(
        tgt,
    ...<7 lines>...
        **kwargs,
    )
  File "/opt/saltstack/salt/lib/python3.14/site-packages/salt/client/__init__.py", line 561, in run_job
    raise SaltClientError(general_exception)
salt.exceptions.SaltClientError: Some exception handling minion payload

Type of salt install

Official rpm

Major version

3007.x

What supported OS are you seeing the problem on? Can select multiple. (If bug appears on an unsupported OS, please open a GitHub Discussion instead)

almalinux-9

salt --versions-report output

[root@saltmaster ~]# salt --versions
Salt Version:
          Salt: 3008.0

Python Version:
        Python: 3.14.5 (main, May 12 2026, 03:56:57) [GCC 11.2.0]

Dependency Versions:
          cffi: 2.0.0
      cherrypy: 18.10.0
  cryptography: 46.0.7
      dateutil: 2.9.0.post0
     docker-py: Not Installed
         gitdb: 4.0.12
     gitpython: 3.1.50
        Jinja2: 3.1.6
       libgit2: Not Installed
  looseversion: 1.3.0
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.1.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 24.0
     pycparser: 3.00
      pycrypto: Not Installed
  pycryptodome: 3.23.0
        pygit2: Not Installed
  python-gnupg: 0.5.6
        PyYAML: 6.0.3
         PyZMQ: 27.1.0
        relenv: 0.22.11
         smmap: 5.0.2
       timelib: 0.3.0
       Tornado: 6.5.5
           ZMQ: 4.3.5

Salt Extensions:
 saltext.vault: 1.6.0

Salt Package Information:
  Package Type: onedir

System Versions:
          dist: almalinux 9.8 Olive Jaguar
        locale: utf-8
       machine: x86_64
       release: 5.14.0-687.5.4.el9_8.x86_64
        system: Linux
       version: AlmaLinux 9.8 Olive Jaguar

[tazaki]

Same issue with 3008.1

[root@saltmaster ~]# salt '*' test.version
minion1:
    3008.1
saltmaster:
    3008.1
[root@saltmaster ~]# salt -b 1 '*' test.ping

Executing run on ['saltmaster']

[ERROR   ] An un-handled exception was caught by Salt's global exception handler:
SaltClientError: Some exception handling minion payload
Traceback (most recent call last):
  File "/opt/saltstack/salt/lib/python3.14/site-packages/salt/client/__init__.py", line 539, in run_job
    pub_data = self.pub(
        tgt,
    ...<7 lines>...
        **kwargs,
    )
  File "/opt/saltstack/salt/lib/python3.14/site-packages/salt/client/__init__.py", line 2149, in pub
    raise PublishError(error)
salt.exceptions.PublishError: Some exception handling minion payload

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/salt", line 12, in <module>
    sys.exit(salt_main())
             ~~~~~~~~~^^
  File "/opt/saltstack/salt/lib/python3.14/site-packages/salt/scripts.py", line 586, in salt_main
    client.run()
    ~~~~~~~~~~^^
  File "/opt/saltstack/salt/lib/python3.14/site-packages/salt/cli/salt.py", line 50, in run
    self._run()
    ~~~~~~~~~^^
  File "/opt/saltstack/salt/lib/python3.14/site-packages/salt/cli/salt.py", line 76, in _run
    self._run_batch()
    ~~~~~~~~~~~~~~~^^
  File "/opt/saltstack/salt/lib/python3.14/site-packages/salt/cli/salt.py", line 322, in _run_batch
    for res, job_retcode in batch.run():
                            ~~~~~~~~~^^
  File "/opt/saltstack/salt/lib/python3.14/site-packages/salt/cli/batch.py", line 194, in run
    new_returns, timed_out = self._poll_iterators(
                             ~~~~~~~~~~~~~~~~~~~~^
        iters, minion_tracker, raw_mode, raw_by_minion
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/opt/saltstack/salt/lib/python3.14/site-packages/salt/cli/batch.py", line 317, in _poll_iterators
    part = next(queue)
  File "/opt/saltstack/salt/lib/python3.14/site-packages/salt/client/__init__.py", line 1117, in cmd_iter_no_block
    pub_data = self.run_job(
        tgt,
    ...<7 lines>...
        **kwargs,
    )
  File "/opt/saltstack/salt/lib/python3.14/site-packages/salt/client/__init__.py", line 561, in run_job
    raise SaltClientError(general_exception)
salt.exceptions.SaltClientError: Some exception handling minion payload
Traceback (most recent call last):
  File "/opt/saltstack/salt/lib/python3.14/site-packages/salt/client/__init__.py", line 539, in run_job
    pub_data = self.pub(
        tgt,
    ...<7 lines>...
        **kwargs,
    )
  File "/opt/saltstack/salt/lib/python3.14/site-packages/salt/client/__init__.py", line 2149, in pub
    raise PublishError(error)
salt.exceptions.PublishError: Some exception handling minion payload

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/salt", line 12, in <module>
    sys.exit(salt_main())
             ~~~~~~~~~^^
  File "/opt/saltstack/salt/lib/python3.14/site-packages/salt/scripts.py", line 586, in salt_main
    client.run()
    ~~~~~~~~~~^^
  File "/opt/saltstack/salt/lib/python3.14/site-packages/salt/cli/salt.py", line 50, in run
    self._run()
    ~~~~~~~~~^^
  File "/opt/saltstack/salt/lib/python3.14/site-packages/salt/cli/salt.py", line 76, in _run
    self._run_batch()
    ~~~~~~~~~~~~~~~^^
  File "/opt/saltstack/salt/lib/python3.14/site-packages/salt/cli/salt.py", line 322, in _run_batch
    for res, job_retcode in batch.run():
                            ~~~~~~~~~^^
  File "/opt/saltstack/salt/lib/python3.14/site-packages/salt/cli/batch.py", line 194, in run
    new_returns, timed_out = self._poll_iterators(
                             ~~~~~~~~~~~~~~~~~~~~^
        iters, minion_tracker, raw_mode, raw_by_minion
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/opt/saltstack/salt/lib/python3.14/site-packages/salt/cli/batch.py", line 317, in _poll_iterators
    part = next(queue)
  File "/opt/saltstack/salt/lib/python3.14/site-packages/salt/client/__init__.py", line 1117, in cmd_iter_no_block
    pub_data = self.run_job(
        tgt,
    ...<7 lines>...
        **kwargs,
    )
  File "/opt/saltstack/salt/lib/python3.14/site-packages/salt/client/__init__.py", line 561, in run_job
    raise SaltClientError(general_exception)
salt.exceptions.SaltClientError: Some exception handling minion payload

[tazaki]

I stumbled upon something interesting... I noticed the jobs were being saved as root?
from the master log

2026-06-12 02:17:51,759 [salt.channel.server:474 ][ERROR   ][12712] Some exception handling a payload from minion
Traceback (most recent call last):
  File "/opt/saltstack/salt/lib/python3.14/site-packages/salt/returners/local_cache.py", line 114, in prep_jid
    with salt.utils.files.fopen(os.path.join(jid_dir, "jid"), "wb+") as fn_:
         ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/saltstack/salt/lib/python3.14/site-packages/salt/utils/files.py", line 531, in fopen
    f_handle = open(  # pylint: disable=resource-leakage,unspecified-encoding
        *args, **kwargs
    )
PermissionError: [Errno 13] Permission denied: '/var/cache/salt/master/jobs/2c/f15c6fe7546a1ca01f8931fd417218323b2aea0393f5f5c93596cb93713253/jid'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/saltstack/salt/lib/python3.14/site-packages/salt/returners/local_cache.py", line 114, in prep_jid
    with salt.utils.files.fopen(os.path.join(jid_dir, "jid"), "wb+") as fn_:
         ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/saltstack/salt/lib/python3.14/site-packages/salt/utils/files.py", line 531, in fopen
    f_handle = open(  # pylint: disable=resource-leakage,unspecified-encoding
        *args, **kwargs
    )
PermissionError: [Errno 13] Permission denied: '/var/cache/salt/master/jobs/2c/f15c6fe7546a1ca01f8931fd417218323b2aea0393f5f5c93596cb93713253/jid'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/saltstack/salt/lib/python3.14/site-packages/salt/returners/local_cache.py", line 114, in prep_jid
    with salt.utils.files.fopen(os.path.join(jid_dir, "jid"), "wb+") as fn_:
         ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/saltstack/salt/lib/python3.14/site-packages/salt/utils/files.py", line 531, in fopen
    f_handle = open(  # pylint: disable=resource-leakage,unspecified-encoding
        *args, **kwargs
    )
PermissionError: [Errno 13] Permission denied: '/var/cache/salt/master/jobs/2c/f15c6fe7546a1ca01f8931fd417218323b2aea0393f5f5c93596cb93713253/jid'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/saltstack/salt/lib/python3.14/site-packages/salt/returners/local_cache.py", line 114, in prep_jid
    with salt.utils.files.fopen(os.path.join(jid_dir, "jid"), "wb+") as fn_:
         ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/saltstack/salt/lib/python3.14/site-packages/salt/utils/files.py", line 531, in fopen
    f_handle = open(  # pylint: disable=resource-leakage,unspecified-encoding
        *args, **kwargs
    )
PermissionError: [Errno 13] Permission denied: '/var/cache/salt/master/jobs/2c/f15c6fe7546a1ca01f8931fd417218323b2aea0393f5f5c93596cb93713253/jid'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/saltstack/salt/lib/python3.14/site-packages/salt/returners/local_cache.py", line 114, in prep_jid
    with salt.utils.files.fopen(os.path.join(jid_dir, "jid"), "wb+") as fn_:
         ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/saltstack/salt/lib/python3.14/site-packages/salt/utils/files.py", line 531, in fopen
    f_handle = open(  # pylint: disable=resource-leakage,unspecified-encoding
        *args, **kwargs
    )
PermissionError: [Errno 13] Permission denied: '/var/cache/salt/master/jobs/2c/f15c6fe7546a1ca01f8931fd417218323b2aea0393f5f5c93596cb93713253/jid'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/saltstack/salt/lib/python3.14/site-packages/salt/channel/server.py", line 446, in handle_message
    ret, req_opts = await self.payload_handler(payload)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/saltstack/salt/lib/python3.14/site-packages/salt/master.py", line 1939, in _handle_payload
    ret = await self._handle_clear(load)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/saltstack/salt/lib/python3.14/site-packages/salt/master.py", line 2004, in _handle_clear
    reply = await method(load)
            ^^^^^^^^^^^^^^^^^^
  File "/opt/saltstack/salt/lib/python3.14/site-packages/salt/master.py", line 3972, in publish
    jid = self._prep_jid(clear_load, extra)
  File "/opt/saltstack/salt/lib/python3.14/site-packages/salt/master.py", line 4029, in _prep_jid
    jid = self.mminion.returners[fstr](nocache=nocache, passed_jid=passed_jid)
  File "/opt/saltstack/salt/lib/python3.14/site-packages/salt/loader/lazy.py", line 177, in __call__
    ret = self.loader.run(run_func, *args, **kwargs)
  File "/opt/saltstack/salt/lib/python3.14/site-packages/salt/loader/lazy.py", line 1401, in run
    return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
           ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/saltstack/salt/lib/python3.14/site-packages/salt/loader/lazy.py", line 1416, in _run_as
    ret = _func_or_method(*args, **kwargs)
  File "/opt/saltstack/salt/lib/python3.14/site-packages/salt/returners/local_cache.py", line 122, in prep_jid
    return prep_jid(
        passed_jid=jid, nocache=nocache, recurse_count=recurse_count + 1
    )
  File "/opt/saltstack/salt/lib/python3.14/site-packages/salt/returners/local_cache.py", line 122, in prep_jid
    return prep_jid(
        passed_jid=jid, nocache=nocache, recurse_count=recurse_count + 1
    )
  File "/opt/saltstack/salt/lib/python3.14/site-packages/salt/returners/local_cache.py", line 122, in prep_jid
    return prep_jid(
        passed_jid=jid, nocache=nocache, recurse_count=recurse_count + 1
    )
  [Previous line repeated 2 more times]
  File "/opt/saltstack/salt/lib/python3.14/site-packages/salt/returners/local_cache.py", line 95, in prep_jid
    raise salt.exceptions.SaltCacheError(err)
salt.exceptions.SaltCacheError: prep_jid could not store a jid after 5 tries.
2026-06-12 02:17:51,774 [salt._logging.impl:1129][ERROR   ][17655] An un-handled exception was caught by Salt's global exception handler:
SaltClientError: Some exception handling minion payload
Traceback (most recent call last):
  File "/opt/saltstack/salt/lib/python3.14/site-packages/salt/client/__init__.py", line 539, in run_job
    pub_data = self.pub(
        tgt,
    ...<7 lines>...
        **kwargs,
    )
  File "/opt/saltstack/salt/lib/python3.14/site-packages/salt/client/__init__.py", line 2149, in pub
    raise PublishError(error)
salt.exceptions.PublishError: Some exception handling minion payload

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/bin/salt", line 12, in <module>
    sys.exit(salt_main())
             ~~~~~~~~~^^
  File "/opt/saltstack/salt/lib/python3.14/site-packages/salt/scripts.py", line 586, in salt_main
    client.run()
    ~~~~~~~~~~^^
  File "/opt/saltstack/salt/lib/python3.14/site-packages/salt/cli/salt.py", line 50, in run
    self._run()
    ~~~~~~~~~^^
  File "/opt/saltstack/salt/lib/python3.14/site-packages/salt/cli/salt.py", line 76, in _run
    self._run_batch()
    ~~~~~~~~~~~~~~~^^
  File "/opt/saltstack/salt/lib/python3.14/site-packages/salt/cli/salt.py", line 322, in _run_batch
    for res, job_retcode in batch.run():
                            ~~~~~~~~~^^
  File "/opt/saltstack/salt/lib/python3.14/site-packages/salt/cli/batch.py", line 194, in run
    new_returns, timed_out = self._poll_iterators(
                             ~~~~~~~~~~~~~~~~~~~~^
        iters, minion_tracker, raw_mode, raw_by_minion
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/opt/saltstack/salt/lib/python3.14/site-packages/salt/cli/batch.py", line 317, in _poll_iterators
    part = next(queue)
  File "/opt/saltstack/salt/lib/python3.14/site-packages/salt/client/__init__.py", line 1117, in cmd_iter_no_block
    pub_data = self.run_job(
        tgt,
    ...<7 lines>...
        **kwargs,
    )
  File "/opt/saltstack/salt/lib/python3.14/site-packages/salt/client/__init__.py", line 561, in run_job
    raise SaltClientError(general_exception)
salt.exceptions.SaltClientError: Some exception handling minion payload
2026-06-12 02:17:57,390 [salt.loaded.int.returner.local_cache:450 ][ERROR   ][12697] Unable to remove /var/cache/salt/master/jobs/2c/f15c6fe7546a1ca01f8931fd417218323b2aea0393f5f5c93596cb93713253: [Errno 13] Permission denied: '/var/cache/salt/master/jobs/2c/f15c6fe7546a1ca01f8931fd417218323b2aea0393f5f5c93596cb93713253'

Checking perms:

2026-06-12 02:17:57,390 [salt.loaded.int.returner.local_cache:450 ][ERROR   ][12697] Unable to remove /var/cache/salt/master/jobs/2c/f15c6fe7546a1ca01f8931fd417218323b2aea0393f5f5c93596cb93713253: [Errno 13] Permission denied: '/var/cache/salt/master/jobs/2c/f15c6fe7546a1ca01f8931fd417218323b2aea0393f5f5c93596cb93713253'
[vagrant@saltmaster ~]$ sudo ls -lha /var/cache/salt/master/jobs/2c/f15c6fe7546a1ca01f8931fd417218323b2aea0393f5f5c93596cb93713253
total 4.0K
drwxr-xr-x. 2 root root  22 Jun 12 02:17 .
drwxr-xr-x. 3 root root  76 Jun 12 02:17 ..
-rw-r--r--. 1 root root 324 Jun 12 02:17 .batch.p

I also noticed if i switch the user to root in /etc/salt/master

user: salt

to

user: root

Then restart the master and try again...

[vagrant@saltmaster ~]$ sudo systemctl restart salt-master
[vagrant@saltmaster ~]$ sudo salt -b 1 '*' test.ping

Executing run on ['minion1']


Executing run on ['saltmaster']

jid:
    20260612022040846180
minion1:
    True
retcode:
    0
jid:
    20260612022040846180
retcode:
    0
saltmaster:
    True

It works! It does seem like its giving the full return by default?