SACD Discrete Soft Actor Critic

[splatter96]

This PR introduces the Soft Actor Critic for discrete actions (SACD) algorithm.

Description

This PR implements the SAC-Discrete algorithm as described in this paper https://arxiv.org/abs/1910.07207. This implementation borrows code from the papers original implementation (https://github.com/p-christ/Deep-Reinforcement-Learning-Algorithms-with-PyTorch) as well as provided by the issues author who requested this feature in stable baselines (https://github.com/toshikwa/sac-discrete.pytorch)

Context

• I have raised an issue to propose this change (required)
• Original issue in the stable baselines repo [Feature request] Implement SAC-Discrete DLR-RM/stable-baselines3#157

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• Documentation (update in the documentation)

Checklist:

• I've read the CONTRIBUTION guide (required)
• The functionality/performance matches that of the source (required for new training algorithms or training-related features).
• I have updated the tests accordingly (required for a bug fix or a new feature).
• I have included an example of using the feature (required for new features).
• I have included baseline results (required for new training algorithms or training-related features).
• I have updated the documentation accordingly.
• I have updated the changelog accordingly (required).
• I have reformatted the code using make format (required)
• I have checked the codestyle using make check-codestyle and make lint (required)
• I have ensured make pytest and make type both pass. (required)

Note: we are using a maximum length of 127 characters per line

[araffin]

Hello,
thanks for the PR =)

The functionality/performance matches that of the source (required for new training algorithms or training-related features).

please don't forget that part (see contributing guide).
I think there are discussion about the results here too: vwxyzjn/cleanrl#270

[splatter96]

Hello,
thanks for the feedback :)
Sorry for the late reply! Should I add the performance comparison to the source similarly as it is done in the official stable baselines3 algorithm pages? As in create a baselines3-zoo config for it and add the plots to this PR?

[araffin]

yes please =)

[daniel-redder]

Hello, thanks for the feedback :) Sorry for the late reply! Should I add the performance comparison to the source similarly as it is done in the official stable baselines3 algorithm pages? As in create a baselines3-zoo config for it and add the plots to this PR?

Do you have the performance results for this? I came across this PR looking for implementations on SACD. Thank you.

[splatter96]

Hello, thanks for the feedback :) Sorry for the late reply! Should I add the performance comparison to the source similarly as it is done in the official stable baselines3 algorithm pages? As in create a baselines3-zoo config for it and add the plots to this PR?

Do you have the performance results for this? I came across this PR looking for implementations on SACD. Thank you.

Unfortunately I never found the time to do the performance benchmark. I however use this implementation in several projects of mine with good results. So the implementation seems to be correct.

[gyunt]

It seems self.features_extractor is None is not handled in forward method.

[alektebel]

@araffin @splatter96

Hi, I’d like to contribute to this PR by adding the rl-baselines3-zoo benchmarks.

Plan:

• Run 5 seeds each on:
  • CartPole-v1 (MlpPolicy)
  • MsPacmanNoFrameskip-v4 (CnnPolicy)
  • BreakoutNoFrameskip-v4 (CnnPolicy)
• Train for 1 million timesteps per environment
• Generate the standard zoo learning-curve plots (mean ± std)
• Compare visually/qualitatively against the original p-christ and toshikwa results

I already tested the PR locally on CartPole and it trains perfectly (reaches ~250 reward very fast).

Before I run the full 5-seed Atari experiments, could you confirm the environments and timestep budget look good, or would you prefer anything different?

I can have the results ready within the next few days and either post the plots here or open a small follow-up PR in rl-baselines3-zoo if that’s cleaner.

Thanks!