[mod & add] fix spo algorithm, add dapo and cispo algorithm in the RLAIF part.

[vanking20000918]

The code in the spo part maybe a little incomplete, that is, it misses the part of Prioritized Prompt Sampling in the original paper.

ps: The modification is refered from my presonal understanding of papaer and Germini. It could be wrong and takes it with caution.

[vanking20000918]

The main results of the test are that the policy_loss and advantage_mean differ from the source code:

1. policy_loss: It can be seen that the variance of the modified policy loss has significantly decreased, indicating a more certain direction for policy optimization, with trajectories leaning towards the correct direction. Personally, I believe this is due to the adoption of priority sampling from the original literature, which clarifies the direction of trajectory optimization and avoids the impact of intra-group degeneration.
2. adventage_mean: After modification, there is a noticeable trend of change opposite to the original, but the variance of the change is almost the same, which cannot be explained..

[vanking20000918]

Actually there is some difference from the original test, such as policy loss... Maybe it's due to the difference of SFT model.

[vanking20000918]

The DAPO algorithm in code is add clip-higher and dynamic sampling compared to GRPO algorithm, which refered to this paper.

In the experiment, I found these will stabilize the training process and policy loss.

[vanking20000918]

This main addition in CISPO algorithm refered to GRPO algorithm : By changing the gradient of out-of-bounds tokens from "directly set to 0" to "bounded clipping", we ensure that high-value exploration tokens can continue to participate in parameter updates while maintaining training stability.
Actually, this change is not very well in experiment, because the ratio is almost nearly at 1, which means seldom out-of-bound.

The refered sourced website

ps: CISPO algorithm in minimind project works very bad in experiment, so this script (train_cispo.py) only be refered.