Is it multilnigual or "en" only?

[zidsi]

Interesting paper. Regarding pretrained model I'm wondering - are they Roberta based or R-XLM? Did you evaluate performance wrt mDeberta as base model?

And finally - how would one use such/this model to score contexts beyond 512 tokens?

[tanyuqian]

Thanks a lot for noticing Cappy!!

Cappy is based on RoBERTa, so it's English-only. We haven't tried other base models for now, while it is expected that Cappy will be enabled with multilingual ability when the training approach is applied on a multilingual base model.

For long contexts beyond 512 tokens, we apply the longest_first truncation strategy mentioned here, which truncates Cappy's input (instruction, response) token by token, removing a token from the longer sequence in the pair until length 512 is reached.

[zidsi]

Thank you for your respone. Any plans for mCappy :) ? Looking forward.

Cross language transfer should work, but I guess you training dataset will not be released. So alternative approach I see to try to replicate your work in multilingual fashion would be to use existing DPO/RLHR datasets as training dataset?

Waiting for mCappy would be my choice of course.

[tanyuqian]

We will release Cappy's pretraining data! Hopefully within April.

However, I guess mCappy is not on the plan of our group :(