Table 4 from the paper

[jsvir]

Thanks for your great work!

How can I reproduce the results from the Table 4? I don't find the references to the datasets.

[VainF]

Hi @jsvir, the datasets listed in Table 4 were extracted from an internal dataset, which has not been made public yet. If you’d like to obtain results across different domains, you can simply select any dataset from HuggingFace and train the mask on it.

[jsvir]

Thanks, I would like to reproduce the table you published. Can you share the statistics of the datasets for each domain? Number of samples? Original source?

[VainF]

Sorry, jsvir. I no longer have access to the dataset. But you can find similar data in open-source datasets such as StarCoder. For training purposes, around 200M tokens should be enough.