Missing `plain_text` attribute in hf dataset

[Swing1wingS]

Hello, thank you for sharing this great work!
I've been trying to run the pipeline using the command given in README.md:

python -m zest.run_pipeline \
  --output_file ./outputs.jsonl \
  --language en \
  --examples_per_language 5 \
  --task full \
  --engine gpt-4.1-mini \
  --data_split dev

And I get the following error:

  File "/path/to/Lemonade/zest/run_pipeline.py", line 286, in main
    await runner.run_full_pipeline(
  File "/path/to/Lemonade/zest/run_pipeline.py", line 114, in run_full_pipeline
    examples, schema = await self.load_data(
                       ^^^^^^^^^^^^^^^^^^^^^
  File "/path/to/Lemonade/zest/run_pipeline.py", line 44, in load_data
    examples = load_dataset(
               ^^^^^^^^^^^^^
  File "/path/to/Lemonade/event_dataset/example.py", line 74, in load_dataset
    article=row["plain_text"],
            ~~~^^^^^^^^^^^^^^
KeyError: 'plain_text'

It seems like the dataset in huggingface lacks the attribute plain_text:

Could you please update the dataset (or corresponding code) to include plain_text?

This would be very helpful for reproducing and extending your results.
Thanks in advance! 🙏

[miro-janosik]

I confirm that I see the same issue.
It seems that this is not a full dataset, but dataset trimmed of the texts.

I agree it would be great if there would be all texts available.
It could be issue to provide these texts as these are probably licensed by the authors.
Anyone could download these news articles referred by the url with some tool like trafilatura for a personal use of course...

[Swing1wingS]

I confirm that I see the same issue. It seems that this is not a full dataset, but dataset trimmed of the texts.

I agree it would be great if there would be all texts available. It could be issue to provide these texts as these are probably licensed by the authors. Anyone could download these news articles referred by the url with some tool like trafilatura for a personal use of course...

@miro-janosik
I agree that manually downloading the texts via urls could be a workaround. However, as shown in the screenshot, some of the URLs are invalid (only contain website names or words like "PDF"), which makes it impossible to access all texts this way.

And I also agree with you that there may also be licensing issues involved here. So maybe a possible solution would be to filter out all the instances with invalid urls?