`"text"` is not included in `it2it` tasks

[KennethEnevoldsen]

When running this dataset EncyclopediaVQAIT2ITRetrieval:

We can check the dataloader:

dl = _create_queries_dataloader(
            dataset,
            task_metadata,
            batch_size=batch_size,
            input_column=input_column,
            num_proc=num_proc,
        )
next(iter(dl)).keys() # dict_keys(['image'])

even though the dataset:

Dataset({
    features: ['id', 'modality', 'text', 'image'],
    num_rows: 3743
})

@Samoed I assume that this is not desired behaviour would expect # dict_keys(['image', 'text'])

[KennethEnevoldsen]

In general I had a look at the dataloader generation code and I would probably suggest a refactor where we do all the dataset specific transformation together, and then afterwards do the dataloader conversion e.g. using a function like:

def _prepare_multimodal_dataset(
        dataset: Dataset,
    task_metadata: TaskMetadata,
    input_column: str | None = None, # TODO: why is this unused? 
    batch_size: int = 32, # TODO: I would generally not set the default at multiple layers - 
    # in case we don't pass it correctly, then this would not fails, but give an incorrect batch size
    num_proc: int | None = None,
):
    document_type = task_metadata.get_modalities(PromptType.document)
    dataset_cols = dataset.column_names
    transform = None # I don't see a case where this is not None am I missing something?
    collate_fn=_custom_collate_fn # TODO: same here - I would imagine this should be an input - might be overwritten later in which case this is fine

    if "text" in document_type: # if we want to fully generalize this, we should fork here for prompt type
        new_ds = dataset.map(
            _corpus_to_dict,
            desc="Standardizing text corpus format",
            num_proc=num_proc,
        )
        assert all(cn in new_ds.column_names for cn in dataset_cols), "ensure that no columns are removed" # TODO remove

    if "image" in document_type:
        new_ds = dataset.map(
            _convert_images_to_rgb,
            fn_kwargs={"image_col_name": "image", "transform": transform},
            desc="Converting images to RGB",
            num_proc=num_proc,
        )

    if "audio" in document_type:
        pass # seemingly no processing is needed or if so this is done in the collate_fn
    return new_ds # returned to a function that then handles the dataloader conversion

[Samoed]

Yes, agree