Dynamically generate the index.json file for datasets

[thinkh]

Current status

At the moment datasets for an application must be pre-defined in the /app/data/index.json. This is handy for a small number of datasets, but becomes very difficult to maintain for a lot of datasets.

Goal

Generate the index.json data structure dynamically.

Pointers

• Extend the dataset loading to not only load a static index.json, but load a python file and run a defined function, e.g., generate(), from that file.
• The function returns a Python dict() with the same structure as the index.json
• Loading of the index.json is described here

Use Case

• Adding and loading datasets on-the-fly for the malevo project