[TabArena] Benchmarking LimiX; Verifying our Implemenetation

[LennartPurucker]

Heyho LimiX Team,

We are working on benchmarking LimiX on TabArena. I have implemented and benchmarked a first version of your model in this pull request: autogluon/tabarena#208; looks great! :)

I have added comments regarding benchmarking and integration in the PR. I'm happy to discuss this further here or in the PR.
Ideally, we aim to maximize the performance of your model and learn how to use it correctly in the kind of data science pipelines we are benchmarking in TabArena (no Docker or DDP setup usually).

Feel free to test the implementation yourself or propose changes I should incorporate.
Once everyone is happy with the code, I can rerun the config and add its results to the public leaderboard.

Thank you!

[limix-ldm]

Hi Lennart,

Thank you for benchmarking and integrating our model! Regarding the issues you mentioned (VRAM optimization, the recommended search space, retrieval improvement, etc.), we will push an update once we fix these things, and we will let you know once it has been done.

Best regards,
The LimiX Team

[limix-ldm]

Hi Lennart,

Following AutoGluon, we release the search space of hyperparameters for LimiX. A new function, sample_inference_params() has been added to utils/inference_utils.py. This function generates a unique configuration with each call. To achieve optimal performance on a specific dataset, we recommend performing inference multiple times using different configurations produced by this function. We conducted experiments on TabArena, identifying the optimal configuration for each dataset through 100 trials, and the results are:

- Classification (mean AUC): 0.8524;
- Regression (mean R2): 0.8031;
- Regression (mean RMSE): 0.3976;

As five datasets were either too large or included unseen classes in the test set, we excluded them from performance evaluation, including APSFailure, SDSS17, customer_satisfaction_in_airline, GiveMeSomeCredit and diamonds. To address the high VRAM and time consumption of the retrieval method, we are working on optimizations and will release an update soon, please follow the future update of our repository ;)

Best regards,
The LimiX Team

[LennartPurucker]

Heyho, great to hear and thank you a lot!

I will add HPO to our experiments following the TabArena protocol!
I am quite busy right now, so it might take a bit longer to add these experiments.

---

Some comments:

included unseen classes in the test set

This is not a problem of the datasets but rather the evaluation protocol. It would lead me to believe you are not stratifying your outer test splits. At the same time, with 3-fold outer cross-validation (as in TabArena), this should very likely not occur even without stratification for the datasets in TabArena.

high VRAM

I was able to address the VRAM problem for 3/5 of these datasets by adding batching, see autogluon/tabarena#208

---

Another side note, I see you updated the default config. Is this the best average config you found on TabArena?

Best regards and thank you,
Lennart

[limix-ldm]

Hi Lennart,

Thanks for your comments!
The default configuration update is not related to TabArena but addresses a general issue. We identified an incorrect setting in the previous patch that harms retrieval performance, which has been resolved in this release.

Best regards,
The LimiX Team

[LennartPurucker]

Heyho @limix-ldm, with the update of the repo and paper, I would be highly interested in reproducing the results in the new version of the paper.

Can you share your TabArena implementation of LimiX? And the TabArena result atrifacts?

[LennartPurucker]

Also let me know how you would want to change the PR with the newest version: autogluon/tabarena#208

[limix-ldm]

Hi Lennart,

We just noticed your new comments on this older issue. Regarding the implementation: there is no specific LimiX integration in TabArena. You can simply run the open-source version of LimiX directly on TabArena. Recently, we were also contacted by other researchers about the evaluation datasets. The list of datasets is now available in the benchmark_list folder.

As for the PR, we’re happy to help. However, we currently have several parallel projects in progress, so we may not be able to work on it immediately. We’ll follow up and handle it once things settle down.