[mrsergazinov]

Summary

This PR adds four models: RealMLP-A, MLP-A, TabM-A and TabICLv2-A. Each model takes the base model then divides it into modules (e.g. numeric embeddings, categorical embeddings, loss, optimizer, etc). For each module, an agent wrote candidate implementations. The candidate choice for each module is a categorical hyperparameter. Index 0 in each module is the base model, so the default config is the base model.

Paper: Agentic Search Spaces for Tabular Machine Learning. Code: yandex-research/agentic-hpset.

Details

Search spaces

• RealMLP-A: the TabArena RealMLP space, plus the candidate indices.
• TabM-A: the TabArena TabM space, plus the candidate indices.
• MLP-A: default MLP with Periodic Embeddings from RTDL of Gorishnyi 2022. MLP-A uses this base MLP HP space, plus the agentic candidates.
• TabICLv2-A: the inference recipes of the package around the frozen TabICLv2. Ensemble is fixed at n_ens=8.

Wrappers

• The three neural networks use the X_val / y_val that TabArena gives.
• TabICLv2-A fits as TabICLv2 does (refit_folds=True). The context is the full training set.
• MLP-A and TabM-A stop training between epochs at the time limit.
• TabICLv2-A uses the same recipe and checkpoints as TabICLv2. Agentic version runs attention in float32.

Failed configurations

On TabArena-Lite:

• RealMLP-A and TabICLv2-A: none have failed.
• MLP-A: 88 failed. TabM-A: 179 failed. These configurations diverge.
• The default config never failed. TabArena uses the default in place of a failed config. Hence, we have adopted the same convention.

Notes

• The package needs Python 3.12 or newer.
• TabICLv2-A loads its checkpoints by file name from jingang/TabICL, as TabICLv2 does.
• On APSFailure, TabICLv2-A needs more GPU memory than on other datasets. We set TABICL_CHUNK_CELL_BUDGET=12000000 for it.

Tests

• ruff check and ruff format --check: pass.
• pytest -m models -k "RealMLP-A or MLP-A or TabM-A": 9 passed, 3 skipped.
• pytest -m models -k "TabICLv2-A": 2 passed, 2 skipped. The skipped tests need more than 10 classes.
• python -m tabarena.tools.sync_pyproject_extras: the four extras agree with info.py.

Model or system submission

Kind: model (four models: RealMLP-A, MLP-A, TabM-A, TabICLv2-A)

Files

• Model: models/<key>/{__init__,model,hpo,info}.py, a lazy entry in models/__init__.py, the extra in packages/tabarena/pyproject.toml (also listed in extended), the family in website/website_format.py. A tests/tabarena/models/smoke_configs.py override only if the toy fit needs it; tests specific to the model in models/<key>/tests/ (outside CI), never in tests/, and no edits to shared test infrastructure.

Implementation

• Model: _fit uses the X_val / y_val that TabArena passes (no internal cross-validation, hold-out split or tuning on the training data), and the search space in hpo.py supports about 200 configurations. Variants are search-space parameters, not separate models.
• time_limit, num_cpus / num_gpus and the seed are wired through; nothing is monkey-patched globally; optional imports stay inside the wrapper; the model pickles for parallel bagging.
• The dependency is pip-installable with an exact pin (PyPI version or git commit, no vendored code) and the extra matches pip_extra in info.py; Hugging Face checkpoints pin a revision; maintainers can run it without credentials or gated weights; the license, the paper and documentation links, and the supported problem types are stated.
• ruff check and ruff format --check pass. Model: pytest -m models -k <Key> passes.

Results (we need these before we re-run the method)

• TabArena-Lite (subset="lite") with the official pipeline: the default plus about 25 random HPO configurations where the method has a search space; or the BeyondArena core2k subset. The run used the arena's official validation protocol (the default); a run made with official_validation_protocol=False says so here and why.
• The hardware (CPU, GPU, RAM) and the entry-point script are stated below.
• Optional: a link to the run's output directory (the expname folder with the results.pkl files, zipped or as a GitHub / Hugging Face release) so we can verify and integrate the results directly.
• Who signs off on the maintainer re-run on behalf of the authors: @mrsergazinov

TabArena-Lite Elo, 51 datasets, default config plus 200 random configs per model:

Method	Default	Tuned	Tuned + ensembled
TabICLv2-A	1591	1618	1635
TabICLv2	1561	—	—
RealMLP-A	1302	1392	1487
RealMLP	1237	1420	1485
TabM-A	1259	1380	1476
TabM	1271	1362	1398
MLP-A	1177	1276	1415

Results: mrsergazinov/agentic-search-spaces-tabarena-lite (one zip per model, expname layout, with predictions).

Hardware: 1 GPU (>= A100 80GB), 8 CPU cores and 96 GB RAM per job.

Command: one job per config and dataset. Each job runs TabArenaV0pt1ExperimentBundle(models=[(<method>, 200)]).build_experiments(num_cpus=8, memory_limit=90), keeps one config, then runs TabArenaContext().build_and_run_jobs([experiment], subset="lite", build_kwargs={"dataset_names": [<dataset>]}).

---

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[LennartPurucker]

Heyho, thanks so much; very cool submissions!

This is kind of special and, to some extent, a resubmission of existing models, but also very different. The TabArena team is thinking about how to integrate or accept these submissions. Below, I raise some questions to help with that process.

---

MLP-A: 88 failed. TabM-A: 179 failed. These configurations diverge.
The default config never failed. TabArena uses the default in place of a failed config. Hence, we have adopted the same convention.

This is not TabArena's convention. All individual configs must be successful. Why did they fail/diverge?

One concern is the differences between the normal models and their agentic versions (like TabICLv2 and RealMLP). The difference might be only noise from using TabArena-Lite. Do you have results for TabArena-Full, e.g., for TabICLv2 and RealMLP (maybe with a smaller budget or just the defaults)?

Some other thoughts:

• We stopped adding tuning for TFMs, mostly due to the high cost for little gains. We could investigate adding it again for TabICL, but so far the TabICL authors haven't wanted to.
• There might be some concerns around leakage of the best configs (in general, due to using LLMs). We need to understand how to flag this on the leaderboard, if at all.

[mrsergazinov]

Hey! Thank you for getting back to us so quickly!

Failed configurations

We (and the agents of course) investigated the failures and found that all of them come from numerical divergence (non-finite loss or predictions).

Model	Category	Runs	Configs	Datasets
MLP-A	warm-up dummy fit fails	39	3 (r58, r150, r164)	13 (all regression)
MLP-A	non-finite predictions	41	36	4
MLP-A	non-finite loss / NaN in metric	8	7	4
TabM-A	non-finite loss	79	7	42
TabM-A	non-finite predictions / NaN in score / no valid checkpoint	100	14	41

The likely culprits are high learning rate and low training prevision (bf16), which combined with the agentic modules leads to instabilities during training. We sampled the configurations randomly, so it's sort of expected that this would happen.

We see a few options to fix it:

1. Divergence guard in the wrapper: at the first non-finite loss, stop training and keep the best checkpoint so far. Also, add a checker for predictions that would catch non-finite predictions and fail gracefully to default value.
2. Resample configurations. We could try to narrow down the search ranges, resample configurations and rerun, but this feels like we are overfitting the benchmark.

Results on full TabArena

We do not have TabArena-Full results yet. We agree that some of the gains could be noise. We think we can run TabICLv2-A or RealMLP-A, but full results would take sometime. In the meantime to proceed with the submission to TabArena, would you consider accepting any of the cheaper options:

1. 3 outer train-test splits
2. subset of smaller datasets

Tuning for TFMs

We agree that out-of-the-box, it's non-trivial to tune the TFMs: they don't have a pre-defined HP set, nor the guidance on what to touch to make them better. In our case, we have found the following combination to work:

1. Each ensemble member uses different augmentation, target / feature preprocessing, and feature permutation. This creates diversity of views.
2. Agent has helped to expand the banks of candidates.

So, possibly with this new recipe, it's a good idea to revisit tuning TFMs.

Leakage

We agree that this needs disclosure. TabArena was not the primary benchmark of our paper. The agents only proposed candidate modules: they did not evaluate or score any model on real datasets, and they did not know that the resulting models would be tested on a benchmark. We cannot fully exclude that an LLM proposed a module that exploits a known dataset, so we are happy with a flag on the leaderboard.

[mrsergazinov]

Hi again!

We ran TabICLv2-A on the full TabArena (all outer train-test splits). We have confirmed that the reported results are most likely not noise from the lite version. TabICLv2-A reaches 1623 Elo vs. 1559 for TabICLv2. Head-to-head, it has a lower error on 39 of 51 datasets. We uploaded these results to HF too: https://huggingface.co/datasets/mrsergazinov/agentic-search-spaces-tabarena-lite. We attach the Elo figures on the full TabArena below.

[LennartPurucker]

Heyho, thanks for the quick replies! A few comments to continue the discussion below.
Note that we still need to discuss this with the larger maintainer group, which I plan to do by the end of next week.

We see a few options to fix it: Divergence guard in the wrapper [...] Resample configurations.

This would require fixes in the wrapper/model code. We want to ensure that no config breaks or crashes. We guarantee this for our TabM and RealMLP model implementations as well, and we also fixed a bug that caused this for/after BeyondArena. So I would advise first fixing the code so this cannot happen again.

Can you replay all the failed configs and fix their crashes? They might need a guard or some dtpye fixes.

TabICLv2-A or RealMLP-A, but full results would take sometime. In the meantime to proceed with the submission to TabArena, would you consider accepting any of the cheaper options: 3 outer train-test splits, subset of smaller datasets

Adding these models and their full search config also costs us a significant amount on our side of the submission (I would assume somewhere between 2k-10k$). So the question is whether it's worth adding them. Hence, it would be great to know the results hold up and differ significantly from the existing RealMLP and TabM submissions to avoid duplicate entries. The current risk is mostly that -Lite can be very noisy, as we know from past experiences (like this one: #298)

To reduce this risk, I think we can check a few cheaper options, such as:

• does the default config generalize to -Full?
• do 10-25 random search configs generalize?

We can also start by looking into the first three folds. However, as most of the noise comes from the tiny datasets, which need many more folds, it is hard to judge. We sadly don't have a -Core subset for TabArena, only for BeyondArena.

Leakage

I sadly haven't had time to read your paper fully yet, so to follow up: was it all zero-shot, or did the agents build the search space through some sort of loop based on real-world data? And then how much overlap between this "training" data and TabArena do we expect? I am not too worried about this, but it would be great to know and explore this aspect.

TabICL Tuning & Results

How large is the delta between the default configurations? And where does this delta in the defaults come from? Based on the current results, I am not super convinced that the tuned + ensembled TabICLv2-A would outperform a standard tuned + ensembled TabICLv2. It is all within the CIs of Elo, although the CIs aren't very meaningful here.

And as long as the results are very similar, the risk of adding "duplicated" entries to the leaderboard remains a problem.