Skip to content

TimeH2O-v1 wrapper and fev-bench submission - #205

Merged
shchur merged 1 commit into
autogluon:mainfrom
Far0n:add-timeh2o-v1
Oct 23, 2026
Merged

shchur merged 1 commit into
autogluon:mainfrom
Far0n:add-timeh2o-v1

Conversation

@Far0n

@Far0n Far0n commented Oct 22, 2026

Copy link
Copy Markdown
Contributor

Hi again,

Following up on #201, we'd like to submit TimeH2O-v1 to fev-bench.
It's a 108.8M-parameter time-series foundation model that runs through the h2ofm package.

On the full benchmark it gets a SQL skill score of 50.8 and a MASE skill score of 40.0, with a median runtime of 2.8 s per 100 series on a single RTX 5090.

This PR adds:

  • models/timeh2o-v1/: the wrapper and requirements.txt (h2ofm==0.43.0)
  • benchmarks/fev_bench/results/timeh2o-v1.csv: results for all 100 tasks
  • an entry in models.yaml

To reproduce:

uv run python models/evaluate.py -m timeh2o-v1 -b benchmarks/fev_bench/tasks.yaml

A few notes:

  • This PR currently sets hidden: true on the timeh2o-v1-tiny entry.
  • The weights are gated on Hugging Face, so access has to be requested on the model page before running.
  • The model's training data has no overlap with the fev-bench datasets.
  • h2ofm requires Python 3.12 or newer (we used 3.14).

The submission tests pass locally.

Thanks again!

Cheers,
Far0n

@shchur
shchur merged commit e750af6 into autogluon:main Oct 23, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants