Skip to content

Add reacher-z UBC Vector 60-environment Track 2 corpus - #35

Open
reacher-z wants to merge 2 commits into
benchflow-ai:mainfrom
reacher-z:submission/reacher-z-ubc-vector-environments
Open

Add reacher-z UBC Vector 60-environment Track 2 corpus#35
reacher-z wants to merge 2 commits into
benchflow-ai:mainfrom
reacher-z:submission/reacher-z-ubc-vector-environments

Conversation

@reacher-z

Copy link
Copy Markdown

Track 2 environment entry: reacher-z-ubc-vector

Entry

  • Team: reacher-z-ubc-vector
  • Contact: reacher@cs.ubc.ca
  • Track: environments
  • Environments: 60
  • Author: Yuxuan Zhang
  • Affiliation: University of British Columbia; Vector Institute

The entry contains 12 deterministic workflow families with five distinct modes each: baseline audit, source-priority conflict resolution, as-of filtering, prior-attempt recovery, and adversarial distractor exclusion.

Validation

  • python3 scripts/check_task.py submissions/reacher-z-ubc-vector/envs: 60/60 valid
  • python3 scripts/check_submission.py: entry valid
  • Double host oracle/verifier runs: 120/120 pass, byte-identical per environment
  • Empty/nop checks: 60/60 rejected
  • Empty-answer mutants: 60/60 rejected
  • Official Docker oracle replays: 60/60 score 1.0
  • Official Docker empty-trial replays: 60/60 score 0.0
  • Prompt-body pairs with Jaccard >= 0.80: 0

Representative run_local.sh oracle tail:

3 passed, 1 warning in 0.02s
reward: 1.0
✓ capacity-schedule-adversarial — oracle scores 1.0

Representative run_local.sh --skip-oracle tail:

3 failed, 2 warnings in 0.16s
reward: 0.0
✓ capacity-schedule-adversarial — empty trial correctly scores 0.0

The environments are offline and use deterministic synthetic fixtures. The managed competition pipeline remains authoritative for leakage, reward-hacking, and difficulty review.

@reacher-z

Copy link
Copy Markdown
Author

Validation update: I retriggered the current base-repository tasks-check workflow at commit 6e690f0; GitHub currently reports the run as action_required, so a maintainer must approve the fork workflow before checks can attach to the PR. The same two self-contained commands pass locally on the exact commit: python3 scripts/check_task.py and python3 scripts/check_submission.py (including reacher-z-ubc-vector — valid). No environment payloads changed in this retrigger commit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant