Repository navigation
agent_world_model_env: 1,681/10,000 code verifiers pass on untouched seeded databases #1475
Description
Environment
- OpenEnv
main@612056b569e99e1c358b02a49f1e8312ed20960c,envs/agent_world_model_env - Ubuntu 24.04 LTS (Linux 6.8), Python 3.12.3
- Data:
Snowflake/AgentWorldModel-1K, as downloaded by the env's data loader
What happens
reset() builds the scenario DB from gen_db.jsonl and gen_sample.jsonl, then copies it to {scenario}_initial.db (awm_environment.py#L224-L228). verify with verifier_mode="code" runs the task's verifier against the initial and final DB (awm_environment.py#L533-L539, verifier.py#L216-L222). A complete result is worth 1.0 (config.py#L31-L36).
Many code verifiers only check that the target row exists in the final DB. For a lot of tasks the seed data already has that row, so the verifier passes before the agent does anything.
For example, accounting_finance_1 task 0 asks to create a new customer with specified attributes. gen_sample.jsonl already inserts a matching customer, so the code verifier returns complete on the unchanged seeded DB. The verifier itself works: if I delete that row from the seed, the no-op returns others, and inserting the customer again returns complete. It just has nothing to check on the shipped seed.
Across the 10,000 tasks in this dataset download, the code verifier returns complete on 1,681 freshly seeded, unchanged DBs. Some verifiers check only the final DB, so they cannot distinguish a newly created row from one present at reset.
The README says code mode "compares initial and final database states" (README.md#L153). The AWM example recipe already filters no-op passes on the client side (task_select.py#L1-L11). The environment's task listing does not flag them.
Reproduce
This builds the DB the same way reset() does and calls the same run_verifier function as verify, without scenario tool calls. It omits the scenario subprocess startup and tool discovery performed by a real reset(), so the sweep measures verifier results on seeded DBs rather than end-to-end episode outcomes.
awm_noop.py
"""agent_world_model_env: build the DB exactly as reset() does, take no actions, run the code verifier.
Run from the repo root: PYTHONPATH=src:envs python awm_noop.py [--all]
"""
import shutil, sys, tempfile
from collections import Counter
from agent_world_model_env.server.data_loader import AWMDataLoader
from agent_world_model_env.server.db_manager import create_database, save_snapshot
from agent_world_model_env.server.verifier import run_verifier
loader = AWMDataLoader() # downloads Snowflake/AgentWorldModel-1K on first use
def noop_verify(scenario, task_idx):
entry = loader.get_verifier(scenario, task_idx, "code")
if entry is None:
return "no_verifier"
d = tempfile.mkdtemp()
try:
db, initial = f"{d}/{scenario}.db", f"{d}/{scenario}_initial.db"
create_database(db, loader.get_db_schema(scenario), loader.get_sample_data(scenario)) # reset()
save_snapshot(db, initial) # reset()
# no tool calls; then verify(verifier_mode="code") with no final_answer
reward_type, _ = run_verifier(entry, "code", initial, db, final_answer=None)
return reward_type
finally:
shutil.rmtree(d, ignore_errors=True)
if "--all" in sys.argv:
counts = Counter()
for s in loader.list_scenarios():
for i in range(s["num_tasks"]):
counts[noop_verify(s["name"], i)] += 1
print(dict(counts), "tasks:", sum(counts.values()))
else:
for scenario, i in [("accounting_finance_1", t) for t in range(10)] + [
("appointment_scheduling_1", 0), ("b2b_commerce_1", 0), ("analytics_dashboard_4", 0)]:
print(f"{scenario} #{i}: {noop_verify(scenario, i):9} {loader.get_tasks(scenario)[i][:90]}")Output excerpt (task descriptions redacted; PYTHONPATH=src:envs python awm_noop.py, network disabled):
accounting_finance_1 #0: complete Create a new customer with specified attributes
accounting_finance_1 #1: others Create an invoice for customer 'Acme Coffee Roasters' dated '2025-01-15' for 10 units of p
accounting_finance_1 #2: complete Add a new vendor with specified attributes
accounting_finance_1 #3: complete Record a new bill from vendor 'Bright Solutions IT' dated '2025-02-01' for invoice #BS-204
accounting_finance_1 #4: others Import or fetch all bank feed transactions from 2025-01-01 to 2025-01-31 for the 'Checking
accounting_finance_1 #5: complete Create a new non-inventory product/service called 'On-site Consultation (Hourly)' with a d
accounting_finance_1 #6: others Set up a recurring invoice schedule for customer 'Northside Fitness' to bill on the 1st of
accounting_finance_1 #7: others Record a manual journal entry dated '2025-02-28' to reclassify $1,200 from 'Office Supplie
accounting_finance_1 #8: others Generate a 'Profit and Loss' report on an accrual basis for the period 2025-01-01 to 2025-
accounting_finance_1 #9: complete Create a bank rule named 'Uber Rides' for the 'Checking Account' that automatically catego
appointment_scheduling_1 #0: complete Create a new pet owner profile with specified attributes
b2b_commerce_1 #0: complete Create a new part record with SKU 'BRK-TOY-001', manufacturer 'Bosch', part_number '0986AB
analytics_dashboard_4 #0: complete Create a new site for my organization named 'Chicago Distribution Center' with address '12
Full sweep (python awm_noop.py --all; stdout only, db_manager logs some seed-insert warnings to stderr):
{'complete': 1681, 'others': 8317, 'judge_error': 2} tasks: 10000
Impact
In code mode, an immediate verify call can receive the default 1.0 reward on these tasks without scenario tool calls. This can affect training or evaluation that uses an unfiltered task list; its effect on aggregate scores depends on task selection and the evaluation protocol.
Possible fix
- Flag or skip tasks whose verifier already passes at reset. The check in
examples/awm_expert_in_the_loop/task_select.pycould move into the env or the data loader, run once over the dataset, and ship as a list of excluded tasks. - For tasks that ask for a write, require a change from
initial_db_path(for example, returnotherswhen the final DB is identical to the initial one). - The seed rows and verifiers come from the Snowflake AgentWorldModel-1K dataset, so the cleaner fix may belong upstream there: don't pre-seed the row a "create" task asks for, and have verifiers compare against the initial DB.
Happy to help test a fix. Found while auditing public RL environments.
Activity
fix(awm): close no-op reward bypasses (openenv#1475)
burtenshaw commentedon Oct 26, 2026
What explicit contract distinguishes answer-verifier tasks from state-changing tasks, and how should empty-answer baseline errors be handled without rejecting valid answers or allowing no-op rewards? See #1477.
jimmy-holdout commentedon Oct 26, 2026
I measured all 10,000 tasks on main (9b2c0be) and #1477 (a5c0bf6). Each code verifier is classified by whether its body reads
final_answer(AST check), then run on the untouched seeded DB._verifier_runnerpassesfinal_answer or "", soNoneand""behave the same.final_answer?completecomplete"done":completeothersguard(1,680 here vs 1,681 in the issue; the env code is identical between 612056b and 9b2c0be, so the one-task gap is probably a date-dependent verifier.)
othersfor an empty answer, so fix: stop awm code verifiers from passing on an untouched database #1477's baseline permits valid completions in those seeded states. The other 12 already pass empty; on an unchanged DB, fix: stop awm code verifiers from passing on an untouched database #1477 now returnsothersfor them whatever the answer (e.g.event_management_6Env code #5, where the seed already holds both of Alice's reservations)."done"passes, 2 are lookup tasks whose verifiers accept"done"on the seed. The third (fleet_management_4Remove envtorch dir nesting and add base env code #2) is a create-or-report-conflict task. These cases need verifier review.completeon fix: stop awm code verifiers from passing on an untouched database #1477.Proposed contract, declared per task next to the verifier:
final_answer(2,358 tasks) is a useful review signal but not enough on its own, since some verifiers read it only on an error path. If required, an empty answer scoresothersbefore the verifier runs.others. Many tasks need both an answer and a write.completeor error for owner review, but don't use a baseline error to reject an otherwise valid submitted answer. With the baseline computed offline rather than re-run at verify time, yourfloat(final_answer)/json.loadscases can't turn a valid answer intojudge_error. I can write both up as regression tests.judge_error(0 reward under the default configuration) and never falls back to the DB or answer check.Upstream, the 1,668 DB-only baseline passes need task-level review: write tasks should check the intended transition from initial to final, while lookup and valid no-write tasks need their own completion criteria.
The sweep extends
awm_noop.pyfrom the issue: samereset()DB build, calling each checkout'srun_verifierwithNone,""and"done". Happy to share the script and per-task results as a gist if useful.