feat(packaging): add DOMjudge problem packager by rsalesc · Pull Request #598 · rsalesc/rbx

rsalesc · 2026-06-12T21:49:46Z

Summary

Adds rbx package domjudge, building a DOMjudge-importable problem zip. The layout follows the ICPC problem package format plus DOMjudge extensions, mirroring what pol2dom produces (known to import cleanly):

domjudge-problem.ini      # short-name, name, exact fractional timelimit, contest color
problem.yaml              # limits {memory, output}, validation[, validator_flags]
problem.pdf               # the already-built main statement PDF
data/sample/, data/secret # 001.in/001.ans per-directory counters
output_validators/        # custom checker (flattened) + patched testlib.h + rbx.h
submissions/              # jury solutions by expected outcome

Design doc: docs/plans/2026-06-12-domjudge-packager-design.md.

Key decisions

Checker mapping: checkers resolving to the bundled builtins map to DOMjudge's default output validator (wcmp/ncmp/yesno → no flags, dcmp → float_tolerance 1e-6). Anything else — including a same-named local file, which may have been user-edited — ships as a custom validator, flattened via flattening.build_flat_namespace (cross-dir includes rewritten).
testlib patch (domjudge/testlib_patch.py): DOMjudge validators speak the Kattis protocol (exit 42/43, team output on stdin, feedback dir), which vanilla testlib doesn't. Ports pol2dom's patch (from cn-xcpc-tools/testlib-for-domjudge), applied to the bundled testlib at package time; raises if an anchor is missing so a testlib upgrade fails tests, not packages. Verified locally: patched wcmp compiles and exits 42/43 with judgemessage.txt under the DOMjudge calling convention.
Solutions: ACCEPTED→accepted, WA→wrong_answer, TLE→time_limit_exceeded, RTE/MLE→run_time_error (DOMjudge reports MLE as RTE); ambiguous outcomes skipped with a console note.
Limits: uses the domjudge limits profile when saved, else package limits (not required, unlike BOCA — DOMjudge has a single per-problem TL).
Scope: BATCH only; rbx COMMUNICATION pairs interactor+checker, which doesn't map onto DOMjudge's single output validator (follow-up). Build only — no upload.

Testing

12 new tests in tests/rbx/box/packaging/test_domjudge.py (testlib patch against bundled testlib, ini/yaml content, builtin-vs-custom checker resolution, flattening, submissions mapping, sample/secret routing, full package() smoke).
tests/rbx/box/packaging + tests/rbx/box/completion: 1139 passed (3 docker-compose BOCA e2e errors are pre-existing local-env failures).
Completion spec regenerated (mise run gen-completion-spec); drift test green.
Docs: new docs/setters/packaging/domjudge.md + index table row + mkdocs nav.

🤖 Generated with Claude Code

rsalesc · 2026-06-13T03:21:46Z

+    'yesno.cpp': None,
+    'dcmp.cpp': 'float_tolerance 1e-6',
+}
+


please let's not resort to domjudge default output validators. let's always honor the rbx checker.

Done in 087f800. Dropped the default-validator mapping entirely — the rbx checker is now always shipped as a custom output validator (validation: custom), so DOMjudge judges with exactly the checker rbx uses locally. This applies to the builtins (wcmp, ncmp, …) too. Verified locally that a builtin checker compiles and exits 42/43 under the DOMjudge calling convention after the testlib patch.

rsalesc · 2026-06-13T03:22:23Z

+    ExpectedOutcome.MEMORY_LIMIT_EXCEEDED: 'run_time_error',
+}
+
+_CPP_SUFFIXES = {'.cpp', '.cc', '.cxx', '.c++'}


please use the language kinds for the checker to identify this

Done in 087f800. Now uses LanguageKind.CPP in environment.language_kinds(code.find_language(checker)) instead of the hardcoded suffix set, so detection derives from the actual toolchain and is robust to custom language names. Added test_output_validators_reject_non_cpp_checker to cover the rejection path.

rsalesc · 2026-06-13T03:25:04Z

+}
+
+# DOMjudge reports MLE as RTE by default, hence the MLE mapping.
+_SUBMISSION_DIRS = {


could you explain this better? i'm not sure i like the idea of solutions with certain outcomes just vanishing from the upload. please research how these submissions are used, when they matter, which ones we can map to their outcomes, etc. please specifically validate that domjudge (not only problem package format) supports what you're stating.

You're right — dropping solutions was wrong. I researched how DOMjudge actually handles jury submissions (verified against the DOMjudge 9.0 source, not just the problem-package spec) and reworked it in 087f800 so no solution is ever dropped.

How DOMjudge uses these: on import it auto-judges every solution in submissions/ and records an expected verdict per submission, then surfaces any mismatch on the jury Judging verifier page (it never blocks the import). The expected verdict comes from one of two places:

Directory name, when it normalizes to a real verdict via DOMjudge's PROBLEM_RESULT_REMAP table (SubmissionService.php). That's 7 dirs, not 4: accepted, wrong_answer, time_limit_exceeded, run_time_error, output_limit, compiler_error, no_output.

An @EXPECTED_RESULTS@: source annotation, which supports multiple acceptable verdicts (e.g. CORRECT, TIMELIMIT). Crucially, in a verdict-named dir DOMjudge collapses the annotation to the dir's single verdict; in a non-verdict dir (e.g. mixed/) it keeps the full list. So multi-verdict expectations must live in a non-verdict dir.

New mapping (every outcome shipped):

rbx outcome placement

ACCEPTED / WRONG_ANSWER / TIME_LIMIT_EXCEEDED / RUNTIME_ERROR matching standard dir, no annotation

OUTPUT_LIMIT_EXCEEDED output_limit/ (DOMjudge extension dir)

ACCEPTED_OR_TLE mixed/ + @EXPECTED_RESULTS@: CORRECT, TIMELIMIT

TLE_OR_RTE mixed/ + TIMELIMIT, RUN-ERROR

INCORRECT mixed/ + all non-CORRECT verdicts

ANY mixed/ + all verdicts

MEMORY_LIMIT_EXCEEDED mixed/ + RUN-ERROR, TIMELIMIT

The one thing DOMjudge genuinely can't represent: there is no memory-limit verdict and no @EXPECTED_RESULTS@ token for it — an over-memory run surfaces as RTE (sometimes TLE). So MLE is the single lossy mapping; I encode it as RUN-ERROR, TIMELIMIT and documented the caveat. Everything else is exact.

Annotation comment prefix is # for Python, // otherwise. Docs/CLAUDE.md updated with the table and a note that mixed/ solutions trigger a harmless "result does not match directory" message on import. Covered by test_submissions_single_verdict_use_standard_dirs and test_submissions_ambiguous_outcomes_use_mixed_dir_with_annotation.

`rbx package domjudge` builds a DOMjudge-importable zip following the ICPC problem package format plus DOMjudge extensions, mirroring the layout pol2dom produces: domjudge-problem.ini + problem.yaml metadata, the already-built statement PDF as problem.pdf, data/sample + data/secret testcases, jury solutions under submissions/, and custom checkers flattened into output_validators/ with a testlib.h patched to speak the Kattis validator protocol (exit 42/43, team output on stdin, feedback dir). Builtin wcmp/ncmp/yesno/dcmp checkers map to DOMjudge's default validator instead. BATCH problems only for now. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

- Always ship the rbx checker as a custom output validator (validation: custom); never fall back to DOMjudge's default validators, so DOMjudge judges with the same checker rbx uses locally. Drops the wcmp/ncmp/yesno/dcmp -> default mapping. - Detect the C++ checker via language kinds (LanguageKind.CPP in environment.language_kinds(find_language)) instead of hardcoded file suffixes. - Ship every solution; none are dropped. Single-verdict outcomes go to the matching standard submission directory (incl. output_limit for OLE); multi-verdict outcomes (MLE, ACCEPTED_OR_TLE, TLE_OR_RTE, INCORRECT, ANY) go to submissions/mixed/ with an @EXPECTED_RESULTS@ annotation listing every acceptable DOMjudge verdict. MLE is the one lossy mapping (DOMjudge has no memory-limit verdict). Updates docs, CLAUDE.md and the design doc to match the corrected DOMjudge submission semantics (directory name -> verdict via remap table; annotation honored verbatim in non-verdict dirs). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

`rbx package domjudge` was missing from the checked-in CLI reference. Inserted by hand rather than regenerating the file, since the generator rewrites the whole page and it has drifted from what is committed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016iAZKnGrrkbA3PvGXx7q86

rsalesc · 2026-08-31T12:01:22Z

Rebased onto current main (the branch was 178 commits behind and conflicting).

Conflicts resolved: mkdocs nav and the packaging CLAUDE.md command table, both of which had gained a MOJ row on main; the packaging docs index table was rebuilt so the newer MOJ row fits the wider column padding this PR introduced.

Drift checks — a clean textual merge does not mean the 6-month-old code still fits today's APIs, so these were verified directly:

Every API the packager calls still exists with a compatible signature (naming.*, limits_info.get_limits, flattening.build_flat_namespace, code.find_language, ...).
The testlib patch still finds all of its anchors in the currently bundled testlib (it raises if one goes missing).
tests/rbx/box/packaging + tests/rbx/box/completion + lazy_cli_test: 1581 passed, 6 skipped, including the completion-spec drift test.

End-to-end verification: built a real package from a fresh --preset default problem with generated secret tests plus WA/INCORRECT solutions. Layout is correct (domjudge-problem.ini, problem.yaml, problem.pdf, data/sample + data/secret, output_validators/, submissions/ routed by outcome with mixed/ carrying the @EXPECTED_RESULTS@ annotation). The shipped validator compiles and obeys the Kattis protocol: exit 42 on accept, 43 on reject, judgemessage.txt written to the feedback dir.

One addition: rbx package domjudge was missing from the checked-in CLI reference. Added by hand rather than regenerating, since the generator rewrites that whole page and it has drifted from what is committed.

Scope is unchanged: BATCH-only, build-only (no upload, no interactive problems).

codecov · 2026-08-31T12:08:56Z

❌ 3 Tests Failed:

Tests completed	Failed	Passed	Skipped
6585	3	6582	9

View the top 3 failed test(s) by shortest run time

tests/rbx/box/test_timing_run_all_cli.py::test_a_passing_extra_run_still_returns_the_profile

Stack Traces | 0.005s run time

tmp_path = PosixPath('.../pytest-of-runner/pytest-0/test_a_passing_extra_run_still0')

    async def test_a_passing_extra_run_still_returns_the_profile(tmp_path: pathlib.Path):
>       profile, _ = await _compute(tmp_path, run_all=True)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

.../rbx/box/test_timing_run_all_cli.py:187: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
.../rbx/box/test_timing_run_all_cli.py:153: in _compute
    profile = await timing.compute_time_limits(
rbx/box/timing.py:1837: in compute_time_limits
    estimated_tl.estimationChecksum = estimation_checksum.compute().encode()
                                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
rbx/box/estimation_checksum.py:294: in compute
    solutions = _solutions_segment()
                ^^^^^^^^^^^^^^^^^^^^
rbx/box/estimation_checksum.py:174: in _solutions_segment
    for solution in package.get_solutions():
                    ^^^^^^^^^^^^^^^^^^^^^^^
rbx/box/package.py:461: in get_solutions
    package = find_problem_package_or_die(root)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

root = PosixPath('.')

    def find_problem_package_or_die(root: pathlib.Path = pathlib.Path()) -> Package:
        package = find_problem_package(root)
        if package is None:
            console.console.print(f'[error]Problem not found in {root.absolute()}[/error]')
>           raise typer.Exit(1)
E           click.exceptions.Exit: 1

rbx/box/package.py:81: Exit

tests/rbx/box/test_timing_run_all_cli.py::test_the_extra_run_only_happens_when_asked

Stack Traces | 0.005s run time

tmp_path = PosixPath('.../pytest-of-runner/pytest-0/test_the_extra_run_only_happen0')

    async def test_the_extra_run_only_happens_when_asked(tmp_path: pathlib.Path):
>       _, run_remaining = await _compute(tmp_path, run_all=False)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

.../rbx/box/test_timing_run_all_cli.py:160: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
.../rbx/box/test_timing_run_all_cli.py:153: in _compute
    profile = await timing.compute_time_limits(
rbx/box/timing.py:1837: in compute_time_limits
    estimated_tl.estimationChecksum = estimation_checksum.compute().encode()
                                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
rbx/box/estimation_checksum.py:294: in compute
    solutions = _solutions_segment()
                ^^^^^^^^^^^^^^^^^^^^
rbx/box/estimation_checksum.py:174: in _solutions_segment
    for solution in package.get_solutions():
                    ^^^^^^^^^^^^^^^^^^^^^^^
rbx/box/package.py:461: in get_solutions
    package = find_problem_package_or_die(root)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

root = PosixPath('.')

    def find_problem_package_or_die(root: pathlib.Path = pathlib.Path()) -> Package:
        package = find_problem_package(root)
        if package is None:
            console.console.print(f'[error]Problem not found in {root.absolute()}[/error]')
>           raise typer.Exit(1)
E           click.exceptions.Exit: 1

rbx/box/package.py:81: Exit

tests/rbx/box/test_timing_run_all_cli.py::test_the_solutions_the_estimate_ran_are_not_run_again

Stack Traces | 0.005s run time

tmp_path = PosixPath('.../pytest-of-runner/pytest-0/test_the_solutions_the_estimat0')

    async def test_the_solutions_the_estimate_ran_are_not_run_again(
        tmp_path: pathlib.Path,
    ):
        lower = [_solution('sols/ac.cpp', ExpectedOutcome.ACCEPTED)]
    
>       _, run_remaining = await _compute(tmp_path, run_all=True, lower=lower)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

.../rbx/box/test_timing_run_all_cli.py:170: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
.../rbx/box/test_timing_run_all_cli.py:153: in _compute
    profile = await timing.compute_time_limits(
rbx/box/timing.py:1837: in compute_time_limits
    estimated_tl.estimationChecksum = estimation_checksum.compute().encode()
                                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
rbx/box/estimation_checksum.py:294: in compute
    solutions = _solutions_segment()
                ^^^^^^^^^^^^^^^^^^^^
rbx/box/estimation_checksum.py:174: in _solutions_segment
    for solution in package.get_solutions():
                    ^^^^^^^^^^^^^^^^^^^^^^^
rbx/box/package.py:461: in get_solutions
    package = find_problem_package_or_die(root)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

root = PosixPath('.')

    def find_problem_package_or_die(root: pathlib.Path = pathlib.Path()) -> Package:
        package = find_problem_package(root)
        if package is None:
            console.console.print(f'[error]Problem not found in {root.absolute()}[/error]')
>           raise typer.Exit(1)
E           click.exceptions.Exit: 1

rbx/box/package.py:81: Exit

To view more test analytics, go to the Test Analytics Dashboard
_{📋 Got 3 mins? Take this short survey to help us improve Test Analytics.}

rsalesc · 2026-08-31T12:35:35Z

CI note: the failing test job is not from this PR. The 3 failures are all in tests/rbx/box/test_timing_run_all_cli.py and reproduce identically on a clean main checkout with none of these changes; bisected to efaf702 (#832). Tracked in #844.

Lint & Format passes, and the packaging/completion/lazy-CLI suites are green (1581 passed).

rsalesc commented Jun 13, 2026

View reviewed changes

Roberto Sales and others added 3 commits August 31, 2026 13:49

rsalesc force-pushed the worktree-domjudge-packager branch from 087f800 to fb94aa1 Compare August 31, 2026 12:00

rsalesc mentioned this pull request Aug 31, 2026

main is red: estimation_checksum.compute() reads the cwd, breaking 3 timing tests #844

Closed

Navigation Menu

feat(packaging): add DOMjudge problem packager - #598

feat(packaging): add DOMjudge problem packager#598
rsalesc wants to merge 3 commits into
mainfrom
worktree-domjudge-packager

rsalesc commented Jun 12, 2026

Uh oh!

rsalesc Jun 13, 2026

Uh oh!

rsalesc Jun 13, 2026

Uh oh!

rsalesc Jun 13, 2026

Uh oh!

rsalesc Jun 13, 2026

Uh oh!

rsalesc Jun 13, 2026

Uh oh!

rsalesc Jun 13, 2026

Uh oh!

rsalesc commented Aug 31, 2026

Uh oh!

codecov Bot commented Aug 31, 2026

Uh oh!

rsalesc commented Aug 31, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

rbx outcome	placement
ACCEPTED / WRONG_ANSWER / TIME_LIMIT_EXCEEDED / RUNTIME_ERROR	matching standard dir, no annotation
OUTPUT_LIMIT_EXCEEDED	`output_limit/` (DOMjudge extension dir)
ACCEPTED_OR_TLE	`mixed/` + `@EXPECTED_RESULTS@: CORRECT, TIMELIMIT`
TLE_OR_RTE	`mixed/` + `TIMELIMIT, RUN-ERROR`
INCORRECT	`mixed/` + all non-`CORRECT` verdicts
ANY	`mixed/` + all verdicts
MEMORY_LIMIT_EXCEEDED	`mixed/` + `RUN-ERROR, TIMELIMIT`

Conversation

rsalesc commented Jun 12, 2026

Summary

Key decisions

Testing

Uh oh!

rsalesc Jun 13, 2026

Choose a reason for hiding this comment

Uh oh!

rsalesc Jun 13, 2026

Choose a reason for hiding this comment

Uh oh!

rsalesc Jun 13, 2026

Choose a reason for hiding this comment

Uh oh!

rsalesc Jun 13, 2026

Choose a reason for hiding this comment

Uh oh!

rsalesc Jun 13, 2026

Choose a reason for hiding this comment

Uh oh!

rsalesc Jun 13, 2026

Choose a reason for hiding this comment

Uh oh!

rsalesc commented Aug 31, 2026

Uh oh!

codecov Bot commented Aug 31, 2026

❌ 3 Tests Failed:

Uh oh!

rsalesc commented Aug 31, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant