docval

Structural validation of scientific documents for automated assessment.

Part of the CodeOcean Autograder project, Assessing Office Documents. Master's research project, TU Ilmenau.

What the project does

Students are expected to build documents using the structural mechanisms their editor provides: heading styles, automatic numbering, a generated table of contents, real lists and real captions. Many instead produce text that only looks like those mechanisms, setting a line in bold in place of a heading, or typing 1.2 in front of a section title.

On the page the two are almost identical. In the file they are completely different, and only the genuine version survives an edit, regenerates its own numbering, converts cleanly to another format, or gives a screen reader an outline to navigate. Telling the two apart by hand is repetitive work that instructors currently do document by document.

docval reads a submitted document, decides for each structural feature whether it was really built or only imitated, and reports every place where structure is faked, missing or misplaced. It supports three formats:

Format	Extension	Read with
Microsoft Word	.docx	python-docx and lxml
OpenDocument Text	.odt	zipfile and lxml
LaTeX	.tex	TexSoup

The tool is meant to run inside a CodeOcean container later on. That is why it is a command-line program with only two dependencies: it has to install and run somewhere with no graphical interface and no manual setup.

The eleven checks

Each check has a stable identifier. That identifier is what appears in the JSON output and in the property files, so it is treated as a published interface and does not change.

Identifier	What it requires
heading-hierarchy	Real heading styles, correctly nested, no level skipped
heading-numbering	Numbers come from automatic numbering, not typed digits
toc-present	A real table-of-contents field or element exists
toc-linked	Its entries actually point at the headings, so it is not out of date
required-sections	The configured sections exist, as real headings
section-order	Those sections appear in the configured order
list-formatting	Lists use real list formatting, not typed bullets or numbers
table-caption	Every table has a real caption directly above it
figure-caption	Every figure has a real caption directly below it
equation-format	Equations use the equation mechanism, not typed text or a pasted picture
inline-image	No uncaptioned image is pasted inside a line of running text

What the tool does not judge: content, argument, citation quality, writing, or visual layout such as margins and fonts. It judges structure.

Installing

pip install -r requirements.txt

Two runtime dependencies, python-docx (which brings lxml) and TexSoup. pytest is listed as well but is only needed to run the tests.

Using it

python docval.py path/to/document.docx
python docval.py path/to/document.odt
python docval.py path/to/document.tex

The format is chosen by the file extension. Add --json for the machine-readable form:

python docval.py path/to/document.docx --json

Point it at a course's own requirements with either or both property files:

python docval.py path/to/document.docx --structure structure.json --styles styles.json

The command sets an exit code, so a script can tell what happened without having to read the output at all:

Code	Meaning
0	The document passed every enabled check
1	Issues were found
2	The file could not be checked, for example an unsupported format

Configuring it for a course

Everything specific to one assignment lives in two JSON property files, so an instructor can change what is required without touching code. Commented templates are in examples/.

Structure (--structure) declares WHAT the document must contain: the required sections, with per-language synonyms, in the order they are expected. A section that appears before one listed earlier is an ordering issue. An empty sections list switches section checking off entirely, while the formatting checks stay fully active.

{
  "sections": [
    {"name": "Introduction", "accept": ["introduction", "einleitung"]},
    {"name": "Methodology"}
  ]
}

accept is optional. A bare entry is accepted under its own name, and the list exists to add other spellings, most usefully translations.

Styles (--styles) declares HOW the document must be formatted: the heading hierarchy rules, where captions belong, the caption vocabulary per language, custom list labels, and which checks are active.

{
  "headings":  {"first_level": 1, "max_deeper_step": 1},
  "captions":  {"table_position": "above", "figure_position": "below",
                "labels": {"table": ["table", "tabelle"],
                           "figure": ["figure", "abbildung"]}},
  "lists":     {"labels": ["RQ"]},
  "checks":    {"toc-present": false}
}

Any check can be switched off by its identifier under checks, and any check not listed there stays on. Both files are optional, since the tool falls back on built-in defaults, and a file only needs to contain the entries it actually changes.

If a file contains a key the tool does not recognise, it stops with an error instead of ignoring it. A typo such as cheks would otherwise look like it had switched a check off while doing nothing at all.

Web front end

user_study/ holds a Streamlit application covering the Word path, for people who have neither Python nor a command line. A participant downloads a template document, fixes it, uploads the result, sees the same report the command line produces, and answers a short UEQ-S questionnaire. Responses are logged to a Google Sheet alongside the checks that fired.

It calls the same extractor, rule engine and reporter as the command line and adds a presentation layer, nothing else.

Running it needs two things beyond the code: a Google service account with the Sheets API enabled, with the target sheet shared to that account's address as an editor, and those credentials plus the sheet URL supplied as Streamlit secrets, locally in user_study/.streamlit/secrets.toml and in the app's own Secrets panel once deployed. The secrets file holds a private key and is excluded by .gitignore.

cd user_study
pip install -r requirements.txt
streamlit run app.py

Tests

python -m pytest

115 tests, a few seconds to run. They work from any directory, since they locate the project from their own position rather than the one the command was run in.

File	Tests	Covers
test_ladders.py	39	Every sample version in all three formats, plus a strict word == latex == odt equality assertion
test_rules.py	38	Each check on its own, against representations built by hand
test_latex_constructs.py	21	Chapters, parts, file inclusion, run-in labels, code listings, nested commands
test_odt_constructs.py	9	Text extraction and numbering resolution, on XML fragments
test_config.py	8	Property-file loading, merging, comment keys, typo detection

The documents behind the ladder tests are in Sample Documents/. They are not nine separate samples but one document in nine stages, where each stage fixes exactly one fault that the stage before it had. That makes the expected result known in advance: every check has a stage where it must start passing, and it must still be failing in the stage before. Guessing that pattern by accident is close to impossible, which is what makes it useful for testing.

The same ladder exists in all three formats, and the tests require the three extractors to report exactly the same thing on it.

Architecture

The whole design rests on one idea: turn every document into the same internal form first, then do all the checking on that. Reading a file is therefore the only part that knows about Word, OpenDocument or LaTeX. Everything after it is written once and works for all three.

  a file              TRANSLATE              JUDGE            PRESENT
 ┌────────┐   ┌──────────────────────┐   ┌────────────┐   ┌─────────────┐
 │ .docx  │──▶│ word_extractor.py    │──▶│            │   │             │
 │ .odt   │──▶│ odt_extractor.py     │──▶│  rules.py  │──▶│ reporter.py │
 │ .tex   │──▶│ latex_extractor.py   │──▶│ 11 checks  │   │ text / JSON │
 └────────┘   └──────────────────────┘   └─────▲──────┘   └─────────────┘
                   all produce a               │
                   model.DocModel      Structure + Styles

• extractors.py: where formats plug in. It defines what an extractor has to provide and keeps the list of which one handles which file extension, so nothing else in the code needs to know a format by name.
• word_extractor.py / odt_extractor.py / latex_extractor.py: one per format, each producing the same representation and nothing else. They read a document; they never judge it.
• model.py: the shared internal form every extractor produces. A list of blocks in reading order, each labelled with what it is (heading, table, caption and so on) and whether it is genuine, plus a few facts about the document as a whole. This is the one thing both halves agree on.
• patterns.py: recognises structure that was typed rather than built ("Table 1:" captions, "RQ1:" list markers, keyboard formulas). Genuine structure differs in every format, but imitation is only text and looks the same everywhere, so this is shared by all three extractors.
• rules.py: the eleven checks and the engine that runs them, written once against the representation.
• config.py: loads the two property files over the built-in defaults.
• reporter.py: the readable and JSON output forms.
• docval.py: the command that ties it together.

The arrangement gives three independent axes of change:

To add	What must be touched
a new format	one extractor module, one registration call, one import line
a new check	one function in rules.py and one registry entry
a new course policy	a JSON property file, and no code at all

ARCHITECTURE.md describes the design in full, including each format's own findings and the verification method, and is written to be read without the code at hand.