BPMNscript evaluation

Measures whether BPMN 2.0 models survive a round trip through BPMNscript: import the XML, print it as a script, compile the script back to XML, and compare the two models. This repository is used for all evaluation processes in the Bachelor Thesis BPMNscript is built for, with all results stored in results/<stage>/report.md.

Running it

Requires Node.js 22.18 or newer, Git, and Docker.

npm ci
npm run tool            # clone and build BPMNscript at the pinned commit
npm run stage -- miwg   # run one stage

There are five stages:

Stage	Input	Engine
generated	programs from the BPMNscript fuzz generator, seeds 1 to 2000, in three statement orderings (--seeds 1-200 for a quick run)	no
purposive	the BPMNscript examples and golden files, and every BPMN file of the Operaton and Camunda 7 repositories, each source reported as its own set	yes
miwg	the MIWG reference models	yes
corpus	BPMN files mined from GitHub (see below)	yes
notation	the BPMNscript examples, counting non-whitespace characters of each program and of its XML without the diagram	no

Each stage writes to results/<stage>/, or to --out <dir>.

The corpus needs its files fetched first. This can take multiple hours due to github rate limits and results are stored in .cache/corpus/:

npm run acquire           # resumes where it stopped; GITHUB_TOKEN is used if set
npm run stage -- corpus

Tests:

npm test
npm run test:e2e
npm run typecheck

Measuring a model

The stage sends the model through the BPMNscript CLI and compares the original with the rebuild using the comparison helper that ships with BPMNscript (tests/helpers/model-equivalence.ts). The helper reports facts, and src/classify.ts turns them into exactly one category per row:

• EQUIVALENT: the models match, at most after bookkeeping such as minted ids or element order.
• EQUIVALENT_BY_EXCEPTION: they match after one or more of the tolerances below, and the row names which.
• DIVERGENT: the models still differ after every tolerance.
• OUT_OF_SCOPE: BPMNscript refused the model and named the construct it can't express.
• TOOL_ERROR: the CLI crashed, timed out, or wrote something it couldn't read back.
• INVALID_INPUT: the input isn't part of the population, for example an original Operaton refuses to deploy.

The stages marked "Engine" deploy both the original and the rebuild to Operaton, then delete them again. An original the engine refuses is INVALID_INPUT, and whether the rebuild deploys is recorded beside the category. Any engine error other than a refusal of the document stops the stage.

Tolerances

	Tolerance	The rebuild may differ by
(i)	synthesized ids	ids the compiler mints (bookkeeping)
(ii)	gateway names	a gateway's label
(iii)	pass-through gateways	a gateway with one way in and one way out
(iv)	element order	the order elements are listed in (bookkeeping)
(v)	expression parse trees	an expression rewritten to an equivalent form
(vi)	default history time to live	P30D where the original sets none
(vii)	implicit end	an unnamed end event after a step with no outgoing flow
(viii)	uncontrolled flow	an explicit gateway where flows entered or left a step directly
(ix)	unconditioned last route	the last route of an exclusive split marked as the default
(x)	nested forks	one flat parallel fork instead of two nested ones

Bookkeeping never makes a row an exception. Under (viii), a run where no condition holds fails with an error naming the inserted gateway instead of the step, so the reports count those rows on their own line. The stage runner refuses to start when this list and the helper's list disagree.

Mined corpus

The sampling frame is the link list of Türker et al. on Zenodo, which names each file by repository, branch, path and SHA-1, but doesn't ship the files. npm run acquire fetches each file from its branch's current head and records whether it is still intact, changed, missing or unavailable. A repository or file only counts as gone when github.com confirms it, and any other failure stops the acquisition. The result is a manifest in corpus/tuerker/. Only the manifest is committed, since each file keeps the licence of the repository it came from. With the manifest in place, npm run acquire fetches exactly those files again from the recorded commits.

The stage keeps intact files that use Camunda or Operaton attributes and that Operaton deploys as written (src/screen.ts). The report counts them at three levels: every frame copy, every distinct file, and every group of files the helper calls structurally the same.

Analyses

Three analyses look closer at results the stages recorded. Each writes results/analyses/<name>/ and leaves the stages and their results untouched.

npm run analysis -- bpmnspector   # or conditions, printed-source
npm run analysis -- conditions --out <dir>

Analysis	Measures
bpmnspector	which golden files and MIWG reference models BPMNspector 1.1.0 accepts, and the rules the others violate
conditions	the cause of each differing condition and route order in the divergent corpus models, and its warnings
printed-source	the goto statements and unstructured-region markers in the source printed for each model the importer accepts

report.md holds the tables, and rows.jsonl one row per model with counts and causes only, since each model keeps the licence of its repository. bpmnspector needs Java and downloads its jars into .cache/bpmnspector-1.1.0/, checking their checksums.

Pins

• tool.json: the BPMNscript commit and tag.
• engine/Dockerfile: the Operaton image by digest, plus the three mail libraries it needs, with checksums.
• src/fetch.ts: the commits of the MIWG suite and the Operaton and Camunda 7 repositories.
• src/frame.ts: the corpus frame by DOI and checksum.

The engine runs with enforceHistoryTimeToLive and the job executor switched off, so a model without a history time to live still deploys and no timer fires during a check.

Results

Each stage writes three files:

• results.jsonl: one row per model, sorted by key. A DIVERGENT row lists what differs in differenceKinds, and in unwarnedDifferences how many of those differences are on an element without a warning (see unwarnedDifferences in src/roundtrip.ts).
• run.json: the pins, the engine settings, and the commit of this repository.
• report.md: the tables.

Two runs on the same pins write the same results.jsonl, byte for byte. The only exception is a timeout on a heavily loaded machine, which turns a slow row into TOOL_ERROR.

License

Apache 2.0, see LICENSE.