di-e2e-mot

English | 简体中文

di-mot is a deployment toolkit for end-to-end multi-object tracking. It takes research implementations of DETR-family trackers — MOTR, CO-MOT, MOTIP, MeMOTR, SambaMOTR, FDTA and DualTemporalMOT — unifies them under one config system and one model registry, and carries them from research weights to a static-shape ONNX graph, a TensorRT engine and a C++ inference runtime. It is built for engineers who need a published tracker running outside its original training repo, on fixed-shape graphs, without Python at inference time.

Current version: 0.7.0 — export, evaluation and C++ deployment work end to end; training and quantization are not implemented yet (see Known limitations).

Features

• One config system, seven registered variants. A single CLI (dimot) covers every family, and upstream configs are accepted as-is (CO-MOT .args files, MOTIP YAML with SUPER_CONFIG_PATH).
• Static-shape ONNX export with a deployment manifest. Each export produces a fixed-shape graph plus a sidecar manifest that declares state initial values, preprocessing and output consumption rules — everything a runtime needs, with no family-specific code on the consumer side.
• A C++17 runtime with ONNX Runtime and TensorRT backends (cpp/). The manifest is the only Python↔C++ interface, so inference needs no Python. A demo turns a folder of frames into MOTChallenge-format tracks.
• Built-in evaluation. dimot eval runs a sequence end to end, writes MOTChallenge-format output and computes HOTA/DetA/AssA through a vendored TrackEval — no external evaluation tooling.
• TensorRT engine build and latency measurement from the same CLI (dimot deploy), with provenance recorded per artifact.
• Traceable porting. Vendored files keep their upstream copyright headers verbatim, and each family documents every departure from upstream in its PROVENANCE.md.

Installation

Python ≥ 3.10. Releases are GitHub-only (no PyPI).

git clone https://github.com/D-Robotics/di-e2e-mot.git
cd di-e2e-mot
pip install -e .              # core needs only torch + pyyaml

Optional extras select a family or a workflow:

Extra	What it installs
motr · motip · memotr · sambamotr	dependencies of one family
onnx	the export stack (onnx, onnxscript, onnxruntime-gpu)
eval	evaluation dependencies
dev	pytest and everything needed to run the test suite

To run the whole test suite, install what CI installs:

pip install -e ".[dev,motip]" onnx onnxscript onnxruntime
python -m pytest tests -q

On a GPU host you can use the onnx extra instead of the explicit CPU trio; the onnxruntime-gpu and onnxruntime wheels conflict within one environment, which is why the CPU trio is spelled out above.

Name map, since the four names differ on purpose: repository di-e2e-mot, distribution di-mot, import di_mot, command dimot.

The C++ runtime has no Python dependency; build it with CMake as shown in the Quickstart.

Quickstart

1. Export a model

From a fresh clone, the shortest path is examples/: a checkpoint maker plus five YAML configs, each really exported before it landed. No released weights are needed to try it:

dimot models                        # list the registered (family, variant) pairs
python examples/make_random_checkpoint.py --family motip --variant fdta \
    -c examples/motip-fdta.yaml --out fdta-random.pth --seed 0
dimot export -c examples/motip-fdta.yaml --family motip --variant fdta \
    --ckpt fdta-random.pth --output fdta.onnx
# -> fdta.onnx + fdta.onnx.json (sidecar manifest)

With a released checkpoint from the corresponding upstream repository, the same command exports the real model:

dimot export -c <upstream-config.yaml> --family sambamotr \
    --ckpt sambamotr.pth --output samba.onnx

--family selects the export recipe and is required; --variant defaults to the variant named like the family; --set key=value overrides config scalars. The first memotr/sambamotr run downloads the ImageNet ResNet50 backbone (~98 MB); the other families need no download. Export details and known pitfalls: docs/families.md.

2. Evaluate on DanceTrack

dimot eval -c <config.yaml> --family motip --variant motip \
    --ckpt motip_dancetrack.pth \
    --data-root <DanceTrack> --seqmap <seqmap.txt> --split val \
    --sequence-hw 1080 1920 --out out/

dimot eval loads the checkpoint, runs the tracker over <split>/<seq>/img1/*.jpg for every sequence in the seqmap, writes MOTChallenge-format tracks and computes HOTA/DetA/AssA in-process, producing eval_report.json. --sequence-hw is the geometry fallback for sequences without a seqinfo.ini; --limit N caps the sequence count. Supported today: the motip family (variants motip, fdta) and sambamotr. Dataset layout declarations live in di_mot/data/spec.py; official pages: DanceTrack · MOT17.

3. Run the C++ runtime

Build and self-test on Linux (ONNX Runtime C++ 1.20.1):

curl -sSL -o /tmp/ort.tgz \
  https://github.com/microsoft/onnxruntime/releases/download/v1.20.1/onnxruntime-linux-x64-1.20.1.tgz
tar -xzf /tmp/ort.tgz -C /tmp
cmake -S cpp -B cpp/build -DDIMOT_ORT_ROOT=/tmp/onnxruntime-linux-x64-1.20.1
cmake --build cpp/build -j
ctest --test-dir cpp/build --output-on-failure

Then track a folder of frames (JPEG/PNG/BMP, decoded and preprocessed exactly as the manifest declares):

cpp/build/dimot-rt-demo --manifest samba.onnx.json --onnx samba.onnx \
    --backend ort --images <seq>/img1 --out mot.txt

--backend trt,fp32 runs a TensorRT engine built from the trt-lowered ONNX that dimot deploy emits; --engine-cache <path> persists the engine across runs and --limit N caps the frame count. Without -DDIMOT_ORT_ROOT the ORT backend, the demo and the ORT tests are skipped loudly at configure time. The TensorRT and Horizon backends, plus fixture generation for the golden tests: docs/deployment.md.

Supported models

Model	family · variant	Graph input	Paper	Status
CO-MOT	comot · motr_co	images	arXiv:2305.12724	✅ export validated
MOTIP	motip · motip	detections	arXiv:2403.16848	✅ metric validated
FDTA	motip · fdta	detections	arXiv:2512.02392	✅ metric validated
SambaMOTR	sambamotr · motr_sambamotr	images	arXiv:2410.01806	✅ metric validated
MeMOTR	memotr · motr_memotr	images	arXiv:2307.15700	🧪 experimental
DualTemporalMOT	dualtemporalmotr · motr_dualtemporalmotr	—	OpenReview	🧪 experimental
MOTR	motr · motr	—	arXiv:2105.03247	source reference

"Graph input" is what the exported graph consumes: image families take frames directly; MOTIP and FDTA take detections from the upstream detector, fed as a sidecar at deployment time. Status legend: ✅ metric validated — reproduces the upstream val metric (see Results); ✅ export validated — exports and deploys end to end, metric comparison not yet wired; 🧪 experimental — MeMOTR and DualTemporalMOT are unverified ports: they run and pass the repository's own consistency checks, but their outputs have never been compared against the upstream implementations. MOTR is vendored as the shared source base and has nothing to export; DualTemporalMOT currently ships a config/contract layer only. Upstream repositories are listed under Citation.

Results

All numbers are HOTA on DanceTrack val.

Accuracy vs upstream — full val split, released checkpoints; the reference is the upstream code's own val run on the same host and software stack, not the paper's test-set number:

Model	HOTA (di-mot)	HOTA (upstream rerun)	Δ
MOTIP	64.164	64.099	+0.065
FDTA	64.818	64.569	+0.249
SambaMOTR	61.902	62.860	−0.958

C++ runtime vs Python ONNX reference — SambaMOTR, released weights, both sides consuming the same exported graph and the same frames (the C++ side is dimot-rt-demo, the reference is scripts/onnx_sequence_runner.py). Run on a subset of val sequences with frame caps, so the absolute HOTA sits below the full-val table above; the meaningful quantity is the delta:

Backend	HOTA (C++)	HOTA (Python)	Δ
ONNX Runtime, CPU EP (5 sequences)	18.95	18.69	+0.26
TensorRT fp32 (2 sequences)	21.10	20.17	+0.93
TensorRT fp16	0.07	20.17	−20.09

TensorRT fp32 measured ≈0.4 s/frame on an RTX 4090. TensorRT fp16 collapses the tracking recurrence on this graph and is not recommended (see Known limitations). For MeMOTR and CO-MOT (random weights) the C++ and Python sides agree exactly at the row level — equal row counts, median IoU 1.000. Methodology, per-component numbers and reproduction commands: docs/families.md · docs/reports/.

Live-id deployment (v0.7) — MOTIP/FDTA, released weights, 400 detection sidecar frames of dancetrack0004, host-side label registry enabled on both sides (v0.6 measured 0 rows on both sides on this exact setup — the empty pass the id wall produced):

Family	rows (C++)	rows (Python)	ΔHOTA	distinct ids
MOTIP	1604	1604	+0.000	17
FDTA	1577	1577	+0.000	14

The deployment rows also match the PyTorch tracker frame-by-frame (40 frames per family, zero mismatches). Full numbers and reproduction commands: docs/reports/live-ids-acceptance-2026-10-14.md.

Documentation

Document	Contents
docs/architecture.md	why this layer exists, the graph boundary, core layout
docs/families.md	per-family export details, weights availability, pitfalls
docs/deployment.md	TensorRT backend, C++ runtime guide, latency numbers, edge plan
docs/known-issues.md	the complete defect and boundary ledger
docs/reports/	dated benchmark and acceptance reports
di_mot/models/<family>/PROVENANCE.md	per-file port records: checksums and every departure from upstream
CHANGELOG.md	release history

The docs are written in Chinese, each with an English TL;DR at the top; there is no docs site. CI (leak scan, full test suite, wheel licence checks, the C++ job) is configured in-repo — the badge may show no status until GitHub Actions is enabled for your fork or organization.

Known limitations

• No training. train, infer, quantize, parity and capacity are stubs; this toolkit ports, exports, evaluates and deploys.
• MOTIP/FDTA deployments emit live track ids through a host-side label registry (v0.7). The exported graph emits the per-frame newborn mask; the runtime contract's id_registry section drives a host-side label→id table in both the C++ runtime and the Python reference runner (detection-order assignment, recycled labels overwrite, misses dropped — mirroring the PyTorch tracker bit for bit). Contract 2 manifests only; v1 artifacts keep replaying. The detector itself is still outside the graph (detection sidecars feed these families).
• TensorRT fp16 is not recommended: on real-weight SambaMOTR graphs the half-precision engine derails the tracking recurrence (see Results). The C++ TensorRT backend also cannot build engines for MOTIP/FDTA graphs (a parser plugin is missing); SambaMOTR, MeMOTR and CO-MOT graphs build fine.
• The Horizon backends are experimental. hb_dnn compiles but has never run on a board; the horizon-s100 deploy backend is a registered placeholder.
• MeMOTR and DualTemporalMOT are unverified ports (see Supported models).
• Validation scope. Metric validation covers three models, on the DanceTrack val split, on a single host and software stack.

Bug reports are welcome — please check docs/known-issues.md first; part of this repository is knowingly not built yet, and that is not a bug.

License

Apache-2.0. See LICENSE. NOTICE lists third-party holders; THIRD_PARTY.md maps each vendored tree to its upstream project and license; upstream copyright headers are preserved verbatim. scripts/scan_leaks.py is a publish gate (internal paths, hostnames, private addresses; non-zero exit on a hit) and runs as part of the test suite and CI.

Citation

If this toolkit is useful to you, cite it as:

@software{dirobotics2026dimot,
  author  = {{D-Robotics}},
  title   = {di-mot},
  year    = {2026},
  version = {0.7.0},
  url     = {https://github.com/D-Robotics/di-e2e-mot},
  note    = {Unified end-to-end multi-object tracking: one registry and state protocol for DETR-family trackers, static-graph ONNX export and deployment}
}

The rule: whichever family you use, cite that family's upstream paper alongside di-mot. Any reported HOTA number additionally requires the HOTA paper (below). A CITATION.cff sits at the repository root.

di-mot family / package	Upstream paper	Venue	Upstream repository
comot (variant motr_co)	yan2025comot	ICLR 2025 Poster	https://github.com/BingfengYan/CO-MOT
motr (package; MOTR source via CO-MOT) and legacy/	zeng2022motr	ECCV 2022	https://github.com/megvii-research/MOTR
motip	gao2025motip	CVPR 2025	https://github.com/MCG-NJU/MOTIP
memotr (variant motr_memotr)	gao2023memotr	ICCV 2023	https://github.com/MCG-NJU/MeMOTR
sambamotr (variant motr_sambamotr)	segu2025samba	ICLR 2025 Spotlight	https://github.com/mattiasegu/sambamotr
fdta (a motip variant)	shao2026fdta	CVPR 2026	https://github.com/Spongebobbbbbbbb/FDTA
dualtemporalmotr (contract layer)	kim2025dualpath	NeurIPS 2025	https://github.com/altkddhfcjs/DualTemporalMOT
eval (vendored TrackEval)	luiten2021hota (the paper)	IJCV 2021	https://github.com/JonathonLuiten/TrackEval
shared components, as used	zhu2021deformabledetr · carion2020detr · zhang2023dino · ge2021yolox · dai2017dcn	—	see THIRD_PARTY.md

The HOTA paper (the metric the vendored TrackEval computes):

@article{luiten2021hota,
  title   = {{HOTA}: A Higher Order Metric for Evaluating Multi-object Tracking},
  author  = {Luiten, Jonathon and Osep, Aljosa and Dendorfer, Patrick and Torr, Philip and Geiger, Andreas and Leal-Taixe, Laura and Leibe, Bastian},
  journal = {International Journal of Computer Vision},
  volume  = {129},
  pages   = {548--578},
  year    = {2021},
  doi     = {10.1007/s11263-020-01375-2},
  note    = {Published online 8 October 2020; official evaluation code: https://github.com/JonathonLuiten/TrackEval}
}

All 12 upstream BibTeX entries (the papers behind the families and components)

@inproceedings{zeng2022motr,
  title         = {{MOTR}: End-to-End Multiple-Object Tracking with Transformer},
  author        = {Zeng, Fangao and Dong, Bin and Zhang, Yuang and Wang, Tiancai and Zhang, Xiangyu and Wei, Yichen},
  booktitle     = {European Conference on Computer Vision (ECCV)},
  year          = {2022},
  eprint        = {2105.03247},
  archivePrefix = {arXiv},
  url           = {https://github.com/megvii-research/MOTR}
}

@inproceedings{yan2025comot,
  title         = {{CO-MOT}: Boosting End-to-end Transformer-based Multi-Object Tracking via Coopetition Label Assignment and Shadow Sets},
  author        = {Yan, Feng and Luo, Weixin and Zhong, Yujie and Gan, Yiyang and Ma, Lin},
  booktitle     = {International Conference on Learning Representations (ICLR)},
  year          = {2025},
  eprint        = {2305.12724},
  archivePrefix = {arXiv},
  url           = {https://openreview.net/forum?id=0ov0dMQ3mN}
}

@inproceedings{gao2025motip,
  title         = {Multiple Object Tracking as {ID} Prediction},
  author        = {Gao, Ruopeng and Qi, Ji and Wang, Limin},
  booktitle     = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  pages         = {27883--27893},
  year          = {2025},
  eprint        = {2403.16848},
  archivePrefix = {arXiv},
  url           = {https://github.com/MCG-NJU/MOTIP}
}

@inproceedings{gao2023memotr,
  title         = {{MeMOTR}: Long-Term Memory-Augmented Transformer for Multi-Object Tracking},
  author        = {Gao, Ruopeng and Wang, Limin},
  booktitle     = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
  pages         = {9901--9910},
  year          = {2023},
  eprint        = {2307.15700},
  archivePrefix = {arXiv},
  url           = {https://github.com/MCG-NJU/MeMOTR}
}

@inproceedings{segu2025samba,
  title         = {Samba: Synchronized Set-of-Sequences Modeling for Multiple Object Tracking},
  author        = {Segu, Mattia and Piccinelli, Luigi and Li, Siyuan and Yang, Yung-Hsu and Van Gool, Luc and Schiele, Bernt},
  booktitle     = {International Conference on Learning Representations (ICLR)},
  year          = {2025},
  note          = {Spotlight},
  eprint        = {2410.01806},
  archivePrefix = {arXiv},
  url           = {https://openreview.net/forum?id=OeBY9XqiTz}
}

@inproceedings{shao2026fdta,
  title         = {From Detection to Association: Learning Discriminative Object Embeddings for Multi-Object Tracking},
  author        = {Shao, Yuqing and Yang, Yuchen and Yu, Rui and Li, Weilong and Guo, Xu and Yan, Huaicheng and Wang, Wei and Sun, Xiao},
  booktitle     = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  year          = {2026},
  eprint        = {2512.02392},
  archivePrefix = {arXiv},
  url           = {https://github.com/Spongebobbbbbbbb/FDTA}
}

@inproceedings{kim2025dualpath,
  title     = {Dual-Path Temporal Decoder for End-to-End Multi-Object Tracking},
  author    = {Kim, Hyunseop and Jeong, Juheon and Kim, Hanul and Koh, Yeong Jun},
  booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
  year      = {2025},
  url       = {https://openreview.net/forum?id=T64Fa2hCZn}
}

@inproceedings{zhu2021deformabledetr,
  title         = {Deformable {DETR}: Deformable Transformers for End-to-End Object Detection},
  author        = {Zhu, Xizhou and Su, Weijie and Lu, Lewei and Li, Bin and Wang, Xiaogang and Dai, Jifeng},
  booktitle     = {International Conference on Learning Representations (ICLR)},
  year          = {2021},
  note          = {Oral},
  eprint        = {2010.04159},
  archivePrefix = {arXiv}
}

@article{carion2020detr,
  title   = {End-to-End Object Detection with Transformers},
  author  = {Carion, Nicolas and Massa, Francisco and Synnaeve, Gabriel and Usunier, Nicolas and Kirillov, Alexander and Zagoruyko, Sergey},
  journal = {arXiv preprint arXiv:2005.12872},
  year    = {2020},
  eprint  = {2005.12872},
  archivePrefix = {arXiv}
}

@inproceedings{zhang2023dino,
  title         = {{DINO}: {DETR} with Improved DeNoising Anchor Boxes for End-to-End Object Detection},
  author        = {Zhang, Hao and Li, Feng and Liu, Shilong and Zhang, Lei and Su, Hang and Zhu, Jun and Ni, Lionel M. and Shum, Heung-Yeung},
  booktitle     = {International Conference on Learning Representations (ICLR)},
  year          = {2023},
  eprint        = {2203.03605},
  archivePrefix = {arXiv}
}

@article{ge2021yolox,
  title   = {{YOLOX}: Exceeding {YOLO} Series in 2021},
  author  = {Ge, Zheng and Liu, Songtao and Wang, Feng and Li, Zeming and Sun, Jian},
  journal = {arXiv preprint arXiv:2107.08430},
  year    = {2021},
  eprint  = {2107.08430},
  archivePrefix = {arXiv}
}

@inproceedings{dai2017dcn,
  title         = {Deformable Convolutional Networks},
  author        = {Dai, Jifeng and Qi, Haozhi and Xiong, Yuwen and Li, Yi and Zhang, Guodong and Hu, Han and Wei, Yichen},
  booktitle     = {Proceedings of the IEEE International Conference on Computer Vision (ICCV)},
  year          = {2017},
  eprint        = {1703.06211},
  archivePrefix = {arXiv}
}