DARDcollect — DETECTOR Archive Data Collector

A GPU-accelerated multi-modal toolkit for downloading, processing, and annotating historical public-domain media from the Internet Archive. Originally developed for the DETECTOR project, it downloads videos, images, audio, and documents organised by language; extracts person detections and pose keypoints; transcribes speech; extracts text from PDFs and plain-text files; and produces 616×616 face crops with rich .json sidecars — bounding boxes, keypoints, quality scores, transcriptions, and full provenance — with FAIR metadata throughout.

Use it two ways:

Complete pipeline for bulk processing of historical media collections.
Modular library — import individual components (detection, transcription, OCR, face crops, quality scoring) into custom workflows.

Key Features

Thirteen decoupled stages — download, video person detection, image person detection, video face crop extraction, image face crop extraction, audio track extraction, frame extraction, face mask generation, quality annotation, quality filtering, video transcription, audio transcription, document text extraction — each resumable and independently re-runnable (12 processing stages run on the fixture; download is auto-skipped there and prepended for full runs).
Pose-based filtering — face visibility, size, frontal orientation, and duplicate suppression all derived from CIGPose 133-keypoint poses; robust to the grain and low resolution of pre-1960 film.
Speech transcription — Whisper-Small transcribes both person-clip audio (video pipeline) and standalone audio files, writing .transcription.json sidecars with language detection.
Document text extraction — extracts text from PDFs (text layer or PaddleOCR fallback for scanned pages) and plain-text files with encoding detection, producing .text.txt + .annotation.json pairs. PP-OCRv5 with per-script model routing supports all 24 EU official languages (Latin/Cyrillic/Greek scripts).
GPU accelerated — YOLOX, CIGPose, Whisper, and PaddleOCR run via ONNX on TensorRT/CUDA 12.1 (Linux/Windows) or MPS (macOS); CPU-only fallback activates automatically. NVIDIA libraries auto-preloaded at import — no manual setup required.
FAIR + EU AI Act — every artifact gets a UUID and full provenance chain; every sidecar carries a shared JSON-LD @context (Dublin Core Terms + PROV-O) so it parses as linked data; every model and rule-based algorithm is documented per Annex IV.

Installation

As a Pipeline

For bulk processing of Archive.org media with the complete 13-stage workflow:

git clone https://github.com/Vicomtech/dardcollect.git && cd dardcollect
uv sync   # Includes TensorRT + CUDA 12.1 on Linux/Windows, MPS on macOS

Note: Python 3.12 required (other versions untested). GPU acceleration works out-of-the-box — NVIDIA libraries are bundled via PyTorch CUDA wheels and auto-preloaded at import. Falls back to CPU automatically on machines without compatible GPUs.

The pipeline processes media through four parallel modality tracks. Video and image face crops converge at quality annotation, then filter, frames, and masks; audio and documents run independently:

Videos  ── download ── clips ──┬─ audio (WAV)
                               ├─ face crops ────┐
                               └─ transcriptions │
                                                 │
Images  ── download ── detections ── face crops ─┴─ quality ── filter ── frames ── masks

Audio   ── download ── transcriptions

Documents ── download ── extracted text

Then follow the step-by-step walkthrough in docs/0-GETTING-STARTED.md.

For existing local datasets (no Archive.org download), set run_pipeline.skip_download: true in your config and run scripts/run_pipeline.py --config <your_config>. To relocate the dataset (e.g. a UNC share or another machine), declare root: <path> at the top of your config and use {root}/subpath in every section — change the single root value to update all paths. See docs/0-GETTING-STARTED.md.

As a Library (Custom Workflows)

To use individual components in your own Python scripts:

uv pip install git+https://github.com/Vicomtech/dardcollect.git

Then import and use components:

# Example: Custom transcription + face detection workflow
from dardcollect import (
    AudioTranscriber,
    DetectorConfig,
    DownloadRequest,
    PersonDetector,
    download_item,
)
from pathlib import Path
import cv2

# Download from archive.org with FAIR metadata
result = download_item(
    DownloadRequest(
        identifier="example_item_id",
        dest_dir=Path("media/"),
        history_file=Path("downloads.csv"),
    )
)

if result["success"]:
    # Locate the file (dest_dir [+ language subdir] / filename from the metadata)
    meta = result["metadata"]
    media_file = next(Path("media/").rglob(meta["filename_downloaded"]))

    # Transcribe audio
    transcriber = AudioTranscriber(model_size="small")
    text = transcriber.transcribe_file(media_file)

    # Detect people in the first video frame
    config = DetectorConfig.from_yaml("configs/config.archive_all.yaml")
    detector = PersonDetector(
        config,
        model_path="dardcollect/models/yolox_tiny_8xb8-300e_humanart-6f3252f9.onnx",
    )
    cap = cv2.VideoCapture(str(media_file))
    ok, frame = cap.read()
    cap.release()
    bboxes, scores = detector.get_detections(frame)

Available components:

PersonDetector, PersonTracker, PoseEstimator — Detection & tracking
AudioTranscriber — Whisper speech-to-text
DocumentExtractor — OCR for scanned PDFs
process_image(), process_video() — Face crop extraction (OFIQ 616×616)
load_models(), score_video() — OFIQ 7-dimensional quality scoring
add_fair_metadata(), generate_uuid() — Provenance tracking
check_face_visibility(), check_frontal_face() — Face validation
extract_frames() — Video to PNG frames
download_item() — Archive.org downloads

For detailed examples and API reference, see docs/5-LIBRARY-API.md.

Usage

uv sync                                                    # install (Python 3.12, uv)
# edit configs/config.archive_all.yaml: media types, search queries, paths
uv run python scripts/run_pipeline.py                      # all stages, Archive.org source
uv run python scripts/run_pipeline.py --config configs/config.mydata.yaml   # your dataset
uv run python pipeline/annotate_face_quality.py            # run one stage on its own

Step-by-step setup and the stage table: docs/0-GETTING-STARTED.md. Configuration keys and defaults: docs/7-CONFIG.md.

Documentation Guide — Which File Should I Read?

Want to…	Read
Get started (install + run the stages)	docs/0-GETTING-STARTED.md
Configure a run (every config key)	docs/7-CONFIG.md
Understand the architecture & FAIR strategy	docs/1-ARCHITECTURE.md
CSV provenance & traceability queries	docs/2-LINEAGE.md
Sidecar JSON annotation formats	docs/3-ANNOTATIONS.md
OFIQ face-crop alignment and stabilization design	docs/DESIGN_landmark_stabilization.md
Metadata contract (CSV ↔ JSON sidecar) — draft plan	docs/DESIGN_metadata_contract.md
GPU setup, dev workflow, the objective gate	docs/4-DEVELOPMENT.md
Use components as a library	docs/5-LIBRARY-API.md
AI-agent harness (rules, skills, gates, work cycle)	docs/6-HARNESS.md
Contribute (style, hooks, PRs)	docs/CONTRIBUTING.md
AI-agent project context (objective, gates, loop; any host)	AGENTS.md

Output Structure

DARD/
├── archive_org_public_domain/            # Downloaded source files
│   ├── videos/eng/, videos/spa/, ...     # Language-organised video downloads (ISO 639-2)
│   ├── videos/und/                       # Videos with no language metadata on Archive.org
│   ├── images/                           # Image downloads (no language subfolder)
│   ├── audio/eng/, audio/spa/, ...       # Language-organised audio downloads (ISO 639-2)
│   ├── audio/und/                        # Audio with no language metadata on Archive.org
│   ├── texts/ger/, texts/fra/, ...       # Language-organised text downloads (ISO 639-2)
│   ├── texts/und/                        # Texts with no language metadata on Archive.org
│   └── downloads.csv                       # Unified metadata (one row per file)
├── extracted_person_clips/               # Person clip videos + JSON sidecars + WAV audio tracks + clips_extraction.csv + transcriptions_extraction.csv
├── extracted_image_detections/           # Per-image detection JSON + image_person_detection.csv
├── video_face_crops/                     # 616×616 OFIQ-aligned crops (video) + video_face_crops_extraction.csv
├── image_face_crops/                     # 616×616 OFIQ-aligned crops (image) + optional *_mask.png + image_face_crops_extraction.csv
├── filtered_video_face_crops/            # Quality-filtered video crops + video_filtered_face_crops.csv (+ .magface.json/.ofiq_attr.json sidecars)
├── filtered_image_face_crops/            # Quality-filtered image crops + optional *_mask.png + image_filtered_face_crops.csv (+ .magface.json/.ofiq_attr.json sidecars)
├── extracted_frames/                     # PNG frames + frame sidecars + optional *_mask.png + frames_extraction.csv
├── audio_transcriptions/                 # Whisper sidecars for audio files + audio_transcriptions_extraction.csv
└── preprocessed_documents/              # Extracted text + annotation JSON + document_text_extraction.csv

Every artifact is linked to its source via UUID: Source manifest → Download → Clip → Crop → Quality scores (the manifest is Archive.org downloads.csv or a custom-registered register_source_files() manifest — the chain is origin-agnostic). Traceability CSVs are lean join indexes (identity + parent UUID + lookup keys); the full per-artifact payload lives in schema-validated JSON sidecars, each carrying a shared JSON-LD @context (Dublin Core Terms + PROV-O) so it parses as linked data. See docs/2-LINEAGE.md for CSV schemas and traceability queries, and docs/3-ANNOTATIONS.md for sidecar JSON formats.

AI Systems (EU AI Act Annex IV)

Each automated component is documented as an AI system per Annex IV, regardless of whether it uses a learned model or a rule-based algorithm. Face quality measures follow ISO/IEC 29794-5 via the OFIQ reference implementation.

Task	System	Type	Implementation	Documentation
Detection	YOLOX-Tiny (HumanArt)	Neural network (ONNX)	`dardcollect/detector.py`	Model card
Tracking	OC-SORT	Algorithm (model-free)	`dardcollect/tracker_ocsort.py`, `dardcollect/tracker_kalman.py`	System card
Pose estimation	CIGPose Wholebody (COCO 133)	Neural network (ONNX)	`dardcollect/poser.py`	Model card
Scene change detection	Luminance histogram + bbox area	Algorithm (rule-based)	`pipeline/extract_person_clips_from_videos.py`	System card
Clip segmentation	Face/duration/frontal rules	Algorithm (rule-based)	`pipeline/extract_person_clips_from_videos.py`	System card
Face crop alignment/stabilization	OFIQ canonical eye anchoring + temporal scale/rotation smoothing	Algorithm (landmark geometry)	`dardcollect/face_geometry.py`, `dardcollect/face_stabilization.py`	System card
Face quality — unified score	MagFace IResNet50 (ISO/IEC 29794-5)	Neural network (ONNX)	`pipeline/filter_face_crops_by_quality.py`, `pipeline/annotate_face_quality.py`	Model card
Face quality — sharpness	Face sharpness random forest (OFIQ `Sharpness`)	Algorithm (random forest)	`pipeline/annotate_face_quality.py`	Model card
Face quality — compression	SSIM CNN (OFIQ `CompressionArtifacts`)	Neural network (ONNX)	`pipeline/annotate_face_quality.py`	Model card
Face quality — expression neutrality	HSEmotion EfficientNet + AdaBoost (OFIQ `ExpressionNeutrality`)	Neural network + algorithm	`pipeline/annotate_face_quality.py`	Model card
Face quality — head coverings / occlusion	BiSeNet face parsing (OFIQ `NoHeadCoverings` / `FaceOcclusionPrevention`)	Neural network (ONNX)	`pipeline/annotate_face_quality.py`	Model card
Face quality — face occlusion segmentation	Face occlusion segmentation CNN	Neural network (ONNX)	`pipeline/annotate_face_quality.py`	Model card
Face quality — head pose	MobileNetV1 3DDFAV2 (OFIQ `HeadPose`)	Neural network (ONNX)	`pipeline/annotate_face_quality.py`	Model card
Audio transcription	Whisper-Small	Neural network (PyTorch)	`pipeline/transcribe_video_clips.py`, `pipeline/transcribe_audio_files.py`	Model card
Document OCR	PaddleOCR PP-OCRv5 (det + cls + per-script rec)	Neural network (ONNX+TRT)	`pipeline/extract_text_from_doc.py`	Model card
Face mask generation	Binary face-region masks — ArcFace quad for crops (one per video-crop frame), OFIQ quad per identity for source frames	Algorithm (rule-based)	`pipeline/generate_face_masks.py`	System card
Audio track extraction	moviepy/ffmpeg WAV demux (16kHz mono PCM)	Algorithm (rule-based)	`pipeline/extract_audio_from_clips.py`	—
Frame extraction	OpenCV video frame decode + sidecar detection reuse	Algorithm (rule-based)	`pipeline/extract_frames_from_videos.py`	—
Colour filter (standalone)	Mean-HSV-saturation classification over sampled keyframes (colour vs B&W)	Algorithm (rule-based)	`pipeline/filter_videos_by_color.py`	docs/2-LINEAGE.md

Contributing

Contributions are welcome. Please read docs/CONTRIBUTING.md for:

Development setup and pre-commit hooks (Ruff + ty + import-linter layer DAG + hygiene hooks incl. a 10 MB large-file guard)
CI (.github/workflows/ci.yml) runs every CPU gate on each push/PR; the GPU objective gate stays manual per AGENTS.md
Code style: Ruff (linting & formatting) + ty (type checking)
PR guidelines — including the requirement to document any new pipeline component as an AI system per EU AI Act Annex IV

License

The source code is licensed under the Apache License 2.0.

The bundled model weights in dardcollect/models/ carry separate licenses and are not covered by the Apache 2.0 license:

Model	File	License
YOLOX-Tiny (HumanArt)	`yolox_tiny_...onnx`	Architecture: Apache 2.0 — weights trained on HumanArt data ⚠ verify before commercial use
CIGPose Wholebody	`cigpose-m_...onnx`	Architecture: Apache 2.0 — trained on COCO WholeBody ⚠ verify before commercial use
Whisper Small	`openai_whisper_small.pt`	MIT
MagFace IResNet50	`magface_iresnet50_norm.onnx`	Apache 2.0 (MagFace); ONNX packaging MIT (OFIQ)
HSEmotion EfficientNet-B0/B2	`enet_b0_...onnx`, `enet_b2_...onnx`	MIT (HSEmotion)
AdaBoost neutrality classifier	`hse_1_2_C_adaboost.yml.gz`	MIT (OFIQ)
BiSeNet, occlusion, head pose, sharpness, SSIM	various `.onnx` / `.xml.gz`	MIT (OFIQ)
PaddleOCR PP-OCRv5 (det + cls + Latin/Cyrillic/Greek rec)	`ch_PP-OCRv5_.onnx`, `_PP-OCRv5_rec_*.onnx`	Apache 2.0 (PaddleOCR)

See NOTICE for full third-party attributions and dependency licenses (including ⚠ PyMuPDF AGPL-3.0).

Name		Name	Last commit message	Last commit date
.agents/skills		.agents/skills
.claude		.claude
.github		.github
.kilo		.kilo
.vscode		.vscode
configs		configs
dardcollect		dardcollect
docs		docs
pipeline		pipeline
schemas		schemas
scripts		scripts
tests		tests
viewer		viewer
.gitattributes		.gitattributes
.gitignore		.gitignore
.pre-commit-config.yaml		.pre-commit-config.yaml
AGENTS.md		AGENTS.md
CLAUDE.md		CLAUDE.md
LICENSE		LICENSE
NOTICE		NOTICE
README.md		README.md
js		js
kilo.json		kilo.json
pyproject.toml		pyproject.toml
uv.lock		uv.lock

Navigation Menu

Repository navigation

About

Repository files navigation

DARDcollect — DETECTOR Archive Data Collector

Key Features

Installation

As a Pipeline

As a Library (Custom Workflows)

Usage

Documentation Guide — Which File Should I Read?

Output Structure

AI Systems (EU AI Act Annex IV)

Contributing

License

About

Releases

Packages

Contributors

Languages

Latest commit

History

Folders and files

Repository files navigation

DARDcollect — DETECTOR Archive Data Collector

Key Features

Installation

As a Pipeline

As a Library (Custom Workflows)

Usage

Documentation Guide — Which File Should I Read?

Output Structure

AI Systems (EU AI Act Annex IV)

Contributing

License

About

Resources

Contributing

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages