URBANITE (UnReal puBlic sAfety syNthetIc daTasEt)

Welcome to the official helper code repository for URBANITE, a large-scale, multi-camera synthetic CCTV dataset designed to advance public safety and surveillance research.

Obtaining real-world anomaly data in public spaces is exceptionally difficult due to the rarity of security-critical events and strict privacy laws. URBANITE bridges this gap by providing a privacy-preserving, richly annotated synthetic benchmark for studying complex human behaviours in CCTV settings. The dataset is generated with Unreal Engine 5.1, which provides fine-grained control over actor behaviours, lighting and weather conditions, together with a fully automated annotation pipeline.

This repository provides the Python utilities and parsing scripts you need to read, visualize and analyse URBANITE's extensive annotations and integrate them into your machine learning pipelines.

License and Usage Restrictions

This dataset is distributed under the Creative Commons Attribution-NonCommercial 4.0 International (CC BY-NC 4.0) license. Please see the LICENSE file for the full legal text.

Justification for Non-Commercial (NC) License: In alignment with open science principles, Vicomtech intends for this data to be as accessible as possible. However, the CC BY-NC 4.0 license is strictly required here because this dataset builds upon and incorporates third-party 3D assets and environments that are released under restrictive, non-commercial licenses. To comply with these upstream legal requirements, the derivative dataset must also carry the Non-Commercial restriction.

By using this dataset, you agree to the license terms and usage limitations described in LICENSE, README.md and assets_source.md.

Additional Restrictions:

Inherited from the licence of used assets.

• No Generative AI Training: You may not use this dataset, in whole or in part, to train, fine-tune, or evaluate generative Artificial Intelligence (AI) models intended to create, synthesize, or generate new content (including but not limited to text, images, audio, or video).

About the dataset

URBANITE captures a diverse range of security scenarios—including armed attacks, fights, people falling, and abandoned objects—alongside everyday pedestrian normality. Every scenario begins in a state of normality before any anomaly occurs, and the high-risk attack scenarios (PanicHit, PanicShot) trigger realistic crowd-panic dynamics in which pedestrians abandon their current tasks and flee towards the nearest escape point.

Key Dataset Statistics:

• Scale: 292 simulated scenarios, over 9 hours of multi-camera video (≈35,000 s in total across all cameras, ≈21 s per camera on average).
• Subjects: 12,000 unique pedestrians and 50,000 distinct tracks (average track length ≈24 frames).
• Actions: 17 types of annotated behaviors, including normal and anomalous situations.
• Maps: six 3D maps built from four environments, each simulated across 9 weather/time-of-day combinations at medium crowd density.
• Capture: 25 fps at 1920×1080, multiple cameras per simulation.
• Format: Standardized labeling scheme that extends the ASAM OpenLABEL format specifically for security contexts.

Annotation volume (summed over every camera of the dataset):

Annotation	Total
2D bounding boxes	≈60,000,000
2D poses (skeletons)	≈20,000,000
Actions	≈75,000 (17 classes)
Tracks	50,000

Areas & weathers

The dataset was created in four environments (City Center, Old Town, Station Entrance, and Mountain Village); some environments were further subdivided into smaller areas of interest, resulting in six 3D maps in total. The figure below shows the six maps:

• Top left: old town main square.
• Top center: square of a snow-covered mountain village.
• Top right: snowy street corner with parked vehicles.
• Bottom left: urban city street intersection.
• Bottom center: urban street next to a parking lot.
• Bottom right: modern urban plaza near a train station entrance.

Each scenario is simulated with one of three times of day (noon, sunset, and night) and one of three weather conditions (Clear_Skies, Overcast and Partly_Cloudy), yielding the 9 weather/time-of-day combinations listed below.

• Clear_Skies_Noon
• Clear_Skies_Sunset
• Clear_Skies_Night
• Overcast_Noon
• Overcast_Sunset
• Overcast_Night
• Partly_Cloudy_Noon
• Partly_Cloudy_Sunset
• Partly_Cloudy_Night

Labels structure

VCD annotation files

The annotations are stored in Video Content Description (VCD) JSON files that follow the ASAM OpenLABEL standard, extended to the public-safety domain. Each simulation produces two complementary levels of annotation:

• vcd_3d_data.json (world level): the complete simulation state in absolute world coordinates — 3D bounding boxes for every labelled object and full 3D skeletal poses for all humans (even when not visible to a specific camera), temporal actions, single-frame events, semantic relationships, and the camera poses.
• {cam}_vcd.json (per camera, e.g. cam1_vcd.json): viewpoint-dependent annotations — localized 2D bounding boxes, 2D skeletal poses, and per-joint visibility states, stored only for the frames in which each object is visible to that camera.

Both files share a persistent object identifier (UID) of the form <3-digit class id><5-digit instance id> (e.g. 25500008 = pedestrian 8; class 255 is Pedestrian), so the same individual keeps the same UID across every camera and in the world-level file, enabling consistent multi-camera tracking.

Key components of the VCD files:

• Coordinate systems: a scene coordinate system (odom) plus one sensor coordinate system per camera, each camera storing its 3D pose relative to the scene with a 4×4 homogeneous transformation matrix (pose_wrt_parent) that maps world-space points into local camera coordinates.
• Streams: per-camera intrinsics, including image dimensions (width and height) and the 3×4 pinhole projection matrix (camera_matrix_3x4) that projects 3D camera coordinates onto the 2D image plane.
• 2D bounding boxes: [x, y, w, h] in pixel coordinates (origin at the top-left), where x and y denote the bounding box centre and w and h its axis-aligned dimensions.
• 3D bounding boxes: cuboids defined by a spatial centre position [cx, cy, cz] and size extents (length, width, height) in meters.
• Skeletons (2D and 3D): since the OpenLABEL standard does not define a native skeleton type, poses are encapsulated within standard poly2d and poly3d vector structures representing the 23-keypoint skeleton model.
• Actions: continuous behaviors defined by frame intervals (frame_start → frame_end); as soon as one action ends, the next frame begins recording the start of the next action.
• Events: single-frame temporal boundaries that mark transitions or milestones within broader actions, indexed under openlabel.events with frame_start equal to frame_end.
• Relationships: RDF-like triples that link entities across the whole simulation — Own (persistent ownership of carried items, kept even after physical contact is broken, e.g. an abandoned object), isSubjectOfAction / isSubjectOfEvent (who performs an action or event), and actsOnObject (who did what to whom, and when).

Instance segmentation maps

The sgm/ PNG maps use a colour-based instance encoding that bypasses the 255-ID limit of the Unreal Engine stencil buffer: the red channel encodes the object class (up to 255 classes; class 255 is Pedestrian), while the green and blue channels jointly encode the instance ID (up to 255×255 instances per class). Two instances of the same class may differ by only 1 in a single channel (e.g. 255,0,1 vs 254,0,1), which is indistinguishable to the human eye — compare the values programmatically rather than visually.

Folder structure

Dataset/
├── <Action>/                        # one of the 6 scenarios (see Scenario list)
│   ├── <Map>/                       # one of the 6 maps
│   │   └── MediumDensity/
│   │       └── <Weather_TimeOfDay>/ # e.g. Clear_Skies_Noon (simulation folder)
│   │           ├── cam1/
│   │           │   ├── rgb/
│   │           │   │   ├── img_000000.jpeg
│   │           │   │   └── ...
│   │           │   ├── sgm/
│   │           │   │   ├── img_000000.png
│   │           │   │   └── ...
│   │           │   └── cam1_vcd.json
│   │           ├── cam2/
│   │           │   └── ...
│   │           ├── camera_config/
│   │           │   ├── BP_camera_cam1_sgm.txt
│   │           │   ├── ...
│   │           │   ├── SceneCapture2D_cam1_rgb.txt
│   │           │   └── ...
│   │           ├── skeleton3D/
│   │           │   ├── img_0.txt
│   │           │   └── ...
│   │           ├── bbox3D/
│   │           │   ├── img_0.txt
│   │           │   └── ...
│   │           ├── actions.txt
│   │           ├── spawned_characters.txt
│   │           └── vcd_3d_data.json

• rgb/ and sgm/ — RGB frames (JPEG) and instance segmentation maps (PNG), one per frame, per camera.
• cam1_vcd.json — per-camera OpenLABEL annotations (2D bounding boxes, 2D skeletons, visibility states).
• vcd_3d_data.json — world-level OpenLABEL annotations (3D bounding boxes, 3D skeletons, actions, events, relationships, camera poses).
• camera_config/ — per-camera configuration files of the RGB and SGM capture setups.
• skeleton3D/ and bbox3D/ — raw per-frame 3D skeleton and bounding-box extractions.
• actions.txt — raw per-frame action extraction.
• spawned_characters.txt — characters spawned in the simulation.

The number of cameras varies per simulation (cam1…camN). For this dataset version only the "medium density" option (the optimum density for each map) is available; the folder structure is designed to accommodate future updates of the dataset with varying crowd densities or additional action classes.

Scenario list

Each top-level folder corresponds to one of the six simulated scenarios:

• Normality: A common, everyday scenario where pedestrians and vehicles behave in a usual and expected way. Typical actions include talking, walking, or sitting. Every scenario begins in a state of normality before any anomaly occurs.
• AbandonedObject: A pedestrian leaves behind an object they were carrying and then walks away.
• Fall: A person unexpectedly falls to the ground.
• Fight: Two people start arguing, and the conflict escalates when one physically strikes the other.
• PanicHit: A person suddenly runs into the scene and attacks another with a weapon, which may be a bladed weapon (e.g., knife) or a blunt weapon (e.g., stick, baseball bat). The victim falls to the ground.
• PanicShot: A person suddenly runs into the scene and shoots another individual. The victim falls to the ground.

Skeleton structure

The skeleton has 23 keypoints covering all primary joints (shoulders, elbows, hands, hips, knees, and feet), plus centre reference points (Pose_ShoulderC, Pose_HipC) and dual points for the hands and feet (e.g. Pose_HandL_L/Pose_HandL_R), which make calculating body orientation and hand/foot angles much easier. It can be mapped onto popular human pose benchmarks (COCO, AIC, CrowdPose, OCHuman, Sub-JHMDB) and easily converted to standard 14-, 15-, or 17-point formats.

See the joints name, order and connections in: code_examples/plot_data/skeleton_description.yaml

Assets list and source

The list of assets used, along with their source links and original licenses, is available in assets_source.md.

Users of the URBANITE dataset must respect the original licenses and attribution requirements of these third-party assets as detailed in the linked document.

Actions and events

Actions

The annotated action classes (names as stored in the VCD files):

• Idle
• Sit
• StandUp
• Walk
• Talk
• Fall
• Lie
• Hit (attacker)
• isHitting (victim)
• Fight (attacker)
• isFought (victim)
• Shot (attacker)
• IsShot (victim)
• PanicFlee (general pedestrians running after an attack)
• AttackerFlee (attacker leaving the scene running)
• Possess (carrying an item)
• AbandonedObject

Actions are recorded with frame-accurate temporal intervals that define their duration, ensuring continuity of activity: as soon as one action ends, the next frame begins recording the start of the next action.

Events

Events are single-frame temporal boundaries that mark transitions or milestones within broader actions, following the naming pattern <action>_<state>, where state typically represents start or end. Complex actions include intermediate milestones (step1):

• AbandonedObject → AbandonedObject_start (initiating the motion to leave the object) / AbandonedObject_step1 (the exact frame the object is released) / AbandonedObject_end (the actor returns to a standing position and begins walking away)
• Shot (the one who shoots) → Shot_start / Shot_step1 / Shot_end
• Hit (the one who hits) → Hit_start / Hit_step1 / Hit_end

Note: isHitting and IsShot start at <action>_step1 (the moment the victim receives the aggression). This can be considered similar to a fall, but it has a different label because it is provoked by another subject. Ongoing activities that lack a captured initiation or end trigger do not generate events.

Code examples

The repository contains Python examples for reading, reporting and visualizing the VCD annotations. By default they operate on the bundled sample world-level file input/vcd_3d_data.json.

Script	Description
A_list_scene_actions_and_events.py	Lists every distinct action and event type present in a VCD label file.
B_report_pedestrian_actions_events_ownership.py	Reads pedestrian actions, events and ownership relations from a VCD label file, detailing who performed an action and when.
C_save_output_imgs.py	Iterates over a dataset (or a single simulation folder) and saves images with 2D bounding boxes, skeletons and head points plotted from the existing per-camera VCD files.
D_visualize_3d_scene_frame.py	Reads 3D bounding boxes and skeletons from a VCD label file and reconstructs the spatial scene with Open3D.
E_project_3d_bboxes_to_frame.py	Projects 3D bounding boxes from a VCD label file onto the corresponding camera frames.
dataset_stats.py	Computes camera-level dataset statistics (poses, bboxes, trajectories, actions) over the whole dataset.

Usage

Install the dependencies (Python ≥ 3.10):

uv sync

Then run any example, e.g.:

python code_examples/get_relations_actions_and_events/A_list_scene_actions_and_events.py --vcd-path /path/to/vcd_3d_data.json

Note: all scripts must be run from the root directory, as shown in the example.

Parsing the annotations: we strongly recommend using the vcd Python library (version 6.0.3, available at https://pypi.org/project/vcd/) when parsing, evaluating, or interacting with the dataset's annotations, as it is the library used to generate and validate them.