From 5de1c145092ec74a612e5d9556c34dc658162e2b Mon Sep 17 00:00:00 2001 From: Md Saiful Islam Date: Tue, 23 Jun 2026 14:48:43 -0400 Subject: [PATCH 1/4] Removed implementaton docs --- .../DATA_HANDLING_FOR_PAPER.md | 1715 ----------------- .../DIRECTORY_DOWNLOAD_DESIGN.md | 192 -- .../FINGERPRINT_IMPLEMENTATION_SUMMARY.md | 236 --- .../FLOABILITY_DATA_OPERATIONS_SUMMARY.md | 540 ------ .../QUICKSTART_FINGERPRINTING.md | 228 --- .../S3_IMPLEMENTATION_SUMMARY.md | 159 -- implementation-docs/TEST_FINGERPRINTING.md | 593 ------ 7 files changed, 3663 deletions(-) delete mode 100644 implementation-docs/DATA_HANDLING_FOR_PAPER.md delete mode 100644 implementation-docs/DIRECTORY_DOWNLOAD_DESIGN.md delete mode 100644 implementation-docs/FINGERPRINT_IMPLEMENTATION_SUMMARY.md delete mode 100644 implementation-docs/FLOABILITY_DATA_OPERATIONS_SUMMARY.md delete mode 100644 implementation-docs/QUICKSTART_FINGERPRINTING.md delete mode 100644 implementation-docs/S3_IMPLEMENTATION_SUMMARY.md delete mode 100644 implementation-docs/TEST_FINGERPRINTING.md diff --git a/implementation-docs/DATA_HANDLING_FOR_PAPER.md b/implementation-docs/DATA_HANDLING_FOR_PAPER.md deleted file mode 100644 index f21cf1c..0000000 --- a/implementation-docs/DATA_HANDLING_FOR_PAPER.md +++ /dev/null @@ -1,1715 +0,0 @@ -# Data Handling in Floability: Implementation Details for Paper - -*Generated for paper on portable execution of data-intensive notebook workflows* -*Date: February 8, 2026* - ---- - -## Table of Contents - -1. [Declarative Data Specification](#section-3-declarative-data-specification) - - [Core Design Principles](#core-design-principles) - - [Data.yml Structure](#datayml-structure) - - [Portability Features](#portability-features) - - [Source Type Abstraction](#source-type-abstraction) - - [Multi-Source Fallback](#multi-source-fallback) - - [Profile-Based Configuration](#profile-based-configuration) - -2. [Implementation in Floability](#section-4-implementation-in-floability) - - [Three Core Operations](#three-core-operations) - - [Local Data Cache Architecture](#local-data-cache-architecture) - - [Content-Addressable Caching](#content-addressable-caching) - - [Source Fingerprinting](#source-fingerprinting) - - [Directory vs Single File Handling](#directory-vs-single-file-handling) - - [Cache Materialization](#cache-materialization) - ---- - -## Section 3: Declarative Data Specification - -### Core Design Principles - -The Floability data specification is designed around three key principles that enable portable execution: - -1. **Separation of Concerns**: Data requirements are specified independently from notebook code -2. **Location Independence**: Data sources are referenced by URI, not filesystem paths -3. **Declarative Configuration**: What data is needed, not how to fetch it - -This design allows the same notebook workflow to access data from different locations across HPC sites without modifying the notebook code itself. - -### Data.yml Structure - -Data requirements are expressed in a YAML file (`data.yml`) typically located at `/data/data.yml`. The specification follows a hierarchical structure: - -```yaml -schema_version: 1.0 -default_profile: profile_name - -data_profiles: - profile_name: - policy: - retry_attempts: 3 - timeout: 60 - size_tolerance_bytes: 1024 - run_operation: fetch - verification_type: strict - - data: - - name: dataset_name - source_type: s3 - source: s3://bucket/path/data.root - target_location: data/samples/dataset.root - expected_size: 1048576 - checksum: sha256:abc123... -``` - -**Key Components:** - -1. **Schema Version**: Enables forward compatibility as the spec evolves -2. **Data Profiles**: Named configurations for different execution environments -3. **Policy**: Runtime behavior (retries, timeouts, validation rules) -4. **Data Items**: Individual datasets with source, target, and integrity information - -### Portability Features - -#### 1. Profile-Based Configuration - -Multiple profiles enable environment-specific configurations within a single specification: - -```yaml -data_profiles: - # Development: local filesystem - local_dev: - data: - - source_type: fs - source: /local/path/to/data.root - target_location: data/samples/dataset.root - - # HPC Site A: S3 access - hpc_site_a: - data: - - source_type: s3 - source: s3://site-a-bucket/data.root - target_location: data/samples/dataset.root - - # HPC Site B: Pelican federation - hpc_site_b: - data: - - source_type: pelican - source: pelican://osg-htc.org:8443/ospool/data.root - target_location: data/samples/dataset.root -``` - -**Portability Benefit**: The same backpack can be executed across different sites by selecting the appropriate profile via `--data-profile` flag. The notebook code remains unchanged because `target_location` is consistent across profiles. - -**Implementation**: Profile selection happens during spec loading: -- Default profile is used unless `--data-profile` explicitly specified -- Profile validation ensures required fields are present -- Normalization applies defaults and infers missing fields - -#### 2. Target Location Consistency - -All profiles specify the same `target_location`, ensuring notebooks can reference data at predictable paths: - -```python -# Notebook code - works across all profiles -data_path = Path("data/samples/dataset.root") -df = load_data(data_path) -``` - -This path-based consistency is critical for portability: -- Notebooks don't need to know about source URIs -- Data appears at the same location regardless of where it came from -- File paths in notebooks are relative to the workflow directory - -### Source Type Abstraction - -Floability abstracts heterogeneous storage systems through a unified source type system: - -| Source Type | Protocol | Use Case | Example | -|------------|----------|----------|---------| -| `http` | HTTP/HTTPS | Public datasets, web-hosted files | `https://data.gov/dataset.csv` | -| `s3` | AWS S3 | Object storage (files & directories) | `s3://bucket/path/data.root` | -| `pelican`/`osdf` | Pelican/OSDF | Federated data systems (files & directories) | `pelican://osg-htc.org:8443/ospool/data/` | -| `fs` | Local filesystem | Site-local storage | `/scratch/shared/data.root` | -| `backpack` | Relative to backpack root | Packaged datasets | `backpack://data/sample.csv` | -| `multi` | Multiple fallback sources | Resilient access | (see below) | - -**Automatic Type Inference**: Source types can be inferred from URI schemes: -```yaml -# Explicit (recommended) -- source_type: s3 - source: s3://bucket/data.root - -# Inferred (convenience) -- source: s3://bucket/data.root # type inferred as "s3" -``` - -**Implementation Details**: -- Type inference occurs during spec normalization (`_normalize_data_item`) -- Backpack-relative paths are resolved against `backpack_root` -- Relative filesystem paths are resolved against `backpack_root` unless absolute - -### Multi-Source Fallback - -The `multi` source type enables resilient data access across heterogeneous environments: - -```yaml -data: - - name: cms_physics_data - source_type: multi - sources: - # Try Pelican federation first (fastest for HPC sites) - - source_type: pelican - source: pelican://osg-htc.org:8443/ospool/cms/data.root - - # Fallback to S3 (works everywhere with internet) - - source_type: s3 - source: s3://cms-open-data/2012/data.root - - # Final fallback to HTTP mirror - - source_type: http - source: https://cms-mirror.cern.ch/data.root - - # Last resort: packaged copy - - source_type: backpack - source: data/fallback/data.root - - target_location: data/cms/data.root - expected_size: 108000000 # ~108 GB - checksum: sha256:def456... -``` - -**Fallback Logic**: -1. Sources are tried in order until one succeeds -2. Metadata checks (if available) are performed first -3. Download only happens if metadata check passes or is unavailable -4. First successful source is used; remaining sources are skipped - -**Portability Benefit**: -- Workflows can adapt to different network topologies -- No manual configuration needed per site -- Automatic failover if primary source unavailable - -**Implementation** (from `_fetch_single_item` and `_download_to_cache`): -```python -if stype == "multi": - success = False - for src_entry in item.get("sources", []): - if _download_to_cache(src_entry, cache_file, backpack_root, verbose): - success = True - break # First success stops iteration -``` - -### Profile-Based Configuration - -Profiles enable environment-specific behavior without changing notebook code: - -#### Policy Configuration - -Runtime behavior is controlled through policy settings: - -```yaml -data_profiles: - production: - policy: - # Retry failed downloads (network resilience) - retry_attempts: 3 - - # Timeout for download operations (seconds) - timeout: 120 - - # Allow size deviation (handles metadata variations) - size_tolerance_bytes: 10 - - # Default operation: 'check', 'fetch', or 'verify' - run_operation: fetch - - # Verification: 'strict' (requires checksum) or 'size_only' - verification_type: strict -``` - -**Default Values** (applied during normalization): -- `retry_attempts: 0` (no retries by default) -- `timeout: None` (no timeout) -- `size_tolerance_bytes: 0` (exact size match required) -- `run_operation: 'fetch'` (download by default) -- `verification_type: 'size_only'` (checksums optional) - -#### Profile Selection Workflow - -1. **Load YAML**: Parse `data.yml` file -2. **Select Profile**: Use `--data-profile` flag or `default_profile` from spec -3. **Validate**: Check required fields (data items, sources, targets) -4. **Normalize**: Apply defaults, infer types, resolve paths -5. **Execute**: Run selected operation with normalized profile - -**Implementation** (`load_and_validate_spec`): -```python -def load_and_validate_spec(data_spec, backpack_root, requested_profile, verbose): - raw = _load_yaml(spec_path) - profile_name, profile, _ = _select_profile(raw, requested_profile) - _validate_required_fields(profile) - normalized = _normalize_data_profile(profile, backpack_root) - return profile_name, normalized -``` - -### Data Item Schema - -Each data item in the `data` list supports the following fields: - -**Required Fields**: -- `source` or `sources`: Where to fetch data from -- `target_location`: Where to materialize data in workflow - -**Optional Fields**: -- `name`: Human-readable identifier (default: basename of target_location) -- `source_type`: Type of source (inferred if omitted) -- `source_object_type`: Explicit "file" or "directory" (for S3/Pelican) -- `expected_size`: Size in bytes (used for validation) -- `checksum`: Integrity check in format "algorithm:hex" -- `content_type`: MIME type (informational) -- `post_process`: Transformation commands (not yet implemented) - -**Integrity Signals**: -```yaml -# Size validation with tolerance -expected_size: 1048576 -policy: - size_tolerance_bytes: 10 # Allow ±10 bytes - -# Checksum validation (strict) -checksum: sha256:e6b7897aa8498b8dac4df0664827f857bc01135c3d9311adb820979bbc44b763 - -# Algorithm-prefixed checksum (recommended) -checksum: sha256:abc123... -checksum: md5:def456... -checksum: sha1:789abc... -``` - -**Checksum Inference**: If algorithm not specified, Floability infers from hex length: -- 32 hex chars → MD5 -- 40 hex chars → SHA1 -- 64 hex chars → SHA256 - -### Directory Support (New in Feb 2026) - -Both S3 and Pelican sources support directory downloads: - -```yaml -# Explicit directory type (recommended) -- name: training_data - source_type: s3 - source: s3://mybucket/datasets/training/ - source_object_type: directory - target_location: data/training - -# Auto-detect via trailing slash -- name: training_data - source: s3://mybucket/datasets/training/ # trailing "/" signals directory - target_location: data/training - -# Single file for comparison -- name: model_weights - source: s3://mybucket/models/weights.h5 - source_object_type: file # explicit file type - target_location: data/model.h5 -``` - -**Directory Detection Priority**: -1. **Explicit field**: `source_object_type: "directory"` or `"file"` -2. **URL trailing slash**: `s3://bucket/prefix/` → directory -3. **Metadata check**: Query S3/Pelican to determine type - -**Implementation** (from `_download_to_cache`): -```python -# Determine if source is a directory -source_obj_type = item.get("source_object_type", "").lower() - -is_dir = False -if source_obj_type == "directory": - is_dir = True -elif source_obj_type == "file": - is_dir = False -else: - # Auto-detect: trailing slash or metadata check - is_dir = source.endswith('/') or is_s3_directory(source) - -if is_dir: - s3_directory_download(source, dest_dir=str(cache_file), ...) -else: - s3_file_download(source, dest_dir=str(cache_file.parent), ...) -``` - ---- - -## Section 4: Implementation in Floability - -### Three Core Operations - -Floability provides three data operations, each serving different use cases in portable execution: - -#### 1. Check (Metadata-Only) - -**Purpose**: Verify data availability without downloading - -**Use Cases**: -- Pre-flight validation before workflow execution -- Checking cache status without triggering downloads -- Verifying data sources are accessible from current site - -**Behavior**: -1. Query source metadata (HEAD requests, S3 head_object, etc.) -2. Validate expected_size within tolerance if specified -3. Report cache status if caching enabled -4. **No downloads or file writes** - -**Command**: -```bash -floability data --data-spec data/data.yml --mode check --verbose -``` - -**Implementation** (`check_data_from_spec`): -```python -def check_data_from_spec(data_spec, backpack_root, ...): - profile_name, profile = load_and_validate_spec(...) - items = profile.get("data", []) - policy = profile.get("policy", {}) - tolerance = policy.get("size_tolerance_bytes", 0) - - results = [] - for item in items: - result = _check_single_item(item, tolerance, backpack_root, ...) - results.append(result) - - success = _print_check_summary(results) - return success -``` - -**Metadata Gathering** (per source type): -```python -def _metadata_for_source(item, backpack_root): - stype = item.get("source_type") - src = item.get("source") - - if stype == "http": - return http_file_metadata(src) # HEAD request - elif stype == "s3": - return s3_file_metadata(src) # head_object - elif stype == "pelican": - return pelican_file_metadata(src) # fs.info() - elif stype in ("fs", "backpack"): - path = resolve_path(src, backpack_root) - return fs_file_metadata(str(path)) # stat() -``` - -**Multi-Source Checking**: -```python -if stype == "multi": - sources = item.get("sources", []) - for s in sources: - meta = _metadata_for_source(s, backpack_root) - if meta.get("exists"): - break # First available source -``` - -**Size Validation**: -```python -expected_size = item.get("expected_size") -actual_size = meta.get("size") - -if expected_size and actual_size: - diff = abs(actual_size - expected_size) - size_ok = diff <= tolerance -``` - -**Output**: -``` -[data:check] Summary: -name exists size_ok expected_size actual_size cache_exists cache_valid -cms_data True True 108000000 108000123 True True -training_set True False 50000000 50005000 False N/A -model_weights False N/A 1048576 N/A N/A N/A -``` - -#### 2. Fetch (Download/Copy) - -**Purpose**: Download/copy data to target locations - -**Use Cases**: -- Initial workflow setup (staging data before execution) -- Populating workflow instances with required datasets -- Standard case for most workflow executions - -**Behavior**: -1. Check if target already exists (skip if present unless `--force`) -2. Download/copy from source to cache (if caching enabled) -3. Materialize from cache to target location using specified mode -4. For multi-source items, try each source until success - -**Command**: -```bash -floability data --data-spec data/data.yml \ - --mode fetch \ - --data-cache-mode symlink \ - --data-cache-dir ~/floability-data-cache \ - --force # Force re-download even if target exists -``` - -**Implementation** (`fetch_data_from_spec`): -```python -def fetch_data_from_spec(data_spec, backpack_root, force=False, ...): - profile_name, profile = load_and_validate_spec(...) - items = profile.get("data", []) - - # Determine target_prefix (where to materialize data) - if target_root: - target_prefix = target_root - elif backpack_root: - target_prefix = backpack_root / "workflow" - - for item in items: - result = _fetch_single_item( - item, backpack_root, target_prefix, - force=force, data_cache_mode=data_cache_mode, ... - ) -``` - -**Fetch Flow** (with caching): -```python -def _fetch_single_item(item, backpack_root, target_prefix, ...): - # 1. Resolve target path - target_path = _resolve_target_path(item, backpack_root, target_prefix) - - # 2. Skip if exists (unless force) - if target_path.exists() and not force: - return success - - # 3. Try cache if enabled - if data_cache_mode != "off": - artifact_spec = _create_artifact_spec(item, backpack_root) - cache_key = _compute_cache_key(artifact_spec) - cache_dir = _get_cache_dir(cache_base_dir, cache_key) - - # Check existing cache - cache_meta = _lookup_cache_entry(cache_dir, artifact_spec, ...) - - if not cache_meta or force_data_cache: - # Build new cache entry - if not _acquire_cache_lock(cache_dir, timeout=300): - # Lock timeout, fall back to direct fetch - return _attempt_fetch_source(item, target_path, ...) - - try: - success = _build_cache_entry(item, cache_dir, backpack_root, ...) - finally: - _release_cache_lock(cache_dir) - - # Materialize from cache to target - return _materialize_from_cache(cache_dir, target_path, mode=data_cache_mode) - - # 4. Direct fetch (no caching) - return _attempt_fetch_source(item, target_path, backpack_root, ...) -``` - -**Direct Fetch** (no caching): -```python -def _attempt_fetch_source(item, target_path, backpack_root, ...): - stype = item.get("source_type") - source = item.get("source") - - if stype == "http": - http_file_download(source, dest_dir=target_path.parent, ...) - elif stype == "s3": - # Check if directory or file - is_dir = is_s3_directory(source) or source.endswith('/') - if is_dir: - s3_directory_download(source, dest_dir=target_path, ...) - else: - s3_file_download(source, dest_dir=target_path.parent, ...) - # ... similar for pelican, fs, backpack -``` - -#### 3. Verify (Fetch + Integrity Check) - -**Purpose**: Ensure data exists and passes integrity checks - -**Use Cases**: -- Production workflows requiring data integrity guarantees -- Validating checksums of critical datasets -- Ensuring reproducibility through cryptographic verification - -**Behavior**: -1. Perform fetch if target doesn't exist -2. Validate size within tolerance (if expected_size specified) -3. Validate checksum (if specified and verification_type="strict") -4. Produce detailed verification report - -**Command**: -```bash -floability data --data-spec data/data.yml \ - --mode verify \ - --data-cache-mode symlink \ - --verbose -``` - -**Implementation** (`verify_data_from_spec`): -```python -def verify_data_from_spec(data_spec, backpack_root, ...): - profile_name, profile = load_and_validate_spec(...) - items = profile.get("data", []) - policy = profile.get("policy", {}) - tolerance = policy.get("size_tolerance_bytes", 0) - - results = [] - for item in items: - # 1. Ensure data exists (fetch if needed) - target_path = _resolve_target_path(item, backpack_root, target_prefix) - if not target_path.exists(): - _fetch_single_item(item, backpack_root, target_prefix, ...) - - # 2. Size validation - size_ok = None - if item.get("expected_size"): - actual_size = get_file_size(target_path) - diff = abs(actual_size - expected_size) - size_ok = diff <= tolerance - - # 3. Checksum validation (if strict) - checksum_ok = None - if policy.get("verification_type") == "strict": - checksum_spec = _extract_checksum_field(item) - if checksum_spec: - alg, expected_hex = _parse_checksum_spec(checksum_spec) - actual_hex = _compute_checksum(target_path, alg) - checksum_ok = (actual_hex == expected_hex) - - results.append({ - "name": item.get("name"), - "exists": target_path.exists(), - "size_ok": size_ok, - "checksum_ok": checksum_ok, - ... - }) - - success = _print_verify_summary(results) - return success -``` - -**Checksum Computation**: -```python -def _compute_checksum(path, alg, chunk_size=1024*1024): - h = hashlib.new(alg) # 'md5', 'sha1', 'sha256', etc. - with open(path, "rb") as f: - while True: - chunk = f.read(chunk_size) - if not chunk: - break - h.update(chunk) - return h.hexdigest() -``` - -**Output**: -``` -[data:verify] Summary: -name exists size_ok checksum_ok expected_size actual_size checksum_alg -cms_data True True True 108000000 108000000 sha256 -training_set True True False 50000000 50000000 sha256 -model True False N/A 1048576 1050000 N/A - -❌ Verification failed: 2 items failed checks -``` - -### Local Data Cache Architecture - -The local data cache is a critical component for portable execution across HPC sites. It addresses several challenges: - -1. **Avoid Redundant Transfers**: Cache data across multiple workflow runs -2. **Share Data**: Multiple instances can reference the same cached data -3. **Decouple Source from Target**: Source changes don't affect workflow paths -4. **Enable Materialization Modes**: Support symlink/hardlink/copy strategies - -#### Cache Directory Structure - -**Cache Root**: `/` (default: `~/floability-data-cache`) - -**Per-Artifact Structure** (Updated Feb 2026): -``` -/ - / # SHA-256 of artifact spec (64 hex chars) - cached_data/ # NEW: Full target structure preserved - / # e.g., "data/samples/file.root" - - .meta.json # Metadata (artifact spec, hashes, timestamps) - .verify.lock # Temporary lock during cache build -``` - -**Key Design Change (Feb 2026)**: -- **Old**: `cached_data/data/` (flat structure) -- **New**: `cached_data/` (full path structure) -- **Benefit**: Simplifies materialization, handles nested directories naturally - -**Example**: -``` -# Data item spec -target_location: "data/samples/test/file.root" - -# Old cache structure -/ - cached_data/ - data/ - file.root # Lost "samples/test" structure - -# New cache structure (Feb 2026) -/ - cached_data/ - data/ - samples/ - test/ - file.root # Full structure preserved -``` - -#### Cache Key Computation (Content-Addressable) - -Cache keys are computed from a normalized "artifact spec" containing only fields that affect the cached bytes: - -**Artifact Spec Fields**: -- `source_type`: Type of source -- `source` or `sources`: Source URI(s) with resolved absolute paths for local sources -- `checksum`: Expected checksum (if specified) -- `expected_size`: Expected size (if specified) -- `content_type`: MIME type (if specified) -- `post_process`: Transformation instructions (if specified) - -**NOT included** (target-specific): -- `name`: Display name -- `target_location`: Where to materialize -- `target_prefix`: Target directory - -**Implementation** (`_create_artifact_spec`): -```python -def _create_artifact_spec(item, backpack_root): - artifact = {} - artifact["source_type"] = item.get("source_type") - - if item["source_type"] == "multi": - # Include all sources in order - sources = [] - for s in item.get("sources", []): - s_type = s.get("source_type") - s_source = s.get("source") - # Resolve relative fs/backpack paths to absolute - if s_type in ("fs", "backpack"): - p = Path(s_source) - if not p.is_absolute(): - s_source = str((backpack_root / p).resolve()) - sources.append({"source_type": s_type, "source": s_source}) - artifact["sources"] = sources - else: - source = item.get("source") - # Resolve relative paths to absolute - if item["source_type"] in ("fs", "backpack"): - p = Path(source) - if not p.is_absolute(): - source = str((backpack_root / p).resolve()) - artifact["source"] = source - - # Optional fields (only if specified) - if item.get("checksum"): - artifact["checksum"] = item["checksum"] - if item.get("expected_size"): - artifact["expected_size"] = item["expected_size"] - if item.get("content_type"): - artifact["content_type"] = item["content_type"] - if item.get("post_process"): - artifact["post_process"] = item["post_process"] - - return artifact -``` - -**Cache Key Computation** (`_compute_cache_key`): -```python -def _compute_cache_key(artifact_spec): - # Canonical JSON (sorted keys, compact) - canonical_json = json.dumps(artifact_spec, sort_keys=True, separators=(",", ":")) - - # SHA-256 hash - cache_key = hashlib.sha256(canonical_json.encode("utf-8")).hexdigest() - - return cache_key # 64 hex characters -``` - -**Content-Addressable Benefits**: -1. **Deduplication**: Same source → same cache entry -2. **Deterministic**: Same artifact spec → same cache key -3. **Safe Sharing**: Multiple workflows can safely share cache entries -4. **Integrity**: Cache key depends on checksums/sizes if specified - -**Example**: -```python -# Two data items with same source but different targets -item1 = { - "source_type": "s3", - "source": "s3://bucket/data.root", - "target_location": "data/run1/data.root", - "checksum": "sha256:abc123..." -} - -item2 = { - "source_type": "s3", - "source": "s3://bucket/data.root", - "target_location": "data/run2/data.root", # Different target - "checksum": "sha256:abc123..." -} - -# Both produce SAME cache key (target_location not in artifact spec) -# -> Both use same cache entry, but materialize to different locations -``` - -### Content-Addressable Caching - -#### Building Cache Entries - -**Cache Build Flow** (`_build_cache_entry`): -```python -def _build_cache_entry(item, cache_dir, backpack_root, fingerprint_mode, ...): - # 1. Create cached_data/ directory - cached_data_dir = cache_dir / "cached_data" - cached_data_dir.mkdir(parents=True, exist_ok=True) - - # 2. Determine cache file path with full target_location structure - target_location = item.get("target_location") - cache_file = cached_data_dir / target_location # e.g., "data/samples/test/file.root" - - # 3. Download/copy data to cache - stype = item.get("source_type") - if stype == "multi": - # Try each source until success - for src_entry in item.get("sources", []): - if _download_to_cache(src_entry, cache_file, backpack_root, ...): - break - else: - _download_to_cache(item, cache_file, backpack_root, ...) - - # 4. Apply post-processing (not yet implemented) - post_process = item.get("post_process") - if post_process: - # TODO: unzip, untar, transform, etc. - pass - - # 5. Compute content hash and size - content_sha256, actual_size = _compute_content_hash(cache_file) - - # 6. Compute source fingerprint (for filesystem sources) - source_fingerprint = None - if stype in ("fs", "backpack"): - from .fingerprint import compute_fingerprint - source_path = resolve_path(item["source"], backpack_root) - source_fingerprint = compute_fingerprint( - str(source_path), - mode=fingerprint_mode, # "meta", "sample", or "strict" - verbose=verbose - ) - - # 7. Write metadata - artifact_spec = _create_artifact_spec(item, backpack_root) - _write_cache_metadata( - cache_dir, artifact_spec, content_sha256, actual_size, source_fingerprint - ) - - return True -``` - -#### Content Hash Computation - -Content hashing differs for files vs directories: - -**Single Files** (`_compute_content_hash`): -```python -def _compute_content_hash(path): - if path.is_file(): - h = hashlib.sha256() - size = 0 - with path.open("rb") as f: - while True: - chunk = f.read(1024 * 1024) # 1MB chunks - if not chunk: - break - h.update(chunk) - size += len(chunk) - return h.hexdigest(), size -``` - -**Directories** (Merkle-like tree hash): -```python -elif path.is_dir(): - h = hashlib.sha256() - total_size = 0 - - # Collect all files with their hashes - file_hashes = [] - for root, dirs, files in os.walk(path): - dirs.sort() # Deterministic order - files.sort() - - for filename in files: - file_path = Path(root) / filename - rel_path = file_path.relative_to(path) - file_hash, file_size = _compute_content_hash(file_path) # Recursive - file_hashes.append((str(rel_path), file_hash, file_size)) - total_size += file_size - - # Hash sorted list of (path, hash, size) - for rel_path, file_hash, file_size in sorted(file_hashes): - h.update(f"{rel_path}:{file_hash}:{file_size}\n".encode("utf-8")) - - return h.hexdigest(), total_size -``` - -**Directory Hash Properties**: -- **Structure-sensitive**: Different file arrangements produce different hashes -- **Deterministic**: Same directory tree → same hash (files sorted) -- **Content-addressed**: Hash depends on file contents, not just metadata -- **Efficient**: Only hashes file paths and individual file hashes (not re-reading content) - -#### Cache Metadata (.meta.json) - -Metadata stored in `.meta.json`: - -```json -{ - "artifact_spec": { - "source_type": "s3", - "source": "s3://bucket/data.root", - "checksum": "sha256:abc123...", - "expected_size": 108000000 - }, - "content_sha256": "def456...", - "actual_size": 108000123, - "created_at": 1707350400.0, - "created_at_iso": "2026-02-08T00:00:00Z", - "source_fingerprint": "789abc...", - "fingerprint_mode": "meta", - "fingerprint_params": { - "size": 108000123, - "mtime_ns": 1707350400000000000 - } -} -``` - -**Fields**: -- `artifact_spec`: Original spec used to compute cache key -- `content_sha256`: Hash of cached content (file or directory tree) -- `actual_size`: Actual size of cached content -- `created_at*`: Timestamps -- `source_fingerprint`: Fingerprint of original source (fs/backpack only) -- `fingerprint_mode`: Mode used ("meta", "sample", "strict") -- `fingerprint_params`: Mode-specific parameters - -#### Cache Lookup and Validation - -**Lookup Flow** (`_lookup_cache_entry`): -```python -def _lookup_cache_entry(cache_dir, artifact_spec, fingerprint_mode, backpack_root, ...): - # 1. Check cache directory exists - if not cache_dir.exists(): - return None # Cache miss - - # 2. Check for lock file (build in progress) - lock_file = cache_dir / ".verify.lock" - if lock_file.exists(): - return None # Cache building - - # 3. Read metadata - meta = _read_cache_metadata(cache_dir) - if not meta: - return None # Invalid/missing metadata - - # 4. Verify artifact spec matches - cached_spec = meta.get("artifact_spec", {}) - if cached_spec != artifact_spec: - return None # Spec mismatch - - # 5. Check cached_data/ directory exists - cached_data_dir = cache_dir / "cached_data" - if not cached_data_dir.exists(): - return None # Data missing - - # 6. Validate size (if specified) - expected_size = artifact_spec.get("expected_size") - actual_size = meta.get("actual_size") - if expected_size and actual_size and expected_size != actual_size: - return None # Size mismatch - - # 7. Validate source fingerprint (for fs/backpack sources) - source_type = artifact_spec.get("source_type") - if source_type in ("fs", "backpack"): - cached_fingerprint = meta.get("source_fingerprint") - if not cached_fingerprint: - return None # Old cache format (no fingerprint) - - # Recompute fingerprint from current source - source = artifact_spec.get("source") - source_path = resolve_path(source, backpack_root) - - if not source_path.exists(): - return None # Source no longer exists - - from .fingerprint import compute_fingerprint - current_fingerprint = compute_fingerprint( - str(source_path), - mode=fingerprint_mode, - verbose=False - ) - - if current_fingerprint["fingerprint"] != cached_fingerprint: - return None # Fingerprint mismatch (source changed) - - # Cache valid! - return meta -``` - -**Cache Invalidation Reasons**: -1. Cache directory doesn't exist (never cached) -2. Lock file exists (build in progress) -3. `.meta.json` missing or corrupt -4. Artifact spec mismatch (different source/checksum/size) -5. `cached_data/` directory missing -6. Size mismatch (expected vs actual) -7. Source fingerprint mismatch (for fs/backpack, source changed) - -#### Concurrent Access Protection - -**Lock Mechanism**: -```python -def _acquire_cache_lock(cache_dir, timeout=300): - """Atomically acquire lock or wait up to timeout seconds.""" - lock_file = cache_dir / ".verify.lock" - cache_dir.mkdir(parents=True, exist_ok=True) - - start_time = time.time() - while True: - try: - # Atomic create (fails if exists) - lock_file.touch(exist_ok=False) - lock_file.write_text(str(os.getpid())) - return True - except FileExistsError: - elapsed = time.time() - start_time - if elapsed > timeout: - return False # Timeout - time.sleep(1) # Wait and retry - -def _release_cache_lock(cache_dir): - """Release lock by removing .verify.lock file.""" - lock_file = cache_dir / ".verify.lock" - lock_file.unlink(missing_ok=True) -``` - -**Lock Usage**: -```python -# Try to acquire lock -if not _acquire_cache_lock(cache_dir, timeout=300): - # Lock timeout, fall back to direct fetch - return _attempt_fetch_source(item, target_path, ...) - -try: - # Build cache entry (protected by lock) - success = _build_cache_entry(...) -finally: - # Always release lock - _release_cache_lock(cache_dir) -``` - -**Concurrency Properties**: -- **Prevents duplicate work**: If one process is building cache, others wait -- **Timeout protection**: If lock held too long (300s), fall back to direct fetch -- **Idempotent**: Multiple processes can safely check same cache entry - -### Source Fingerprinting - -Source fingerprinting enables cache validation for filesystem sources where content may change over time. - -#### Motivation - -**Problem**: Content-addressable caching assumes sources are immutable. For filesystem sources, this isn't always true: -- Local files may be updated (new experiments, corrected datasets) -- Network filesystems may have clock skew (mtime unreliable) -- Cache may be stale if source changed after caching - -**Solution**: Compute and store a "fingerprint" of the source when building cache, then revalidate on cache lookup. - -#### Three Fingerprinting Modes - -Floability supports three modes with different performance/reliability tradeoffs: - -##### Mode 1: "meta" (Metadata-Based) - -**Files**: -```python -def fs_fingerprint_file_meta(path): - stat = path.stat() - size = stat.st_size - mtime_ns = stat.st_mtime_ns # Nanosecond precision - - record = f"size:{size}|mtime_ns:{mtime_ns}" - fingerprint = hashlib.sha256(record.encode("utf-8")).hexdigest() - - return { - "fingerprint": fingerprint, - "mode": "meta", - "params": {"size": size, "mtime_ns": mtime_ns} - } -``` - -**Directories**: -```python -def fs_fingerprint_dir_meta(root): - # Collect (relpath, size, mtime_ns) for all files - h = hashlib.sha256() - for relpath, size, mtime_ns, _ in collect_files(root): - record = f"{relpath}|{size}|{mtime_ns}\n" - h.update(record.encode("utf-8")) - - return {"fingerprint": h.hexdigest(), "mode": "meta", ...} -``` - -**Properties**: -- **Fast**: Only stat() calls, no file reads -- **Low overhead**: Suitable for large directories (thousands of files) -- **Clock-dependent**: Relies on mtime (may be unreliable on NFS) -- **Good for**: Development workflows, frequently-accessed local data - -##### Mode 2: "sample" (Content Sampling) - -**Files**: -```python -def fs_fingerprint_file_sample(path, sample_bytes=200): - stat = path.stat() - size = stat.st_size - mtime_ns = stat.st_mtime_ns - - # Read first N bytes - h = hashlib.sha256() - with path.open("rb") as f: - chunk = f.read(min(sample_bytes, size)) - h.update(chunk) - sample_hash = h.hexdigest() - - record = f"size:{size}|mtime_ns:{mtime_ns}|sample_sha256:{sample_hash}" - fingerprint = hashlib.sha256(record.encode("utf-8")).hexdigest() - - return { - "fingerprint": fingerprint, - "mode": "sample", - "params": { - "size": size, - "mtime_ns": mtime_ns, - "sample_bytes": len(chunk), - "sample_sha256": sample_hash - } - } -``` - -**Directories**: -```python -def fs_fingerprint_dir_sample(root, sample_bytes=200): - # Hash (relpath, size, mtime_ns, sha256(first N bytes)) for each file - h = hashlib.sha256() - for relpath, size, mtime_ns, content_hash in collect_files(root, sample_bytes): - record = f"{relpath}|{size}|{mtime_ns}|{content_hash}\n" - h.update(record.encode("utf-8")) - - return {"fingerprint": h.hexdigest(), "mode": "sample", ...} -``` - -**Properties**: -- **Balanced**: Metadata + partial content -- **Detects corruption**: Catches changes to file headers/structure -- **Moderate overhead**: Reads small amount from each file -- **Good for**: Production workflows with moderate file counts - -##### Mode 3: "strict" (Full Content Hash) - -**Files**: -```python -def fs_fingerprint_file_strict(path): - h = hashlib.sha256() - with path.open("rb") as f: - while True: - chunk = f.read(1024 * 1024) - if not chunk: - break - h.update(chunk) - - content_hash = h.hexdigest() - return { - "fingerprint": content_hash, # Fingerprint IS content hash - "mode": "strict", - "params": {"content_sha256": content_hash} - } -``` - -**Directories**: -```python -def fs_fingerprint_dir_strict(root): - # Hash (relpath, sha256(full content)) for each file - h = hashlib.sha256() - for relpath, _, _, content_hash in collect_files(root, full_content=True): - record = f"{relpath}|{content_hash}\n" - h.update(record.encode("utf-8")) - - return {"fingerprint": h.hexdigest(), "mode": "strict", ...} -``` - -**Properties**: -- **Cryptographic integrity**: Guaranteed detection of any change -- **High overhead**: Reads all content -- **Slow for large datasets**: Not suitable for TB-scale data -- **Good for**: Critical datasets, small files, verification-heavy workflows - -#### Fingerprinting Integration - -**CLI Flag**: -```bash -floability data --data-spec data/data.yml \ - --mode fetch \ - --data-cache-mode symlink \ - --fingerprint-mode meta # or "sample", "strict" -``` - -**During Cache Build**: -```python -def _build_cache_entry(item, cache_dir, backpack_root, fingerprint_mode, ...): - # ... download data ... - - # Compute source fingerprint (for fs/backpack only) - source_fingerprint = None - if item["source_type"] in ("fs", "backpack"): - from .fingerprint import compute_fingerprint - source_path = resolve_path(item["source"], backpack_root) - source_fingerprint = compute_fingerprint( - str(source_path), - mode=fingerprint_mode, # "meta", "sample", or "strict" - verbose=verbose - ) - - # Write to .meta.json - _write_cache_metadata( - cache_dir, artifact_spec, content_sha256, actual_size, - source_fingerprint # Includes fingerprint, mode, and params - ) -``` - -**During Cache Lookup**: -```python -def _lookup_cache_entry(cache_dir, artifact_spec, fingerprint_mode, backpack_root, ...): - # ... validate cache exists ... - - # Validate source fingerprint (for fs/backpack) - if artifact_spec["source_type"] in ("fs", "backpack"): - cached_fingerprint = meta.get("source_fingerprint") - if not cached_fingerprint: - return None # Old cache format - - # Recompute fingerprint from current source - from .fingerprint import compute_fingerprint - source_path = resolve_path(artifact_spec["source"], backpack_root) - current_fingerprint = compute_fingerprint( - str(source_path), - mode=fingerprint_mode, # SAME mode as when cached - verbose=False - ) - - if current_fingerprint["fingerprint"] != cached_fingerprint: - return None # Source changed, invalidate cache -``` - -#### Fingerprinting Performance Comparison - -Example: Directory with 1000 files, 10 GB total - -| Mode | Time | Overhead | Detects | -|------|------|----------|---------| -| **meta** | ~0.1s | stat() only | Size/mtime changes | -| **sample** (200 bytes) | ~1s | Read 200KB total | Header corruption, size/mtime | -| **strict** | ~30s | Read 10GB | Any byte change | - -**Recommendation**: -- **Development**: `meta` (fastest, good enough for local changes) -- **Production**: `sample` (balanced, catches most issues) -- **Critical/Small**: `strict` (full verification) - -### Directory vs Single File Handling - -Floability handles directories and files differently throughout the stack: - -#### Detection and Download - -**S3 Sources**: -```python -def _download_to_cache(item, cache_file, backpack_root, ...): - if stype == "s3": - # Three-tier detection - source_obj_type = item.get("source_object_type", "").lower() - - is_dir = False - if source_obj_type == "directory": - is_dir = True # Explicit type - elif source_obj_type == "file": - is_dir = False - else: - # Auto-detect: trailing slash or metadata check - is_dir = source.endswith('/') or is_s3_directory(source) - - if is_dir: - # Directory: recursive download preserving structure - s3_directory_download( - source, - dest_dir=str(cache_file), # cache_file is directory - overwrite=True, - show_progress=verbose - ) - else: - # Single file: download to parent directory - s3_file_download( - source, - dest_dir=str(cache_file.parent), - filename=cache_file.name, - overwrite=True - ) -``` - -**S3 Directory Detection** (`is_s3_directory`): -```python -def is_s3_directory(uri, anonymous=None): - """Check if S3 URI represents a directory (prefix with multiple objects).""" - # Parse s3://bucket/prefix/ - bucket, key = parse_s3_uri(uri) - - # List objects with prefix - s3 = boto3.client('s3') - response = s3.list_objects_v2(Bucket=bucket, Prefix=key, MaxKeys=2) - - # If multiple objects with this prefix, it's a directory - return response.get('KeyCount', 0) > 1 -``` - -**S3 Directory Download** (`s3_directory_download`): -```python -def s3_directory_download(uri, dest_dir, overwrite=False, show_progress=False): - """Recursively download all objects under S3 prefix.""" - bucket, prefix = parse_s3_uri(uri) - dest_path = Path(dest_dir) - dest_path.mkdir(parents=True, exist_ok=True) - - # List all objects with prefix - objects = s3_list_objects(uri, recursive=True) - - for obj in objects: - # Get relative path within prefix - rel_path = obj['Key'][len(prefix):].lstrip('/') - target_file = dest_path / rel_path - - # Download object - target_file.parent.mkdir(parents=True, exist_ok=True) - s3_file_download( - f"s3://{bucket}/{obj['Key']}", - dest_dir=str(target_file.parent), - filename=target_file.name, - overwrite=overwrite - ) - - if show_progress: - print(f"Downloaded {rel_path} ({obj['Size']} bytes)") -``` - -**Pelican Sources**: Similar three-tier detection with `is_pelican_directory` and `pelican_directory_download`. - -**Filesystem Sources**: -```python -elif stype in ("fs", "backpack"): - source_path = resolve_path(source, backpack_root) - - if source_path.is_file(): - shutil.copy2(source_path, cache_file) - else: # Directory - if cache_file.exists(): - shutil.rmtree(cache_file) # Remove existing - shutil.copytree(source_path, cache_file) -``` - -#### Cache Structure - -**Single File**: -``` -target_location: "data/model.h5" - -Cache structure: -/ - cached_data/ - data/ - model.h5 # Single file - .meta.json -``` - -**Directory**: -``` -target_location: "data/samples" - -Cache structure: -/ - cached_data/ - data/ - samples/ # Directory with full tree - file1.root - file2.root - subdir/ - file3.root - .meta.json -``` - -#### Content Hashing - -**Single File**: Direct SHA-256 of file content - -**Directory**: Merkle-like tree hash: -1. Recursively compute SHA-256 for each file -2. Build sorted list of `(relative_path, file_hash, file_size)` -3. Hash the concatenation: `SHA256(path1:hash1:size1\npath2:hash2:size2\n...)` - -**Benefits**: -- Detects structural changes (files added/removed/renamed) -- Detects content changes (any file modified) -- Deterministic (same tree → same hash) -- Efficient (doesn't re-read content during tree hash) - -#### Materialization - -Both files and directories use the same materialization logic (since cache structure preserves hierarchy): - -```python -def _materialize_from_cache(cache_dir, target_path, mode="symlink", ...): - cached_data_dir = cache_dir / "cached_data" - - # Get top-level items from cached_data/ - items = list(cached_data_dir.iterdir()) - - # Find workflow root by locating first item name in target_path - first_item_name = items[0].name # e.g., "data" - workflow_root = find_parent_of(target_path, first_item_name) - - # Materialize each item - for cached_item in items: - rel_path = cached_item.relative_to(cached_data_dir) - target_item = workflow_root / rel_path - - if mode == "symlink": - target_item.symlink_to(cached_item.resolve()) - elif mode == "hardlink": - if cached_item.is_file(): - os.link(cached_item, target_item) - else: - shutil.copytree(cached_item, target_item) # Fall back to copy - elif mode == "copy": - if cached_item.is_file(): - shutil.copy2(cached_item, target_item) - else: - shutil.copytree(cached_item, target_item) -``` - -**Key Insight**: Because cache structure mirrors target structure exactly, materialization doesn't need to distinguish between files and directories. The same top-level symlink works for both. - -### Cache Materialization - -Cache materialization is the process of making cached data available at target locations in the workflow directory. - -#### Three Materialization Modes - -##### Mode 1: symlink (Recommended Default) - -**Creation**: -```python -target_item.symlink_to(cached_item.resolve()) -``` - -**Properties**: -- **Zero copy**: No data duplication -- **Instant**: O(1) operation regardless of size -- **Read-only convention**: Workflow shouldn't modify data -- **Space efficient**: Multiple instances can share cache - -**Use Cases**: -- Read-only workflows (most scientific applications) -- Large datasets (TB-scale) -- Multiple concurrent instances - -**Limitations**: -- Requires symlink support (not all filesystems) -- Workflow must respect read-only convention - -##### Mode 2: hardlink - -**Creation**: -```python -if cached_item.is_file(): - os.link(cached_item, target_item) -else: - # Hardlinks don't work for directories, fall back to copy - shutil.copytree(cached_item, target_item) -``` - -**Properties**: -- **Shared inode**: File appears in both locations -- **Space efficient**: No data duplication (for files) -- **Independent**: Can be deleted independently -- **Modification affects both**: Changes visible through both links - -**Use Cases**: -- Workflows that need writable paths (in-place updates) -- When symlink support unavailable -- Filesystem-aware applications (detect hardlinks) - -**Limitations**: -- Only works for files on same filesystem -- Directories must be copied (no directory hardlinks) -- Modification affects cache (potentially dangerous) - -##### Mode 3: copy - -**Creation**: -```python -if cached_item.is_file(): - shutil.copy2(cached_item, target_item) # Preserves metadata -else: - shutil.copytree(cached_item, target_item) -``` - -**Properties**: -- **Full independence**: Target is separate copy -- **Writable**: Workflow can modify freely -- **Space overhead**: Doubles storage requirement -- **Slow**: O(size) operation - -**Use Cases**: -- Workflows that modify data in-place -- Testing (don't want to affect cache) -- When symlink/hardlink not supported - -**Limitations**: -- Doubles storage requirement -- Slow for large datasets (minutes to hours) -- No longer benefits from shared cache - -#### Materialization Algorithm (Feb 2026) - -The simplified materialization algorithm: - -```python -def _materialize_from_cache(cache_dir, target_path, mode, verbose): - cached_data_dir = cache_dir / "cached_data" - - # 1. Get all top-level items from cached_data/ - items = list(cached_data_dir.iterdir()) - # Example: ["data"] for cached_data/data/samples/file.root - - # 2. Find workflow root by locating first item name in target_path - first_item_name = items[0].name # "data" - workflow_root = target_path - - # Walk up until we find where "data" should be - while workflow_root.name != first_item_name and workflow_root.parent != workflow_root: - workflow_root = workflow_root.parent - - # If we found "data", go up one more to get the root - if workflow_root.name == first_item_name: - workflow_root = workflow_root.parent - - # 3. Materialize each top-level item - for cached_item in items: - rel_path = cached_item.relative_to(cached_data_dir) - target_item = workflow_root / rel_path - - # Ensure parent exists - target_item.parent.mkdir(parents=True, exist_ok=True) - - # Remove existing (stale symlinks, old data) - if target_item.exists() or target_item.is_symlink(): - if target_item.is_dir() and not target_item.is_symlink(): - shutil.rmtree(target_item) - else: - target_item.unlink() - - # Create link/copy based on mode - if mode == "symlink": - target_item.symlink_to(cached_item.resolve()) - elif mode == "hardlink": - # ... (see above) - elif mode == "copy": - # ... (see above) - - return True -``` - -**Example**: -``` -Cache: - cached_data/data/samples/test/file.root - -Target path: /instance/workflow/data/samples/test - -Step 1: Items = ["data"] -Step 2: First item = "data", walk up from target_path: - - /instance/workflow/data/samples/test - - /instance/workflow/data/samples - - /instance/workflow/data ← name matches "data" - - /instance/workflow ← go up one more, this is workflow_root -Step 3: Materialize: - - Symlink /instance/workflow/data → cached_data/data -Result: /instance/workflow/data/samples/test/file.root accessible via symlink -``` - -**Key Properties**: -- **Single top-level symlink**: Not per-file, but per-data item's top directory -- **Structure preserved**: Full nested hierarchy works automatically -- **Predictable**: Always creates symlink at the level where item name appears in target path -- **Handles multiple items**: If cache has multiple top-level items, each gets its own symlink - -#### Cache Cleanup - -**Manual Cleanup**: -```bash -# Remove all cache entries -rm -rf ~/floability-data-cache - -# Remove specific cache entry -rm -rf ~/floability-data-cache/ - -# Remove only data, keep metadata -rm -rf ~/floability-data-cache//cached_data -``` - -**Automated Cleanup** (Future): -- LRU eviction when cache size exceeds threshold -- Age-based pruning (remove entries older than N days) -- Reference counting (remove if no instances using) - -**Cache Size Estimation**: -```bash -# Total cache size -du -sh ~/floability-data-cache - -# Per-entry size -du -sh ~/floability-data-cache/* - -# Find largest cache entries -du -sh ~/floability-data-cache/* | sort -h | tail -10 -``` - ---- - -## Implementation Summary - -### Key Design Decisions - -1. **Declarative Specification**: Data requirements separate from notebook code -2. **Content-Addressable Caching**: Same source → same cache entry -3. **Profile-Based Configuration**: Environment-specific sources without code changes -4. **Multi-Source Fallback**: Resilient access across network topologies -5. **Three Operation Modes**: Check (metadata), Fetch (download), Verify (integrity) -6. **Fingerprinting**: Validate filesystem sources haven't changed -7. **Flexible Materialization**: Symlink (fast), hardlink (independent), copy (isolated) -8. **Directory Support**: First-class support for directory trees (S3, Pelican, fs) - -### Portability Mechanisms - -| Challenge | Solution | Benefit | -|-----------|----------|---------| -| Heterogeneous storage | Source type abstraction | Unified access to HTTP/S3/Pelican/fs | -| Site-specific paths | Profile-based configuration | Same backpack, different sources | -| Network topology | Multi-source fallback | Automatic adaptation | -| Data duplication | Content-addressable cache | Shared cache across instances | -| Large datasets | Symlink materialization | Zero-copy instantiation | -| Source changes | Fingerprinting | Detect stale cache | -| Integrity | Checksums + verification | Reproducibility guarantees | - -### Performance Characteristics - -| Operation | Time Complexity | I/O | Network | -|-----------|----------------|-----|---------| -| **Check** | O(items) | Metadata only | HEAD requests | -| **Fetch (cache hit)** | O(items) | Symlink creation | None | -| **Fetch (cache miss)** | O(total_size) | Full download | Full download | -| **Verify** | O(total_size) | Full read | None (if cached) | -| **Fingerprint (meta)** | O(files) | stat() only | None | -| **Fingerprint (sample)** | O(files × sample_bytes) | Partial read | None | -| **Fingerprint (strict)** | O(total_size) | Full read | None | - -### Implementation Files - -- **`floability/data/data_handler.py`**: Core orchestration (check/fetch/verify) -- **`floability/data/http_file_utils.py`**: HTTP download utilities -- **`floability/data/s3_file_utils.py`**: S3 file/directory operations -- **`floability/data/pelican_file_utils.py`**: Pelican/OSDF file/directory operations -- **`floability/data/fs_file_utils.py`**: Filesystem utilities -- **`floability/data/fingerprint.py`**: Source fingerprinting (meta/sample/strict) -- **`floability/ops/data.py`**: CLI operation handlers - ---- - -## For Paper Writing - -### Suggested Structure for Section 3 (Declarative Data Spec) - -1. **Introduction**: Motivation for declarative specs (portability, separation of concerns) -2. **YAML Structure**: Brief overview with example -3. **Portability Features**: - - Profile-based configuration (figure: same backpack, 3 different profiles) - - Source type abstraction (table of supported types) - - Multi-source fallback (example with 4 fallback sources) - - Target location consistency (code snippet showing notebook independence) -4. **Real-World Example**: One of your applications with 2-3 profiles - -### Suggested Structure for Section 4 (Implementation) - -1. **Three Operations**: - - Check: Metadata-only pre-flight - - Fetch: Download with caching - - Verify: Integrity validation - - Table comparing when to use each -2. **Local Data Cache** (Big subsection): - - Architecture: Content-addressable, lock-based concurrency - - Cache key computation: Artifact spec, deterministic hashing - - Cache structure: Diagram showing cache_key/cached_data/.meta.json - - Building vs. lookup: Flow diagram -3. **Source Fingerprinting**: - - Problem: Filesystem sources can change - - Solution: Three modes (table comparing meta/sample/strict) - - Integration: When computed, when validated -4. **Directory Handling**: - - Detection: Three-tier priority - - Download: Recursive with structure preservation - - Hashing: Merkle-like tree hash - - Materialization: Single top-level symlink -5. **Performance**: Table showing I/O characteristics - -### Figures/Tables to Include - -1. **Figure: Profile-based portability** - Same backpack with local/S3/Pelican profiles -2. **Table: Source types** - Type, protocol, use case, example -3. **Figure: Cache architecture** - Directory structure with annotations -4. **Table: Fingerprinting modes** - Mode, time, overhead, detects -5. **Figure: Materialization flow** - Cache → workflow symlink diagram -6. **Table: Operation comparison** - Check/fetch/verify characteristics - -### Metrics to Highlight - -- **Cache hit rate**: X% in your experiments across 3 sites -- **Materialization time**: <1s vs. Xmin for copy (for Y GB dataset) -- **Storage savings**: N instances sharing 1 cache vs. N copies -- **Network efficiency**: Multi-source fallback used Z% of time at Site B - ---- - -*Document generated: February 8, 2026* -*For internal use in paper writing - not for direct inclusion* diff --git a/implementation-docs/DIRECTORY_DOWNLOAD_DESIGN.md b/implementation-docs/DIRECTORY_DOWNLOAD_DESIGN.md deleted file mode 100644 index aafb8f6..0000000 --- a/implementation-docs/DIRECTORY_DOWNLOAD_DESIGN.md +++ /dev/null @@ -1,192 +0,0 @@ -# Directory Download Support - Design Document - -## Problem Statement - -Currently, the data handler treats all sources as individual files. When a user specifies a directory URL (e.g., `pelican://server/path/to/dir/`), the system: -1. Tries to download it as a single file -2. Creates a file named after the target_path instead of a directory with contents - -**Example problematic spec:** -```yaml -- name: data_dir - source_type: pelican - source: pelican://disc-head-002.crc.nd.edu:443/nd/disc2/apps/floability/examples/cms-physics-dv5/data/ - target_path: data -``` - -**Current behavior:** Creates a single file `data` -**Expected behavior:** Creates directory `data/` with all files from remote directory recursively - -## Root Causes - -1. **No directory detection**: System doesn't check if source is file or directory -2. **No recursive download function**: `pelican_file_download()` only handles single files -3. **No directory listing function**: No way to recursively list Pelican directory contents -4. **No spec field to indicate directory**: No `is_directory` or similar field - -## Proposed Solution - -### 1. Add Directory Detection - -**Option A: Explicit field** (Recommended for clarity) -- Add optional `source_object_type` field with values: `"file"`, `"directory"` -- If specified, skips auto-detection -- Pros: Explicit, no network call, clear intent, works for all source types -- Cons: Requires user to specify - -**Option B: URL-based heuristic** (Simple, fast) -- If URL ends with `/`, treat as directory -- Pros: No network call needed, clear intent -- Cons: User must remember trailing slash - -**Option C: Metadata-based detection** (More robust) -- Call `fs.info(path)` and check `type` field -- If `type == 'directory'`, it's a directory -- Pros: Works regardless of URL format -- Cons: Extra network call - -**Recommendation: Use all three with priority order** -1. Check `source_object_type` field first (if present) -2. Check for trailing `/` in URL -3. Call `is_pelican_directory()` as fallback (metadata check) - -### 2. Add Pelican Directory Listing Function - -```python -def pelican_list_directory(url: str, recursive: bool = True) -> List[Dict[str, Any]]: - """ - List all files in a Pelican directory. - - Args: - url: Pelican directory URL (e.g., pelican://server/path/to/dir/) - recursive: If True, list all files recursively - - Returns: - List of dicts with keys: path, size, type, name - """ -``` - -Implementation using PelicanFileSystem: -- `fs.ls(path, detail=True)` - list directory with details -- `fs.walk(path)` - recursive traversal -- Filter out directories, return only files - -### 3. Add Pelican Directory Download Function - -```python -def pelican_directory_download( - url: str, - dest_dir: str = ".", - *, - overwrite: bool = False, - show_progress: bool = True, -) -> Path: - """ - Download all files from a Pelican directory recursively. - - Args: - url: Pelican directory URL - dest_dir: Local destination directory - overwrite: Overwrite existing files - show_progress: Show progress bar - - Returns: - Path to destination directory - """ -``` - -Implementation: -1. List all files recursively using `pelican_list_directory()` -2. For each file, download using `pelican_file_download()` -3. Preserve directory structure in dest_dir -4. Optional: aggregate progress across all files - -### 4. Update Data Handler Logic - -In `_fetch_source_to_target()`: - -```python -if stype == "pelican": - # Check if source is a directory - if src.endswith('/') or _is_pelican_directory(src): - # Directory download - pelican_directory_download( - src, - dest_dir=str(target_path), - overwrite=force, - ) - else: - # Single file download - pelican_file_download( - src, - dest_dir=str(target_path.parent), - filename=target_path.name, - overwrite=force, - ) - return True -``` - -### 5. Add Spec Field for Explicit Control - -Add optional field to data item: -```yaml -- name: data_dir - source_type: pelican - source: pelican://server/path/to/dir/ - target_path: data - source_object_type: directory # Optional: "file" or "directory" -``` - -**Benefits:** -- Explicit control over file vs directory handling -- Avoids auto-detection overhead (no network metadata call) -- Works for ambiguous URLs (no trailing slash needed) -- Self-documenting spec - -**Implementation:** -- Priority order: `source_object_type` → URL trailing `/` → metadata check -- Valid values: `"file"`, `"directory"`, or omitted for auto-detect -- Applied to both direct fetch and cache build paths - -### 6. Extend to Other Source Types - -Apply same pattern to: -- **S3**: Add `s3_directory_download()` using `s3_list_objects(recursive=True)` -- **HTTP**: Limited support (requires directory listing endpoint) -- **FS/Backpack**: Already supported via `shutil.copytree()` - -## Implementation Plan - -### Phase 1: Pelican Directory Support (This PR) -1. ✅ Add `pelican_list_directory()` to `pelican_file_utils.py` -2. ✅ Add `pelican_directory_download()` to `pelican_file_utils.py` -3. ✅ Update `_fetch_source_to_target()` to detect and handle directories -4. ✅ Add tests for directory operations -5. ✅ Update documentation - -### Phase 2: S3 Directory Support (Future) -1. Add `s3_directory_download()` using existing `s3_list_objects()` -2. Update data handler to handle S3 directories -3. Add tests - -### Phase 3: Metadata & Verification (Future) -1. Support `expected_size` for directories (sum of all files) -2. Support checksums for directories (manifest-based) -3. Cache directory downloads as units - -## Testing Strategy - -1. **Unit tests**: Test directory listing and download functions -2. **Integration tests**: Test with real Pelican server -3. **Edge cases**: - - Empty directories - - Nested directories - - Large directories (many files) - - Partial downloads/resume - - Permission errors - -## Migration Path - -- **Backward compatible**: Existing single-file specs work unchanged -- **Opt-in**: Users add trailing `/` to enable directory mode -- **No breaking changes**: All existing functionality preserved diff --git a/implementation-docs/FINGERPRINT_IMPLEMENTATION_SUMMARY.md b/implementation-docs/FINGERPRINT_IMPLEMENTATION_SUMMARY.md deleted file mode 100644 index b6d375e..0000000 --- a/implementation-docs/FINGERPRINT_IMPLEMENTATION_SUMMARY.md +++ /dev/null @@ -1,236 +0,0 @@ -# Filesystem Fingerprinting Implementation - Summary - -## Completed Implementation - -### Core Fingerprinting Module -✅ **File**: `floability/data/fingerprint.py` -- Implements three fingerprint modes for filesystem sources: - - `meta`: Fast, uses file size + mtime (metadata only) - - `sample`: Medium speed, uses size + mtime + SHA256 of first 200 bytes - - `strict`: Slow but thorough, full content SHA256 hash -- Supports both files and directories -- Directory fingerprinting: recursive walk with sorted paths for determinism -- Handles symlinks (skips them to avoid issues) -- Future extension hooks for HTTP, S3, Pelican sources - -### Cache Integration -✅ **File**: `floability/data/data_handler.py` -- Extended `.meta.json` schema with: - - `source_fingerprint`: Hex digest of source fingerprint - - `fingerprint_mode`: Mode used (meta/sample/strict) - - `fingerprint_params`: Parameters (size, mtime, sample_bytes, etc.) -- Updated `_write_cache_metadata()`: Stores fingerprint data -- Updated `_lookup_cache_entry()`: Validates fingerprints on cache reuse -- Updated `_build_cache_entry()`: Computes fingerprints during cache creation -- Updated `_fetch_single_item()`: Passes fingerprint_mode through call chain - -### CLI Integration -✅ **Files**: `floability/cli.py`, `floability/ops/data.py`, `floability/ops/run.py`, `floability/ops/instance.py` - -Added `--fingerprint-mode` flag to: -- ✅ `floability data` command (check, fetch, verify modes) -- ✅ `floability run` command -- ✅ `floability execute` command -- ✅ `floability instance create` command - -All commands support: -- `--fingerprint-mode meta` (default - fast, metadata only) -- `--fingerprint-mode sample` (first N bytes + metadata) -- `--fingerprint-mode strict` (full content hash) -- Works alongside `--data-cache-mode` (off/symlink/hardlink/copy) - -### Key Features - -#### 1. Cache Key Unchanged -- Existing cache-key generation untouched (based on artifact spec hash) -- Fingerprint is for validation only, not cache key computation -- Multiple runs with same spec use same cache directory - -#### 2. Automatic Invalidation -- **Meta mode**: Invalidates on mtime or size change -- **Sample mode**: Invalidates on header changes or metadata changes -- **Strict mode**: Invalidates on any content change -- Directory changes: Detects file add/remove/rename/modify - -#### 3. Intelligent Warnings -- Large directories with strict mode: warns about performance -- Many files with sample mode: warns about processing time -- Missing source fingerprint: indicates legacy cache format - -#### 4. Backward Compatibility -- Old cache entries without fingerprints gracefully invalidated -- Logs show "no source fingerprint (old cache format)" -- Cache rebuilt with new fingerprint metadata - -#### 5. Filesystem-Only Implementation -- Currently implements `fs` and `backpack` source types -- HTTP, Pelican, S3 sources have TODO stub functions -- Easy to extend for additional source types - -### Usage Examples - -#### Data Command -```bash -# Fetch with meta mode (default) -floability data --mode fetch --data-spec data/data.yml --backpack . \ - --data-cache-mode symlink --fingerprint-mode meta --verbose - -# Verify with sample mode -floability data --mode verify --data-spec data/data.yml --backpack . \ - --data-cache-mode symlink --fingerprint-mode sample --verbose - -# Strict mode for critical data -floability data --mode fetch --data-spec data/data.yml --backpack . \ - --data-cache-mode copy --fingerprint-mode strict --verbose -``` - -#### Run Command -```bash -# Run backpack with caching and fingerprinting -floability run --backpack example/matrix-multiplication \ - --data-cache-mode symlink --fingerprint-mode meta --verbose - -# Disable caching entirely -floability run --backpack example/matrix-multiplication \ - --data-cache-mode off -``` - -#### Instance Create -```bash -# Create instance with sample mode -floability instance create --backpack example/matrix-multiplication \ - --name my-instance --data-cache-mode symlink \ - --fingerprint-mode sample --verbose -``` - -### Testing Strategy - -✅ **File**: `TEST_FINGERPRINTING.md` -- 14 comprehensive test scenarios -- Covers all fingerprint modes -- Tests cache reuse and invalidation -- Directory operations (add/remove/rename) -- Integration with run/execute/instance commands -- Performance comparisons -- Legacy cache migration -- Debugging tips and validation checklist - -### Performance Characteristics - -**Meta Mode** (Default) -- Speed: Fastest (milliseconds) -- Use case: Development, frequent iteration -- Detects: File modifications (mtime/size changes) -- Overhead: Minimal (stat calls only) - -**Sample Mode** -- Speed: Medium (seconds for many files) -- Use case: Balance between speed and safety -- Detects: Header changes, metadata changes -- Overhead: Reads first 200 bytes per file - -**Strict Mode** -- Speed: Slowest (minutes for large directories) -- Use case: Production, critical data validation -- Detects: Any content changes anywhere in files -- Overhead: Full file reads, complete hash computation - -### What's NOT Included (Future Work) - -❌ HTTP source fingerprinting (ETag, Last-Modified headers) -❌ S3 source fingerprinting (ETag, metadata) -❌ Pelican source fingerprinting -❌ Post-processing support (unzip, untar) -❌ Configurable sample size per data item -❌ Parallel fingerprinting for large directories - -### Files Modified - -1. **New**: `floability/data/fingerprint.py` (555 lines) -2. **Modified**: `floability/data/data_handler.py` (+150 lines) -3. **Modified**: `floability/cli.py` (+15 lines) -4. **Modified**: `floability/ops/data.py` (+3 lines) -5. **Modified**: `floability/ops/run.py` (+1 line) -6. **Modified**: `floability/ops/instance.py` (+1 line) -7. **New**: `TEST_FINGERPRINTING.md` (test strategy) -8. **Existing**: `FLOABILITY_DATA_OPERATIONS_SUMMARY.md` (documentation) - -### Design Principles Followed - -✅ No changes to existing cache-key generation -✅ Source-type specific implementation (fs/backpack only) -✅ Reusable and extensible design -✅ Clear separation: cache key vs validation -✅ Graceful degradation (old caches work) -✅ Comprehensive logging -✅ User-configurable modes -✅ Backward compatible - -### Next Steps (Future PRs) - -1. **HTTP Fingerprinting** - - Use ETag header for validation - - HEAD request for metadata - - Partial GET for sample mode - -2. **Pelican Fingerprinting** - - Pelican-specific APIs - - Fall back to HTTP methods - -3. **S3 Fingerprinting** - - Use S3 ETag and metadata - - head_object for meta mode - - get_object with Range for sample - -4. **Performance Optimizations** - - Parallel fingerprinting - - Incremental directory hashing - - Fingerprint caching - -5. **Configuration** - - Per-item fingerprint mode in data.yml - - Configurable sample size - - Fingerprint cache TTL - ---- - -## Testing Instructions - -See `TEST_FINGERPRINTING.md` for complete testing strategy. - -Quick smoke test: -```bash -# Setup test backpack -mkdir -p test-backpack/data -echo "test content" > test-backpack/data/file.txt - -# Create data spec -cat > test-backpack/data/data.yml << 'EOF' -schema_version: 1.0 -default_profile: test -data_profiles: - test: - data: - - name: test_file - source_type: backpack - source: data/file.txt - target_location: data/file.txt -EOF - -# Test with fingerprinting -floability data --mode fetch --data-spec test-backpack/data/data.yml \ - --backpack test-backpack --data-cache-mode symlink \ - --fingerprint-mode meta --verbose - -# Check cache metadata -find flo_data_cache -name ".meta.json" -exec cat {} \; | jq . - -# Cleanup -rm -rf test-backpack flo_data_cache -``` - ---- - -**Status**: ✅ Complete and ready for testing -**Branch**: dev/data-handling -**Target**: Filesystem sources only (HTTP/S3/Pelican in future PRs) diff --git a/implementation-docs/FLOABILITY_DATA_OPERATIONS_SUMMARY.md b/implementation-docs/FLOABILITY_DATA_OPERATIONS_SUMMARY.md deleted file mode 100644 index 31b3096..0000000 --- a/implementation-docs/FLOABILITY_DATA_OPERATIONS_SUMMARY.md +++ /dev/null @@ -1,540 +0,0 @@ -# Floability Data Operations & Caching Summary - -## Overview -Floability manages data dependencies for distributed workflows through a declarative YAML specification system with built-in content-addressable caching. Data can be sourced from multiple locations (HTTP, S3, Pelican/OSDF, local filesystem, backpack-relative paths) and materialized into workflow instances with integrity verification. - -**Latest Updates (Feb 2026)**: -- **S3 Directory Support**: Full support for downloading S3 directories recursively -- **Pelican Directory Support**: Download entire directories from Pelican/OSDF federations -- **New Cache Structure**: Simplified `cached_data/` structure that mirrors target paths -- **Improved Materialization**: Simplified symlink logic for predictable behavior -- **Directory Detection**: Auto-detect directories via trailing slash or metadata - -## Core Concepts - -### 1. Data Specification (data.yml) -- **Location**: Typically at `/data/data.yml` -- **Structure**: YAML file containing: - - `schema_version`: Version identifier (e.g., "1.0") - - `default_profile`: Name of default profile to use - - `data_profiles`: Map of profile names to data profiles - -### 2. Data Profiles -Each profile contains: -- **`policy`** (optional): Runtime behavior configuration - - `retry_attempts`: Number of retry attempts for failed downloads - - `timeout`: Timeout in seconds for download operations - - `size_tolerance_bytes`: Allowed deviation from expected size - - `run_operation`: Default operation ("check", "fetch", "verify") - - `verification_type`: "strict" (requires checksum) or "size_only" - -- **`data`** (required): List of data items to manage - -### 3. Data Item Schema -Each item in the `data` list can specify: -- **`name`**: Optional identifier for the item -- **`source_type`**: Type of source ("http", "s3", "pelican", "osdf", "fs", "backpack", or "multi") -- **`source`**: Source URI/path (for single source) -- **`sources`**: List of fallback sources (for multi-source items) -- **`source_object_type`**: Optional explicit type ("file" or "directory") - NEW -- **`target_location`** or **`target_path`**: Where to materialize the data (relative to workflow directory) -- **`target_prefix`**: Optional override for target directory prefix -- **`expected_size`**: Expected file size in bytes (for validation) -- **`checksum`**: Expected checksum in format "algorithm:hex" (e.g., "sha256:abc123...") -- **`content_type`**: Optional MIME type -- **`post_process`**: Optional post-processing instructions (not yet implemented) - -### 4. Source Types - -#### Single File/Directory Sources -- **`http`**: Download from HTTP/HTTPS URL -- **`s3`**: Download from S3 bucket (supports files and directories) -- **`pelican`/`osdf`**: Download from Pelican/OSDF federation (supports files and directories) -- **`fs`**: Local filesystem path (absolute or relative to backpack) -- **`backpack`**: Relative to backpack root directory (e.g., "backpack://data/file.csv") - -#### Multi-Source Fallback -- **`multi`**: Multiple fallback sources (tries each until one succeeds) - -#### Directory Detection (S3 and Pelican) -Three-tier priority for detecting directories: -1. **Explicit field**: `source_object_type: "directory"` or `source_object_type: "file"` -2. **URL trailing slash**: `s3://bucket/prefix/` or `pelican://server/path/` → directory -3. **Metadata check**: Query S3/Pelican to determine if source is a directory - -## Data Operations - -### 1. **Check** (Metadata-Only) -- **Command**: `floability data --mode check` -- **Purpose**: Verify data sources exist and match expected metadata without downloading -- **Actions**: - - Queries remote sources for metadata (size, existence) - - Validates expected_size within tolerance - - Reports cache status if caching enabled - - No file downloads or writes -- **Returns**: Success if all items exist and match expected size - -### 2. **Fetch** (Download/Copy) -- **Command**: `floability data --mode fetch` -- **Purpose**: Download/copy data to target locations -- **Actions**: - - Downloads data from sources to cache (if caching enabled) - - Materializes data to target locations using specified cache mode - - For multi-source items, tries each source until success - - Skips existing targets unless `--force` specified -- **Returns**: Success if all items fetched successfully - -### 3. **Verify** (Fetch + Integrity Check) -- **Command**: `floability data --mode verify` -- **Purpose**: Ensure data exists and passes integrity checks -- **Actions**: - - Performs fetch if target doesn't exist - - Validates checksums (if specified and verification_type="strict") - - Validates file sizes within tolerance - - Produces detailed verification report -- **Returns**: Success if all items exist and pass integrity checks - -## Caching System - -### Cache Architecture (Updated Feb 2026) - -#### Cache Location -- **Base directory**: Specified via `--data-cache-dir` (defaults to `~/floability-data-cache`) -- **Cache root**: `/` -- **Per-artifact**: `//` - -#### Cache Entry Structure (NEW) -``` -/ - / # Deterministic hash of artifact spec (SHA-256) - cached_data/ # NEW: Stores data with full target_location path - / # e.g., data/samples/file.root - - .meta.json # Metadata (artifact spec, SHA-256, size, timestamps) - .verify.lock # Temporary lock during build/verify (prevents concurrent writes) -``` - -**Key Change**: The cache now stores data under `cached_data/` instead of `cached_data/data/`. This preserves the full directory structure and enables predictable materialization. - -**Example**: -- **Spec**: `target_location: "data/samples/test"` -- **Cache**: `/cached_data/data/samples/test/file.root` -- **Workflow**: `workflow/data/samples/test/file.root` (symlinked to cache) - -#### Cache Key Computation -- Deterministic hash computed from "artifact spec" including: - - Source(s) with resolved absolute paths for local sources - - Expected size (if specified) - - Checksum (if specified) - - Content type (if specified) - - Post-process settings (if specified) -- Multi-source items include all sources in order -- Cache key is SHA-256 hex digest (64 chars) of normalized artifact spec JSON - -### Materialization Modes -### Materialization Logic (Updated Feb 2026) - -The new materialization approach is simpler and more predictable: - -1. **Read top-level items** from `cached_data/` (e.g., `data/`) -2. **Find workflow root** by locating where the first item name appears in `target_path` -3. **Create symlinks** from `workflow_root/{item_name}` to `cached_data/{item_name}` - -**Example**: -- Cache: `cached_data/data/samples/test/file.root` -- Target path: `/instance/workflow/data/samples/test` -- First item: `data` -- Workflow root: `/instance/workflow` (parent of `data` in target path) -- Symlink: `/instance/workflow/data` → `cached_data/data` - -This ensures the entire directory structure is preserved with a single symlink at the top level. - -### Cache Operations Flow - -#### Building Cache Entry (First Access) -1. Compute cache key from artifact spec -2. Check if cache entry exists and is valid -3. If not valid or `--force-data-cache`: - - Acquire `.verify.lock` (timeout: 300s) - - Download/copy data to `/cached_data//` - - For S3/Pelican directories: recursively download all files - - Apply post_process if specified (not yet implemented) - - Compute content SHA-256 hash and size - - Write `.meta.json` with metadata - - Release lock -4. Materialize from cache to workflow using specified mode - -#### Using Existing Cache Entry -1. Compute cache key -2. Lookup cache entry: - - Check cache directory exists - - Check `.meta.json` exists and is valid - - Verify artifact spec matches - - Verify size matches (if expected_size specified) - - Check `cached_data/` directory exists -3. If valid: materialize from cache to target -4. If invalid: rebuild cache entry - -#### Concurrent Access Protection -- `.verify.lock` file prevents duplicate work during concurrent runs -- Lock timeout: 300 seconds -- Falls back to direct fetch if lock timeout occurs - -### Cache Metadata (.meta.json) -Contains: -- `artifact_spec`: Normalized spec used to compute cache key -- `content_sha256`: SHA-256 hash of cached content -- `actual_size`: Actual size in bytes -- `created_at_iso`: ISO timestamp of cache entry creation -- `source_fingerprint`: Optional fingerprint for filesystem sources -- For directories: composite hash of all files with sorted paths - -### CLI Flags for Caching - -- **`--data-cache-mode `**: Set materialization mode (off|symlink|hardlink|copy) -- **`--force-data-cache`**: Force rebuild of cache entries even if valid -- **`--data-cache-dir `**: Set cache directory (default: `~/floability-data-cache`) -- **`--data-profile `**: Select which data profile to use - -## Integration with Workflow Execution - -### `floability run` and `floability instance create` -- Automatically pass `--data-cache-dir`, `--data-cache-mode`, and `--force-data-cache` to data operations -- Execute data operation phase when `--data-spec` provided -- Data materialized into instance workflow directory - -### Target Path Resolution -- **Default**: `/workflow/` -- **Override**: Use `target_prefix` in item or `--target-root` CLI flag -- Relative `target_location` paths resolved against target prefix - -## Example Data Specifications - -### Example 1: HTTP Sources with Checksums - -```yaml -schema_version: 1.0 -default_profile: gutenberg_data - -data_profiles: - gutenberg_data: - policy: - retry_attempts: 3 - timeout: 60 - size_tolerance_bytes: 1024 - run_operation: fetch - verification_type: strict - - data: - - name: gatsby - source_type: http - source: https://www.gutenberg.org/cache/epub/64317/pg64317.txt - content_type: text/plain - expected_size: 306594 - checksum: sha256:e6b7897aa8498b8dac4df0664827f857bc01135c3d9311adb820979bbc44b763 - target_location: data/pg64317.txt - - - name: frankenstein - source_type: http - source: https://www.gutenberg.org/files/84/84-0.txt - content_type: text/plain - expected_size: 421633 - checksum: sha256:06c37d2c52d208d3d81eb12c3b10b5edbd7728b73554325ddceadbe2fb427e77 - target_location: data/frankenstein.txt -``` - -### Example 2: S3 Directory Download - -```yaml -schema_version: 1.0 -default_profile: s3_data - -data_profiles: - s3_data: - policy: - retry_attempts: 0 - timeout: 30 - size_tolerance_bytes: 10 - run_operation: fetch - verification_type: size_only - - data: - # Explicit directory type - - name: sample_data - source_type: s3 - source: s3://floability/reyer_data/ - source_object_type: directory - target_location: data/samples - - # Auto-detect via trailing slash - - name: training_data - source_type: s3 - source: s3://mybucket/datasets/training/ - target_location: data/training - - # Single file - - name: model_weights - source_type: s3 - source: s3://mybucket/models/weights.h5 - source_object_type: file - target_location: data/model.h5 -``` - -### Example 3: Pelican/OSDF Directory Download - -```yaml -schema_version: 1.0 -default_profile: pelican_data - -data_profiles: - pelican_data: - policy: - retry_attempts: 2 - timeout: 60 - run_operation: fetch - verification_type: size_only - - data: - # Pelican directory with explicit type - - name: cms_data - source_type: pelican - source: pelican://osg-htc.org:8443/ospool/uc-shared/public/OSG-Staff/validation/test-data/ - source_object_type: directory - target_location: data/cms - - # Auto-detect via trailing slash - - name: physics_samples - source_type: osdf - source: osdf://ospool/datasets/physics/samples/ - target_location: data/physics_samples -``` - -### Example 4: Multi-Source Fallback - -```yaml -schema_version: 1.0 -default_profile: resilient_data - -data_profiles: - resilient_data: - policy: - retry_attempts: 2 - timeout: 60 - verification_type: strict - - data: - - name: dataset - source_type: multi - sources: - # Try Pelican first - - source_type: pelican - source: pelican://server.example.org:443/datasets/data.csv - # Fallback to S3 - - source_type: s3 - source: s3://backup-bucket/datasets/data.csv - # Final fallback to HTTP - - source_type: http - source: https://backup.example.org/datasets/data.csv - # Last resort: local backpack copy - - source_type: backpack - source: data/fallback/data.csv - expected_size: 1048576 - checksum: sha256:abc123def456... - target_location: data/dataset.csv -``` - -### Example 5: Mixed Files and Directories - -```yaml -schema_version: 1.0 -default_profile: mixed_data - -data_profiles: - mixed_data: - policy: - retry_attempts: 0 - timeout: 30 - run_operation: fetch - verification_type: size_only - - data: - # Directory from S3 - - name: s3_samples - source_type: s3 - source: s3://floability/dv5-sample-data/ - source_object_type: directory - target_location: data/s3_samples - - # Directory from Pelican - - name: pelican_data - source_type: pelican - source: pelican://osg-htc.org:8443/ospool/data/ - source_object_type: directory - target_location: data/pelican - - # Single file from HTTP - - name: config - source_type: http - source: https://example.org/config.json - target_location: config/app.json - - # Local file - - name: readme - source_type: backpack - source: README.md - target_location: docs/README.md -``` - -## Best Practices - -## Best Practices - -1. **Use symlink mode** for read-only workflows (default, most efficient) -2. **Use copy mode** if workflow modifies files in-place -3. **Always specify checksums** for production data (enables strict verification) -4. **Use multi-source fallbacks** for reliability across environments -5. **Set appropriate size_tolerance_bytes** to account for metadata variations -6. **Share cache across runs** by using consistent `--data-cache-dir` -7. **Use profiles** to switch between local/remote/development/production sources -8. **Use trailing slashes** for directories (`s3://bucket/dir/`) to avoid ambiguity -9. **Specify source_object_type** explicitly when auto-detection might be unclear -10. **Use descriptive names** for data items to improve logging clarity - -## Directory Download Specifics - -### S3 Directories -- **Detection**: Trailing `/`, explicit `source_object_type: directory`, or metadata check -- **Features**: - - Recursive download preserving structure - - Resume support for interrupted downloads - - Progress bars for each file - - Anonymous access via `AWS_NO_SIGN_REQUEST=true` -- **Example**: `s3://floability/dv5-sample-data/` downloads all objects under that prefix - -### Pelican/OSDF Directories -- **Detection**: Trailing `/`, explicit `source_object_type: directory`, or metadata check via `fs.walk()` -- **Features**: - - Recursive download using PelicanFileSystem - - Structure preservation - - Progress tracking - - SSL bypass mode for testing (DISABLE_SSL=True) -- **Example**: `pelican://osg-htc.org:8443/ospool/data/` downloads all files recursively - -### Cache Structure for Directories -- Directories are cached with full path: `cached_data//` -- Example: `target_location: "data/samples"` → `cached_data/data/samples/file1.root` -- Materialization creates symlink at top level: `workflow/data` → `cached_data/data` - -## Example Workflow Commands - -```bash -# Check data availability (no download) -floability data --data-spec data/data.yml --mode check --verbose - -# Fetch data with caching (symlink mode) -floability data --data-spec data/data.yml \ - --mode fetch \ - --data-cache-mode symlink \ - --data-cache-dir ~/floability-data-cache \ - --verbose - -# Fetch specific profile -floability data --data-spec data/data.yml \ - --data-profile s3_data \ - --mode fetch \ - --data-cache-mode symlink \ - --verbose - -# Force cache rebuild -floability data --data-spec data/data.yml \ - --mode fetch \ - --data-cache-mode symlink \ - --force-data-cache \ - --verbose - -# Run workflow with automatic data fetch -floability run --backpack example/cms-physics-lfv \ - --data-spec data/data.yml \ - --data-profile s3_data \ - --data-cache-mode symlink \ - --data-cache-dir ~/floability-data-cache -``` - -## Testing Directory Downloads - -### S3 Test Script -```bash -conda activate floability-env -python scripts/test-s3-dir-download.py -``` - -Tests: -1. S3 directory detection -2. Object listing -3. Directory download -4. Caching with data handler - -### Pelican Test Script -```bash -conda activate floability-env -python scripts/test-pelican-dir-download.py -``` - -## Troubleshooting - -### Common Issues - -**Issue**: Double directory nesting (e.g., `workflow/data/samples/samples/`) -- **Cause**: Old cache structure or materialization logic -- **Fix**: Clear cache and re-download with latest code - -**Issue**: S3 anonymous access fails -- **Solution**: Set environment variable `export AWS_NO_SIGN_REQUEST=true` - -**Issue**: Pelican SSL errors -- **Solution**: For testing only, set `DISABLE_SSL=True` in code - -**Issue**: Cache depth calculation incorrect -- **Cause**: Multiple items from different operations in same cache -- **Fix**: Latest code finds leaf directories correctly by examining target_path structure - -**Issue**: Permission denied when creating symlinks -- **Solution**: Use `--data-cache-mode copy` instead of symlink - -## Implementation Files - -Key source files: -- `floability/data/data_handler.py`: Core data operations and caching logic -- `floability/data/http_file_utils.py`: HTTP download utilities -- `floability/data/s3_file_utils.py`: S3 file and directory operations -- `floability/data/pelican_file_utils.py`: Pelican/OSDF file and directory operations -- `floability/data/fs_file_utils.py`: Filesystem utilities -- `floability/ops/data.py`: CLI operation handlers -- `docs/concept/data-caching.md`: User-facing caching documentation -- `docs/reference/data.md`: Complete data specification reference - -### New Functions (Feb 2026) -- `s3_directory_download()`: Download entire S3 directories -- `is_s3_directory()`: Detect if S3 URI is a directory -- `s3_list_objects()`: List all objects in S3 prefix -- `pelican_directory_download()`: Download entire Pelican directories -- `is_pelican_directory()`: Detect if Pelican URI is a directory -- `pelican_list_directory()`: List all files in Pelican directory - -## Key Implementation Details - -- **Content addressing**: Cache keys ensure same data specs share cache entries -- **Atomic cache builds**: Lock mechanism prevents race conditions -- **Multi-source failover**: Automatic fallback to alternative sources -- **Directory support**: Handles both files and directory trees for S3 and Pelican -- **Integrity verification**: SHA-256 checksums for strict validation -- **Flexible materialization**: Choose between symlink/hardlink/copy based on workflow needs -- **Idempotent operations**: Safe to re-run fetch/verify operations -- **Concurrent execution safe**: Lock files prevent cache corruption -- **Predictable structure**: Cached data mirrors target_location exactly -- **Simple materialization**: Top-level symlinks preserve entire directory structure - ---- - -*Last Updated: February 7, 2026* -*For sharing with LLMs or team members to understand Floability's data management system* diff --git a/implementation-docs/QUICKSTART_FINGERPRINTING.md b/implementation-docs/QUICKSTART_FINGERPRINTING.md deleted file mode 100644 index c53b604..0000000 --- a/implementation-docs/QUICKSTART_FINGERPRINTING.md +++ /dev/null @@ -1,228 +0,0 @@ -# Quick Start: Filesystem Fingerprinting in Floability - -## What is it? - -Filesystem fingerprinting validates cached data by checking if source files have changed. This ensures your cached data stays fresh while avoiding unnecessary re-downloads when sources haven't changed. - -## Three Modes - -| Mode | Speed | Detects | Best For | -|------|-------|---------|----------| -| `meta` | ⚡ Fastest | File size/time changes | Development, frequent iterations | -| `sample` | ⚡⚡ Medium | Header + metadata changes | Balanced workflows | -| `strict` | 🐢 Slowest | Any content changes | Production, critical data | - -## Quick Examples - -### 1. Enable Caching with Fingerprinting (Recommended) - -```bash -# Fetch data with caching enabled (default: meta mode) -floability data --mode fetch \ - --data-spec example/rag-lite-bm25/data/data.yml \ - --backpack example/rag-lite-bm25 \ - --data-cache-mode symlink \ - --verbose -``` - -### 2. Run a Backpack with Caching - -```bash -# Run with default caching and fingerprinting -floability run \ - --backpack example/matrix-multiplication \ - --data-cache-mode symlink \ - --fingerprint-mode meta \ - --verbose -``` - -### 3. Strict Mode for Production - -```bash -# Use strict validation for critical data -floability data --mode verify \ - --data-spec data/data.yml \ - --backpack . \ - --data-cache-mode copy \ - --fingerprint-mode strict \ - --verbose -``` - -### 4. Disable Caching (Legacy Behavior) - -```bash -# Turn off caching entirely -floability run \ - --backpack example/matrix-multiplication \ - --data-cache-mode off -``` - -## Command Options - -All commands (`data`, `run`, `execute`, `instance create`) support: - -```bash ---data-cache-mode [off|symlink|hardlink|copy] - off: No caching (direct download each time) - symlink: Cache and link (default, read-only) - hardlink: Cache and hard-link (same filesystem) - copy: Cache and copy (isolated, modifiable) - ---fingerprint-mode [meta|sample|strict] - meta: Fast, metadata only (default) - sample: First 200 bytes + metadata - strict: Full content hash - ---force-data-cache - Rebuild cache even if valid - ---verbose - Show detailed fingerprinting logs -``` - -## How It Works - -1. **First Run**: Downloads data → Computes fingerprint → Stores in cache -2. **Second Run**: Checks cache → Recomputes fingerprint → Compares - - If fingerprints match: Reuse cache ✅ - - If fingerprints differ: Invalidate & rebuild ♻️ - -## What Gets Fingerprinted? - -- ✅ Filesystem files (`source_type: fs` or `backpack`) -- ❌ HTTP downloads (not yet implemented) -- ❌ Pelican sources (not yet implemented) -- ❌ S3 sources (not yet implemented) - -## Cache Location - -``` -/flo_data_cache/ - / - data/ # Cached content - your-file.csv - .meta.json # Metadata + fingerprint -``` - -## Inspecting Cache - -```bash -# View cache metadata -find flo_data_cache -name ".meta.json" -exec cat {} \; | jq . - -# Check specific fingerprint -cat flo_data_cache//.meta.json | jq '{ - fingerprint: .source_fingerprint, - mode: .fingerprint_mode, - params: .fingerprint_params -}' -``` - -## Common Scenarios - -### Development: Fast Iteration -```bash -floability run --backpack . \ - --data-cache-mode symlink \ - --fingerprint-mode meta -``` -- Fastest mode -- Detects file modifications via mtime -- Good for development cycles - -### Testing: Balance Speed and Safety -```bash -floability run --backpack . \ - --data-cache-mode symlink \ - --fingerprint-mode sample -``` -- Medium speed -- Detects header changes -- Good for testing phase - -### Production: Maximum Validation -```bash -floability run --backpack . \ - --data-cache-mode copy \ - --fingerprint-mode strict -``` -- Full content validation -- Isolated data (copy mode) -- Good for production runs - -### Debugging: See What's Happening -```bash -floability data --mode fetch \ - --data-spec data/data.yml \ - --backpack . \ - --data-cache-mode symlink \ - --fingerprint-mode meta \ - --verbose \ - --force-data-cache -``` -- Shows fingerprint computation -- Shows cache operations -- Rebuilds cache for inspection - -## Troubleshooting - -### Cache Not Reusing? -- Check fingerprint_mode matches previous run -- Verify source files haven't changed -- Use `--verbose` to see why cache invalidated - -### Slow Performance? -- Switch from `strict` to `sample` or `meta` -- Use `meta` mode for large directories -- Consider `--data-cache-mode off` if caching overhead too high - -### Old Cache Format? -- Old caches without fingerprints automatically invalidated -- Will see: "Cache invalid: no source fingerprint (old cache format)" -- Cache will rebuild with new fingerprint metadata - -### Source Changed But Cache Not Invalidating? -- Meta mode only checks mtime/size (use sample or strict) -- Sample mode only checks first 200 bytes (use strict) -- Strict mode checks full content (slowest but catches everything) - -## Performance Tips - -1. **Use meta mode by default** - Fast enough for most cases -2. **Use sample for critical headers** - CSV files, JSON with version info -3. **Reserve strict for small, critical data** - Checksums, signatures, configs -4. **Consider cache mode**: - - `symlink`: Fastest, read-only - - `hardlink`: Fast, same filesystem only - - `copy`: Slower, but isolated - -## Verification - -Run the verification script: -```bash -python3 verify_fingerprinting.py -``` - -This checks: -- Module imports work -- Function signatures correct -- Fingerprinting computes correctly -- All three modes functional - -## Learn More - -- **Complete Testing Strategy**: `TEST_FINGERPRINTING.md` -- **Implementation Details**: `FINGERPRINT_IMPLEMENTATION_SUMMARY.md` -- **Data Operations Overview**: `FLOABILITY_DATA_OPERATIONS_SUMMARY.md` - -## Questions? - -- Does caching work without fingerprinting? **No** - fingerprinting is required for validation -- Can I disable fingerprinting? **Yes** - use `--data-cache-mode off` -- Does it work for HTTP sources? **Not yet** - filesystem only for now -- Will old caches work? **Yes** - they'll be invalidated and rebuilt with fingerprints -- Is it backward compatible? **Yes** - no breaking changes to existing workflows - ---- - -**Ready to test?** Start with `TEST_FINGERPRINTING.md` Scenario 1! diff --git a/implementation-docs/S3_IMPLEMENTATION_SUMMARY.md b/implementation-docs/S3_IMPLEMENTATION_SUMMARY.md deleted file mode 100644 index 2da4f21..0000000 --- a/implementation-docs/S3_IMPLEMENTATION_SUMMARY.md +++ /dev/null @@ -1,159 +0,0 @@ -# S3 Directory Download Implementation Summary - -## Overview -Implemented S3 directory download and caching support in Floability CLI, mirroring the Pelican directory download feature. - -## Changes Made - -### 1. S3 File Utils (`floability/data/s3_file_utils.py`) - -Added two new functions: - -#### `is_s3_directory(uri, anonymous=None)` -- Checks if an S3 URI represents a directory (prefix with multiple objects) -- Detection logic: - 1. If URI ends with "/", assume directory - 2. Otherwise, list up to 2 objects with the prefix - 3. If exactly one object with matching key → file - 4. Otherwise (multiple objects or different key) → directory -- Returns `True` if directory, `False` otherwise - -#### `s3_directory_download(uri, dest_dir, ...)` -- Downloads entire S3 directory recursively -- Uses `s3_list_objects()` to get all files -- Downloads each file preserving directory structure -- Supports: - - Overwrite mode - - Resume support - - Progress bars - - Anonymous access - - Structure preservation (can flatten if desired) -- Returns Path to destination directory - -### 2. Data Handler (`floability/data/data_handler.py`) - -Updated two functions to support S3 directories: - -#### `_attempt_fetch_source()` -- Added S3 directory detection using three-tier priority: - 1. Explicit `source_object_type` field - 2. URL trailing slash - 3. Metadata check via `is_s3_directory()` -- If directory: calls `s3_directory_download()` with target_path as dest_dir -- If file: calls `s3_file_download()` with target_path as filename - -#### `_download_to_cache()` (in `_build_cache_entry`) -- Added same S3 directory detection logic -- If directory: downloads to cache_file (which is a directory path) -- If file: downloads to cache_file.parent with cache_file.name -- Consistent with Pelican directory handling - -### 3. Imports -- Updated `data_handler.py` imports to include: - - `s3_directory_download` - - `is_s3_directory` - -### 4. Documentation & Examples - -Created: -- `example/S3_DIRECTORY_DOWNLOAD.md`: Comprehensive documentation -- `example/s3-directory-examples.yml`: Example data specs -- `scripts/test-s3-dir-download.py`: Test script with 4 test cases - -## Test URL -`s3://floability/dv5-sample-data/` - -## Test Script -`scripts/test-s3-dir-download.py` - -Tests: -1. S3 directory detection -2. Object listing -3. Directory download -4. Caching with data handler - -## Usage Example - -```yaml -- name: dv5_data - target_location: data/dv5 - source: s3://floability/dv5-sample-data/ - source_type: s3 - source_object_type: directory # Optional -``` - -## Cache Structure - -``` -cache/ -├── cached_data/ -│ └── data/ -│ └── dv5/ # Full target_location path -│ ├── file1.root -│ └── subdir/ -│ └── file2.root -└── .meta.json -``` - -## Detection Priority - -1. **Explicit field**: `source_object_type: "directory"` -2. **Trailing slash**: Source URL ends with `/` -3. **Metadata check**: Calls `is_s3_directory()` to check S3 - -## Anonymous Access - -Set environment variable: -```bash -export AWS_NO_SIGN_REQUEST=true -``` - -Or in data spec (future enhancement): -```yaml -source_config: - anonymous: true -``` - -## Consistency with Pelican - -The S3 implementation exactly mirrors the Pelican directory download: -- Same detection methods -- Same cache structure (cached_data/{target_location}) -- Same source_object_type field -- Same function signatures and behavior - -## Running Tests - -```bash -conda activate floability-env -python scripts/test-s3-dir-download.py -``` - -Or you can test manually and report results. - -## Next Steps (Optional) - -1. Add parallel downloads for S3 directories -2. Add bandwidth limiting -3. Add progress tracking for overall directory download -4. Add support for source_config field for per-item anonymous setting -5. Add integration tests with actual backpacks - -## Files Modified - -1. `floability/data/s3_file_utils.py`: Added 2 functions (~150 lines) -2. `floability/data/data_handler.py`: Updated 2 functions (~50 lines changed) -3. `example/S3_DIRECTORY_DOWNLOAD.md`: Created documentation -4. `example/s3-directory-examples.yml`: Created examples -5. `scripts/test-s3-dir-download.py`: Created test script (~250 lines) - -## Verification Needed - -The test script should verify: -- ✅ Directory detection works -- ✅ Object listing works -- ✅ Download preserves structure -- ✅ Cache uses cached_data/{target_location} -- ✅ Anonymous access works with public buckets - -Ready for testing! diff --git a/implementation-docs/TEST_FINGERPRINTING.md b/implementation-docs/TEST_FINGERPRINTING.md deleted file mode 100644 index 42189fa..0000000 --- a/implementation-docs/TEST_FINGERPRINTING.md +++ /dev/null @@ -1,593 +0,0 @@ -# Testing Strategy for Filesystem Fingerprinting - -## Overview -This document outlines the testing strategy for the new filesystem fingerprinting feature in Floability's data caching system. - -## Test Environment Setup - -### 1. Create Test Backpack Structure -```bash -cd /users/mislam5/floability-project/floability-cli -mkdir -p test-fingerprint-backpack/{data,workflow,software,compute} -cd test-fingerprint-backpack -``` - -### 2. Create Test Data Files -```bash -# Create a simple text file -echo "Hello, World!" > data/test-file.txt - -# Create a directory with multiple files -mkdir -p data/test-dir -echo "File 1 content" > data/test-dir/file1.txt -echo "File 2 content" > data/test-dir/file2.txt -echo "File 3 content" > data/test-dir/file3.txt - -# Create a larger file for sample testing -dd if=/dev/urandom of=data/large-file.dat bs=1M count=5 - -# Create a nested directory structure -mkdir -p data/nested/subdir1/subdir2 -echo "Deep file" > data/nested/subdir1/subdir2/deep.txt -echo "Shallow file" > data/nested/file.txt -``` - -### 3. Create Data Specification (data/data.yml) -```bash -cat > data/data.yml << 'EOF' -schema_version: 1.0 -default_profile: local_files - -data_profiles: - local_files: - policy: - retry_attempts: 0 - timeout: 30 - size_tolerance_bytes: 10 - run_operation: fetch - verification_type: size_only - - data: - - name: simple_file - source_type: backpack - source: data/test-file.txt - target_location: data/test-file.txt - expected_size: 14 - - - name: test_directory - source_type: backpack - source: data/test-dir - target_location: data/test-dir - expected_size: 1000 - - - name: large_file - source_type: backpack - source: data/large-file.dat - target_location: data/large-file.dat - expected_size: 5242880 - - - name: nested_directory - source_type: backpack - source: data/nested - target_location: data/nested -EOF -``` - ---- - -## Test Scenarios - -### Scenario 1: Initial Cache Build with Fingerprinting - -**Objective**: Verify that cache entries are created with source fingerprints - -**Test Steps**: -```bash -# Clean any existing cache -rm -rf flo_data_cache/ - -# Fetch data with meta mode (default) -floability data \ - --mode fetch \ - --data-spec data/data.yml \ - --backpack . \ - --data-cache-mode symlink \ - --fingerprint-mode meta \ - --verbose - -# Verify cache was created with fingerprints -ls -la flo_data_cache/ -cat flo_data_cache/*/.[m]eta.json | grep -i fingerprint -``` - -**Expected Results**: -- Cache directories created under `flo_data_cache/` -- Each `.meta.json` contains `source_fingerprint`, `fingerprint_mode`, and `fingerprint_params` fields -- Log shows fingerprint computation messages -- Files materialized in `workflow/data/` - ---- - -### Scenario 2: Cache Reuse with Unchanged Sources - -**Objective**: Verify that cache is reused when source fingerprints match - -**Test Steps**: -```bash -# First fetch (builds cache) -floability data --mode fetch --data-spec data/data.yml --backpack . \ - --data-cache-mode symlink --fingerprint-mode meta --verbose - -# Second fetch (should reuse cache) -floability data --mode fetch --data-spec data/data.yml --backpack . \ - --data-cache-mode symlink --fingerprint-mode meta --verbose \ - --force-fetch -``` - -**Expected Results**: -- First run: "Building cache entry" messages -- Second run: "Cache hit" and "Source fingerprint valid" messages -- Second run: No download/copy operations, only materialization from cache -- Faster execution on second run - ---- - -### Scenario 3: Cache Invalidation on Content Change (Meta Mode) - -**Objective**: Verify meta mode detects mtime changes - -**Test Steps**: -```bash -# Initial fetch -floability data --mode fetch --data-spec data/data.yml --backpack . \ - --data-cache-mode symlink --fingerprint-mode meta --verbose - -# Touch file (change mtime but not content) -sleep 2 -touch data/test-file.txt - -# Fetch again -floability data --mode fetch --data-spec data/data.yml --backpack . \ - --data-cache-mode symlink --fingerprint-mode meta --verbose \ - --force-fetch -``` - -**Expected Results**: -- Log shows "Cache invalid: source fingerprint mismatch" -- Cache rebuilt with new fingerprint -- New mtime_ns in fingerprint_params - ---- - -### Scenario 4: Cache Invalidation on Content Change (Sample Mode) - -**Objective**: Verify sample mode detects header changes - -**Test Steps**: -```bash -# Initial fetch with sample mode -floability data --mode fetch --data-spec data/data.yml --backpack . \ - --data-cache-mode symlink --fingerprint-mode sample --verbose - -# Modify beginning of file (affects sample) -echo "Modified content" > data/test-file.txt - -# Fetch again -floability data --mode fetch --data-spec data/data.yml --backpack . \ - --data-cache-mode symlink --fingerprint-mode sample --verbose \ - --force-fetch -``` - -**Expected Results**: -- Cache invalidated due to sample hash change -- Log shows different sample_sha256 values -- Cache rebuilt - ---- - -### Scenario 5: Cache Invalidation on Content Change (Strict Mode) - -**Objective**: Verify strict mode detects any content changes - -**Test Steps**: -```bash -# Initial fetch with strict mode -floability data --mode fetch --data-spec data/data.yml --backpack . \ - --data-cache-mode symlink --fingerprint-mode strict --verbose - -# Modify end of file (sample mode wouldn't catch this) -echo "Appended content" >> data/large-file.dat - -# Fetch again -floability data --mode fetch --data-spec data/data.yml --backpack . \ - --data-cache-mode symlink --fingerprint-mode strict --verbose \ - --force-fetch -``` - -**Expected Results**: -- Cache invalidated due to full content hash change -- Warning about large file size with strict mode -- Cache rebuilt with new content hash - ---- - -### Scenario 6: Directory Structure Changes (Meta Mode) - -**Objective**: Verify directory fingerprinting detects file additions/deletions - -**Test Steps**: -```bash -# Initial fetch -floability data --mode fetch --data-spec data/data.yml --backpack . \ - --data-cache-mode symlink --fingerprint-mode meta --verbose - -# Add a new file to directory -echo "New file" > data/test-dir/file4.txt - -# Fetch again -floability data --mode fetch --data-spec data/data.yml --backpack . \ - --data-cache-mode symlink --fingerprint-mode meta --verbose \ - --force-fetch - -# Remove a file -rm data/test-dir/file4.txt - -# Fetch again -floability data --mode fetch --data-spec data/data.yml --backpack . \ - --data-cache-mode symlink --fingerprint-mode meta --verbose \ - --force-fetch -``` - -**Expected Results**: -- Cache invalidated when file added (file_count changes) -- Cache invalidated when file removed (file_count changes) -- Logs show fingerprint mismatches - ---- - -### Scenario 7: Directory File Rename Detection - -**Objective**: Verify fingerprinting detects file renames within directories - -**Test Steps**: -```bash -# Initial fetch -floability data --mode fetch --data-spec data/data.yml --backpack . \ - --data-cache-mode symlink --fingerprint-mode meta --verbose - -# Rename a file -mv data/test-dir/file1.txt data/test-dir/renamed.txt - -# Fetch again -floability data --mode fetch --data-spec data/data.yml --backpack . \ - --data-cache-mode symlink --fingerprint-mode meta --verbose \ - --force-fetch -``` - -**Expected Results**: -- Cache invalidated (relative path changed in fingerprint) -- Log shows source fingerprint mismatch -- Cache rebuilt - ---- - -### Scenario 8: Force Cache Rebuild - -**Objective**: Verify --force-data-cache flag bypasses fingerprint validation - -**Test Steps**: -```bash -# Initial fetch -floability data --mode fetch --data-spec data/data.yml --backpack . \ - --data-cache-mode symlink --fingerprint-mode meta --verbose - -# Force rebuild without changing source -floability data --mode fetch --data-spec data/data.yml --backpack . \ - --data-cache-mode symlink --fingerprint-mode meta --verbose \ - --force-fetch --force-data-cache -``` - -**Expected Results**: -- Cache rebuilt even though source unchanged -- Log shows "Building cache entry" instead of cache lookup - ---- - -### Scenario 9: Different Fingerprint Modes on Same Data - -**Objective**: Verify that different fingerprint modes create separate validations - -**Test Steps**: -```bash -# Fetch with meta mode -floability data --mode fetch --data-spec data/data.yml --backpack . \ - --data-cache-mode symlink --fingerprint-mode meta --verbose - -# Examine cache metadata -cat flo_data_cache/*/.[m]eta.json | jq '.fingerprint_mode' - -# Fetch with sample mode (same cache key, different validation) -floability data --mode fetch --data-spec data/data.yml --backpack . \ - --data-cache-mode symlink --fingerprint-mode sample --verbose \ - --force-fetch - -# Fetch with strict mode -floability data --mode fetch --data-spec data/data.yml --backpack . \ - --data-cache-mode symlink --fingerprint-mode strict --verbose \ - --force-fetch -``` - -**Expected Results**: -- Cache entry reused (same cache key from spec hash) -- Fingerprint recomputed with new mode -- If source unchanged, cache still valid -- Cache metadata shows last used fingerprint mode - ---- - -### Scenario 10: Verify Command with Fingerprinting - -**Objective**: Test fingerprinting with verify operation - -**Test Steps**: -```bash -# Verify with meta mode -floability data --mode verify --data-spec data/data.yml --backpack . \ - --data-cache-mode symlink --fingerprint-mode meta --verbose - -# Modify source -echo "Modified" > data/test-file.txt - -# Verify again (should invalidate and refetch) -floability data --mode verify --data-spec data/data.yml --backpack . \ - --data-cache-mode symlink --fingerprint-mode meta --verbose -``` - -**Expected Results**: -- First verify: Cache valid, verification passes -- After modification: Cache invalidated, data refetched, verification passes -- Integrity checks (size, checksum) still work correctly - ---- - -### Scenario 11: Legacy Cache Migration - -**Objective**: Verify handling of old cache entries without fingerprints - -**Test Steps**: -```bash -# Create a mock old cache entry (without fingerprints) -mkdir -p flo_data_cache/test_cache/data -echo "Test" > flo_data_cache/test_cache/data/test.txt -cat > flo_data_cache/test_cache/.meta.json << 'EOF' -{ - "artifact_spec": {"source": "data/test-file.txt", "source_type": "backpack"}, - "content_sha256": "abc123", - "actual_size": 5, - "created_at_iso": "2025-01-01T00:00:00Z" -} -EOF - -# Try to use cache with fingerprinting enabled -floability data --mode fetch --data-spec data/data.yml --backpack . \ - --data-cache-mode symlink --fingerprint-mode meta --verbose -``` - -**Expected Results**: -- Log shows "Cache invalid: no source fingerprint (old cache format)" -- Cache entry rebuilt with new fingerprint metadata -- Old cache entries gracefully invalidated - ---- - -### Scenario 12: Performance Comparison - -**Objective**: Compare performance across fingerprint modes - -**Test Steps**: -```bash -# Create larger test data -mkdir -p data/perf-test -for i in {1..100}; do - echo "File $i content" > data/perf-test/file$i.txt -done - -# Test meta mode -time floability data --mode fetch --data-spec data/data.yml --backpack . \ - --data-cache-mode symlink --fingerprint-mode meta --force-data-cache - -# Test sample mode -time floability data --mode fetch --data-spec data/data.yml --backpack . \ - --data-cache-mode symlink --fingerprint-mode sample --force-data-cache - -# Test strict mode -time floability data --mode fetch --data-spec data/data.yml --backpack . \ - --data-cache-mode symlink --fingerprint-mode strict --force-data-cache -``` - -**Expected Results**: -- Meta mode: Fastest (metadata only) -- Sample mode: Medium (reads first N bytes) -- Strict mode: Slowest (reads all content) -- Warning messages for strict mode with many/large files - ---- - -## Integration with Run/Execute Commands - -### Scenario 13: Run Backpack with Fingerprinting - -**Objective**: Verify fingerprinting works when running a backpack with data - -**Test Steps**: -```bash -# Run with default meta mode -floability run \ - --backpack test-fingerprint-backpack \ - --data-cache-mode symlink \ - --fingerprint-mode meta \ - --verbose - -# Run with sample mode -floability run \ - --backpack test-fingerprint-backpack \ - --data-cache-mode symlink \ - --fingerprint-mode sample \ - --verbose - -# Run with strict mode -floability run \ - --backpack test-fingerprint-backpack \ - --data-cache-mode symlink \ - --fingerprint-mode strict \ - --verbose - -# Run with cache disabled -floability run \ - --backpack test-fingerprint-backpack \ - --data-cache-mode off \ - --verbose -``` - -**Expected Results**: -- Data fetched during run with appropriate fingerprinting -- Cache reused correctly across runs -- Logs show fingerprint computation during data phase -- Workflow executes successfully with cached data - ---- - -### Scenario 14: Instance Create with Fingerprinting - -**Objective**: Verify fingerprinting works when creating instances - -**Test Steps**: -```bash -# Create instance with meta mode -floability instance create \ - --backpack test-fingerprint-backpack \ - --name test-instance-meta \ - --data-cache-mode symlink \ - --fingerprint-mode meta \ - --verbose - -# Create instance with sample mode -floability instance create \ - --backpack test-fingerprint-backpack \ - --name test-instance-sample \ - --data-cache-mode symlink \ - --fingerprint-mode sample \ - --verbose - -# Create instance with cache disabled -floability instance create \ - --backpack test-fingerprint-backpack \ - --name test-instance-nocache \ - --data-cache-mode off \ - --verbose -``` - -**Expected Results**: -- Instance created with data materialized using fingerprinting -- Cache shared across different instances -- Each instance has properly materialized data -- Fingerprint validation ensures data integrity - ---- - -## Validation Checklist - -After running test scenarios, verify: - -- [ ] Cache entries contain fingerprint metadata -- [ ] Fingerprints are deterministic (same source = same fingerprint) -- [ ] Cache invalidates on real changes -- [ ] Cache reuses when source unchanged -- [ ] All three modes (meta, sample, strict) work correctly -- [ ] Directory fingerprinting works for files and directories -- [ ] Warnings shown for large directories with strict mode -- [ ] Legacy cache entries are handled gracefully -- [ ] CLI flag `--fingerprint-mode` works correctly -- [ ] Verbose logging provides useful information -- [ ] No errors or crashes during normal operations - ---- - -## Debugging Tips - -### Inspect Cache Metadata -```bash -# View all cache entries -find flo_data_cache -name ".meta.json" -exec cat {} \; | jq . - -# Check fingerprint for specific cache entry -cat flo_data_cache//.meta.json | jq '{ - fingerprint: .source_fingerprint, - mode: .fingerprint_mode, - params: .fingerprint_params -}' -``` - -### Monitor Cache Operations -```bash -# Watch cache directory during operations -watch -n 1 'ls -lR flo_data_cache/' - -# Monitor logs -floability data --mode fetch --data-spec data/data.yml --backpack . \ - --data-cache-mode symlink --fingerprint-mode meta --verbose 2>&1 | tee fetch.log -``` - -### Test Fingerprint Module Directly -```bash -# Test Python module directly -python3 -c " -from floability.data.fingerprint import compute_fingerprint -import json - -result = compute_fingerprint('data/test-file.txt', 'meta', verbose=True) -print(json.dumps(result, indent=2)) -" -``` - ---- - -## Expected Warnings - -During testing, you may see expected warnings: - -- `"[fingerprint:strict] Warning: Directory is X MB, strict mode will read all content"` - Normal for large directories with strict mode -- `"[fingerprint:sample] Warning: Directory has X files, sampling may take time"` - Normal for directories with many files -- `"[cache] Cache invalid: no source fingerprint (old cache format)"` - Expected when using old cache entries -- `"[cache] Warning: Failed to validate source fingerprint"` - May occur if source was deleted - ---- - -## Cleanup - -After testing: -```bash -# Remove test backpack -cd /users/mislam5/floability-project/floability-cli -rm -rf test-fingerprint-backpack/ - -# Clean cache -rm -rf flo_data_cache/ -``` - ---- - -## Success Criteria - -The implementation is successful if: - -1. ✅ All test scenarios pass without errors -2. ✅ Cache invalidates on real source changes -3. ✅ Cache reuses correctly when sources unchanged -4. ✅ All three fingerprint modes work as expected -5. ✅ Directory fingerprinting handles files and directories -6. ✅ Performance is acceptable (meta < sample < strict) -7. ✅ Logging is clear and helpful -8. ✅ No breaking changes to existing cache functionality -9. ✅ Legacy cache entries handled gracefully -10. ✅ CLI integration works smoothly From 3cdc13977f203c9113189fa5be9dd65e28357dbd Mon Sep 17 00:00:00 2001 From: Md Saiful Islam Date: Wed, 24 Jun 2026 15:15:10 -0400 Subject: [PATCH 2/4] Added site defaults to more commands --- floability/commands/backpack.py | 4 +++- floability/commands/data.py | 4 +++- floability/commands/workers.py | 4 +++- 3 files changed, 9 insertions(+), 3 deletions(-) diff --git a/floability/commands/backpack.py b/floability/commands/backpack.py index 34a9859..aa6ec0b 100644 --- a/floability/commands/backpack.py +++ b/floability/commands/backpack.py @@ -110,7 +110,9 @@ def add_arguments(self, parser: argparse.ArgumentParser) -> None: def execute(self, args: argparse.Namespace, cleanup_manager=None) -> None: """Execute backpack command.""" from ..ops.backpack import run_backpack_command - + from floability.sites import apply_site_defaults + + apply_site_defaults(args, explicit_args=getattr(args, "_explicit_args", None)) run_backpack_command(args) def get_examples(self) -> list: diff --git a/floability/commands/data.py b/floability/commands/data.py index 777b881..349f95d 100644 --- a/floability/commands/data.py +++ b/floability/commands/data.py @@ -91,7 +91,9 @@ def add_arguments(self, parser: argparse.ArgumentParser) -> None: def execute(self, args: argparse.Namespace, cleanup_manager=None) -> None: """Execute data command.""" from ..ops.data import run_data_command - + from floability.sites import apply_site_defaults + + apply_site_defaults(args, explicit_args=getattr(args, "_explicit_args", None)) run_data_command(args) def get_examples(self) -> list: diff --git a/floability/commands/workers.py b/floability/commands/workers.py index 6002da6..137bf86 100644 --- a/floability/commands/workers.py +++ b/floability/commands/workers.py @@ -88,7 +88,9 @@ def add_arguments(self, parser: argparse.ArgumentParser) -> None: def execute(self, args: argparse.Namespace, cleanup_manager=None) -> None: """Execute workers command.""" from ..ops.workers import run_workers_command - + from floability.sites import apply_site_defaults + + apply_site_defaults(args, explicit_args=getattr(args, "_explicit_args", None)) run_workers_command(args) def get_examples(self) -> list: From cc86eca206c09bee72a8c693d7e60c27e0f48d49 Mon Sep 17 00:00:00 2001 From: Md Saiful Islam Date: Thu, 25 Jun 2026 17:17:22 -0400 Subject: [PATCH 3/4] Remvoed fingerprint mode --- docs/reference/cli.md | 3 - docs/reference/data-spec.md | 1 - floability/commands/argument_groups.py | 7 - floability/commands/data.py | 6 - floability/commands/instance.py | 6 - floability/data/data_handler.py | 10 - floability/ops/data.py | 2 - floability/ops/instance.py | 1 - floability/ops/run.py | 1 - scripts/benchmark-cache-key.py | 1 - scripts/test-cache-structure.py | 1 - scripts/test-s3-dir-download.py | 1 - tests/test_data_handler.py | 713 ------------------------- verify_fingerprinting.py | 130 ----- 14 files changed, 883 deletions(-) delete mode 100644 tests/test_data_handler.py delete mode 100644 verify_fingerprinting.py diff --git a/docs/reference/cli.md b/docs/reference/cli.md index e9cce7d..33ae2f9 100644 --- a/docs/reference/cli.md +++ b/docs/reference/cli.md @@ -72,7 +72,6 @@ floability run --instance - `--data-cache-mode off|symlink|hardlink|copy` (default: `symlink`) - `--data-cache-dir DIR`: override default `/floability-data-cache` - `--force-data-cache`: rebuild cache entries even if they already exist -- `--fingerprint-mode meta|sample|strict` (default: `meta`) - `--continue-on-data-failure`: proceed even if data operations fail **Workers/factory:** @@ -129,7 +128,6 @@ Options: - `--data-profile NAME`: override the default profile in the data spec - `--data-cache-mode off|symlink|hardlink|copy` (default: `off`) - `--force-data-cache`: rebuild cache entries even if they already exist -- `--fingerprint-mode meta|sample|strict` (default: `meta`) - `--environment PATH`: manager environment spec - `--worker-environment PATH`: worker environment spec - `--manager-name NAME`: TaskVine manager name (auto-generated if omitted) @@ -238,7 +236,6 @@ Options: - `--data-cache-mode off|symlink|hardlink|copy` (default: `off`) - `--data-cache-dir DIR`: override default `/floability-data-cache` - `--force-data-cache`: rebuild cache entries even if they already exist -- `--fingerprint-mode meta|sample|strict` (default: `meta`) - `--base-dir DIR` (default: `~/floability-base-dir`) --- diff --git a/docs/reference/data-spec.md b/docs/reference/data-spec.md index 2c21937..6009bb6 100644 --- a/docs/reference/data-spec.md +++ b/docs/reference/data-spec.md @@ -218,7 +218,6 @@ Supported options: - `--data-cache-mode off|symlink|hardlink|copy` (default `off`) - `--data-cache-dir` - `--force-data-cache` -- `--fingerprint-mode meta|sample|strict` (default `meta`) - `--base-dir` ### `floability run` / `floability execute` diff --git a/floability/commands/argument_groups.py b/floability/commands/argument_groups.py index dab6550..15ecb71 100644 --- a/floability/commands/argument_groups.py +++ b/floability/commands/argument_groups.py @@ -102,13 +102,6 @@ def add_execution_args(parser: argparse.ArgumentParser) -> None: action="store_true", help="Force rebuild of cache entries even if they already exist.", ) - - parser.add_argument( - "--fingerprint-mode", - default="meta", - choices=["meta", "sample", "strict"], - help="Fingerprint mode for filesystem source validation: meta (fast, metadata only), sample (first N bytes), strict (full content hash).", - ) parser.add_argument( "--cache-lookup-mode", diff --git a/floability/commands/data.py b/floability/commands/data.py index 349f95d..394b45d 100644 --- a/floability/commands/data.py +++ b/floability/commands/data.py @@ -68,12 +68,6 @@ def add_arguments(self, parser: argparse.ArgumentParser) -> None: action="store_true", help="Force rebuild of cache entries even if they already exist.", ) - parser.add_argument( - "--fingerprint-mode", - default="meta", - choices=["meta", "sample", "strict"], - help="Fingerprint mode for filesystem source validation: meta (fast, metadata only), sample (first N bytes), strict (full content hash).", - ) parser.add_argument( "--cache-lookup-mode", diff --git a/floability/commands/instance.py b/floability/commands/instance.py index 5da12c1..e948ce3 100644 --- a/floability/commands/instance.py +++ b/floability/commands/instance.py @@ -62,12 +62,6 @@ def add_arguments(self, parser: argparse.ArgumentParser) -> None: action="store_true", help="Force rebuild of cache entries.", ) - create_parser.add_argument( - "--fingerprint-mode", - default="meta", - choices=["meta", "sample", "strict"], - help="Fingerprint mode for filesystem source validation: meta (fast, metadata only), sample (first N bytes), strict (full content hash).", - ) create_parser.add_argument( "--environment", help="Path to environment.yml (optional).", diff --git a/floability/data/data_handler.py b/floability/data/data_handler.py index af862e9..1e13f44 100644 --- a/floability/data/data_handler.py +++ b/floability/data/data_handler.py @@ -48,7 +48,6 @@ def execute_default_data_operation( base_dir: Path | None = None, cache_base_dir: Path | None = None, target_root: Path | None = None, - fingerprint_mode: str = "meta", cache_lookup_mode: str = "strict", perf: Optional[Any] = None, _out_cache_dirs: Optional[List[str]] = None, @@ -70,7 +69,6 @@ def execute_default_data_operation( base_dir: Floability base directory for cache storage (default: current directory) cache_base_dir: Floability data cache directory (overrides default /floability-data-cache) target_root: Target directory for materialized data (typically instance/workflow). If None, defaults to backpack_root/workflow - fingerprint_mode: Fingerprint mode for filesystem sources: 'meta', 'sample', or 'strict' Returns: bool: True if the operation succeeded, False otherwise. @@ -122,7 +120,6 @@ def execute_default_data_operation( base_dir=base_dir, cache_base_dir=cache_base_dir, target_root=target_root, - fingerprint_mode=fingerprint_mode, cache_lookup_mode=cache_lookup_mode, perf=perf, _out_cache_dirs=_out_cache_dirs, @@ -139,7 +136,6 @@ def execute_default_data_operation( base_dir=base_dir, cache_base_dir=cache_base_dir, target_root=target_root, - fingerprint_mode=fingerprint_mode, cache_lookup_mode=cache_lookup_mode, ) else: @@ -250,7 +246,6 @@ def fetch_data_from_spec( base_dir: Path | None = None, cache_base_dir: Path | None = None, target_root: Path | None = None, - fingerprint_mode: str = "meta", cache_lookup_mode: str = "strict", perf: Optional[Any] = None, _out_cache_dirs: Optional[List[str]] = None, @@ -351,7 +346,6 @@ def fetch_data_from_spec( force_data_cache=force_data_cache, cache_base_dir=cache_base_dir, target_prefix=target_prefix, - fingerprint_mode=fingerprint_mode, perf=perf, _out_cache_dirs=_out_cache_dirs, ) @@ -397,7 +391,6 @@ def verify_data_from_spec( base_dir: Path | None = None, cache_base_dir: Path | None = None, target_root: Path | None = None, - fingerprint_mode: str = "meta", cache_lookup_mode: str = "strict", ) -> bool: """Verify data items: ensure present (download/copy if needed) then validate integrity. @@ -496,7 +489,6 @@ def verify_data_from_spec( force_data_cache=force_data_cache, cache_base_dir=cache_base_dir, target_prefix=target_prefix, - fingerprint_mode=fingerprint_mode, cache_lookup_mode=cache_lookup_mode, ) # Evaluate integrity on local target @@ -1417,7 +1409,6 @@ def _fetch_single_item( force_data_cache: bool = False, cache_base_dir: Path | None = None, target_prefix: Path | None = None, - fingerprint_mode: str = "meta", # TODO: remove — only "meta" mode remains cache_lookup_mode: str = "strict", perf: Optional[Any] = None, _out_cache_dirs: Optional[List[str]] = None, @@ -2005,7 +1996,6 @@ def _write_cache_metadata( # Fingerprint only written for local sources (fs/backpack, "meta" mode). if source_fingerprint: meta["source_fingerprint"] = source_fingerprint.get("fingerprint") - meta["fingerprint_mode"] = "meta" meta_file = cache_dir / ".meta.json" with meta_file.open("w", encoding="utf-8") as f: diff --git a/floability/ops/data.py b/floability/ops/data.py index 9609ec1..fc1d7c6 100644 --- a/floability/ops/data.py +++ b/floability/ops/data.py @@ -190,7 +190,6 @@ def run_data_command(args): base_dir=base_dir, cache_base_dir=cache_base_dir, target_root=target_root, - fingerprint_mode=getattr(args, "fingerprint_mode", "meta"), ) elif args.mode == "verify": print( @@ -207,7 +206,6 @@ def run_data_command(args): base_dir=base_dir, cache_base_dir=cache_base_dir, target_root=target_root, - fingerprint_mode=getattr(args, "fingerprint_mode", "meta"), ) if success and instance_root: diff --git a/floability/ops/instance.py b/floability/ops/instance.py index de55723..e6227d1 100644 --- a/floability/ops/instance.py +++ b/floability/ops/instance.py @@ -135,7 +135,6 @@ def _create_instance_impl(args): base_dir=Path(args.base_dir), cache_base_dir=Path(args.cache_base_dir), target_root=target_root, - fingerprint_mode=getattr(args, "fingerprint_mode", "meta"), perf=perf, _out_cache_dirs=cache_dirs, ) diff --git a/floability/ops/run.py b/floability/ops/run.py index 0d7ac7c..b3075c5 100644 --- a/floability/ops/run.py +++ b/floability/ops/run.py @@ -454,7 +454,6 @@ def _materialize_data( base_dir=Path(args.base_dir), cache_base_dir=Path(args.cache_base_dir), target_root=target_root, - fingerprint_mode=getattr(args, "fingerprint_mode", "meta"), cache_lookup_mode=getattr(args, "cache_lookup_mode", "strict"), perf=perf if perf_enabled else None, _out_cache_dirs=cache_dirs, diff --git a/scripts/benchmark-cache-key.py b/scripts/benchmark-cache-key.py index 71f4db3..7a5aa17 100755 --- a/scripts/benchmark-cache-key.py +++ b/scripts/benchmark-cache-key.py @@ -192,7 +192,6 @@ def benchmark_cache_key(path, backpack_root=None, sample_bytes=200, verbose=Fals # Create artifact spec with fingerprint info artifact_spec = artifact_spec_base.copy() artifact_spec["fingerprint"] = fp_result["fingerprint"] - artifact_spec["fingerprint_mode"] = mode # Benchmark cache key computation cache_key_start = time.time() diff --git a/scripts/test-cache-structure.py b/scripts/test-cache-structure.py index adf77e3..1264d00 100644 --- a/scripts/test-cache-structure.py +++ b/scripts/test-cache-structure.py @@ -41,7 +41,6 @@ def test_cache_structure(): cache_dir=cache_dir, backpack_root=Path(tmpdir), verbose=True, - fingerprint_mode='meta' ) if not success: diff --git a/scripts/test-s3-dir-download.py b/scripts/test-s3-dir-download.py index ee2ae02..c17b6ae 100755 --- a/scripts/test-s3-dir-download.py +++ b/scripts/test-s3-dir-download.py @@ -156,7 +156,6 @@ def test_s3_caching_with_data_handler(): cache_dir=cache_dir, backpack_root=Path(tmpdir), verbose=True, - fingerprint_mode='meta' ) if not success: diff --git a/tests/test_data_handler.py b/tests/test_data_handler.py deleted file mode 100644 index 88de75e..0000000 --- a/tests/test_data_handler.py +++ /dev/null @@ -1,713 +0,0 @@ -""" -Functional tests for data_handler.py - -Tests high-level data operations: check, fetch, verify with data specs. -""" -import pytest -import hashlib -import time -from pathlib import Path -from floability.data.data_handler import ( - check_data_from_spec, - fetch_data_from_spec, - verify_data_from_spec, - execute_default_data_operation, -) - - -@pytest.mark.network -class TestCheckDataFromSpec: - """Test check_data_from_spec() function.""" - - def test_check_returns_true_accessible_file(self, test_data_spec_path, test_backpack_root): - """Test check returns True for accessible file.""" - result = check_data_from_spec( - data_spec=str(test_data_spec_path), - backpack_root=test_backpack_root, - verbose=True, - data_cache_mode="off" - ) - - assert result is True, "Check should return True for accessible file" - print(f"✓ check_data_from_spec returned True for accessible file") - - def test_check_metadata_only_no_download(self, test_data_spec_path, test_backpack_root, mock_base_dir): - """Test check fetches metadata without downloading.""" - workflow_dir = mock_base_dir / "workflow" - - # Ensure workflow dir is empty before check - assert not list(workflow_dir.glob("**/*.root")), "No .root files should exist before check" - - result = check_data_from_spec( - data_spec=str(test_data_spec_path), - backpack_root=test_backpack_root, - verbose=True, - data_cache_mode="off", - base_dir=mock_base_dir - ) - - # Verify no files were downloaded - assert not list(workflow_dir.glob("**/*.root")), "Check should not download files" - assert result is True, "Check should succeed" - print(f"✓ check_data_from_spec verified metadata without downloading") - - def test_check_size_tolerance(self, tmp_test_dir, test_backpack_root): - """Test size tolerance logic in check.""" - # Create a test spec with strict size tolerance - test_spec = tmp_test_dir / "test_size_tolerance.yml" - test_spec.write_text(""" -schema_version: 1.0 -default_profile: strict_size - -profiles: - strict_size: - policy: - size_tolerance_bytes: 0 - data: - - name: test_file - source_type: pelican - source: pelican://disc-head-002.crc.nd.edu:443/nd/disc2/apps/floability/examples/cms-physics-dv5/data/samples/diboson/zz/nano_mc2017_6.root - expected_size: 190634 - target_path: data/test.root -""") - - result = check_data_from_spec( - data_spec=str(test_spec), - backpack_root=test_backpack_root, - verbose=True, - data_cache_mode="off" - ) - - assert result is True, "Check should pass with exact size match" - print(f"✓ Size tolerance check passed") - - def test_check_cache_mode_off(self, test_data_spec_path, test_backpack_root, mock_base_dir): - """Test check with cache_mode='off'.""" - result = check_data_from_spec( - data_spec=str(test_data_spec_path), - backpack_root=test_backpack_root, - verbose=True, - data_cache_mode="off", - base_dir=mock_base_dir - ) - - cache_dir = mock_base_dir / "flo_data_cache" - # With cache_mode='off', cache should not be created during check - # (check is metadata-only anyway) - assert result is True - print(f"✓ check with cache_mode='off' completed") - - def test_check_cache_mode_symlink(self, test_data_spec_path, test_backpack_root, mock_base_dir): - """Test check with cache_mode='symlink'.""" - result = check_data_from_spec( - data_spec=str(test_data_spec_path), - backpack_root=test_backpack_root, - verbose=True, - data_cache_mode="symlink", - base_dir=mock_base_dir, - show_details=True - ) - - assert result is True - print(f"✓ check with cache_mode='symlink' completed") - - def test_check_nonexistent_file(self, test_backpack_root): - """Test check returns False for non-existent file.""" - nonexistent_spec = Path(__file__).parent / "fixtures" / "data" / "pelican_nonexistent.yml" - - result = check_data_from_spec( - data_spec=str(nonexistent_spec), - backpack_root=test_backpack_root, - verbose=True, - data_cache_mode="off" - ) - - assert result is False, "Check should return False for non-existent file" - print(f"✓ check_data_from_spec correctly returned False for non-existent file") - - -@pytest.mark.network -@pytest.mark.slow -class TestFetchDataFromSpec: - """Test fetch_data_from_spec() function.""" - - def test_fetch_basic(self, test_data_spec_path, test_backpack_root, mock_base_dir): - """Test basic fetch operation.""" - target_root = mock_base_dir / "workflow" - - result = fetch_data_from_spec( - data_spec=str(test_data_spec_path), - backpack_root=test_backpack_root, - verbose=True, - data_cache_mode="off", - base_dir=mock_base_dir, - target_root=target_root - ) - - assert result is True, "Fetch should succeed" - - # Verify file materialized - expected_file = target_root / "data" / "samples" / "diboson" / "zz" / "nano_mc2017_6.root" - assert expected_file.exists(), f"Expected file at {expected_file}" - print(f"✓ File fetched successfully to {expected_file}") - - def test_fetch_file_integrity(self, test_data_spec_path, test_backpack_root, mock_base_dir, test_pelican_size, test_pelican_checksum): - """Test fetched file has correct size and checksum.""" - target_root = mock_base_dir / "workflow" - - result = fetch_data_from_spec( - data_spec=str(test_data_spec_path), - backpack_root=test_backpack_root, - verbose=True, - data_cache_mode="off", - base_dir=mock_base_dir, - target_root=target_root - ) - - assert result is True - - target_file = target_root / "data" / "samples" / "diboson" / "zz" / "nano_mc2017_6.root" - - # Check size - actual_size = target_file.stat().st_size - assert actual_size == test_pelican_size, f"Size mismatch: expected {test_pelican_size}, got {actual_size}" - - # Check checksum - sha256_hash = hashlib.sha256() - with open(target_file, "rb") as f: - for chunk in iter(lambda: f.read(8192), b""): - sha256_hash.update(chunk) - actual_checksum = f"sha256:{sha256_hash.hexdigest()}" - - assert actual_checksum == test_pelican_checksum, f"Checksum mismatch" - print(f"✓ File integrity verified: size={actual_size}, checksum={actual_checksum}") - - def test_fetch_force_false_skip_existing(self, test_data_spec_path, test_backpack_root, mock_base_dir): - """Test force=False skips existing file.""" - target_root = mock_base_dir / "workflow" - - # First fetch - result1 = fetch_data_from_spec( - data_spec=str(test_data_spec_path), - backpack_root=test_backpack_root, - verbose=True, - force=False, - data_cache_mode="off", - base_dir=mock_base_dir, - target_root=target_root - ) - assert result1 is True - - target_file = target_root / "data" / "samples" / "diboson" / "zz" / "nano_mc2017_6.root" - mtime1 = target_file.stat().st_mtime - - # Wait a moment - time.sleep(0.1) - - # Second fetch with force=False - result2 = fetch_data_from_spec( - data_spec=str(test_data_spec_path), - backpack_root=test_backpack_root, - verbose=True, - force=False, - data_cache_mode="off", - base_dir=mock_base_dir, - target_root=target_root - ) - assert result2 is True - - mtime2 = target_file.stat().st_mtime - assert mtime1 == mtime2, "File should not be re-fetched with force=False" - print(f"✓ force=False skipped existing file (mtime unchanged)") - - def test_fetch_force_true_redownload(self, test_data_spec_path, test_backpack_root, mock_base_dir): - """Test force=True forces re-download.""" - target_root = mock_base_dir / "workflow" - - # First fetch - result1 = fetch_data_from_spec( - data_spec=str(test_data_spec_path), - backpack_root=test_backpack_root, - verbose=True, - force=False, - data_cache_mode="off", - base_dir=mock_base_dir, - target_root=target_root - ) - assert result1 is True - - target_file = target_root / "data" / "samples" / "diboson" / "zz" / "nano_mc2017_6.root" - original_size = target_file.stat().st_size - - # Corrupt file - with open(target_file, "ab") as f: - f.write(b"CORRUPTED") - corrupted_size = target_file.stat().st_size - assert corrupted_size > original_size - - # Second fetch with force=True - result2 = fetch_data_from_spec( - data_spec=str(test_data_spec_path), - backpack_root=test_backpack_root, - verbose=True, - force=True, - data_cache_mode="off", - base_dir=mock_base_dir, - target_root=target_root - ) - assert result2 is True - - restored_size = target_file.stat().st_size - assert restored_size == original_size, f"force=True should restore original file" - print(f"✓ force=True forced re-download and restored file") - - def test_fetch_cache_mode_off(self, test_data_spec_path, test_backpack_root, mock_base_dir): - """Test fetch with cache_mode='off' (direct download).""" - target_root = mock_base_dir / "workflow" - - result = fetch_data_from_spec( - data_spec=str(test_data_spec_path), - backpack_root=test_backpack_root, - verbose=True, - data_cache_mode="off", - base_dir=mock_base_dir, - target_root=target_root - ) - - assert result is True - - # Verify file exists in target - target_file = target_root / "data" / "samples" / "diboson" / "zz" / "nano_mc2017_6.root" - assert target_file.exists() - - # Verify NO cache was created - cache_dir = mock_base_dir / "flo_data_cache" - cache_entries = list(cache_dir.glob("**/*.root")) - assert len(cache_entries) == 0, "No cache should be created with cache_mode='off'" - print(f"✓ fetch with cache_mode='off' bypassed cache") - - def test_fetch_cache_mode_symlink(self, test_data_spec_path, test_backpack_root, mock_base_dir, cleanup_cache): - """Test fetch with cache_mode='symlink' (cache then symlink).""" - target_root = mock_base_dir / "workflow" - - result = fetch_data_from_spec( - data_spec=str(test_data_spec_path), - backpack_root=test_backpack_root, - verbose=True, - data_cache_mode="symlink", - base_dir=mock_base_dir, - target_root=target_root, - fingerprint_mode="meta" - ) - - assert result is True - - target_file = target_root / "data" / "samples" / "diboson" / "zz" / "nano_mc2017_6.root" - assert target_file.exists() - - # Verify it's a symlink - assert target_file.is_symlink(), "Target should be a symlink with cache_mode='symlink'" - - # Verify cache exists - cache_dir = mock_base_dir / "flo_data_cache" - cache_entries = list(cache_dir.glob("**/*.root")) - assert len(cache_entries) >= 1, "Cache entry should exist" - - # Verify symlink points to cache - link_target = target_file.resolve() - assert str(link_target).startswith(str(cache_dir)), "Symlink should point to cache" - print(f"✓ fetch with cache_mode='symlink' created symlink to cache: {link_target}") - - def test_fetch_cache_mode_hardlink(self, test_data_spec_path, test_backpack_root, mock_base_dir, cleanup_cache): - """Test fetch with cache_mode='hardlink'.""" - target_root = mock_base_dir / "workflow" - - result = fetch_data_from_spec( - data_spec=str(test_data_spec_path), - backpack_root=test_backpack_root, - verbose=True, - data_cache_mode="hardlink", - base_dir=mock_base_dir, - target_root=target_root, - fingerprint_mode="meta" - ) - - assert result is True - - target_file = target_root / "data" / "samples" / "diboson" / "zz" / "nano_mc2017_6.root" - assert target_file.exists() - - # Verify it's NOT a symlink - assert not target_file.is_symlink(), "Target should not be a symlink with cache_mode='hardlink'" - - # Verify cache exists - cache_dir = mock_base_dir / "flo_data_cache" - cache_entries = list(cache_dir.glob("**/*.root")) - assert len(cache_entries) >= 1, "Cache entry should exist" - - # Verify shared inode (hardlink) - cache_file = cache_entries[0] - target_inode = target_file.stat().st_ino - cache_inode = cache_file.stat().st_ino - assert target_inode == cache_inode, "Target and cache should share inode (hardlink)" - print(f"✓ fetch with cache_mode='hardlink' created hardlink (shared inode: {target_inode})") - - def test_fetch_cache_mode_copy(self, test_data_spec_path, test_backpack_root, mock_base_dir, cleanup_cache): - """Test fetch with cache_mode='copy'.""" - target_root = mock_base_dir / "workflow" - - result = fetch_data_from_spec( - data_spec=str(test_data_spec_path), - backpack_root=test_backpack_root, - verbose=True, - data_cache_mode="copy", - base_dir=mock_base_dir, - target_root=target_root, - fingerprint_mode="meta" - ) - - assert result is True - - target_file = target_root / "data" / "samples" / "diboson" / "zz" / "nano_mc2017_6.root" - assert target_file.exists() - - # Verify it's NOT a symlink - assert not target_file.is_symlink(), "Target should not be a symlink with cache_mode='copy'" - - # Verify cache exists - cache_dir = mock_base_dir / "flo_data_cache" - cache_entries = list(cache_dir.glob("**/*.root")) - assert len(cache_entries) >= 1, "Cache entry should exist" - - # Verify independent file (different inodes) - cache_file = cache_entries[0] - target_inode = target_file.stat().st_ino - cache_inode = cache_file.stat().st_ino - assert target_inode != cache_inode, "Target and cache should have different inodes (copy)" - print(f"✓ fetch with cache_mode='copy' created independent copy (target inode: {target_inode}, cache inode: {cache_inode})") - - def test_fetch_cache_structure(self, test_data_spec_path, test_backpack_root, mock_base_dir, cleanup_cache): - """Test cache directory structure is created correctly.""" - target_root = mock_base_dir / "workflow" - - result = fetch_data_from_spec( - data_spec=str(test_data_spec_path), - backpack_root=test_backpack_root, - verbose=True, - data_cache_mode="symlink", - base_dir=mock_base_dir, - target_root=target_root, - fingerprint_mode="meta" - ) - - assert result is True - - cache_dir = mock_base_dir / "flo_data_cache" - assert cache_dir.exists(), "Cache directory should exist" - assert cache_dir.is_dir(), "Cache should be a directory" - - # Check for cache structure (cache entries organized by hash) - cache_entries = list(cache_dir.glob("**/*")) - assert len(cache_entries) > 0, "Cache should contain entries" - print(f"✓ Cache structure created: {len(cache_entries)} entries") - - def test_fetch_fingerprint_mode_meta(self, test_data_spec_path, test_backpack_root, mock_base_dir, cleanup_cache): - """Test fetch with fingerprint_mode='meta'.""" - target_root = mock_base_dir / "workflow" - - result = fetch_data_from_spec( - data_spec=str(test_data_spec_path), - backpack_root=test_backpack_root, - verbose=True, - data_cache_mode="symlink", - base_dir=mock_base_dir, - target_root=target_root, - fingerprint_mode="meta" - ) - - assert result is True - print(f"✓ fetch with fingerprint_mode='meta' successful") - - def test_fetch_fingerprint_mode_sample(self, test_data_spec_path, test_backpack_root, mock_base_dir, cleanup_cache): - """Test fetch with fingerprint_mode='sample'.""" - target_root = mock_base_dir / "workflow" - - result = fetch_data_from_spec( - data_spec=str(test_data_spec_path), - backpack_root=test_backpack_root, - verbose=True, - data_cache_mode="symlink", - base_dir=mock_base_dir, - target_root=target_root, - fingerprint_mode="sample" - ) - - assert result is True - print(f"✓ fetch with fingerprint_mode='sample' successful") - - def test_fetch_fingerprint_mode_strict(self, test_data_spec_path, test_backpack_root, mock_base_dir, cleanup_cache): - """Test fetch with fingerprint_mode='strict'.""" - target_root = mock_base_dir / "workflow" - - result = fetch_data_from_spec( - data_spec=str(test_data_spec_path), - backpack_root=test_backpack_root, - verbose=True, - data_cache_mode="symlink", - base_dir=mock_base_dir, - target_root=target_root, - fingerprint_mode="strict" - ) - - assert result is True - print(f"✓ fetch with fingerprint_mode='strict' successful") - - -@pytest.mark.network -@pytest.mark.slow -class TestVerifyDataFromSpec: - """Test verify_data_from_spec() function.""" - - def test_verify_downloads_and_validates(self, test_data_spec_path, test_backpack_root, mock_base_dir): - """Test verify downloads and validates checksum.""" - target_root = mock_base_dir / "workflow" - - result = verify_data_from_spec( - data_spec=str(test_data_spec_path), - backpack_root=test_backpack_root, - verbose=True, - data_cache_mode="off", - base_dir=mock_base_dir, - target_root=target_root - ) - - assert result is True, "Verify should succeed with matching checksum" - - # Verify file exists - target_file = target_root / "data" / "samples" / "diboson" / "zz" / "nano_mc2017_6.root" - assert target_file.exists() - print(f"✓ verify downloaded and validated file successfully") - - def test_verify_matching_checksum_pass(self, test_data_spec_path, test_backpack_root, mock_base_dir): - """Test verify passes with matching checksum.""" - target_root = mock_base_dir / "workflow" - - result = verify_data_from_spec( - data_spec=str(test_data_spec_path), - backpack_root=test_backpack_root, - verbose=True, - data_cache_mode="off", - base_dir=mock_base_dir, - target_root=target_root - ) - - assert result is True, "Verify should pass with correct checksum" - print(f"✓ Checksum validation passed") - - def test_verify_wrong_checksum_fail(self, tmp_test_dir, test_backpack_root, mock_base_dir): - """Test verify fails with wrong checksum.""" - # Create spec with wrong checksum - bad_checksum_spec = tmp_test_dir / "bad_checksum.yml" - bad_checksum_spec.write_text(""" -schema_version: 1.0 -default_profile: bad_checksum - -profiles: - bad_checksum: - policy: - retry_attempts: 0 - timeout: 30 - size_tolerance_bytes: 10 - data: - - name: test_bad_checksum - source_type: pelican - source: pelican://disc-head-002.crc.nd.edu:443/nd/disc2/apps/floability/examples/cms-physics-dv5/data/samples/diboson/zz/nano_mc2017_6.root - expected_size: 190634 - checksum: sha256:0000000000000000000000000000000000000000000000000000000000000000 - target_path: data/test.root -""") - - target_root = mock_base_dir / "workflow" - - result = verify_data_from_spec( - data_spec=str(bad_checksum_spec), - backpack_root=test_backpack_root, - verbose=True, - data_cache_mode="off", - base_dir=mock_base_dir, - target_root=target_root - ) - - assert result is False, "Verify should fail with wrong checksum" - print(f"✓ Wrong checksum correctly detected and failed") - - def test_verify_size_tolerance(self, tmp_test_dir, test_backpack_root, mock_base_dir): - """Test verify size validation with tolerance.""" - # Create spec with size tolerance - tolerance_spec = tmp_test_dir / "size_tolerance.yml" - tolerance_spec.write_text(""" -schema_version: 1.0 -default_profile: with_tolerance - -profiles: - with_tolerance: - policy: - retry_attempts: 0 - timeout: 30 - size_tolerance_bytes: 100 - data: - - name: test_tolerance - source_type: pelican - source: pelican://disc-head-002.crc.nd.edu:443/nd/disc2/apps/floability/examples/cms-physics-dv5/data/samples/diboson/zz/nano_mc2017_6.root - expected_size: 190700 - checksum: sha256:4c976188f38ffbd755267acb9d5b431b9208c7376afa5a26f9a62412e2f33bb0 - target_path: data/test.root -""") - - target_root = mock_base_dir / "workflow" - - result = verify_data_from_spec( - data_spec=str(tolerance_spec), - backpack_root=test_backpack_root, - verbose=True, - data_cache_mode="off", - base_dir=mock_base_dir, - target_root=target_root - ) - - # Should pass because actual size 190634 is within tolerance of expected 190700 - assert result is True, "Verify should pass within size tolerance" - print(f"✓ Size tolerance validation passed") - - def test_verify_integrity_report(self, test_data_spec_path, test_backpack_root, mock_base_dir): - """Test verify produces integrity report output.""" - target_root = mock_base_dir / "workflow" - - # Capture would require redirect, so we just verify it completes - result = verify_data_from_spec( - data_spec=str(test_data_spec_path), - backpack_root=test_backpack_root, - verbose=True, - data_cache_mode="off", - base_dir=mock_base_dir, - target_root=target_root - ) - - assert result is True - print(f"✓ Verify completed and produced integrity report") - - -@pytest.mark.network -@pytest.mark.slow -class TestExecuteDefaultDataOperation: - """Test execute_default_data_operation() function.""" - - def test_execute_with_policy_check(self, tmp_test_dir, test_backpack_root, mock_base_dir): - """Test execute with policy.run_operation='check'.""" - check_spec = tmp_test_dir / "policy_check.yml" - check_spec.write_text(""" -schema_version: 1.0 -default_profile: check_policy - -profiles: - check_policy: - policy: - run_operation: check - retry_attempts: 0 - timeout: 30 - size_tolerance_bytes: 10 - data: - - name: test_check - source_type: pelican - source: pelican://disc-head-002.crc.nd.edu:443/nd/disc2/apps/floability/examples/cms-physics-dv5/data/samples/diboson/zz/nano_mc2017_6.root - expected_size: 190634 - target_path: data/test.root -""") - - result = execute_default_data_operation( - data_spec=str(check_spec), - backpack_root=test_backpack_root, - verbose=True, - data_cache_mode="off", - base_dir=mock_base_dir - ) - - assert result is True - print(f"✓ execute_default_data_operation with policy='check' successful") - - def test_execute_with_policy_fetch(self, tmp_test_dir, test_backpack_root, mock_base_dir): - """Test execute with policy.run_operation='fetch'.""" - fetch_spec = tmp_test_dir / "policy_fetch.yml" - fetch_spec.write_text(""" -schema_version: 1.0 -default_profile: fetch_policy - -profiles: - fetch_policy: - policy: - run_operation: fetch - retry_attempts: 0 - timeout: 30 - size_tolerance_bytes: 10 - data: - - name: test_fetch - source_type: pelican - source: pelican://disc-head-002.crc.nd.edu:443/nd/disc2/apps/floability/examples/cms-physics-dv5/data/samples/diboson/zz/nano_mc2017_6.root - expected_size: 190634 - target_path: data/test.root -""") - - target_root = mock_base_dir / "workflow" - - result = execute_default_data_operation( - data_spec=str(fetch_spec), - backpack_root=test_backpack_root, - verbose=True, - data_cache_mode="off", - base_dir=mock_base_dir, - target_root=target_root - ) - - assert result is True - - # Verify file was fetched - target_file = target_root / "data" / "test.root" - assert target_file.exists() - print(f"✓ execute_default_data_operation with policy='fetch' successful") - - def test_execute_with_policy_verify(self, tmp_test_dir, test_backpack_root, mock_base_dir): - """Test execute with policy.run_operation='verify'.""" - verify_spec = tmp_test_dir / "policy_verify.yml" - verify_spec.write_text(""" -schema_version: 1.0 -default_profile: verify_policy - -profiles: - verify_policy: - policy: - run_operation: verify - retry_attempts: 0 - timeout: 30 - size_tolerance_bytes: 10 - data: - - name: test_verify - source_type: pelican - source: pelican://disc-head-002.crc.nd.edu:443/nd/disc2/apps/floability/examples/cms-physics-dv5/data/samples/diboson/zz/nano_mc2017_6.root - expected_size: 190634 - checksum: sha256:4c976188f38ffbd755267acb9d5b431b9208c7376afa5a26f9a62412e2f33bb0 - target_path: data/test.root -""") - - target_root = mock_base_dir / "workflow" - - result = execute_default_data_operation( - data_spec=str(verify_spec), - backpack_root=test_backpack_root, - verbose=True, - data_cache_mode="off", - base_dir=mock_base_dir, - target_root=target_root - ) - - assert result is True - print(f"✓ execute_default_data_operation with policy='verify' successful") diff --git a/verify_fingerprinting.py b/verify_fingerprinting.py deleted file mode 100644 index 4e2ca44..0000000 --- a/verify_fingerprinting.py +++ /dev/null @@ -1,130 +0,0 @@ -#!/usr/bin/env python3 -""" -Quick verification script for fingerprinting implementation. -Tests that all components are properly connected. -""" - -import sys -from pathlib import Path - -# Add parent directory to path -sys.path.insert(0, str(Path(__file__).parent)) - -print("=" * 70) -print("Floability Fingerprinting Implementation Verification") -print("=" * 70) - -# Test 1: Import fingerprint module -print("\n[1/5] Testing fingerprint module import...") -try: - from floability.data.fingerprint import compute_fingerprint - print("✅ fingerprint module imported successfully") -except ImportError as e: - print(f"❌ Failed to import fingerprint module: {e}") - sys.exit(1) - -# Test 2: Import data_handler updates -print("\n[2/5] Testing data_handler integration...") -try: - from floability.data.data_handler import ( - execute_default_data_operation, - fetch_data_from_spec, - verify_data_from_spec, - ) - print("✅ data_handler functions imported successfully") -except ImportError as e: - print(f"❌ Failed to import data_handler: {e}") - sys.exit(1) - -# Test 3: Check function signatures -print("\n[3/5] Checking function signatures...") -import inspect - -# Check execute_default_data_operation has fingerprint_mode parameter -sig = inspect.signature(execute_default_data_operation) -if 'fingerprint_mode' in sig.parameters: - print("✅ execute_default_data_operation has fingerprint_mode parameter") -else: - print("❌ execute_default_data_operation missing fingerprint_mode parameter") - sys.exit(1) - -# Check fetch_data_from_spec has fingerprint_mode parameter -sig = inspect.signature(fetch_data_from_spec) -if 'fingerprint_mode' in sig.parameters: - print("✅ fetch_data_from_spec has fingerprint_mode parameter") -else: - print("❌ fetch_data_from_spec missing fingerprint_mode parameter") - sys.exit(1) - -# Check verify_data_from_spec has fingerprint_mode parameter -sig = inspect.signature(verify_data_from_spec) -if 'fingerprint_mode' in sig.parameters: - print("✅ verify_data_from_spec has fingerprint_mode parameter") -else: - print("❌ verify_data_from_spec missing fingerprint_mode parameter") - sys.exit(1) - -# Test 4: Test fingerprint computation -print("\n[4/5] Testing fingerprint computation...") -import tempfile -import os - -# Create a test file -test_dir = Path(tempfile.mkdtemp(prefix='flo_verify_')) -test_file = test_dir / "test.txt" -test_file.write_text("Test content for verification\n") - -try: - # Test all three modes - for mode in ['meta', 'sample', 'strict']: - result = compute_fingerprint(str(test_file), mode, verbose=False) - assert 'fingerprint' in result, f"Missing fingerprint in {mode} result" - assert 'mode' in result, f"Missing mode in {mode} result" - assert 'params' in result, f"Missing params in {mode} result" - assert result['mode'] == mode, f"Mode mismatch: expected {mode}, got {result['mode']}" - print(f" ✅ {mode} mode: {result['fingerprint'][:16]}...") - - print("✅ Fingerprint computation working correctly") -except Exception as e: - print(f"❌ Fingerprint computation failed: {e}") - sys.exit(1) -finally: - # Cleanup - import shutil - shutil.rmtree(test_dir) - -# Test 5: Test directory fingerprinting -print("\n[5/5] Testing directory fingerprinting...") -test_dir = Path(tempfile.mkdtemp(prefix='flo_verify_dir_')) -(test_dir / "file1.txt").write_text("Content 1\n") -(test_dir / "file2.txt").write_text("Content 2\n") -subdir = test_dir / "subdir" -subdir.mkdir() -(subdir / "file3.txt").write_text("Content 3\n") - -try: - for mode in ['meta', 'sample', 'strict']: - result = compute_fingerprint(str(test_dir), mode, verbose=False) - assert 'fingerprint' in result - assert result['mode'] == mode - print(f" ✅ {mode} mode: {result['fingerprint'][:16]}...") - - print("✅ Directory fingerprinting working correctly") -except Exception as e: - print(f"❌ Directory fingerprinting failed: {e}") - sys.exit(1) -finally: - import shutil - shutil.rmtree(test_dir) - -# Final summary -print("\n" + "=" * 70) -print("✅ All verification tests passed!") -print("=" * 70) -print("\nImplementation is ready for testing with floability commands.") -print("\nNext steps:") -print(" 1. Review TEST_FINGERPRINTING.md for comprehensive test scenarios") -print(" 2. Run: floability data --help (to see --fingerprint-mode option)") -print(" 3. Run: floability run --help (to see --fingerprint-mode option)") -print(" 4. Test with example backpacks") -print("=" * 70) From 675f48e712409e04396c3b8a3622268964ea5e14 Mon Sep 17 00:00:00 2001 From: Md Saiful Islam Date: Thu, 25 Jun 2026 17:18:41 -0400 Subject: [PATCH 4/4] Cleanuped examples --- .../directory-download-examples.yml | 88 -- .../s3-directory-examples.yml | 45 - example/rag-lite-bm25/compute/compute.yml | 6 - example/rag-lite-bm25/data/data.yml | 42 - .../rag-lite-bm25/software/environment.yml | 11 - .../workflow/rag-lite-bm25.ipynb | 946 ------------------ 6 files changed, 1138 deletions(-) delete mode 100644 example/example-data-specs/directory-download-examples.yml delete mode 100644 example/example-data-specs/s3-directory-examples.yml delete mode 100644 example/rag-lite-bm25/compute/compute.yml delete mode 100644 example/rag-lite-bm25/data/data.yml delete mode 100644 example/rag-lite-bm25/software/environment.yml delete mode 100644 example/rag-lite-bm25/workflow/rag-lite-bm25.ipynb diff --git a/example/example-data-specs/directory-download-examples.yml b/example/example-data-specs/directory-download-examples.yml deleted file mode 100644 index b104e89..0000000 --- a/example/example-data-specs/directory-download-examples.yml +++ /dev/null @@ -1,88 +0,0 @@ -schema_version: 1.0 -default_profile: example_profile - -# Examples of Pelican directory download specifications -# Three ways to indicate a source is a directory: -# 1. Explicit source_object_type field (recommended for clarity) -# 2. Trailing slash in URL (convention-based) -# 3. Auto-detection via metadata (automatic fallback) - -data_profiles: - # Example 1: Explicit source_object_type field (RECOMMENDED) - explicit_directory: - policy: - retry_attempts: 0 - timeout: 30 - data: - - name: data_dir_explicit - source_type: pelican - source: pelican://disc-head-002.crc.nd.edu:443/nd/disc2/apps/floability/examples/cms-physics-dv5/data - target_path: data - source_object_type: directory # Explicitly mark as directory - - # Example 2: Trailing slash convention - trailing_slash: - policy: - retry_attempts: 0 - timeout: 30 - data: - - name: data_dir_slash - source_type: pelican - source: pelican://disc-head-002.crc.nd.edu:443/nd/disc2/apps/floability/examples/cms-physics-dv5/data/ - target_path: data - # No source_object_type needed - trailing slash indicates directory - - # Example 3: Auto-detection (makes metadata call) - auto_detect: - policy: - retry_attempts: 0 - timeout: 30 - data: - - name: data_dir_auto - source_type: pelican - source: pelican://disc-head-002.crc.nd.edu:443/nd/disc2/apps/floability/examples/cms-physics-dv5/data - target_path: data - # No source_object_type, no trailing slash - will check metadata - - # Example 4: Single file (explicit) - explicit_file: - policy: - retry_attempts: 0 - timeout: 30 - data: - - name: single_file - source_type: pelican - source: pelican://server.example.org:443/path/to/file.root - target_path: data/file.root - source_object_type: file # Explicitly mark as file - - # Example 5: Mixed - directories and files together - mixed_example: - policy: - retry_attempts: 0 - timeout: 30 - data: - - name: data_directory - source_type: pelican - source: pelican://server.example.org:443/datasets/raw/ - target_path: data/raw - source_object_type: directory - - - name: metadata_file - source_type: pelican - source: pelican://server.example.org:443/datasets/metadata.json - target_path: data/metadata.json - source_object_type: file - - - name: results_dir - source_type: pelican - source: pelican://server.example.org:443/datasets/results/ - target_path: data/results - # Trailing slash - detected as directory - -# Notes: -# - source_object_type takes priority over all other detection methods -# - Valid values: "file" or "directory" (case-insensitive) -# - Omit source_object_type to use auto-detection -# - Trailing slash (/) is convention for directories -# - Auto-detection makes a metadata network call (slower but automatic) diff --git a/example/example-data-specs/s3-directory-examples.yml b/example/example-data-specs/s3-directory-examples.yml deleted file mode 100644 index e44a169..0000000 --- a/example/example-data-specs/s3-directory-examples.yml +++ /dev/null @@ -1,45 +0,0 @@ -# S3 Directory Download Examples -# Example data specs demonstrating S3 directory downloads - -# Example 1: Explicit source_object_type = "directory" -- name: dv5_data_explicit - target_location: data/dv5 - source: s3://floability/dv5-sample-data/ - source_type: s3 - source_object_type: directory - description: "Download DV5 sample data with explicit directory type" - -# Example 2: Auto-detect via trailing slash -- name: dv5_data_trailing_slash - target_location: data/dv5-auto - source: s3://floability/dv5-sample-data/ - source_type: s3 - description: "Auto-detect directory via trailing slash" - -# Example 3: Single file (for comparison) -- name: single_file - target_location: data/single/file.root - source: s3://floability/dv5-sample-data/some-file.root - source_type: s3 - source_object_type: file - description: "Download a single file from S3" - -# Example 4: Nested directory -- name: nested_dir - target_location: data/nested/samples - source: s3://mybucket/path/to/samples/ - source_type: s3 - source_object_type: directory - description: "Download nested directory structure" - -# Example 5: Mixed files and directories in same spec -- name: mixed_example - target_location: data/mixed - sources: - - source: s3://floability/dv5-sample-data/ - source_type: s3 - source_object_type: directory - - source: s3://otherbucket/file.txt - source_type: s3 - source_object_type: file - description: "Mix of directory and file sources" diff --git a/example/rag-lite-bm25/compute/compute.yml b/example/rag-lite-bm25/compute/compute.yml deleted file mode 100644 index 6cab721..0000000 --- a/example/rag-lite-bm25/compute/compute.yml +++ /dev/null @@ -1,6 +0,0 @@ -vine_factory_config: - min-workers: 2 - max-workers: 4 - cores: 4 - memory: 1024 - disk: 2000 \ No newline at end of file diff --git a/example/rag-lite-bm25/data/data.yml b/example/rag-lite-bm25/data/data.yml deleted file mode 100644 index 1716dad..0000000 --- a/example/rag-lite-bm25/data/data.yml +++ /dev/null @@ -1,42 +0,0 @@ -schema_version: 1.0 -default_profile: gutenberg_data - -profiles: - gutenberg_data: - policy: - retry_attempts: 3 - timeout: 60 - size_tolerance_bytes: 1024 - - data: - - name: gatsby - source_type: http - source: https://www.gutenberg.org/cache/epub/64317/pg64317.txt - content_type: text/plain - expected_size: 306594 - checksum: sha256:e6b7897aa8498b8dac4df0664827f857bc01135c3d9311adb820979bbc44b763 - target_path: data/pg64317.txt - - - name: frankenstein - source_type: http - source: https://www.gutenberg.org/files/84/84-0.txt - content_type: text/plain - expected_size: 421633 - checksum: sha256:06c37d2c52d208d3d81eb12c3b10b5edbd7728b73554325ddceadbe2fb427e77 - target_path: data/frankenstein.txt - - - name: alice - source_type: http - source: https://www.gutenberg.org/files/11/11-0.txt - content_type: text/plain - expected_size: 151191 - checksum: sha256:a3a27f8edbf7fcd9b8ba8435494440e24952deaa3e2f2d65192d4cb7ca403754 - target_path: data/alice.txt - - - name: shakespeare - source_type: http - source: https://www.gutenberg.org/cache/epub/100/pg100.txt - content_type: text/plain - expected_size: 5638525 - checksum: sha256:4291cb282e90f6580fa683148f7c94a55276acb1757d725a2e84caf8c00cb9a5 - target_path: data/shakespeare_complete.txt diff --git a/example/rag-lite-bm25/software/environment.yml b/example/rag-lite-bm25/software/environment.yml deleted file mode 100644 index 5495b05..0000000 --- a/example/rag-lite-bm25/software/environment.yml +++ /dev/null @@ -1,11 +0,0 @@ -name: rag-lite-bm25 -channels: - - conda-forge - -dependencies: - - python=3.11.14 # consistent with your working Floability instance - - ndcctools=7.15.14 # TaskVine Manager, Worker, PythonTask - - langchain=1.0.3 # core LLM orchestration - - langchain-community=0.3.31 # retrievers, including BM25 - - langchain-text-splitters=0.3.11 # text splitting for chunking - - rank-bm25=0.2.2 # classical BM25 scoring backend \ No newline at end of file diff --git a/example/rag-lite-bm25/workflow/rag-lite-bm25.ipynb b/example/rag-lite-bm25/workflow/rag-lite-bm25.ipynb deleted file mode 100644 index 9a5aedc..0000000 --- a/example/rag-lite-bm25/workflow/rag-lite-bm25.ipynb +++ /dev/null @@ -1,946 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "id": "0dd23872", - "metadata": {}, - "source": [ - "# RAG-Lite with TaskVine + BM25\n", - "\n", - "This notebook shows how to build a **simple RAG-style pipeline** using:\n", - "\n", - "- **TaskVine** to clean and chunk the books in parallel.\n", - "- **BM25** (keyword-based retrieval) to find relevant chunks for a question.\n", - "\n", - "This version is **RAG-lite**: no embeddings, no GPU, just classical IR + distributed preprocessing.\n" - ] - }, - { - "cell_type": "markdown", - "id": "c5585687", - "metadata": {}, - "source": [ - "## Imports and basic setup" - ] - }, - { - "cell_type": "code", - "execution_count": 1, - "id": "dc66fd8a", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Python + TaskVine imports OK\n", - "Found books:\n", - " - alice.txt (151191 bytes)\n", - " - frankenstein.txt (421633 bytes)\n", - " - pg64317.txt (306594 bytes)\n", - " - shakespeare_complete.txt (5638525 bytes)\n", - "\n", - "Book IDs: ['alice', 'frankenstein', 'pg64317', 'shakespeare_complete']\n" - ] - } - ], - "source": [ - "import os\n", - "import math\n", - "import json\n", - "from pathlib import Path\n", - "\n", - "import ndcctools.taskvine as vine\n", - "\n", - "print(\"Python + TaskVine imports OK\")\n", - "\n", - "# Data directory with your local Gutenberg files\n", - "DATA_DIR = Path(\"data\")\n", - "\n", - "book_paths = sorted(DATA_DIR.glob(\"*.txt\"))\n", - "\n", - "if not book_paths:\n", - " raise RuntimeError(f\"No .txt files found in {DATA_DIR}\")\n", - "\n", - "print(\"Found books:\")\n", - "for p in book_paths:\n", - " print(f\" - {p.name} ({p.stat().st_size} bytes)\")\n", - "\n", - "# Simple book_id from filename (without extension)\n", - "def book_id_from_path(path: Path) -> str:\n", - " return path.stem\n", - "\n", - "book_ids = [book_id_from_path(p) for p in book_paths]\n", - "print(\"\\nBook IDs:\", book_ids)" - ] - }, - { - "cell_type": "markdown", - "id": "d9293538", - "metadata": {}, - "source": [ - "## Worker Function\n", - "This function will be executed on TaskVine workers to process each book: clean the text, split into chunks and return the chunks for rag retrieval.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 2, - "id": "82e4f36e", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "✓ worker function clean_and_chunk_book() defined\n" - ] - } - ], - "source": [ - "def clean_and_chunk_book(local_filename: str, book_id: str):\n", - " \"\"\"\n", - " Read a local Gutenberg .txt file (staged by TaskVine on the worker),\n", - " clean it, and chunk it into ~1000-character segments for RAG-style use.\n", - "\n", - " Args:\n", - " local_filename: filename as seen on the worker (e.g., \"book.txt\")\n", - " book_id: identifier for the book (e.g., \"1960\")\n", - "\n", - " Returns:\n", - " list[dict]: one dict per chunk, e.g.\n", - " {\n", - " \"book_id\": ...,\n", - " \"chunk_id\": ...,\n", - " \"total_chunks\": ...,\n", - " \"relative_position\": 0.0–1.0,\n", - " \"text\": \"...\",\n", - " \"chunk_length\": ...,\n", - " \"n_chars\": ...,\n", - " \"n_words\": ...,\n", - " \"preview\": \"...\",\n", - " }\n", - " \"\"\"\n", - " # Imports INSIDE so the function is self-contained on the worker\n", - " import re\n", - " from langchain_text_splitters import RecursiveCharacterTextSplitter\n", - "\n", - " # --- 1. Read full file ---\n", - " with open(local_filename, \"r\", encoding=\"utf-8\", errors=\"ignore\") as f:\n", - " text = f.read()\n", - "\n", - " # --- 2. Strip Gutenberg boilerplate (best-effort) ---\n", - "\n", - " # Header: everything before a \"START OF\" marker\n", - " start_markers = [\n", - " \"*** START OF THIS PROJECT GUTENBERG\",\n", - " \"*** START OF THE PROJECT GUTENBERG\",\n", - " \"***START OF THE PROJECT GUTENBERG\",\n", - " \"*END*THE SMALL PRINT\", # older texts\n", - " ]\n", - " for marker in start_markers:\n", - " idx = text.find(marker)\n", - " if idx != -1:\n", - " text = text[idx + len(marker):]\n", - " break\n", - "\n", - " # Footer: everything after an \"END OF\" marker\n", - " end_markers = [\n", - " \"*** END OF THIS PROJECT GUTENBERG\",\n", - " \"*** END OF THE PROJECT GUTENBERG\",\n", - " \"***END OF THE PROJECT GUTENBERG\",\n", - " ]\n", - " for marker in end_markers:\n", - " idx = text.find(marker)\n", - " if idx != -1:\n", - " text = text[:idx]\n", - " break\n", - "\n", - " # --- 3. Basic normalization ---\n", - "\n", - " # Normalize line endings\n", - " text = text.replace(\"\\r\\n\", \"\\n\").replace(\"\\r\", \"\\n\")\n", - "\n", - " # Collapse multiple blank lines\n", - " text = re.sub(r\"\\n\\s*\\n\\s*\\n+\", \"\\n\\n\", text)\n", - "\n", - " # Collapse multiple spaces\n", - " text = re.sub(r\" +\", \" \", text)\n", - "\n", - " # Strip leading/trailing whitespace\n", - " text = text.strip()\n", - "\n", - " # --- 4. Chunking with LangChain's RecursiveCharacterTextSplitter ---\n", - "\n", - " splitter = RecursiveCharacterTextSplitter(\n", - " chunk_size=1000,\n", - " chunk_overlap=200,\n", - " separators=[\"\\n\\n\", \"\\n\", \". \", \" \", \"\"],\n", - " length_function=len,\n", - " )\n", - "\n", - " chunks = splitter.split_text(text)\n", - " total_chunks = len(chunks)\n", - "\n", - " # --- 5. Build result records with richer metadata ---\n", - "\n", - " results = []\n", - " for i, chunk in enumerate(chunks):\n", - " # Word count\n", - " words = re.findall(r\"\\b\\w+\\b\", chunk)\n", - " n_words = len(words)\n", - "\n", - " # Relative position of this chunk in the book (0.0 = start, 1.0 = end)\n", - " if total_chunks > 1:\n", - " relative_position = i / (total_chunks - 1)\n", - " else:\n", - " relative_position = 0.0\n", - "\n", - " # Short single-line preview\n", - " preview = chunk[:160].replace(\"\\n\", \" \")\n", - "\n", - " results.append({\n", - " \"book_id\": book_id,\n", - " \"chunk_id\": i,\n", - " \"total_chunks\": total_chunks,\n", - " \"relative_position\": relative_position,\n", - " \"text\": chunk,\n", - " \"chunk_length\": len(chunk),\n", - " \"n_chars\": len(chunk),\n", - " \"n_words\": n_words,\n", - " \"preview\": preview,\n", - " })\n", - "\n", - " return results\n", - "\n", - "print(\"✓ worker function clean_and_chunk_book() defined\")" - ] - }, - { - "cell_type": "markdown", - "id": "7aec7bd3-83b3-4a19-8375-5ee416768423", - "metadata": {}, - "source": [ - "## Setup Vine Manager" - ] - }, - { - "cell_type": "code", - "execution_count": 3, - "id": "aaa23eee-2359-4571-b22f-721310ff4b1c", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Manager name: floability-910fe741-dcac-447c-9f5e-5171d1cbd423\n", - "Manager Ports: [9123, 9150]\n" - ] - } - ], - "source": [ - "import os\n", - "\n", - "manager_name = api_key = os.environ.get(\"VINE_MANAGER_NAME\")\n", - "print(f\"Manager name: {manager_name}\")\n", - "\n", - "ports_str = os.environ.get(\"VINE_MANAGER_PORTS\", \"9123, 9150\")\n", - "ports = [int(p.strip()) for p in ports_str.split(\",\")]\n", - "\n", - "if len(ports) == 1:\n", - " ports = ports[0]\n", - "else:\n", - " ports = [int(p) for p in ports]\n", - "\n", - "print(f\"Manager Ports: {ports}\")" - ] - }, - { - "cell_type": "code", - "execution_count": 4, - "id": "32aa0f34-c7ef-40e1-8664-43f66c7d6c3a", - "metadata": {}, - "outputs": [], - "source": [ - "import ndcctools.taskvine as vine\n", - "m = vine.Manager(ports, name=manager_name)" - ] - }, - { - "cell_type": "code", - "execution_count": 5, - "id": "9d93bfcb-33a3-4415-bd62-50f813256a90", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "TaskVine Manager listening on port 9123, floability-910fe741-dcac-447c-9f5e-5171d1cbd423\n" - ] - } - ], - "source": [ - "print(f\"TaskVine Manager listening on port {m.port}, {m.name}\")" - ] - }, - { - "cell_type": "markdown", - "id": "dfbac170-64c2-4455-8f57-f7e336cc3941", - "metadata": {}, - "source": [ - "## Submit Tasks" - ] - }, - { - "cell_type": "code", - "execution_count": 6, - "id": "765a88d0-a845-4084-bf3a-450f02c41d20", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Submitted task 1 for book alice (alice.txt, 151191 bytes)\n", - "Submitted task 2 for book frankenstein (frankenstein.txt, 421633 bytes)\n", - "Submitted task 3 for book pg64317 (pg64317.txt, 306594 bytes)\n", - "Submitted task 4 for book shakespeare_complete (shakespeare_complete.txt, 5638525 bytes)\n", - "\n", - "Total tasks submitted: 4\n" - ] - } - ], - "source": [ - "task_meta = {}\n", - "total_tasks = 0\n", - "\n", - "for path in book_paths:\n", - " book_id = book_id_from_path(path)\n", - " file_size = path.stat().st_size\n", - "\n", - " # Declare the local file; TaskVine will cache it on workers\n", - " f = m.declare_file(str(path), cache=\"worker\")\n", - "\n", - " # Create PythonTask: function + arguments\n", - " # first arg: the filename as it will appear on the worker (\"book.txt\")\n", - " # second arg: book_id\n", - " task = vine.PythonTask(\n", - " clean_and_chunk_book,\n", - " \"book.txt\",\n", - " book_id,\n", - " )\n", - "\n", - " # Attach declared file as input, staged on worker as \"book.txt\"\n", - " task.add_input(f, \"book.txt\")\n", - "\n", - " # Optionally set cores\n", - " task.set_cores(1)\n", - "\n", - " t_id = m.submit(task)\n", - " task_meta[t_id] = {\n", - " \"book_id\": book_id,\n", - " \"path\": str(path),\n", - " \"file_size\": file_size,\n", - " }\n", - " total_tasks += 1\n", - "\n", - " print(f\"Submitted task {t_id} for book {book_id} ({path.name}, {file_size} bytes)\")\n", - "\n", - "print(f\"\\nTotal tasks submitted: {total_tasks}\")" - ] - }, - { - "cell_type": "markdown", - "id": "8b7b9a86-b0f7-492f-8a42-04dc3af6df5a", - "metadata": {}, - "source": [ - "## Wait for tasks and collect the corpus" - ] - }, - { - "cell_type": "code", - "execution_count": 7, - "id": "a0b80d3a-06d8-4902-b99d-374f5fdb159b", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "[1/4] ✓ Task 1 for book alice -> 189 chunks\n", - "[2/4] ✓ Task 3 for book pg64317 -> 359 chunks\n", - "[3/4] ✓ Task 2 for book frankenstein -> 616 chunks\n", - "[4/4] ✓ Task 4 for book shakespeare_complete -> 7198 chunks\n", - "\n", - "All tasks done.\n", - "Total chunks collected: 8362\n" - ] - } - ], - "source": [ - "corpus = []\n", - "completed = 0\n", - "\n", - "while not m.empty():\n", - " t = m.wait(5) # wait up to 5 seconds\n", - " if not t:\n", - " continue\n", - "\n", - " completed += 1\n", - " meta = task_meta[t.id]\n", - " book_id = meta[\"book_id\"]\n", - "\n", - " if t.successful():\n", - " book_chunks = t.output\n", - " corpus.extend(book_chunks)\n", - " print(f\"[{completed}/{total_tasks}] ✓ Task {t.id} for book {book_id} -> {len(book_chunks)} chunks\")\n", - " else:\n", - " print(f\"[{completed}/{total_tasks}] ✗ Task {t.id} for book {book_id} FAILED: {t.result}\")\n", - "\n", - "print(\"\\nAll tasks done.\")\n", - "print(f\"Total chunks collected: {len(corpus)}\")" - ] - }, - { - "cell_type": "markdown", - "id": "5628b10d-8dbb-400b-b687-b6dee47e1216", - "metadata": {}, - "source": [ - "## Save corpus and basic stats" - ] - }, - { - "cell_type": "code", - "execution_count": 8, - "id": "44763716-c97c-45bc-b0a7-b2ec56671005", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "✓ Saved 8362 chunks to gutenberg_corpus.json\n", - "\n", - "Chunks per book:\n", - " alice: 189 chunks\n", - " frankenstein: 616 chunks\n", - " pg64317: 359 chunks\n", - " shakespeare_complete: 7198 chunks\n", - "\n", - "Chunk length stats:\n", - " min: 35\n", - " max: 999\n", - " avg: 822.2\n" - ] - } - ], - "source": [ - "import json\n", - "from collections import Counter\n", - "\n", - "output_file = \"gutenberg_corpus.json\"\n", - "with open(output_file, \"w\", encoding=\"utf-8\") as f:\n", - " json.dump(corpus, f, ensure_ascii=False, indent=2)\n", - "\n", - "print(f\"✓ Saved {len(corpus)} chunks to {output_file}\")\n", - "\n", - "# Quick stats\n", - "book_counts = Counter(c[\"book_id\"] for c in corpus)\n", - "print(\"\\nChunks per book:\")\n", - "for b, cnt in sorted(book_counts.items()):\n", - " print(f\" {b}: {cnt} chunks\")\n", - "\n", - "chunk_lengths = [c[\"chunk_length\"] for c in corpus]\n", - "if chunk_lengths:\n", - " print(\"\\nChunk length stats:\")\n", - " print(f\" min: {min(chunk_lengths)}\")\n", - " print(f\" max: {max(chunk_lengths)}\")\n", - " print(f\" avg: {sum(chunk_lengths)/len(chunk_lengths):.1f}\")" - ] - }, - { - "cell_type": "code", - "execution_count": 9, - "id": "ff5c48c6-2f96-4413-a732-03f2b8c129e1", - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "{'book_id': 'alice',\n", - " 'chunk_id': 0,\n", - " 'total_chunks': 189,\n", - " 'relative_position': 0.0,\n", - " 'text': 'EBOOK 11 ***\\n\\n[Illustration]\\n\\nAlice’s Adventures in Wonderland\\n\\nby Lewis Carroll\\n\\nTHE MILLENNIUM FULCRUM EDITION 3.0\\n\\nContents\\n\\n CHAPTER I. Down the Rabbit-Hole\\n CHAPTER II. The Pool of Tears\\n CHAPTER III. A Caucus-Race and a Long Tale\\n CHAPTER IV. The Rabbit Sends in a Little Bill\\n CHAPTER V. Advice from a Caterpillar\\n CHAPTER VI. Pig and Pepper\\n CHAPTER VII. A Mad Tea-Party\\n CHAPTER VIII. The Queen’s Croquet-Ground\\n CHAPTER IX. The Mock Turtle’s Story\\n CHAPTER X. The Lobster Quadrille\\n CHAPTER XI. Who Stole the Tarts?\\n CHAPTER XII. Alice’s Evidence\\n\\nCHAPTER I.\\nDown the Rabbit-Hole\\n\\nAlice was beginning to get very tired of sitting by her sister on the\\nbank, and of having nothing to do: once or twice she had peeped into\\nthe book her sister was reading, but it had no pictures or\\nconversations in it, “and what is the use of a book,” thought Alice\\n“without pictures or conversations?”',\n", - " 'chunk_length': 893,\n", - " 'n_chars': 893,\n", - " 'n_words': 158,\n", - " 'preview': 'EBOOK 11 *** [Illustration] Alice’s Adventures in Wonderland by Lewis Carroll THE MILLENNIUM FULCRUM EDITION 3.0 Contents CHAPTER I. Down the Rabbit-Hole'}" - ] - }, - "execution_count": 9, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "corpus[0]" - ] - }, - { - "cell_type": "markdown", - "id": "903df920-d9c6-4f32-b4fa-d6280b2c25d8", - "metadata": {}, - "source": [ - "## BM25 retriever + simple RAG helper" - ] - }, - { - "cell_type": "code", - "execution_count": 10, - "id": "91c42855-b094-4041-98c9-fe973cc1d263", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Built 8362 Documents from corpus\n", - "✓ BM25 retriever ready (k=4)\n" - ] - } - ], - "source": [ - "# STEP: Build BM25 retriever and a simple RAG-style query function\n", - "\n", - "from langchain_core.documents import Document\n", - "from langchain_community.retrievers import BM25Retriever # uses rank-bm25 under the hood\n", - "\n", - "# Convert corpus -> LangChain Documents\n", - "documents = [\n", - " Document(\n", - " page_content=c[\"text\"],\n", - " metadata={\n", - " \"book_id\": c[\"book_id\"],\n", - " \"chunk_id\": c[\"chunk_id\"],\n", - " \"relative_position\": c[\"relative_position\"],\n", - " \"n_words\": c[\"n_words\"],\n", - " },\n", - " )\n", - " for c in corpus\n", - "]\n", - "\n", - "print(f\"Built {len(documents)} Documents from corpus\")\n", - "\n", - "# Create BM25 retriever\n", - "retriever = BM25Retriever.from_documents(documents)\n", - "retriever.k = 4 # top-k context chunks per query\n", - "\n", - "print(\"✓ BM25 retriever ready (k=4)\")\n", - "\n", - "def rag_query(query: str, k: int = 4):\n", - " \"\"\"\n", - " RAG-style helper using modern LangChain 'invoke' interface.\n", - " \"\"\"\n", - " retriever.k = k\n", - " results = retriever.invoke(query) # invoke() is correct for LC 1.x\n", - "\n", - " print(\"\\n\" + \"=\"*70)\n", - " print(f\"RAG query: {query!r}\")\n", - " print(\"=\"*70)\n", - " print(f\"Retrieved {len(results)} chunks:\\n\")\n", - "\n", - " for i, doc in enumerate(results, 1):\n", - " meta = doc.metadata\n", - " pos_pct = f\"{100 * meta.get('relative_position', 0.0):.1f}%\"\n", - " print(f\"[{i}] book_id={meta['book_id']} chunk_id={meta['chunk_id']} pos={pos_pct}\")\n", - " preview = doc.page_content[:200].replace(\"\\n\", \" \")\n", - " print(f\" {preview}...\")\n", - " print()\n", - "\n", - " context = \"\\n\\n\".join(\n", - " f\"[book {doc.metadata['book_id']} | chunk {doc.metadata['chunk_id']}] {doc.page_content}\"\n", - " for doc in results\n", - " )\n", - "\n", - " prompt = f\"\"\"You are a helpful assistant answering questions about classic literature.\n", - "\n", - "Use ONLY the following context to answer the question. If the answer is not in the context, say you don't know.\n", - "\n", - "Context:\n", - "{context}\n", - "\n", - "Question: {query}\n", - "Answer:\"\"\"\n", - "\n", - " # print(\"=\"*70)\n", - " # print(\"Example LLM prompt (truncated):\")\n", - " # print(\"=\"*70)\n", - " # print(prompt[:800] + (\"...\" if len(prompt) > 800 else \"\"))\n", - "\n", - " return results, prompt\n" - ] - }, - { - "cell_type": "markdown", - "id": "1d41b997-263e-4609-a10f-c7d39df02380", - "metadata": {}, - "source": [ - "## Sample Rag Query" - ] - }, - { - "cell_type": "code", - "execution_count": 11, - "id": "df11bee7-ed23-490f-b525-e6430ab14876", - "metadata": {}, - "outputs": [], - "source": [ - "query_alice = \"What happens when Alice falls down the rabbit hole?\"\n", - "query_hamlet = \"What does Hamlet mean when he says ‘To be or not to be’?\"" - ] - }, - { - "cell_type": "code", - "execution_count": 12, - "id": "80bf9a96-079f-4173-ae52-65207ffb040b", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "\n", - "======================================================================\n", - "RAG query: 'What happens when Alice falls down the rabbit hole?'\n", - "======================================================================\n", - "Retrieved 4 chunks:\n", - "\n", - "[1] book_id=alice chunk_id=2 pos=1.1%\n", - " There was nothing so _very_ remarkable in that; nor did Alice think it so _very_ much out of the way to hear the Rabbit say to itself, “Oh dear! Oh dear! I shall be late!” (when she thought it over af...\n", - "\n", - "[2] book_id=shakespeare_complete chunk_id=5999 pos=83.4%\n", - " QUINTUS. My sight is very dull, whate’er it bodes. MARTIUS. And mine, I promise you. Were it not for shame, Well could I leave our sport to sleep awhile. [_He falls into the pit._] QUINTUS. What, a...\n", - "\n", - "[3] book_id=alice chunk_id=104 pos=55.3%\n", - " “Well, I’d hardly finished the first verse,” said the Hatter, “when the Queen jumped up and bawled out, ‘He’s murdering the time! Off with his head!’” “How dreadfully savage!” exclaimed Alice. “And ...\n", - "\n", - "[4] book_id=alice chunk_id=8 pos=4.3%\n", - " Alice was not a bit hurt, and she jumped up on to her feet in a moment: she looked up, but it was all dark overhead; before her was another long passage, and the White Rabbit was still in sight, hurry...\n", - "\n" - ] - } - ], - "source": [ - "r,p = rag_query(query_alice)" - ] - }, - { - "cell_type": "code", - "execution_count": 13, - "id": "a03d4a81-2425-465a-9368-c487d5c1a9c6", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "You are a helpful assistant answering questions about classic literature.\n", - "\n", - "Use ONLY the following context to answer the question. If the answer is not in the context, say you don't know.\n", - "\n", - "Context:\n", - "[book alice | chunk 2] There was nothing so _very_ remarkable in that; nor did Alice think it\n", - "so _very_ much out of the way to hear the Rabbit say to itself, “Oh\n", - "dear! Oh dear! I shall be late!” (when she thought it over afterwards,\n", - "it occurred to her that she ought to have wondered at this, but at the\n", - "time it all seemed quite natural); but when the Rabbit actually _took a\n", - "watch out of its waistcoat-pocket_, and looked at it, and then hurried\n", - "on, Alice started to her feet, for it flashed across her mind that she\n", - "had never before seen a rabbit with either a waistcoat-pocket, or a\n", - "watch to take out of it, and burning with curiosity, she ran across the\n", - "field after it, and fortunately was just in time to see it pop down a\n", - "large rabbit-hole under the hedge.\n", - "\n", - "In another moment down went Alice after it, never once considering how\n", - "in the world she was to get out again.\n", - "\n", - "[book shakespeare_complete | chunk 5999] QUINTUS.\n", - "My sight is very dull, whate’er it bodes.\n", - "\n", - "MARTIUS.\n", - "And mine, I promise you. Were it not for shame,\n", - "Well could I leave our sport to sleep awhile.\n", - "\n", - "[_He falls into the pit._]\n", - "\n", - "QUINTUS.\n", - "What, art thou fallen? What subtle hole is this,\n", - "Whose mouth is covered with rude-growing briers,\n", - "Upon whose leaves are drops of new-shed blood\n", - "As fresh as morning dew distilled on flowers?\n", - "A very fatal place it seems to me.\n", - "Speak, brother, hast thou hurt thee with the fall?\n", - "\n", - "MARTIUS.\n", - "O brother, with the dismall’st object hurt\n", - "That ever eye with sight made heart lament!\n", - "\n", - "AARON.\n", - "[_Aside_.] Now will I fetch the king to find them here,\n", - "That he thereby may have a likely guess\n", - "How these were they that made away his brother.\n", - "\n", - "[_Exit._]\n", - "\n", - "MARTIUS.\n", - "Why dost not comfort me, and help me out\n", - "From this unhallowed and blood-stained hole?\n", - "\n", - "QUINTUS.\n", - "I am surprised with an uncouth fear;\n", - "A chilling sweat o’er-runs my trembling joints.\n", - "My heart suspects more than mine eye can see.\n", - "\n", - "[book alice | chunk 104] “Well, I’d hardly finished the first verse,” said the Hatter, “when the\n", - "Queen jumped up and bawled out, ‘He’s murdering the time! Off with his\n", - "head!’”\n", - "\n", - "“How dreadfully savage!” exclaimed Alice.\n", - "\n", - "“And ever since that,” the Hatter went on in a mournful tone, “he won’t\n", - "do a thing I ask! It’s always six o’clock now.”\n", - "\n", - "A bright idea came into Alice’s head. “Is that the reason so many\n", - "tea-things are put out here?” she asked.\n", - "\n", - "“Yes, that’s it,” said the Hatter with a sigh: “it’s always tea-time,\n", - "and we’ve no time to wash the things between whiles.”\n", - "\n", - "“Then you keep moving round, I suppose?” said Alice.\n", - "\n", - "“Exactly so,” said the Hatter: “as the things get used up.”\n", - "\n", - "“But what happens when you come to the beginning again?” Alice ventured\n", - "to ask.\n", - "\n", - "“Suppose we change the subject,” the March Hare interrupted, yawning.\n", - "“I’m getting tired of this. I vote the young lady tells us a story.”\n", - "\n", - "“I’m afraid I don’t know one,” said Alice, rather alarmed at the\n", - "proposal.\n", - "\n", - "[book alice | chunk 8] Alice was not a bit hurt, and she jumped up on to her feet in a moment:\n", - "she looked up, but it was all dark overhead; before her was another\n", - "long passage, and the White Rabbit was still in sight, hurrying down\n", - "it. There was not a moment to be lost: away went Alice like the wind,\n", - "and was just in time to hear it say, as it turned a corner, “Oh my ears\n", - "and whiskers, how late it’s getting!” She was close behind it when she\n", - "turned the corner, but the Rabbit was no longer to be seen: she found\n", - "herself in a long, low hall, which was lit up by a row of lamps hanging\n", - "from the roof.\n", - "\n", - "There were doors all round the hall, but they were all locked; and when\n", - "Alice had been all the way down one side and up the other, trying every\n", - "door, she walked sadly down the middle, wondering how she was ever to\n", - "get out again.\n", - "\n", - "Question: What happens when Alice falls down the rabbit hole?\n", - "Answer:\n" - ] - } - ], - "source": [ - "print(p)" - ] - }, - { - "cell_type": "code", - "execution_count": 27, - "id": "68d95cbb-443e-48ed-baa4-25560a6f3a90", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "\n", - "======================================================================\n", - "RAG query: 'What does Hamlet mean when he says ‘To be or not to be’?'\n", - "======================================================================\n", - "Retrieved 4 chunks:\n", - "\n", - "[1] book_id=shakespeare_complete chunk_id=1491 pos=20.7%\n", - " Enter King, Queen, Laertes, Lords, Osric and Attendants with foils &c. KING. Come, Hamlet, come, and take this hand from me. [_The King puts Laertes’s hand into Hamlet’s._] HAMLET. Give me your par...\n", - "\n", - "[2] book_id=shakespeare_complete chunk_id=1464 pos=20.3%\n", - " SECOND CLOWN. Go to. FIRST CLOWN. What is he that builds stronger than either the mason, the shipwright, or the carpenter? SECOND CLOWN. The gallows-maker; for that frame outlives a thousand tenants...\n", - "\n", - "[3] book_id=shakespeare_complete chunk_id=469 pos=6.5%\n", - " CLEOPATRA. [_Aside to Enobarbus_.] What does he mean? ENOBARBUS. [_Aside to Cleopatra_.] To make his followers weep. ANTONY. Tend me tonight; May be it is the period of your duty. Haply you shall no...\n", - "\n", - "[4] book_id=shakespeare_complete chunk_id=6581 pos=91.4%\n", - " into the company of three or four gentleman-like dogs under the Duke’s table; he had not been there—bless the mark!—a pissing-while but all the chamber smelt him. “Out with the dog!” says one; “What c...\n", - "\n" - ] - } - ], - "source": [ - "r,p = rag_query(query_hamlet)" - ] - }, - { - "cell_type": "code", - "execution_count": 28, - "id": "decbd5bd-0105-477a-a432-83cbbcd996de", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "You are a helpful assistant answering questions about classic literature.\n", - "\n", - "Use ONLY the following context to answer the question. If the answer is not in the context, say you don't know.\n", - "\n", - "Context:\n", - "[book shakespeare_complete | chunk 1491] Enter King, Queen, Laertes, Lords, Osric and Attendants with foils &c.\n", - "\n", - "KING.\n", - "Come, Hamlet, come, and take this hand from me.\n", - "\n", - "[_The King puts Laertes’s hand into Hamlet’s._]\n", - "\n", - "HAMLET.\n", - "Give me your pardon, sir. I have done you wrong;\n", - "But pardon’t as you are a gentleman.\n", - "This presence knows, and you must needs have heard,\n", - "How I am punish’d with sore distraction.\n", - "What I have done\n", - "That might your nature, honour, and exception\n", - "Roughly awake, I here proclaim was madness.\n", - "Was’t Hamlet wrong’d Laertes? Never Hamlet.\n", - "If Hamlet from himself be ta’en away,\n", - "And when he’s not himself does wrong Laertes,\n", - "Then Hamlet does it not, Hamlet denies it.\n", - "Who does it, then? His madness. If’t be so,\n", - "Hamlet is of the faction that is wrong’d;\n", - "His madness is poor Hamlet’s enemy.\n", - "Sir, in this audience,\n", - "Let my disclaiming from a purpos’d evil\n", - "Free me so far in your most generous thoughts\n", - "That I have shot my arrow o’er the house\n", - "And hurt my brother.\n", - "\n", - "[book shakespeare_complete | chunk 1464] SECOND CLOWN.\n", - "Go to.\n", - "\n", - "FIRST CLOWN.\n", - "What is he that builds stronger than either the mason, the shipwright,\n", - "or the carpenter?\n", - "\n", - "SECOND CLOWN.\n", - "The gallows-maker; for that frame outlives a thousand tenants.\n", - "\n", - "FIRST CLOWN.\n", - "I like thy wit well in good faith, the gallows does well. But how does\n", - "it well? It does well to those that do ill. Now, thou dost ill to say\n", - "the gallows is built stronger than the church; argal, the gallows may\n", - "do well to thee. To’t again, come.\n", - "\n", - "SECOND CLOWN.\n", - "Who builds stronger than a mason, a shipwright, or a carpenter?\n", - "\n", - "FIRST CLOWN.\n", - "Ay, tell me that, and unyoke.\n", - "\n", - "SECOND CLOWN.\n", - "Marry, now I can tell.\n", - "\n", - "FIRST CLOWN.\n", - "To’t.\n", - "\n", - "SECOND CLOWN.\n", - "Mass, I cannot tell.\n", - "\n", - "Enter Hamlet and Horatio, at a distance.\n", - "\n", - "FIRST CLOWN.\n", - "Cudgel thy brains no more about it, for your dull ass will not mend his\n", - "pace with beating; and when you are asked this question next, say ‘a\n", - "grave-maker’. The houses he makes last till doomsday. Go, get thee to\n", - "Yaughan; fetch me a stoup of liquor.\n", - "\n", - "[book shakespeare_complete | chunk 469] CLEOPATRA.\n", - "[_Aside to Enobarbus_.] What does he mean?\n", - "\n", - "ENOBARBUS.\n", - "[_Aside to Cleopatra_.] To make his followers weep.\n", - "\n", - "ANTONY.\n", - "Tend me tonight;\n", - "May be it is the period of your duty.\n", - "Haply you shall not see me more, or if,\n", - "A mangled shadow. Perchance tomorrow\n", - "You’ll serve another master. I look on you\n", - "As one that takes his leave. Mine honest friends,\n", - "I turn you not away, but, like a master\n", - "Married to your good service, stay till death.\n", - "Tend me tonight two hours, I ask no more,\n", - "And the gods yield you for’t!\n", - "\n", - "ENOBARBUS.\n", - "What mean you, sir,\n", - "To give them this discomfort? Look, they weep,\n", - "And I, an ass, am onion-eyed. For shame,\n", - "Transform us not to women.\n", - "\n", - "[book shakespeare_complete | chunk 6581] into the company of three or four gentleman-like dogs under the Duke’s\n", - "table; he had not been there—bless the mark!—a pissing-while but all\n", - "the chamber smelt him. “Out with the dog!” says one; “What cur is\n", - "that?” says another; “Whip him out”, says the third; “Hang him up”,\n", - "says the Duke. I, having been acquainted with the smell before, knew it\n", - "was Crab, and goes me to the fellow that whips the dogs. “Friend,”\n", - "quoth I, “you mean to whip the dog?” “Ay, marry do I,” quoth he. “You do\n", - "him the more wrong,” quoth I. “’Twas I did the thing you wot of.” He\n", - "makes me no more ado but whips me out of the chamber. How many masters\n", - "would do this for his servant? Nay, I’ll be sworn I have sat in the\n", - "stock for puddings he hath stolen, otherwise he had been executed. I\n", - "have stood on the pillory for geese he hath killed, otherwise he had\n", - "suffered for’t. [_To Crab_.] Thou think’st not of this now. Nay, I\n", - "remember the trick you served me when I took my leave of Madam Silvia.\n", - "\n", - "Question: What does Hamlet mean when he says ‘To be or not to be’?\n", - "Answer:\n" - ] - } - ], - "source": [ - "print(p)" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3 (ipykernel)", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.11.14" - } - }, - "nbformat": 4, - "nbformat_minor": 5 -}