ADR: Terra requestor-pays

[bwalsh]

Contextual Routing of googleProject for Requester-Pays DRS Resolution

Status

Proposed

Context

When data consumers query the Data Repository Service (DRS) layer to access controlled-access files, the underlying objects are frequently housed in "requester-pays" Google Cloud Storage (GCS) buckets. To fulfill Google's billing architecture constraints, an active Google Cloud Billing Project must be explicitly declared to absorb data operations and egress costs.
To keep data pipelines modular, reusable, and free from direct platform dependencies, we must isolate users from manually passing storage-level parameters (gsutil -u or gcloud storage --billing-project). Instead, we must pass a standard googleProject string parameter down through our client runtime context, which then binds directly to the orchestration layer handled by Broad Institute's server infrastructure.

Decision

We will establish a unified end-to-end data route for the googleProject tracking parameter. The execution lifecycle transitions linearly across the following client and server boundaries using terra's pattern:

sequenceDiagram
    autonumber
    participant Env as Client Environment (OS/Env)
    participant Client as Client Factory (terra-notebook-utils)
    participant Hub as DRS Server (terra-drs-hub)
    participant GCS as Google Cloud Storage API

    Env->>Client: 1. Reads os.environ.get("GOOGLE_PROJECT")
    Client->>Hub: 2. POST /api/v4/drs/resolve with JSON Body {"googleProject": "id"}
    Hub->>Hub: 3. Marshals string parameter into internal Java Service Record
    Hub->>GCS: 4. Injects project into StorageOptions.newBuilder().setProjectId()
    GCS-->>Client: 5. Dispatches Signed URL containing userProject query parameter

Loading

1. Client Mechanics (terra-notebook-utils)

When bypassing the native tnu interactive CLI shell, the execution script will rely entirely on standardized automation bindings:

• Extraction: The runtime client queries the local environment for an active workspace profile configuration by calling os.environ.get("GOOGLE_PROJECT").
• Hydration: The client factory populates the explicit parameter into the payload dictionary of an outgoing HTTP request targeting the centralized resolution hub. [1]

2. Server Mechanics (terra-drs-hub)

The upstream server engine processes requests via a stateless Java framework:

• Ingestion: The server handles incoming resolutions through endpoints defined inside the core service application layer. It deserializes the JSON request directly into an internal data model.
• Storage Allocation: The service initializes downstream storage handlers by passing the string to Google Cloud's underlying Java SDK (StorageOptions.newBuilder().setProjectId(googleProject)).
• Signed Egress Delivery: If a direct downloadable URL stream is requested, the system maps the configuration parameter into the output signed metadata string using a clean target query structure (?userProject=your-billing-project-id).

Consequences

Positive (Pros)

• Isolated Script Workflows: Data engineering scripts do not require localized tracking definitions or platform context injections.
• Unified Billing Pipeline: Ensures that egress costs for protected multi-tenant pipelines are cleanly tracked and absorbed by the requesting consumer workspace environment.

Negative / Risks (Cons)

• Missing Variable Failures: If GOOGLE_PROJECT is completely omitted from the execution environment, resolution attempts against managed structural endpoints will systematically throw hard operational errors. [1]

References & Code Maps## Client Implementation Links

• Repository Architecture: Review baseline project patterns in the primary DataBiosphere/terra-notebook-utils Workspace Client
• Variable Definition Specs: View documentation on environment mapping rules in the terra-notebook-utils Client Setup Guide [1, 2]

Server Implementation Links

• Repository Architecture: Track structural configurations in the primary [DataBiosphere/terra-drs-hub Orchestration Service Engine](https://github.com/DataBiosphere/terra-drs-hub).
• API Documentation: View the JSON body layout schema inside the DrsHub Operational Request Specification.
• Service Code Layer: Track object validation logic and routing packages within the terra-drs-hub Service Module Core.

[1] https://github.com terra-notebook-utils repo
[2] https://github.com terra-notebook-utils README