[FEATURE] VNext Phase 5 — Deep Research Workforce and non-repository resource proof

[Joncallim]

Parent: #333
Execution mode: implementation
Depends on: #338
Transitive foundations: #334, #335, #336
Later migration/generalization: #342

Problem Statement

Forge is not proven general-purpose while its only successful Workforce is tied to Project/Task/Git concepts. A second Workforce must complete useful work using non-repository Resources and the same Mission/Execution/Grant/Budget/Gate contracts, without reintroducing an always-on parent agent or leaking private Resource content to public/model providers.

Deep Research is the smallest high-value proof because its core actions are bounded read/evidence/synthesis rather than broad external mutation.

Desired Outcome

An installable official Deep Research Workforce accepts a bounded research Mission with non-repository Resources, deterministically retrieves/versions evidence, uses bounded parallel cognitive work only where useful, verifies material claims independently, surfaces contradictions/uncertainty, and produces a cited Artifact whose provenance can be reconstructed from Forge evidence.

No Project/Git record is required.

User Story

As the Forge operator,
I want to run deep, adversarial research through the same governed runtime as coding,
So that Forge proves non-coding Workforces without unbounded context, hidden egress or unsupported evidence claims.

Requirements

A. Official package

Ship Deep Research as a normal #338 declarative Workforce package. Keep the role set small/purposeful; illustrative roles are Research Lead, Source Scout(s), Primary-source Researcher, Evidence Verifier, Contrarian Reviewer and Synthesizer. The package manifest/workflow is canonical, not the role names.

B. Non-repository Mission

The Mission is creatable through generic APIs/contracts with objective/question, source/resource policy, Resource bindings, egress/classification policy, time/cost/token/context/concurrency budget, completion/verification policy and output Artifact requirements. It must not need projectId, repository, branch or fake filesystem root.

C. Minimal Phase-5 Resource readers

Implement only the minimum read-only Resource access required for this proof using the generic #334 Resource/Capability/Grant contracts plus #336 confined/read Operation semantics:

• bounded public/approved web retrieval;
• bounded uploaded/local document Resource;
• optional existing knowledge-source read only if it is already safely available.

Do not pre-build the full Phase-8 adapter ecosystem here. Define a narrow generic read boundary/service whose behaviors/evidence are sufficient for Deep Research; #342 later generalizes/migrates these proven readers into the common adapter/credential/conformance plane.

Reads use Principal + Grant + Resource scope, response byte/time/item limits, content-type/redirect validation and provenance. Public web retrieval obeys #353/#335 egress policy and may not receive confidential Resource text merely because a package asks for search.

D. Evidence model

Persist/source Artifacts containing safe metadata such as stable source Resource ref, retrieval time/version/fingerprint/URL when policy allows, content digest and bounded extract identity, classification, claim/evidence relation, verification state, contradiction/staleness notes.

Do not persist raw secrets or unbounded source dumps as ordinary evidence. Historical claim provenance remains reconstructable even when payload retention policy removes content.

E. Bounded retrieval/context

• deterministically dedupe identical/near-identical Resources where safe;
• build run-scoped context packets under explicit token/byte/item budgets;
• no shared accumulated conversation inheritance;
• reuse validated content-addressed extracts/indexes when Resource versions are unchanged;
• treat retrieval/search text as untrusted data that cannot alter policy, Grants, routing or package instructions.

F. Parallel discovery

Parallel Source Scouts are optional/policy-bound. Fan out only when independent discovery materially helps. Enforce Mission concurrency/model-call/token/cost ceilings and avoid sending the full corpus to every worker.

G. Claim/verification contract

The Synthesizer cannot self-certify. Material claims carry structured evidence refs. Deterministic checks run first (source availability, duplicate refs, citation integrity, schema/required evidence). An independent verifier/contrarian pass evaluates support, contradictions and source quality under fresh bounded context. A trusted Gate decides passed | needs_revision | needs_human_review | insufficient_evidence from versioned policy/evidence.

H. Uncertainty/source hierarchy

Prefer primary/authoritative sources where allowed; distinguish fact, interpretation and unresolved contradiction; never fabricate consensus; inaccessible sources produce explicit insufficiency; stale evidence cannot silently satisfy freshness requirements.

I. Output

Produce a human-readable report Artifact plus machine-readable claim/evidence manifest. Citations resolve to underlying evidence/Resource identity and verification state. Prose formatting does not confer authority.

J. Phase-8 handoff contract

Before close, document the exact read behaviors, Resource/Grant semantics, egress rules, evidence schema and failure classes that #342 must preserve when migrating the Phase-5 readers into the generic adapter ecosystem. #342 must be able to migrate without changing the Phase-5 release outcome.

Implementation Sequence

1. Deep Research package/workflow schema under [FEATURE] VNext Phase 4 — declarative Workforce packages and Software Engineering extraction #338.
2. Non-repository Mission API/fixture with no Project/Git dependency.
3. Minimal document Resource reader under [FEATURE][ARCH] VNext Phase 0 — generic runtime contracts and compatibility seam #334/[FEATURE] VNext Phase 2 — secure generic execution envelope and side-effect recovery #336 boundaries.
4. Minimal approved web Resource reader under [FEATURE] VNext Phase 1 — deterministic budget, routing, and context economics #335 egress + [FEATURE] VNext Phase 2 — secure generic execution envelope and side-effect recovery #336 bounds.
5. Evidence/claim contracts: source Artifact, claim relation, contradiction/freshness.
6. Retrieval/context selector: deterministic dedupe, budgets and content-addressed reuse.
7. Research Agent Runs with provider-neutral cognitive requirements.
8. Independent verification/contrarian Gate and bounded revision loop.
9. Report + machine-readable provenance/citations.
10. Hostile release fixture: contradictory/stale/prompt-injected/duplicated/inaccessible sources + budget/egress restrictions.
11. Phase-8 migration contract documenting what [FEATURE] VNext Phase 8 — general Resource/Capability adapter ecosystem #342 must generalize/preserve.

Primary Code Seams To Inspect First

• [FEATURE][ARCH] VNext Phase 0 — generic runtime contracts and compatibility seam #334 Mission/Resource/Artifact/Grant contracts
• [FEATURE] VNext Phase 1 — deterministic budget, routing, and context economics #335 invocation/context/egress broker
• [FEATURE] VNext Phase 2 — secure generic execution envelope and side-effect recovery #336 read-only Operation/confinement boundary
• [FEATURE] VNext Phase 4 — declarative Workforce packages and Software Engineering extraction #338 package/workflow loader
• generic Artifact/Gate/evidence stores
• existing web/worker/architect-context.ts web-search behavior only as migration/reference, not architecture

Orthogonal Checkpoints

1. Non-repository purity: hidden Project/Git assumptions, foreign keys, API/UI serialization.
2. Egress/privacy: sentinel confidential strings, public search, provider restrictions, redirects/content types.
3. Prompt/data injection: hostile source requests tools/policy changes/authority.
4. Evidence correctness: duplicate/missing/stale/contradictory sources, citation mismatch, unsupported claims.
5. Budget/fan-out: parallel oversubscription, duplicate context, retry/remediation cost.
6. Verification independence: Synthesizer self-pass attempts, missing evidence, revision-loop scope.
7. Reader boundary: Phase-5 readers do not become a hidden full adapter framework or expose credentials/broad network authority.
8. Release reproducibility: stable evidence identities where deterministic policy promises them; explicit residual uncertainty.
9. Future migration: hostile review of the [FEATURE] VNext Phase 8 — general Resource/Capability adapter ecosystem #342 handoff contract for semantic gaps that would force a breaking rewrite.

Acceptance Criteria

• A Deep Research Mission completes with no Project/Git record.
• The Workforce is a normal [FEATURE] VNext Phase 4 — declarative Workforce packages and Software Engineering extraction #338 package; Core contains no research-only orchestration branch.
• Every Resource access is attributable to Principal + Grant + Resource version/scope.
• Phase-5 web/document readers use generic [FEATURE][ARCH] VNext Phase 0 — generic runtime contracts and compatibility seam #334/[FEATURE] VNext Phase 2 — secure generic execution envelope and side-effect recovery #336 read boundaries and do not prematurely implement a second full adapter ecosystem.
• Public/model egress is denied before sensitive request construction when Resource policy disallows it.
• Source/prompt-injection text cannot widen authority, tools, routing or egress.
• Hard cost/token/context/model-call/concurrency budgets are enforced.
• Parallel workers do not each receive the full corpus by default.
• Evidence provenance/fingerprints are reconstructable according to retention policy.
• Final material claims trace to evidence refs and explicit verification state.
• Contradictory/stale/insufficient evidence is surfaced rather than forged into certainty.
• A trusted Gate, not the Synthesizer, determines completion.
• Validated extract/index reuse is deterministic when inputs are unchanged; arbitrary model-response caching is not required.
• A versioned Phase-8 handoff contract exists so [FEATURE] VNext Phase 8 — general Resource/Capability adapter ecosystem #342 can migrate/generalize these readers without altering the proven outcome/security semantics.
• Hostile fixture + normal release suite are green.

Out of Scope

• Full generic adapter/credential ecosystem ([FEATURE] VNext Phase 8 — general Resource/Capability adapter ecosystem #342).
• Every academic/web/enterprise connector.
• Persistent monitoring/Triggers ([FEATURE] VNext Phase 6 — persistent Missions, checkpoints, leases, and bounded autonomy #340/[FEATURE] VNext Phase 7 — Trigger/Event runtime with dedupe, causality, and zero-token idle #341).
• Broad write-capable external actions.
• Model ensembles.

Implementation Scope

Large - second official Workforce plus minimal read-only Resource proof, expected as 6-9 small PRs.

Technical Notes

This phase proves generic runtime reuse, not a premature connector platform. If research requires if (deepResearch) orchestration in Core, or a whole parallel adapter framework, stop and correct the generic seam.

[github-actions]

FORGE issue validation

This Feature issue now matches the required FORGE template structure.
The validator applied ready-for-agent and removed needs-clarification if it was present.

[Joncallim]

Hostile Deep Research addendum — retrieval is an adapter/security boundary and citations bind evidence snapshots, not live URLs

A fresh non-repository pass found several correctness/security requirements not explicit enough in the original Phase-5 contract. “Read-only research” still processes attacker-controlled files/web responses and can misstate evidence if URLs, parser outputs, caches or dedupe semantics are treated too casually.

1. Web retrieval is a separate SSRF/destination boundary from public search

#353's fixed DuckDuckGo query endpoint does not make arbitrary source retrieval safe. Following a search result or operator-supplied URL is a new Resource-read Operation and may target arbitrary network locations.

The Phase-5 web reader must:

• normalize/canonicalize the requested URL under a closed supported scheme set;
• apply [FEATURE] VNext Phase 1 — deterministic budget, routing, and context economics #335 destination/classification/Grant policy before request construction;
• validate the actual socket destination against allowed public/private network classes, including DNS rebinding and IPv4/IPv6 special ranges;
• block loopback, link-local/metadata, private/internal and other forbidden address classes unless the Resource is explicitly approved as that private-service class;
• disable or re-authorize redirects destination-by-destination; no transparent public→private redirect;
• not inherit ambient proxy behavior into an egress safety claim unless the proxy itself is trusted policy;
• bound DNS, connect, TLS, headers, body, redirects and total wall-clock;
• inject no unrelated provider/service credentials into generic web retrieval.

Reuse the #335 trusted network/destination transport where possible. Do not create a research-only HTTP client that becomes a second SSRF policy plane.

2. Retrieval identity is the observed evidence snapshot, not the URL

A URL is a locator, not immutable evidence.

Every successful retrieval creates an immutable source-observation Artifact/record binding at least:

• Resource identity + requested locator;
• final admitted locator after redirects;
• retrieval Operation identity;
• database-authoritative observed time;
• HTTP status/content type and policy-approved validators (ETag/Last-Modified when present, as untrusted source metadata rather than absolute truth);
• exact bounded content/extract digest/version;
• parser/extractor version;
• classification + provenance;
• retention/availability state.

Two retrievals from the same URL with different content are distinct evidence revisions. A citation in a report resolves to the exact observed evidence Artifact/revision; the UI may also show the live URL as navigation, but the live page is not substituted silently for what the model/verifier actually saw.

If the source exposes no stable version, say so. Do not fabricate one from the URL.

3. Source time and Forge observation time are different authorities

For freshness:

• observed_at uses Forge/database clock and is authoritative for when Forge saw the evidence;
• source Date, publication date, Last-Modified, embedded timestamps and document metadata are untrusted evidence fields that may be absent/false;
• policy declares whether freshness is based on observation age, asserted publication date, source revision, or a combination;
• clock skew/source future dates cannot make stale evidence magically fresh;
• a cached observation retains its original observed/retrieved identity; cache reuse does not rewrite observed_at to now.

4. Download/decompression/parser handling is hostile code/data processing

The uploaded/local-document reader must not assume PDF/DOCX/HTML/text parsing is inert.

For each supported type:

• magic/content validation and size bounds precede expensive parsing;
• cap compressed and expanded bytes, archive members, nesting/depth, image/object counts and parser time/memory;
• reject archive/path traversal, symlinks/special entries and decompression bombs;
• Office documents are parsed as data: no macros/OLE/script execution, external-template/link fetches, DDE, embedded executable launch or automatic network retrieval;
• HTML/XML parsers disable active scripting and unsafe external entity/resource resolution;
• PDFs/document converters run in the lowest-privilege confined parser boundary available, with no ambient filesystem/network/credentials;
• parser crash/timeout yields explicit unreadable/blocked evidence, never partial text presented as complete without a partial/truncation marker;
• malformed text encoding/Unicode is normalized deterministically with the original content digest retained.

Do not let “minimal reader” become a privileged generic file parser inside Forge Core.

5. Extracts are derived Artifacts with exact source coverage

A bounded extract/index must record:

• exact source observation/revision;
• extraction policy/parser version;
• byte/page/section/span coverage or explicit truncation;
• source classification inherited per SPEC-0005;
• content digest;
• whether it is complete, partial or sampled.

A summary/model output is not a source extract and cannot declassify or replace the source evidence.

When required claim evidence falls outside the retained/extracted coverage, verification is insufficient rather than assuming the rest of the document agrees.

6. Exact dedupe and near-duplicate clustering have different semantics

Safe exact dedupe:

• identical canonical content digest under compatible parser/version/classification may share one content object while preserving every source-observation/provenance edge.

Near-duplicate/fuzzy similarity:

• is advisory clustering only;
• must not collapse two sources into one authoritative evidence identity;
• must not discard a small but material contradiction/change merely because 99% of the text is shared;
• similarity thresholds/algorithm versions are pinned and bounded;
• the verifier can inspect distinct source observations in a cluster when their claims differ.

Never turn “five mirrors of one press release” into five independent corroborating sources merely because URLs differ; likewise never turn two revisions with one changed sentence into one source and lose the contradiction.

7. Independence/source diversity is evidence metadata, not raw source count

For claim support, distinguish:

• multiple observations of the same underlying source/content;
• derivative/reposted material;
• genuinely independent source lineages where provenance can be established;
• unknown independence.

Do not fabricate independence from domains/URLs alone. The source-quality/diversity assessment is evidence/advisory policy and cannot become a model-created Grant/Gate shortcut.

8. Claim/citation IDs are closed references selected from the admitted context

The Synthesizer should produce a structured claim manifest where evidence references must resolve to exact source/extract Artifact IDs present in its admitted context/evidence registry.

Validation rejects:

• fabricated IDs/URLs;
• citation to a source the Agent Run did not receive;
• source revision mismatch;
• citation pointing to a different claim/span than the verifier expects;
• duplicate refs counted as independent support;
• citation to an evidence Artifact whose content was unreadable/insufficient for the claimed proposition.

Do not parse bracket numbers in prose as authority. Prose citations are rendered from the validated machine-readable manifest where practical.

9. Support, contradiction and insufficiency are explicit relations

For each material claim, model the evidence relation rather than only a list of citations:

• supports;
• contradicts;
• contextual/qualifying;
• insufficient/ambiguous.

The verifier evaluates whether the claim wording is no stronger than the support.

Rules:

• contradiction cannot be erased by a larger count of supporting sources;
• missing primary source where policy requires it => insufficient/human/revision;
• inaccessible evidence is not negative proof unless the claim/policy says absence itself is meaningful and the observation method supports that inference;
• model confidence cannot replace evidence sufficiency.

10. Retention/deletion changes evidence availability, not historical truth

If policy later removes raw source payloads:

• immutable provenance/digest/claim relation may remain according to retention policy;
• record payload/extract availability state explicitly;
• do not say the claim is currently independently re-verifiable from Forge if the supporting bytes/extract are gone and the live source cannot reproduce the snapshot;
• existing report provenance remains historical evidence of what Forge processed, not proof that the source still says it now;
• Gate/re-verification policy may require fresh retrieval when retained evidence is unavailable.

“Provenance reconstructable” must not overclaim “content indefinitely reconstructable.”

11. Retrieval caches are authority-neutral and version/classification fenced

Reuse keys include exact source content/version, parser/extractor version, classification revision and relevant policy, as already required by #335.

Additionally:

• cache failure/eviction never changes source truth;
• cached untrusted content remains untrusted;
• cache hit must not skip current destination/Grant/classification authorization for a consumer who may not be allowed to read the cached Artifact;
• raising classification invalidates lower-authority reuse immediately;
• stale cache cannot satisfy a freshness-constrained research Mission merely because content hash is unchanged unless policy explicitly permits it.

12. Research fan-out has per-source and aggregate backpressure

Beyond Mission model-call concurrency, bound:

• web/document retrieval concurrency;
• per-origin request concurrency/rate policy;
• bytes downloaded/parsed;
• number of candidate sources considered;
• redirects and parser work;
• context/extract storage volume.

A hostile query/source graph cannot create a denial-of-wallet/disk/network even before model calls begin.

The Research Lead/model cannot raise these limits by asking for “more exhaustive research.”

13. Search queries and source fetches have separate privacy admission

An authorized static/public search query does not authorize fetching every returned URL, and authorization to read a confidential local document does not authorize using its terms in public search.

Keep the sequence explicit:

• authorize/minimize outbound search terms;
• receive untrusted result locators;
• for each chosen locator, perform a separate Resource/destination read admission;
• classify retrieved content before later model/provider egress;
• do not feed the whole fetched body into the next provider merely because web retrieval itself was public.

14. Hostile release fixture additions

Include:

• public URL redirects to localhost/metadata/private IP -> no request to forbidden sink;
• DNS rebinding source;
• gzip/zip/document decompression bomb;
• DOCX macro/external-template/link bait -> no execution/network;
• XML external entity / HTML active content -> inert;
• PDF/parser timeout/crash -> bounded explicit unreadable result;
• same URL changes between retrievals -> two evidence revisions, old citation stable;
• source asserts a future publication date -> observation freshness not forged;
• near-identical documents differ in one material conclusion -> contradiction preserved;
• five mirror URLs same digest -> not counted as five independent confirmations;
• synthesizer invents citation ID -> schema rejects;
• citation points at source not in context -> rejects;
• retained payload deleted -> report states historical provenance but re-verification availability honestly;
• cache hit by a Principal without read Grant -> denied;
• hostile source explosion exceeds retrieval budget -> bounded deterministic stop before model fan-out.

The strengthened invariant is:

Deep Research cites the exact evidence snapshot Forge actually retrieved and was authorized to use. “Read-only” does not mean parser-safe, network-safe, immutable, independent, or sufficient until those properties are proven explicitly.

[Joncallim]

Roadmap alignment addendum — 2026-09-10

Deep Research should prove reuse of #335 + #367 rather than invent research-specific orchestration:

• Source Scout fan-out/join uses [FEATURE][ARCH] VNext Phase 1B — deterministic Workflow kernel and artifact-routed dispatch #367 dependency/readiness semantics.
• Each cognitive Agent Run receives a fresh [FEATURE] VNext Phase 1 — deterministic budget, routing, and context economics #335 ContextPacket selected from Resource/Artifact/evidence refs; do not send the whole corpus to every worker by default.
• Synthesizer/verifier handoff is artifact-routed, not a supervisor-written transcript summary.
• Research Workflow/package content cannot become a second mutable graph/checkpoint authority.

No new hard dependency is required because #339 already inherits #367 through #338 -> #337.

[Joncallim]

Architecture reconciliation — final report identity and honest evidence limits

Reviewed 1 October 2026 against this issue, the retrieval/evidence addendum and later #367 alignment. Keep the existing SSRF/parser, evidence snapshot, source-independence, cache and egress rules. This is a design review, not a completed research run.

Remaining synthesis/verification contracts

1. Use the common Gate vocabulary. The body proposes research labels passed | needs_revision | needs_human_review | insufficient_evidence. Keep those, if useful, as research-specific disposition/reason fields mapped explicitly to [FEATURE] Add independent Verification Workforce execution #188's trusted pass | rework | human_required | blocked Gate result. Do not introduce a second authoritative research Gate enum or evaluator.
2. Pin the final report and claim manifest together. Verification binds exact report bytes/version, claim wording and evidence manifest. Editing prose after verification can strengthen a claim or remove a qualifier without changing any cited source. That creates a new subject requiring the applicable recheck; a polished final rendering cannot inherit a pass for different wording.
3. Citation integrity is not entailment. A valid source ID and digest prove where the reference points, not that the source supports the sentence. Preserve the independent semantic support check, especially scope, causal claims, numerical comparisons, negative claims and qualifications. One citation at paragraph end must not silently validate several unrelated propositions.
4. Derived numerical claims need calculation evidence. Record source values/units/time basis, transformation or formula, rounding and result provenance for computed findings. A calculation Artifact is distinct from a source extract and does not become primary evidence for its own inputs. Missing denominators or incompatible periods become uncertainty rather than a confident chart/table.
5. Negative and completeness claims require coverage. “No evidence found” is bounded by searched sources, query/scope, access, dates and retrieval limits; it is not “does not exist.” Report inaccessible/partial material and the stopping reason. Hard budgets produce an honestly limited report, not an exhaustive-review claim.
6. Retained text and delivered excerpts have separate policies. Authorised retrieval does not automatically permit unlimited quotation, redistribution or externally sharing private sources. Keep citations, bounded excerpts and final-output disclosure within source rights/classification and the approved audience. A synthesis model cannot declassify a confidential source by paraphrasing it.
7. Claim-level uncertainty survives summarisation. Executive summaries, titles and channel previews must not omit a material contradiction/limit that changes the conclusion. Render them from the same reviewed report/manifest or give the derivative its own validation step; no ambient extra model call on every view.

Acceptance additions

• Verified cautious sentence is edited into a stronger causal claim: final subject no longer matches its Gate.
• Citation resolves correctly but supports only part of a compound claim: verification refuses blanket support.
• Same numbers with different units/periods or a missing denominator: derived comparison fails validation.
• Search hits its source/time/access bound: output states coverage and does not claim exhaustive absence.
• Public-facing summary attempts to paraphrase protected-source details: output disclosure policy still applies.
• Conflicting evidence remains visible in the short summary as well as the detailed appendix.
• Report, machine manifest and rendered citations disagree after formatting: release blocks until they match.

Next step

Prove one non-repository fixture through #367/#335/#336/#338 with exact final-subject verification, then the hostile source matrix. Preserve the narrow Phase5 reader boundary and its explicit #342 migration contract. No new connector, research-only scheduler, model ensemble, code change or issue-state change was introduced.