Harden agent file handling and TypeScript runtime packaging

[noise64]

• adds golem agent file-contents ... --output - for raw stdout bytes while routing progress and errors to stderr with a scoped guard
• streams local, remote, and archive-backed initial files into component upload ZIPs, deduplicating repeated sources and reusing their hashes
• streams only referenced ZIP entries under a shared 16-file semaphore, rejects duplicate normalized paths, and enforces configurable 512 MiB per-file and 1 GiB aggregate uncompressed limits with size and CRC validation on every pass
• acquires existing workers without recreating missing agents and makes each worker own shared deletion completion, monotonic cleanup retries, and identity-safe active-cache retirement
• uses a dedicated component-upload HTTP client controlled only by GOLEM_HTTP_COMPONENT_UPLOAD_*, with no higher-level upload retry policy
• updates the workspace, CI, benchmark, publishing, TypeScript declarations, and Scala guidance to wasm-rquickjs 0.4.4
• documents fresh JavaScript execution for TypeScript and Scala agents while limiting runtime TypeScript transformation guidance to TypeScript templates

[netlify]

✅ Deploy Preview for golemcloud canceled.

Name	Link
🔨 Latest commit	3e9046d
🔍 Latest deploy log	https://app.netlify.com/projects/golemcloud/deploys/6aaae896d76daa00098b1493

[github-actions]

📖 Docs preview: https://docs-pu6jgqb6n-golem-cloud.vercel.app

Built from commit 3e9046d3c1b81762533cd3f77e91a45cc394eb57 by docs.yaml.

[vigoo]

I don't like the additional "deletion lock" introduced in ActiveWorkers. There was already a deletion state in Worker that is supposed to mark the deleting state for overlapping operations, and all operations have been moved from the gRPC layer to be part of Worker itself, making the gRPC API just a facade.

There seems to be two major problems this lock is trying to solve:

• the Worker::get_existing_suspended() is not doing what it's name is doing and can accidentally recreate the worker. That is a bug and all callers of it are actually expecting it to work as its name tells. Should be fixed.
• that the deletion does additional cleanup that can overlap. I would like the Worker deleting state to be the owner of this and overlapping callers to just (optionally) await it, similar to idempotent invocations etc.

Detailed agent-written spec about this follows:

Replace the external deletion lock with worker-owned deletion

Remove the newly added AgentDeletionLocks registry, AgentDeletionLock, and ActiveAgents::deletion_lock(). Do not move the same keyed lock registry elsewhere.

Make deletion an operation owned by the existing Worker, with one executing attempt and a shared completion result.

Keep the separate DefaultWorkerService metadata read/write gate: it protects storage cleanup against concurrent metadata reconstruction and is not the mechanism being replaced.

1. Add genuinely existing-only worker acquisition

“Existing-only” means:

• It may construct an in-memory Worker representation of persisted agent state.
• It must never create a new logical agent, write a new Create oplog entry, generate a new fingerprint, or initialize fresh agent configuration.
• It must not start guest execution. Returning an already cached/running worker is fine; “suspended” does not mean forcibly suspending it.
• A cache miss with no persisted agent returns typed not-found.

Implement this through the actual constructor/metadata-loading path, inside the cache’s coordinated initialization. A separate existence check followed by get-or-create is not sufficient.

Keep intentional create-or-load behavior for normal creation/invocation paths. Do not overload invocation freshness semantics merely to express existing-only lifecycle operations.

Update all six helper call sites and resume. Preserve their status-specific policies:

• Interrupt treats disappearance during acquisition as a no-op, just as initial absence is.
• Other operations return not-found if acquisition discovers absence.
• Resume acquires the existing worker and starts that object, rather than invoking another get-or-create lookup.
• Where validation needs metadata, obtain it from the acquired worker so validation and mutation refer to the same worker identity.

Remove creation-only arguments from the existing-only API. In particular, an update’s target revision must not become a fallback creation revision.

2. Make deletion ownership distinguishable from deletion completion

An illustrative internal API is:

enum DeleteOutcome {
    Started(DeletionHandle),
    AlreadyDeleting(DeletionHandle),
}

impl DeletionHandle {
    async fn wait(&self) -> Result<(), WorkerExecutorError>;
}

Exact names are flexible; the contract is not:

• The first caller atomically claims deletion and starts the operation.
• An overlapping caller gets AlreadyDeleting with a handle to that same attempt.
• That caller does not repeat interruption, fencing, shutdown, stream cleanup, metadata removal, or cache removal.
• The handle is cloneable, supports multiple waiters, and retains the result for callers that start waiting after completion.
• Completion contains success or the actual deletion failure. A notification without a stored result is insufficient.

The existing gRPC delete adapter should await either outcome and return its result. This does not require a new wire/API response variant.

3. Claim ownership before the first deletion side effect

Claim deletion before calling set_interrupting() or doing any asynchronous fencing or cleanup.

Publish the claim under a short worker-state critical section, then release the lock before awaiting work. From that point:

• New execution/start requests observe that deletion has begun.
• Other delete requests discover the shared attempt.
• Stopping with a deleting destination and Deleting participate in the same deletion operation.

Do not simply replace a Running state with Deleting and lose its execution resources. Integrate the operation’s ownership/completion with the existing shutdown transitions.

Do not hold worker.instance, the metadata write lock, or a cache lock while waiting for interruption or invocation-loop exit.

4. Completion means the entire deletion has finished

A successful deletion handle resolves only after:

1. Interruption and execution fencing/draining.
2. Status-flusher and checkpointer barriers.
3. Worker shutdown and required invocation-loop completion.
4. Durable-stream dependency cleanup.
5. Durable metadata/oplog/index removal.
6. Removal of this worker’s active-cache entry.

The existing stopping notification is therefore not the deletion completion handle. Shutdown can complete while durable cleanup is still running.

5. Keep the deleting worker authoritative until cleanup finishes

This is essential to removing the external registry.

A deleting worker must remain discoverable in the active cache until its deletion owner finishes cleanup—even after persistent metadata has been removed. A concurrent delete must be able to find the worker and join its operation during that interval.

Audit the other retirement paths:

• Explicit interrupt.
• Ephemeral invocation-loop cleanup.
• Unloaded-worker eviction.

They must not remove an entry whose deletion operation owns cleanup. They also must not wait for deletion from inside the invocation loop, because deletion may itself be waiting for that loop to exit; defer retirement to the deletion owner instead.

Make removal identity-checked: a stale Worker reference must never remove a replacement worker’s cache entry or associated bookkeeping merely because the agent ID matches.

A completed old worker must retain its terminal deletion result. Calling deletion through an old Arc<Worker> must never rerun cleanup against a replacement agent.

6. Define cancellation and failure explicitly

Recommended behavior:

• Once deletion is claimed, it is worker/executor-owned, not owned by the initiating request’s future.
• Dropping the initiating request or any waiter does not cancel the deletion attempt.
• No gap may leave “deleting” published without an operation that will complete it.
• Every surviving waiter receives a terminal result, including unexpected task failure.
• Do not execute deletion on a serialized task queue if completing shutdown requires work on that same queue.

If cleanup fails:

• Complete all handles for that attempt with the same failure.
• Keep the worker fenced; do not make partially deleted state runnable.
• Permit a later explicit delete to claim one retry attempt.
• Overlapping callers join the retry rather than starting more attempts.
• Old handles retain their original failure.
• Retrying cleanup must tolerate stages already completed and must never recreate missing durable state.

This specifies in-process ownership and retries, not a new persisted deletion/recovery protocol.

7. Required regression tests

Use deterministic barriers around the actual acquisition/deletion paths, not only atomics modeling metadata existence.

Scenario
Required assertion
Existing-only acquisition, absent agent
Not-found; no new oplog, fingerprint, metadata, or cached worker.
Existing-only acquisition, cold persisted agent
Loads the same identity/revision/configuration without starting execution.
Agent disappears between an earlier check and acquisition
Never recreates it. Cover the lifecycle callers’ error/no-op policies.
Two deletes on one worker
One Started, one AlreadyDeleting; one execution of each cleanup stage.
Worker stopped, durable cleanup blocked
Both deletion waiters remain pending.
Metadata removed, cache removal blocked
Another delete joins the existing attempt rather than creating or prematurely returning not-found.
Initiating request or waiter cancelled
Deletion continues; remaining waiters complete.
Cleanup fails, then explicit retry
Original waiters see the failure; exactly one retry executes.
Interrupt/ephemeral retirement overlaps deletion
Entry remains authoritative; no loop-exit/deletion wait cycle.
Replacement created after successful deletion
Old handles/references cannot delete its storage or cache entry.
Different agents deleted concurrently
No unnecessary cross-agent serialization.

Also retain coverage for the previous concern: interruption/shutdown must be able to perform metadata reads before deletion acquires the metadata write lock.

---

The acceptance invariant is: once a caller has acquired a particular worker and claimed its deletion, exactly one operation owns cleanup of that worker; overlapping callers can observe or await it, no lifecycle lookup creates an absent agent, and old cleanup cannot affect a replacement.

[vigoo]

Other than the above, all looks good!

[vigoo]

Can we keep the inherit_test_dep!s at the top of the file please? :)

[vigoo]

This has a very weird name :) and I doubt it is necessary. (If it is, it should live in our Cache type, and definitely not implemented like this). I don't even understand what this is trying to solve

[vigoo]

The way the "get but existing only" has been implemented in this service is not good enough. The cache is not supposed to be called with different construction codes in get_or_insert and that's what the above hack is trying to workaround.

The proper implementation would be to introduce a new initial state to the Worker state machine, from where the caller can check existance or ensure existance, while the worker instance is already registered the cache.

Something like:

// Illustrative API:
let worker = active_agents.get_or_add_unresolved(agent_id).await;

// Request-specific operation, serialized by the Worker state machine:
worker.ensure_existing(principal).await?;
// or:
worker.ensure_created(create_parameters).await?;

[vigoo]

Could this be a one-shot channel or something similar higher level abstraction?

[vigoo]

I expected this state to live in WorkerInstance::Deleting or a new state of the worker, not as a wrapper on top of it

[vigoo]

I'm not sure where to comment about this, it is about all the changes in the oplog submodules not just this line.

Adding this generation, and the explicit removes is trying to fix a race condition where the old open oplog instance's drop closes a recreated one, after an agent gets deleted and reopened.

Even though it solves that problem, if this happens, we have a much bigger problem! It means we have two open oplog instances simulateneously pointing to the same physical oplog. This is the split brain situation we can have among executors, just locally. What happens with drop is just a small part of it; it makes it possible to correupt the whole oplog.

This global Oplog cache with the weak references etc was exactly crated to prevent this - the "singleton-ness" of an oplog instance is stronger than the lifecycle of workers, anything accessing the oplog in any way must get the same instance.

So I don't want this patch; we either never have this problem in practice (just not proven by the type system) in which case we don't need to modify the oplog implementation at all. If we do have this race condition, we have to solve it properly and not just protect against a late drop