Tool rework

.env.example

@@ -26,6 +26,13 @@
 # (Qwen, Gemini, Grok, Mistral, GLM, ChatGPT, Claude, Kimi).
 BROWSER_BRIDGE_ENABLED=0
 
+# Provider error blocking (default 1). When 1, durable pause and unresolved
+# pending attempts block future requests until `python -m providers.access resume <provider>`.
+# When 0, stored provider pauses/pending blocks are ignored and new failures are
+# not latched; this does not disable provider quotas, local budgets, or chat-history
+# consistency checks, and may retry uncertain requests / duplicate provider work.
+PROVIDER_ACCESS_PAUSE=1
+
 # How long to wait for one provider answer, in seconds (1–18000). Default 600.
 # Used by browser tabs, Kimi HTTP and Gemini CLI (after a browser stream starts,
 # the same budget applies).
@@ -40,6 +47,17 @@ PROVIDER_MIN_INTERVAL=30
 # and one-use continuation protection.
 CONVERSATION_MAX_REPLIES=20
 
+# Repeated initial prompts with exactly identical history produce the same cache key.
+# The bridge does not open a new provider chat while a matching link is retained; it may
+# reuse that link if still unused, or reject the repeat when already consumed.
+# On startup, optionally delete closed/consumed links so a repeated initial prompt
+# can start a new chat: 0 (default) keeps them; 1 deletes rows with used=1 only;
+# unused links (used=0) remain. The same prompt may then create a new upstream chat,
+# and its later continuation may bind to the wrong chat and fail.
+# For a complete manual cleanup of all chat-cache tables, run:
+#   python -m server.conversation_admin -clear
+CONVERSATION_CLEAR_USED_ON_STARTUP=0
+
 # Opt-in: one automatic context migration per failed request, in the SAME
 # Harness goal. Only an empty, explicitly cleared context refusal qualifies.
 # Private checkpoints in session/context-checkpoints; no counter reset/replay.
@@ -50,27 +68,50 @@ DEEPSEEK_CONTEXT_ROLLOVER=0
 # RATE_LIMIT_PER_MINUTE=30               # per-client-IP limit on /v1 endpoints
 
 # --- TOOL / TOOLCALL ---
+# Optional raw per-chat traces (prompts, tool calls/results and replies) in ./llog/.
+# Files can contain private project data. Disabled by default.
+TOOLCALL_LOG=0
+
 # How OpenAI `tools` are flattened into the provider prompt.
 # full  — full JSON Schema + tool_calls rules on every turn.
 # light — recommended; full block on the first turn of a provider chat; short
 #         name/format reminder on continuations (smaller follow-ups).
+# medium — like light, but the continuation reminder also lists each tool's
+#         argument shape (field names, nesting, `?` optional; no descriptions), so
+#         a model does not guess arguments from memory. Longer follow-up prompts.
 # none  — like light on the first turn; no tools block on continuations.
 # HTTP transports (DeepSeek, Qwen API, Kimi direct, Gemini CLI): default full.
 # Browser tab bridge (grok/mistral/glm/chatgpt/claude/kimi/qwen/gemini): default light.
 TOOL_INSTRUCTIONS_HTTP=full
 TOOL_INSTRUCTIONS_BROWSER=light
 
-# DeepSeek's web output limit can cut a large tool call off mid-JSON. When that
-# happens it stops without returning partial calls. Continuations are off by
-# default. An explicit opt-in (>0) only continues a provider-confirmed output
-# limit, never an otherwise finished reply with a missing Markdown fence.
+# Temporary: which tool contract to use for every provider.
+#   1 (default) — main-era in api.py + tool_protocol.py (single-block validate,
+#     DeepSeek-only format repair, always invalid_tool_response). Will be removed.
+#   0 — tool-rework in api_new.py + tool_protocol_new.py for all providers
+#     (demote, multi-block, pointed repair, INVALID_MODE).
+TOOLCALL_LEGACY=1
+
+# Provider output limit can cut a tool call off mid-JSON. Continuations are off
+# by default. With TOOLCALL_LEGACY=0 an opt-in (>0) continues when a top-level
+# tool_calls fence is still open anywhere in the reply; with LEGACY=1 only when
+# the whole reply starts with an open fence (main-era). Never continues a normal
+# stop that merely omitted a Markdown fence.
 TOOLCALL_MAX_CONTINUATIONS=0
 
-# Optional: one format correction after a normally completed invalid DeepSeek
-# tool reply, in the same conversation. Adds at most one paced request. Never
-# retries provider refusals, output limits, or uncertain dispatches. Default off.
+# Optional: one format correction after a normally completed erroneous tool reply.
+# LEGACY=1: DeepSeek only (main-era prompt). LEGACY=0: every provider (catalog
+# on unknown tool, argument shape on schema). Default off.
 TOOLCALL_FORMAT_REPAIR=0
 
+# Outcome after an invalid tool reply (or a failed single format correction).
+# Only when TOOLCALL_LEGACY=0 (ignored under LEGACY=1 — always error).
+#   error — invalid_tool_response (default; fail-closed)
+#   text  — live provider chat + explicit failed step for the agent:
+#           content notice (BRIDGE_TOOL_NOT_EXECUTED), top-level bridge_notice,
+#           message.refusal; next user turn → provider as Tool result (name): …
+TOOLCALL_INVALID_MODE=error
+
 # Dump the raw model reply to the log whenever a tool call can't be parsed.
 # TOOLCALL_DEBUG=1
 

README.en.md

@@ -237,7 +237,93 @@ its context. It is not a local budget or expired session; resetting counters or
 signing in again will not fix it. The bridge preserves history and stops, without
 creating a new chat. Migrating the goal requires a separate manual decision.
 
-Disabling `TOOLCALL_FORMAT_REPAIR` prevents an extra format-correction generation;
+Optional `TOOLCALL_LOG=1` enables detailed conversation traces in `logs/` (default
+`0`, disabled). Each conversation gets one file named `MMDDHHmmss-<provider>.log`;
+it accumulates agent requests, assembled provider prompts, raw provider replies,
+final JSON/SSE responses to the agent, and errors, separated by `from agent`, `to provider`,
+`from provider`, and `to agent` markers. Logs can contain source code, secrets and
+tool results; enable only for local diagnosis and do not publish them. Git ignores
+the directory. Implementation checks are offline; live-account exchanges were not
+verified. Continuations are associated via the persisted conversation ID; if a
+provider returns no ID, a later request cannot be linked to this trace.
+
+New mode `TOOL_INSTRUCTIONS_BROWSER=medium` (the default stays `light`, unchanged:
+tool names only). On chat continuations the reminder also includes each tool's
+argument shape (field names, nesting, `?` = optional; no descriptions) so the model
+does not guess arguments from memory; the first turn still gets the full block, as
+in `light`. Continuation prompts are longer. A reply like `{"question": ..., "options": [...]}` instead of `{"questions": [...]}`
+is rejected with "Tool arguments do not satisfy the advertised schema". No extra
+generation.
+
+Temporary switch `TOOLCALL_LEGACY` (default `1`): `1` — main-era contract for
+**every** provider (`server/api.py` + `server/tool_protocol.py`: one block,
+DeepSeek-only repair, always `invalid_tool_response`; will be removed); `0` — one
+tool-rework path in `server/api_new.py` + `server/tool_protocol_new.py` for **every**
+provider, including DeepSeek (demote, multi-block, pointed repair,
+`TOOLCALL_INVALID_MODE`). Summary in
+[SETUP §5](docs/SETUP.en.md#5-why-the-model-doesnt-make-edits-emulated-tool-calling);
+longer notes (Russian):
+[`docs/toolcall-format-repair-legacy.md`](docs/toolcall-format-repair-legacy.md)
+(`LEGACY=1`) and
+[`docs/toolcall-format-repair.md`](docs/toolcall-format-repair.md) (`LEGACY=0`).
+With `TOOLCALL_LEGACY=0`, tool calls are handled like in any agent. Only a top-level
+fenced `tool_calls` block is a call, wherever it stands in the message. Prose, text
+after the block, several blocks, arrays without that fence (also inside a
+```` ```json ```` block) and quoted blocks are plain text. Before validate/repair the
+bridge may demote advice mistagged as a call: if a block has no call-level `name`
+(JSON whole or broken) or the name is unknown and there is accompanying prose, the
+opening fence is relabeled from `tool_calls` to `text` and the agent gets ordinary
+successful `content` — no repair and no `TOOLCALL_INVALID_MODE`. Otherwise every
+remaining block must be valid: broken JSON (including a known tool name), an unknown
+name with no prose, or arguments off-schema stop the whole message (repair /
+`TOOLCALL_INVALID_MODE`); there is no silent validate pass. When every block is
+valid, all calls run in order: the agent receives them as `tool_calls` and the rest
+of the message, with the call blocks cut out, as ordinary content.
+`tool_choice: none` never turns text into an error. Verified offline.
+
+With `TOOLCALL_FORMAT_REPAIR=1` (off by default), under `LEGACY=0` an erroneous call
+gets one pointed correction in the same chat (one extra generation): for an unknown
+tool the full list of tools with their argument shapes (field names, nesting, `?` =
+optional; not shortened), for wrong arguments that tool's argument shape (no values).
+No second correction is requested. If the corrected reply is valid the agent sees only
+the final call, with no notice. What happens next is `TOOLCALL_INVALID_MODE` (below).
+Under `LEGACY=1`, repair remains DeepSeek-only (main-era prompt). Verified offline.
+
+`TOOLCALL_INVALID_MODE` (default `error`; only when `LEGACY=0`): what to do with an
+invalid call after validation or after one failed correction — for **any** provider,
+including DeepSeek. With `error` the agent gets
+`invalid_tool_response` (fail-closed). With `text` — HTTP success and a live provider
+chat, but an explicit failed tool step for the agent:
+
+- `content`: raw reply plus `Bridge notice: BRIDGE_TOOL_NOT_EXECUTED: …` (tool name at
+  most; no schemas or values);
+- top-level `bridge_notice` (`executed: false`, `code`, `kind`, `tool`, `as_tool_error`
+  shaped like a tool-role error);
+- `message.refusal`: the same notice text (for clients that honor refusal);
+- the next user message is sent to the provider tab as
+  `Tool result (name): <notice>` plus the follow-up — as if the tool returned an error.
+
+Executable `tool_calls` are not invented on reject (the agent would run them). Even with
+`text`, these still error: `tool_choice` required/specific tool, a reply without a
+conversation id, and an unfinished reply. Mistagged advice (no call-level name, or
+unknown name with prose) is demoted to a `text` fence before this path.
+Verified offline.
+
+A call that repeats, with identical arguments, the call whose result the request
+carries (the model answers a tool result with the same call again) is not executed a
+second time: with `TOOLCALL_INVALID_MODE=text` the agent gets the same kind of message
+(raw reply plus a line saying the call repeats the one just executed); with `error` it
+raises. A different call, or a call after a new user message, runs normally. Verified
+offline.
+
+The tool result the agent returns (for example the option you picked) goes to the
+provider as just `Tool result (name): <result JSON>`. It carries no tools reminder
+(`TOOL_INSTRUCTIONS_BROWSER` still applies to ordinary user messages), no repeated
+question or call and no bridge call id; results of several calls to the same tool keep
+the identifier mapping. Invalid-call outcome follows `TOOLCALL_INVALID_MODE` (above).
+Verified offline.
+
+For DeepSeek, disabling `TOOLCALL_FORMAT_REPAIR` prevents an extra format-correction generation;
 it does not disable tool-markup validation. Malformed tool blocks stop the step
 instead of being returned as successful plain text.
 After a normal DeepSeek stop, commentary may precede one explicit terminal
@@ -251,9 +337,10 @@ fences and quotations do not become calls. Prose quotations may precede a separa
 top-level `tool_calls` block with a blank line before its opening fence. Quoted,
 nested and ambiguous code blocks are not executed; `>` markers are never stripped
 from code. An explicit `tool_calls` block is an
-action protocol: do not use it for explanatory examples. Remote schemas are never
-fetched; local `$ref` references work. On update, install `requirements.txt`
-(the new dependency is `jsonschema`) and restart the bridge.
+action protocol: do not use it for explanatory examples. Browser providers follow the
+rules above instead. Remote schemas are never fetched; local `$ref` references work.
+On update, install `requirements.txt` (the new dependency is `jsonschema`) and
+restart the bridge.
 
 A normal stop with a closed fence also permits swapping the final `]}` to `}]`
 only when a string-aware scan confirms that arguments are complete and only the
@@ -422,7 +509,10 @@ DeepSeek defaults are **12/60 generations, 3/6 new chats and 40/200 HTTP request
 per hour/day**, with a shared 20 accepted replies per conversation
 (`CONVERSATION_MAX_REPLIES`) and a 30-second interval. `DEEPSEEK_MAX_*` budget
 ceilings and `CONVERSATION_MAX_REPLIES` belong to the bridge process environment.
-Raising a local ceiling does not raise the website's allowance.
+Raising a local ceiling does not raise the website's allowance. `PROVIDER_ACCESS_PAUSE=0`
+disables provider pause and pending blocks, including persisted ones; it does not
+bypass budgets, provider quotas or conversation-history safeguards, and can allow
+retries after an uncertain outcome.
 [Configuration and error guide](docs/HARNESS.en.md#local-limits-and-errors).
 
 ## Connect OpenCode
@@ -431,6 +521,8 @@ Use the same bridge and account setup above. Put the example
 [`opencode.json`](examples/opencode.json) in your project. For existing configs,
 merge settings while preserving your `provider`, `model` and `small_model`.
 Set `small_model` explicitly: title generation can also consume messages.
+Do not point title/`small_model` at a browser model — that opens a second tab
+chat (Harness disables `session-title-llm` in its profile).
 
 ```bash
 cd /path/to/your-project
@@ -474,7 +566,9 @@ and reset time have not been established.
 **Harness or OpenCode runs tools on your computer.** Prompts and tool-read data
 are sent to the selected provider. The bridge validates responses and
 `tool_calls`, saves conversation state and sends new tool results to the existing
-chat. Unknown history stops instead of silently creating a new chat.
+chat. Continuation matching ignores client `system` prompts and OpenCode
+`<system-reminder>` wrappers (they change between turns). Unknown or already
+consumed history stops instead of silently creating a new chat.
 
 Refusal pauses survive restart. Client setup, ordinary sign-in and local budget
 reset do not clear them. Safari transfer also preserves the entire pause, pending
@@ -495,6 +589,14 @@ the tab, not to an authenticated answer source. DeepSeek/Qwen streams stop after
 300 seconds total or 15 seconds without bytes; such a stop pauses the provider
 and is not replayed. A first request (no assistant message yet) has no history
 link, so resending it can open a second chat; continuations are one-use.
+Set `CONVERSATION_CLEAR_USED_ON_STARTUP=1` in the bridge process environment to
+delete consumed/closed history links at startup; the default `0` preserves them.
+A repeated first request with exactly identical history otherwise has the same
+cache key, so the bridge will not open another provider chat for it while the link
+is retained. It may reuse an unused link or reject an already-consumed one. Deleting
+used links allows a repeat to start another upstream chat,
+but a later continuation may bind to the wrong chat and fail. Links with
+`used=0` are not deleted.
 
 ## OpenCode demonstrations
 
@@ -507,6 +609,18 @@ These are **not Harness screenshots** or proof of current account availability.
 
 [GLM/Kimi captures and recording method](docs/media/README.md).
 
+A repeated initial request with **exactly identical history** has the same cache key.
+While its link is retained, the bridge does not open a new provider chat for that
+history: it may reuse a still-unused link or reject a repeat whose link was already
+consumed. By default consumed/closed links are retained to avoid binding a repeat to
+another provider chat. With `CONVERSATION_CLEAR_USED_ON_STARTUP=1`, startup deletes
+only consumed links (`used=1`), so a repeated prompt may create a new upstream chat;
+a later continuation may then bind to the wrong chat. For a complete manual
+cleanup, run `python -m server.conversation_admin -clear`: it deletes all `links`,
+associated log-path mappings (`trace_links`), and migration records (`rollovers`)
+from `session/deepseek-conversations.sqlite3`. It does not delete log files or chats
+at the provider. This operation is destructive.
+
 ## Documentation and development
 
 - [Harness: setup, sessions, budgets, offline verification](docs/HARNESS.en.md)
@@ -526,7 +640,7 @@ Claude requires a network terminal marker; truncated, partial or malformed answe
 not reported as complete. Uncertain dispatched outcomes retain pending state
 and a safety pause across restart. There is no automatic retry.
 Verified offline without live accounts. Re-export and update the userscripts
-to apply the browser changes (controller 0.3.34).
+to apply the browser changes (controller 0.3.35).
 
 Mistral accepts only a request-bound stream with a terminal trailer.
 Harness `--without-browser` excludes browser models and their aliases.