feat: 保留进程崩溃诊断日志并升级至 2.0.7

README.md

@@ -704,6 +704,16 @@ Example output:
 
 ---
 
+### Process diagnostics (2.0.7)
+
+In addition to the existing business logs, `console.log` mirrors Python stdout/stderr
+(including Uvicorn errors and tracebacks), `process-events.jsonl` records process
+startup, exit codes, signals and unhandled exceptions, and `crash.log` captures
+Python fatal fault stacks. These files are written alongside `--log-file`.
+The console and process-event files rotate by size; see
+[process diagnostics](docs/process-diagnostics-2.0.7.md) for details.
+**Console logs may contain sensitive data: review before sharing.**
+
 ### Logging
 
 Logs include:

README_zh.md

@@ -769,6 +769,15 @@ grep projection_applied "$HOME/.workbuddy2api/codebuddy-proxy.jsonl" | jq .
 
 ---
 
+### 进程诊断（2.0.7）
+
+除原有业务日志外，`console.log` 留存 Python stdout/stderr（包括 Uvicorn 异常和
+traceback），`process-events.jsonl` 记录进程启动、退出码、信号及未处理异常，
+`crash.log` 用于 Python 底层致命错误堆栈。文件均存放在 `--log-file` 所在目录。
+控制台和进程事件日志按大小轮转。详细说明见
+[进程诊断指南](docs/process-diagnostics-2.0.7.md)。
+**控制台日志可能包含敏感信息，公开前请先脱敏。**
+
 ### 日志
 
 日志包含：

docs/process-diagnostics-2.0.7.md

@@ -0,0 +1,70 @@
+# Process diagnostics (2.0.7)
+
+This release adds process-level evidence for unattended proxy failures without
+changing the HTTP/LLM business logic. Files use the directory of `--log-file`
+(default `~/.workbuddy2api/`).
+
+## Files
+
+| File | Content | Retention |
+| --- | --- | --- |
+| `console.log` | Python-level stdout/stderr, including ordinary tracebacks and Uvicorn error logs, while still displaying them in the terminal | 20 MiB per file, 4 rotations (up to ~100 MiB) |
+| `process-events.jsonl` | Structured process start, exit code/reason, caught signal, fatal exception class, memory measurements, unhandled asyncio/thread exceptions and diagnostic log failures | 2 MiB per file, 3 rotations |
+| `crash.log` | `faulthandler` native/Python thread stacks for fatal Python runtime signals | Append-only; inspect/truncate manually after incidents |
+| `proxy.log`, `codebuddy-proxy.jsonl` | Existing request, proxy and operational logs (no semantic changes) | Existing daily retention |
+
+The new files are created on startup, and on POSIX new files use mode `0600`.
+On Windows their ACL is inherited from the containing directory; protect that
+directory if multiple users share the machine.
+
+**Warning:** Unlike the structured process events, console and crash outputs
+can include exception content, upstream fragments, tokens or other private data
+printed by dependencies. Do not attach raw files to public issues without
+reviewing and redacting them.
+
+## How to inspect an incident
+
+1. Find the last `process_start` in `process-events.jsonl` and its `run_id`.
+2. Check for `process_fatal_exception`, `process_signal_received`,
+   `asyncio_unhandled_exception`, `thread_unhandled_exception`, or
+   `diagnostic_log_failure` for that run.
+3. Match with its `process_exit` and `exit_code`. A normal graceful
+   `uvicorn.run()` return records code `0`. A caught uncaught exception
+   records `1`, and SIGTERM can record `143` (128+15).
+4. Read `console.log` and `crash.log` for the traceback, without relying on
+   exception messages in the JSONL journal.
+5. If the process terminates abruptly (e.g. Windows TerminateProcess, kill -9,
+   host shutdown), Python cannot run an exit handler. On its **next** launch,
+   `previous_run_exit_unobserved` records the previous PID/run ID if the
+   most recent start has no matching exit. This is not proof of a crash;
+   simultaneous proxy instances can also result in an unmatched run.
+
+## Signal and output capture limitations
+
+Signals handled by Uvicorn are observed with a delegating wrapper so its
+original signal behavior runs first. SIGTERM is recorded even if Uvicorn
+gracefully stops and replays the signal. Windows force termination bypasses
+Python handlers, so it is detectable only retrospectively. The Python tee
+captures `sys.stdout` and `sys.stderr`; writes directly to native file
+descriptors, subprocess handles or the original binary stream are not fully
+mirrored. Use a terminal/supervisor-level redirection if those must also be
+preserved.
+
+Logs are best-effort by design: a failed console or JSONL write is reported
+in `process-events.jsonl` if possible and to the original stderr without
+crashing the proxy. If the entire log volume becomes inaccessible, diagnostics
+may be incomplete.
+
+## Version
+
+`pyproject.toml` and FastAPI metadata are `2.0.7`. From a source checkout,
+run `uv sync` before starting to refresh installed distribution metadata
+used by the runtime startup event. The current running proxy is not restarted
+by implementing this change.
+
+## Tests
+
+`test_process_diagnostics.py` exercises console mirroring, rotation, normal
+and abnormal exits, SIGTERM, forced child-process termination, unobserved exit
+detection, traceback retention, asyncio/thread exception forwarding, memory
+snapshots, log failure fallback, and the version contract.

pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "workbuddy2api"
-version = "2.0.6"
+version = "2.0.7"
 description = "Local OpenAI/Responses/Anthropic compatible proxy for CodeBuddy"
 readme = "README.md"
 requires-python = ">=3.10"

src/codebuddy_proxy/__main__.py

@@ -50,6 +50,7 @@
 from codebuddy_proxy.billing_proxy import forward_billing
 from codebuddy_proxy.upstream_request import ChatRequestPreparer
 from codebuddy_proxy.protocol_response import format_nonstream_response
+from codebuddy_proxy.process_diagnostics import ProcessDiagnostics
 
 # 尝试导入高级功能模块（可选）
 try:
@@ -93,6 +94,17 @@ def anthropic_to_chat(body):
 # 日志配置
 # ============================================================================
 
+class ObservedTimedRotatingFileHandler(logging.handlers.TimedRotatingFileHandler):
+    """Make otherwise-suppressed Python logging handler failures observable."""
+
+    def handleError(self, record: logging.LogRecord) -> None:
+        monitor = globals().get("process_diagnostics")
+        exc = sys.exc_info()[1]
+        if monitor is not None and exc is not None:
+            monitor.log_failure("file_logger", exc)
+        super().handleError(record)
+
+
 def setup_logging(log_dir: pathlib.Path) -> logging.Logger:
     """配置滚动日志：按天分片，保留30天。"""
     log_dir.mkdir(parents=True, exist_ok=True)
@@ -103,7 +115,7 @@ def setup_logging(log_dir: pathlib.Path) -> logging.Logger:
     logger.propagate = False
     logger.handlers.clear()
 
-    handler = logging.handlers.TimedRotatingFileHandler(
+    handler = ObservedTimedRotatingFileHandler(
         log_file, when='midnight', interval=1, backupCount=30, encoding='utf-8'
     )
     handler.suffix = "%Y-%m-%d"
@@ -124,7 +136,7 @@ def setup_json_logging(log_file: pathlib.Path) -> logging.Logger:
     logger.propagate = False
     logger.handlers.clear()
 
-    handler = logging.handlers.TimedRotatingFileHandler(
+    handler = ObservedTimedRotatingFileHandler(
         log_file, when="midnight", interval=1, backupCount=30, encoding="utf-8"
     )
     handler.suffix = "%Y-%m-%d"
@@ -211,8 +223,10 @@ def write_log(self, event: str, **kwargs) -> None:
                 **kwargs,
             }
             self.json_logger.info(json.dumps(record, ensure_ascii=False))
-        except Exception:
-            pass
+        except Exception as exc:
+            monitor = globals().get("process_diagnostics")
+            if monitor is not None:
+                monitor.log_failure("jsonl_write", exc)
 
     def write_body_log(self, event: str, body: bytes, **kwargs) -> None:
         if self.json_logger is None:
@@ -228,18 +242,31 @@ def write_body_log(self, event: str, body: bytes, **kwargs) -> None:
                 **kwargs
             }
             self.json_logger.info(json.dumps(record, ensure_ascii=False))
-        except Exception:
-            pass
+        except Exception as exc:
+            monitor = globals().get("process_diagnostics")
+            if monitor is not None:
+                monitor.log_failure("jsonl_body_write", exc)
 
 
 # ============================================================================
 # FastAPI 应用
 # ============================================================================
 
-app = FastAPI(title="CodeBuddy Proxy (FastAPI)", version="2.0")
+app = FastAPI(title="CodeBuddy Proxy (FastAPI)", version="2.0.7")
 
 # 全局状态（在 main() 中初始化）
 proxy_state: ProxyState | None = None
+process_diagnostics: ProcessDiagnostics | None = None
+
+
+async def install_process_asyncio_diagnostics() -> None:
+    """Capture unhandled background task errors in the active Uvicorn loop."""
+    if process_diagnostics is not None:
+        process_diagnostics.install_asyncio(asyncio.get_running_loop())
+        process_diagnostics.install_signal_observers()
+
+
+app.router.add_event_handler("startup", install_process_asyncio_diagnostics)
 
 
 def get_state() -> ProxyState:
@@ -1990,10 +2017,8 @@ def parse_args(argv: list[str] | None = None) -> argparse.Namespace:
     return args
 
 
-def main():
+def _run_proxy(args: argparse.Namespace) -> int:
     global proxy_state
-
-    args = parse_args()
     try:
         resolved_backend = resolve_backend_from_args(args)
         client = CodeBuddyClient(backend=resolved_backend)
@@ -2068,6 +2093,19 @@ def main():
 
     # 启动 uvicorn
     run_http_server(host=args.host, port=args.port, windows_loop=args.windows_loop)
+    return 0
+
+
+def main() -> int:
+    global process_diagnostics
+    args = parse_args()
+    monitor = ProcessDiagnostics(args.log_file.parent)
+    process_diagnostics = monitor
+    try:
+        return monitor.run(lambda: _run_proxy(args))
+    finally:
+        process_diagnostics = None
+
 
 if __name__ == "__main__":
     raise SystemExit(main())