Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions deploy/kubernetes/BETA_CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -76,3 +76,22 @@ Publish matching Core, provider helpers and a newly built Runtime template toget
| Required release assets | Matching Core/Web release; retain the selected Runtime template |

History sampling observes Worker ownership without using its short-lived contexts on the execution lease connection. Execution retains authoritative lease checks and fails closed on ownership loss. Worker shutdown immediately invalidates the sampling observation. Lease failure logs identify cancellation, timeout and connection state without SQL or error text, and Worker failure logs identify the exiting stage. [Runtime observability](../../contracts/agents-api/runtime-observability.md#sampling-ownership) owns these rules.


## 2026-10-07 — Environment file rejection diagnostics

| Item | Value |
| --- | --- |
| Fork beta baseline | `8aebb6c8bf531b90828bc7efd369f4e49535c67a` |
| Feature branch | `codex/core-safe-http-rejection-diagnostics` |
| Integrated source commit | `914e1d359b0bc0e3dd6f6913313481a5abb7e509` |
| Integration | Cherry-pick; translated operations source hash recomputed against the merged English document |
| Schema migrations / DDL / SQL changes | None |
| Runtime bootstrap / provider helpers / node wire / native pins | Unchanged |
| Required release assets | Core and Web built from the same integrated beta commit; retain the selected Runtime template |

The Environment file creation handler adds request- and trace-correlated static rejection diagnostics without changing API responses or execution. [Operations](../../docs/getting-started/operations.md#environment-file-rejection-logs) owns the event fields and coverage. Existing beta readiness and runtime behavior remain in place.

Deploy through **core-deploy** from workflow branch `main`, selecting `source_branch=beta` and the full integrated commit already merged into beta, as described in the [Kubernetes guide](README.md#deploy). This replaces the existing production Core and Web; it does not create a separate beta environment. No Runtime template build or activation is required by this diagnostic change.

Verify both rollout revisions and ready Service endpoints, then check public health and authenticated API behavior. A safe rejected Environment file request can establish that the new event is correlated with its response IDs; it does not prove a successful upload or identify the cause of earlier 400 responses. This entry records integration requirements, not deployment success. Do not replay uncertain uploads or create paid execution solely to produce diagnostic evidence.
8 changes: 8 additions & 0 deletions docs/getting-started/operations.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,14 @@ docker compose -f "$HOME/.oac/core/compose.yaml" logs --tail 200 core

Don't paste `docker compose config`, `docker inspect` or raw logs into public issue reports.

### Environment file rejection logs

The Environment file creation handler emits one `api_request_rejected` event when it writes a response with HTTP status 400 or higher. Match its `request_id` to the response's `X-Request-Id`; Core generates this ID, so it need not match a caller's request ID. The existing log context also supplies `trace_id` and `span_id`, matching the response's `traceparent` whether the request carries a valid incoming trace or Core generates a new trace. This event covers `operation=environment_file_create` and the fixed route template `/v1/agents/environments/{environment_id}/files`. Requests rejected before this handler are outside its coverage.

The event contains `operation`, `route`, `status`, an allowlisted `code` (unrecognized codes become `unknown`), and a static `reason`. Reasons are `unknown`, `invalid_payload`, `unknown_field`, `invalid_path`, `invalid_base64`, `inline_too_large`, `hosted_environment_provisioning`, `destination_directory`, and `destination_unsafe`. `unknown` means the rejection has no classified reason; a 400 alone does not establish that an Environment is still provisioning.

The event does not record request or response bodies, query strings, raw URL paths, resource IDs, user file paths, field values, arbitrary error messages or credentials. It does not change the HTTP response or execution behavior. Check that the deployed Core version includes this diagnostic before relying on it; missing events do not establish success, and diagnostics alone do not show that file uploads work.

## Stop and restart

Let active work settle before a planned restart:
Expand Down
10 changes: 9 additions & 1 deletion docs/zh/getting-started/operations.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
title: "管理你的安装"
source: docs/getting-started/operations.md
source_hash: 06de77447d1647b7ab546bd41bc2c4c042abdb47053d20b0a4bdf25465bca2f7
source_hash: 9012f887f10171fb2abe27dc49770d429485673559c0d9661bd2100a34adce5f
---

安装运维人员负责 Core 主机、存储和可用性。节点主机运行各自的服务;参阅[节点](nodes.md)。设置见[配置参考](../configuration.md)。
Expand Down Expand Up @@ -47,6 +47,14 @@ docker compose -f "$HOME/.oac/core/compose.yaml" logs --tail 200 core

不要将 `docker compose config`、`docker inspect` 或原始日志粘贴到公开问题报告。

### Environment 文件拒绝日志 {#environment-file-rejection-logs}

Environment 文件创建 handler 写出HTTP状态码为400或以上的响应时,会发出一条 `api_request_rejected` 事件。用其中的 `request_id` 关联响应的 `X-Request-Id`;该ID由Core生成,不一定与调用方的请求ID相同。既有日志上下文还会注入 `trace_id` 和 `span_id`;无论请求携带有效的入站trace,还是由Core生成新trace,这两个字段均与响应的 `traceparent` 一致。事件覆盖 `operation=environment_file_create`,路由固定为模板 `/v1/agents/environments/{environment_id}/files`。进入该handler之前被拒绝的请求不在其覆盖范围内。

事件包含 `operation`、`route`、`status`、白名单内的 `code`(未识别的code记为 `unknown`)以及静态 `reason`。reason取值为 `unknown`、`invalid_payload`、`unknown_field`、`invalid_path`、`invalid_base64`、`inline_too_large`、`hosted_environment_provisioning`、`destination_directory` 和 `destination_unsafe`。`unknown` 表示拒绝原因尚未分类;仅凭400不能判定Environment仍在准备中。

事件不记录请求或响应正文、查询字符串、原始URL路径、资源ID、用户文件路径、字段值、任意错误文案或凭据,也不改变HTTP响应和执行行为。使用前应核实已部署的Core版本包含此诊断;缺少事件不能证明成功,诊断信息本身也不能证明文件上传可用。

## 停止与重启 {#stop-and-restart}

计划重启前,先等待活动工作结束:
Expand Down
92 changes: 92 additions & 0 deletions services/core/internal/api/api_rejection.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
package api

import (
"context"
"net/http"

"github.com/MiniMax-AI/OpenAgentCore/internal/obs/log"
)

// Reasons are assigned only at explicit rejection branches, never from error
// messages or request data. The zero value leaves an unclassified failure unknown.
type apiRejectionReason uint8

const (
rejectionUnknown apiRejectionReason = iota
rejectionInvalidPayload
rejectionUnknownField
rejectionInvalidPath
rejectionInvalidBase64
rejectionInlineTooLarge
rejectionHostedEnvironmentProvisioning
rejectionDestinationDirectory
rejectionDestinationUnsafe
)

func (reason apiRejectionReason) String() string {
switch reason {
case rejectionInvalidPayload:
return "invalid_payload"
case rejectionUnknownField:
return "unknown_field"
case rejectionInvalidPath:
return "invalid_path"
case rejectionInvalidBase64:
return "invalid_base64"
case rejectionInlineTooLarge:
return "inline_too_large"
case rejectionHostedEnvironmentProvisioning:
return "hosted_environment_provisioning"
case rejectionDestinationDirectory:
return "destination_directory"
case rejectionDestinationUnsafe:
return "destination_unsafe"
default:
return "unknown"
}
}

// The existing response writer owns diagnostics as well as response timing.
// Unwrapping preserves the controller's deadline/flush path and adds no wrapper.
// A non-nil context enables the diagnostic using the handler context after
// tracing middleware has run. Later rejection branches pass nil to only mark a reason.
func environmentFileDiagnostic(w http.ResponseWriter, ctx context.Context, reason apiRejectionReason) {
for w != nil {
if writer, ok := w.(*processingTimeWriter); ok {
if !writer.stamped && (ctx != nil || writer.environmentFile) {
if ctx != nil {
writer.ctx = ctx
}
writer.environmentFile = true
writer.rejectionReason = reason
}
return
}
unwrapper, ok := w.(interface{ Unwrap() http.ResponseWriter })
if !ok {
return
}
w = unwrapper.Unwrap()
}
}

func safeAPIErrorCode(code string) string {
switch code {
case "invalid_request", "invalid_request_error", "not_found_error", "request_too_large", "conflict_error", "server_error", "execution_unavailable":
return code
default:
return "unknown"
}
}

func (w *processingTimeWriter) logRejection(status int) {
if !w.environmentFile || status < 400 || w.stamped {
return
}
log.Warn(w.ctx, "api_request_rejected",
"operation", "environment_file_create",
"route", "/v1/agents/environments/{environment_id}/files",
"status", status,
"code", safeAPIErrorCode(w.errorCode),
"reason", w.rejectionReason.String())
}
221 changes: 221 additions & 0 deletions services/core/internal/api/api_rejection_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,221 @@
package api

import (
"bytes"
"encoding/json"
"fmt"
"log/slog"
"net/http"
"net/http/httptest"
"reflect"
"strings"
"testing"
"time"

obslog "github.com/MiniMax-AI/OpenAgentCore/internal/obs/log"
"github.com/MiniMax-AI/OpenAgentCore/services/core/internal/execution"
)

func rejectionLogBuffer(t *testing.T) *bytes.Buffer {
t.Helper()
var output bytes.Buffer
previous := slog.Default()
slog.SetDefault(slog.New(obslog.NewContextHandler(slog.NewJSONHandler(&output, nil))))
t.Cleanup(func() { slog.SetDefault(previous) })
return &output
}

func rejectionRecords(t *testing.T, output *bytes.Buffer) []map[string]any {
t.Helper()
var records []map[string]any
for _, line := range strings.Split(strings.TrimSpace(output.String()), "\n") {
if line == "" {
continue
}
var record map[string]any
if err := json.Unmarshal([]byte(line), &record); err != nil {
t.Fatal(err)
}
if record["msg"] == "api_request_rejected" {
records = append(records, record)
}
}
return records
}

func TestEnvironmentFileRejectionDiagnostics(t *testing.T) {
output := rejectionLogBuffer(t)
const canary = "private-canary-value"
const valid = `{"type":"inline","data":"YWJj","path":"/workspace/private-canary-value"}`
for _, tc := range []struct {
name, body, reason, code, message string
status int
err error
pending bool
param string
}{
{name: "payload", body: `{"type":"inline","path":"/workspace/private-canary-value"}`, reason: "invalid_payload", code: "invalid_request", message: invalidInputMessage, status: 400},
{name: "unknown field", body: `{"type":"inline","data":"YWJj","path":"/workspace/a","private-canary-value":"secret"}`, reason: "unknown_field", code: "invalid_request_error", message: "Unknown parameter: 'private-canary-value'.", status: 400, param: canary},
{name: "path", body: `{"type":"inline","data":"YWJj","path":"/private-canary-value"}`, reason: "invalid_path", code: "invalid_request_error", message: errEnvironmentFileCreatePath.message, status: 400},
{name: "base64", body: `{"type":"inline","data":"private-canary-value","path":"/workspace/a"}`, reason: "invalid_base64", code: "invalid_request", message: invalidInputMessage, status: 400},
{name: "inline size", body: inlineCreateBody((5 << 20) + 1), reason: "inline_too_large", code: "invalid_request_error", message: errEnvironmentFileInlineTooLarge.message, status: 400},
{name: "provisioning", body: `{"type":"file_id","file_id":"private-canary-value","path":"/workspace/private-canary-value"}`, reason: "hosted_environment_provisioning", code: "invalid_request_error", message: errHostedEnvironmentProvisioning.message, status: 400, pending: true},
{name: "directory", body: valid, reason: "destination_directory", code: "invalid_request_error", message: errEnvironmentFileConflict.message, status: 400, err: fmt.Errorf("%s: %w", canary, execution.ErrEnvironmentFileDirectory)},
{name: "unsafe", body: valid, reason: "destination_unsafe", code: "invalid_request_error", message: errEnvironmentFileUnsafe.message, status: 400, err: fmt.Errorf("%s: %w", canary, execution.ErrEnvironmentFileUnsafe)},
{name: "unknown", body: valid, reason: "unknown", code: "execution_unavailable", message: "Execution is not available on this service.", status: 503, err: fmt.Errorf("%s: %w", canary, execution.ErrExecutionUnavailable)},
} {
t.Run(tc.name, func(t *testing.T) {
output.Reset()
unavailable := 0
handler, f := environmentFileCreateHandler(t, countEnvironmentFilesUnavailable(&unavailable))
f.err = tc.err
if tc.pending {
f.environment.Status = "pending"
}
request := httptest.NewRequest(http.MethodPost, "/v1/agents/environments/"+f.environment.ID+"/files?secret="+canary, strings.NewReader(tc.body))
request.Header.Set("Authorization", "Bearer files-key")
request.Header.Set("OpenAI-Beta", "agents=v1")
request.Header.Set("Content-Type", "application/json")
request.Header.Set("X-Request-Id", canary)
response := httptest.NewRecorder()
handler.ServeHTTP(environmentFilesRecorder{response}, request)
expected := httptest.NewRecorder()
var param []string
if tc.param != "" {
param = []string{tc.param}
}
writeError(expected, tc.status, tc.code, tc.message, param...)
if response.Code != expected.Code || response.Body.String() != expected.Body.String() {
t.Fatalf("response changed: %d %s; expected %d %s", response.Code, response.Body, expected.Code, expected.Body)
}
records := rejectionRecords(t, output)
if len(records) != 1 {
t.Fatalf("expected one diagnostic, got %d", len(records))
}
record := records[0]
want := map[string]any{"operation": "environment_file_create", "route": "/v1/agents/environments/{environment_id}/files", "status": float64(tc.status), "code": tc.code, "reason": tc.reason, "request_id": response.Header().Get("X-Request-Id")}
carrier, err := obslog.ParseTraceparent(response.Header().Get(obslog.HeaderName))
if err != nil {
t.Fatal(err)
}
want["trace_id"], want["span_id"] = carrier.Trace.String(), carrier.Span.String()
for key, value := range want {
if record[key] != value {
t.Fatalf("%s=%v want %v", key, record[key], value)
}
}
if !requestIDPattern.MatchString(want["request_id"].(string)) {
t.Fatal("missing server request ID")
}
for key := range record {
if key != "time" && key != "level" && key != "msg" {
if _, ok := want[key]; !ok {
t.Fatalf("unapproved diagnostic field: %s", key)
}
}
}
for _, secret := range []string{canary, "files-key", f.environment.ID, f.environment.TenantID} {
if strings.Contains(output.String(), secret) {
t.Fatalf("sensitive data leaked: %s", secret)
}
}
})
}
}

func TestRejectionDiagnosticsPreserveResponseWriter(t *testing.T) {
output := rejectionLogBuffer(t)
for _, stream := range []bool{false, true} {
output.Reset()
out := &detailDeadlineWriter{ResponseRecorder: httptest.NewRecorder()}
var observed []string
handler := responseHeadersWithErrors(coreErrorResponses(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
environmentFileDiagnostic(w, r.Context(), rejectionUnknown)
deadline := time.Unix(1234, 0)
if err := http.NewResponseController(w).SetWriteDeadline(deadline); err != nil {
t.Fatal(err)
}
if stream {
w.Header().Set("Content-Type", "text/event-stream")
if err := http.NewResponseController(w).Flush(); err != nil {
t.Fatal(err)
}
_, _ = w.Write([]byte("data: safe\n\n"))
environmentFileDiagnostic(w, nil, rejectionInvalidPath)
reportAPIError(w, "invalid_request_error")
} else {
writeJSON(w, 201, map[string]string{"result": "safe"})
}
})), func(code string) { observed = append(observed, code) })
handler.ServeHTTP(out, httptest.NewRequest("POST", "/fixture", nil))
if !out.deadline.Equal(time.Unix(1234, 0)) || out.Header().Get("Openai-Processing-Ms") == "" || len(observed) != 0 || len(rejectionRecords(t, output)) != 0 {
t.Fatal("success/stream behavior changed")
}
if stream && (!out.Flushed || out.Body.String() != "data: safe\n\n") {
t.Fatal("SSE changed")
}
}
output.Reset()
var observed []string
handler := responseHeadersWithErrors(coreErrorResponses(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
environmentFileDiagnostic(w, r.Context(), apiRejectionReason(255))
writeError(w, 400, "secret-code-canary", "secret-message-canary", "secret-param-canary")
reportAPIError(w, "late-code")
w.WriteHeader(500)
})), func(code string) { observed = append(observed, code) })
response := httptest.NewRecorder()
handler.ServeHTTP(response, httptest.NewRequest("POST", "/secret-path-canary", nil))
records := rejectionRecords(t, output)
if len(records) != 1 || records[0]["code"] != "unknown" || records[0]["reason"] != "unknown" || strings.Contains(output.String(), "canary") || !reflect.DeepEqual(observed, []string{"secret-code-canary"}) {
t.Fatal("observer/allowlist changed")
}
}

func TestEnvironmentFileDiagnosticsStayWithinCreateHandler(t *testing.T) {
output := rejectionLogBuffer(t)
h, f := environmentFileCreateHandler(t)
f.environment.Status = "pending"
if response := requestEnvironmentFiles(h, f.environment.ID, "", "files-key"); response.Code != 400 {
t.Fatal("expected provisioning rejection")
}
if response := requestCreateEnvironmentFile(h, f.environment.ID, `{"type":"inline","data":"","path":"/workspace/a"}`, ""); response.Code != 401 {
t.Fatal("expected authentication rejection")
}
if len(rejectionRecords(t, output)) != 0 {
t.Fatal("diagnostic escaped create handler scope")
}
f.environment.Status = "connected"
if response := requestCreateEnvironmentFile(h, f.environment.ID, `{"type":"inline","data":"","path":"/workspace/a"}`, "files-key"); response.Code != 201 || f.writes != 1 {
t.Fatal("success changed")
}
if len(rejectionRecords(t, output)) != 0 {
t.Fatal("success logged as rejection")
}
}

func TestEnvironmentFileDiagnosticUsesHandlerTraceContext(t *testing.T) {
output := rejectionLogBuffer(t)
const inbound = "00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01"
for _, parent := range []string{"", inbound} {
output.Reset()
h, f := environmentFileCreateHandler(t)
f.environment.Status = "pending"
r := httptest.NewRequest("POST", "/v1/agents/environments/"+f.environment.ID+"/files", strings.NewReader(`{"type":"inline","data":"","path":"/workspace/a"}`))
r.Header.Set("Authorization", "Bearer files-key")
r.Header.Set("OpenAI-Beta", "agents=v1")
r.Header.Set(obslog.HeaderName, parent)
out := httptest.NewRecorder()
h.ServeHTTP(environmentFilesRecorder{out}, r)
records := rejectionRecords(t, output)
carrier, err := obslog.ParseTraceparent(out.Header().Get(obslog.HeaderName))
if err != nil || out.Code != 400 || len(records) != 1 {
t.Fatalf("missing response/diagnostic trace: %v", err)
}
if parent != "" && out.Header().Get(obslog.HeaderName) != parent {
t.Fatal("inbound carrier changed")
}
if records[0]["request_id"] != out.Header().Get("X-Request-Id") || records[0]["trace_id"] != carrier.Trace.String() || records[0]["span_id"] != carrier.Span.String() {
t.Fatal("diagnostic did not use the traced handler context")
}
}
}
1 change: 1 addition & 0 deletions services/core/internal/api/environment_files.go
Original file line number Diff line number Diff line change
Expand Up @@ -102,6 +102,7 @@ func environmentFilesAccessible(w http.ResponseWriter, environment sessions.Envi
Type string `json:"type"`
}
if environment.Status == "pending" && json.Unmarshal(environment.Configuration, &configuration) == nil && configuration.Type == "openai_hosted" {
environmentFileDiagnostic(w, nil, rejectionHostedEnvironmentProvisioning)
writeFieldError(w, errHostedEnvironmentProvisioning)
return false
}
Expand Down
Loading
Loading