diff --git a/.agents/skills/loom-framework/SKILL.md b/.agents/skills/loom-framework/SKILL.md index e95df8d71..14dffd3ad 100644 --- a/.agents/skills/loom-framework/SKILL.md +++ b/.agents/skills/loom-framework/SKILL.md @@ -75,6 +75,43 @@ consumer validation. ## Architecture Boundaries +- `admission.Rate` serializes the maintained token bucket's Allow and complete + rate/burst policy updates. Preserve accrued allowance, clamp on reductions and + never reset quota on reconfiguration. `RateAdmission` owns bounded cooperative + backend checks and safe failures; HTTP and protected endpoint adapters retain + their respective placement and authored error contracts. Grouping, key storage, + distributed policy and retries remain application-owned. Keep the rate model, + fractional accounting tests and generated example aligned. + +- `admission.Concurrency` owns local unary permits, live capacity updates and + coherent counters under one mutex. Endpoint exit releases the permit; + cancellation and resizing never do. Existing protected endpoint composition + owns validation/access ordering, and authored errors own transport mappings. + Keep the bounded model under `admission/tla/concurrency` and the runnable + generated example aligned. Streaming lifetime, distributed admission, + adaptive policy, configuration endpoints and metrics backends remain outside + this helper. + +- Reusable OpenAPI body, response, and header references share one typed importer + walk. Resolve onto occurrence copies before planning; keep component identity + and annotation edges separate from baseline values. The IR allocates authored + identities before inferred components and reserves their names. Versioned + serializers preserve absent versus explicitly empty reference annotations. + Keep the reusable-reference model and cross-kind round-trip tests aligned. + + +- OpenAPI import normalizes reference assertions with the target's carrier + context, retaining separate reference identity and local constraints. Persist + rejected intersections for planner and renderer admission. Normalize effective + target nullability through membership before supplying carrier context. + Object overlays use Extend; CompositionBases retains declaration identity for + the OpenAPI allOf reference marker, while the composed shape owns physical + fields. Preserve closed-object key sets and do not remove inherited nullability. + Object field facts + distinguish semantic Required from LayoutRequired on the physical declaration; + named occurrence requirements validate the existing carrier. Keep the + reference-sibling model and generated required-field regression aligned. + - `expr.IsNullable` supplies effective nullability to prepared JSON schema matching and lowered `JSONShape` descriptions. Nullable null bypasses value predicates, including enums, while ordinary Any null still obeys its enum. @@ -340,7 +377,21 @@ consumer validation. IR consume this projection; keep required scopes on each requirement's fresh scheme data and preserve them through Dup. Globally deduplicated transport schemes do not own OR-branch authentication decisions. Keep union-based - unknown-scope admission and OpenAPI's protocol-specific scope-array rules. + unknown-scope admission. SchemeRequirements binds explicit names to scheme + identity: OAuth scopes must be declared; non-OAuth roles need no inventory. + ScopeFor and RoleFor add to shared Scope requirements. Deep-copy these arrays + on inheritance and include them in requirement identity. Import planning owns + normalized requirement validation; OpenAPI exports both OAuth scope arrays and + non-OAuth role arrays. Keep expr/tla/security_requirements and the generated + credential regressions aligned; role assignment remains application-owned. + +- `expr.SchemeExpr.FlowScopes` owns effective OAuth inventories: nil inherits + scheme defaults, nonnil empty replaces them. ScopeNames projects available + names only for requirements and callback discovery. Import, OpenAPI export + and generated OAuthFlow maps retain each flow's names and descriptions. When + any flow overrides defaults, resolve every runtime flow explicitly so the + available-name union cannot leak into inherited inventories. Keep the bounded + model in expr/tla/oauth_flow_scopes aligned; token grants remain app-owned. - `service.assignClientMembers` owns the service Client receiver: reserve all authored methods, record synthetic mixed-stream method names, then allocate @@ -1395,6 +1446,18 @@ filter, and serialization rules belong here. branches remain unsupported. `expr.ValuePlan.JSONShape` lowers captured effective predicates. `codegen/unionjson` projects service or HTTP member layouts and renders typed adapters; `pkg.MatchUntaggedJSON` owns orchestration. + Runtime DSL branches additionally admit string-keyed map fields (#1041) and + nested untagged unions (#1036); import's narrower admission remains separate. + Lower nested alternatives into `JSONShape.Branches`, traverse their member + policies and preserve exact-one matching at every level. Only internal typed + union contract mismatches become ordinary failed decoder candidates; malformed + descriptions, evaluator errors and opaque codec failures stay fatal. Reject + descriptor cycles that revisit a wire value without consuming a member or + collection element, while retaining recursive object/collection compositions. Lower map values through + `JSONShape.Child`, preserve entry-count rules on named map occurrences, and + visit map values when projecting member spelling, visibility and closed-object + policy. Emit named map validators alongside object/array validators. Current + map schemas constrain values; do not add key-schema semantics here. Independently require unique schema and actual Go matches with the same identity in both directions. Do not use numeric widths as schema discriminators, select the first successful branch, or mutate a destination before matching succeeds. diff --git a/.agents/skills/loom-framework/references/formal-models.md b/.agents/skills/loom-framework/references/formal-models.md index 3891dc263..4e3cb1390 100644 --- a/.agents/skills/loom-framework/references/formal-models.md +++ b/.agents/skills/loom-framework/references/formal-models.md @@ -20,6 +20,18 @@ The [documentation schema admission model](../../../../expr/tla/documentation_sc separates valid documentation schemas from unsupported generated codecs and checks that validation visits cannot leak between those contexts. +## Unary admission + +The [concurrency model](../../../../admission/tla/concurrency/README.md) checks +permit retention across cancellation and live capacity updates. It rejects +release-on-cancellation and reset-on-resize candidates. Direct race tests cover +Go exit paths; the generated example checks protected HTTP/gRPC admission. + +The [rate policy model](../../../../admission/tla/rate/README.md) checks quota +conservation and coherent rate/burst updates. Direct tests cover fractional +accounting and backend deadlines; generated HTTP checks cover early and +authenticated quota placement. Group storage and distributed policy are external. + ## Compilation value ownership The [service semantic ownership model](../../../../codegen/service/tla/semantic_ownership/README.md) @@ -261,6 +273,7 @@ Rendered schema validation and actual decoder behavior are separate obligations. | Public message ownership, history masking, diagnostic exclusion, transparent wrappers and independent joins | [Public disclosure](../../../../pkg/tla/error_disclosure/README.md); finite owner/message model, with HTTP/gRPC/JSON-RPC unary/SSE/WebSocket wire tests | | gRPC aggregate error status and detail ownership | [Error contracts](../../../../grpc/tla/error_contract/README.md); bounded branch-order and complete-failure checks, with wrappers and generated unary/streaming mappings tested in Go | | Nearest raw forwarding boundary and prefix independence | [Forwarded client trust](../../../../http/tla/forwarded_client/README.md); bounded chains, with IP parsing and repeated-header grouping checked in Go | +| Per-scheme security scopes and roles with AND/OR/anonymous alternatives | [Security requirements](../../../../expr/tla/security_requirements/README.md); flattened-name counterexample and generated credential checks | | Authentication alternatives, optional credential extraction and failed-context isolation | [Credential alternatives](../../../../internal/securitygen/tla/credential_alternatives/README.md); finite extraction/control model, with actual framing and HTTP/JSON-RPC/gRPC wire regressions | | gRPC pre-frame admission, complete application input, original cancellation and middleware rechecks; ordered AND/OR contexts and failed-branch reset | [Stream admission](../../../../security/tla/stream_admission/README.md); positive and deliberate negative models paired with runtime and generated transport regressions | | Scalar query/header/cookie absence, defaults and conversion admission | [Scalar binding presence](../../../../http/codegen/tla/scalar_binding_presence/README.md); supplied-empty counterexample, shared decoder and mounted HTTP/JSON-RPC controls | @@ -343,3 +356,11 @@ The [discriminator admission model](../../../../http/codegen/openapi/internal/ir records optional/fallback admission and 3.1 projection. Go tests cover reference resolution and graph retention outside its Boolean assumptions. | JSON-RPC envelope ID source, presence and validation ordering | [Envelope ID admission](../../../../http/codegen/tla/jsonrpc_id_admission/README.md); params-first and late-handler counterexamples, shared carrier/CLI validation and mounted unary/SSE/WebSocket controls | + +OAuth flow inventories: [independent defaults and overrides](../../../../expr/tla/oauth_flow_scopes/README.md), with a flattening counterexample and generated callback checks. + +Schema reference siblings: [conjunction and declaration storage](../../../../internal/openapiimport/tla/reference_siblings/README.md), with rejection, replacement, mutation and occurrence-storage counterexamples. + +The [reusable-reference model](../../../../internal/openapiimport/tla/reusable_references/README.md) +checks component identity, occurrence overlays, immutable definitions and bounded +forward-reference/cycle admission across request bodies, responses and headers. diff --git a/.agents/skills/loom-framework/references/repo-map.md b/.agents/skills/loom-framework/references/repo-map.md index c9d32cc91..1581f027c 100644 --- a/.agents/skills/loom-framework/references/repo-map.md +++ b/.agents/skills/loom-framework/references/repo-map.md @@ -5,6 +5,10 @@ belong in the `loom` skill and the canonical guides under `docs/`. ## Ownership +- `admission/`: local concurrency and rate budgets, atomic settings updates and + bounded cooperative quota checks; existing protected hooks retain access + ordering, and `http/middleware/rate.go` owns early HTTP admission + - `dsl/`: public DSL functions and DSL-time validation entry points - `expr/`: evaluated design model, preparation, finalization, and semantic validation diff --git a/.agents/skills/loom/SKILL.md b/.agents/skills/loom/SKILL.md index 6fa16065f..a33f64643 100644 --- a/.agents/skills/loom/SKILL.md +++ b/.agents/skills/loom/SKILL.md @@ -39,6 +39,33 @@ fields in the body, map them explicitly, or declare `ErrorNameFromStatus()` on an HTTP error response with a unique status. Views retain their projected contract. Generation rejects incomplete mappings; add the missing source and regenerate. +## Unary concurrency admission + +Use `admission.NewConcurrency(capacity)` and install `budget.Middleware` through +generated `Endpoints.Use` or protected unary per-method constructors. Share a +budget only across methods that should share a process-local limit; exclude +streams. Declare `overloaded` and map saturation explicitly (HTTP 503, for +example), separately from quota errors. `SetCapacity` accepts positive values, +retains active permits and rejects invalid updates without changing the limit. +Cancellation releases capacity only when the wrapped invocation exits. Export +bounded counts and gauges from `Snapshot`; applications own metrics, update +authorization and any cross-replica policy. See [admission](../../../docs/admission.md). + +## Rate and burst admission + +Use `admission.Rate` for a local token bucket and `SetPolicy` for an atomic +rate/burst update. Updates retain allowance after old-rate refill, clamp it to a +smaller burst and apply the new rate going forward; unchanged updates never +reset spent quota. `NewRateAdmission(check, timeout)` supplies the backend +boundary. Checks must honor their context deadline and own coherent accounting. +Use HTTP `middleware.RateLimit` for early admission and the control's endpoint +`Middleware` through protected constructors for authenticated quotas. Return a +declared denial error, map 429 plus known Retry-After, and declare/map +`rate_limit_unavailable` for backend failures. Applications own trusted grouping, +bounded key storage and local versus shared quota scope. See +[rate limiting](../../../docs/rate-limiting.md) for the runnable example and update +contract. No streams, automatic retries or fleet-wide policy activation are added. + ## Typed background activities Use `execution.Activity[Input, Output]` for named, typed activity calls without @@ -574,11 +601,17 @@ and receive a blocking diagnostic before a design is written. See [OpenAPI import coverage](../../../docs/openapi-import-coverage.md) for supported recursive shapes and limitations. +Reusable request body, response, and header references retain component names, +forward aliases, and OpenAPI 3.1/3.2 occurrence summary/description annotations, +including empty strings. Shared definitions remain unchanged. Unused definitions, +cycles and wrong-kind references still block strict import. Re-import and +regenerate to adopt this support. + OpenAPI 3.2 Media Type `description` requires `--allow-lossy`. `prefixEncoding` remains a strict import error. If you explicitly accept omitting non-contract -metadata, the documented `allOf` flattening and inline array-item promotion, +metadata, the documented inline array-item promotion, unrecognized `format` values, or a parameter/header (not schema) `deprecated` flag the HTTP DSL cannot express per-parameter, add `--allow-lossy`; it warns on stderr for each omission but still refuses any @@ -587,11 +620,12 @@ operations. It reports skipped operations and omitted root members separately. These root members include servers and unsupported info metadata. Tag metadata, response summaries, supported serialization, structured examples, and reusable examples import losslessly. API-key, HTTP basic, HTTP bearer, and supported -OAuth 2.0 schemes also import losslessly. OAuth 2.0 flows must use the same -scope map. Root and operation security requirements retain OAuth scopes and +OAuth 2.0 schemes also import losslessly. OAuth 2.0 flows retain independent scope maps, including empty inventories. +Author flow-local scopes in an optional final block on each flow constructor; +no block inherits scheme-level scopes, while an empty block replaces them. Root and operation security requirements retain OAuth scopes and AND or alternative semantics. Schemes using the same wire credential share one -payload field while retaining each authentication check. A conjunction whose -schemes require different scope sets is rejected; equal scope sets and separate +payload field while retaining each authentication check. Each scheme retains its own required names: OAuth scopes render with +`ScopeFor`, and non-OAuth roles render with `RoleFor`. Empty lists and separate alternatives remain supported. Partial import omits unused root security when every retained operation explicitly overrides it. An anonymous `{}` alternative renders as `Security()` and an explicit operation `security: []` @@ -927,7 +961,14 @@ together; these rules also cover JSON-RPC HTTP metadata, not gRPC metadata. and decoding require the same unique schema and Go branch; overlapping schemas fail even when Go numeric widths distinguish them. Nil slices encode as `[]`, which is ambiguous if multiple array branches admit empty arrays. Put nullability - on the enclosing union, not a branch. Opaque codecs and nested maps are rejected. + on the enclosing union, not a branch. Fields may contain string-keyed maps + with constrained primitive, named object, array or map values. Nullability and + entry-count bounds retain their declared meaning. Regenerate service and + transport code when adopting map fields or nested untagged unions. Every + nested union must have exactly one matching named object/array branch; invalid + or ambiguous rows reject the enclosing candidate, and empty arrays stay valid + unless constrained. Map root branches, tagged nested unions and opaque codecs + remain rejected. - Explicit discriminator tags control wire values independently of schema and Go type names. - Optional object unions generate as pointers; required unions remain values. @@ -1212,8 +1253,13 @@ projected onto each scheme's declared scopes, including Basic and API keys. Callbacks can use `scheme.Validate(grantedScopes)` directly: an unscoped key alongside a scoped JWT receives no JWT scopes. Shared scopes apply to each scheme that declares them; undeclared scopes fail design validation. Separate `Security` -alternatives retain their own scope sets. Regenerate endpoints and OpenAPI to -adopt this correction. +alternatives retain their own scope sets. Inside Security, use +`ScopeFor(oauth, "read")` to bind OAuth scopes to one scheme and +`RoleFor(key, "admin")` for non-OAuth application roles. Both are additive to +shared Scope declarations. Role names need no scheme inventory. Non-OAuth +callbacks receive them in RequiredScopes and must validate application-owned +grants; they are not extra wire credentials. Regenerate endpoints and OpenAPI; +non-OAuth required names now export as role arrays. `CookieTransport(scheme, "", fn...)` is the transport-owned browser-cookie mode. It emits the security contract without synthesizing a payload field or @@ -1638,6 +1684,13 @@ loom example /design loom vet /design ``` +OpenAPI 3.1/3.2 imports preserve supported bounds, enums, required property +names and annotations beside local schema references. These refine each use +without changing the shared type. Object-property overlays preserve the base +reference through `allOf`; they may refine inherited fields and add fields to +open objects. Changes to the referenced type or nullability and general schema +intersections remain blocked; see `docs/openapi-import-coverage.md`. + ## Canonical Guides For Pulse applications, read `docs/pulse.md`. Client creation and custom hooks diff --git a/admission/concurrency.go b/admission/concurrency.go new file mode 100644 index 000000000..e6b40f69f --- /dev/null +++ b/admission/concurrency.go @@ -0,0 +1,118 @@ +// Package admission provides request admission controls for existing middleware +// hooks. Applications own budgets, grouping, placement and transport mappings. +package admission + +import ( + "context" + "errors" + "sync" + + loom "github.com/CaliLuke/loom/pkg" +) + +type ( + // Concurrency bounds simultaneous endpoint invocations. Construct it with + // NewConcurrency and share the pointer across the endpoints in one budget. + // It must not be copied. Completion, including panic unwinding, releases the + // permit; cancellation alone does not. It is unsuitable for streams or work + // that continues after the wrapped endpoint returns. + Concurrency struct { + mu sync.Mutex + state ConcurrencySnapshot + } + + // ConcurrencySnapshot is a coherent local observation with no request labels. + // Counters cover calls reaching this admission boundary since construction. + ConcurrencySnapshot struct { + // Capacity is the current positive admission limit. + Capacity int + // InFlight counts invocations that have acquired a permit and not exited. + // It may exceed Capacity after a reduction. + InFlight int + // Admitted counts acquired permits, including calls that later fail or panic. + Admitted uint64 + // Rejected counts calls denied because capacity was unavailable. + Rejected uint64 + // Canceled counts calls denied because their context was already canceled + // or expired at the admission decision. These are not also Rejected. + Canceled uint64 + } +) + +// NewConcurrency constructs an admission budget. Capacity must be positive; +// invalid configuration returns an error before any middleware is constructed. +// No queues, worker goroutines, retries or cross-process coordination are used. +func NewConcurrency(capacity int) (*Concurrency, error) { + c := new(Concurrency) + if err := c.SetCapacity(capacity); err != nil { + return nil, err + } + return c, nil +} + +// SetCapacity atomically changes the local limit without changing permits or +// counters. Nonpositive values return an error and leave the prior limit intact. +// Reductions affect new admissions only; existing invocations finish normally. +func (c *Concurrency) SetCapacity(capacity int) error { + if capacity <= 0 { + return errors.New("admission: concurrency capacity must be positive") + } + c.mu.Lock() + defer c.mu.Unlock() + c.state.Capacity = capacity + return nil +} + +// Snapshot returns the effective capacity and accounting in one synchronized read. +// Applications may export these bounded counters and gauges to their metrics system. +func (c *Concurrency) Snapshot() ConcurrencySnapshot { + c.mu.Lock() + defer c.mu.Unlock() + return c.state +} + +// Middleware wraps a unary endpoint with immediate admission. Saturation returns +// a *loom.ServiceError named "overloaded" with Temporary set. Declare that error +// in the design and map it explicitly, for example to HTTP 503. No retry occurs. +// A context canceled at admission returns ctx.Err() instead, even when saturated. +// After admission the original context and request are passed through unchanged; +// the endpoint's result, error and panic propagate unchanged. +// +// Install through generated Endpoints.Use or a protected per-method constructor +// to retain validation and authorization before admission. This does not protect +// transport decoding or authentication. It panics for an uninitialized budget +// or nil endpoint, before requests can be served. +func (c *Concurrency) Middleware(next loom.Endpoint) loom.Endpoint { + if next == nil || c.Snapshot().Capacity <= 0 { + panic("admission: initialized concurrency and non-nil endpoint required") + } + return func(ctx context.Context, request any) (any, error) { + if err := c.acquire(ctx); err != nil { + return nil, err + } + defer c.release() + return next(ctx, request) + } +} + +func (c *Concurrency) acquire(ctx context.Context) error { + c.mu.Lock() + defer c.mu.Unlock() + if err := ctx.Err(); err != nil { + c.state.Canceled++ + return err + } + if c.state.InFlight >= c.state.Capacity { + c.state.Rejected++ + return loom.TemporaryError("overloaded", "service capacity exhausted") + } + c.state.InFlight++ + c.state.Admitted++ + return nil +} + +func (c *Concurrency) release() { + c.mu.Lock() + defer c.mu.Unlock() + c.state.InFlight-- +} diff --git a/admission/concurrency_test.go b/admission/concurrency_test.go new file mode 100644 index 000000000..2f39262b1 --- /dev/null +++ b/admission/concurrency_test.go @@ -0,0 +1,252 @@ +package admission_test + +import ( + "context" + "errors" + "sync" + "testing" + "testing/synctest" + "time" + + "github.com/stretchr/testify/require" + + "github.com/CaliLuke/loom/admission" + loom "github.com/CaliLuke/loom/pkg" +) + +func TestConcurrencyConfiguration(t *testing.T) { + for _, capacity := range []int{-1, 0} { + c, err := admission.NewConcurrency(capacity) + require.Error(t, err) + require.Nil(t, c) + } + c, err := admission.NewConcurrency(2) + require.NoError(t, err) + for _, capacity := range []int{-1, 0} { + require.Error(t, c.SetCapacity(capacity)) + require.Equal(t, 2, c.Snapshot().Capacity) + } + require.Panics(t, func() { + c.Middleware(nil) + }) + var zero admission.Concurrency + require.Panics(t, func() { + zero.Middleware(func(context.Context, any) (any, error) { + return nil, nil + }) + }) +} + +func TestConcurrencyReleasesOnEveryExit(t *testing.T) { + serviceError := errors.New("service failure") + for _, outcome := range []string{"success", "error", "panic"} { + t.Run(outcome, func(t *testing.T) { + c, err := admission.NewConcurrency(1) + require.NoError(t, err) + type traceKey struct{} + ctx, cancel := context.WithDeadline(context.WithValue(context.Background(), traceKey{}, "trace"), time.Now().Add(time.Minute)) + defer cancel() + request := new(int) + ep := c.Middleware(func(got context.Context, req any) (any, error) { + require.Same(t, ctx, got) + require.Same(t, request, req) + require.Equal(t, 1, c.Snapshot().InFlight) + switch outcome { + case "error": + return request, serviceError + case "panic": + panic("service panic") + default: + return request, nil + } + }) + for range 2 { + if outcome == "panic" { + require.PanicsWithValue(t, "service panic", func() { + _, err := ep(ctx, request) + require.NoError(t, err) + }) + } else { + result, err := ep(ctx, request) + require.Same(t, request, result) + if outcome == "error" { + require.ErrorIs(t, err, serviceError) + } else { + require.NoError(t, err) + } + } + require.Zero(t, c.Snapshot().InFlight) + } + require.Equal(t, uint64(2), c.Snapshot().Admitted) + }) + } +} + +func TestConcurrencyCancellationRetainsPermit(t *testing.T) { + synctest.Test(t, func(t *testing.T) { + c, err := admission.NewConcurrency(1) + require.NoError(t, err) + release := make(chan struct{}) + unblock := sync.OnceFunc(func() { + close(release) + }) + defer unblock() + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + ep := c.Middleware(func(ctx context.Context, _ any) (any, error) { + <-ctx.Done() + <-release + return nil, ctx.Err() + }) + go func() { + _, err := ep(ctx, nil) + require.ErrorIs(t, err, context.Canceled) + }() + synctest.Wait() + cancel() + synctest.Wait() + require.Equal(t, 1, c.Snapshot().InFlight) + _, err = ep(context.Background(), nil) + requireOverloaded(t, err) + _, err = ep(ctx, nil) + require.ErrorIs(t, err, context.Canceled) + expired, stop := context.WithDeadline(context.Background(), time.Now().Add(-time.Second)) + defer stop() + _, err = ep(expired, nil) + require.ErrorIs(t, err, context.DeadlineExceeded) + require.Equal(t, admission.ConcurrencySnapshot{Capacity: 1, InFlight: 1, Admitted: 1, Rejected: 1, Canceled: 2}, c.Snapshot()) + unblock() + synctest.Wait() + require.Zero(t, c.Snapshot().InFlight) + // Once the canceled invocation exits, its permit can be reused. + _, err = c.Middleware(func(context.Context, any) (any, error) { + return "recovered", nil + })(context.Background(), nil) + require.NoError(t, err) + require.Equal(t, uint64(2), c.Snapshot().Admitted) + }) +} + +func TestConcurrencySimultaneousCalls(t *testing.T) { + synctest.Test(t, func(t *testing.T) { + c, err := admission.NewConcurrency(4) + require.NoError(t, err) + start, release := make(chan struct{}), make(chan struct{}) + defer close(release) + ep := c.Middleware(func(context.Context, any) (any, error) { + <-release + return nil, nil + }) + for range 20 { + go func() { + <-start + _, err := ep(context.Background(), nil) + if err != nil { + requireOverloaded(t, err) + } + }() + } + close(start) + synctest.Wait() + require.Equal(t, admission.ConcurrencySnapshot{Capacity: 4, InFlight: 4, Admitted: 4, Rejected: 16}, c.Snapshot()) + }) +} + +func TestConcurrencyResizeRetainsActiveCalls(t *testing.T) { + synctest.Test(t, func(t *testing.T) { + c, err := admission.NewConcurrency(2) + require.NoError(t, err) + finish := []chan struct{}{make(chan struct{}), make(chan struct{}), make(chan struct{})} + ep := c.Middleware(func(_ context.Context, request any) (any, error) { + <-finish[request.(int)] + return nil, nil + }) + start := func(i int) { + go func() { + _, err := ep(context.Background(), i) + require.NoError(t, err) + }() + synctest.Wait() + } + start(0) + start(1) + require.NoError(t, c.SetCapacity(3)) + start(2) + require.NoError(t, c.SetCapacity(1)) + require.NoError(t, c.SetCapacity(1)) + require.Error(t, c.SetCapacity(0)) + for _, done := range finish { + _, err := ep(context.Background(), 0) + requireOverloaded(t, err) + close(done) + synctest.Wait() + } + _, err = ep(context.Background(), 0) + require.NoError(t, err) + require.Equal(t, admission.ConcurrencySnapshot{Capacity: 1, Admitted: 4, Rejected: 3}, c.Snapshot()) + }) +} + +func TestConcurrencyUpdateCompletionAndCancellationRace(t *testing.T) { + c, err := admission.NewConcurrency(4) + require.NoError(t, err) + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + start := make(chan struct{}) + entered, finish := make(chan struct{}), make(chan struct{}) + unblock := sync.OnceFunc(func() { + close(finish) + }) + defer unblock() + var workers sync.WaitGroup + ep := c.Middleware(func(_ context.Context, request any) (any, error) { + if request == "held" { + close(entered) + <-finish + } + return nil, nil + }) + workers.Go(func() { + _, err := ep(ctx, "held") + require.NoError(t, err) + }) + <-entered + require.Equal(t, 1, c.Snapshot().InFlight) + // A permit is already held before updates, cancellation and its release + // become runnable. Cancellation cannot avoid the completion/update race. + for i := range 100 { + workers.Go(func() { + <-start + require.NoError(t, c.SetCapacity(i%4+1)) + _, err := ep(ctx, nil) + if err != nil && !errors.Is(err, context.Canceled) { + requireOverloaded(t, err) + } + require.Error(t, c.SetCapacity(-1)) + }) + } + workers.Go(func() { + <-start + cancel() + }) + workers.Go(func() { + <-start + unblock() + }) + close(start) + workers.Wait() + snapshot := c.Snapshot() + require.Zero(t, snapshot.InFlight) + require.GreaterOrEqual(t, snapshot.Admitted, uint64(1)) + require.Equal(t, uint64(101), snapshot.Admitted+snapshot.Rejected+snapshot.Canceled) + require.GreaterOrEqual(t, snapshot.Capacity, 1) + require.LessOrEqual(t, snapshot.Capacity, 4) +} + +func requireOverloaded(t *testing.T, err error) { + t.Helper() + var serviceError *loom.ServiceError + require.ErrorAs(t, err, &serviceError) + require.Equal(t, "overloaded", serviceError.Name) + require.True(t, serviceError.Temporary) +} diff --git a/admission/rate.go b/admission/rate.go new file mode 100644 index 000000000..fe1955706 --- /dev/null +++ b/admission/rate.go @@ -0,0 +1,142 @@ +package admission + +import ( + "context" + "errors" + "math" + "sync" + "time" + + "golang.org/x/time/rate" +) + +type ( + // RatePolicy configures one local token bucket. Both fields must be positive, + // and PerSecond must be below math.MaxFloat64 (the dependency's unlimited + // sentinel). One admitted request consumes one token. + RatePolicy struct { + // PerSecond is the number of tokens replenished per second. + PerSecond float64 + // Burst is the maximum token allowance, initially available in full. + Burst int + } + + // RateDecision reports one admission decision. A denied decision with no error + // means quota exhaustion; backend failures are returned separately as errors. + RateDecision struct { + // Allowed permits the request to proceed. + Allowed bool + // RetryAfter is a positive advisory delay until allowance may be available. + // Zero means unknown. A successful decision must leave this field zero. + RetryAfter time.Duration + // Version identifies the coherent local policy used for this decision. + // External checkers may use zero when they do not expose a version. + Version uint64 + } + + // Rate wraps a maintained token bucket with atomic policy updates. Construct + // it with NewRate; it must not be copied. A Rate is one process-local budget, + // not a keyed store. Applications own trusted grouping and bounded storage of + // separate budgets, or supply a shared backend to NewRateAdmission. + Rate struct { + mu sync.Mutex + limiter *rate.Limiter + policy RatePolicy + version uint64 + } +) + +// NewRate validates policy and constructs a full local bucket at version 1. +func NewRate(policy RatePolicy) (*Rate, error) { + if err := validateRatePolicy(policy); err != nil { + return nil, err + } + return &Rate{ + limiter: rate.NewLimiter(rate.Limit(policy.PerSecond), policy.Burst), + policy: policy, + version: 1, + }, nil +} + +// SetPolicy atomically applies both settings. Accrue allowance under the old +// rate up to the update, retain that allowance (clamped to the new burst), then +// use the new rate for future refill. Increasing burst adds refill headroom, +// not immediate allowance. Reapplying identical settings does not reset +// accounting or advance the version. Invalid updates leave all state unchanged. +func (r *Rate) SetPolicy(policy RatePolicy) error { + if err := validateRatePolicy(policy); err != nil { + return err + } + r.mu.Lock() + defer r.mu.Unlock() + if r.limiter == nil { + return errors.New("admission: rate budget must be constructed with NewRate") + } + if r.policy == policy { + return nil + } + now := time.Now() + r.limiter.SetLimitAt(now, rate.Limit(policy.PerSecond)) + r.limiter.SetBurstAt(now, policy.Burst) + r.policy = policy + r.version++ + return nil +} + +// Policy returns both effective settings and their version in one coherent read. +func (r *Rate) Policy() (RatePolicy, uint64) { + r.mu.Lock() + defer r.mu.Unlock() + return r.policy, r.version +} + +// Allow consumes one token if available, without waiting or reserving future +// allowance. A canceled context consumes nothing. Consumed tokens are not +// refunded if later validation, service execution or request cancellation fails. +// RetryAfter is advisory: other calls and policy changes can alter availability. +func (r *Rate) Allow(ctx context.Context) (RateDecision, error) { + r.mu.Lock() + defer r.mu.Unlock() + if err := rateContextError(ctx); err != nil { + return RateDecision{}, err + } + if r.limiter == nil { + return RateDecision{}, errors.New("admission: rate budget must be constructed with NewRate") + } + now := time.Now() + decision := RateDecision{Allowed: r.limiter.AllowN(now, 1), Version: r.version} + if !decision.Allowed { + missing := 1 - r.limiter.TokensAt(now) + nanos := math.Ceil(missing / r.policy.PerSecond * float64(time.Second)) + switch { + case nanos >= float64(math.MaxInt64): + decision.RetryAfter = time.Duration(math.MaxInt64) + case nanos < 1: + decision.RetryAfter = time.Nanosecond + default: + decision.RetryAfter = time.Duration(nanos) + } + } + return decision, nil +} + +// RetryAfterSeconds rounds a known positive delay up to whole seconds without +// overflowing. It returns zero for an unknown delay. HTTP adapters can publish +// a positive value as Retry-After; unknown delays must omit that header. +func (d RateDecision) RetryAfterSeconds() int64 { + if d.RetryAfter <= 0 { + return 0 + } + seconds := int64(d.RetryAfter / time.Second) + if d.RetryAfter%time.Second != 0 { + seconds++ + } + return seconds +} + +func validateRatePolicy(policy RatePolicy) error { + if policy.PerSecond <= 0 || math.IsNaN(policy.PerSecond) || policy.PerSecond >= float64(rate.Inf) || policy.Burst <= 0 { + return errors.New("admission: rate must be positive and below math.MaxFloat64 and burst must be positive") + } + return nil +} diff --git a/admission/rate_admission.go b/admission/rate_admission.go new file mode 100644 index 000000000..fbcdb2a87 --- /dev/null +++ b/admission/rate_admission.go @@ -0,0 +1,116 @@ +package admission + +import ( + "context" + "errors" + "time" + + loom "github.com/CaliLuke/loom/pkg" +) + +type ( + // RateCheck checks and accounts for one request in an application-selected + // group. It must be safe for concurrent use and return promptly when ctx ends. + // A backend owns coherent policy/accounting and the scope of its quota. Keys + // must not be included in returned error messages, logs or metric labels. + RateCheck func(ctx context.Context, key string) (RateDecision, error) + + // RateAdmission bounds calls to a quota backend and rejects its errors or + // invalid decisions. Construct it with NewRateAdmission. It creates no + // background goroutines, retries, queues or key storage. Its timeout bounds + // cooperative backends; it cannot forcibly stop a callback that ignores ctx. + RateAdmission struct { + check RateCheck + timeout time.Duration + } +) + +// NewRateAdmission validates the checker and its positive per-check time budget. +// Applications must configure network/storage timeouts in their checker too. +func NewRateAdmission(check RateCheck, timeout time.Duration) (*RateAdmission, error) { + if check == nil || timeout <= 0 { + return nil, errors.New("admission: rate checker and positive timeout required") + } + return &RateAdmission{check: check, timeout: timeout}, nil +} + +// Check runs one backend check with a child deadline, retaining the caller's +// values and earlier deadline. Caller cancellation returns ctx.Err(). Backend +// errors, backend timeouts, malformed decisions and an uninitialized admission +// return a safe *loom.ServiceError named "rate_limit_unavailable". Backend error +// text and grouping keys are never included. No error permits execution. +func (a *RateAdmission) Check(ctx context.Context, key string) (RateDecision, error) { + if err := rateContextError(ctx); err != nil { + return RateDecision{}, err + } + if a == nil || a.check == nil || a.timeout <= 0 { + return RateDecision{}, rateUnavailable() + } + checkCtx, cancel := context.WithTimeout(ctx, a.timeout) + defer cancel() + decision, err := a.check(checkCtx, key) + if callerErr := rateContextError(ctx); callerErr != nil { + return RateDecision{}, callerErr + } + if err != nil || rateContextError(checkCtx) != nil || decision.RetryAfter < 0 || (decision.Allowed && decision.RetryAfter != 0) { + return RateDecision{}, rateUnavailable() + } + return decision, nil +} + +// Middleware admits unary endpoint calls using an application-owned grouping +// function and declared denial error. Install it through generated Endpoints.Use +// or protected constructors so grouping uses authenticated identity. A key error +// becomes "rate_limit_unavailable" without exposing its text. Denial invokes +// denied with the decision; it must return an error matching the design, including +// any known RetryAfter mapping. A nil denial error fails closed. +// +// The next endpoint receives the original context, not the bounded backend +// context. Callback and next panics propagate. Invalid construction dependencies +// panic before serving. Streams and per-message quotas are outside this helper. +func (a *RateAdmission) Middleware(key func(context.Context, any) (string, error), denied func(RateDecision) error) func(loom.Endpoint) loom.Endpoint { + if a == nil || a.check == nil || a.timeout <= 0 || key == nil || denied == nil { + panic("admission: initialized rate admission, grouping and denial functions required") + } + return func(next loom.Endpoint) loom.Endpoint { + if next == nil { + panic("admission: non-nil endpoint required") + } + return func(ctx context.Context, request any) (any, error) { + if err := rateContextError(ctx); err != nil { + return nil, err + } + group, err := key(ctx, request) + if err != nil { + return nil, rateUnavailable() + } + decision, err := a.Check(ctx, group) + if err != nil { + return nil, err + } + if !decision.Allowed { + if err := denied(decision); err != nil { + return nil, err + } + return nil, rateUnavailable() + } + return next(ctx, request) + } + } +} + +// rateContextError checks clock expiry independently of timer delivery. A busy +// backend can return after its deadline before ctx.Err publishes that expiry. +func rateContextError(ctx context.Context) error { + if err := ctx.Err(); err != nil { + return err + } + if deadline, ok := ctx.Deadline(); ok && !time.Now().Before(deadline) { + return context.DeadlineExceeded + } + return nil +} + +func rateUnavailable() error { + return loom.TemporaryError("rate_limit_unavailable", "rate limit check unavailable") +} diff --git a/admission/rate_admission_test.go b/admission/rate_admission_test.go new file mode 100644 index 000000000..c255c7e86 --- /dev/null +++ b/admission/rate_admission_test.go @@ -0,0 +1,210 @@ +package admission_test + +import ( + "context" + "errors" + "testing" + "testing/synctest" + "time" + + "github.com/stretchr/testify/require" + + "github.com/CaliLuke/loom/admission" + loom "github.com/CaliLuke/loom/pkg" +) + +func TestRateAdmissionRejectsBackendFailures(t *testing.T) { + for _, tc := range []struct { + name string + decision admission.RateDecision + err error + }{ + {"backend error", admission.RateDecision{}, errors.New("sensitive key and backend details")}, + {"allow with error", admission.RateDecision{Allowed: true}, errors.New("sensitive failure")}, + {"negative retry", admission.RateDecision{RetryAfter: -time.Second}, nil}, + {"allow with retry", admission.RateDecision{Allowed: true, RetryAfter: time.Second}, nil}, + } { + t.Run(tc.name, func(t *testing.T) { + control, err := admission.NewRateAdmission(func(context.Context, string) (admission.RateDecision, error) { + return tc.decision, tc.err + }, time.Second) + require.NoError(t, err) + decision, err := control.Check(context.Background(), "sensitive-key") + require.False(t, decision.Allowed) + requireRateUnavailable(t, err) + require.NotContains(t, err.Error(), "sensitive") + }) + } +} + +func TestRateAdmissionDeadlineAndContext(t *testing.T) { + for _, parentDeadline := range []time.Duration{0, 50 * time.Millisecond} { + synctest.Test(t, func(t *testing.T) { + type contextKey struct{} + ctx := context.WithValue(context.Background(), contextKey{}, "trace") + if parentDeadline > 0 { + var cancel context.CancelFunc + ctx, cancel = context.WithTimeout(ctx, parentDeadline) + defer cancel() + } + start := time.Now() + control, err := admission.NewRateAdmission(func(check context.Context, key string) (admission.RateDecision, error) { + require.Equal(t, "group", key) + require.Equal(t, "trace", check.Value(contextKey{})) + <-check.Done() + // Even a late successful answer cannot authorize execution. + return admission.RateDecision{Allowed: true}, nil + }, 100*time.Millisecond) + require.NoError(t, err) + decision, err := control.Check(ctx, "group") + require.False(t, decision.Allowed) + if parentDeadline > 0 { + require.ErrorIs(t, err, context.DeadlineExceeded) + require.Equal(t, parentDeadline, time.Since(start)) + } else { + requireRateUnavailable(t, err) + require.Equal(t, 100*time.Millisecond, time.Since(start)) + } + }) + } +} + +func TestRateAdmissionMiddlewarePreservesContextAndGroups(t *testing.T) { + synctest.Test(t, func(t *testing.T) { + first, err := admission.NewRate(admission.RatePolicy{PerSecond: 1, Burst: 1}) + require.NoError(t, err) + second, err := admission.NewRate(admission.RatePolicy{PerSecond: 1, Burst: 1}) + require.NoError(t, err) + groups := map[string]*admission.Rate{"first": first, "second": second} + control, err := admission.NewRateAdmission(func(ctx context.Context, key string) (admission.RateDecision, error) { + return groups[key].Allow(ctx) + }, time.Second) + require.NoError(t, err) + denial := errors.New("declared quota error") + wrap := control.Middleware(func(_ context.Context, request any) (string, error) { + return request.(string), nil + }, func(decision admission.RateDecision) error { + require.Equal(t, time.Second, decision.RetryAfter) + return denial + }) + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + calls := 0 + ep := wrap(func(got context.Context, request any) (any, error) { + require.Same(t, ctx, got, "backend deadline must not replace the service context") + require.NoError(t, got.Err()) + calls++ + return request, nil + }) + result, err := ep(ctx, "first") + require.NoError(t, err) + require.Equal(t, "first", result) + _, err = ep(ctx, "first") + require.ErrorIs(t, err, denial) + _, err = ep(ctx, "second") + require.NoError(t, err) + require.Equal(t, 2, calls) + cancel() + _, err = ep(ctx, "first") + require.ErrorIs(t, err, context.Canceled) + require.Equal(t, 2, calls) + }) +} + +func TestRateAdmissionMiddlewareFailsClosed(t *testing.T) { + for _, failure := range []string{"grouping", "backend", "nil denial", "canceled"} { + t.Run(failure, func(t *testing.T) { + checks := 0 + control, err := admission.NewRateAdmission(func(context.Context, string) (admission.RateDecision, error) { + checks++ + if failure == "backend" { + return admission.RateDecision{}, errors.New("sensitive backend") + } + return admission.RateDecision{}, nil + }, time.Second) + require.NoError(t, err) + wrap := control.Middleware(func(context.Context, any) (string, error) { + if failure == "grouping" { + return "", errors.New("sensitive identity") + } + return "group", nil + }, func(admission.RateDecision) error { + return nil + }) + calls := 0 + ep := wrap(func(context.Context, any) (any, error) { + calls++ + return nil, nil + }) + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + if failure == "canceled" { + cancel() + } + _, err = ep(ctx, nil) + if failure == "canceled" { + require.ErrorIs(t, err, context.Canceled) + } else { + requireRateUnavailable(t, err) + } + require.Zero(t, calls) + if failure == "grouping" || failure == "canceled" { + require.Zero(t, checks) + } + }) + } +} + +func TestRateAdmissionCanceledCheck(t *testing.T) { + checks := 0 + control, err := admission.NewRateAdmission(func(context.Context, string) (admission.RateDecision, error) { + checks++ + return admission.RateDecision{Allowed: true}, nil + }, time.Second) + require.NoError(t, err) + ctx, cancel := context.WithCancel(context.Background()) + cancel() + decision, err := control.Check(ctx, "group") + require.ErrorIs(t, err, context.Canceled) + require.False(t, decision.Allowed) + require.Zero(t, checks) +} + +func TestRateAdmissionConfiguration(t *testing.T) { + check := func(context.Context, string) (admission.RateDecision, error) { + return admission.RateDecision{}, nil + } + for _, timeout := range []time.Duration{0, -time.Second} { + _, err := admission.NewRateAdmission(check, timeout) + require.Error(t, err) + } + _, err := admission.NewRateAdmission(nil, time.Second) + require.Error(t, err) + var zero admission.RateAdmission + _, err = zero.Check(context.Background(), "") + requireRateUnavailable(t, err) + control, err := admission.NewRateAdmission(check, time.Second) + require.NoError(t, err) + require.Panics(t, func() { + control.Middleware(nil, nil) + }) + key := func(context.Context, any) (string, error) { + return "", nil + } + denied := func(admission.RateDecision) error { + return errors.New("quota exhausted") + } + require.Panics(t, func() { + zero.Middleware(key, denied) + }) + require.Panics(t, func() { + control.Middleware(key, denied)(nil) + }) +} + +func requireRateUnavailable(t *testing.T, err error) { + t.Helper() + var serviceError *loom.ServiceError + require.ErrorAs(t, err, &serviceError) + require.Equal(t, "rate_limit_unavailable", serviceError.Name) +} diff --git a/admission/rate_deadline_test.go b/admission/rate_deadline_test.go new file mode 100644 index 000000000..b5999e3ba --- /dev/null +++ b/admission/rate_deadline_test.go @@ -0,0 +1,72 @@ +package admission_test + +import ( + "context" + "runtime" + "testing" + "time" + + "github.com/stretchr/testify/require" + + "github.com/CaliLuke/loom/admission" +) + +// pendingDeadlineContext models the interval after a fixed deadline has elapsed +// but before its timer callback closes Done and publishes Err. +type pendingDeadlineContext struct { + context.Context + deadline time.Time + done <-chan struct{} +} + +// Deadline returns the already elapsed deadline independently of timer delivery. +func (c pendingDeadlineContext) Deadline() (time.Time, bool) { + return c.deadline, true +} + +// Done remains open while the modeled timer callback is waiting to run. +func (c pendingDeadlineContext) Done() <-chan struct{} { + return c.done +} + +func TestRateExpiredDeadlineBeforeTimerSignal(t *testing.T) { + ctx := pendingDeadlineContext{Context: context.Background(), deadline: time.Now().Add(-time.Second), done: make(chan struct{})} + t.Run("local bucket", func(t *testing.T) { + budget, err := admission.NewRate(admission.RatePolicy{PerSecond: 1, Burst: 1}) + require.NoError(t, err) + decision, err := budget.Allow(ctx) + require.ErrorIs(t, err, context.DeadlineExceeded) + require.False(t, decision.Allowed) + allowRate(t, budget, true) + }) + t.Run("backend", func(t *testing.T) { + calls := 0 + control, err := admission.NewRateAdmission(func(context.Context, string) (admission.RateDecision, error) { + calls++ + return admission.RateDecision{Allowed: true}, nil + }, time.Second) + require.NoError(t, err) + decision, err := control.Check(ctx, "group") + require.ErrorIs(t, err, context.DeadlineExceeded) + require.False(t, decision.Allowed) + require.Zero(t, calls) + }) +} + +func TestRateAdmissionLateBackendAnswer(t *testing.T) { + previous := runtime.GOMAXPROCS(1) + defer runtime.GOMAXPROCS(previous) + control, err := admission.NewRateAdmission(func(ctx context.Context, _ string) (admission.RateDecision, error) { + deadline, ok := ctx.Deadline() + require.True(t, ok) + // Avoid waiting on Done: the timer may not have been scheduled when the + // backend returns. The result must be denied with either timer ordering. + for time.Now().Before(deadline.Add(time.Millisecond)) { + } + return admission.RateDecision{Allowed: true}, nil + }, time.Millisecond) + require.NoError(t, err) + decision, err := control.Check(context.Background(), "group") + require.False(t, decision.Allowed) + requireRateUnavailable(t, err) +} diff --git a/admission/rate_test.go b/admission/rate_test.go new file mode 100644 index 000000000..47995f69d --- /dev/null +++ b/admission/rate_test.go @@ -0,0 +1,165 @@ +package admission_test + +import ( + "context" + "math" + "sync/atomic" + "testing" + "testing/synctest" + "time" + + "github.com/stretchr/testify/require" + + "github.com/CaliLuke/loom/admission" +) + +func TestRatePolicyValidation(t *testing.T) { + valid := admission.RatePolicy{PerSecond: 1, Burst: 2} + r, err := admission.NewRate(valid) + require.NoError(t, err) + for _, policy := range []admission.RatePolicy{ + {PerSecond: 0, Burst: 1}, {PerSecond: -1, Burst: 1}, + {PerSecond: math.NaN(), Burst: 1}, {PerSecond: math.Inf(1), Burst: 1}, + {PerSecond: math.MaxFloat64, Burst: 1}, + {PerSecond: 1, Burst: 0}, {PerSecond: 1, Burst: -1}, + } { + invalid, err := admission.NewRate(policy) + require.Error(t, err) + require.Nil(t, invalid) + require.Error(t, r.SetPolicy(policy)) + got, version := r.Policy() + require.Equal(t, valid, got) + require.Equal(t, uint64(1), version) + } + var zero admission.Rate + require.Error(t, zero.SetPolicy(valid)) + _, err = zero.Allow(context.Background()) + require.Error(t, err) +} + +func TestRateUpdatesPreserveAllowance(t *testing.T) { + synctest.Test(t, func(t *testing.T) { + r, err := admission.NewRate(admission.RatePolicy{PerSecond: 2, Burst: 2}) + require.NoError(t, err) + allowRate(t, r, true) + allowRate(t, r, true) + require.Equal(t, 500*time.Millisecond, allowRate(t, r, false).RetryAfter) + time.Sleep(250 * time.Millisecond) + larger := admission.RatePolicy{PerSecond: 4, Burst: 4} + require.NoError(t, r.SetPolicy(larger)) + decision := allowRate(t, r, false) + require.Equal(t, uint64(2), decision.Version) + require.Equal(t, 125*time.Millisecond, decision.RetryAfter, "old rate refills half a token before the update") + time.Sleep(125 * time.Millisecond) + allowRate(t, r, true) + require.NoError(t, r.SetPolicy(larger)) + require.Equal(t, uint64(2), allowRate(t, r, false).Version, "unchanged updates neither refill nor change version") + time.Sleep(time.Second) + require.NoError(t, r.SetPolicy(admission.RatePolicy{PerSecond: 4, Burst: 1})) + allowRate(t, r, true) + allowRate(t, r, false) + require.NoError(t, r.SetPolicy(admission.RatePolicy{PerSecond: 4, Burst: 3})) + allowRate(t, r, false) + time.Sleep(250 * time.Millisecond) + allowRate(t, r, true) + allowRate(t, r, false) + }) +} + +func TestRateDecreaseUsesOldRefillThenNewRate(t *testing.T) { + synctest.Test(t, func(t *testing.T) { + r, err := admission.NewRate(admission.RatePolicy{PerSecond: 2, Burst: 1}) + require.NoError(t, err) + allowRate(t, r, true) + time.Sleep(250 * time.Millisecond) + require.NoError(t, r.SetPolicy(admission.RatePolicy{PerSecond: 1, Burst: 1})) + require.Equal(t, 500*time.Millisecond, allowRate(t, r, false).RetryAfter) + time.Sleep(500 * time.Millisecond) + allowRate(t, r, true) + allowRate(t, r, false) + }) +} + +func TestRateCancellationConsumesNothing(t *testing.T) { + r, err := admission.NewRate(admission.RatePolicy{PerSecond: 1, Burst: 1}) + require.NoError(t, err) + ctx, cancel := context.WithCancel(context.Background()) + cancel() + _, err = r.Allow(ctx) + require.ErrorIs(t, err, context.Canceled) + allowRate(t, r, true) +} + +func TestRateInvalidUpdatePreservesSpentQuota(t *testing.T) { + synctest.Test(t, func(t *testing.T) { + r, err := admission.NewRate(admission.RatePolicy{PerSecond: 1, Burst: 1}) + require.NoError(t, err) + allowRate(t, r, true) + for _, invalid := range []admission.RatePolicy{{PerSecond: 2, Burst: 0}, {PerSecond: math.NaN(), Burst: 2}} { + require.Error(t, r.SetPolicy(invalid)) + decision := allowRate(t, r, false) + require.Equal(t, uint64(1), decision.Version) + require.Equal(t, time.Second, decision.RetryAfter) + } + }) +} + +func TestRateConcurrentPolicyUpdatesDoNotRefill(t *testing.T) { + synctest.Test(t, func(t *testing.T) { + first := admission.RatePolicy{PerSecond: 1, Burst: 2} + second := admission.RatePolicy{PerSecond: 4, Burst: 1} + r, err := admission.NewRate(first) + require.NoError(t, err) + start := make(chan struct{}) + var admitted atomic.Int64 + for i := range 100 { + go func() { + <-start + policy := first + if i%2 == 0 { + policy = second + } + require.NoError(t, r.SetPolicy(policy)) + decision, err := r.Allow(context.Background()) + require.NoError(t, err) + require.Positive(t, decision.Version) + if decision.Allowed { + admitted.Add(1) + } + observed, version := r.Policy() + require.Contains(t, []admission.RatePolicy{first, second}, observed) + require.Positive(t, version) + }() + } + close(start) + synctest.Wait() + require.GreaterOrEqual(t, admitted.Load(), int64(1)) + require.LessOrEqual(t, admitted.Load(), int64(2), "updates cannot recreate spent allowance") + }) +} + +func TestRateRetryDelayBoundaries(t *testing.T) { + for _, tc := range []struct { + delay time.Duration + seconds int64 + }{ + {0, 0}, {time.Nanosecond, 1}, {time.Second, 1}, + {time.Second + time.Nanosecond, 2}, {time.Duration(math.MaxInt64), 9223372037}, + } { + require.Equal(t, tc.seconds, (admission.RateDecision{RetryAfter: tc.delay}).RetryAfterSeconds()) + } + synctest.Test(t, func(t *testing.T) { + r, err := admission.NewRate(admission.RatePolicy{PerSecond: math.SmallestNonzeroFloat64, Burst: 1}) + require.NoError(t, err) + allowRate(t, r, true) + require.Equal(t, time.Duration(math.MaxInt64), allowRate(t, r, false).RetryAfter) + }) +} + +func allowRate(t *testing.T, r *admission.Rate, allowed bool) admission.RateDecision { + t.Helper() + decision, err := r.Allow(context.Background()) + require.NoError(t, err) + require.Equal(t, allowed, decision.Allowed) + return decision +} diff --git a/admission/tla/concurrency/Concurrency.tla b/admission/tla/concurrency/Concurrency.tla new file mode 100644 index 000000000..3c3dfb75a --- /dev/null +++ b/admission/tla/concurrency/Concurrency.tla @@ -0,0 +1,54 @@ +------------------------- MODULE Concurrency ------------------------- +EXTENDS Naturals, FiniteSets +CONSTANTS Requests, MaxCapacity, Mode +VARIABLES capacity, running, owned, admitted, finished, rejected, canceled, + safeAdmission +vars == <> +Init == /\ capacity = 1 + /\ running = {} + /\ owned = {} + /\ admitted = {} + /\ finished = {} + /\ rejected = {} + /\ canceled = {} + /\ safeAdmission = TRUE +Pending == Requests \ (admitted \cup rejected) +Admit(r) == /\ r \in Pending + /\ r \notin canceled + /\ Cardinality(owned) < capacity + /\ safeAdmission' = (Cardinality(running) < capacity) + /\ running' = running \cup {r} + /\ owned' = owned \cup {r} + /\ admitted' = admitted \cup {r} + /\ UNCHANGED <> +Reject(r) == /\ r \in Pending + /\ r \in canceled \/ Cardinality(owned) >= capacity + /\ rejected' = rejected \cup {r} + /\ UNCHANGED <> +Cancel(r) == /\ r \notin canceled + /\ canceled' = canceled \cup {r} + /\ owned' = IF Mode = "release-on-cancel" THEN owned \ {r} + ELSE owned + /\ UNCHANGED <> +Finish(r) == /\ r \in running + /\ running' = running \ {r} + /\ owned' = owned \ {r} + /\ finished' = finished \cup {r} + /\ UNCHANGED <> +Resize(n) == /\ capacity' = IF n = 0 THEN capacity ELSE n + /\ owned' = IF Mode = "reset-on-resize" /\ n # 0 THEN {} ELSE owned + /\ UNCHANGED <> +Next == (\E r \in Requests : Admit(r) \/ Reject(r) \/ Cancel(r) \/ Finish(r)) + \/ (\E n \in 0..MaxCapacity : Resize(n)) +Spec == Init /\ [][Next]_vars +PermitLifetime == owned = running +Accounting == /\ admitted = running \cup finished + /\ running \intersect finished = {} + /\ admitted \intersect rejected = {} +AdmissionBound == safeAdmission +ValidCapacity == capacity \in 1..MaxCapacity +====================================================================== diff --git a/admission/tla/concurrency/README.md b/admission/tla/concurrency/README.md new file mode 100644 index 000000000..0ebc30f74 --- /dev/null +++ b/admission/tla/concurrency/README.md @@ -0,0 +1,55 @@ +# Unary admission and resizing + +`Concurrency.tla` explores permit lifetime and live capacity changes before the +Go implementation. This is a new capability, not a reproduction of a previously +supported limiter defect. One mutex owns admission, capacity updates, completion +and snapshots in `../../concurrency.go`. + +The checked model allows arbitrary ordering of admission, rejection, cancellation, +completion and capacity updates. Nonpositive capacity is represented by zero and +leaves state intact. Capacity reductions can leave more running calls than the +new limit; the safety requirement applies to each new admission, not retroactively +to running calls. Repeated updates never discard permits. + +Assumptions and scope: + +- Each of three request identities invokes once. Finish represents success, + error or panic unwinding; implementation tests cover those concrete paths. +- Admission and resize are atomic actions, corresponding to mutex ownership. + Cancellation can precede or follow admission. It cannot terminate an invocation + by itself. There is no fairness or completion-time guarantee. +- `owned = running` checks permit retention, and `Accounting` checks disjoint + active/completed/rejected identities. Counts are abstracted as sets; this is not + a proof of Go defer semantics, integer overflow or exactly-once execution of + arbitrary application middleware. +- The model excludes streams, process failures, transport/authentication behavior, + distributed limits, queues and background work after endpoint completion. + +Using the repository's pinned TLC distribution (`tla2tools-1.7.4.jar`, reporting +TLC2 2.19, revision 5a47802) and Java 25.0.5, from this directory: + +```sh +java -XX:+UseParallelGC -cp "$HOME/.local/share/tlaplus/tla2tools-1.7.4.jar" \ + tlc2.TLC -workers 1 -metadir /tmp/loom-admission-model \ + -config checked.cfg Concurrency.tla +``` + +Run `cancel.cfg` and `resize.cfg` explicitly as negative controls. Neither is a +legacy implementation: they represent rejected candidate designs. The first +releases a permit on cancellation, and the second forgets permits when resizing. +Both violate `PermitLifetime` after admission followed by the named action. +Checked exploration passes 1,425 distinct states (10,072 generated). Cancellation +fails after 10 distinct states; resize fails after 16. The model informed the +choice to release only on endpoint exit and update capacity in the same owner. + +`../../concurrency_test.go` checks configuration, context identity, simultaneous +admission, each exit path, cancellation, resizing below usage and concurrent +updates/completions. `../../../codegen/generator/admission_integration_test.go` +(relative to the repository root: `codegen/generator/`) generates the runnable +example and checks HTTP/gRPC mappings and protected validation/access ordering. +The model does not establish those transport properties. Run the direct Go tests +with `go test -race ./admission` and the generated test with local Loom source +selected while developing an unpushed change. + +All formal commands are opt-in design tools. They are not CI or release gates. +Remove checker output and traces after review; keep these sources and findings. diff --git a/admission/tla/concurrency/cancel.cfg b/admission/tla/concurrency/cancel.cfg new file mode 100644 index 000000000..bfb5a69c5 --- /dev/null +++ b/admission/tla/concurrency/cancel.cfg @@ -0,0 +1,6 @@ +CONSTANTS Requests = {r1, r2, r3} + MaxCapacity = 3 + Mode = "release-on-cancel" +SPECIFICATION Spec +INVARIANTS PermitLifetime Accounting AdmissionBound ValidCapacity +CHECK_DEADLOCK FALSE diff --git a/admission/tla/concurrency/checked.cfg b/admission/tla/concurrency/checked.cfg new file mode 100644 index 000000000..4a011b757 --- /dev/null +++ b/admission/tla/concurrency/checked.cfg @@ -0,0 +1,6 @@ +CONSTANTS Requests = {r1, r2, r3} + MaxCapacity = 3 + Mode = "checked" +SPECIFICATION Spec +INVARIANTS PermitLifetime Accounting AdmissionBound ValidCapacity +CHECK_DEADLOCK FALSE diff --git a/admission/tla/concurrency/resize.cfg b/admission/tla/concurrency/resize.cfg new file mode 100644 index 000000000..d83546aac --- /dev/null +++ b/admission/tla/concurrency/resize.cfg @@ -0,0 +1,6 @@ +CONSTANTS Requests = {r1, r2, r3} + MaxCapacity = 3 + Mode = "reset-on-resize" +SPECIFICATION Spec +INVARIANTS PermitLifetime Accounting AdmissionBound ValidCapacity +CHECK_DEADLOCK FALSE diff --git a/admission/tla/rate/CheckDeadline.tla b/admission/tla/rate/CheckDeadline.tla new file mode 100644 index 000000000..c01afd210 --- /dev/null +++ b/admission/tla/rate/CheckDeadline.tla @@ -0,0 +1,28 @@ +------------------------ MODULE CheckDeadline ------------------------ +CONSTANT Mode +VARIABLES phase, expired, signaled, admitted +vars == <> +Init == /\ phase = "backend" + /\ expired = FALSE + /\ signaled = FALSE + /\ admitted = FALSE +Expire == /\ phase # "checked" + /\ ~expired + /\ expired' = TRUE + /\ UNCHANGED <> +Signal == /\ expired + /\ ~signaled + /\ signaled' = TRUE + /\ UNCHANGED <> +Answer == /\ phase = "backend" + /\ phase' = "answered" + /\ UNCHANGED <> +Check == /\ phase = "answered" + /\ phase' = "checked" + /\ admitted' = IF Mode = "err-only" THEN ~signaled + ELSE ~signaled /\ ~expired + /\ UNCHANGED <> +Next == Expire \/ Signal \/ Answer \/ Check +Spec == Init /\ [][Next]_vars +DeadlineEnforced == phase = "checked" /\ expired => ~admitted +====================================================================== diff --git a/admission/tla/rate/README.md b/admission/tla/rate/README.md new file mode 100644 index 000000000..cba33e770 --- /dev/null +++ b/admission/tla/rate/README.md @@ -0,0 +1,74 @@ +# Atomic rate policy and quota conservation + +`RatePolicy.tla` compares the accepted live-update rule with two rejected +candidate designs before implementation. This is a new integration, not a claim +of a defect in the maintained token-bucket dependency. + +The model represents one local budget with integer refill ticks and three +rate/burst pairs. Consume, refill and update actions interleave for five steps. +The checked update retains available allowance, clamped to the new burst. It +never refills solely because settings changed, including unchanged settings. +`Conservation` requires available plus spent tokens to stay within initial +allowance plus actual refill. `CoherentPolicy` rejects a partially published +rate/burst pair; `Bounded` checks the effective allowance. + +Correspondence and limits: + +- `../../rate.go` serializes Allow, SetPolicy and Policy under one mutex. The + maintained `golang.org/x/time/rate` v0.16.0 owns token accounting. SetLimitAt + settles old-rate refill before SetBurstAt at the same timestamp. The library + clamps its stored allowance lazily on the next observation; this model's + `tokens` is the observable allowance, not the library's private field. +- Direct tests cover fractional refill, rate/burst increases and decreases, + unchanged and invalid updates, cancellation, bounded retry delays, coherent + versions and simultaneous updates/admissions. The model does not prove Go + mutex refinement, floating-point arithmetic, time conversion or counter + overflow. It abstracts versions as policy identity and excludes invalid + policies; tests cover those obligations. +- Grouping and storage belong to the application. Independent group budgets + are tested through the runtime and generated example. The model does not + establish a fleet-wide quota or a cache eviction policy. +- `RateAdmission` checks are synchronous and require a backend honoring context + cancellation. Direct and generated tests check deadline failure, safe error + handling, preserved contexts and protected access ordering. Those properties + are outside this quota model. No goroutine abandonment or retry is modeled. + +Use the pinned repository TLC distribution (`tla2tools-1.7.4.jar`, TLC2 2.19, +revision 5a47802) with Java 25.0.5, from this directory: + +```sh +java -XX:+UseParallelGC -cp "$HOME/.local/share/tlaplus/tla2tools-1.7.4.jar" \ + tlc2.TLC -workers 1 -metadir /tmp/loom-rate-model \ + -config checked.cfg RatePolicy.tla +``` + +The checked model passes 122 distinct states (349 generated). `reset.cfg` +replenishes the bucket on update and violates conservation after 4 distinct +states. `split.cfg` publishes rate before burst and violates coherent policy +after 6 distinct states. These controls informed preserving allowance and +locking the complete policy transition. + +`CheckDeadline.tla` models an independently delayed context timer callback. The +initial implementation checked only `ctx.Err()` after a backend returned. An +independent review probe reproduced late admission in 9 of 10 calls with one Go +scheduler processor and a backend returning after its deadline. The +`deadline-err-only.cfg` model reproduces that defect in 8 distinct states: +deadline expiry, a successful backend answer, then admission before the timer +signals. `deadline-checked.cfg` passes 9 distinct states (12 generated) by checking +clock expiry as well as the published signal. Run it with the command above, +replacing the module with `CheckDeadline.tla` and selecting the named config. + +The shared private `rateContextError` owns that predicate for the rate bucket +and admission boundary. `rate_deadline_test.go` checks a fixed elapsed deadline +with pending timer publication deterministically, plus an actual late backend +answer without waiting on Done. The latter must reject with either timer +ordering. Both regressions failed before the fix. The model ends at the decision; +it does not claim that time cannot advance afterward or forcibly stop a callback. + +Run Go tests with `go test -race ./admission ./http/middleware`. The generated +boundary is `TestRateAdmissionExample` in +`codegen/generator/admission_integration_test.go`; select local Loom source +while developing an unpushed change. It generates the exact public example and +runs it under the race detector. Formal tools remain opt-in, outside all normal +build, test, hook, CI and release gates. Delete task-owned checker output after +validation and independent review; keep these sources and findings. diff --git a/admission/tla/rate/RatePolicy.tla b/admission/tla/rate/RatePolicy.tla new file mode 100644 index 000000000..55e6b0508 --- /dev/null +++ b/admission/tla/rate/RatePolicy.tla @@ -0,0 +1,47 @@ +------------------------- MODULE RatePolicy ------------------------- +EXTENDS Naturals, FiniteSets +CONSTANT Mode +VARIABLES rate, burst, tokens, spent, refilled, steps, target, phase +vars == <> +Policies == {<<1, 2>>, <<2, 1>>, <<1, 3>>} +Min(a, b) == IF a < b THEN a ELSE b +Init == /\ rate = 1 + /\ burst = 2 + /\ tokens = 2 + /\ spent = 0 + /\ refilled = 0 + /\ steps = 0 + /\ target = <<1, 2>> + /\ phase = "ready" +Consume == /\ tokens > 0 + /\ tokens' = tokens - 1 + /\ spent' = spent + 1 + /\ steps' = steps + 1 + /\ UNCHANGED <> +Refill == /\ tokens' = Min(burst, tokens + rate) + /\ refilled' = refilled + (tokens' - tokens) + /\ steps' = steps + 1 + /\ UNCHANGED <> +Update(p) == /\ phase = "ready" + /\ rate' = p[1] + /\ burst' = IF Mode = "split" THEN burst ELSE p[2] + /\ tokens' = CASE Mode = "reset" -> p[2] + [] Mode = "split" -> tokens + [] OTHER -> Min(tokens, p[2]) + /\ target' = p + /\ phase' = IF Mode = "split" THEN "updating" ELSE "ready" + /\ steps' = steps + 1 + /\ UNCHANGED <> +FinishUpdate == /\ phase = "updating" + /\ burst' = target[2] + /\ tokens' = Min(tokens, target[2]) + /\ phase' = "ready" + /\ steps' = steps + 1 + /\ UNCHANGED <> +Next == /\ steps < 5 + /\ Consume \/ Refill \/ (\E p \in Policies : Update(p)) \/ FinishUpdate +Spec == Init /\ [][Next]_vars +Conservation == spent + tokens <= 2 + refilled +CoherentPolicy == <> \in Policies +Bounded == tokens <= burst +===================================================================== diff --git a/admission/tla/rate/checked.cfg b/admission/tla/rate/checked.cfg new file mode 100644 index 000000000..71dccd212 --- /dev/null +++ b/admission/tla/rate/checked.cfg @@ -0,0 +1,4 @@ +CONSTANT Mode = "checked" +SPECIFICATION Spec +INVARIANTS Conservation CoherentPolicy Bounded +CHECK_DEADLOCK FALSE diff --git a/admission/tla/rate/deadline-checked.cfg b/admission/tla/rate/deadline-checked.cfg new file mode 100644 index 000000000..374308bb4 --- /dev/null +++ b/admission/tla/rate/deadline-checked.cfg @@ -0,0 +1,4 @@ +CONSTANT Mode = "checked" +SPECIFICATION Spec +INVARIANT DeadlineEnforced +CHECK_DEADLOCK FALSE diff --git a/admission/tla/rate/deadline-err-only.cfg b/admission/tla/rate/deadline-err-only.cfg new file mode 100644 index 000000000..9ff0a2466 --- /dev/null +++ b/admission/tla/rate/deadline-err-only.cfg @@ -0,0 +1,4 @@ +CONSTANT Mode = "err-only" +SPECIFICATION Spec +INVARIANT DeadlineEnforced +CHECK_DEADLOCK FALSE diff --git a/admission/tla/rate/reset.cfg b/admission/tla/rate/reset.cfg new file mode 100644 index 000000000..1b58cfa24 --- /dev/null +++ b/admission/tla/rate/reset.cfg @@ -0,0 +1,4 @@ +CONSTANT Mode = "reset" +SPECIFICATION Spec +INVARIANTS Conservation CoherentPolicy Bounded +CHECK_DEADLOCK FALSE diff --git a/admission/tla/rate/split.cfg b/admission/tla/rate/split.cfg new file mode 100644 index 000000000..43618ba49 --- /dev/null +++ b/admission/tla/rate/split.cfg @@ -0,0 +1,4 @@ +CONSTANT Mode = "split" +SPECIFICATION Spec +INVARIANTS Conservation CoherentPolicy Bounded +CHECK_DEADLOCK FALSE diff --git a/codegen/field_facts_regression_test.go b/codegen/field_facts_regression_test.go index 0efb37c4a..8765bfc1b 100644 --- a/codegen/field_facts_regression_test.go +++ b/codegen/field_facts_regression_test.go @@ -5,8 +5,9 @@ import ( "strings" "testing" - "github.com/CaliLuke/loom/expr" "github.com/stretchr/testify/require" + + "github.com/CaliLuke/loom/expr" ) func TestRequiredFieldQueriesDoNotCopyDescendants(t *testing.T) { @@ -84,3 +85,22 @@ func fieldFactsValidationAttribute(width int) *expr.AttributeExpr { object.Set("unrelated", &expr.AttributeExpr{Type: nested}) return &expr.AttributeExpr{Type: object, Validation: &expr.ValidationExpr{Required: required}} } + +func TestReferenceRequirementPreservesDeclaredJSONStorage(t *testing.T) { + field := &expr.AttributeExpr{Type: expr.String} + declaration := &expr.UserTypeExpr{TypeName: "Value", UID: "Value", AttributeExpr: &expr.AttributeExpr{ + Type: &expr.Object{{Name: "name", Attribute: field}}, + }} + occurrence := &expr.AttributeExpr{Type: declaration, Validation: &expr.ValidationExpr{Required: []string{"name"}}} + for _, pointer := range []bool{false, true} { + context := NewAttributeContext(pointer, false, false, "", NewNameScope()) + context.JSONPresence = true + facts := expr.NewObjectFields(occurrence) + require.True(t, facts.FieldFacts("name").Required) + require.Equal(t, OptionalPresence, context.FieldPresence(facts, "name", field)) + code := ValidationCode(occurrence, nil, context, true, false, false, "body") + require.Contains(t, code, "!body.Name.Present()") + require.NotContains(t, code, "body.Name == nil") + } + require.False(t, declaration.IsRequired("name")) +} diff --git a/codegen/generator/admission_integration_test.go b/codegen/generator/admission_integration_test.go new file mode 100644 index 000000000..926a93ea2 --- /dev/null +++ b/codegen/generator/admission_integration_test.go @@ -0,0 +1,59 @@ +package generator + +import ( + "fmt" + "os" + "path/filepath" + "testing" + + "github.com/stretchr/testify/require" + + "github.com/CaliLuke/loom/internal/testingx" +) + +// TestConcurrencyAdmissionExample generates the exact runnable public example, +// then checks protected admission and its declared HTTP and gRPC error mappings. +// No generator behavior changes; this is a consumer of the existing hooks. +func TestConcurrencyAdmissionExample(t *testing.T) { + runAdmissionExample(t, "concurrency") +} + +// TestRateAdmissionExample verifies early HTTP and protected endpoint quotas +// using the exact public example, including live policy and backend failures. +func TestRateAdmissionExample(t *testing.T) { + runAdmissionExample(t, "rate") +} + +func runAdmissionExample(t *testing.T, example string) { + t.Helper() + dir := t.TempDir() + require.NoError(t, os.Mkdir(filepath.Join(dir, "design"), 0o700)) + exampleDir := filepath.Join("../../docs/examples", example) + design, err := os.ReadFile(filepath.Join(exampleDir, "design.go.txt")) + require.NoError(t, err) + require.NoError(t, os.WriteFile(filepath.Join(dir, "design/design.go"), design, 0o600)) + mod := fmt.Sprintf("module example.com/admission\n\ngo 1.27.3\n\nrequire github.com/CaliLuke/loom v1.0.0\n\nreplace github.com/CaliLuke/loom => %s\n", loomModuleSource(t)) + require.NoError(t, os.WriteFile(filepath.Join(dir, "go.mod"), []byte(mod), 0o600)) + for _, args := range [][]string{ + {"get", "-tool", "github.com/CaliLuke/loom/cmd/loom"}, + {"tool", "loom", "gen", "example.com/admission/design"}, + } { + output, err := testingx.RunCmd(dir, "go", args...) + require.NoError(t, err, "%v\n%s", args, output) + } + for _, file := range []struct{ source, target string }{ + {filepath.Join(exampleDir, "main.go.txt"), "main.go"}, + {filepath.Join("testdata", example+"_admission_test.go.txt"), "admission_test.go"}, + } { + source, err := os.ReadFile(file.source) + require.NoError(t, err) + require.NoError(t, os.WriteFile(filepath.Join(dir, file.target), source, 0o600)) + } + for _, args := range [][]string{ + {"mod", "tidy"}, + {"test", "-race", "./..."}, + } { + output, err := testingx.RunCmd(dir, "go", args...) + require.NoError(t, err, "%v\n%s", args, output) + } +} diff --git a/codegen/generator/testdata/concurrency_admission_test.go.txt b/codegen/generator/testdata/concurrency_admission_test.go.txt new file mode 100644 index 000000000..176fdb0cc --- /dev/null +++ b/codegen/generator/testdata/concurrency_admission_test.go.txt @@ -0,0 +1,159 @@ +package main + +import ( + "context" + "net/http" + "net/http/httptest" + "strings" + "sync" + "testing" + "testing/synctest" + "time" + + workpb "example.com/admission/gen/grpc/work/pb" + grpcserver "example.com/admission/gen/grpc/work/server" + httpserver "example.com/admission/gen/http/work/server" + "example.com/admission/gen/work" + "github.com/CaliLuke/loom/admission" + loomhttp "github.com/CaliLuke/loom/http" + loom "github.com/CaliLuke/loom/pkg" + "github.com/stretchr/testify/require" + "google.golang.org/grpc/codes" + "google.golang.org/grpc/metadata" + "google.golang.org/grpc/status" + "google.golang.org/protobuf/proto" +) + +type testService struct { + workService + finish <-chan struct{} + calls int + checks int + deny bool + deadline time.Time +} + +func (s *testService) Run(ctx context.Context, _ *work.RunPayload) (string, error) { + s.calls++ + if ctx.Value(actorKey{}) != true { + return "", loom.Fault("authenticated context lost") + } + s.deadline, _ = ctx.Deadline() + <-s.finish + return "done", nil +} + +func (s *testService) AuthorizeWorkRun(ctx context.Context) error { + s.checks++ + if s.deny { + return loom.PermanentError("forbidden", "work permission denied") + } + return s.workService.AuthorizeWorkRun(ctx) +} + +func TestGeneratedAdmission(t *testing.T) { + synctest.Test(t, func(t *testing.T) { + budget, err := admission.NewConcurrency(1) + require.NoError(t, err) + finish := make(chan struct{}) + unblock := sync.OnceFunc(func() { + close(finish) + }) + defer unblock() + service := &testService{workService: workService{key: "valid"}, finish: finish} + endpoints := work.NewEndpoints(service, service) + endpoints.Use(budget.Middleware) + mux := loomhttp.NewMuxer() + httpserver.Mount(mux, httpserver.New(endpoints, mux, loomhttp.RequestDecoder, loomhttp.ResponseEncoder, nil, nil)) + request := func(key, body string) *httptest.ResponseRecorder { + req := httptest.NewRequest(http.MethodPost, "/work", strings.NewReader(body)) + req.Header.Set("Content-Type", "application/json") + req.Header.Set("X-API-Key", key) + recorder := httptest.NewRecorder() + mux.ServeHTTP(recorder, req) + return recorder + } + ctx, cancel := context.WithTimeout(context.Background(), time.Minute) + defer cancel() + go func() { + result, err := endpoints.Run(ctx, &work.RunPayload{Key: "valid", Seconds: 1}) + require.NoError(t, err) + require.Equal(t, "done", result) + }() + synctest.Wait() + deadline, ok := ctx.Deadline() + require.True(t, ok) + require.Equal(t, deadline, service.deadline) + require.GreaterOrEqual(t, service.checks, 2, "access is checked before admission and again before service") + cancel() + synctest.Wait() + require.Equal(t, 1, budget.Snapshot().InFlight) + for _, tc := range []struct { + name, key, body string + status int + deny bool + }{ + {"saturated", "valid", "{\"seconds\":1}", 503, false}, + {"invalid", "valid", "{\"seconds\":0}", 400, false}, + {"unauthenticated", "invalid", "{\"seconds\":1}", 401, false}, + {"forbidden", "valid", "{\"seconds\":1}", 403, true}, + } { + service.deny = tc.deny + recorder := request(tc.key, tc.body) + require.Equal(t, tc.status, recorder.Code, "%s: %s", tc.name, recorder.Body.String()) + require.Equal(t, 1, service.calls, tc.name) + if tc.status == 503 { + require.Contains(t, recorder.Body.String(), "overloaded") + require.Contains(t, recorder.Body.String(), "service capacity exhausted") + require.Empty(t, recorder.Header().Get("Retry-After")) + } + } + service.deny = false + require.Equal(t, uint64(1), budget.Snapshot().Rejected, "invalid and denied requests never reach the limiter") + grpc := grpcserver.New(endpoints, nil) + grpcCtx := metadata.NewIncomingContext(context.Background(), metadata.Pairs("x-api-key", "valid")) + _, err = grpc.Run(grpcCtx, &workpb.RunRequest{Seconds: proto.Int64(1)}) + require.Equal(t, codes.Unavailable, status.Code(err)) + require.Equal(t, 1, service.calls) + _, err = endpoints.Run(ctx, &work.RunPayload{Key: "valid", Seconds: 1}) + require.ErrorIs(t, err, context.Canceled) + require.Equal(t, uint64(2), budget.Snapshot().Rejected) + unblock() + synctest.Wait() + require.Zero(t, budget.Snapshot().InFlight) + recorder := request("valid", "{\"seconds\":1}") + require.Equal(t, http.StatusOK, recorder.Code, recorder.Body.String()) + require.Equal(t, 2, service.calls) + require.Equal(t, uint64(2), budget.Snapshot().Admitted) + }) +} + +func TestGeneratedChecksRepeatInsidePermit(t *testing.T) { + for _, mutation := range []string{"payload", "permission", "cancel"} { + t.Run(mutation, func(t *testing.T) { + budget, err := admission.NewConcurrency(1) + require.NoError(t, err) + service := &testService{workService: workService{key: "valid"}} + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + mutate := func(next loom.Endpoint) loom.Endpoint { + return func(ctx context.Context, req any) (any, error) { + switch mutation { + case "payload": + req.(*work.RunPayload).Seconds = 0 + case "permission": + service.deny = true + case "cancel": + cancel() + } + return next(ctx, req) + } + } + endpoint := work.NewRunEndpoint(service, service, service.APIKeyAuth, mutate, budget.Middleware) + _, err = endpoint(ctx, &work.RunPayload{Key: "valid", Seconds: 1}) + require.Error(t, err) + require.Zero(t, service.calls) + require.Equal(t, admission.ConcurrencySnapshot{Capacity: 1, Admitted: 1}, budget.Snapshot()) + }) + } +} diff --git a/codegen/generator/testdata/rate_admission_test.go.txt b/codegen/generator/testdata/rate_admission_test.go.txt new file mode 100644 index 000000000..3c4e5ec92 --- /dev/null +++ b/codegen/generator/testdata/rate_admission_test.go.txt @@ -0,0 +1,201 @@ +package main + +import ( + "context" + "errors" + "net/http" + "net/http/httptest" + "strings" + "testing" + "testing/synctest" + "time" + + workserver "example.com/admission/gen/http/work/server" + "example.com/admission/gen/work" + "github.com/CaliLuke/loom/admission" + loomhttp "github.com/CaliLuke/loom/http" + "github.com/CaliLuke/loom/http/middleware" + loom "github.com/CaliLuke/loom/pkg" + "github.com/stretchr/testify/require" +) + +type testService struct { + workService + calls, checks int + denied bool +} + +func (s *testService) Run(_ context.Context, payload *work.RunPayload) (string, error) { + s.calls++ + return payload.Message, nil +} + +func (s *testService) AuthorizeWorkRun(ctx context.Context) error { + s.checks++ + if s.denied { + return loom.PermanentError("forbidden", "permission denied") + } + return s.workService.AuthorizeWorkRun(ctx) +} + +func TestEarlyRateAdmission(t *testing.T) { + synctest.Test(t, func(t *testing.T) { + early := newRate(t, 1, 1) + service := &testService{workService: workService{secret: "valid"}} + backendChecks := 0 + handler := quotaHTTP(t, service, early, func(context.Context, string) (admission.RateDecision, error) { + backendChecks++ + return admission.RateDecision{Allowed: true}, nil + }) + response := quotaRequest(handler, "valid-alpha", "{\"message\":\"one\"}") + require.Equal(t, 200, response.Code, response.Body.String()) + checks := service.checks + response = quotaRequest(handler, "bad-key", "{malformed") + require.Equal(t, 429, response.Code) + require.Equal(t, "1", response.Header().Get("Retry-After")) + require.Equal(t, checks, service.checks, "early rejection precedes authentication and decoding") + require.Equal(t, 1, backendChecks) + require.Equal(t, 1, service.calls) + require.NoError(t, early.SetPolicy(admission.RatePolicy{PerSecond: 2, Burst: 2})) + response = quotaRequest(handler, "valid-alpha", "{\"message\":\"two\"}") + require.Equal(t, 429, response.Code, "an increase must not refill spent allowance") + time.Sleep(500 * time.Millisecond) + response = quotaRequest(handler, "valid-alpha", "{\"message\":\"two\"}") + require.Equal(t, 200, response.Code, response.Body.String()) + require.Equal(t, 2, service.calls) + }) +} + +func TestAuthenticatedRateAdmission(t *testing.T) { + synctest.Test(t, func(t *testing.T) { + groups := map[string]*admission.Rate{"alpha": newRate(t, 1, 1), "beta": newRate(t, 1, 1)} + service := &testService{workService: workService{secret: "valid"}} + backendChecks := 0 + handler := quotaHTTP(t, service, newRate(t, 100, 100), func(ctx context.Context, key string) (admission.RateDecision, error) { + backendChecks++ + bucket, ok := groups[key] + require.True(t, ok, "only trusted bounded groups reach the backend") + return bucket.Allow(ctx) + }) + for _, tc := range []struct { + key, body string + status int + }{ + {"valid-alpha", "{\"message\":\"\"}", 400}, + {"invalid", "{\"message\":\"ok\"}", 401}, + } { + response := quotaRequest(handler, tc.key, tc.body) + require.Equal(t, tc.status, response.Code, response.Body.String()) + } + require.Zero(t, backendChecks) + response := quotaRequest(handler, "valid-alpha", "{\"message\":\"ok\"}") + require.Equal(t, 200, response.Code, response.Body.String()) + response = quotaRequest(handler, "valid-alpha", "{\"message\":\"ok\"}") + require.Equal(t, 429, response.Code, response.Body.String()) + require.Equal(t, "1", response.Header().Get("Retry-After")) + require.Contains(t, response.Body.String(), "authenticated quota exhausted") + response = quotaRequest(handler, "valid-beta", "{\"message\":\"ok\"}") + require.Equal(t, 200, response.Code, response.Body.String()) + require.Equal(t, 2, service.calls) + checks := backendChecks + service.denied = true + response = quotaRequest(handler, "valid-alpha", "{\"message\":\"ok\"}") + require.Equal(t, 403, response.Code, response.Body.String()) + require.Equal(t, checks, backendChecks, "authorization still precedes an exhausted quota") + service.denied = false + policy := admission.RatePolicy{PerSecond: 2, Burst: 4} + require.NoError(t, groups["alpha"].SetPolicy(policy)) + response = quotaRequest(handler, "valid-alpha", "{\"message\":\"ok\"}") + require.Equal(t, 429, response.Code) + time.Sleep(500 * time.Millisecond) + response = quotaRequest(handler, "valid-alpha", "{\"message\":\"ok\"}") + require.Equal(t, 200, response.Code, response.Body.String()) + require.NoError(t, groups["alpha"].SetPolicy(policy)) + response = quotaRequest(handler, "valid-alpha", "{\"message\":\"ok\"}") + require.Equal(t, 429, response.Code) + require.Equal(t, 3, service.calls) + }) +} + +func TestRateBackendFailure(t *testing.T) { + for _, timeout := range []bool{false, true} { + synctest.Test(t, func(t *testing.T) { + service := &testService{workService: workService{secret: "valid"}} + handler := quotaHTTP(t, service, newRate(t, 100, 100), func(ctx context.Context, _ string) (admission.RateDecision, error) { + if timeout { + <-ctx.Done() + return admission.RateDecision{Allowed: true}, nil + } + return admission.RateDecision{}, errors.New("sensitive database and identity") + }) + response := quotaRequest(handler, "valid-alpha", "{\"message\":\"ok\"}") + require.Equal(t, 503, response.Code, response.Body.String()) + require.Contains(t, response.Body.String(), "rate_limit_unavailable") + require.NotContains(t, response.Body.String(), "sensitive") + require.Empty(t, response.Header().Get("Retry-After")) + require.Zero(t, service.calls) + }) + } +} + +func TestRateAdmissionRetainsProtectedRecheck(t *testing.T) { + synctest.Test(t, func(t *testing.T) { + service := &testService{workService: workService{secret: "valid"}} + bucket := newRate(t, 1, 1) + handler := quotaHTTP(t, service, newRate(t, 100, 100), func(ctx context.Context, _ string) (admission.RateDecision, error) { + decision, err := bucket.Allow(ctx) + service.denied = true + return decision, err + }) + response := quotaRequest(handler, "valid-alpha", "{\"message\":\"ok\"}") + require.Equal(t, 403, response.Code, response.Body.String()) + require.Zero(t, service.calls) + decision, err := bucket.Allow(context.Background()) + require.NoError(t, err) + require.False(t, decision.Allowed, "later rejection does not refund an accounted attempt") + }) +} + +func TestRateUnknownDelayOmitsHeader(t *testing.T) { + service := &testService{workService: workService{secret: "valid"}} + handler := quotaHTTP(t, service, newRate(t, 100, 100), func(context.Context, string) (admission.RateDecision, error) { + return admission.RateDecision{}, nil + }) + response := quotaRequest(handler, "valid-alpha", "{\"message\":\"ok\"}") + require.Equal(t, 429, response.Code, response.Body.String()) + require.Empty(t, response.Header().Get("Retry-After")) + require.Zero(t, service.calls) +} + +func newRate(t *testing.T, perSecond float64, burst int) *admission.Rate { + t.Helper() + bucket, err := admission.NewRate(admission.RatePolicy{PerSecond: perSecond, Burst: burst}) + require.NoError(t, err) + return bucket +} + +func quotaHTTP(t *testing.T, service *testService, bucket *admission.Rate, check admission.RateCheck) http.Handler { + t.Helper() + quota, err := admission.NewRateAdmission(check, 10*time.Millisecond) + require.NoError(t, err) + early, err := admission.NewRateAdmission(func(ctx context.Context, _ string) (admission.RateDecision, error) { + return bucket.Allow(ctx) + }, 10*time.Millisecond) + require.NoError(t, err) + endpoints := work.NewEndpoints(service, service) + endpoints.Use(quota.Middleware(authenticatedGroup, quotaError)) + mux := loomhttp.NewMuxer() + workserver.Mount(mux, workserver.New(endpoints, mux, loomhttp.RequestDecoder, loomhttp.ResponseEncoder, nil, nil)) + return middleware.RateLimit(early, func(*http.Request) (string, error) { + return "process", nil + })(mux) +} + +func quotaRequest(handler http.Handler, key, body string) *httptest.ResponseRecorder { + req := httptest.NewRequest(http.MethodPost, "/work", strings.NewReader(body)) + req.Header.Set("Content-Type", "application/json") + req.Header.Set("X-API-Key", key) + response := httptest.NewRecorder() + handler.ServeHTTP(response, req) + return response +} diff --git a/codegen/generator/testdata/security_required_scopes.go.txt b/codegen/generator/testdata/security_required_scopes.go.txt index 66987675e..61a0eed64 100644 --- a/codegen/generator/testdata/security_required_scopes.go.txt +++ b/codegen/generator/testdata/security_required_scopes.go.txt @@ -23,6 +23,11 @@ func TestScopeCallbacks(t *testing.T) { wantCalls []string wantError bool }{ + {"machine scope", "flow", map[string][]string{"oauth": {"machine"}}, []string{"oauth"}, false}, + {"machine scope missing", "flow", map[string][]string{"oauth": {"oauth"}}, []string{"oauth"}, true}, + {"isolated valid", "isolated", map[string][]string{"oauth": {"shared"}, "second": {"oauth"}, "key": {"admin"}}, []string{"oauth", "second", "key"}, false}, + {"isolated missing role", "isolated", map[string][]string{"oauth": {"shared"}, "second": {"oauth"}}, []string{"oauth", "second", "key"}, true}, + {"isolated wrong scheme grant", "isolated", map[string][]string{"oauth": {"oauth"}, "second": {"shared"}, "key": {"admin"}}, []string{"oauth"}, true}, {"compound valid", "compound", map[string][]string{"basic": {"shared", "basic"}, "key": {"shared", "key"}, "jwt": {"shared", "read"}, "oauth": {"shared", "oauth"}}, []string{"basic", "key", "jwt", "oauth"}, false}, {"compound missing", "compound", map[string][]string{"basic": {"basic"}}, []string{"basic"}, true}, {"first alternative", "alternative", map[string][]string{"jwt": {"read"}}, []string{"jwt", "unscoped"}, false}, @@ -35,6 +40,9 @@ func TestScopeCallbacks(t *testing.T) { check := func(name string, declared, required []string) { authCalls = append(authCalls, strings.TrimPrefix(name, "scope_")) for _, scope := range required { + if tc.method == "isolated" && name == "key" { + continue + } require.True(t, slices.Contains(declared, scope), "%s requires undeclared %s", name, scope) } } @@ -52,10 +60,29 @@ func TestScopeCallbacks(t *testing.T) { } oauth := func(ctx context.Context, _ string, s *security.OAuth2Scheme) (context.Context, error) { check(s.Name, s.Scopes, s.RequiredScopes) + if s.Name == "oauth" { + require.Len(t, s.Flows, 3) + require.Equal(t, map[string]string{"oauth": "User consent", "shared": "User grant"}, s.Flows[0].Scopes) + require.Equal(t, map[string]string{"machine": "Machine grant", "shared": "Machine permission"}, s.Flows[1].Scopes) + require.NotNil(t, s.Flows[2].Scopes) + require.Empty(t, s.Flows[2].Scopes) + } + if s.Name == "second" { + require.Len(t, s.Flows, 2) + require.Equal(t, map[string]string{"oauth": "no description", "shared": "no description"}, s.Flows[0].Scopes) + require.NotNil(t, s.Flows[1].Scopes) + require.Empty(t, s.Flows[1].Scopes) + } return ctx, s.Validate(tc.grants[strings.TrimPrefix(s.Name, "scope_")]) } var err error switch tc.method { + case "flow": + endpoint := scopes.NewFlowEndpoint(svc, oauth) + _, err = endpoint(context.Background(), &scopes.FlowPayload{Token: "token"}) + case "isolated": + endpoint := scopes.NewIsolatedEndpoint(svc, oauth, key) + _, err = endpoint(context.Background(), &scopes.IsolatedPayload{Token: "token", Key: "key"}) case "compound": endpoint := scopes.NewCompoundEndpoint(svc, basic, key, jwt, oauth) _, err = endpoint(context.Background(), &scopes.CompoundPayload{User: "user", Password: "pass", Key: "key", Token: "jwt", Access: "oauth"}) @@ -87,3 +114,13 @@ func (s *scopeService) Alternative(context.Context, *scopes.AlternativePayload) s.calls = append(s.calls, "alternative") return nil } + +func (s *scopeService) Isolated(context.Context, *scopes.IsolatedPayload) error { + s.calls = append(s.calls, "isolated") + return nil +} + +func (s *scopeService) Flow(context.Context, *scopes.FlowPayload) error { + s.calls = append(s.calls, "flow") + return nil +} diff --git a/codegen/service/endpoint_method_section.go b/codegen/service/endpoint_method_section.go index a8d2a63ea..f03f4d7a9 100644 --- a/codegen/service/endpoint_method_section.go +++ b/codegen/service/endpoint_method_section.go @@ -170,6 +170,13 @@ func buildOAuth2SchemeAuth(group *jen.Group, scheme *SchemeData, payload, contex for _, flow := range scheme.Flows { flows.Op("&").Add(codegen.Expr("security.OAuthFlow")).CustomFunc(multilineValues, func(fields *jen.Group) { fields.Id("Type").Op(":").Lit(flow.Type()) + if flow.Scopes != nil { + scopes := make(jen.Dict, len(flow.Scopes)) + for _, scope := range flow.Scopes { + scopes[jen.Lit(scope.Name)] = jen.Lit(scope.Description) + } + fields.Id("Scopes").Op(":").Map(jen.String()).String().Values(scopes) + } if flow.AuthorizationURL != "" { fields.Id("AuthorizationURL").Op(":").Lit(flow.AuthorizationURL) } diff --git a/codegen/service/security_required_scopes_test.go b/codegen/service/security_required_scopes_test.go index 3ca97ae7f..de1c50f76 100644 --- a/codegen/service/security_required_scopes_test.go +++ b/codegen/service/security_required_scopes_test.go @@ -20,6 +20,8 @@ func TestEndpointRequiredScopeOwnership(t *testing.T) { method string want map[string][][]string }{ + {"flow", map[string][][]string{"oauth": {{"machine"}}}}, + {"isolated", map[string][][]string{"oauth": {{"shared"}}, "second": {{"oauth"}}, "key": {{"admin"}}}}, {"compound", map[string][][]string{"scope_basic": {{"shared", "basic"}}, "key": {{"shared", "key"}}, "scope_jwt": {{"shared", "read"}}, "oauth": {{"shared", "oauth"}}}}, {"alternative", map[string][][]string{"scope_jwt": {{"read"}, {"write"}}, "unscoped": {{}, {}}}}, } { diff --git a/codegen/service/service_data_methods.go b/codegen/service/service_data_methods.go index 9907629bd..5aaa67923 100644 --- a/codegen/service/service_data_methods.go +++ b/codegen/service/service_data_methods.go @@ -2,6 +2,7 @@ package service import ( "fmt" + "slices" "github.com/CaliLuke/loom/codegen" "github.com/CaliLuke/loom/expr" @@ -413,7 +414,7 @@ func BuildSchemeData(s *expr.SchemeExpr, m *expr.MethodExpr) *SchemeData { if !expr.IsObject(m.Payload.Type) { return nil } - scopes := schemeScopeNames(s) + scopes := s.ScopeNames() switch s.Kind { case expr.BasicAuthKind: userAtt := expr.TaggedAttribute(m.Payload, "security:username") @@ -445,23 +446,25 @@ func BuildSchemeData(s *expr.SchemeExpr, m *expr.MethodExpr) *SchemeData { data := buildCredentialSchemeData(s, m, "security:accesstoken", scopes) if data != nil { data.Flows = s.Flows + if slices.ContainsFunc(s.Flows, func(flow *expr.FlowExpr) bool { + return flow.Scopes != nil + }) { + data.Flows = make([]*expr.FlowExpr, len(s.Flows)) + for i, flow := range s.Flows { + copy := *flow + copy.Scopes = s.FlowScopes(flow) + if copy.Scopes == nil { + copy.Scopes = []*expr.ScopeExpr{} + } + data.Flows[i] = © + } + } } return data } return nil } -func schemeScopeNames(s *expr.SchemeExpr) []string { - if len(s.Scopes) == 0 { - return nil - } - scopes := make([]string, len(s.Scopes)) - for i, scope := range s.Scopes { - scopes[i] = scope.Name - } - return scopes -} - func buildCredentialSchemeData(s *expr.SchemeExpr, m *expr.MethodExpr, tag string, scopes []string) *SchemeData { keyAtt := expr.TaggedAttribute(m.Payload, tag) if keyAtt == "" { diff --git a/codegen/service/service_data_security.go b/codegen/service/service_data_security.go index e3ece6b09..5eaf08728 100644 --- a/codegen/service/service_data_security.go +++ b/codegen/service/service_data_security.go @@ -74,7 +74,8 @@ type ( KeyAttr string // Scopes lists the scopes that apply to the scheme. Scopes []string - // RequiredScopes lists this requirement's scopes declared by the scheme. + // RequiredScopes holds this requirement's names for the scheme: declared + // OAuth scopes or application permission/role names for non-OAuth schemes. RequiredScopes []string // Flows describes the OAuth2 flows. Flows []*expr.FlowExpr diff --git a/codegen/service/service_data_types.go b/codegen/service/service_data_types.go index 7390b083d..ddf95891b 100644 --- a/codegen/service/service_data_types.go +++ b/codegen/service/service_data_types.go @@ -279,7 +279,7 @@ func buildUntaggedUnionValidations(union *expr.Union, scope *codegen.NameScope, types := collectTypes(&expr.AttributeExpr{Type: union}, scope, make(map[string]struct{}), loc) validations := make([]*unionValidationData, 0, len(types)) for _, data := range types { - if !expr.IsObject(data.Type) && !expr.IsArray(data.Type) { + if !expr.IsObject(data.Type) && !expr.IsArray(data.Type) && !expr.IsMap(data.Type) { continue } ctx := typeContext(scope) diff --git a/codegen/service/service_data_union_order_test.go b/codegen/service/service_data_union_order_test.go index 64309edad..7dbdc4335 100644 --- a/codegen/service/service_data_union_order_test.go +++ b/codegen/service/service_data_union_order_test.go @@ -343,3 +343,30 @@ func makeOptionalObjectUnionForFormTest() *expr.Union { }, } } + +func TestUntaggedUnionEmitsNamedMapValidation(t *testing.T) { + root := expr.RunDSL(t, func() { + labels := dsl.Type("Labels", dsl.MapOf(dsl.String, dsl.String), func() { + dsl.MinLength(1) + }) + page := dsl.Type("Page", func() { + dsl.Attribute("labels", labels) + }) + items := dsl.Type("Items", dsl.ArrayOf(dsl.String)) + dsl.Service("maps", func() { + dsl.Method("read", func() { + dsl.Result(dsl.OneOf(page, items), dsl.Untagged) + }) + }) + }) + data := NewServicesData(root).Get("maps") + require.Len(t, data.unions, 1) + names := make(map[string]bool) + for _, validation := range data.unions[0].validations { + names[validation.data.Name] = true + if validation.data.Name == "ValidateLabels" { + require.Contains(t, validation.data.Validate, "len(result)") + } + } + require.True(t, names["ValidateLabels"]) +} diff --git a/codegen/service/testdata/security_required_scopes.go b/codegen/service/testdata/security_required_scopes.go index 748316bb5..0c23e94dc 100644 --- a/codegen/service/testdata/security_required_scopes.go +++ b/codegen/service/testdata/security_required_scopes.go @@ -2,6 +2,7 @@ package testdata import ( . "github.com/CaliLuke/loom/dsl" + "github.com/CaliLuke/loom/expr" ) // SecurityRequiredScopesDSL exercises compound and alternative scope ownership. @@ -20,12 +21,9 @@ func SecurityRequiredScopesDSL() { Scope("write") Scope("shared") }) - oauth := OAuth2Security("oauth", func() { - AuthorizationCodeFlow("/authorize", "/token", "/refresh") - Scope("oauth") - Scope("shared") - }) + oauth, second := securityFlowScopeSchemes() Service("scopes", func() { + securityFlowScopeMethods(oauth, second, key) Method("compound", func() { Security(basic, key, jwt, oauth, func() { Scope("shared") @@ -100,3 +98,57 @@ func SecurityOAuthScopesDSL() { }) }) } + +func securityFlowScopeSchemes() (*expr.SchemeExpr, *expr.SchemeExpr) { + oauth := OAuth2Security("oauth", func() { + AuthorizationCodeFlow("/authorize", "/token", "/refresh", func() { + Scope("oauth", "Superseded description") + Scope("oauth", "User consent") + Scope("shared", "User grant") + }) + ClientCredentialsFlow("https://example.com/token", "", func() { + Scope("machine", "Machine grant") + Scope("shared", "Machine permission") + }) + ImplicitFlow("https://example.com/authorize", "", func() {}) + }) + second := OAuth2Security("second", func() { + Scope("oauth", "Superseded default") + ClientCredentialsFlow("https://example.com/token", "") + ImplicitFlow("https://example.com/authorize", "", func() {}) + Scope("oauth") + Scope("shared") + }) + return oauth, second +} + +func securityFlowScopeMethods(oauth, second, key *expr.SchemeExpr) { + Method("flow", func() { + Security(oauth, func() { + ScopeFor(oauth, "machine") + }) + Payload(func() { + AccessToken("token") + Required("token") + }) + HTTP(func() { + GET("/flow") + }) + }) + Method("isolated", func() { + Security(oauth, second, key, func() { + ScopeFor(oauth, "shared") + ScopeFor(second, "oauth") + RoleFor(key, "admin") + }) + Payload(func() { + AccessToken("token") + APIKey("key", "key") + Required("token", "key") + }) + HTTP(func() { + GET("/isolated") + Header("key:X-Key") + }) + }) +} diff --git a/codegen/transformer.go b/codegen/transformer.go index 817071f3a..bbd6380ff 100644 --- a/codegen/transformer.go +++ b/codegen/transformer.go @@ -317,7 +317,7 @@ func (a *AttributeContext) FieldPresence(parent expr.FieldFactsProvider, name st } if a.JSONPresence && parent != nil { facts := parent.FieldFacts(name) - if facts.ParentPresent && (!facts.Required || facts.HasDefault) { + if facts.ParentPresent && (!facts.LayoutRequired || facts.HasDefault) { return OptionalPresence } } diff --git a/codegen/unionjson/README.md b/codegen/unionjson/README.md index 2f72b856a..184931799 100644 --- a/codegen/unionjson/README.md +++ b/codegen/unionjson/README.md @@ -92,17 +92,19 @@ continues to use the existing discriminator/value envelope. The evaluated `expr` graph, effective validation, and occurrence-specific value plans are the semantic source of truth. Extend their supported-branch admission to concrete named arrays whose elements use the currently admitted object-field -grammar: primitives, concrete named objects, or recursively arrays of these. -Apply the same structural restrictions recursively to named objects reached -through array elements; crossing an array must not bypass validation. Reject -maps, nested union fields, inline object leaves, and scalar root branches. +grammar: primitives, concrete named objects, and recursively arrays or +string-keyed maps of these, plus nested untagged unions whose own branches obey +the same contract. Apply the same restrictions through every collection and +union. Reject non-string map keys, tagged nested unions, inline object leaves, +and scalar or map root branches. Map fields were added in #1041 and nested +untagged unions in #1036; importer admission remains separately bounded. Existing primitive `Any` fields remain opaque leaves; they do not become an implicit union branch or rescue a failed match. Shared codegen analysis projects an immutable JSON matching description from that evaluated contract and the actual emitted representation. The description includes root shape, member names, presence/nullability, unknown-member policy, -array element shape, and effective constraints. HTTP supplies its finalized +array element and map value shapes, and effective constraints. HTTP supplies its finalized layout and body/view selection; it does not select another semantic policy. Service codegen supplies its own emitted layout through the same analysis. @@ -337,3 +339,42 @@ Redocly 2.46.1. Direct matcher/projection tests also pass with the race detector The TLC results and their assumptions are recorded beside the model. This is bounded ticket validation, not an exhaustive exported-design audit. + +## Map field extension (#1041) + +Map membership requires an object and every value must match the captured value +shape. Empty maps satisfy that universal predicate; entry-count bounds can still +reject them. Named map aliases preserve effective bounds, nullable maps and +nullable values remain distinct, and recursive object values reuse descriptor +identity. Key spelling is not a schema predicate in the existing map projection; +typed key decoding and authored validators retain their independent obligation. +No `propertyNames` behavior is introduced by this ticket. + +`Plan` visits map values before capturing per-member spelling, visibility and +closed-object rules. `buildUntaggedUnionValidations` includes named maps so every +emitted validation call has a definition. The generated HTTP fixture originally +failed compilation with an undefined `ValidateLabels`; the shared collection +filter now emits it. Direct service analysis covers that discovery boundary. + +The map model and bounded verification evidence live beside +[the matching model](../../pkg/tla/untagged_json/README.md). + +## Nested untagged composition (#1036) + +A nested union is an exact-one predicate over its alternatives. Arrays and maps +require every supplied value to satisfy that predicate. Empty collections retain +their ordinary length semantics. Outer matching still requires the same unique +schema and Go candidate, and assignment remains transactional. Alias/union-only +descriptor cycles are invalid because they revisit the same wire value; recursive +objects and collections consume input and remain supported. + +Nested generated codecs return a private typed mismatch for zero/multiple matches +or differing schema/decoder identities. `DecodeJSONCandidate` recognizes that +classification through JSON error wrapping and rejects just that candidate. +Malformed descriptions, predicate evaluator failures and unexpected codec errors +retain their fatal causes. An undifferentiated error would let a rejected nested +candidate block an otherwise valid outer alternative. The generated fixture's +`Fallback` union exercises that case with API-level closed objects. + +See the [nested models](../../pkg/tla/untagged_json/README.md) for the bounded +exact-one and error-classification alternatives and their implementation seams. diff --git a/codegen/unionjson/shape.go b/codegen/unionjson/shape.go index e837abbf3..e6e89c8ad 100644 --- a/codegen/unionjson/shape.go +++ b/codegen/unionjson/shape.go @@ -21,6 +21,9 @@ func renderShape(b *strings.Builder, name string, root *loom.JSONShape) { indexes[node] = len(nodes) nodes = append(nodes, node) visit(node.Child) + for _, branch := range node.Branches { + visit(branch) + } for _, field := range node.Fields { visit(field.Shape) } @@ -32,6 +35,13 @@ func renderShape(b *strings.Builder, name string, root *loom.JSONShape) { if node.Child != nil { fmt.Fprintf(b, "Child: &%sNodes[%d],\n", name, indexes[node.Child]) } + if len(node.Branches) > 0 { + b.WriteString("Branches: []*loom.JSONShape{\n") + for _, branch := range node.Branches { + fmt.Fprintf(b, "&%sNodes[%d],\n", name, indexes[branch]) + } + b.WriteString("},\n") + } if len(node.Fields) > 0 { b.WriteString("Fields: []loom.JSONShapeField{\n") for _, field := range node.Fields { diff --git a/codegen/unionjson/union.go b/codegen/unionjson/union.go index af7044fcf..21bb7de36 100644 --- a/codegen/unionjson/union.go +++ b/codegen/unionjson/union.go @@ -76,10 +76,21 @@ func Plan(source *expr.AttributeExpr, fieldName func(string) string, schema, clo walk(ut.Attribute()) return } + if union := expr.AsUnion(att.Type); union != nil { + for _, branch := range union.Values { + walk(branch.Attribute) + } + return + } if array := expr.AsArray(att.Type); array != nil { walk(array.ElemType) return } + if mapping := expr.AsMap(att.Type); mapping != nil { + walk(mapping.KeyType) + walk(mapping.ElemType) + return + } if object := expr.AsObject(att.Type); object != nil { if schema && closedObjects { if att.Meta == nil { @@ -88,23 +99,7 @@ func Plan(source *expr.AttributeExpr, fieldName func(string) string, schema, clo att.Meta["openapi:additionalProperties"] = []string{"false"} } for _, field := range *object { - visible := true - if hidden, ok := field.Attribute.Meta.Last("openapi:generate"); schema && ok && hidden == "false" { - visible = false - } - var numeric expr.Kind - if expr.IsPrimitive(field.Attribute.Type) { - kind := field.Attribute.Type.Kind() - if kind >= expr.IntKind && kind <= expr.Float64Kind { - numeric = kind - } - } - request.Fields = append(request.Fields, expr.ValueFieldPolicy{ - Parent: att, Target: field.Attribute, Name: field.Name, - WireName: expr.JSONFieldName(fieldName(field.Name), field.Attribute), - Visible: visible, Required: att.IsRequired(field.Name), - Presence: expr.ValueFieldRetain, NumericKind: numeric, - }) + request.Fields = append(request.Fields, matchingFieldPolicy(att, field, fieldName, schema)) walk(field.Attribute) } } @@ -149,6 +144,26 @@ func (u *Union) Matcher() string { return b.String() } +func matchingFieldPolicy(parent *expr.AttributeExpr, field *expr.NamedAttributeExpr, fieldName func(string) string, schema bool) expr.ValueFieldPolicy { + visible := true + if hidden, ok := field.Attribute.Meta.Last("openapi:generate"); schema && ok && hidden == "false" { + visible = false + } + var numeric expr.Kind + if expr.IsPrimitive(field.Attribute.Type) { + kind := field.Attribute.Type.Kind() + if kind >= expr.IntKind && kind <= expr.Float64Kind { + numeric = kind + } + } + return expr.ValueFieldPolicy{ + Parent: parent, Target: field.Attribute, Name: field.Name, + WireName: expr.JSONFieldName(fieldName(field.Name), field.Attribute), + Visible: visible, Required: parent.IsRequired(field.Name), + Presence: expr.ValueFieldRetain, NumericKind: numeric, + } +} + func (u *Union) candidates(b *strings.Builder) { for index, branch := range u.Branches { fmt.Fprintf(b, "var value%d %s\n", index, branch.Type) diff --git a/codegen/unionjson/union_test.go b/codegen/unionjson/union_test.go index b23d3addf..948bfe3e6 100644 --- a/codegen/unionjson/union_test.go +++ b/codegen/unionjson/union_test.go @@ -1,6 +1,7 @@ package unionjson import ( + "fmt" "testing" "github.com/stretchr/testify/require" @@ -31,15 +32,28 @@ func TestPlanOwnsRepresentationPolicy(t *testing.T) { {"hidden schema field", expr.ElementName, true, true, `{"count":0,"hidden":"x"}`, false}, {"runtime field", expr.ElementName, false, true, `{"count":0,"hidden":"x"}`, true}, } { - t.Run(tc.name, func(t *testing.T) { - plan, err := Plan(source, tc.fieldName, tc.schema, tc.closed) - require.NoError(t, err) - shape, err := plan.JSONShape() - require.NoError(t, err) - matched, err := shape.Match([]byte(tc.wire)) - require.NoError(t, err) - require.Equal(t, tc.want, matched) - }) + for _, container := range []string{"object", "map", "union"} { + target, wire := source, tc.wire + if container == "map" { + target = &expr.AttributeExpr{Type: &expr.Map{KeyType: &expr.AttributeExpr{Type: expr.String}, ElemType: source}} + wire = `{"entry":` + wire + `}` + } + if container == "union" { + target = &expr.AttributeExpr{Type: &expr.Union{TypeName: "Choice", Untagged: true, Values: []*expr.NamedAttributeExpr{ + {Name: "Page", Attribute: source}, + {Name: "Strings", Attribute: &expr.AttributeExpr{Type: &expr.UserTypeExpr{TypeName: "Strings", AttributeExpr: &expr.AttributeExpr{Type: &expr.Array{ElemType: &expr.AttributeExpr{Type: expr.String}}}}}}, + }}} + } + t.Run(fmt.Sprintf("%s/%s", tc.name, container), func(t *testing.T) { + plan, err := Plan(target, tc.fieldName, tc.schema, tc.closed) + require.NoError(t, err) + shape, err := plan.JSONShape() + require.NoError(t, err) + matched, err := shape.Match([]byte(wire)) + require.NoError(t, err) + require.Equal(t, tc.want, matched) + }) + } } require.Empty(t, source.Type.(expr.UserType).Attribute().Meta, "projection must not mutate the authored type") } diff --git a/codegen/validation.go b/codegen/validation.go index 600f77bc3..938b37bce 100644 --- a/codegen/validation.go +++ b/codegen/validation.go @@ -66,16 +66,18 @@ func generatedRequiredValidation(att *expr.AttributeExpr, attCtx *AttributeConte } func generatedRequiredValidationFrom(att *expr.AttributeExpr, validation *expr.ValidationExpr, attCtx *AttributeContext) (res []string) { - if validation == nil { + if validation == nil || len(validation.Required) == 0 { return } obj := expr.AsObject(att.Type) + fields := expr.NewObjectFields(att) for _, req := range validation.Required { reqAtt := obj.Attribute(req) if reqAtt == nil { continue } - if !attCtx.Pointer && expr.IsPrimitive(reqAtt.Type) && + if attCtx.FieldPresence(fields, req, reqAtt) == NativePresence && + !attCtx.IsPrimitivePointer(req, att) && expr.IsPrimitive(reqAtt.Type) && reqAtt.Type.Kind() != expr.BytesKind && !expr.IsAny(reqAtt.Type) { continue diff --git a/docs/admission.md b/docs/admission.md new file mode 100644 index 000000000..a3a16e83f --- /dev/null +++ b/docs/admission.md @@ -0,0 +1,139 @@ +# Unary concurrency admission + +`github.com/CaliLuke/loom/admission` bounds active unary endpoint invocations +within one process. A shared `Concurrency` instance supplies one budget across +the endpoints using it. Separate instances and separate replicas have independent +budgets. Applications choose capacity and which methods share it. + +```go +budget, err := admission.NewConcurrency(32) +if err != nil { + return err +} +endpoints := work.NewEndpoints(service, authorizer) +endpoints.Use(budget.Middleware) +``` + +Use `Endpoints.Use` only when all affected methods are unary and should share +the budget. For a mixed service, pass `budget.Middleware` to the generated +protected per-method constructor for each selected unary method. Do not wrap or +replace a protected endpoint field from outside its access boundary. + +Protected endpoints validate, authenticate and authorize before admission. +Access checks still run on saturated requests, and repeat before service +execution if admission succeeds. Admission preserves the context, payload, +deadline and cancellation. See [authorization ordering](authorization.md#invocation-boundary). +Unprotected methods retain their existing validation and middleware ordering; +the limiter does not add authorization or move validation to a new boundary. + +The budget protects work inside the wrapped endpoint. Protect transport decoding +and authentication with separately positioned transport controls. Request-rate +and burst quotas are separate from the count of active calls; see +[rate admission](rate-limiting.md) for early and authenticated quotas. + +## Saturation and completion + +When all permits are occupied, middleware returns immediately without invoking +the next endpoint. The returned `*loom.ServiceError` is named `overloaded`, has +`Temporary` set, and contains no request identity. Declare and map it explicitly: + +```go +Error("overloaded", func() { + Temporary() +}) +HTTP(func() { + Response("overloaded", StatusServiceUnavailable) +}) +GRPC(func() { + Response("overloaded", CodeUnavailable) +}) +``` + +HTTP 503 describes service saturation. Caller-quota exhaustion uses a separate +error and HTTP 429. Other transports use the mappings declared in the design; +the middleware does not select transport statuses. It adds no retry or +`Retry-After` header because it cannot predict when an active call will finish. + +An already canceled or expired context is rejected with `ctx.Err()` before +capacity is considered. Once admitted, a call retains its permit until the +wrapped endpoint exits, including service errors and panic unwinding. Panics +propagate to the caller's recovery boundary. Cancellation does **not** free +capacity while an invocation continues running. A cancellation racing admission +may occur after its context check; the endpoint still receives the original +context and remains responsible for honoring it. + +Unary endpoint completion is the lifetime boundary. Do not apply this helper to +streams, subscriptions, or background work that outlives endpoint return. The +helper creates no queue, background goroutine, distributed lease or adaptive +capacity policy. + +## Updating and observing capacity + +```go +if err := budget.SetCapacity(16); err != nil { + return err +} +snapshot := budget.Snapshot() +``` + +Capacity must be positive at construction and on every update. An invalid +update preserves the prior capacity and all accounting. Updates and admissions +are synchronized, so each decision sees one effective capacity. Reapplying a +capacity does not reset counters or permits. + +Reducing capacity leaves active invocations running. `InFlight` may temporarily +exceed the new capacity; no new call enters until it is below that limit. +Increasing capacity permits more admissions immediately. The framework does not +provide an operator endpoint, configuration store, authorization for updates or +coordination across replicas. + +`Snapshot` returns capacity, in-flight calls and cumulative admitted, saturated +(`Rejected`) and pre-admission canceled (`Canceled`) counts in one coherent read. +Export those fields through your metrics system with application-chosen bounded +labels. Requests rejected by earlier validation or access checks are outside +these counters. An admitted call that later fails or is canceled remains counted +as admitted. There are no caller identities or metrics-backend dependencies. + +## Runnable generated service + +The [design](examples/concurrency/design.go.txt) and +[application](examples/concurrency/main.go.txt) demonstrate a shared local budget +of two calls with API-key authentication, a protected unary method and explicit +HTTP/gRPC overload mappings. The application mounts HTTP on loopback. This guide +tracks `main`; select a Loom release or exact commit containing `admission`. +Use Go 1.27.3 and the normal gRPC generation prerequisites described in the +[gRPC guide](grpc-guide.md). + +In a new directory, save the design as `design/design.go`, then run: + +```sh +go mod init example.com/admission +go get -tool github.com/CaliLuke/loom/cmd/loom@"$LOOM_VERSION" +go tool loom gen example.com/admission/design +``` + +Set `LOOM_VERSION` to the selected release or commit before running the commands. +Save the application as `main.go` after generation, then run: + +```sh +go mod tidy +EXAMPLE_API_KEY=local-demo-key go run . +``` + +From another terminal, start two long calls, then send a third while both are +active: + +```sh +curl -s -H 'X-API-Key: local-demo-key' -H 'Content-Type: application/json' \ + -d '{"seconds":30}' http://127.0.0.1:8080/work & +curl -s -H 'X-API-Key: local-demo-key' -H 'Content-Type: application/json' \ + -d '{"seconds":30}' http://127.0.0.1:8080/work & +curl -i -H 'X-API-Key: local-demo-key' -H 'Content-Type: application/json' \ + -d '{"seconds":1}' http://127.0.0.1:8080/work +wait +``` + +The saturated request receives 503 without service execution. An invalid payload +still receives 400, and an invalid key receives 401, even while capacity is full. +Use application-owned authentication and authorization policies in deployed +services; this example grants work permission to its one configured demo key. diff --git a/docs/authorization.md b/docs/authorization.md index 8e2bd585e..76e31afa6 100644 --- a/docs/authorization.md +++ b/docs/authorization.md @@ -100,8 +100,35 @@ that scheme, in requirement order. An unscoped API key combined with a JWT that declares `read` receives no required scopes; the JWT receives `read`. A scope declared by both schemes is required by both. Scopes declared by none are rejected in the design. Separate `Security` calls remain independent alternatives. -Regenerate endpoints and OpenAPI after upgrading; callbacks can continue using -`scheme.Validate(grantedScopes)` without filtering another scheme's scopes. +Use `ScopeFor(oauth, "read")` inside `Security` when OAuth schemes have +independent requirements, including overlapping scope inventories. Use +`RoleFor(key, "admin")` for non-OAuth role names. Both calls add requirements only +to their selected scheme; they add to any shared `Scope` declarations. Empty +per-scheme lists require authentication without extra grants. A name required +by one scheme never becomes required by another merely because both declare it. + +Role names have no protocol-defined inventory and are not exchanged on the +wire. Basic, API-key and JWT callbacks receive these application permission +names in `RequiredScopes`; `scheme.Validate(grantedNames)` requires all of them. +The application obtains the granted names from its authenticated principal and +must call `Validate` or enforce equivalent checks before returning success. +OAuth callbacks continue receiving declared OAuth scopes in the same field. +ScopeFor on non-OAuth schemes, RoleFor on OAuth schemes, unknown selected +schemes and unknown OAuth scopes are rejected. + +OAuth flow constructors accept an optional final block of `Scope(name, description)` +declarations. A flow without a block inherits the scheme-level scopes. A supplied +block replaces those defaults; `func() {}` declares an empty inventory. Different +flows may use disjoint names or different descriptions for the same name. +Requirements may reference names available in any effective flow. The generated +OAuth callback receives each flow's inventory in `Flows`; `Scopes` lists the +available names across flows. Neither inventory grants access: the application +validates the token, its issuer and flow, and its actual granted scopes. + +Regenerate endpoints and OpenAPI after upgrading. OpenAPI now exports required +non-OAuth names as role arrays, including requirements authored with shared +`Scope`. Existing credential wire formats and the callbacks' validation rule +are unchanged. `StrictAuthorization()` at API or service scope requires every method to declare `Authorize`, `AuthorizeBy`, or `NoAccessCheck`. Existing designs remain unchanged diff --git a/docs/code-generation.md b/docs/code-generation.md index 96f8034ad..fd73695f5 100644 --- a/docs/code-generation.md +++ b/docs/code-generation.md @@ -459,8 +459,8 @@ own `deprecated` keyword. It never downgrades contract-affecting diagnostics such as servers, security, extensions, callbacks, links, custom serialization, media encodings, or unsupported schema composition and structural keywords. The documented -`allOf` flattening and inline array-item promotion are explicit lossy -exceptions. Without `--skip-unrenderable`, other diagnostics prevent output. +inline array-item promotion is an explicit lossy exception. Supported reference +and object intersections preserve their `allOf` contributors. Without `--skip-unrenderable`, other diagnostics prevent output. With that flag, the importer omits each affected operation. The document-level omissions use the separate behavior above. diff --git a/docs/dsl-reference.md b/docs/dsl-reference.md index 53680361f..5fb17a670 100644 --- a/docs/dsl-reference.md +++ b/docs/dsl-reference.md @@ -274,9 +274,16 @@ are validated. Regenerate affected clients and servers to pick up this correctio Selected-view validation remains separate from full union-branch matching. Branches must be concrete named objects or arrays. Their recursive contents may -be primitives, named objects, and arrays of these shapes. Inline object leaves, -maps, nested unions, opaque custom codecs, and schema encoding overrides are -rejected throughout the branch. Matching preserves exact JSON names, required +be primitives, named objects, untagged unions, arrays, and string-keyed maps of +these shapes. Every nested union keeps its own exact-one rule and concrete named +object/array branches. Empty arrays of union values remain valid unless constrained; +a row matching zero or multiple branches rejects its enclosing candidate. JSON +contains each nested value directly, without a synthetic discriminator. +Map values retain their declared constraints, including named types, nullable +values and recursive objects. Map length constraints count entries; an empty +map is valid unless a minimum length rejects it. Map keys must be strings or +named string types. Inline object leaves, tagged nested unions, opaque custom +codecs, and schema encoding overrides are rejected throughout the branch. Matching preserves exact JSON names, required membership before defaults, nullability, closed-object policy, and effective value constraints. A selected nil slice encodes as `[]`; a nil object branch is invalid. Multiple array branches that accept `[]` make that value ambiguous. @@ -286,8 +293,9 @@ OpenAPI renders branch component references directly under `oneOf` and omits the discriminator. This encoding is for JSON bodies; string-encoded parameters, headers, cookies, forms, and multipart bodies do not support it. Protobuf keeps its existing explicitly numbered `oneof` alternatives and collection wrappers. -Regenerate consumers to adopt these checks; previously emitted ambiguous objects -and nested-map branches are no longer accepted. +Regenerate consumers to adopt map fields and these checks. Ambiguous objects +remain invalid even when both branches contain maps. Map root branches remain +unsupported; place a dictionary inside a named object or array branch. #### Result View Requiredness @@ -1765,6 +1773,26 @@ semantics and chain the context returned by each successful authorizer. Only the context from the requirement that ultimately succeeds reaches the service method. +Within a requirement, `Scope("read")` applies to each scheme that declares that +name. Use `ScopeFor(oauth, "read")` to require it from only one OAuth scheme, +and `RoleFor(key, "admin")` to require an application role from one non-OAuth +scheme. These calls add to shared Scope declarations and accept a scheme name +or reference already listed in Security. Unknown OAuth scopes and references to +schemes outside that requirement are rejected; role names need no inventory. +Callbacks receive their own required names in `RequiredScopes` and validate +application-owned grants with `scheme.Validate(grantedNames)`. Roles are not +extra credentials exchanged on the wire. Regenerate transports and OpenAPI; +required non-OAuth names are now emitted as OpenAPI role arrays. + +OAuth flow constructors accept an optional final block of `Scope(name, description)` +declarations. A flow without a block inherits the scheme-level scopes. A supplied +block replaces those defaults; `func() {}` declares an empty inventory. Different +flows may use disjoint names or different descriptions for the same name. +Requirements may reference names available in any effective flow. The generated +OAuth callback receives each flow's inventory in `Flows`; `Scopes` lists the +available names across flows. Neither inventory grants access: the application +validates the token, its issuer and flow, and its actual granted scopes. + Each separate `Security(...)` call adds an OR alternative. `Security()` with no scheme adds an anonymous alternative, so callers may satisfy the endpoint without credentials while the other alternatives remain available. This maps diff --git a/docs/examples/concurrency/design.go.txt b/docs/examples/concurrency/design.go.txt new file mode 100644 index 000000000..379cb3821 --- /dev/null +++ b/docs/examples/concurrency/design.go.txt @@ -0,0 +1,50 @@ +package design + +import . "github.com/CaliLuke/loom/dsl" + +var _ = API("capacity", func() { + Title("Unary admission example") +}) + +var key = APIKeySecurity("key") +var run = Authorization("work.run", Empty) + +var _ = Service("work", func() { + StrictAuthorization() + Security(key) + HTTP(func() { + AuthErrorResponses() + }) + Method("run", func() { + Payload(func() { + APIKey("key", "key", String) + Field(1, "seconds", Int, func() { + Minimum(1) + Maximum(30) + }) + Required("key", "seconds") + }) + Authorize(run) + Result(String) + Error("overloaded", func() { + Temporary() + }) + Error("unauthorized") + Error("forbidden") + HTTP(func() { + POST("/work") + Header("key:X-API-Key") + Response(StatusOK) + Response("overloaded", StatusServiceUnavailable) + }) + GRPC(func() { + Metadata(func() { + Attribute("key:x-api-key") + }) + Response(CodeOK) + Response("overloaded", CodeUnavailable) + Response("unauthorized", CodeUnauthenticated) + Response("forbidden", CodePermissionDenied) + }) + }) +}) diff --git a/docs/examples/concurrency/main.go.txt b/docs/examples/concurrency/main.go.txt new file mode 100644 index 000000000..6eb7ed5ff --- /dev/null +++ b/docs/examples/concurrency/main.go.txt @@ -0,0 +1,71 @@ +package main + +import ( + "context" + "crypto/subtle" + "errors" + "log" + "net/http" + "os" + "time" + + workserver "example.com/admission/gen/http/work/server" + "example.com/admission/gen/work" + "github.com/CaliLuke/loom/admission" + loomhttp "github.com/CaliLuke/loom/http" + loom "github.com/CaliLuke/loom/pkg" + "github.com/CaliLuke/loom/security" +) + +type ( + workService struct { + key string + } + actorKey struct{} +) + +func main() { + key := os.Getenv("EXAMPLE_API_KEY") + if key == "" { + log.Fatal("set EXAMPLE_API_KEY before starting the example") + } + budget, err := admission.NewConcurrency(2) + if err != nil { + log.Fatal(err) + } + service := &workService{key: key} + endpoints := work.NewEndpoints(service, service) + endpoints.Use(budget.Middleware) + mux := loomhttp.NewMuxer() + workserver.Mount(mux, workserver.New(endpoints, mux, loomhttp.RequestDecoder, loomhttp.ResponseEncoder, nil, nil)) + server := &http.Server{Addr: "127.0.0.1:8080", Handler: mux, ReadHeaderTimeout: 5 * time.Second} + log.Print("listening on 127.0.0.1:8080; local unary capacity is 2") + if err := server.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) { + log.Fatal(err) + } +} + +func (*workService) Run(ctx context.Context, payload *work.RunPayload) (string, error) { + timer := time.NewTimer(time.Duration(payload.Seconds) * time.Second) + defer timer.Stop() + select { + case <-ctx.Done(): + return "", ctx.Err() + case <-timer.C: + return "finished", nil + } +} + +func (s *workService) APIKeyAuth(ctx context.Context, key string, _ *security.APIKeyScheme) (context.Context, error) { + if subtle.ConstantTimeCompare([]byte(key), []byte(s.key)) != 1 { + return nil, loom.PermanentError("unauthorized", "invalid API key") + } + return context.WithValue(ctx, actorKey{}, true), nil +} + +func (*workService) AuthorizeWorkRun(ctx context.Context) error { + if ctx.Value(actorKey{}) != true { + return loom.PermanentError("forbidden", "work permission required") + } + return nil +} diff --git a/docs/examples/rate/design.go.txt b/docs/examples/rate/design.go.txt new file mode 100644 index 000000000..223bf111d --- /dev/null +++ b/docs/examples/rate/design.go.txt @@ -0,0 +1,47 @@ +package design + +import . "github.com/CaliLuke/loom/dsl" + +var _ = API("quota", func() { + Title("Rate admission example") +}) + +var key = APIKeySecurity("key") +var run = Authorization("work.run", Empty) +var rateDenied = Type("RateDenied", func() { + Attribute("message", String) + Attribute("retry_after", Int64, "Whole seconds until allowance may be available; absent when unknown") + Required("message") +}) + +var _ = Service("work", func() { + StrictAuthorization() + Security(key) + HTTP(func() { + AuthErrorResponses() + }) + Method("run", func() { + Payload(func() { + APIKey("key", "key", String) + Attribute("message", String, func() { + MinLength(1) + }) + Required("key", "message") + }) + Authorize(run) + Result(String) + Error("rate_limited", rateDenied) + Error("rate_limit_unavailable", func() { + Temporary() + }) + HTTP(func() { + POST("/work") + Header("key:X-API-Key") + Response(StatusOK) + Response("rate_limited", StatusTooManyRequests, func() { + Header("retry_after:Retry-After") + }) + Response("rate_limit_unavailable", StatusServiceUnavailable) + }) + }) +}) diff --git a/docs/examples/rate/main.go.txt b/docs/examples/rate/main.go.txt new file mode 100644 index 000000000..bbf7bdc78 --- /dev/null +++ b/docs/examples/rate/main.go.txt @@ -0,0 +1,112 @@ +package main + +import ( + "context" + "crypto/subtle" + "errors" + "log" + "net/http" + "os" + "time" + + workserver "example.com/admission/gen/http/work/server" + "example.com/admission/gen/work" + "github.com/CaliLuke/loom/admission" + loomhttp "github.com/CaliLuke/loom/http" + "github.com/CaliLuke/loom/http/middleware" + loom "github.com/CaliLuke/loom/pkg" + "github.com/CaliLuke/loom/security" +) + +type ( + workService struct { + secret string + } + groupKey struct{} +) + +func main() { + secret := os.Getenv("EXAMPLE_API_KEY") + if secret == "" { + log.Fatal("set EXAMPLE_API_KEY before starting the example") + } + global, err := admission.NewRate(admission.RatePolicy{PerSecond: 10, Burst: 20}) + if err != nil { + log.Fatal(err) + } + groups := make(map[string]*admission.Rate, 2) + for _, name := range []string{"alpha", "beta"} { + groups[name], err = admission.NewRate(admission.RatePolicy{PerSecond: 1, Burst: 2}) + if err != nil { + log.Fatal(err) + } + } + early, err := admission.NewRateAdmission(func(ctx context.Context, _ string) (admission.RateDecision, error) { + return global.Allow(ctx) + }, time.Second) + if err != nil { + log.Fatal(err) + } + quota, err := admission.NewRateAdmission(func(ctx context.Context, key string) (admission.RateDecision, error) { + bucket, ok := groups[key] + if !ok { + return admission.RateDecision{}, errors.New("unknown quota group") + } + return bucket.Allow(ctx) + }, time.Second) + if err != nil { + log.Fatal(err) + } + service := &workService{secret: secret} + endpoints := work.NewEndpoints(service, service) + endpoints.Use(quota.Middleware(authenticatedGroup, quotaError)) + mux := loomhttp.NewMuxer() + workserver.Mount(mux, workserver.New(endpoints, mux, loomhttp.RequestDecoder, loomhttp.ResponseEncoder, nil, nil)) + handler := middleware.RateLimit(early, func(*http.Request) (string, error) { + return "process", nil + })(mux) + server := &http.Server{Addr: "127.0.0.1:8080", Handler: handler, ReadHeaderTimeout: 5 * time.Second} + log.Print("listening on 127.0.0.1:8080 with early and authenticated local quotas") + if err := server.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) { + log.Fatal(err) + } +} + +// Run echoes an admitted request after the generated validation and access checks. +func (*workService) Run(_ context.Context, payload *work.RunPayload) (string, error) { + return payload.Message, nil +} + +// APIKeyAuth maps the example's two configured keys to a bounded trusted group. +func (s *workService) APIKeyAuth(ctx context.Context, key string, _ *security.APIKeyScheme) (context.Context, error) { + for _, name := range []string{"alpha", "beta"} { + if subtle.ConstantTimeCompare([]byte(key), []byte(s.secret+"-"+name)) == 1 { + return context.WithValue(ctx, groupKey{}, name), nil + } + } + return nil, loom.PermanentError("unauthorized", "invalid API key") +} + +// AuthorizeWorkRun grants the demo's work permission to authenticated groups. +func (*workService) AuthorizeWorkRun(ctx context.Context) error { + if ctx.Value(groupKey{}) == nil { + return loom.PermanentError("forbidden", "work permission required") + } + return nil +} + +func authenticatedGroup(ctx context.Context, _ any) (string, error) { + group, ok := ctx.Value(groupKey{}).(string) + if !ok || group == "" { + return "", errors.New("authenticated quota group missing") + } + return group, nil +} + +func quotaError(decision admission.RateDecision) error { + err := &work.RateDenied{Message: "authenticated quota exhausted"} + if seconds := decision.RetryAfterSeconds(); seconds > 0 { + err.RetryAfter = &seconds + } + return err +} diff --git a/docs/openapi-import-coverage.md b/docs/openapi-import-coverage.md index 24998fb3b..79a9561d9 100644 --- a/docs/openapi-import-coverage.md +++ b/docs/openapi-import-coverage.md @@ -86,12 +86,12 @@ reads. Specification extensions are patterns, not fixed fields. | Operation | `tags`, `summary`, `description`, `operationId`, `parameters`, `requestBody`, `responses`, `deprecated`, `security`, supported `x-*` | `externalDocs` | `callbacks`, `servers` | | Components | Supported schemas, parameters, request bodies, responses, headers, security schemes, and examples | None | Links, callbacks, path items, media types, and `x-*` component entries | | Parameter | Identity, location, requiredness, schema, description, supported style, equivalent `explode` defaults, `allowReserved`, named query-map JSON `content`, and supported `x-*` | `deprecated`, unsupported examples | Incompatible serialization or content | -| Request Body | `description`, `required`, supported content, and supported `x-*` | None | direct `$ref` plans that cannot retain component identity | +| Request Body | `description`, `required`, supported content, and supported `x-*` | None | unresolved, cyclic, or wrong-kind `$ref` plans | | Media Type | Supported schema, examples, and equivalent URL-encoded field encoding | OpenAPI 3.2 `description` | `itemSchema`, unsupported encoding fields, `prefixEncoding`, and `x-*` | | Responses | Concrete status-code entries | None | `default` and `x-*` | -| Response | `summary`, `description`, supported headers and content, and supported `x-*` | None | Direct `$ref` plans and links | -| Header | `description`, `required`, schema, `style: simple`, and `allowReserved` | `deprecated`, unsupported examples | Other serialization, content, direct `$ref`, and `x-*` | -| Security Scheme | API keys, HTTP basic, HTTP bearer, and supported OAuth 2.0 flows and scopes | None | OpenID Connect, bearer formats, device authorization, and incompatible OAuth scope maps | +| Response | `summary`, `description`, supported headers and content, and supported `x-*` | None | Unresolved, cyclic, or wrong-kind `$ref` plans and links | +| Header | `description`, `required`, schema, `style: simple`, and `allowReserved` | `deprecated`, unsupported examples | Other serialization, content, unresolved/cyclic/wrong-kind `$ref`, and `x-*` | +| Security Scheme | API keys, HTTP basic, HTTP bearer, and supported OAuth 2.0 flows and scopes | None | OpenID Connect, bearer formats, device authorization | | Example | JSON-compatible inline, structured, and reusable component examples | Unrenderable examples | External values and incompatible value forms | @@ -106,16 +106,24 @@ in application code. Duplicate credentials no longer become unrelated request body fields. Each security requirement object is a conjunction; separate objects remain -alternatives. Loom applies one scope list to every scheme in a conjunction, so -import rejects different per-scheme lists rather than changing authentication. -For example, `[{Bearer: []}, {OAuth: [read]}]` is supported, while -`[{Bearer: [], OAuth: [read]}]` is not. Two OAuth schemes with the same required -scopes and conjunctions with empty lists remain supported. This restriction also -applies with `--allow-lossy`. Partial import skips operations that inherit an -unsupported root requirement while retaining public or secured operations that -explicitly override it. The retained design includes root security only when a -retained operation inherits it; strict whole-document import still rejects the -unsupported root declaration. +alternatives. Import preserves each scheme's independent array, including empty +arrays and overlapping OAuth scope inventories. For example, +`[{Bearer: [admin], OAuth: [read]}]` requires both the application's `admin` +role and the OAuth `read` scope. Import renders `ScopeFor` for OAuth scopes and +`RoleFor` for non-OAuth role names. Roles have no protocol-defined inventory; +unknown OAuth scopes still fail closed. Non-OAuth role arrays require OpenAPI +3.1 or 3.2; OpenAPI 3.0 requires those arrays to be empty. Anonymous alternatives and explicit +operation overrides remain distinct. Partial import skips operations that +inherit an invalid root requirement while retaining public or secured overrides. +The retained design includes root security only when a retained operation +inherits it. `security_per_scheme_test.go` and the generated credential fixture +cover per-scheme fidelity and runtime enforcement. + +OAuth flows retain their own scope names and descriptions, including disjoint, +overlapping and empty inventories. Import emits a flow-local Scope block for +each flow. Regeneration preserves the exact flow maps; requirement validation +checks names across the effective inventories without assigning token grants. +`security_flow_scopes_test.go` covers the imported and regenerated contract. Path and operation extensions use a documented allowlist. Unknown extensions at those supported scopes produce diagnostics. Document, schema, parameter, @@ -155,6 +163,17 @@ component description for that occurrence, including an explicitly empty string. Other occurrences retain the component description. Regeneration may use separate parameter components to preserve these effective descriptions. +Reusable request bodies, responses, and headers retain their component names and +forward alias references. In OpenAPI 3.1 and 3.2, Reference Object descriptions +and summaries remain occurrence annotations, including explicitly empty strings; +shared definitions are unchanged. Summaries have an effect only on target object +kinds that support them. OpenAPI 3.0 ignores Reference Object siblings. + +Referenced definitions must fit the same media, schema, and header subset as +inline objects. Unused definitions, cyclic aliases, unresolved references, and +references to the wrong component kind remain blocking errors. Regenerate an +imported design to obtain preserved component identities and annotation edges. + An extension at an unsupported scope is a `vendor-extension` diagnostic. Strict import fails before it writes output. With `--allow-lossy`, the importer omits the extension and reports it as a warning. @@ -258,7 +277,8 @@ The following constructs are conditional: - A one-member `allOf` with a local reference is lossless. Numeric bounds and compatible scalar defaults are preserved on the reference occurrence; other contract siblings remain rejected. -- A `$ref` plus inline-object `allOf` requires `--allow-lossy`. +- A local `$ref` plus an inline object in `allOf` preserves the reference and + the object intersection. It does not require `--allow-lossy`. - An inline object used as array items requires `--allow-lossy`. Loom promotes it to a deterministic component. - A two-member `anyOf` with a bare `{type: null}` branch maps to `Nullable()`. @@ -321,8 +341,24 @@ local definitions. Structured `const` values and schemas that combine `const` with `enum` are also rejected. -A Schema Object with `$ref` siblings is rejected. Use a supported `allOf` -shape when the sibling constraint must remain part of the contract. +OpenAPI 3.1 and 3.2 Schema Objects can place supported scalar assertions, +array length bounds, required property names and annotations beside a local +`$ref`. Loom retains the reference and occurrence constraints independently: +`{$ref: '#/components/schemas/Name', minLength: 2}` enforces both the Name +contract and the extra minimum, without changing other uses of Name. Required +names must exist in the referenced object. Regenerate imported designs to pick +up corrected validation of optional fields required by a named occurrence. + +Object `properties` beside a reference refine the inherited fields and can add +fields to an open object. Omitted property types use the inherited carrier. +Regeneration retains the base reference in `allOf` alongside the composed +object. A closed base cannot gain fields, and a closed sibling must declare all +inherited property names; these checks prevent changing the accepted key set. + +Reference-level `items`, typed `additionalProperties`, arbitrary composition, +type changes, removal of inherited nullability and changes to numeric carrier +formats remain blocked. `--allow-lossy` cannot bypass these boundaries. OpenAPI +3.0 direct reference siblings remain unsupported. OpenAPI 3.1 and 3.2 permit custom JSON Schema vocabularies. Loom rejects every unknown non-`x-*` schema keyword because it cannot prove that the keyword is diff --git a/docs/production.md b/docs/production.md index cee2e486f..84363b8e6 100644 --- a/docs/production.md +++ b/docs/production.md @@ -431,7 +431,8 @@ var _ = Service("users", func() { 1. **Always use HTTPS in production** 2. **Define security at API level** for consistent defaults 3. **Use `NoSecurity()` explicitly** for public endpoints -4. **Implement rate limiting** for API key authentication +4. **Implement [rate admission](rate-limiting.md)** before expensive work and + authenticated quotas after identity is trusted 5. **Use appropriate token expiration** for JWT tokens 6. **Regularly rotate secrets and keys** 7. **Log and monitor authentication failures** @@ -441,6 +442,15 @@ var _ = Service("users", func() { ## Common Patterns +### Bound active unary calls + +Use `admission.NewConcurrency` with generated endpoint middleware to bound active +unary invocations in one process. Capacity can change without rebuilding the +middleware; cancellation retains a permit until the invocation exits. Declare +`overloaded` and map service saturation explicitly, for example to HTTP 503. +See [concurrency admission](admission.md) for placement, observation and a runnable +generated service. Request-rate quotas and streaming lifetimes are separate. + ### Graceful Shutdown ```go diff --git a/docs/rate-limiting.md b/docs/rate-limiting.md new file mode 100644 index 000000000..2d23ca51a --- /dev/null +++ b/docs/rate-limiting.md @@ -0,0 +1,149 @@ +# Rate and burst admission + +Rate admission counts attempts over time. It is separate from +[concurrency admission](admission.md), which bounds active unary calls. +Loom uses existing HTTP and protected endpoint middleware for both controls. +Applications choose budgets, trusted grouping keys, storage and quota scope. + +## Local budgets and live policy + +`admission.Rate` wraps `golang.org/x/time/rate` with synchronized policy updates. +One instance is one process-local token bucket. It starts full; each allowed +attempt consumes one token. It does not queue, reserve future tokens or refund +tokens after a later service failure, access recheck failure or cancellation. + +```go +budget, err := admission.NewRate(admission.RatePolicy{PerSecond: 5, Burst: 10}) +if err != nil { + return err +} +decision, err := budget.Allow(ctx) +``` + +`Allow` reports `Allowed`, an advisory `RetryAfter`, and the policy `Version` used +for that decision. Cancellation observed before admission consumes nothing. +A concurrent cancellation can happen after that decision; the request still +retains its original context and spent quota. + +Use `SetPolicy` to update rate and burst together without rebuilding middleware: + +```go +if err := budget.SetPolicy(admission.RatePolicy{PerSecond: 2, Burst: 4}); err != nil { + return err +} +policy, version := budget.Policy() +``` + +Both settings must be positive and the rate must be finite and below +`math.MaxFloat64`, which is reserved for unlimited admission. Invalid updates +preserve the previous settings, version and accounting. Each decision and policy +read sees a coherent version. Reapplying unchanged settings does not reset quota +or advance the version. + +An update first accrues tokens under the **old** rate up to the update time, capped +by the old burst. It retains that allowance, capped by the **new** burst, then +uses the new rate for future refill. Increasing burst adds room for future refill; +it does not immediately replenish allowance. For example, an exhausted 2-token +bucket raised to a burst of 10 remains exhausted until refill. Decreasing burst +from 10 to 2 with 6 tokens available leaves 2 tokens available. This is a local +atomic update, not a fleet-wide policy activation protocol. + +## Backend admission boundary + +`admission.NewRateAdmission(check, timeout)` calls an application-supplied +`RateCheck` with a grouping key and a bounded child context. A check can select +one of several local `Rate` budgets or use an application-owned shared backend. +The backend owns atomic accounting and policy changes for shared quotas. + +The check must be safe for concurrent use and return promptly on cancellation. +Configure its network/storage operations to honor the supplied deadline. Loom +does not launch a goroutine to abandon an uncooperative backend. The configured +timeout bounds cooperative checks; it cannot forcibly terminate arbitrary Go +code. No requests proceed after backend errors, timeout, malformed decisions or +late successful answers observed after the deadline. Caller cancellation returns +the caller's context error. Other failures become the safe named error +`rate_limit_unavailable`; original backend text is not exposed. + +The child context is used only for the check. An admitted service receives the +original request context, preserving values, cancellation and its own deadline. +There is no automatic retry. + +## Placement and error mappings + +Use `http/middleware.RateLimit(control, key)` outside the generated HTTP handler +for an early limit protecting decoding and authentication. Quota denial returns +HTTP 429. A known delay produces a positive, rounded-up whole-second `Retry-After`; +an unknown delay omits the header. Backend or grouping failure returns HTTP 503 +with no backend details. + +Use `control.Middleware(key, denied)` through generated `Endpoints.Use` or a +protected unary per-method constructor for an authenticated quota. Generated +validation and access checks run before the quota, including on rejected calls, +and repeat before service execution. Grouping must read identity established by +authentication. A grouping failure or nil denial error fails closed. The denial +callback returns the application's declared error, allowing a typed response +with the known delay mapped to a `Retry-After` header. Declare +`rate_limit_unavailable` too and map it explicitly, such as HTTP 503. Other +transports retain their authored error mappings. + +An early quota may be spent on malformed or unauthorized requests because it +protects that work. An authenticated quota is not spent when earlier protected +validation/access fails. A later access recheck failure does not refund an +already consumed attempt. Place observation at these boundaries if you need +separate counters; use bounded labels, never credentials or user identifiers. + +## Grouping and storage ownership + +Local budgets are independent across replicas. Applications that need a shared +quota must supply a shared backend; a local version does not imply fleet-wide +consistency. The framework provides no tenant policy store, operator endpoint, +audit history, distribution protocol or streaming/message quota. + +`RateAdmission` stores no grouping keys. The runnable example uses an immutable +map of two known groups, so key storage is bounded. Production applications must +bound their own group registry or backend. Unknown groups should be rejected; +do not turn backend failures or capacity pressure into permission to proceed. +Evicting a bucket and recreating it full can reset spent quota, so an eviction +policy must preserve accounting. Never create unlimited buckets from arbitrary +request input. + +For an early network-based key, `RemoteAddr` identifies the connected peer. Behind +a proxy that may be the proxy itself. Only use forwarded IP headers under an +explicit trusted proxy policy that strips untrusted input; Loom does not infer +that trust. Authenticated quotas should use a trusted stable group from context, +not an arbitrary tenant header, raw credential or request body field. + +## Runnable generated service + +Save the [design](examples/rate/design.go.txt) as `design/design.go` in a new +directory. With Go 1.27.3 and `LOOM_VERSION` set to a release or exact commit +containing rate admission: + +```sh +go mod init example.com/admission +go get -tool github.com/CaliLuke/loom/cmd/loom@"$LOOM_VERSION" +go tool loom gen example.com/admission/design +``` + +Save the [application](examples/rate/main.go.txt) as `main.go` after generation: + +```sh +go mod tidy +EXAMPLE_API_KEY=local-demo-key go run . +``` + +The loopback HTTP server has a shared early budget of 10 requests/second with a +burst of 20. Its two authenticated groups each have 1 request/second and a burst +of 2. Send three quick requests using `local-demo-key-alpha` to observe a 429; +`local-demo-key-beta` has separate allowance: + +```sh +for attempt in 1 2 3; do + curl -i -H 'X-API-Key: local-demo-key-alpha' -H 'Content-Type: application/json' \ + -d '{"message":"hello"}' http://127.0.0.1:8080/work +done +``` + +The demo grants work permission to either configured key. Replace that demo +policy with application authentication and authorization. Policy persistence, +operator authorization and activation across replicas remain application-owned. diff --git a/dsl/oneof_test.go b/dsl/oneof_test.go index 5b7b5302e..20db39a73 100644 --- a/dsl/oneof_test.go +++ b/dsl/oneof_test.go @@ -227,7 +227,7 @@ func TestUntaggedRejectsNestedBranchFields(t *testing.T) { }) require.Error(t, err) - require.Contains(t, err.Error(), `untagged OneOf branch "NestedUntagged" field "child" must be primitive, a concrete named object type, or an array of either`) + require.Contains(t, err.Error(), `untagged OneOf branch "NestedUntagged" field "child" must be primitive, a concrete named object type, an untagged union, or arrays or string-keyed maps of these`) } func TestUntaggedAcceptsNamedObjectBranchFieldsAndArrays(t *testing.T) { diff --git a/dsl/security.go b/dsl/security.go index f9b16169b..9313001cd 100644 --- a/dsl/security.go +++ b/dsl/security.go @@ -236,7 +236,9 @@ func CookieName(name string) { // The requirement refers to one or more OAuth2Security, BasicAuthSecurity, // APIKeySecurity or JWTSecurity security scheme. Required scopes may be listed by // name in the Security DSL. Each scheme receives the required scopes that it -// declares; scopes declared by no scheme are rejected. All the listed schemes +// declares; scopes declared by no scheme are rejected. ScopeFor binds OAuth2 +// scopes to one scheme; RoleFor binds application roles to one non-OAuth scheme. +// These explicit requirements add to the shared Scope requirements. All listed schemes // must be validated for the request to be authorized. Security may appear multiple times // in the same scope in which case the client may validate any one of the // requirements for the request to be authorized. diff --git a/dsl/security_flow_scopes_test.go b/dsl/security_flow_scopes_test.go new file mode 100644 index 000000000..2425c8bfa --- /dev/null +++ b/dsl/security_flow_scopes_test.go @@ -0,0 +1,104 @@ +package dsl_test + +import ( + "testing" + + "github.com/stretchr/testify/require" + + . "github.com/CaliLuke/loom/dsl" + "github.com/CaliLuke/loom/expr" +) + +func TestOAuthFlowScopeDSL(t *testing.T) { + root := expr.RunDSL(t, func() { + auth := OAuth2Security("auth", func() { + AuthorizationCodeFlow("https://example.com/auth", "https://example.com/token", "", func() { + Scope("read", "User consent") + }) + ImplicitFlow("https://example.com/auth", "", func() {}) + PasswordFlow("https://example.com/token", "") + ClientCredentialsFlow("https://example.com/token", "", func() { + Scope("write", "Machine grant") + }) + DeviceAuthorizationFlow("https://example.com/device", "https://example.com/token", "", func() { + Scope("device", "Device grant") + }) + Scope("default", "Shared default declared after flows") + }) + Service("flows", func() { + Method("read", func() { + Security(auth, func() { + ScopeFor(auth, "read", "write", "device", "default") + }) + Payload(func() { + AccessToken("token") + }) + }) + }) + }) + scheme := root.Schemes[0] + require.Len(t, scheme.Flows, 5) + for i, want := range [][]string{{"read"}, {}, {"default"}, {"write"}, {"device"}} { + scopes := scheme.FlowScopes(scheme.Flows[i]) + names := make([]string, 0, len(scopes)) + for _, scope := range scopes { + names = append(names, scope.Name) + } + require.ElementsMatch(t, want, names) + } +} + +func TestOAuthFlowScopeBlockErrors(t *testing.T) { + for _, tc := range []struct { + name string + body func() + want string + }{ + {"multiple blocks", func() { + ClientCredentialsFlow("https://example.com/token", "", func() {}, func() {}) + }, "too many"}, + {"nil block", func() { + ClientCredentialsFlow("https://example.com/token", "", nil) + }, "must not be nil"}, + {"invalid nested DSL", func() { + ClientCredentialsFlow("https://example.com/token", "", func() { + RoleFor("auth", "admin") + }) + }, "invalid use"}, + } { + t.Run(tc.name, func(t *testing.T) { + err := expr.RunInvalidDSL(t, func() { + OAuth2Security("auth", tc.body) + }) + require.ErrorContains(t, err, tc.want) + }) + } +} + +func TestOAuthFlowScopeDefaultsCannotGrantUnavailableNames(t *testing.T) { + for _, explicit := range []bool{false, true} { + t.Run(map[bool]string{false: "shared requirement", true: "scheme requirement"}[explicit], func(t *testing.T) { + err := expr.RunInvalidDSL(t, func() { + auth := OAuth2Security("auth", func() { + Scope("default") + ClientCredentialsFlow("https://example.com/token", "", func() {}) + }) + Service("flows", func() { + Method("read", func() { + Security(auth, func() { + if explicit { + ScopeFor(auth, "default") + } else { + Scope("default") + } + }) + Payload(func() { + AccessToken("token") + }) + }) + }) + }) + require.ErrorContains(t, err, "default") + }) + } +} diff --git a/dsl/security_oauth_flows.go b/dsl/security_oauth_flows.go index fda532e85..dc232b33b 100644 --- a/dsl/security_oauth_flows.go +++ b/dsl/security_oauth_flows.go @@ -5,13 +5,14 @@ import ( "github.com/CaliLuke/loom/expr" ) -// Scope has two uses: in a security scheme it defines a scope supported by -// that scheme. In Security it lists required scopes. +// Scope defines a supported scope in a security scheme, or a default for +// OAuth flows. In an OAuth flow block it defines +// that flow's complete inventory. In Security it lists required scopes. // -// Scope must appear in Security, BasicAuthSecurity, APIKeySecurity, JWTSecurity or OAuth2Security. +// Scope must appear in Security, a security scheme, or an OAuth flow block. // // Scope accepts one or two arguments: the first argument is the scope name and -// when used in a scheme the optional second argument is a description. +// when used in a scheme or flow the optional second argument is a description. // // Example: // @@ -26,24 +27,25 @@ import ( // }) // }) func Scope(name string, desc ...string) { + if len(desc) > 1 { + eval.TooManyArgError() + return + } + description := "no description" + if len(desc) == 1 { + description = desc[0] + } switch current := eval.Current().(type) { case *expr.SecurityExpr: - if len(desc) >= 1 { + if len(desc) > 0 { eval.TooManyArgError() return } current.Scopes = append(current.Scopes, name) case *expr.SchemeExpr: - if len(desc) > 1 { - eval.TooManyArgError() - return - } - d := "no description" - if len(desc) == 1 { - d = desc[0] - } - current.Scopes = append(current.Scopes, - &expr.ScopeExpr{Name: name, Description: d}) + current.Scopes = append(current.Scopes, &expr.ScopeExpr{Name: name, Description: description}) + case *expr.FlowExpr: + current.Scopes = append(current.Scopes, &expr.ScopeExpr{Name: name, Description: description}) default: eval.IncompatibleDSL() } @@ -55,23 +57,15 @@ func Scope(name string, desc ...string) { // AuthorizationCodeFlow must be used in OAuth2Security. // // AuthorizationCodeFlow accepts three arguments: the authorization, token and -// refresh URLs. -func AuthorizationCodeFlow(authorizationURL, tokenURL, refreshURL string) { - current, ok := eval.Current().(*expr.SchemeExpr) - if !ok { - eval.IncompatibleDSL() - return - } - if current.Kind != expr.OAuth2Kind { - eval.ReportError("cannot specify flow for non-oauth2 security scheme.") - return - } - current.Flows = append(current.Flows, &expr.FlowExpr{ +// refresh URLs, followed by an optional flow-local Scope block. A supplied +// block replaces scheme-level scope defaults, including when empty. +func AuthorizationCodeFlow(authorizationURL, tokenURL, refreshURL string, fn ...func()) { + oauthFlow(&expr.FlowExpr{ Kind: expr.AuthorizationCodeFlowKind, AuthorizationURL: authorizationURL, TokenURL: tokenURL, RefreshURL: refreshURL, - }) + }, fn) } // ImplicitFlow defines an implicit OAuth2 flow as described in section 1.3.2 @@ -79,22 +73,14 @@ func AuthorizationCodeFlow(authorizationURL, tokenURL, refreshURL string) { // // ImplicitFlow must be used in OAuth2Security. // -// ImplicitFlow accepts two arguments: the authorization and refresh URLs. -func ImplicitFlow(authorizationURL, refreshURL string) { - current, ok := eval.Current().(*expr.SchemeExpr) - if !ok { - eval.IncompatibleDSL() - return - } - if current.Kind != expr.OAuth2Kind { - eval.ReportError("cannot specify flow for non-oauth2 security scheme.") - return - } - current.Flows = append(current.Flows, &expr.FlowExpr{ +// ImplicitFlow accepts the authorization and refresh URLs and an optional +// flow-local Scope block. A supplied block replaces scheme-level defaults. +func ImplicitFlow(authorizationURL, refreshURL string, fn ...func()) { + oauthFlow(&expr.FlowExpr{ Kind: expr.ImplicitFlowKind, AuthorizationURL: authorizationURL, RefreshURL: refreshURL, - }) + }, fn) } // PasswordFlow defines an Resource Owner Password Credentials OAuth2 flow as @@ -102,22 +88,14 @@ func ImplicitFlow(authorizationURL, refreshURL string) { // // PasswordFlow must be used in OAuth2Security. // -// PasswordFlow accepts two arguments: the token and refresh URLs. -func PasswordFlow(tokenURL, refreshURL string) { - current, ok := eval.Current().(*expr.SchemeExpr) - if !ok { - eval.IncompatibleDSL() - return - } - if current.Kind != expr.OAuth2Kind { - eval.ReportError("cannot specify flow for non-oauth2 security scheme.") - return - } - current.Flows = append(current.Flows, &expr.FlowExpr{ +// PasswordFlow accepts the token and refresh URLs and an optional flow-local +// Scope block. A supplied block replaces scheme-level defaults. +func PasswordFlow(tokenURL, refreshURL string, fn ...func()) { + oauthFlow(&expr.FlowExpr{ Kind: expr.PasswordFlowKind, TokenURL: tokenURL, RefreshURL: refreshURL, - }) + }, fn) } // ClientCredentialsFlow defines an clientCredentials OAuth2 flow as described @@ -125,27 +103,29 @@ func PasswordFlow(tokenURL, refreshURL string) { // // ClientCredentialsFlow must be used in OAuth2Security. // -// ClientCredentialsFlow accepts two arguments: the token and refresh URLs. -func ClientCredentialsFlow(tokenURL, refreshURL string) { - current, ok := eval.Current().(*expr.SchemeExpr) - if !ok { - eval.IncompatibleDSL() - return - } - if current.Kind != expr.OAuth2Kind { - eval.ReportError("cannot specify flow for non-oauth2 security scheme.") - return - } - current.Flows = append(current.Flows, &expr.FlowExpr{ +// ClientCredentialsFlow accepts the token and refresh URLs and an optional +// flow-local Scope block. A supplied block replaces scheme-level defaults. +func ClientCredentialsFlow(tokenURL, refreshURL string, fn ...func()) { + oauthFlow(&expr.FlowExpr{ Kind: expr.ClientCredentialsFlowKind, TokenURL: tokenURL, RefreshURL: refreshURL, - }) + }, fn) } // DeviceAuthorizationFlow defines an OAuth 2 device authorization flow as -// described by RFC 8628. It must be used in OAuth2Security. -func DeviceAuthorizationFlow(deviceAuthorizationURL, tokenURL, refreshURL string) { +// described by RFC 8628. It must be used in OAuth2Security. An optional Scope +// block supplies the complete flow inventory instead of scheme-level defaults. +func DeviceAuthorizationFlow(deviceAuthorizationURL, tokenURL, refreshURL string, fn ...func()) { + oauthFlow(&expr.FlowExpr{ + Kind: expr.DeviceAuthorizationFlowKind, + DeviceAuthorizationURL: deviceAuthorizationURL, + TokenURL: tokenURL, + RefreshURL: refreshURL, + }, fn) +} + +func oauthFlow(flow *expr.FlowExpr, fn []func()) { current, ok := eval.Current().(*expr.SchemeExpr) if !ok { eval.IncompatibleDSL() @@ -155,10 +135,18 @@ func DeviceAuthorizationFlow(deviceAuthorizationURL, tokenURL, refreshURL string eval.ReportError("cannot specify flow for non-oauth2 security scheme.") return } - current.Flows = append(current.Flows, &expr.FlowExpr{ - Kind: expr.DeviceAuthorizationFlowKind, - DeviceAuthorizationURL: deviceAuthorizationURL, - TokenURL: tokenURL, - RefreshURL: refreshURL, - }) + if len(fn) > 1 { + eval.TooManyArgError() + return + } + if len(fn) == 1 { + flow.Scopes = []*expr.ScopeExpr{} + if fn[0] == nil || !eval.Execute(fn[0], flow) { + if fn[0] == nil { + eval.ReportError("OAuth flow scope block must not be nil") + } + return + } + } + current.Flows = append(current.Flows, flow) } diff --git a/dsl/security_requirements.go b/dsl/security_requirements.go new file mode 100644 index 000000000..bc6c763f9 --- /dev/null +++ b/dsl/security_requirements.go @@ -0,0 +1,58 @@ +package dsl + +import ( + "github.com/CaliLuke/loom/eval" + "github.com/CaliLuke/loom/expr" +) + +// ScopeFor requires OAuth2 scopes from one scheme in the enclosing Security +// requirement. The scheme is a name or a scheme reference already listed in +// Security. Each scope must be declared by that scheme. Repeated calls add +// requirements; they do not replace Scope or another scheme's requirements. +func ScopeFor(scheme any, scopes ...string) { + requireSchemeNames(scheme, scopes, true) +} + +// RoleFor requires application role names from one non-OAuth scheme in the +// enclosing Security requirement. The scheme is a name or a scheme reference +// already listed in Security. Roles need no scheme-level inventory and are not +// exchanged on the wire. The application's authentication callback receives +// them in RequiredScopes and validates them against its own role assignments. +func RoleFor(scheme any, roles ...string) { + requireSchemeNames(scheme, roles, false) +} + +func requireSchemeNames(scheme any, names []string, oauth bool) { + current, ok := eval.Current().(*expr.SecurityExpr) + if !ok { + eval.IncompatibleDSL() + return + } + var name string + switch value := scheme.(type) { + case string: + name = value + case *expr.SchemeExpr: + if value != nil { + name = value.SchemeName + } + default: + eval.InvalidArgError("security scheme or security scheme name", scheme) + return + } + for _, selected := range current.Schemes { + if selected.SchemeName != name { + continue + } + if (selected.Kind == expr.OAuth2Kind) != oauth || selected.Kind == expr.NoKind { + eval.ReportError("ScopeFor requires an OAuth2 scheme; RoleFor requires a non-OAuth scheme") + return + } + if current.SchemeRequirements == nil { + current.SchemeRequirements = make(map[string][]string) + } + current.SchemeRequirements[name] = append(current.SchemeRequirements[name], names...) + return + } + eval.ReportError("security scheme %q is not in this requirement", name) +} diff --git a/dsl/security_requirements_test.go b/dsl/security_requirements_test.go new file mode 100644 index 000000000..74e6fe488 --- /dev/null +++ b/dsl/security_requirements_test.go @@ -0,0 +1,63 @@ +package dsl_test + +import ( + "testing" + + "github.com/stretchr/testify/require" + + . "github.com/CaliLuke/loom/dsl" + "github.com/CaliLuke/loom/expr" +) + +func TestPerSchemeRequirementsDSL(t *testing.T) { + design := func(requirements func()) func() { + return func() { + OAuth2Security("oauth", func() { + ClientCredentialsFlow("https://example.com/token", "") + Scope("read") + }) + APIKeySecurity("key") + Service("secure", func() { + Security("oauth", "key", requirements) + Method("read", func() { + Payload(func() { + AccessToken("token") + APIKey("key", "key") + }) + }) + }) + } + } + root := expr.RunDSL(t, design(func() { + ScopeFor("oauth", "read") + RoleFor("key", "admin") + })) + req := root.Services[0].Methods[0].EffectiveRequirements()[0] + require.Equal(t, map[string][]string{"oauth": {"read"}, "key": {"admin"}}, req.SchemeRequirements) + for _, tc := range []struct { + name string + dsl func() + want string + }{ + {"unknown scope", func() { + ScopeFor("oauth", "unknown") + }, "not found in scheme"}, + {"wrong scheme", func() { + ScopeFor("missing", "read") + }, "not in this requirement"}, + {"roles on oauth", func() { + RoleFor("oauth", "admin") + }, "non-OAuth scheme"}, + {"scopes on key", func() { + ScopeFor("key", "read") + }, "OAuth2 scheme"}, + {"nil scheme", func() { + RoleFor((*expr.SchemeExpr)(nil), "admin") + }, "not in this requirement"}, + } { + t.Run(tc.name, func(t *testing.T) { + err := expr.RunInvalidDSL(t, design(tc.dsl)) + require.ErrorContains(t, err, tc.want) + }) + } +} diff --git a/dsl/untagged_maps_test.go b/dsl/untagged_maps_test.go new file mode 100644 index 000000000..0458e6efa --- /dev/null +++ b/dsl/untagged_maps_test.go @@ -0,0 +1,120 @@ +package dsl_test + +import ( + "testing" + + "github.com/stretchr/testify/require" + + . "github.com/CaliLuke/loom/dsl" + "github.com/CaliLuke/loom/expr" +) + +func TestUntaggedAcceptsStringMapFields(t *testing.T) { + root := expr.RunDSL(t, func() { + key := Type("Key", String) + item := Type("Item", func() { + Attribute("id", String) + Required("id") + }) + lookup := Type("Lookup", MapOf(key, item)) + tree := Type("Tree", func() { + Attribute("id", String) + Attribute("children", MapOf(String, "Tree")) + Required("id") + }) + record := Type("Record", func() { + Attribute("id", String) + Attribute("properties", MapOf(String, Any)) + Attribute("labels", MapOf(String, String)) + Attribute("records", lookup) + Attribute("arrays", ArrayOf(MapOf(String, ArrayOf(item)))) + Attribute("nested", MapOf(String, MapOf(String, item))) + Attribute("tree", tree) + Required("id") + }) + absent := Type("Absent", func() { + Attribute("missing", Boolean) + Required("missing") + }) + Service("maps", func() { + Method("read", func() { + Result(OneOf(record, absent), func() { + Untagged() + }) + }) + }) + }) + require.NotNil(t, root.Service("maps")) +} + +func TestUntaggedRejectsNonStringMapKeys(t *testing.T) { + for _, key := range []expr.DataType{Int, Boolean} { + err := expr.RunInvalidDSL(t, func() { + record := Type("Record", func() { + Attribute("values", MapOf(key, String)) + }) + absent := Type("Absent", func() { + Attribute("missing", Boolean) + }) + Service("maps", func() { + Method("read", func() { + Result(OneOf(record, absent), func() { + Untagged() + }) + }) + }) + }) + require.ErrorContains(t, err, "map keys must be strings") + } +} + +func TestUntaggedRejectsUnsupportedMapValues(t *testing.T) { + for _, tc := range []struct { + name string + field func() + message string + }{ + {"opaque key", func() { + Attribute("values", MapOf(String, String), func() { + Key(func() { + Meta("struct:field:type", "Custom", "example.com/custom") + }) + }) + }, "opaque codec"}, + {"opaque value", func() { + Attribute("values", MapOf(String, String), func() { + Elem(func() { + Meta("openapi:contentEncoding", "hex") + }) + }) + }, "schema encoding override"}, + {"inline object", func() { + Attribute("values", MapOf(String, &expr.Object{{Name: "name", Attribute: &expr.AttributeExpr{Type: String}}})) + }, "map values must be primitive"}, + {"tagged union", func() { + Attribute("values", MapOf(String, "Choice")) + }, "map values must be primitive"}, + } { + t.Run(tc.name, func(t *testing.T) { + err := expr.RunInvalidDSL(t, func() { + first := Type("First", func() { + Attribute("name", String) + }) + second := Type("Second", func() { + Attribute("id", String) + }) + Type("Choice", OneOf(first, second)) + record := Type("Record", tc.field) + other := Type("Other", func() { + Attribute("missing", Boolean) + }) + Service("maps", func() { + Method("read", func() { + Result(OneOf(record, other), Untagged) + }) + }) + }) + require.ErrorContains(t, err, tc.message) + }) + } +} diff --git a/dsl/untagged_nested_test.go b/dsl/untagged_nested_test.go new file mode 100644 index 000000000..0c8af582a --- /dev/null +++ b/dsl/untagged_nested_test.go @@ -0,0 +1,85 @@ +package dsl_test + +import ( + "testing" + + "github.com/stretchr/testify/require" + + . "github.com/CaliLuke/loom/dsl" + "github.com/CaliLuke/loom/expr" +) + +func TestUntaggedNestedComposition(t *testing.T) { + root := expr.RunDSL(t, func() { + person := Type("Person", func() { + Attribute("email", String) + Required("email") + }) + group := Type("Group", func() { + Attribute("label", String) + Required("label") + }) + entry := Type("Entry", OneOf(person, group), Untagged) + page := Type("Page", func() { + Attribute("results", ArrayOf(entry)) + Attribute("lookup", MapOf(String, entry)) + Attribute("single", entry) + Required("results") + }) + Service("directory", func() { + Method("read", func() { + Result(OneOf(person, page), Untagged) + }) + }) + }) + require.NotNil(t, root.Service("directory")) +} + +func TestUntaggedRecursiveNestedComposition(t *testing.T) { + root := expr.RunDSL(t, func() { + node := Type("Node", func() { + Attribute("children", ArrayOf("Tree")) + Required("children") + }) + leaf := Type("Leaf", func() { + Attribute("name", String) + Required("name") + }) + tree := Type("Tree", OneOf(node, leaf), Untagged) + Service("tree", func() { + Method("read", func() { + Result(tree) + }) + }) + }) + require.NotNil(t, root.Service("tree")) +} + +func TestUntaggedNestedRejectsUnsupportedBranches(t *testing.T) { + for _, tagged := range []bool{false, true} { + err := expr.RunInvalidDSL(t, func() { + person := Type("Person", func() { + Attribute("name", String) + Required("name") + }) + choice := Type("Choice", OneOf(person, String), func() { + if !tagged { + Untagged() + } + }) + page := Type("Page", func() { + Attribute("rows", ArrayOf(choice)) + }) + Service("unsupported", func() { + Method("read", func() { + Result(OneOf(person, page), Untagged) + }) + }) + }) + if tagged { + require.ErrorContains(t, err, "array elements must be primitive") + } else { + require.ErrorContains(t, err, "must be a concrete named object or array type") + } + } +} diff --git a/expr/attribute_composition.go b/expr/attribute_composition.go index 70e8bb95d..66b2f1653 100644 --- a/expr/attribute_composition.go +++ b/expr/attribute_composition.go @@ -33,6 +33,19 @@ func (a *AttributeExpr) DSLSource() func() { return a.DSLFunc } +// CompositionBases returns the named object contracts retained by Extend after +// composition. The returned slice is detached; its declarations are shared and +// must not be mutated. Before composition it returns the authored bases. +func (a *AttributeExpr) CompositionBases() []DataType { + if a == nil { + return nil + } + if len(a.compositionBases) > 0 { + return slices.Clone(a.compositionBases) + } + return slices.Clone(a.Bases) +} + // composeDesign establishes object shape before any transport prepares a // mapping or validation inspects a field set. Finalize still owns defaults, // views and transport lowering that depend on successful validation. diff --git a/expr/attribute_composition_reference_test.go b/expr/attribute_composition_reference_test.go new file mode 100644 index 000000000..bba8250e0 --- /dev/null +++ b/expr/attribute_composition_reference_test.go @@ -0,0 +1,21 @@ +package expr + +import ( + "testing" + + "github.com/stretchr/testify/require" +) + +func TestCompositionBasesRetainDeclarationIdentity(t *testing.T) { + base := &UserTypeExpr{TypeName: "Base", AttributeExpr: &AttributeExpr{Type: &Object{}}} + attribute := &AttributeExpr{Type: &Object{}, compositionBases: []DataType{base}} + copied := DupAtt(attribute) + for _, occurrence := range []*AttributeExpr{attribute, copied} { + bases := occurrence.CompositionBases() + require.Equal(t, []DataType{base}, bases) + require.Same(t, base, bases[0]) + bases[0] = String + require.Equal(t, []DataType{base}, occurrence.CompositionBases()) + } + require.Empty(t, (*AttributeExpr)(nil).CompositionBases()) +} diff --git a/expr/attribute_validate.go b/expr/attribute_validate.go index 55e1c3040..ff04a088b 100644 --- a/expr/attribute_validate.go +++ b/expr/attribute_validate.go @@ -334,11 +334,17 @@ func (v *attributeValidator) validateChildTypes(a *AttributeExpr, ctx string, pa } func isUntaggedBranchFieldType(dataType DataType) bool { + if union := AsUnion(dataType); union != nil { + return union.Untagged + } if IsPrimitive(dataType) { return true } if userType, named := dataType.(UserType); named { - return !IsAlias(userType) && (IsObject(userType) || IsArray(userType)) + return !IsAlias(userType) && (IsObject(userType) || IsArray(userType) || IsMap(userType)) + } + if mapping := AsMap(dataType); mapping != nil { + return isUntaggedBranchFieldType(mapping.ElemType.Type) } array := AsArray(dataType) return array != nil && isUntaggedBranchFieldType(array.ElemType.Type) diff --git a/expr/field_layout.go b/expr/field_layout.go index 0b4533c57..cfa73ec7e 100644 --- a/expr/field_layout.go +++ b/expr/field_layout.go @@ -6,6 +6,7 @@ import "github.com/CaliLuke/loom/internal/encodingmeta" // Composition uses the same owner after inherited defaults and requiredness have // been finalized. Type names remain the code generator's responsibility. func ServiceFieldLayout(parent *AttributeExpr, name string, attribute *AttributeExpr, pointer, useDefault bool) encodingmeta.FieldLayout { + parent = fieldLayoutDeclaration(parent) return serviceFieldLayout(attribute, parent.IsRequired(name), pointer, func() bool { return parent.IsPrimitivePointer(name, useDefault) }) @@ -15,7 +16,7 @@ func ServiceFieldLayout(parent *AttributeExpr, name string, attribute *Attribute // The attribute supplies current explicit type and codec decisions; callers must // pair facts and attributes from the same stable object analysis. func (f ObjectFieldFacts) ServiceLayout(attribute *AttributeExpr, pointer, useDefault bool) encodingmeta.FieldLayout { - return serviceFieldLayout(attribute, f.Required, pointer, func() bool { + return serviceFieldLayout(attribute, f.LayoutRequired, pointer, func() bool { return f.PrimitivePointer(useDefault) }) } diff --git a/expr/method.go b/expr/method.go index 6541bcb94..d3cf7e1b6 100644 --- a/expr/method.go +++ b/expr/method.go @@ -3,6 +3,7 @@ package expr import ( "errors" "fmt" + "slices" "github.com/CaliLuke/loom/eval" ) @@ -419,22 +420,7 @@ func copyReqs(reqs []*SecurityExpr) []*SecurityExpr { } reqs2 := make([]*SecurityExpr, len(reqs)) for i, req := range reqs { - req2 := &SecurityExpr{Scopes: req.Scopes} - schs := make([]*SchemeExpr, len(req.Schemes)) - for j, sch := range req.Schemes { - schs[j] = &SchemeExpr{ - Kind: sch.Kind, - SchemeName: sch.SchemeName, - Description: sch.Description, - In: sch.In, - Name: sch.Name, - Scopes: sch.Scopes, - Flows: sch.Flows, - Meta: sch.Meta, - } - } - req2.Schemes = schs - reqs2[i] = req2 + reqs2[i] = DupRequirement(req) } return reqs2 } @@ -532,7 +518,7 @@ func containsRequirement(requirements []*SecurityExpr, candidate *SecurityExpr) } func requirementEqual(left *SecurityExpr, right *SecurityExpr) bool { - if len(left.Scopes) != len(right.Scopes) || len(left.Schemes) != len(right.Schemes) { + if len(left.Scopes) != len(right.Scopes) || len(left.Schemes) != len(right.Schemes) || len(left.SchemeRequirements) != len(right.SchemeRequirements) { return false } for i, scope := range left.Scopes { @@ -540,6 +526,12 @@ func requirementEqual(left *SecurityExpr, right *SecurityExpr) bool { return false } } + for name, names := range left.SchemeRequirements { + other, ok := right.SchemeRequirements[name] + if !ok || !slices.Equal(names, other) { + return false + } + } for i, scheme := range left.Schemes { other := right.Schemes[i] if scheme.Kind != other.Kind || scheme.SchemeName != other.SchemeName || scheme.In != other.In || scheme.Name != other.Name { diff --git a/expr/method_validate_security.go b/expr/method_validate_security.go index 505526350..bebbf2592 100644 --- a/expr/method_validate_security.go +++ b/expr/method_validate_security.go @@ -2,6 +2,8 @@ package expr import ( "fmt" + "maps" + "slices" "strings" "github.com/CaliLuke/loom/eval" @@ -162,6 +164,28 @@ func (m *MethodExpr) validateRequirementScopes(r *SecurityExpr) *eval.Validation } verr.Add(m, "security scope %q not found in any of the security schemes.", scope) } + for _, name := range slices.Sorted(maps.Keys(r.SchemeRequirements)) { + names := r.SchemeRequirements[name] + var selected *SchemeExpr + for _, scheme := range r.Schemes { + if scheme.SchemeName == name { + selected = scheme + break + } + } + if selected == nil || selected.Kind == NoKind { + verr.Add(m, "security scheme %q is not in this requirement", name) + continue + } + if selected.Kind != OAuth2Kind { + continue + } + for _, scope := range names { + if !requirementHasScope(&SecurityExpr{Schemes: []*SchemeExpr{selected}}, scope) { + verr.Add(m, "security scope %q not found in scheme %q", scope, name) + } + } + } return verr } @@ -170,10 +194,8 @@ func requirementHasScope(r *SecurityExpr, scope string) bool { if !schemeSupportsScopes(s) { continue } - for _, se := range s.Scopes { - if se.Name == scope { - return true - } + if slices.Contains(s.ScopeNames(), scope) { + return true } } return false diff --git a/expr/object_field_facts.go b/expr/object_field_facts.go index 3b46ee8a5..ae620773e 100644 --- a/expr/object_field_facts.go +++ b/expr/object_field_facts.go @@ -10,6 +10,9 @@ type ( ParentPresent bool // Required records normalized requiredness, including dangling names. Required bool + // LayoutRequired records requiredness on the type that declares the field, + // before requirements added by a named occurrence. It owns field storage. + LayoutRequired bool // HasDefault records selected default presence without materializing it. HasDefault bool // NativePrimitive records eligibility for native primitive pointer storage. @@ -94,13 +97,19 @@ func NewObjectFields(attribute *AttributeExpr) *ObjectFields { facts.Required = true result.fields[name] = facts } + for _, required := range fieldLayoutDeclaration(attribute).AllRequired() { + name := AttributeName(required) + facts := result.FieldFacts(name) + facts.LayoutRequired = true + result.fields[name] = facts + } return result } // PrimitivePointer reports whether an optional primitive uses pointer storage // under the selected default policy. Transport-wide pointer flags are separate. func (f ObjectFieldFacts) PrimitivePointer(useDefault bool) bool { - return f.NativePrimitive && !f.Required && (!f.HasDefault || !useDefault) + return f.NativePrimitive && !f.LayoutRequired && (!f.HasDefault || !useDefault) } // FieldFacts returns captured facts by logical name. Unknown names retain parent @@ -137,7 +146,7 @@ func (ma *MappedAttributeExpr) FieldFacts(name string) ObjectFieldFacts { if ma == nil { return ObjectFieldFacts{Index: -1} } - facts := ObjectFieldFacts{ParentPresent: true, Required: ma.IsRequired(name), Index: -1} + facts := ObjectFieldFacts{ParentPresent: true, Required: ma.IsRequired(name), LayoutRequired: ma.IsRequired(name), Index: -1} for index, field := range *AsObject(ma.Type) { if field.Name != name { continue @@ -161,3 +170,16 @@ func primitivePointerEligible(attribute *AttributeExpr) bool { kind := unalias(attribute.Type).Kind() return kind != BytesKind && kind != AnyKind } + +func fieldLayoutDeclaration(attribute *AttributeExpr) *AttributeExpr { + seen := make(map[UserType]bool) + for attribute != nil { + named, ok := attribute.Type.(UserType) + if !ok || seen[named] { + return attribute + } + seen[named] = true + attribute = named.Attribute() + } + return nil +} diff --git a/expr/object_field_facts_test.go b/expr/object_field_facts_test.go index b0224f999..60b4c0f45 100644 --- a/expr/object_field_facts_test.go +++ b/expr/object_field_facts_test.go @@ -84,8 +84,8 @@ func TestObjectFieldFactsDuplicateSelection(t *testing.T) { order = append(order, name) } require.Equal(t, []string{"n", "other"}, order) - require.Equal(t, ObjectFieldFacts{ParentPresent: true, Required: true, HasDefault: true, NativePrimitive: true, ElementName: "second:tail", OriginalKey: "n", Index: 3}, facts.FieldFacts("n")) - require.Equal(t, ObjectFieldFacts{ParentPresent: true, Required: true, Index: -1}, facts.FieldFacts("missing")) + require.Equal(t, ObjectFieldFacts{ParentPresent: true, Required: true, LayoutRequired: true, HasDefault: true, NativePrimitive: true, ElementName: "second:tail", OriginalKey: "n", Index: 3}, facts.FieldFacts("n")) + require.Equal(t, ObjectFieldFacts{ParentPresent: true, Required: true, LayoutRequired: true, Index: -1}, facts.FieldFacts("missing")) } func TestObjectFieldFactsSnapshotAndMutableQueries(t *testing.T) { @@ -167,3 +167,22 @@ func TestObjectFieldFactsRepeatedExactKeys(t *testing.T) { } require.Equal(t, []string{"n"}, order) } + +func TestObjectFieldFactsNamedOccurrenceStorage(t *testing.T) { + for _, required := range []bool{false, true} { + field := &AttributeExpr{Type: String} + decl := &AttributeExpr{Type: &Object{{Name: "name", Attribute: field}}} + if required { + decl.Validation = &ValidationExpr{Required: []string{"name"}} + } + named := &UserTypeExpr{TypeName: "Value", AttributeExpr: decl} + alias := &UserTypeExpr{TypeName: "Alias", AttributeExpr: &AttributeExpr{Type: named}} + occurrence := &AttributeExpr{Type: alias, Validation: &ValidationExpr{Required: []string{"name"}}} + facts := NewObjectFields(occurrence).FieldFacts("name") + require.True(t, facts.Required) + require.Equal(t, required, facts.LayoutRequired) + require.Equal(t, !required, facts.PrimitivePointer(false)) + require.Equal(t, ServiceFieldLayout(decl, "name", field, false, true), facts.ServiceLayout(field, false, true)) + require.Equal(t, ServiceFieldLayout(decl, "name", field, false, true), ServiceFieldLayout(occurrence, "name", field, false, true)) + } +} diff --git a/expr/security.go b/expr/security.go index c5874ef81..b3c2a5031 100644 --- a/expr/security.go +++ b/expr/security.go @@ -66,6 +66,10 @@ type ( Schemes []*SchemeExpr // Scopes list the required scopes if any. Scopes []string + // SchemeRequirements holds names required only by the named scheme. + // OAuth2 values are declared scopes; other values are application roles. + // These requirements are additive to Scopes, never shared across schemes. + SchemeRequirements map[string][]string } // SessionAuthExpr defines a logical auth contract backed by one or more @@ -110,7 +114,7 @@ type ( // Name refers to a header or parameter name, based on In's // value. Name string - // Scopes lists the Basic, APIKey, JWT or OAuth2 scopes. + // Scopes lists Basic, APIKey or JWT scopes, or default OAuth2 flow scopes. Scopes []*ScopeExpr // Flows determine the oauth2 flows supported by this scheme. Flows []*FlowExpr @@ -132,6 +136,9 @@ type ( RefreshURL string // DeviceAuthorizationURL starts a device authorization flow. DeviceAuthorizationURL string + // Scopes is this flow's complete scope inventory. Nil inherits the + // scheme defaults; a nonnil empty slice declares no scopes. + Scopes []*ScopeExpr } // ScopeExpr defines a security scope. @@ -263,8 +270,9 @@ func (s *SecurityExpr) EvalName() string { // DupRequirement creates a copy of the given security requirement. func DupRequirement(req *SecurityExpr) *SecurityExpr { dup := &SecurityExpr{ - Scopes: req.Scopes, - Schemes: make([]*SchemeExpr, 0, len(req.Schemes)), + Scopes: append([]string(nil), req.Scopes...), + Schemes: make([]*SchemeExpr, 0, len(req.Schemes)), + SchemeRequirements: dupSchemeRequirements(req.SchemeRequirements), } for _, s := range req.Schemes { dup.Schemes = append(dup.Schemes, DupScheme(s)) @@ -279,10 +287,16 @@ func DupScheme(sch *SchemeExpr) *SchemeExpr { SchemeName: sch.SchemeName, Description: sch.Description, In: sch.In, - Scopes: sch.Scopes, - Flows: sch.Flows, + Name: sch.Name, + Scopes: dupScopeDeclarations(sch.Scopes), + Flows: append([]*FlowExpr(nil), sch.Flows...), Meta: sch.Meta, } + for i, flow := range sch.Flows { + copy := *flow + copy.Scopes = dupScopeDeclarations(flow.Scopes) + dup.Flows[i] = © + } return &dup } diff --git a/expr/security_flow_scopes_test.go b/expr/security_flow_scopes_test.go new file mode 100644 index 000000000..6b4e1c380 --- /dev/null +++ b/expr/security_flow_scopes_test.go @@ -0,0 +1,48 @@ +package expr + +import ( + "testing" + + "github.com/stretchr/testify/require" +) + +func TestOAuthFlowScopeOwnership(t *testing.T) { + scheme := &SchemeExpr{Kind: OAuth2Kind, Scopes: []*ScopeExpr{{Name: "shared", Description: "default"}}, Flows: []*FlowExpr{ + {Kind: PasswordFlowKind}, + {Kind: ImplicitFlowKind, Scopes: []*ScopeExpr{}}, + {Kind: ClientCredentialsFlowKind, Scopes: []*ScopeExpr{{Name: "write", Description: "machine"}, {Name: "shared", Description: "client"}}}, + }} + require.Equal(t, scheme.Scopes, scheme.FlowScopes(scheme.Flows[0])) + require.Empty(t, scheme.FlowScopes(scheme.Flows[1])) + require.NotNil(t, scheme.FlowScopes(scheme.Flows[1])) + require.Equal(t, "client", scheme.FlowScopes(scheme.Flows[2])[1].Description) + require.Equal(t, []string{"shared", "write"}, scheme.ScopeNames()) + require.Equal(t, []string{"write"}, (&SecurityExpr{Scopes: []string{"write"}}).RequiredScopes(scheme)) + copy := DupScheme(scheme) + copy.Scopes[0].Description = "changed" + copy.Flows[2].Scopes[0].Name = "changed" + require.Equal(t, "default", scheme.Scopes[0].Description) + require.Equal(t, "write", scheme.Flows[2].Scopes[0].Name) + require.NotNil(t, copy.Flows[1].Scopes) + projected := scheme.FlowScopes(scheme.Flows[0]) + projected[0].Name = "changed" + require.Equal(t, "shared", scheme.Scopes[0].Name) +} + +func TestOAuthFlowScopeDuplicateDescriptions(t *testing.T) { + for _, local := range []bool{false, true} { + t.Run(map[bool]string{false: "defaults", true: "local"}[local], func(t *testing.T) { + inventory := []*ScopeExpr{{Name: "read", Description: "first"}, {Name: "write", Description: "other"}, {Name: "read", Description: "last"}} + scheme := &SchemeExpr{Scopes: inventory} + flow := &FlowExpr{} + if local { + flow.Scopes = inventory + } + got := scheme.FlowScopes(flow) + require.Equal(t, []*ScopeExpr{{Name: "read", Description: "last"}, {Name: "write", Description: "other"}}, got) + require.Equal(t, "first", inventory[0].Description) + got[0].Description = "changed" + require.Equal(t, "last", inventory[2].Description) + }) + } +} diff --git a/expr/security_scopes.go b/expr/security_scopes.go index 75c65641f..51a11adcd 100644 --- a/expr/security_scopes.go +++ b/expr/security_scopes.go @@ -1,17 +1,95 @@ package expr +import "slices" + // RequiredScopes returns the requirement scopes declared by scheme, in authored -// requirement order. The result is independently owned. Requirement validation -// separately rejects scopes absent from every scheme in the requirement. +// requirement order, followed by its explicit per-scheme requirements. For +// non-OAuth schemes the explicit names are application roles. The result is +// independently owned. Validation separately rejects unknown OAuth scopes. func (s *SecurityExpr) RequiredScopes(scheme *SchemeExpr) []string { var scopes []string + inventory := scheme.ScopeNames() for _, required := range s.Scopes { - for _, declared := range scheme.Scopes { - if required == declared.Name { - scopes = append(scopes, required) - break - } + if slices.Contains(inventory, required) { + scopes = append(scopes, required) + } + } + for _, name := range s.SchemeRequirements[scheme.SchemeName] { + if !slices.Contains(scopes, name) { + scopes = append(scopes, name) } } return scopes } + +func dupSchemeRequirements(source map[string][]string) map[string][]string { + if source == nil { + return nil + } + result := make(map[string][]string, len(source)) + for scheme, names := range source { + result[scheme] = slices.Clone(names) + } + return result +} + +// FlowScopes returns a flow's effective scope inventory. A nil flow inventory +// inherits the scheme defaults; a nonnil empty inventory remains empty. The +// returned scope declarations are independently owned. Repeated names retain +// their first position and last description, matching OpenAPI map semantics. +func (s *SchemeExpr) FlowScopes(flow *FlowExpr) []*ScopeExpr { + inventory := flow.Scopes + if inventory == nil { + inventory = s.Scopes + } + if inventory == nil { + return nil + } + result := make([]*ScopeExpr, 0, len(inventory)) + positions := make(map[string]int, len(inventory)) + for _, scope := range inventory { + if position, exists := positions[scope.Name]; exists { + result[position].Description = scope.Description + continue + } + positions[scope.Name] = len(result) + copy := *scope + result = append(result, ©) + } + return result +} + +// ScopeNames returns the names available to requirement validation and auth +// callbacks, in first-declaration order. For OAuth2 it collects names from the +// effective flow inventories without combining their descriptions or changing +// any flow. Flow selection and granted permissions remain application-owned. +func (s *SchemeExpr) ScopeNames() []string { + inventories := [][]*ScopeExpr{s.Scopes} + if s.Kind == OAuth2Kind && len(s.Flows) > 0 { + inventories = make([][]*ScopeExpr, len(s.Flows)) + for i, flow := range s.Flows { + inventories[i] = s.FlowScopes(flow) + } + } + var names []string + for _, inventory := range inventories { + for _, scope := range inventory { + if !slices.Contains(names, scope.Name) { + names = append(names, scope.Name) + } + } + } + return names +} + +func dupScopeDeclarations(source []*ScopeExpr) []*ScopeExpr { + if source == nil { + return nil + } + result := make([]*ScopeExpr, len(source)) + for i, scope := range source { + value := *scope + result[i] = &value + } + return result +} diff --git a/expr/security_scopes_test.go b/expr/security_scopes_test.go index 81a6ce77d..767941455 100644 --- a/expr/security_scopes_test.go +++ b/expr/security_scopes_test.go @@ -24,3 +24,22 @@ func TestRequiredScopeProjection(t *testing.T) { }) } } + +func TestPerSchemeRequirementOwnership(t *testing.T) { + left := &SchemeExpr{SchemeName: "left", Kind: OAuth2Kind, Name: "Authorization", Scopes: []*ScopeExpr{{Name: "read"}}} + right := &SchemeExpr{SchemeName: "right", Kind: OAuth2Kind, Scopes: left.Scopes} + key := &SchemeExpr{SchemeName: "key", Kind: APIKeyKind} + req := &SecurityExpr{Schemes: []*SchemeExpr{left, right, key}, SchemeRequirements: map[string][]string{"left": {"read"}, "right": {}, "key": {"admin"}}} + require.Equal(t, []string{"read"}, req.RequiredScopes(left)) + require.Empty(t, req.RequiredScopes(right), "a shared inventory must not share requirements") + require.Equal(t, []string{"admin"}, req.RequiredScopes(key), "roles need no inventory") + copy := DupRequirement(req) + require.True(t, requirementEqual(req, copy)) + require.Equal(t, "Authorization", copy.Schemes[0].Name) + copy.SchemeRequirements["left"][0] = "write" + require.False(t, requirementEqual(req, copy)) + require.Equal(t, []string{"read"}, req.RequiredScopes(left)) + req.Scopes = []string{"read"} + require.Equal(t, []string{"read"}, req.RequiredScopes(left), "shared and explicit requirements are additive without duplicates") + require.Equal(t, []string{"read"}, req.RequiredScopes(right)) +} diff --git a/expr/testdata/security_scope.go b/expr/testdata/security_scope.go index bc2b58d6e..d95df4456 100644 --- a/expr/testdata/security_scope.go +++ b/expr/testdata/security_scope.go @@ -157,3 +157,56 @@ func matrixMethod(transport, path string, security func(), credentials ...string JSONRPC(func() {}) } } + +// SecurityPerSchemeDSL checks separate requirements on schemes with overlapping +// scope inventories and separate alternatives on one physical credential. +func SecurityPerSchemeDSL(transport string) func() { + return func() { + left := OAuth2Security("left", func() { + ClientCredentialsFlow("https://example.com/token", "", func() { + Scope("read") + Scope("write") + }) + }) + right := OAuth2Security("right", func() { + ClientCredentialsFlow("https://example.com/token", "", func() { + Scope("read") + Scope("write") + }) + }) + API("per_scheme", func() { + if transport == "jsonrpc" { + JSONRPC(func() {}) + } + }) + Service("per_scheme", func() { + matrixTransportService(transport, "per_scheme") + Method("read", func() { + Security(left, right, func() { + ScopeFor(left, "read") + ScopeFor(right, "write") + }) + Security(right, func() { + ScopeFor(right, "read") + }) + Security() + Payload(func() { + AccessToken("token") + if transport == "jsonrpc" { + ID("id", String) + } + }) + switch transport { + case "http": + HTTP(func() { + GET("/per_scheme") + }) + case "grpc": + GRPC(func() {}) + case "jsonrpc": + JSONRPC(func() {}) + } + }) + }) + } +} diff --git a/expr/tla/oauth_flow_scopes/Inventories.tla b/expr/tla/oauth_flow_scopes/Inventories.tla new file mode 100644 index 000000000..942f7137f --- /dev/null +++ b/expr/tla/oauth_flow_scopes/Inventories.tla @@ -0,0 +1,15 @@ +------------------------- MODULE Inventories ------------------------- +EXTENDS TLC +CONSTANT Flatten +VARIABLES shared, local, explicit +Flows == {"code", "client"} +Names == {"read", "write"} +Init == /\ shared \in SUBSET Names + /\ local \in [Flows -> SUBSET Names] + /\ explicit \in [Flows -> BOOLEAN] +Expected(f) == IF explicit[f] THEN local[f] ELSE shared +Inventory(f) == IF Flatten THEN UNION {Expected(g): g \in Flows} ELSE Expected(f) +Isolation == \A f \in Flows: Inventory(f) = Expected(f) +Next == UNCHANGED <> +Spec == Init /\ [][Next]_<> +======================================================================= diff --git a/expr/tla/oauth_flow_scopes/README.md b/expr/tla/oauth_flow_scopes/README.md new file mode 100644 index 000000000..02eb0db8d --- /dev/null +++ b/expr/tla/oauth_flow_scopes/README.md @@ -0,0 +1,26 @@ +# OAuth flow inventories + +`Inventories.tla` checks two flows and two scope names with shared defaults, +flow-local inventories, and an explicit-block flag. `FlowScopes` in +`expr/security_scopes.go` implements the selected inventory. A supplied empty +block overrides defaults; a missing block inherits them. `ScopeNames` computes +availability for requirement admission, never a replacement flow inventory. + +Run explicitly with a locally installed TLA+ 1.7.4 jar: + +```sh +java -cp "$TLA_JAR" tlc2.TLC -config checked.cfg Inventories.tla +java -cp "$TLA_JAR" tlc2.TLC -config flatten.cfg Inventories.tla +``` + +The checked model explores 256 distinct states (512 generated) and preserves +Isolation. The flattening candidate fails initially: an empty inherited flow +receives a name declared only by another flow. These are opt-in design checks. +No proof tools are required by ordinary builds, hooks or CI. + +The model abstracts descriptions, URLs and authentication policy. Expression +and DSL tests cover inheritance, explicit empty inventories and deep copies; +import round trips and rendered OpenAPI tests cover exact names/descriptions; +the generated security fixture checks callback inventories, required grants +and denial. Authentication, issuer validation and token grants remain owned +by the application. The finite model does not prove the Go implementation. diff --git a/expr/tla/oauth_flow_scopes/checked.cfg b/expr/tla/oauth_flow_scopes/checked.cfg new file mode 100644 index 000000000..409dabf2a --- /dev/null +++ b/expr/tla/oauth_flow_scopes/checked.cfg @@ -0,0 +1,3 @@ +CONSTANT Flatten = FALSE +SPECIFICATION Spec +INVARIANT Isolation diff --git a/expr/tla/oauth_flow_scopes/flatten.cfg b/expr/tla/oauth_flow_scopes/flatten.cfg new file mode 100644 index 000000000..4b00ed0c9 --- /dev/null +++ b/expr/tla/oauth_flow_scopes/flatten.cfg @@ -0,0 +1,3 @@ +CONSTANT Flatten = TRUE +SPECIFICATION Spec +INVARIANT Isolation diff --git a/expr/tla/security_requirements/README.md b/expr/tla/security_requirements/README.md new file mode 100644 index 000000000..793ef2059 --- /dev/null +++ b/expr/tla/security_requirements/README.md @@ -0,0 +1,35 @@ +# Per-scheme security requirements + +`Requirements.tla` enumerates two schemes, two permission names, every list of +AND branches (including empty/anonymous branches), and every per-scheme grant +set. `checked.cfg` preserves each scheme's required names and passed 4,096 +states (8,192 generated) with TLC 1.7.4. `flatten.cfg` deliberately combines +requirements across a conjunction; its initial-state counterexample requires +no grants from scheme a and read from b, but incorrectly requires read from a. +This compares design candidates; the prior importer explicitly rejected unequal +lists rather than admitting that faulty projection. + +Run explicitly from this directory with Java and the installed TLA+ 1.7.4 jar: + +```sh +java -XX:+UseParallelGC -cp "$TLA_JAR" tlc2.TLC -workers 1 -config checked.cfg -metadir /tmp/loom-security-requirements Requirements.tla +``` + +Substitute `flatten.cfg` and a separate temporary metadir for the negative case. +This is an opt-in design tool, never a build, hook, CI or release gate. + +`expr.SecurityExpr.SchemeRequirements` owns explicit names by scheme identity. +`RequiredScopes` adds shared declared requirements without crossing scheme +boundaries. `ScopeFor` admits OAuth scopes; `RoleFor` admits non-OAuth roles. +Shared Scope remains a convenience for names required by every declaring scheme; +explicit requirements add to it rather than silently replacing authored intent. +Import uses explicit per-scheme requirements. The application determines grants; +callbacks receive them through the existing RequiredScopes/Validate contract. + +Direct DSL/expression tests cover admission, inheritance copies and alternative +identity. The generated scope harness checks exact callback arrays, rejected +roles and grants attached to the wrong scheme. The imported credential fixture +checks wire execution and export. HTTP/gRPC/JSON-RPC projection tests cover shared +service ownership. Wire credential parsing, callbacks, OAuth inventory validity, +list ordering and copying are outside this finite set model and remain Go test +obligations. Keep raw checker output and temporary state out of the repository. diff --git a/expr/tla/security_requirements/Requirements.tla b/expr/tla/security_requirements/Requirements.tla new file mode 100644 index 000000000..fa2bc2e37 --- /dev/null +++ b/expr/tla/security_requirements/Requirements.tla @@ -0,0 +1,20 @@ +--------------------------- MODULE Requirements --------------------------- +EXTENDS TLC +CONSTANT Flatten +VARIABLES requirements, required, granted +Schemes == {"a", "b"} +Names == {"read", "admin"} +Init == /\ requirements \in SUBSET (SUBSET Schemes) + /\ required \in [Schemes -> SUBSET Names] + /\ granted \in [Schemes -> SUBSET Names] +Expected == \E branch \in requirements: \A scheme \in branch: + required[scheme] \subseteq granted[scheme] +Projected(branch, scheme) == IF Flatten + THEN UNION {required[s]: s \in branch} + ELSE required[scheme] +Actual == \E branch \in requirements: \A scheme \in branch: + Projected(branch, scheme) \subseteq granted[scheme] +Agreement == Actual = Expected +Next == UNCHANGED <> +Spec == Init /\ [][Next]_<> +============================================================================= diff --git a/expr/tla/security_requirements/checked.cfg b/expr/tla/security_requirements/checked.cfg new file mode 100644 index 000000000..449cf511e --- /dev/null +++ b/expr/tla/security_requirements/checked.cfg @@ -0,0 +1,3 @@ +CONSTANT Flatten = FALSE +SPECIFICATION Spec +INVARIANT Agreement diff --git a/expr/tla/security_requirements/flatten.cfg b/expr/tla/security_requirements/flatten.cfg new file mode 100644 index 000000000..4838614b2 --- /dev/null +++ b/expr/tla/security_requirements/flatten.cfg @@ -0,0 +1,3 @@ +CONSTANT Flatten = TRUE +SPECIFICATION Spec +INVARIANT Agreement diff --git a/expr/untagged_array_test.go b/expr/untagged_array_test.go index d69802682..c9422112f 100644 --- a/expr/untagged_array_test.go +++ b/expr/untagged_array_test.go @@ -59,7 +59,7 @@ func TestUntaggedRejectsUnsupportedNestedShapes(t *testing.T) { }) }) }) - require.ErrorContains(t, err, `field "mapping" must be primitive`) + require.ErrorContains(t, err, `map keys must be strings`) } } diff --git a/expr/untagged_validation.go b/expr/untagged_validation.go index befed85c9..5b9d17c5b 100644 --- a/expr/untagged_validation.go +++ b/expr/untagged_validation.go @@ -33,13 +33,23 @@ func validateUntaggedShape(att *AttributeExpr, ctx, branch string, parent eval.E validateUntaggedShape(ut.Attribute(), ctx, branch, parent, errors, seen) return } + if union := AsUnion(att.Type); union != nil { + for _, nested := range union.Values { + validateUntaggedShape(nested.Attribute, ctx, branch, parent, errors, seen) + } + return + } if array := AsArray(att.Type); array != nil { if !isUntaggedBranchFieldType(array.ElemType.Type) { - errors.Add(parent, "%suntagged OneOf branch %q array elements must be primitive, a concrete named object type, or an array of either", ctx, branch) + errors.Add(parent, "%suntagged OneOf branch %q array elements must be primitive, a concrete named object type, an untagged union, or arrays or string-keyed maps of these", ctx, branch) } validateUntaggedShape(array.ElemType, ctx, branch, parent, errors, seen) return } + if mapping := AsMap(att.Type); mapping != nil { + validateUntaggedMap(mapping, ctx, branch, parent, errors, seen) + return + } object := AsObject(att.Type) if object == nil { return @@ -50,7 +60,7 @@ func validateUntaggedShape(att *AttributeExpr, ctx, branch string, parent eval.E names := make(map[string]bool) for _, field := range *object { if !isUntaggedBranchFieldType(field.Attribute.Type) { - errors.Add(parent, "%suntagged OneOf branch %q field %q must be primitive, a concrete named object type, or an array of either", ctx, branch, field.Name) + errors.Add(parent, "%suntagged OneOf branch %q field %q must be primitive, a concrete named object type, an untagged union, or arrays or string-keyed maps of these", ctx, branch, field.Name) } name := JSONFieldName(AttributeName(field.Name), field.Attribute) switch { @@ -66,6 +76,25 @@ func validateUntaggedShape(att *AttributeExpr, ctx, branch string, parent eval.E } } +func validateUntaggedMap(mapping *Map, ctx, branch string, parent eval.Expression, errors *eval.ValidationErrors, seen map[*AttributeExpr]bool) { + key := mapping.KeyType.Type + for { + ut, named := key.(UserType) + if !named { + break + } + key = ut.Attribute().Type + } + if key != String { + errors.Add(parent, "%suntagged OneOf branch %q map keys must be strings", ctx, branch) + } + if !isUntaggedBranchFieldType(mapping.ElemType.Type) { + errors.Add(parent, "%suntagged OneOf branch %q map values must be primitive, a concrete named object type, an untagged union, or arrays or string-keyed maps of these", ctx, branch) + } + validateUntaggedShape(mapping.KeyType, ctx, branch, parent, errors, seen) + validateUntaggedShape(mapping.ElemType, ctx, branch, parent, errors, seen) +} + func validateUntaggedCodec(att *AttributeExpr, ctx, branch string, parent eval.Expression, errors *eval.ValidationErrors) { if tag, exists := attributeJSONTag(att); exists { for _, option := range strings.Split(tag, ",")[1:] { diff --git a/expr/value_plan_json_shape.go b/expr/value_plan_json_shape.go index 7562f31b8..f1f377bee 100644 --- a/expr/value_plan_json_shape.go +++ b/expr/value_plan_json_shape.go @@ -18,7 +18,7 @@ func (p ValuePlan) MatchJSONSchema(wire jsontext.Value) (bool, error) { // JSONShape lowers the captured JSON representation into a detached matching // description for generated untagged codecs. It rejects opaque codecs and shapes -// outside the supported object/array/primitive grammar. Enum values are projected +// outside the supported object/array/map/untagged-union/primitive grammar. Enum values are projected // through this exact plan, retaining effective constraints and wire identity. func (p ValuePlan) JSONShape() (*loom.JSONShape, error) { if p.root == nil || p.root.schemaOnly { @@ -64,6 +64,12 @@ func lowerJSONShape(node *valuePlanNode, seen map[*valuePlanNode]*loom.JSONShape case ArrayKind: shape.Kind = "array" shape.Child, err = lowerJSONShape(node.element, seen) + case MapKind: + shape.Kind = "map" + shape.Child, err = lowerJSONShape(node.element, seen) + case UnionKind: + shape.Kind = "union" + shape.Branches, err = lowerJSONShapeBranches(node, seen) case ObjectKind: shape.Kind = "object" for _, field := range node.members { @@ -82,6 +88,21 @@ func lowerJSONShape(node *valuePlanNode, seen map[*valuePlanNode]*loom.JSONShape return shape, err } +func lowerJSONShapeBranches(node *valuePlanNode, seen map[*valuePlanNode]*loom.JSONShape) ([]*loom.JSONShape, error) { + if !node.untagged { + return nil, fmt.Errorf("JSON matching cannot lower a tagged union") + } + branches := make([]*loom.JSONShape, 0, len(node.branches)) + for _, branch := range node.branches { + child, err := lowerJSONShape(branch.node, seen) + if err != nil { + return nil, err + } + branches = append(branches, child) + } + return branches, nil +} + func jsonShapeBound(value *NumericBound) string { if value == nil { return "" diff --git a/expr/value_plan_json_shape_test.go b/expr/value_plan_json_shape_test.go index 35be6e382..d90492ddd 100644 --- a/expr/value_plan_json_shape_test.go +++ b/expr/value_plan_json_shape_test.go @@ -51,3 +51,80 @@ func TestJSONShapeLoweringMatchesPreparedSchema(t *testing.T) { require.NoError(t, err) require.True(t, canonical, "lowered description must not mutate its source plan") } + +func TestJSONShapeMapLoweringMatchesPreparedSchema(t *testing.T) { + item := &UserTypeExpr{TypeName: "Item", AttributeExpr: &AttributeExpr{ + Type: &Object{{Name: "name", Attribute: &AttributeExpr{Type: String, Validation: &ValidationExpr{MinLength: new(2)}}}}, + Validation: &ValidationExpr{Required: []string{"name"}}, + Meta: MetaExpr{"openapi:additionalProperties": {"false"}}, + }} + dictionary := &UserTypeExpr{TypeName: "Dictionary", AttributeExpr: &AttributeExpr{ + Type: &Map{KeyType: &AttributeExpr{Type: String}, ElemType: &AttributeExpr{Type: item, Nullable: true}}, + Validation: &ValidationExpr{MinLength: new(1)}, + }} + source := &AttributeExpr{Type: dictionary, Nullable: true, Validation: &ValidationExpr{MaxLength: new(2)}} + context := NewValueContext() + occurrence, err := context.NewOccurrence(source) + require.NoError(t, err) + plan, err := context.NewValuePlan(occurrence, ValuePlanRequest{Target: DupAtt(source), Codec: ValueCodecJSON, Use: ValuePlanDocumentation}) + require.NoError(t, err) + shape, err := plan.JSONShape() + require.NoError(t, err) + for _, tc := range []struct { + wire string + want bool + }{ + {`null`, true}, {`{}`, false}, {`[]`, false}, + {`{"a":{"name":"ok"}}`, true}, {`{"a":null}`, true}, + {`{"a":{"name":"ok"},"b":null}`, true}, + {`{"a":null,"b":null,"c":null}`, false}, + {`{"a":{"name":"x"}}`, false}, {`{"a":{}}`, false}, + {`{"a":{"name":"ok","extra":1}}`, false}, + {`{"a":{"name":"ok"},"b":{"name":"x"}}`, false}, + } { + matched, err := shape.Match([]byte(tc.wire)) + require.NoError(t, err) + require.Equal(t, tc.want, matched, tc.wire) + canonical, err := plan.MatchJSONSchema([]byte(tc.wire)) + require.NoError(t, err) + require.Equal(t, canonical, matched, tc.wire) + } +} + +func TestJSONShapeNestedUnionMatchesPreparedSchema(t *testing.T) { + first := &UserTypeExpr{TypeName: "Person", AttributeExpr: &AttributeExpr{ + Type: &Object{{Name: "name", Attribute: &AttributeExpr{Type: String}}}, + Validation: &ValidationExpr{Required: []string{"name"}}, + }} + second := &UserTypeExpr{TypeName: "Group", AttributeExpr: &AttributeExpr{ + Type: &Object{{Name: "group", Attribute: &AttributeExpr{Type: String}}}, + Validation: &ValidationExpr{Required: []string{"group"}}, + }} + choice := &Union{TypeName: "Entry", Untagged: true, Values: []*NamedAttributeExpr{ + {Name: "person", Attribute: &AttributeExpr{Type: first}}, + {Name: "group", Attribute: &AttributeExpr{Type: second}}, + }} + source := &AttributeExpr{Type: &Array{ElemType: &AttributeExpr{Type: choice}}} + context := NewValueContext() + occurrence, err := context.NewOccurrence(source) + require.NoError(t, err) + plan, err := context.NewValuePlan(occurrence, ValuePlanRequest{Target: DupAtt(source), Codec: ValueCodecJSON, Use: ValuePlanDocumentation}) + require.NoError(t, err) + shape, err := plan.JSONShape() + require.NoError(t, err) + for _, tc := range []struct { + wire string + want bool + }{ + {`[]`, true}, {`[{"name":"person"},{"group":"team"}]`, true}, + {`[{}]`, false}, {`[{"name":"person","group":"team"}]`, false}, + {`[null]`, false}, + } { + actual, err := shape.Match([]byte(tc.wire)) + require.NoError(t, err) + require.Equal(t, tc.want, actual, tc.wire) + canonical, err := plan.MatchJSONSchema([]byte(tc.wire)) + require.NoError(t, err) + require.Equal(t, canonical, actual) + } +} diff --git a/go.mod b/go.mod index 46746fe9b..a3a6831ce 100644 --- a/go.mod +++ b/go.mod @@ -41,6 +41,7 @@ require ( golang.org/x/net v0.61.0 golang.org/x/term v0.47.0 golang.org/x/text v0.43.0 + golang.org/x/time v0.16.0 golang.org/x/tools v0.52.0 google.golang.org/genproto/googleapis/rpc v0.0.0-20261005182115-fad411399dd8 google.golang.org/grpc v1.84.0 diff --git a/go.sum b/go.sum index aaaf216de..5f44fe39f 100644 --- a/go.sum +++ b/go.sum @@ -243,6 +243,8 @@ golang.org/x/term v0.47.0 h1:cpyOwQkr2Nzxe2p6KnrTG7Ei2wtzjsuSjhpXXepZ/BM= golang.org/x/term v0.47.0/go.mod h1:efAfAuG6RKABvg1ArGebRkvX6LXul14SM67o2e4mUCY= golang.org/x/text v0.43.0 h1:1QivrlhwAsnMOcqgOdl9my7s9OyZl8O5/UGUTJVXoWc= golang.org/x/text v0.43.0/go.mod h1:hbSIYA/amXcRXTFSZqh8tWHEoqRvvbKPJ2uz/KU+a+s= +golang.org/x/time v0.16.0 h1:vMb6ptszcQMkcwiRTAuNNU50gom6++Q/6gY2hDM6VDE= +golang.org/x/time v0.16.0/go.mod h1:rVKOqvZeKvrDKTQiAHJ7wmwP0RzleSphoEA9RcdLA0s= golang.org/x/tools v0.52.0 h1:ryxg4U63oaUYk7uOLTF37n43f4qgtGvqV4Ioit1BaEs= golang.org/x/tools v0.52.0/go.mod h1:ItcP5UNaVGG+Yc4eIYmaFLNJ4soEopkx5Y2d4lcEurs= gonum.org/v1/gonum v0.17.0 h1:VbpOemQlsSMrYmn7T2OUvQ4dqxQXU+ouZFQsZOx50z4= diff --git a/grpc/codegen/security_scope_test.go b/grpc/codegen/security_scope_test.go index cc10f32bc..a75222114 100644 --- a/grpc/codegen/security_scope_test.go +++ b/grpc/codegen/security_scope_test.go @@ -86,3 +86,14 @@ func grpcRequirementSchemeNames(requirements service.RequirementsData) []string } return names } + +func TestGRPCPerSchemeSecurityProjection(t *testing.T) { + root := RunGRPCDSL(t, testdata.SecurityPerSchemeDSL("grpc")) + endpoint := CreateGRPCServices(root).Get("per_scheme").Endpoint("read") + requirements := endpoint.Method.Requirements + require.Len(t, requirements, 3) + require.Equal(t, []string{"read"}, requirements[0].Schemes[0].RequiredScopes) + require.Equal(t, []string{"write"}, requirements[0].Schemes[1].RequiredScopes) + require.Equal(t, []string{"read"}, requirements[1].Schemes[0].RequiredScopes) + require.Empty(t, requirements[2].Schemes) +} diff --git a/http/codegen/openapi/internal/ir/analyzer.go b/http/codegen/openapi/internal/ir/analyzer.go index a04d720ff..62a6faa8d 100644 --- a/http/codegen/openapi/internal/ir/analyzer.go +++ b/http/codegen/openapi/internal/ir/analyzer.go @@ -218,6 +218,7 @@ func (a *Analyzer) analyzeSchema(attr *expr.AttributeExpr, context string, noref s, note := a.analyzeInlineType(attr, context) a.applySchemaAttributeDetails(s, attr, note, context) + s = a.analyzeCompositionReferences(s, attr, context) if expr.IsNullable(attr) { a.applyNullableSchema(s) } diff --git a/http/codegen/openapi/internal/ir/analyzer_composition.go b/http/codegen/openapi/internal/ir/analyzer_composition.go new file mode 100644 index 000000000..97e6c13c7 --- /dev/null +++ b/http/codegen/openapi/internal/ir/analyzer_composition.go @@ -0,0 +1,28 @@ +package ir + +import "github.com/CaliLuke/loom/expr" + +// analyzeCompositionReferences retains the separately authored base contract +// when an intersection asks for reference identity. The physical object still +// uses the composed fields and constraints; neither layer replaces the other. +func (a *Analyzer) analyzeCompositionReferences(schema *Schema, attribute *expr.AttributeExpr, context string) *Schema { + if !metaBool(attribute.Meta, "openapi:allOf:reference") { + return schema + } + bases := attribute.CompositionBases() + if len(bases) == 0 { + return schema + } + result := &Schema{} + slots := make([]schemaSlot, 0, len(bases)+1) + for _, base := range bases { + contributor := a.analyzeSchemaPositions(&expr.AttributeExpr{Type: base}, context, + expr.ValuePlanNode{}, expr.ValuePlanNode{}) + slots = append(slots, schemaSlot{location: schemaAllOf, index: len(result.AllOf)}) + result.AllOf = append(result.AllOf, contributor) + } + slots = append(slots, schemaSlot{location: schemaAllOf, index: len(result.AllOf)}) + result.AllOf = append(result.AllOf, schema) + a.constructions[result] = schemaConstruction{slots: slots} + return result +} diff --git a/http/codegen/openapi/internal/ir/document.go b/http/codegen/openapi/internal/ir/document.go index 97eb450e4..ac2255e02 100644 --- a/http/codegen/openapi/internal/ir/document.go +++ b/http/codegen/openapi/internal/ir/document.go @@ -43,7 +43,9 @@ func BuildDocument(api *expr.APIExpr, types []expr.UserType, resultTypes []*expr if err := buildDocumentPaths(doc, api, bodyTypes, exampleGenerator, closeObjects, bindings); err != nil { return nil, err } - componentizeDocument(doc) + if err := componentizeDocument(doc); err != nil { + return nil, err + } return doc, nil } @@ -192,10 +194,11 @@ func buildRequestBody(endpointIR *transportir.Endpoint, bodies *EndpointBodies, applyFormFieldEncoding(endpointIR.Request, content[contentType]) } return &RequestBody{ - Description: requestBodyDescription(bodyAttr), - Required: required, - ComponentName: componentMetaValue(bodyAttr, "openapi:component:requestBody"), - Content: content, + ReferenceMetadata: componentMetadata(bodyAttr, "openapi:reference:requestBody"), + Description: requestBodyDescription(bodyAttr), + Required: required, + ComponentName: componentMetaValue(bodyAttr, "openapi:component:requestBody"), + Content: content, Extensions: openapi.MergeExtensions( openapi.ExtensionsFromExpr(bodyAttr.Meta), openapi.ScopedExtensionsFromExpr(bodyAttr.Meta, "requestBody"), @@ -207,6 +210,11 @@ func requestBodyDescription(bodyAttr *expr.AttributeExpr) string { if bodyAttr == nil { return "" } + if len(componentMetadata(bodyAttr, "openapi:reference:requestBody")) > 0 { + if values := componentMetadata(bodyAttr, "openapi:description:requestBody"); len(values) > 0 { + return values[len(values)-1] + } + } if desc := componentMetaValue(bodyAttr, "openapi:description:requestBody"); desc != "" { return strings.TrimSpace(desc) } @@ -287,17 +295,18 @@ func buildResponse( } desc := resp.Description - if desc == "" { + if desc == "" && len(resp.Meta["openapi:reference:response"]) == 0 { desc = fmt.Sprintf("%s response.", http.StatusText(statusCode)) } return &Response{ - Description: desc, - Summary: metaValue(resp.Meta, "openapi:summary"), - OmitDescription: metaBool(resp.Meta, "openapi:description:omit"), - ComponentName: metaValue(resp.Meta, "openapi:component:response"), - Headers: headers, - Content: buildResponseContent(resp, statusCode, bodies, closeObjects, websocketHandshake, mediaSchemas), - Links: buildResponseLinks(resp.Links, currentService), + ReferenceMetadata: resp.Meta["openapi:reference:response"], + Description: desc, + Summary: metaValue(resp.Meta, "openapi:summary"), + OmitDescription: metaBool(resp.Meta, "openapi:description:omit"), + ComponentName: metaValue(resp.Meta, "openapi:component:response"), + Headers: headers, + Content: buildResponseContent(resp, statusCode, bodies, closeObjects, websocketHandshake, mediaSchemas), + Links: buildResponseLinks(resp.Links, currentService), Extensions: openapi.MergeExtensions( openapi.ExtensionsFromExpr(resp.Meta), openapi.ScopedExtensionsFromExpr(resp.Meta, "response"), @@ -436,11 +445,12 @@ func headersFromIR( } headerContext := attributeExampleContext(child, closeObjects, "response-header", headerIR.HTTPName) header := &Header{ - Description: child.Description, - Required: child.IsRequiredNoDefault(headerIR.Name), - AllowReserved: metaBool(child.Meta, "openapi:allowReserved"), - Schema: preparedLocationSchema(child, headerContext, rand, closeObjects, prepared), - Extensions: openapi.ScopedExtensionsFromExpr(child.Meta, "header"), + ReferenceMetadata: componentMetadata(child, "openapi:reference:header"), + Description: child.Description, + Required: child.IsRequiredNoDefault(headerIR.Name), + AllowReserved: metaBool(child.Meta, "openapi:allowReserved"), + Schema: preparedLocationSchema(child, headerContext, rand, closeObjects, prepared), + Extensions: openapi.ScopedExtensionsFromExpr(child.Meta, "header"), } initExamples(header, child, closeObjects, headerIR.Value) headers[headerIR.HTTPName] = &HeaderRef{Value: header} @@ -641,3 +651,18 @@ func preparedLocationSchema(attribute *expr.AttributeExpr, context string, rando } return schema } + +// componentMetadata preserves local presence, including empty values, before +// consulting a named declaration. Reference edge metadata is never combined. +func componentMetadata(attr *expr.AttributeExpr, key string) []string { + if attr == nil { + return nil + } + if values, present := attr.Meta[key]; present { + return values + } + if userType, ok := attr.Type.(expr.UserType); ok { + return userType.Attribute().Meta[key] + } + return nil +} diff --git a/http/codegen/openapi/internal/ir/model.go b/http/codegen/openapi/internal/ir/model.go index c6ac7bb04..b818bc11a 100644 --- a/http/codegen/openapi/internal/ir/model.go +++ b/http/codegen/openapi/internal/ir/model.go @@ -76,12 +76,20 @@ type ( // RequestBodyRef is a request body reference or value. RequestBodyRef struct { + // Description overrides the target description, including an explicit empty string. + Description *string `json:",omitzero"` + // Summary overrides the target summary when supported by that object. + Summary *string `json:",omitzero"` + Ref string Value *RequestBody } // RequestBody describes an IR request body. RequestBody struct { + // ReferenceMetadata retains authored reusable-object edges until component allocation. + ReferenceMetadata []string `json:"-"` + Description string Required bool ComponentName string `json:"-"` @@ -91,7 +99,9 @@ type ( // ResponseRef is a response reference or value. ResponseRef struct { - Ref string + // Summary overrides the referenced summary, including an explicit empty string. + Summary *string `json:",omitzero"` + Ref string // Description overrides the component description, including an explicit empty string. Description *string Value *Response @@ -99,6 +109,9 @@ type ( // Response describes an IR response. Response struct { + // ReferenceMetadata retains authored reusable-object edges until component allocation. + ReferenceMetadata []string `json:"-"` + Description string Summary string OmitDescription bool @@ -137,12 +150,20 @@ type ( // HeaderRef is a header reference or value. HeaderRef struct { + // Description overrides the target description, including an explicit empty string. + Description *string `json:",omitzero"` + // Summary overrides the target summary when supported by that object. + Summary *string `json:",omitzero"` + Ref string Value *Header } // Header describes an IR header. Header struct { + // ReferenceMetadata retains authored reusable-object edges until component allocation. + ReferenceMetadata []string `json:"-"` + Description string Required bool AllowReserved bool diff --git a/http/codegen/openapi/internal/ir/response_allocation_test.go b/http/codegen/openapi/internal/ir/response_allocation_test.go index 2e732498d..78a8f0c3c 100644 --- a/http/codegen/openapi/internal/ir/response_allocation_test.go +++ b/http/codegen/openapi/internal/ir/response_allocation_test.go @@ -83,7 +83,7 @@ func responseAllocationUse(semantic, allocation, base string, forced bool) respo } func responseAllocatedNames(uses []responseComponentUse) ([]string, *responseNameAllocator) { - allocator := newResponseNameAllocator(uses) + allocator := newResponseNameAllocator(uses, nil) names := make([]string, len(uses)) for i, use := range uses { if use.forced || allocator.counts[use.keys.semantic] >= 2 { diff --git a/http/codegen/openapi/internal/ir/response_description_test.go b/http/codegen/openapi/internal/ir/response_description_test.go index 8cc64f704..9593a24d0 100644 --- a/http/codegen/openapi/internal/ir/response_description_test.go +++ b/http/codegen/openapi/internal/ir/response_description_test.go @@ -16,7 +16,7 @@ func TestResponseDescriptionsDoNotSplitComponents(t *testing.T) { right.Value.Description = "second: Second failure.\n\nthird: Third failure." original := *left.Value paths := responseDescriptionPaths(left, right, status, status) - components := componentizeResponses(paths, nil) + components := componentizeResponses(paths, nil, nil) require.Len(t, components, 1) require.Equal(t, left.Ref, right.Ref) require.Equal(t, original.Description, *left.Description) @@ -26,7 +26,7 @@ func TestResponseDescriptionsDoNotSplitComponents(t *testing.T) { require.Equal(t, original.Content, component.Content) require.Nil(t, left.Value) // Already referenced responses are not componentized again. - require.Empty(t, componentizeResponses(paths, nil)) + require.Empty(t, componentizeResponses(paths, nil, nil)) }) } } @@ -71,7 +71,7 @@ func TestResponseDescriptionSharingPreservesOtherDifferences(t *testing.T) { right := responseSchemaValue(&Schema{Type: "string"}) right.Value.Description = "different description" tc.change(right.Value) - components := componentizeResponses(responseDescriptionPaths(left, right, "400", "400"), nil) + components := componentizeResponses(responseDescriptionPaths(left, right, "400", "400"), nil, nil) require.Len(t, components, 2) require.NotEqual(t, left.Ref, right.Ref) }) @@ -81,7 +81,7 @@ func TestResponseDescriptionSharingPreservesOtherDifferences(t *testing.T) { func TestResponseDescriptionsKeepStatusesSeparate(t *testing.T) { left := responseSchemaValue(&Schema{Type: "string"}) right := responseSchemaValue(&Schema{Type: "string"}) - components := componentizeResponses(responseDescriptionPaths(left, right, "400", "503"), nil) + components := componentizeResponses(responseDescriptionPaths(left, right, "400", "503"), nil, nil) require.Len(t, components, 2) require.NotEqual(t, left.Ref, right.Ref) } @@ -92,7 +92,7 @@ func TestResponseDescriptionChangesKeepAutomaticName(t *testing.T) { right := responseSchemaValue(&Schema{Ref: "#/components/schemas/Error"}) left.Value.ComponentName, right.Value.ComponentName = "", "" left.Value.Description, right.Value.Description = description, "other: Other error." - components := componentizeResponses(responseDescriptionPaths(left, right, "503", "503"), nil) + components := componentizeResponses(responseDescriptionPaths(left, right, "503", "503"), nil, nil) require.Len(t, components, 1) return left.Ref, components["ServiceUnavailableError"].Value } @@ -108,7 +108,7 @@ func TestResponseDescriptionEmptyAndSingleUse(t *testing.T) { right := responseSchemaValue(&Schema{Type: "string"}) left.Value.Description = "" right.Value.Description = "OK response." - componentizeResponses(responsePaths(left, right, false), nil) + componentizeResponses(responsePaths(left, right, false), nil, nil) require.NotNil(t, left.Description) require.Empty(t, *left.Description) require.Nil(t, right.Description) @@ -116,7 +116,7 @@ func TestResponseDescriptionEmptyAndSingleUse(t *testing.T) { single := responseSchemaValue(&Schema{Type: "string"}) single.Value.ComponentName = "" paths := responsePaths(single, nil, false) - require.Empty(t, componentizeResponses(paths, nil)) + require.Empty(t, componentizeResponses(paths, nil, nil)) require.NotNil(t, single.Value, "unnamed single-use responses stay inline") } @@ -126,7 +126,7 @@ func TestResponseDescriptionOmissionPreservesFallback(t *testing.T) { left.Value.OmitDescription, right.Value.OmitDescription = true, true left.Value.Description, right.Value.Description = "first fallback", "second fallback" original := left.Value - components := componentizeResponses(responsePaths(left, right, false), nil) + components := componentizeResponses(responsePaths(left, right, false), nil, nil) require.Len(t, components, 2, "3.1 fallback descriptions remain part of omitted-description shapes") require.Nil(t, left.Description) require.Nil(t, right.Description) diff --git a/http/codegen/openapi/internal/ir/response_identity_test.go b/http/codegen/openapi/internal/ir/response_identity_test.go index 4fcb53723..f6de22eba 100644 --- a/http/codegen/openapi/internal/ir/response_identity_test.go +++ b/http/codegen/openapi/internal/ir/response_identity_test.go @@ -15,7 +15,7 @@ func TestResponseRetainsHistoricalSlotAfterSemanticSplit(t *testing.T) { add("/b", "second", nil) add("/c", nil, "later") add("/d", nil, "later") - components := componentizeResponses(paths, map[string]*Schema{"Shared": {Type: "string"}}) + components := componentizeResponses(paths, map[string]*Schema{"Shared": {Type: "string"}}, nil) require.NotEmpty(t, components) // Status/shape naming still reserves the slot before semantic-class splitting. require.Equal(t, "#/components/responses/SharedStatus200Response_b630d408", paths["/c"].Operations["GET"].Responses["200"].Ref) diff --git a/http/codegen/openapi/internal/ir/response_keys_test.go b/http/codegen/openapi/internal/ir/response_keys_test.go index 3477127a9..295edb375 100644 --- a/http/codegen/openapi/internal/ir/response_keys_test.go +++ b/http/codegen/openapi/internal/ir/response_keys_test.go @@ -41,7 +41,7 @@ func TestResponseKeysPreserveReferenceSiblingSemantics(t *testing.T) { require.Equal(t, a.allocation, b.allocation, "historical naming erased these siblings") require.NotEqual(t, a.semantic, b.semantic, "naming identity must not permit semantic sharing") paths := responsePaths(left, right, false) - components := componentizeResponses(paths, schemas) + components := componentizeResponses(paths, schemas, nil) require.Len(t, components, 2) require.NotEqual(t, left.Ref, right.Ref) }) @@ -94,7 +94,7 @@ func TestResponseAllocationIsIndependentOfMapInsertion(t *testing.T) { a := responseSchemaValue(&Schema{Ref: "#/components/schemas/Shared", Example: "first"}) b := responseSchemaValue(&Schema{Ref: "#/components/schemas/Shared", Example: "second"}) paths := responsePaths(a, b, reverse) - return componentizeResponses(paths, map[string]*Schema{"Shared": {Type: "string"}}), paths + return componentizeResponses(paths, map[string]*Schema{"Shared": {Type: "string"}}, nil), paths } a, pathsA := build(false) b, pathsB := build(true) diff --git a/http/codegen/openapi/internal/ir/reusable_authored.go b/http/codegen/openapi/internal/ir/reusable_authored.go new file mode 100644 index 000000000..4ef45729d --- /dev/null +++ b/http/codegen/openapi/internal/ir/reusable_authored.go @@ -0,0 +1,173 @@ +package ir + +import ( + "encoding/json/v2" + "fmt" + + "github.com/CaliLuke/loom/internal/openapiref" +) + +type reusableAdapter[V, R any] struct { + value func(R) *V + metadata func(*V) []string + inline func(*V) R + reference func(openapiref.Object) R + set func(R, openapiref.Object) +} + +var ( + headerReferenceAdapter = reusableAdapter[Header, *HeaderRef]{ + value: func(r *HeaderRef) *Header { + return r.Value + }, + metadata: func(v *Header) []string { + return v.ReferenceMetadata + }, + inline: func(v *Header) *HeaderRef { + return &HeaderRef{Value: v} + }, + reference: func(r openapiref.Object) *HeaderRef { + return &HeaderRef{Ref: r.Ref, Description: r.Description, Summary: r.Summary} + }, + set: func(target *HeaderRef, r openapiref.Object) { + *target = HeaderRef{Ref: r.Ref, Description: r.Description, Summary: r.Summary} + }, + } + bodyReferenceAdapter = reusableAdapter[RequestBody, *RequestBodyRef]{ + value: func(r *RequestBodyRef) *RequestBody { + return r.Value + }, + metadata: func(v *RequestBody) []string { + return v.ReferenceMetadata + }, + inline: func(v *RequestBody) *RequestBodyRef { + return &RequestBodyRef{Value: v} + }, + reference: func(r openapiref.Object) *RequestBodyRef { + return &RequestBodyRef{Ref: r.Ref, Description: r.Description, Summary: r.Summary} + }, + set: func(target *RequestBodyRef, r openapiref.Object) { + *target = RequestBodyRef{Ref: r.Ref, Description: r.Description, Summary: r.Summary} + }, + } + responseReferenceAdapter = reusableAdapter[Response, *ResponseRef]{ + value: func(r *ResponseRef) *Response { + return r.Value + }, + metadata: func(v *Response) []string { + return v.ReferenceMetadata + }, + inline: func(v *Response) *ResponseRef { + return &ResponseRef{Value: v} + }, + reference: func(r openapiref.Object) *ResponseRef { + return &ResponseRef{Ref: r.Ref, Description: r.Description, Summary: r.Summary} + }, + set: func(target *ResponseRef, r openapiref.Object) { + *target = ResponseRef{Ref: r.Ref, Description: r.Description, Summary: r.Summary} + }, + } +) + +// authoredReusableComponents allocates canonical identities before inferred +// components. All three families share edge admission and conflict detection. +func authoredReusableComponents(paths map[string]*PathItem) (map[string]*HeaderRef, map[string]*RequestBodyRef, map[string]*ResponseRef, error) { + headersUses := collectHeaderUsages(paths) + headers := make([]*HeaderRef, len(headersUses)) + for index, use := range headersUses { + headers[index] = use.ref + } + h, err := hoistAuthoredReferences(headers, "headers", headerReferenceAdapter) + if err != nil { + return nil, nil, nil, err + } + bodiesUses := collectRequestBodyUsages(paths) + bodies := make([]*RequestBodyRef, len(bodiesUses)) + for index, use := range bodiesUses { + bodies[index] = use.ref + } + b, err := hoistAuthoredReferences(bodies, "requestBodies", bodyReferenceAdapter) + if err != nil { + return nil, nil, nil, err + } + responsesUses := collectResponseUsages(paths) + responses := make([]*ResponseRef, len(responsesUses)) + for index, use := range responsesUses { + responses[index] = use.ref + } + r, err := hoistAuthoredReferences(responses, "responses", responseReferenceAdapter) + return h, b, r, err +} + +func hoistAuthoredReferences[V, R any](refs []R, kind string, adapter reusableAdapter[V, R]) (map[string]R, error) { + components := make(map[string]R) + hashes := make(map[string]string) + install := func(name string, reference R) error { + hash, err := hashReusableValue(reference) + if err != nil { + return err + } + if previous, exists := hashes[name]; exists && previous != hash { + return fmt.Errorf("conflicting authored %s component %q", kind, name) + } + hashes[name], components[name] = hash, reference + return nil + } + for _, ref := range refs { + value := adapter.value(ref) + metadata := adapter.metadata(value) + if len(metadata) == 0 { + continue + } + if len(metadata) != 1 { + return nil, fmt.Errorf("%s reference metadata requires one edge list", kind) + } + var chain []openapiref.Object + if err := json.Unmarshal([]byte(metadata[0]), &chain, json.RejectUnknownMembers(true)); err != nil { + return nil, fmt.Errorf("invalid %s reference metadata: %w", kind, err) + } + if len(chain) == 0 { + return nil, fmt.Errorf("empty %s reference chain", kind) + } + names := make([]string, len(chain)) + seen := make(map[string]bool) + for i, edge := range chain { + name, err := openapiref.Name(edge.Ref, "#/components/"+kind+"/") + if err != nil { + return nil, fmt.Errorf("invalid %s reference: %w", kind, err) + } + if seen[name] { + return nil, fmt.Errorf("cyclic %s reference %q", kind, name) + } + seen[name], names[i] = true, name + } + if err := install(names[len(names)-1], adapter.inline(value)); err != nil { + return nil, err + } + for i := len(chain) - 1; i > 0; i-- { + if err := install(names[i-1], adapter.reference(chain[i])); err != nil { + return nil, err + } + } + adapter.set(ref, chain[0]) + } + return components, nil +} + +func reservedReusableNames[R any](components map[string]R) map[string]bool { + names := make(map[string]bool, len(components)) + for name := range components { + names[name] = true + } + return names +} + +func mergeReusableComponents[R any](authored, inferred map[string]R) map[string]R { + if len(authored) == 0 { + return inferred + } + for name, value := range inferred { + authored[name] = value + } + return authored +} diff --git a/http/codegen/openapi/internal/ir/reusable_authored_test.go b/http/codegen/openapi/internal/ir/reusable_authored_test.go new file mode 100644 index 000000000..869e9fa2a --- /dev/null +++ b/http/codegen/openapi/internal/ir/reusable_authored_test.go @@ -0,0 +1,69 @@ +package ir + +import ( + "encoding/json/v2" + "testing" + + "github.com/CaliLuke/loom/internal/openapiref" + "github.com/stretchr/testify/require" +) + +func TestAuthoredReusableNamesAreReserved(t *testing.T) { + metadata := func(kind, name string) []string { + wire, err := json.Marshal([]openapiref.Object{{Ref: "#/components/" + kind + "/" + name}}, json.Deterministic(true)) + require.NoError(t, err) + return []string{string(wire)} + } + headerName := headerComponentBase("X-Trace") + authored := &Operation{ + OperationID: "authored", + RequestBody: &RequestBodyRef{Value: &RequestBody{ReferenceMetadata: metadata("requestBodies", "Collision"), Description: "original"}}, + Responses: map[string]*ResponseRef{"200": {Value: &Response{ + ReferenceMetadata: metadata("responses", "Collision"), Description: "original", + Headers: map[string]*HeaderRef{"X-Trace": {Value: &Header{ReferenceMetadata: metadata("headers", headerName), Description: "original", Schema: &Schema{Type: "string"}}}}, + }}}, + } + paths := map[string]*PathItem{"/a": {Operations: map[string]*Operation{"POST": authored}}} + for _, path := range []string{"/b", "/c"} { + paths[path] = &PathItem{Operations: map[string]*Operation{"POST": { + OperationID: path, + RequestBody: &RequestBodyRef{Value: &RequestBody{ComponentName: "Collision", Description: "inferred"}}, + Responses: map[string]*ResponseRef{"200": {Value: &Response{ + ComponentName: "Collision", Description: "inferred", + Headers: map[string]*HeaderRef{"X-Trace": {Value: &Header{Description: "inferred", Schema: &Schema{Type: "integer"}}}}, + }}}, + }}} + } + doc := &Document{Paths: paths} + require.NoError(t, componentizeDocument(doc)) + require.Equal(t, "original", doc.Components.Headers[headerName].Value.Description) + require.Equal(t, "original", doc.Components.RequestBodies["Collision"].Value.Description) + require.Equal(t, "original", doc.Components.Responses["Collision"].Value.Description) + require.Len(t, doc.Components.Headers, 2) + require.Len(t, doc.Components.RequestBodies, 2) + require.Len(t, doc.Components.Responses, 2) + require.NotEqual(t, authored.RequestBody.Ref, paths["/b"].Operations["POST"].RequestBody.Ref) + require.NotEqual(t, authored.Responses["200"].Ref, paths["/b"].Operations["POST"].Responses["200"].Ref) +} + +func TestAuthoredReusableReferenceAdmission(t *testing.T) { + for _, value := range []string{"", `[]`, `[{"$ref":"#/components/headers/Wrong"}]`, `[{"$ref":"#/components/requestBodies/A"},{"$ref":"#/components/requestBodies/A"}]`, `[{"$ref":"#/components/requestBodies/A","unknown":true}]`} { + doc := &Document{Paths: map[string]*PathItem{"/x": {Operations: map[string]*Operation{"POST": { + RequestBody: &RequestBodyRef{Value: &RequestBody{ReferenceMetadata: []string{value}}}, + }}}}} + require.Error(t, componentizeDocument(doc), value) + } +} + +func TestAuthoredReusableConflictingDefinitions(t *testing.T) { + doc := &Document{Paths: map[string]*PathItem{}} + for _, description := range []string{"first", "second"} { + doc.Paths["/"+description] = &PathItem{Operations: map[string]*Operation{"POST": { + RequestBody: &RequestBodyRef{Value: &RequestBody{ + ReferenceMetadata: []string{`[{"$ref":"#/components/requestBodies/Shared"}]`}, + Description: description, + }}, + }}} + } + require.ErrorContains(t, componentizeDocument(doc), `conflicting authored requestBodies component "Shared"`) +} diff --git a/http/codegen/openapi/internal/ir/reusable_components.go b/http/codegen/openapi/internal/ir/reusable_components.go index cf7ae430b..63f6b5614 100644 --- a/http/codegen/openapi/internal/ir/reusable_components.go +++ b/http/codegen/openapi/internal/ir/reusable_components.go @@ -51,6 +51,7 @@ type ( } responseNameAllocator struct { + authored map[string]bool counts map[string]int historical map[string]responseHistoricalSlot reserved map[string]bool @@ -68,18 +69,23 @@ type ( } ) -func componentizeDocument(doc *Document) { +func componentizeDocument(doc *Document) error { if doc == nil { - return + return nil } if doc.Components == nil { doc.Components = &Components{} } doc.Components.Examples = componentizeExamples(doc.Paths) - doc.Components.Headers = componentizeHeaders(doc.Paths) - doc.Components.RequestBodies = componentizeRequestBodies(doc.Paths, doc.Components.Schemas) - doc.Components.Responses = componentizeResponses(doc.Paths, doc.Components.Schemas) + headers, bodies, responses, err := authoredReusableComponents(doc.Paths) + if err != nil { + return err + } + doc.Components.Headers = mergeReusableComponents(headers, componentizeHeaders(doc.Paths, reservedReusableNames(headers))) + doc.Components.RequestBodies = mergeReusableComponents(bodies, componentizeRequestBodies(doc.Paths, doc.Components.Schemas, reservedReusableNames(bodies))) + doc.Components.Responses = mergeReusableComponents(responses, componentizeResponses(doc.Paths, doc.Components.Schemas, reservedReusableNames(responses))) doc.Components.Parameters = componentizeParameters(doc.Paths) + return nil } func componentizeParameters(paths map[string]*PathItem) map[string]*ParameterRef { @@ -146,10 +152,11 @@ func componentizeExamples(paths map[string]*PathItem) map[string]*ExampleRef { example := valueOfExampleRef(usage.ref) return example != nil && strings.TrimSpace(example.ComponentName) != "" }, + nil, ) } -func componentizeHeaders(paths map[string]*PathItem) map[string]*HeaderRef { +func componentizeHeaders(paths map[string]*PathItem, reserved map[string]bool) map[string]*HeaderRef { usages := collectHeaderUsages(paths) return componentizeReusableRefs( usages, @@ -168,10 +175,11 @@ func componentizeHeaders(paths map[string]*PathItem) map[string]*HeaderRef { }, func(value *Header) *HeaderRef { return &HeaderRef{Value: value} }, func(componentUsage[Header, *HeaderRef]) bool { return false }, + reserved, ) } -func componentizeRequestBodies(paths map[string]*PathItem, schemas map[string]*Schema) map[string]*RequestBodyRef { +func componentizeRequestBodies(paths map[string]*PathItem, schemas map[string]*Schema, reserved map[string]bool) map[string]*RequestBodyRef { usages := collectRequestBodyUsages(paths) return componentizeReusableRefs( usages, @@ -196,12 +204,13 @@ func componentizeRequestBodies(paths map[string]*PathItem, schemas map[string]*S func(usage componentUsage[RequestBody, *RequestBodyRef]) bool { return shouldForceComponentizeRequestBody(requestBodyValue(usage.ref)) }, + reserved, ) } -func componentizeResponses(paths map[string]*PathItem, schemas map[string]*Schema) map[string]*ResponseRef { +func componentizeResponses(paths map[string]*PathItem, schemas map[string]*Schema, reserved map[string]bool) map[string]*ResponseRef { uses := collectResponseComponentUses(paths, schemas) - allocator := newResponseNameAllocator(uses) + allocator := newResponseNameAllocator(uses, reserved) components := make(map[string]*ResponseRef) for _, use := range uses { if !use.forced && allocator.counts[use.keys.semantic] < 2 { @@ -251,8 +260,9 @@ func collectResponseComponentUses(paths map[string]*PathItem, schemas map[string return uses } -func newResponseNameAllocator(uses []responseComponentUse) *responseNameAllocator { +func newResponseNameAllocator(uses []responseComponentUse, reserved map[string]bool) *responseNameAllocator { a := &responseNameAllocator{ + authored: reserved, counts: make(map[string]int), historical: make(map[string]responseHistoricalSlot), reserved: make(map[string]bool), @@ -268,6 +278,9 @@ func newResponseNameAllocator(uses []responseComponentUse) *responseNameAllocato a.reserved[use.base] = true } } + for name := range reserved { + a.reserved[name] = true + } a.reserveHistorical(uses) return a } @@ -482,6 +495,9 @@ func (a *responseNameAllocator) reserveHistorical(uses []responseComponentUse) { counts[use.keys.allocation]++ } hashesByName := make(map[string]string) + for name := range a.authored { + hashesByName[name] = "reserved" + } for _, use := range uses { if !use.forced && counts[use.keys.allocation] < 2 { continue diff --git a/http/codegen/openapi/internal/ir/reusable_components_helpers.go b/http/codegen/openapi/internal/ir/reusable_components_helpers.go index 587726dea..4204e42e3 100644 --- a/http/codegen/openapi/internal/ir/reusable_components_helpers.go +++ b/http/codegen/openapi/internal/ir/reusable_components_helpers.go @@ -78,6 +78,7 @@ func componentizeReusableRefs[U any, V any, R any]( setRef func(R, string), newRef func(*V) R, shouldForce func(componentUsage[V, R]) bool, + reserved map[string]bool, ) map[string]R { if len(usages) == 0 { return nil @@ -99,6 +100,9 @@ func componentizeReusableRefs[U any, V any, R any]( components := make(map[string]R) namesByHash := make(map[string]string) hashesByName := make(map[string]string) + for name := range reserved { + hashesByName[name] = "reserved" + } for _, usage := range mapped { value := valueOf(usage.ref) hash, err := hashReusableValue(value) diff --git a/http/codegen/openapi/internal/ir/reusable_components_test.go b/http/codegen/openapi/internal/ir/reusable_components_test.go index 60e14d59a..0838307b9 100644 --- a/http/codegen/openapi/internal/ir/reusable_components_test.go +++ b/http/codegen/openapi/internal/ir/reusable_components_test.go @@ -72,7 +72,7 @@ func TestComponentizeResponsesReusesEquivalentPayloadResponsesAcrossSchemaAliase }, } - componentizeDocument(doc) + require.NoError(t, componentizeDocument(doc)) require.Contains(t, doc.Components.Responses, "SharedShapeStatus200Response") require.Equal(t, "#/components/responses/SharedShapeStatus200Response", doc.Paths["/one"].Operations["GET"].Responses["200"].Ref) @@ -124,7 +124,7 @@ func TestComponentizeRequestBodiesUsesPublicSchemaNameWhenAvailable(t *testing.T }, } - componentizeDocument(doc) + require.NoError(t, componentizeDocument(doc)) require.Contains(t, doc.Components.RequestBodies, "PayloadRequestBody") require.Equal(t, "#/components/requestBodies/PayloadRequestBody", doc.Paths["/one"].Operations["POST"].RequestBody.Ref) @@ -173,7 +173,7 @@ func TestComponentizeRequestBodiesUsesExplicitComponentNameWhenAvailable(t *test }, } - componentizeDocument(doc) + require.NoError(t, componentizeDocument(doc)) require.Contains(t, doc.Components.RequestBodies, "SearchFiltersRequest") require.Equal(t, "#/components/requestBodies/SearchFiltersRequest", doc.Paths["/one"].Operations["POST"].RequestBody.Ref) @@ -205,7 +205,7 @@ func TestComponentizeRequestBodiesHoistsSingleUseExplicitComponentNameAndDescrip }, } - componentizeDocument(doc) + require.NoError(t, componentizeDocument(doc)) require.Contains(t, doc.Components.RequestBodies, "SearchFiltersRequest") require.Equal(t, "#/components/requestBodies/SearchFiltersRequest", doc.Paths["/search"].Operations["POST"].RequestBody.Ref) @@ -238,7 +238,7 @@ func TestComponentizeParametersUsesExplicitComponentNameWhenAvailable(t *testing }, } - componentizeDocument(doc) + require.NoError(t, componentizeDocument(doc)) require.Contains(t, doc.Components.Parameters, "WidgetIDParam") require.Equal(t, "#/components/parameters/WidgetIDParam", doc.Paths["/one/{widgetID}"].Operations["GET"].Parameters[0].Ref) @@ -261,7 +261,7 @@ func TestComponentizeParametersHoistsSingleUseExplicitComponentName(t *testing.T }, } - componentizeDocument(doc) + require.NoError(t, componentizeDocument(doc)) require.Contains(t, doc.Components.Parameters, "WidgetIDParam") require.Equal(t, "#/components/parameters/WidgetIDParam", doc.Paths["/widgets/{widgetID}"].Operations["GET"].Parameters[0].Ref) @@ -303,7 +303,7 @@ func TestComponentizeExamplesUsesExplicitComponentNameWhenAvailable(t *testing.T Components: &Components{}, } - componentizeDocument(doc) + require.NoError(t, componentizeDocument(doc)) require.Contains(t, doc.Components.Examples, "ArtifactThreadExample") exampleRef := doc.Paths["/threads"].Operations["POST"].RequestBody.Value.Content["application/json"].Examples["artifact-thread"] @@ -360,7 +360,7 @@ func TestComponentizeResponsesUsesPublicSchemaNameWhenAvailable(t *testing.T) { }, } - componentizeDocument(doc) + require.NoError(t, componentizeDocument(doc)) require.Contains(t, doc.Components.Responses, "ResultStatus200Response") require.Equal(t, "#/components/responses/ResultStatus200Response", doc.Paths["/one"].Operations["GET"].Responses["200"].Ref) @@ -408,7 +408,7 @@ func TestComponentizeResponsesHoistsSingleUseExplicitComponentName(t *testing.T) }, } - componentizeDocument(doc) + require.NoError(t, componentizeDocument(doc)) require.Contains(t, doc.Components.Responses, "ThreadResponse") require.Equal(t, "#/components/responses/ThreadResponse", doc.Paths["/thread"].Operations["GET"].Responses["200"].Ref) @@ -484,7 +484,7 @@ func TestComponentizeResponsesIncludesLinksInReuseHash(t *testing.T) { }, } - componentizeDocument(doc) + require.NoError(t, componentizeDocument(doc)) require.Contains(t, doc.Components.Responses, "ResultStatus202Response") require.Equal(t, "#/components/responses/ResultStatus202Response", doc.Paths["/one"].Operations["POST"].Responses["202"].Ref) @@ -522,7 +522,7 @@ func assertSharedNoBodyResponseComponent(t *testing.T, method, status, descripti }, } - componentizeDocument(doc) + require.NoError(t, componentizeDocument(doc)) require.Contains(t, doc.Components.Responses, componentName) expectedRef := "#/components/responses/" + componentName diff --git a/http/codegen/openapi/internal/ir/security_bindings.go b/http/codegen/openapi/internal/ir/security_bindings.go index 9a756f9cd..02bb1c1dd 100644 --- a/http/codegen/openapi/internal/ir/security_bindings.go +++ b/http/codegen/openapi/internal/ir/security_bindings.go @@ -147,10 +147,7 @@ func (bindings *securityBindings) requirements(requirements []*expr.SecurityExpr if scheme.Kind == expr.NoKind { continue } - scopes := []string{} - if scheme.Kind == expr.OAuth2Kind { - scopes = append(scopes, requirement.RequiredScopes(scheme)...) - } + scopes := append([]string{}, requirement.RequiredScopes(scheme)...) result[i][bindings.names[securityBindingFor(scheme)]] = scopes } } diff --git a/http/codegen/openapi/v3/builder.go b/http/codegen/openapi/v3/builder.go index cbbe4b72c..9c3960e20 100644 --- a/http/codegen/openapi/v3/builder.go +++ b/http/codegen/openapi/v3/builder.go @@ -538,12 +538,13 @@ func buildSecurityScheme(se *expr.SchemeExpr) *SecurityScheme { } func buildOAuthFlows(se *expr.SchemeExpr) *OAuthFlows { - scopes := make(map[string]string, len(se.Scopes)) - for _, scope := range se.Scopes { - scopes[scope.Name] = scope.Description - } flows := new(OAuthFlows) for _, flow := range se.Flows { + inventory := se.FlowScopes(flow) + scopes := make(map[string]string, len(inventory)) + for _, scope := range inventory { + scopes[scope.Name] = scope.Description + } assignOAuthFlow(flows, flow, scopes) } return flows diff --git a/http/codegen/openapi/v3/effective_composition_test.go b/http/codegen/openapi/v3/effective_composition_test.go index 7b58b7a0d..e18a9a703 100644 --- a/http/codegen/openapi/v3/effective_composition_test.go +++ b/http/codegen/openapi/v3/effective_composition_test.go @@ -50,3 +50,24 @@ func lintEffectiveCompositionArtifacts(t *testing.T, artifacts renderedOpenAPIAr } } } + +func TestReferenceCompositionOpenAPI(t *testing.T) { + artifacts := renderOpenAPIArtifacts(t, testdata.ReferenceCompositionDSL) + spec := decodeOpenAPIJSON(t, artifacts.JSON) + schema := requireComponentSchema(t, spec, "CompositionDerived") + branches, ok := schema["allOf"].([]any) + require.True(t, ok) + require.Len(t, branches, 2) + require.Equal(t, "#/components/schemas/CompositionBase", branches[0].(map[string]any)["$ref"]) + properties := branches[1].(map[string]any)["properties"].(map[string]any) + require.Equal(t, float64(100), properties["count"].(map[string]any)["maximum"]) + base := requireComponentSchema(t, spec, "CompositionBase") + require.NotContains(t, base["properties"].(map[string]any)["count"], "maximum") + for _, object := range []map[string]any{base["properties"].(map[string]any), properties} { + digest := object["digest"].(map[string]any) + require.Equal(t, "string", digest["type"]) + require.Equal(t, "base64", digest["contentEncoding"]) + require.Len(t, digest["anyOf"], 3) + } + lintEffectiveCompositionArtifacts(t, artifacts) +} diff --git a/http/codegen/openapi/v3/ir_adapter.go b/http/codegen/openapi/v3/ir_adapter.go index 4e6671f12..7d6932831 100644 --- a/http/codegen/openapi/v3/ir_adapter.go +++ b/http/codegen/openapi/v3/ir_adapter.go @@ -69,7 +69,7 @@ func requestBodyRefFromIR(body *openapiir.RequestBodyRef) *RequestBodyRef { return nil } if body.Ref != "" { - return &RequestBodyRef{Ref: body.Ref} + return &RequestBodyRef{Ref: body.Ref, Description: body.Description, Summary: body.Summary} } return requestBodyFromIR(body.Value) } @@ -95,7 +95,7 @@ func responseRefFromIR(response *openapiir.ResponseRef) *ResponseRef { return nil } if response.Ref != "" { - return &ResponseRef{Ref: response.Ref, Description: response.Description} + return &ResponseRef{Ref: response.Ref, Description: response.Description, Summary: response.Summary} } return &ResponseRef{Value: responseFromIR(response.Value)} } @@ -177,7 +177,7 @@ func headerRefFromIR(header *openapiir.HeaderRef) *HeaderRef { return nil } if header.Ref != "" { - return &HeaderRef{Ref: header.Ref} + return &HeaderRef{Ref: header.Ref, Description: header.Description, Summary: header.Summary} } return &HeaderRef{Value: &Header{ Description: header.Value.Description, diff --git a/http/codegen/openapi/v3/object_ref.go b/http/codegen/openapi/v3/object_ref.go new file mode 100644 index 000000000..82481a4ed --- /dev/null +++ b/http/codegen/openapi/v3/object_ref.go @@ -0,0 +1,120 @@ +package openapiv3 + +import "encoding/json/v2" + +type objectReference struct { + Ref string `json:"$ref" yaml:"$ref"` + Description *string `json:"description,omitzero" yaml:"description,omitempty"` + Summary *string `json:"summary,omitzero" yaml:"summary,omitempty"` +} + +func serializedObjectReference(ref string, description, summary *string, value any) any { + if ref != "" { + return &objectReference{Ref: ref, Description: description, Summary: summary} + } + return value +} + +func decodeObjectReference[V any](unmarshal func(any) error) (objectReference, *V, error) { + var ref objectReference + if err := unmarshal(&ref); err != nil { + return ref, nil, err + } + var value *V + if ref.Ref == "" { + if err := unmarshal(&value); err != nil { + return ref, nil, err + } + } + return ref, value, nil +} + +// MarshalJSON preserves present reference annotations and inline object content. +func (r *ResponseRef) MarshalJSON() ([]byte, error) { + return json.Marshal(serializedObjectReference(r.Ref, r.Description, r.Summary, r.Value), json.Deterministic(true)) +} + +// MarshalYAML preserves present reference annotations and inline object content. +func (r *ResponseRef) MarshalYAML() (any, error) { + return serializedObjectReference(r.Ref, r.Description, r.Summary, r.Value), nil +} + +// UnmarshalJSON decodes an inline object or a reference with occurrence annotations. +func (r *ResponseRef) UnmarshalJSON(data []byte) error { + return r.UnmarshalYAML(func(value any) error { + return json.Unmarshal(data, value) + }) +} + +// UnmarshalYAML decodes an inline object or a reference with occurrence annotations. +func (r *ResponseRef) UnmarshalYAML(unmarshal func(any) error) error { + ref, value, err := decodeObjectReference[Response](unmarshal) + if err != nil { + return err + } + *r = ResponseRef{Ref: ref.Ref, Value: value} + if ref.Ref != "" { + r.Description, r.Summary = ref.Description, ref.Summary + } + return nil +} + +// MarshalJSON preserves present reference annotations and inline object content. +func (r *HeaderRef) MarshalJSON() ([]byte, error) { + return json.Marshal(serializedObjectReference(r.Ref, r.Description, r.Summary, r.Value), json.Deterministic(true)) +} + +// MarshalYAML preserves present reference annotations and inline object content. +func (r *HeaderRef) MarshalYAML() (any, error) { + return serializedObjectReference(r.Ref, r.Description, r.Summary, r.Value), nil +} + +// UnmarshalJSON decodes an inline object or a reference with occurrence annotations. +func (r *HeaderRef) UnmarshalJSON(data []byte) error { + return r.UnmarshalYAML(func(value any) error { + return json.Unmarshal(data, value) + }) +} + +// UnmarshalYAML decodes an inline object or a reference with occurrence annotations. +func (r *HeaderRef) UnmarshalYAML(unmarshal func(any) error) error { + ref, value, err := decodeObjectReference[Header](unmarshal) + if err != nil { + return err + } + *r = HeaderRef{Ref: ref.Ref, Value: value} + if ref.Ref != "" { + r.Description, r.Summary = ref.Description, ref.Summary + } + return nil +} + +// MarshalJSON preserves present reference annotations and inline object content. +func (r *RequestBodyRef) MarshalJSON() ([]byte, error) { + return json.Marshal(serializedObjectReference(r.Ref, r.Description, r.Summary, r.Value), json.Deterministic(true)) +} + +// MarshalYAML preserves present reference annotations and inline object content. +func (r *RequestBodyRef) MarshalYAML() (any, error) { + return serializedObjectReference(r.Ref, r.Description, r.Summary, r.Value), nil +} + +// UnmarshalJSON decodes an inline object or a reference with occurrence annotations. +func (r *RequestBodyRef) UnmarshalJSON(data []byte) error { + return r.UnmarshalYAML(func(value any) error { + return json.Unmarshal(data, value) + }) +} + +// UnmarshalYAML decodes an inline object or a reference with occurrence annotations. +func (r *RequestBodyRef) UnmarshalYAML(unmarshal func(any) error) error { + ref, value, err := decodeObjectReference[RequestBody](unmarshal) + if err != nil { + return err + } + *r = RequestBodyRef{Ref: ref.Ref, Value: value} + if ref.Ref != "" { + r.Description, r.Summary = ref.Description, ref.Summary + } + return nil +} diff --git a/http/codegen/openapi/v3/object_ref_test.go b/http/codegen/openapi/v3/object_ref_test.go new file mode 100644 index 000000000..ef58c3649 --- /dev/null +++ b/http/codegen/openapi/v3/object_ref_test.go @@ -0,0 +1,59 @@ +package openapiv3 + +import ( + "encoding/json/v2" + "testing" + + "github.com/stretchr/testify/require" + "go.yaml.in/yaml/v3" +) + +func TestReusableObjectReferenceAnnotations(t *testing.T) { + for _, tc := range []struct { + name string + value any + fresh func() any + }{ + {"body", &RequestBodyRef{Ref: "#/components/requestBodies/Input", Description: new(""), Summary: new("hint")}, func() any { + return &RequestBodyRef{} + }}, + {"header", &HeaderRef{Ref: "#/components/headers/Trace", Description: new("local"), Summary: new("")}, func() any { + return &HeaderRef{} + }}, + {"response", &ResponseRef{Ref: "#/components/responses/Result", Description: new(""), Summary: new("")}, func() any { + return &ResponseRef{} + }}, + {"inherited-body", &RequestBodyRef{Ref: "#/components/requestBodies/Input"}, func() any { + return &RequestBodyRef{} + }}, + {"inherited-header", &HeaderRef{Ref: "#/components/headers/Trace"}, func() any { + return &HeaderRef{} + }}, + {"inline-body", &RequestBodyRef{Value: &RequestBody{Description: "body"}}, func() any { + return &RequestBodyRef{Ref: "stale", Description: new("stale"), Summary: new("stale")} + }}, + {"inline-header", &HeaderRef{Value: &Header{Description: "header"}}, func() any { + return &HeaderRef{Ref: "stale", Description: new("stale"), Summary: new("stale")} + }}, + } { + t.Run(tc.name, func(t *testing.T) { + for _, format := range []string{"json", "yaml"} { + t.Run(format, func(t *testing.T) { + got := tc.fresh() + var wire []byte + var err error + if format == "json" { + wire, err = json.Marshal(tc.value, json.Deterministic(true)) + require.NoError(t, err) + require.NoError(t, json.Unmarshal(wire, got)) + } else { + wire, err = yaml.Marshal(tc.value) + require.NoError(t, err) + require.NoError(t, yaml.Unmarshal(wire, got)) + } + require.Equal(t, tc.value, got) + }) + } + }) + } +} diff --git a/http/codegen/openapi/v3/ref.go b/http/codegen/openapi/v3/ref.go index f59cd1d33..c2b8894af 100644 --- a/http/codegen/openapi/v3/ref.go +++ b/http/codegen/openapi/v3/ref.go @@ -14,6 +14,8 @@ type ( // ResponseRef holds an inline response or an OpenAPI 3.1/3.2 Reference Object. ResponseRef struct { + // Summary overrides the target summary, including an explicit empty string. + Summary *string // Ref identifies the response component; when set it takes precedence over Value. Ref string // Description overrides the referenced description. Nil inherits it; a pointer @@ -26,6 +28,11 @@ type ( // HeaderRef represents an OpenAPI reference to a Header object as defined in // https://github.com/OAI/OpenAPI-Specification/blob/master/versions/3.0.3.md#referenceObject HeaderRef struct { + // Description overrides the target description, including an explicit empty string. + Description *string + // Summary overrides the target summary when supported by the target object. + Summary *string + Ref string Value *Header } @@ -57,6 +64,11 @@ type ( // RequestBodyRef represents an OpenAPI reference to a RequestBody object as defined in // https://github.com/OAI/OpenAPI-Specification/blob/master/versions/3.0.3.md#referenceObject RequestBodyRef struct { + // Description overrides the target description, including an explicit empty string. + Description *string + // Summary overrides the target summary when supported by the target object. + Summary *string + Ref string Value *RequestBody } @@ -76,29 +88,23 @@ type ( ) func (r *ParameterRef) MarshalJSON() ([]byte, error) { return marshalJSONRef(r.Ref, r.Value) } -func (r *HeaderRef) MarshalJSON() ([]byte, error) { return marshalJSONRef(r.Ref, r.Value) } func (r *CallbackRef) MarshalJSON() ([]byte, error) { return marshalJSONRef(r.Ref, r.Value) } func (r *ExampleRef) MarshalJSON() ([]byte, error) { return marshalJSONRef(r.Ref, r.Value) } func (r *LinkRef) MarshalJSON() ([]byte, error) { return marshalJSONRef(r.Ref, r.Value) } -func (r *RequestBodyRef) MarshalJSON() ([]byte, error) { return marshalJSONRef(r.Ref, r.Value) } func (r *SecuritySchemeRef) MarshalJSON() ([]byte, error) { return marshalJSONRef(r.Ref, r.Value) } func (r *MediaTypeRef) MarshalJSON() ([]byte, error) { return marshalJSONRef(r.Ref, r.Value) } func (r *ParameterRef) MarshalYAML() (any, error) { return marshalYAMLRef(r.Ref, r.Value) } -func (r *HeaderRef) MarshalYAML() (any, error) { return marshalYAMLRef(r.Ref, r.Value) } func (r *CallbackRef) MarshalYAML() (any, error) { return marshalYAMLRef(r.Ref, r.Value) } func (r *ExampleRef) MarshalYAML() (any, error) { return marshalYAMLRef(r.Ref, r.Value) } func (r *LinkRef) MarshalYAML() (any, error) { return marshalYAMLRef(r.Ref, r.Value) } -func (r *RequestBodyRef) MarshalYAML() (any, error) { return marshalYAMLRef(r.Ref, r.Value) } func (r *SecuritySchemeRef) MarshalYAML() (any, error) { return marshalYAMLRef(r.Ref, r.Value) } func (r *MediaTypeRef) MarshalYAML() (any, error) { return marshalYAMLRef(r.Ref, r.Value) } -func (r *ParameterRef) UnmarshalJSON(d []byte) error { return unmarshalJSONRef(d, &r.Ref, &r.Value) } -func (r *HeaderRef) UnmarshalJSON(d []byte) error { return unmarshalJSONRef(d, &r.Ref, &r.Value) } -func (r *CallbackRef) UnmarshalJSON(d []byte) error { return unmarshalJSONRef(d, &r.Ref, &r.Value) } -func (r *ExampleRef) UnmarshalJSON(d []byte) error { return unmarshalJSONRef(d, &r.Ref, &r.Value) } -func (r *LinkRef) UnmarshalJSON(d []byte) error { return unmarshalJSONRef(d, &r.Ref, &r.Value) } -func (r *RequestBodyRef) UnmarshalJSON(d []byte) error { return unmarshalJSONRef(d, &r.Ref, &r.Value) } +func (r *ParameterRef) UnmarshalJSON(d []byte) error { return unmarshalJSONRef(d, &r.Ref, &r.Value) } +func (r *CallbackRef) UnmarshalJSON(d []byte) error { return unmarshalJSONRef(d, &r.Ref, &r.Value) } +func (r *ExampleRef) UnmarshalJSON(d []byte) error { return unmarshalJSONRef(d, &r.Ref, &r.Value) } +func (r *LinkRef) UnmarshalJSON(d []byte) error { return unmarshalJSONRef(d, &r.Ref, &r.Value) } func (r *SecuritySchemeRef) UnmarshalJSON(d []byte) error { return unmarshalJSONRef(d, &r.Ref, &r.Value) } @@ -109,9 +115,6 @@ func (r *MediaTypeRef) UnmarshalJSON(d []byte) error { func (r *ParameterRef) UnmarshalYAML(u func(any) error) error { return unmarshalYAMLRef(u, &r.Ref, &r.Value) } -func (r *HeaderRef) UnmarshalYAML(u func(any) error) error { - return unmarshalYAMLRef(u, &r.Ref, &r.Value) -} func (r *CallbackRef) UnmarshalYAML(u func(any) error) error { return unmarshalYAMLRef(u, &r.Ref, &r.Value) } @@ -121,9 +124,6 @@ func (r *ExampleRef) UnmarshalYAML(u func(any) error) error { func (r *LinkRef) UnmarshalYAML(u func(any) error) error { return unmarshalYAMLRef(u, &r.Ref, &r.Value) } -func (r *RequestBodyRef) UnmarshalYAML(u func(any) error) error { - return unmarshalYAMLRef(u, &r.Ref, &r.Value) -} func (r *SecuritySchemeRef) UnmarshalYAML(u func(any) error) error { return unmarshalYAMLRef(u, &r.Ref, &r.Value) } diff --git a/http/codegen/openapi/v3/response_ref.go b/http/codegen/openapi/v3/response_ref.go deleted file mode 100644 index 46698d332..000000000 --- a/http/codegen/openapi/v3/response_ref.go +++ /dev/null @@ -1,57 +0,0 @@ -package openapiv3 - -import "encoding/json/v2" - -type responseReference struct { - Ref string `json:"$ref" yaml:"$ref"` - Description *string `json:"description,omitzero" yaml:"description,omitempty"` -} - -// MarshalJSON preserves the description override on a response reference. -func (r *ResponseRef) MarshalJSON() ([]byte, error) { - return json.Marshal(r.serialized(), json.Deterministic(true)) -} - -// MarshalYAML preserves the description override on a response reference. -func (r *ResponseRef) MarshalYAML() (any, error) { - return r.serialized(), nil -} - -// UnmarshalJSON decodes an inline response or a reference with its description. -func (r *ResponseRef) UnmarshalJSON(data []byte) error { - var ref responseReference - if err := json.Unmarshal(data, &ref); err != nil { - return err - } - value := ResponseRef{Ref: ref.Ref} - if ref.Ref != "" { - value.Description = ref.Description - } else if err := json.Unmarshal(data, &value.Value); err != nil { - return err - } - *r = value - return nil -} - -// UnmarshalYAML decodes an inline response or a reference with its description. -func (r *ResponseRef) UnmarshalYAML(unmarshal func(any) error) error { - var ref responseReference - if err := unmarshal(&ref); err != nil { - return err - } - value := ResponseRef{Ref: ref.Ref} - if ref.Ref != "" { - value.Description = ref.Description - } else if err := unmarshal(&value.Value); err != nil { - return err - } - *r = value - return nil -} - -func (r *ResponseRef) serialized() any { - if r.Ref != "" { - return &responseReference{Ref: r.Ref, Description: r.Description} - } - return r.Value -} diff --git a/http/codegen/openapi/v3/reusable_reference_test.go b/http/codegen/openapi/v3/reusable_reference_test.go new file mode 100644 index 000000000..421430ac4 --- /dev/null +++ b/http/codegen/openapi/v3/reusable_reference_test.go @@ -0,0 +1,23 @@ +package openapiv3_test + +import ( + "testing" + + "github.com/CaliLuke/loom/http/codegen/testdata" + "github.com/stretchr/testify/require" +) + +func TestReusableReferenceComponentsOpenAPI(t *testing.T) { + artifacts := renderOpenAPIArtifacts(t, testdata.OpenAPIReferencedComponentsDSL) + spec := decodeOpenAPIJSON(t, artifacts.JSON) + components := requireMap(t, spec["components"], "components") + bodies := requireMap(t, components["requestBodies"], "requestBodies") + require.Contains(t, bodies, "SearchInput") + require.Equal(t, "Search filters.", requireMap(t, bodies["SearchFiltersRequest"], "body")["description"]) + responses := requireMap(t, components["responses"], "responses") + require.Equal(t, "#/components/responses/SearchResults", requireMap(t, responses["Results"], "response")["$ref"]) + require.Equal(t, "Search results.", requireMap(t, responses["SearchResults"], "response")["description"]) + headers := requireMap(t, components["headers"], "headers") + require.Equal(t, "Trace identifier.", requireMap(t, headers["Trace"], "header")["description"]) + lintEffectiveCompositionArtifacts(t, artifacts) +} diff --git a/http/codegen/openapi/v3/security_required_scopes_test.go b/http/codegen/openapi/v3/security_required_scopes_test.go index 644531c3e..6da350532 100644 --- a/http/codegen/openapi/v3/security_required_scopes_test.go +++ b/http/codegen/openapi/v3/security_required_scopes_test.go @@ -1,6 +1,8 @@ package openapiv3_test import ( + "os" + "path/filepath" "testing" "github.com/stretchr/testify/require" @@ -31,3 +33,29 @@ func TestRenderedOAuthRequiredScopeOwnership(t *testing.T) { }) } } + +func TestRenderedPerSchemeRequirements(t *testing.T) { + var vacuum contractVacuum + if os.Getenv("LOOM_OPENAPI_CONTRACT") != "" { + vacuum = installContractVacuum(t) + } + for _, version := range []struct{ target, want string }{ + {"3.1", openapiv3.OpenAPICompatibilityVersion}, {"3.2", openapiv3.OpenAPIVersion}, + } { + t.Run(version.target, func(t *testing.T) { + artifacts := renderOpenAPIArtifactsForVersion(t, servicetestdata.SecurityRequiredScopesDSL, version.target, version.want) + spec := decodeOpenAPIJSON(t, artifacts.JSON) + operation := spec["paths"].(map[string]any)["/isolated"].(map[string]any)["get"].(map[string]any) + require.Equal(t, []any{map[string]any{"oauth": []any{"shared"}, "second": []any{"oauth"}, "key": []any{"admin"}}}, operation["security"]) + flows := spec["components"].(map[string]any)["securitySchemes"].(map[string]any)["oauth"].(map[string]any)["flows"].(map[string]any) + require.Equal(t, map[string]any{"oauth": "User consent", "shared": "User grant"}, flows["authorizationCode"].(map[string]any)["scopes"]) + require.Equal(t, map[string]any{"machine": "Machine grant", "shared": "Machine permission"}, flows["clientCredentials"].(map[string]any)["scopes"]) + require.Empty(t, flows["implicit"].(map[string]any)["scopes"]) + if os.Getenv("LOOM_OPENAPI_CONTRACT") != "" { + path := filepath.Join(t.TempDir(), "openapi.yaml") + require.NoError(t, os.WriteFile(path, artifacts.YAML, 0o600)) + require.NoError(t, vacuum.lint(path)) + } + }) + } +} diff --git a/http/codegen/openapi/v3/testdata/golden/security_file0.golden b/http/codegen/openapi/v3/testdata/golden/security_file0.golden index edd7df5ed..6b03028ec 100644 --- a/http/codegen/openapi/v3/testdata/golden/security_file0.golden +++ b/http/codegen/openapi/v3/testdata/golden/security_file0.golden @@ -71,7 +71,9 @@ { "api_key_query_k": [], "basic": [], - "jwt_header_X-Authorization": [] + "jwt_header_X-Authorization": [ + "api:read" + ] } ], "summary": "testEndpointA testService", diff --git a/http/codegen/openapi/v3/testdata/golden/security_file1.golden b/http/codegen/openapi/v3/testdata/golden/security_file1.golden index e1a38bc8e..b137f7fc5 100644 --- a/http/codegen/openapi/v3/testdata/golden/security_file1.golden +++ b/http/codegen/openapi/v3/testdata/golden/security_file1.golden @@ -20,7 +20,8 @@ paths: security: - api_key_query_k: [] basic: [] - jwt_header_X-Authorization: [] + jwt_header_X-Authorization: + - api:read post: tags: - testService diff --git a/http/codegen/openapi/v3/untagged_arrays_test.go b/http/codegen/openapi/v3/untagged_arrays_test.go index 29f99155d..d9077324c 100644 --- a/http/codegen/openapi/v3/untagged_arrays_test.go +++ b/http/codegen/openapi/v3/untagged_arrays_test.go @@ -68,6 +68,81 @@ func TestUntaggedArrayObjectContract(t *testing.T) { } } require.True(t, foundRecord, "missing nullable enum record") + foundMap := false + for name, raw := range schemas { + schema := raw.(map[string]any) + properties, ok := schema["properties"].(map[string]any) + if !ok || properties["lookup"] == nil { + continue + } + foundMap = true + lookup := properties["lookup"].(map[string]any) + require.Equal(t, "object", lookup["type"]) + require.IsType(t, map[string]any{}, lookup["additionalProperties"]) + root := map[string]any{"components": spec["components"], "$ref": "#/components/schemas/" + name} + for _, value := range []struct { + wire string + want bool + }{ + {"{\"total\":0,\"properties\":{}}", true}, + {"{\"total\":1,\"properties\":{\"a\":[1,null,{\"nested\":true}]},\"labels\":{\"first\":\"ok\"},\"lookup\":{\"item\":{\"name\":\"ok\",\"label\":null}}}", true}, + {"{\"total\":0,\"nested\":{\"outer\":{\"inner\":[{\"name\":\"ok\"}]}}}", true}, + {"{\"total\":0,\"nullable\":null}", true}, + {"{\"total\":0,\"nullable\":{\"a\":null,\"b\":{\"name\":\"ok\"}}}", true}, + {"{\"total\":0,\"tree\":{\"name\":\"root\",\"children\":{\"leaf\":{\"name\":\"leaf\"}}}}", true}, + {"{\"total\":0,\"properties\":null}", false}, + {"{\"total\":0,\"properties\":[]}", false}, + {"{\"total\":0,\"labels\":{}}", false}, + {"{\"total\":0,\"labels\":{\"a\":\"ok\",\"b\":\"ok\",\"c\":\"ok\"}}", false}, + {"{\"total\":0,\"labels\":{\"a\":\"x\"}}", false}, + {"{\"total\":0,\"lookup\":{\"a\":{\"name\":\"ok\"},\"b\":{\"name\":\"x\"}}}", false}, + {"{\"total\":0,\"lookup\":{\"a\":null}}", false}, + {"{\"total\":0,\"lookup\":{\"a\":{\"name\":\"ok\",\"extra\":1}}}", false}, + {"{\"total\":0,\"nested\":{\"outer\":{\"inner\":[{\"name\":\"x\"}]}}}", false}, + {"{\"total\":0,\"tree\":{\"name\":\"root\",\"children\":{\"leaf\":{}}}}", false}, + {"{\"total\":0,\"nullable\":{\"a\":{\"name\":\"x\"}}}", false}, + } { + result, err := gojsonschema.Validate(gojsonschema.NewGoLoader(root), gojsonschema.NewStringLoader(value.wire)) + require.NoError(t, err) + require.Equal(t, value.want, result.Valid(), "%s: %s", name, value.wire) + } + } + require.True(t, foundMap, "missing map-valued page") + foundNested := false + for name, raw := range schemas { + schema := raw.(map[string]any) + properties, ok := schema["properties"].(map[string]any) + if !ok || properties["entries"] == nil { + continue + } + foundNested = true + root := map[string]any{"components": spec["components"], "$ref": "#/components/schemas/" + name} + for _, value := range []struct { + wire string + want bool + }{ + {"{\"total\":0,\"entries\":[]}", true}, + {"{\"total\":2,\"entries\":[{\"name\":\"ok\"},{\"group\":\"team\"}]}", true}, + {"{\"total\":1,\"entries\":[{\"group\":\"team\",\"members\":[{\"name\":\"ok\"},{\"group\":\"nested\",\"members\":[]}]}]}", true}, + {"{\"total\":0,\"single\":null}", true}, + {"{\"total\":0,\"single\":{\"name\":\"ok\"}}", true}, + {"{\"total\":0,\"entry_lookup\":{\"a\":{\"name\":\"ok\"},\"b\":{\"group\":\"team\"}}}", true}, + {"{\"total\":1,\"entries\":[{}]}", false}, + {"{\"total\":1,\"entries\":[{\"name\":\"ok\",\"group\":\"team\"}]}", false}, + {"{\"total\":1,\"entries\":[null]}", false}, + {"{\"total\":1,\"entries\":[{\"name\":\"x\"}]}", false}, + {"{\"total\":1,\"entries\":[{\"group\":\"team\",\"members\":[{}]}]}", false}, + {"{\"total\":0,\"single\":{}}", false}, + {"{\"total\":0,\"entry_lookup\":{\"a\":{}}}", false}, + {"{\"total\":0,\"overlap\":{}}", false}, + } { + result, err := gojsonschema.Validate(gojsonschema.NewGoLoader(root), gojsonschema.NewStringLoader(value.wire)) + require.NoError(t, err) + require.Equal(t, value.want, result.Valid(), "%s: %s", name, value.wire) + } + } + require.True(t, foundNested, "missing nested union page") + if os.Getenv("LOOM_OPENAPI_CONTRACT") != "" { dir := t.TempDir() path := filepath.Join(dir, "openapi.yaml") diff --git a/http/codegen/openapi_operation_security_test.go b/http/codegen/openapi_operation_security_test.go index c296802e0..8d8dc235a 100644 --- a/http/codegen/openapi_operation_security_test.go +++ b/http/codegen/openapi_operation_security_test.go @@ -48,7 +48,7 @@ func TestOpenAPIOperationSecurityRequirements(t *testing.T) { require.Empty(t, security.([]any)) }) - t.Run("HTTP bearer requirements do not publish scopes", func(t *testing.T) { + t.Run("HTTP bearer requirements publish application roles", func(t *testing.T) { root := RunHTTPDSL(t, scopedBearerOperationDSL) spec := renderOpenAPIJSON(t, openapiv3.Files, root) @@ -58,7 +58,7 @@ func TestOpenAPIOperationSecurityRequirements(t *testing.T) { security, ok := op["security"].([]any) require.True(t, ok) require.Len(t, security, 1) - require.Equal(t, map[string]any{"jwt": []any{}}, security[0]) + require.Equal(t, map[string]any{"jwt": []any{"api:read"}}, security[0]) }) t.Run("service session security applies to secured meal planner operations only", func(t *testing.T) { diff --git a/http/codegen/security_scope_test.go b/http/codegen/security_scope_test.go index 86d0908f9..b4758ca05 100644 --- a/http/codegen/security_scope_test.go +++ b/http/codegen/security_scope_test.go @@ -456,3 +456,14 @@ func assertUnauthorizedProblem(t *testing.T, rec *httptest.ResponseRecorder) { } } ` + +func TestHTTPPerSchemeSecurityProjection(t *testing.T) { + root := RunHTTPDSL(t, testdata.SecurityPerSchemeDSL("http")) + endpoint := CreateHTTPServices(root).Get("per_scheme").Endpoint("read") + requirements := endpoint.Method.Requirements + require.Len(t, requirements, 3) + require.Equal(t, []string{"read"}, requirements[0].Schemes[0].RequiredScopes) + require.Equal(t, []string{"write"}, requirements[0].Schemes[1].RequiredScopes) + require.Equal(t, []string{"read"}, requirements[1].Schemes[0].RequiredScopes) + require.Empty(t, requirements[2].Schemes) +} diff --git a/http/codegen/testdata/effective_composition_dsl.go b/http/codegen/testdata/effective_composition_dsl.go index f56bdccdb..b24b05953 100644 --- a/http/codegen/testdata/effective_composition_dsl.go +++ b/http/codegen/testdata/effective_composition_dsl.go @@ -4,40 +4,56 @@ import . "github.com/CaliLuke/loom/dsl" // EffectiveCompositionDSL exercises field refinement through both HTTP and // gRPC: inherited requiredness, numeric conjunction and an authored default. -var EffectiveCompositionDSL = func() { - base := Type("CompositionBase", func() { - Field(1, "count", Int, func() { - Minimum(0) - }) - Field(2, "label", String, func() { - MinLength(1) - }) - Field(3, "retries", Int, func() { - Minimum(0) - Default(2) - }) - Required("count", "label") - }) - derived := Type("CompositionDerived", func() { - Extend(base) - Field(1, "count", Int, func() { - Maximum(100) - }) - Field(3, "retries", Int, func() { - Maximum(10) - Default(5) +var EffectiveCompositionDSL = effectiveCompositionDSL(false) + +// ReferenceCompositionDSL preserves the named contract contributor while +// exercising the same runtime intersection as EffectiveCompositionDSL. +var ReferenceCompositionDSL = effectiveCompositionDSL(true) + +func effectiveCompositionDSL(reference bool) func() { + return func() { + base := Type("CompositionBase", func() { + Field(1, "count", Int, func() { + Minimum(0) + }) + Field(2, "label", String, func() { + MinLength(1) + }) + Field(3, "retries", Int, func() { + Minimum(0) + Default(2) + }) + if reference { + Field(4, "digest", Bytes, func() { + MinLength(2) + }) + } + Required("count", "label") }) - }) - Service("composition", func() { - Method("read", func() { - Payload(derived) - Result(derived) - HTTP(func() { - POST("/read") + derived := Type("CompositionDerived", func() { + if reference { + Meta("openapi:allOf:reference", "true") + } + Extend(base) + Field(1, "count", Int, func() { + Maximum(100) }) - GRPC(func() { + Field(3, "retries", Int, func() { + Maximum(10) + Default(5) + }) + }) + Service("composition", func() { + Method("read", func() { + Payload(derived) + Result(derived) + HTTP(func() { + POST("/read") + }) + GRPC(func() { + }) }) }) - }) + } } diff --git a/http/codegen/testdata/openapi_dsls_02.go b/http/codegen/testdata/openapi_dsls_02.go index cd51d8752..0a55ad5ea 100644 --- a/http/codegen/testdata/openapi_dsls_02.go +++ b/http/codegen/testdata/openapi_dsls_02.go @@ -1,50 +1,71 @@ package testdata -import . "github.com/CaliLuke/loom/dsl" +import ( + . "github.com/CaliLuke/loom/dsl" + "github.com/CaliLuke/loom/expr" +) -var OpenAPIExplicitReusableComponentNamesDSL = func() { - var SearchFilters = Type("SearchFilters", func() { - Meta("openapi:component:requestBody", "SearchFiltersRequest") - Attribute("query", String, func() { - Example("soup") - }) - Required("query") - }) +var OpenAPIExplicitReusableComponentNamesDSL = openAPIExplicitReusableComponentNamesDSL(false) - Service("componentNames", func() { - Method("searchRecipes", func() { - Payload(func() { - Attribute("widgetID", String, func() { - Meta("openapi:component:parameter", "WidgetIDParam") - Example("widget-123") - }) - Attribute("body", SearchFilters) - Required("widgetID", "body") - }) - Result(String) - HTTP(func() { - POST("/recipes/{widgetID}/search") - Param("widgetID") - Body("body") - Response(StatusOK) +// OpenAPIReferencedComponentsDSL exercises imported reusable transport identities. +var OpenAPIReferencedComponentsDSL = openAPIExplicitReusableComponentNamesDSL(true) + +func openAPIExplicitReusableComponentNamesDSL(references bool) func() { + return func() { + var SearchFilters = Type("SearchFilters", func() { + Meta("openapi:component:requestBody", "SearchFiltersRequest") + if references { + Meta("openapi:reference:requestBody", `[{"$ref":"#/components/requestBodies/SearchInput","description":""},{"$ref":"#/components/requestBodies/SearchFiltersRequest"}]`) + Meta("openapi:description:requestBody", "Search filters.") + } + Attribute("query", String, func() { + Example("soup") }) + Required("query") }) - Method("searchGadgets", func() { - Payload(func() { - Attribute("widgetID", String, func() { - Meta("openapi:component:parameter", "WidgetIDParam") - Example("widget-123") + Service("componentNames", func() { + explicitReusableComponentMethod("searchRecipes", "/recipes/{widgetID}/search", SearchFilters, references) + explicitReusableComponentMethod("searchGadgets", "/gadgets/{widgetID}/search", SearchFilters, references) + }) + } +} + +func explicitReusableComponentMethod(name, path string, filters expr.UserType, references bool) { + Method(name, func() { + Payload(func() { + Attribute("widgetID", String, func() { + Meta("openapi:component:parameter", "WidgetIDParam") + Example("widget-123") + }) + Attribute("body", filters) + Required("widgetID", "body") + }) + if references { + Result(func() { + Attribute("value", String, func() { + Example("found") + }) + Attribute("trace", String, "Trace identifier.", func() { + Example("trace-123") + Meta("openapi:reference:header", `[{"$ref":"#/components/headers/Trace","description":""}]`) }) - Attribute("body", SearchFilters) - Required("widgetID", "body") + Required("value", "trace") }) + } else { Result(String) - HTTP(func() { - POST("/gadgets/{widgetID}/search") - Param("widgetID") - Body("body") - Response(StatusOK) + } + HTTP(func() { + POST(path) + Param("widgetID") + Body("body") + Response(StatusOK, func() { + if references { + Description("Search results.") + Meta("openapi:reference:response", `[{"$ref":"#/components/responses/Results","summary":"Search"},{"$ref":"#/components/responses/SearchResults"}]`) + Header("trace:X-Trace") + Body("value") + } }) }) }) diff --git a/http/codegen/testdata/untagged_arrays_dsl.go b/http/codegen/testdata/untagged_arrays_dsl.go index c5de41326..c125ccbdd 100644 --- a/http/codegen/testdata/untagged_arrays_dsl.go +++ b/http/codegen/testdata/untagged_arrays_dsl.go @@ -23,10 +23,57 @@ var UntaggedArraysDSL = func() { Required("name") Meta("openapi:additionalProperties", "false") }) + group := Type("Group", func() { + Attribute("group", String) + Attribute("members", ArrayOf("Entry")) + Required("group") + }) + entry := Type("Entry", OneOf(item, group), Untagged) + structured := Type("Structured", func() { + Attribute("row", entry) + Required("row") + }) + extraRow := Type("ExtraRow", func() { + Attribute("group", String) + Attribute("extra", Boolean) + Required("group", "extra") + }) + raw := Type("Raw", func() { + Attribute("row", extraRow) + Required("row") + }) + fallback := Type("Fallback", OneOf(structured, raw), Untagged) + labels := Type("Labels", MapOf(String, String), func() { + MinLength(1) + MaxLength(2) + Elem(func() { + MinLength(2) + }) + }) + tree := Type("Tree", func() { + Attribute("name", String) + Attribute("children", MapOf(String, "Tree")) + Required("name") + }) items := Type("Items", ArrayOf(item)) page := Type("Page", func() { Attribute("items", ArrayOf(item)) + Attribute("entries", ArrayOf(entry)) + Attribute("single", entry, Nullable) + Attribute("overlap", "Overlap") + Attribute("entry_lookup", MapOf(String, entry)) Attribute("total", Int) + Attribute("properties", MapOf(String, Any)) + Attribute("labels", labels) + Attribute("lookup", MapOf(String, item)) + Attribute("nested", MapOf(String, MapOf(String, ArrayOf(item)))) + Attribute("tree", tree) + Attribute("nullable", MapOf(String, item), func() { + Nullable() + Elem(func() { + Nullable() + }) + }) Required("total") }) result := Type("Result", OneOf(items, page), func() { @@ -49,9 +96,11 @@ var UntaggedArraysDSL = func() { }) left := Type("Left", func() { Attribute("value", String) + Attribute("mapping", MapOf(String, String)) }) right := Type("Right", func() { Attribute("value", String) + Attribute("mapping", MapOf(String, String)) }) overlap := Type("Overlap", OneOf(left, right), func() { Untagged() @@ -67,6 +116,12 @@ var UntaggedArraysDSL = func() { Untagged() }) Service("listing", func() { + Method("fallback", func() { + Result(fallback) + HTTP(func() { + GET("/fallback") + }) + }) Method("record", func() { Result(recordOrPage) HTTP(func() { GET("/record") }) diff --git a/http/codegen/testdata/untagged_arrays_harness.go b/http/codegen/testdata/untagged_arrays_harness.go index 55ac1b301..bafd7e120 100644 --- a/http/codegen/testdata/untagged_arrays_harness.go +++ b/http/codegen/testdata/untagged_arrays_harness.go @@ -78,6 +78,9 @@ func TestServiceCodecs(t *testing.T) { require.ErrorContains(t, err, "matched 2 branches in schema") var overlapping svc.Overlap require.ErrorContains(t, json.Unmarshal([]byte("{}"), &overlapping), "matched 2 branches in schema") + require.ErrorContains(t, json.Unmarshal([]byte("{\"mapping\":{\"key\":\"value\"}}"), &overlapping), "matched 2 branches in schema") + _, err = json.Marshal(svc.NewOverlapLeft(&svc.Left{Mapping: map[string]string{"key":"value"}}), json.Deterministic(true)) + require.ErrorContains(t, err, "matched 2 branches in schema") } @@ -96,13 +99,23 @@ func TestHTTPCodecs(t *testing.T) { return nil, nil } mux := loomhttp.NewMuxer() - server.Mount(mux, server.New(&svc.Endpoints{Echo: echo, Optional: optional}, mux, loomhttp.RequestDecoder, loomhttp.ResponseEncoder, nil, nil)) + fallbackValue := svc.NewFallbackRaw(&svc.Raw{Row: &svc.ExtraRow{Group: "team", Extra: true}}) + fallback := func(context.Context, any) (any, error) { + return &fallbackValue, nil + } + server.Mount(mux, server.New(&svc.Endpoints{Echo: echo, Optional: optional, Fallback: fallback}, mux, loomhttp.RequestDecoder, loomhttp.ResponseEncoder, nil, nil)) host := httptest.NewServer(mux) defer host.Close() address, err := url.Parse(host.URL) require.NoError(t, err) c := client.NewClient(address.Scheme, address.Host, host.Client(), loomhttp.RequestEncoder, loomhttp.ResponseDecoder, false) - for _, wire := range []string{"[]", "[{\"name\":\"ok\"}]", "[{\"name\":\"ok\",\"label\":null}]", "[{\"name\":\"ok\",\"label\":\"open\"}]", "{\"total\":0}", "{\"total\":1,\"items\":[{\"name\":\"ok\",\"label\":null}]}"} { + // API-level closed objects reject the structured nested candidate while Raw + // remains the unique match. Both generated HTTP codecs must retain Raw. + fallbackResult, err := c.Fallback()(t.Context(), nil) + require.NoError(t, err) + require.Equal(t, &fallbackValue, fallbackResult) + + for _, wire := range []string{"[]", "[{\"name\":\"ok\"}]", "[{\"name\":\"ok\",\"label\":null}]", "[{\"name\":\"ok\",\"label\":\"open\"}]", "{\"total\":0}", "{\"total\":1,\"items\":[{\"name\":\"ok\",\"label\":null}]}", "{\"total\":0,\"properties\":{}}", "{\"total\":1,\"properties\":{\"a\":[1,null,{\"nested\":true}]},\"labels\":{\"first\":\"ok\"},\"lookup\":{\"item\":{\"name\":\"ok\",\"label\":null}}}", "{\"total\":0,\"nested\":{\"outer\":{\"inner\":[{\"name\":\"ok\"}]}}}", "{\"total\":0,\"nullable\":null}", "{\"total\":0,\"nullable\":{\"a\":null,\"b\":{\"name\":\"ok\"}}}", "{\"total\":0,\"tree\":{\"name\":\"root\",\"children\":{\"leaf\":{\"name\":\"leaf\"}}}}", "{\"total\":0,\"entries\":[]}", "{\"total\":2,\"entries\":[{\"name\":\"ok\"},{\"group\":\"team\"}]}", "{\"total\":1,\"entries\":[{\"group\":\"team\",\"members\":[{\"name\":\"ok\"},{\"group\":\"nested\",\"members\":[]}]}]}", "{\"total\":0,\"single\":null}", "{\"total\":0,\"single\":{\"name\":\"ok\"}}", "{\"total\":0,\"entry_lookup\":{\"a\":{\"name\":\"ok\"},\"b\":{\"group\":\"team\"}}}"} { payload, err := client.BuildEchoPayload(wire) require.NoError(t, err) result, err := c.Echo()(t.Context(), payload) @@ -117,6 +130,25 @@ func TestHTTPCodecs(t *testing.T) { {"", true, false}, {" \t\r\n", true, false}, {"{x}", false, false}, {"[", false, false}, {"null", false, false}, {"[null]", false, false}, {"{\"total\":0,\"extra\":1}", false, false}, + {"{\"total\":1,\"entries\":[{}]}", false, false}, + {"{\"total\":1,\"entries\":[{\"name\":\"ok\",\"group\":\"team\"}]}", false, false}, + {"{\"total\":1,\"entries\":[null]}", false, false}, + {"{\"total\":1,\"entries\":[{\"name\":\"x\"}]}", false, false}, + {"{\"total\":1,\"entries\":[{\"group\":\"team\",\"members\":[{}]}]}", false, false}, + {"{\"total\":0,\"single\":{}}", false, false}, + {"{\"total\":0,\"entry_lookup\":{\"a\":{}}}", false, false}, + {"{\"total\":0,\"overlap\":{}}", false, false}, + {"{\"total\":0,\"properties\":null}", false, false}, + {"{\"total\":0,\"properties\":[]}", false, false}, + {"{\"total\":0,\"labels\":{}}", false, false}, + {"{\"total\":0,\"labels\":{\"a\":\"ok\",\"b\":\"ok\",\"c\":\"ok\"}}", false, false}, + {"{\"total\":0,\"labels\":{\"a\":\"x\"}}", false, false}, + {"{\"total\":0,\"lookup\":{\"a\":{\"name\":\"ok\"},\"b\":{\"name\":\"x\"}}}", false, false}, + {"{\"total\":0,\"lookup\":{\"a\":null}}", false, false}, + {"{\"total\":0,\"lookup\":{\"a\":{\"name\":\"ok\",\"extra\":1}}}", false, false}, + {"{\"total\":0,\"nested\":{\"outer\":{\"inner\":[{\"name\":\"x\"}]}}}", false, false}, + {"{\"total\":0,\"tree\":{\"name\":\"root\",\"children\":{\"leaf\":{}}}}", false, false}, + {"{\"total\":0,\"nullable\":{\"a\":{\"name\":\"x\"}}}", false, false}, {"[{\"name\":\"ok\",\"label\":\"other\"}]", false, false}, {"[{\"name\":\"x\"}]", false, false}, {"[]", false, true}, } { @@ -177,4 +209,93 @@ func TestExactNumericUnionDescriptors(t *testing.T) { require.Error(t, json.Unmarshal([]byte(wire), &second), wire) } } + +func TestMapFields(t *testing.T) { + for _, wire := range []string{"{\"total\":0,\"properties\":{}}", +"{\"total\":1,\"properties\":{\"a\":[1,null,{\"nested\":true}]},\"labels\":{\"first\":\"ok\"},\"lookup\":{\"item\":{\"name\":\"ok\",\"label\":null}}}", +"{\"total\":0,\"nested\":{\"outer\":{\"inner\":[{\"name\":\"ok\"}]}}}", +"{\"total\":0,\"nullable\":null}", +"{\"total\":0,\"nullable\":{\"a\":null,\"b\":{\"name\":\"ok\"}}}", +"{\"total\":0,\"tree\":{\"name\":\"root\",\"children\":{\"leaf\":{\"name\":\"leaf\"}}}}"} { + var value svc.Result + require.NoError(t, json.Unmarshal([]byte(wire), &value), wire) + encoded, err := json.Marshal(value, json.Deterministic(true)) + require.NoError(t, err, wire) + require.JSONEq(t, wire, string(encoded)) + var decoded svc.Result + require.NoError(t, json.Unmarshal(encoded, &decoded)) + require.Equal(t, value, decoded) + } + for _, wire := range []string{"{\"total\":0,\"properties\":null}", +"{\"total\":0,\"properties\":[]}", +"{\"total\":0,\"labels\":{}}", +"{\"total\":0,\"labels\":{\"a\":\"ok\",\"b\":\"ok\",\"c\":\"ok\"}}", +"{\"total\":0,\"labels\":{\"a\":\"x\"}}", +"{\"total\":0,\"lookup\":{\"a\":{\"name\":\"ok\"},\"b\":{\"name\":\"x\"}}}", +"{\"total\":0,\"lookup\":{\"a\":null}}", +"{\"total\":0,\"lookup\":{\"a\":{\"name\":\"ok\",\"extra\":1}}}", +"{\"total\":0,\"nested\":{\"outer\":{\"inner\":[{\"name\":\"x\"}]}}}", +"{\"total\":0,\"tree\":{\"name\":\"root\",\"children\":{\"leaf\":{}}}}", +"{\"total\":0,\"nullable\":{\"a\":{\"name\":\"x\"}}}"} { + original := svc.NewResultItems(nil) + value := original + require.Error(t, json.Unmarshal([]byte(wire), &value), wire) + require.Equal(t, original, value) + } + for _, wire := range []string{ + "{\"total\":0,\"labels\":{}}", + "{\"total\":0,\"labels\":{\"a\":\"x\"}}", + "{\"total\":0,\"lookup\":{\"a\":null}}", + "{\"total\":0,\"lookup\":{\"a\":{\"name\":\"x\"}}}", + "{\"total\":0,\"nested\":{\"outer\":{\"inner\":[{\"name\":\"x\"}]}}}", + } { + var page svc.Page + require.NoError(t, json.Unmarshal([]byte(wire), &page), wire) + _, err := json.Marshal(svc.NewResultPage(&page), json.Deterministic(true)) + require.Error(t, err, wire) + } +} + +func TestNestedUnions(t *testing.T) { + for _, wire := range []string{"{\"total\":0,\"entries\":[]}", +"{\"total\":2,\"entries\":[{\"name\":\"ok\"},{\"group\":\"team\"}]}", +"{\"total\":1,\"entries\":[{\"group\":\"team\",\"members\":[{\"name\":\"ok\"},{\"group\":\"nested\",\"members\":[]}]}]}", +"{\"total\":0,\"single\":null}", +"{\"total\":0,\"single\":{\"name\":\"ok\"}}", +"{\"total\":0,\"entry_lookup\":{\"a\":{\"name\":\"ok\"},\"b\":{\"group\":\"team\"}}}"} { + var value svc.Result + require.NoError(t, json.Unmarshal([]byte(wire), &value), wire) + encoded, err := json.Marshal(value, json.Deterministic(true)) + require.NoError(t, err, wire) + require.JSONEq(t, wire, string(encoded)) + var decoded svc.Result + require.NoError(t, json.Unmarshal(encoded, &decoded)) + require.Equal(t, value, decoded) + } + for _, wire := range []string{"{\"total\":1,\"entries\":[{}]}", +"{\"total\":1,\"entries\":[null]}", +"{\"total\":1,\"entries\":[{\"name\":\"x\"}]}", +"{\"total\":1,\"entries\":[{\"group\":\"team\",\"members\":[{}]}]}", +"{\"total\":0,\"single\":{}}", +"{\"total\":0,\"entry_lookup\":{\"a\":{}}}", +"{\"total\":0,\"overlap\":{}}"} { + original := svc.NewResultItems(nil) + value := original + require.Error(t, json.Unmarshal([]byte(wire), &value), wire) + require.Equal(t, original, value) + } + // Group is open in the service schema; HTTP's API-level closed-object policy + // rejects this same extra-member input in the request matrix above. + var open svc.Result + require.NoError(t, json.Unmarshal([]byte("{\"total\":1,\"entries\":[{\"name\":\"ok\",\"group\":\"team\"}]}"), &open)) + + var page svc.Page + require.NoError(t, json.Unmarshal([]byte("{\"total\":1,\"entries\":[{\"name\":\"ok\"}]}"), &page)) + item, ok := page.Entries[0].AsItem() + require.True(t, ok) + item.Name = "x" + _, err := json.Marshal(svc.NewResultPage(&page), json.Deterministic(true)) + require.Error(t, err, "encoding must validate the selected nested branch") + +} ` diff --git a/http/middleware/rate.go b/http/middleware/rate.go new file mode 100644 index 000000000..c9194e5e8 --- /dev/null +++ b/http/middleware/rate.go @@ -0,0 +1,52 @@ +package middleware + +import ( + "net/http" + "strconv" + + "github.com/CaliLuke/loom/admission" +) + +// RateLimit admits HTTP requests before invoking the next handler. Applications +// choose a trusted grouping function and local or shared quota backend. Quota +// denial returns 429 and a whole-second Retry-After when known. Grouping/backend +// errors and timeouts return 503 with no sensitive details. No retry occurs. +// +// Place early limits outside the generated handler to protect decoding and +// authentication. Authenticated quotas belong at protected endpoint middleware. +// The request/context is passed unchanged after admission. Do not derive keys +// from forwarded headers without a trusted proxy policy, or expose keys as metric +// labels. Construction panics for nil dependencies. +func RateLimit(control *admission.RateAdmission, key func(*http.Request) (string, error)) func(http.Handler) http.Handler { + if control == nil || key == nil { + panic("rate limit: admission and grouping function required") + } + return func(next http.Handler) http.Handler { + if next == nil { + panic("rate limit: non-nil handler required") + } + return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.Context().Err() != nil { + return + } + group, err := key(r) + if err != nil { + http.Error(w, http.StatusText(http.StatusServiceUnavailable), http.StatusServiceUnavailable) + return + } + decision, err := control.Check(r.Context(), group) + if err != nil { + http.Error(w, http.StatusText(http.StatusServiceUnavailable), http.StatusServiceUnavailable) + return + } + if !decision.Allowed { + if seconds := decision.RetryAfterSeconds(); seconds > 0 { + w.Header().Set("Retry-After", strconv.FormatInt(seconds, 10)) + } + http.Error(w, http.StatusText(http.StatusTooManyRequests), http.StatusTooManyRequests) + return + } + next.ServeHTTP(w, r) + }) + } +} diff --git a/http/middleware/rate_test.go b/http/middleware/rate_test.go new file mode 100644 index 000000000..cd8a13207 --- /dev/null +++ b/http/middleware/rate_test.go @@ -0,0 +1,113 @@ +package middleware_test + +import ( + "context" + "errors" + "net/http" + "net/http/httptest" + "testing" + "testing/synctest" + "time" + + "github.com/stretchr/testify/require" + + "github.com/CaliLuke/loom/admission" + "github.com/CaliLuke/loom/http/middleware" +) + +func TestRateLimitHTTP(t *testing.T) { + for _, tc := range []struct { + name string + decision admission.RateDecision + backendError, keyError error + status, calls int + retry string + }{ + {"admitted", admission.RateDecision{Allowed: true}, nil, nil, 204, 1, ""}, + {"known delay", admission.RateDecision{RetryAfter: time.Second + time.Nanosecond}, nil, nil, 429, 0, "2"}, + {"subsecond", admission.RateDecision{RetryAfter: time.Nanosecond}, nil, nil, 429, 0, "1"}, + {"unknown delay", admission.RateDecision{}, nil, nil, 429, 0, ""}, + {"backend error", admission.RateDecision{Allowed: true}, errors.New("sensitive backend"), nil, 503, 0, ""}, + {"key error", admission.RateDecision{}, nil, errors.New("sensitive grouping"), 503, 0, ""}, + } { + t.Run(tc.name, func(t *testing.T) { + checks := 0 + control, err := admission.NewRateAdmission(func(_ context.Context, key string) (admission.RateDecision, error) { + checks++ + require.Equal(t, "bounded-group", key) + return tc.decision, tc.backendError + }, time.Second) + require.NoError(t, err) + req := httptest.NewRequest(http.MethodGet, "/", nil) + calls := 0 + handler := middleware.RateLimit(control, func(*http.Request) (string, error) { + return "bounded-group", tc.keyError + })(http.HandlerFunc(func(w http.ResponseWriter, got *http.Request) { + calls++ + require.Same(t, req, got) + require.NoError(t, got.Context().Err()) + w.WriteHeader(http.StatusNoContent) + })) + response := httptest.NewRecorder() + handler.ServeHTTP(response, req) + require.Equal(t, tc.status, response.Code) + require.Equal(t, tc.calls, calls) + require.Equal(t, tc.retry, response.Header().Get("Retry-After")) + require.NotContains(t, response.Body.String(), "sensitive") + if tc.keyError != nil { + require.Zero(t, checks) + } else { + require.Equal(t, 1, checks) + } + }) + } +} + +func TestRateLimitHTTPTimeoutAndCancellation(t *testing.T) { + synctest.Test(t, func(t *testing.T) { + checks, calls := 0, 0 + control, err := admission.NewRateAdmission(func(ctx context.Context, _ string) (admission.RateDecision, error) { + checks++ + <-ctx.Done() + return admission.RateDecision{}, ctx.Err() + }, 100*time.Millisecond) + require.NoError(t, err) + handler := middleware.RateLimit(control, func(*http.Request) (string, error) { + return "global", nil + })(http.HandlerFunc(func(http.ResponseWriter, *http.Request) { + calls++ + })) + req := httptest.NewRequest(http.MethodGet, "/", nil) + response := httptest.NewRecorder() + start := time.Now() + handler.ServeHTTP(response, req) + require.Equal(t, 100*time.Millisecond, time.Since(start)) + require.Equal(t, 503, response.Code) + require.Equal(t, 1, checks) + require.Zero(t, calls) + ctx, cancel := context.WithCancel(context.Background()) + cancel() + handler.ServeHTTP(httptest.NewRecorder(), req.WithContext(ctx)) + require.Equal(t, 1, checks) + require.Zero(t, calls) + }) +} + +func TestRateLimitHTTPConfiguration(t *testing.T) { + key := func(*http.Request) (string, error) { + return "", nil + } + require.Panics(t, func() { + middleware.RateLimit(nil, key) + }) + control, err := admission.NewRateAdmission(func(context.Context, string) (admission.RateDecision, error) { + return admission.RateDecision{}, nil + }, time.Second) + require.NoError(t, err) + require.Panics(t, func() { + middleware.RateLimit(control, nil) + }) + require.Panics(t, func() { + middleware.RateLimit(control, key)(nil) + }) +} diff --git a/internal/openapiimport/analyze.go b/internal/openapiimport/analyze.go index 6c3227ec2..061d06647 100644 --- a/internal/openapiimport/analyze.go +++ b/internal/openapiimport/analyze.go @@ -63,17 +63,19 @@ func analyzeSelectedDocument(source []byte, selection Selection) (*Document, Dia if err != nil { return nil, nil, SelectionReport{}, fmt.Errorf("inspect numeric source: %w", err) } + positions := numericSourcePositions(parsed.GetSpecInfo().RootNode, numbers) model, err := parsed.BuildV3Model() if err != nil { return nil, nil, SelectionReport{}, fmt.Errorf("build OpenAPI %s model: %w", version, err) } analyzer := analyzer{ - diagnostics: diagnostics, - selection: selection, - examples: make(map[string]Example), - schemas: make(map[string]*Schema), - version: version, - numbers: numbers, + diagnostics: diagnostics, + selection: selection, + examples: make(map[string]Example), + schemas: make(map[string]*Schema), + version: version, + numbers: numbers, + numericPositions: positions, } document := analyzer.document(&model.Model) if selection.Active() { @@ -84,13 +86,14 @@ func analyzeSelectedDocument(source []byte, selection Selection) (*Document, Dia } type analyzer struct { - diagnostics Diagnostics - selection Selection - report SelectionReport - examples map[string]Example - schemas map[string]*Schema - version string - numbers map[string]numericSource + diagnostics Diagnostics + selection Selection + report SelectionReport + examples map[string]Example + schemas map[string]*Schema + version string + numbers map[string]numericSource + numericPositions map[numericPosition]numericSource } var concreteHTTPStatus = regexp.MustCompile(`^[1-5][0-9]{2}$`) @@ -404,11 +407,17 @@ func (a *analyzer) requestBody(source *v3.RequestBody, path string) RequestBody if source == nil { return RequestBody{} } - if reference := source.Reference; reference != "" { - return RequestBody{Ref: reference} + reference := source.Reference + var referenceNode *yaml.Node + if low := source.GoLow(); low != nil { + if reference == "" && low.IsReference() { + reference = low.GetReference() + } + referenceNode = low.GetReferenceNode() } - if low := source.GoLow(); low != nil && low.IsReference() { - return RequestBody{Ref: low.GetReference()} + if reference != "" { + description, summary := a.referenceAnnotations(referenceNode) + return RequestBody{Ref: reference, ReferenceDescription: description, ReferenceSummary: summary} } contentTypes, schema, examples := a.requestContent(source.Content, path+"/content") body := RequestBody{ @@ -424,11 +433,17 @@ func (a *analyzer) response(source *v3.Response, path string) Response { if source == nil { return Response{} } - if reference := source.Reference; reference != "" { - return Response{Ref: reference} + reference := source.Reference + var referenceNode *yaml.Node + if low := source.GoLow(); low != nil { + if reference == "" && low.IsReference() { + reference = low.GetReference() + } + referenceNode = low.GetReferenceNode() } - if low := source.GoLow(); low != nil && low.IsReference() { - return Response{Ref: low.GetReference()} + if reference != "" { + description, summary := a.referenceAnnotations(referenceNode) + return Response{Ref: reference, ReferenceDescription: description, ReferenceSummary: summary} } contentType, schema, examples := a.content(source.Content, path+"/content") response := Response{ @@ -458,11 +473,17 @@ func (a *analyzer) header(source *v3.Header, path string) Header { if source == nil { return Header{} } - if reference := source.Reference; reference != "" { - return Header{Ref: reference} + reference := source.Reference + var referenceNode *yaml.Node + if low := source.GoLow(); low != nil { + if reference == "" && low.IsReference() { + reference = low.GetReference() + } + referenceNode = low.GetReferenceNode() } - if low := source.GoLow(); low != nil && low.IsReference() { - return Header{Ref: low.GetReference()} + if reference != "" { + description, summary := a.referenceAnnotations(referenceNode) + return Header{Ref: reference, ReferenceDescription: description, ReferenceSummary: summary} } header := Header{ Description: source.Description, Required: source.Required, @@ -527,14 +548,7 @@ func (a *analyzer) schema(proxy *base.SchemaProxy, path string) *Schema { } a.schemaUnknownKeywordDiagnostics(proxy, path) if proxy.IsReference() { - if proxy.IsTransformedRefWithSiblings() { - a.unsupported( - "schema-reference-siblings", - path, - "schema $ref siblings are not in the strict import subset; wrap the reference in a supported allOf shape", - ) - } - return &Schema{Ref: proxy.GetReference()} + return a.schemaReference(proxy, path) } if node := semanticSchemaNode(proxy); node != nil && node.Kind == yaml.ScalarNode && node.Tag == "!!bool" { var value bool @@ -549,9 +563,13 @@ func (a *analyzer) schema(proxy *base.SchemaProxy, path string) *Schema { a.unsupported("empty-schema", path, "schema could not be resolved") return &Schema{} } + return a.schemaSource(source, path) +} + +func (a *analyzer) schemaSource(source *base.Schema, path string) *Schema { schema := newNormalizedSchema(source) a.schemaPattern(schema, path) - a.schemaNumericBounds(schema, path) + a.schemaNumericBounds(schema, source, path) supportedNullableComposition := a.schemaNullableAnyOf(schema, source, path) supportedNullableOneOf := schemaNullableReferenceOneOf(schema, source) supportedUntaggedOneOf := false diff --git a/internal/openapiimport/analyze_reference.go b/internal/openapiimport/analyze_reference.go index 82b4f4534..1a431b89b 100644 --- a/internal/openapiimport/analyze_reference.go +++ b/internal/openapiimport/analyze_reference.go @@ -1,6 +1,9 @@ package openapiimport -import "github.com/pb33f/go-yaml" +import ( + "github.com/pb33f/go-yaml" + "strings" +) // referenceDescriptionOverride preserves absent versus explicitly empty // description fields on reference occurrences. @@ -16,3 +19,22 @@ func referenceDescriptionOverride(node *yaml.Node) *string { } return nil } + +func (a *analyzer) referenceAnnotations(node *yaml.Node) (*string, *string) { + if strings.HasPrefix(a.version, "3.0.") { + return nil, nil + } + var description, summary *string + if node != nil && node.Kind == yaml.MappingNode { + for index := 0; index+1 < len(node.Content); index += 2 { + value := node.Content[index+1].Value + switch node.Content[index].Value { + case "description": + description = &value + case "summary": + summary = &value + } + } + } + return description, summary +} diff --git a/internal/openapiimport/analyze_reference_object.go b/internal/openapiimport/analyze_reference_object.go new file mode 100644 index 000000000..b7b4faa68 --- /dev/null +++ b/internal/openapiimport/analyze_reference_object.go @@ -0,0 +1,116 @@ +package openapiimport + +import ( + "slices" + "sort" + + "github.com/CaliLuke/libopenapi/datamodel/high/base" + "github.com/CaliLuke/libopenapi/orderedmap" +) + +// schemaObjectReference uses object composition for physical fields while +// retaining the reference as a separate contract contributor. +func (a *analyzer) schemaObjectReference(local, target *base.Schema, ref, path string) *Schema { + fail := func(message string) *Schema { + return &Schema{Ref: ref, referenceError: &Diagnostic{Code: "schema-reference-siblings", Path: path, Message: message}} + } + if !slices.Equal(target.Type, []string{"object"}) || len(local.Type) > 0 && !slices.Equal(local.Type, target.Type) { + return fail("object reference properties require a nonnullable object target") + } + if local.Items != nil || len(local.AllOf) > 0 || len(local.AnyOf) > 0 || len(local.OneOf) > 0 { + return fail("object reference properties cannot be combined with other structural applicators") + } + if message := referenceObjectAdmission(local, target); message != "" { + return fail(message) + } + schema := a.schemaPropertyRefinement(local, target, path) + schema.Bases = []*Schema{{Ref: ref}} + schema.referenceSiblings = true + return schema +} + +// schemaPropertyRefinement supplies missing carrier context only inside an +// established intersection. Constraints still belong to their authored layer. +func (a *analyzer) schemaPropertyRefinement(local, target *base.Schema, path string) *Schema { + fail := func(message string) *Schema { + return &Schema{referenceError: &Diagnostic{Code: "schema-reference-siblings", Path: path, Message: message}} + } + if message := referenceObjectAdmission(local, target); message != "" { + return fail(message) + } + source := *local + types := a.referenceCarrierTypes(target, path) + if len(local.Type) > 0 && !sameReferenceTypes(local.Type, types) && !sameReferenceTypes(local.Type, target.Type) { + return fail("property refinement cannot change the inherited type or nullability") + } + source.Type = types + if local.Format != "" && local.Format != target.Format && !slices.Contains(types, "string") { + return fail("property refinement cannot change the inherited Go carrier format") + } + if source.Format == "" { + source.Format = target.Format + } + if source.Items == nil { + source.Items = target.Items + } + source.Properties = nil + schema := a.schemaSource(&source, path) + if slices.Contains(types, "null") && !schema.Nullable { + return fail("property membership cannot remove inherited nullability") + } + for name, property := range local.Properties.FromOldest() { + propertyPath := path + "/properties/" + escapeJSONPointer(name) + refined := a.schemaInheritedProperty(property, target.Properties.GetOrZero(name), propertyPath) + schema.Properties = append(schema.Properties, NamedProperty{Name: name, Schema: refined}) + } + sort.Slice(schema.Properties, func(i, j int) bool { return schema.Properties[i].Name < schema.Properties[j].Name }) + return schema +} + +func (a *analyzer) schemaInheritedProperty(property, inherited *base.SchemaProxy, path string) *Schema { + if inherited == nil || property == nil || property.IsReference() { + return a.schema(property, path) + } + a.schemaUnknownKeywordDiagnostics(property, path) + shape := referenceSchemaShape(inherited, make(map[*base.Schema]bool)) + if shape == nil || property.Schema() == nil { + return a.schema(property, path) + } + if inherited.IsReference() && orderedmap.Len(property.Schema().Properties) > 0 { + return a.schemaObjectReference(property.Schema(), shape, inherited.GetReference(), path) + } + refined := a.schemaPropertyRefinement(property.Schema(), shape, path) + if inherited.IsReference() { + refined.Ref = inherited.GetReference() + refined.referenceSiblings = true + } + return refined +} + +func referenceObjectAdmission(local, target *base.Schema) string { + closed := target.AdditionalProperties != nil && target.AdditionalProperties.IsB() && !target.AdditionalProperties.B + if orderedmap.Len(local.Properties) > 0 && orderedmap.Len(target.Properties) == 0 && !closed { + return "structured properties cannot refine a free-form map" + } + if closed { + for name := range local.Properties.FromOldest() { + if target.Properties.GetOrZero(name) == nil { + return "a reference to a closed object cannot add properties" + } + } + } + if local.AdditionalProperties == nil { + return "" + } + if local.AdditionalProperties.IsA() { + return "typed additionalProperties intersections require general schema composition support" + } + if !local.AdditionalProperties.B { + for name := range target.Properties.FromOldest() { + if local.Properties.GetOrZero(name) == nil { + return "a closed sibling must declare every inherited property" + } + } + } + return "" +} diff --git a/internal/openapiimport/analyze_reference_siblings.go b/internal/openapiimport/analyze_reference_siblings.go new file mode 100644 index 000000000..08102f368 --- /dev/null +++ b/internal/openapiimport/analyze_reference_siblings.go @@ -0,0 +1,100 @@ +package openapiimport + +import ( + "slices" + "strings" + + "github.com/CaliLuke/libopenapi/datamodel/high/base" + "github.com/CaliLuke/libopenapi/orderedmap" +) + +func (a *analyzer) schemaReference(proxy *base.SchemaProxy, path string) *Schema { + ref := proxy.GetReference() + fail := func(message string) *Schema { + return &Schema{Ref: ref, referenceError: &Diagnostic{Code: "schema-reference-siblings", Path: path, Message: message}} + } + if !proxy.IsTransformedRefWithSiblings() { + return &Schema{Ref: ref} + } + if strings.HasPrefix(a.version, "3.0.") { + return fail("schema reference siblings require OpenAPI 3.1 or 3.2") + } + sibling := proxy.Schema() + semantic, err := proxy.BuildTransformedRefSemanticSchema(sibling) + if err != nil || sibling == nil || semantic == nil || len(semantic.AllOf) != 2 { + return fail("schema reference siblings could not be resolved") + } + target := referenceSchemaShape(semantic.AllOf[1], make(map[*base.Schema]bool)) + if target == nil { + return fail("referenced schema has no supported concrete shape") + } + if orderedmap.Len(sibling.Properties) > 0 { + return a.schemaObjectReference(sibling, target, ref, path) + } + if sibling.Items != nil || sibling.AdditionalProperties != nil || + len(sibling.AllOf) > 0 || len(sibling.AnyOf) > 0 || len(sibling.OneOf) > 0 { + return fail("structural reference intersections require general schema composition support") + } + targetTypes := a.referenceCarrierTypes(target, ref) + if len(sibling.Type) > 0 && !sameReferenceTypes(sibling.Type, target.Type) && !sameReferenceTypes(sibling.Type, targetTypes) { + return fail("reference sibling type must preserve the referenced concrete type and nullability") + } + if sibling.Format != "" && sibling.Format != target.Format && !slices.Contains(target.Type, "string") { + return fail("reference siblings cannot change the referenced Go carrier format") + } + // Type and format provide the carrier context needed to normalize enums, + // defaults and lengths. They do not replace or mutate the referenced schema. + source := *sibling + source.Type = targetTypes + inheritedFormat := source.Format == "" + if inheritedFormat { + source.Format = target.Format + } + schema := a.schemaSource(&source, path) + if slices.Contains(targetTypes, "null") && !schema.Nullable { + return fail("reference sibling membership cannot remove inherited nullability") + } + schema.Ref = ref + schema.referenceSiblings = true + schema.inheritedFormat = inheritedFormat + return schema +} + +func referenceSchemaShape(proxy *base.SchemaProxy, seen map[*base.Schema]bool) *base.Schema { + if proxy == nil { + return nil + } + source := proxy.Schema() + if source == nil || seen[source] { + return nil + } + seen[source] = true + if len(source.Type) > 0 { + return source + } + if len(source.AllOf) == 1 { + return referenceSchemaShape(source.AllOf[0], seen) + } + return nil +} + +func sameReferenceTypes(left, right []string) bool { + left, right = slices.Clone(left), slices.Clone(right) + slices.Sort(left) + slices.Sort(right) + return slices.Equal(left, right) +} + +// referenceCarrierTypes applies the same membership normalization as declarations +// before supplying context to an occurrence. A nullable spelling alone does not +// authorize null when enum or const has already excluded it from the base. +func (a *analyzer) referenceCarrierTypes(target *base.Schema, path string) []string { + if valueType, nullable := nullableUnionType(target.Type); nullable { + domain := &Schema{Type: valueType, Format: target.Format, Nullable: true} + a.schemaValueDomain(domain, target, path) + if !domain.Nullable { + return []string{valueType} + } + } + return slices.Clone(target.Type) +} diff --git a/internal/openapiimport/analyze_schema_composition.go b/internal/openapiimport/analyze_schema_composition.go index ca0527d79..95aeb0ff9 100644 --- a/internal/openapiimport/analyze_schema_composition.go +++ b/internal/openapiimport/analyze_schema_composition.go @@ -2,6 +2,7 @@ package openapiimport import ( "fmt" + "slices" "github.com/CaliLuke/libopenapi/datamodel/high/base" "github.com/CaliLuke/libopenapi/orderedmap" @@ -142,7 +143,7 @@ func (a *analyzer) schemaAllOf(schema *Schema, source *base.Schema, path string) } if len(source.AllOf) == 1 && !hasUnsupportedSingleReferenceAllOfSiblings(source) { proxy := source.AllOf[0] - if proxy != nil && proxy.IsReference() { + if proxy != nil && proxy.IsReference() && !proxy.IsTransformedRefWithSiblings() { ref := proxy.GetReference() if _, err := openapiref.Name(ref, "#/components/schemas/"); err == nil { schema.Ref = ref @@ -156,41 +157,38 @@ func (a *analyzer) schemaAllOf(schema *Schema, source *base.Schema, path string) if len(source.AllOf) != 2 || hasDirectAllOfSchemaShape(source) { return false } - var baseSchema, inlineSchema *Schema + var reference *base.SchemaProxy + var inline *base.Schema + inlinePath := path for index, proxy := range source.AllOf { if proxy == nil { return false } - partPath := fmt.Sprintf("%s/allOf/%d", path, index) if proxy.IsReference() { - ref := proxy.GetReference() - if _, err := openapiref.Name(ref, "#/components/schemas/"); err != nil || baseSchema != nil { + if proxy.IsTransformedRefWithSiblings() { + return false + } + if _, err := openapiref.Name(proxy.GetReference(), "#/components/schemas/"); err != nil || reference != nil { return false } - baseSchema = &Schema{Ref: ref} + reference = proxy continue } - if inlineSchema != nil { + if inline != nil { return false } - inlineSchema = a.schema(proxy, partPath) + inline = proxy.Schema() + inlinePath = fmt.Sprintf("%s/allOf/%d", path, index) + a.schemaUnknownKeywordDiagnostics(proxy, inlinePath) } - if baseSchema == nil || inlineSchema == nil || inlineSchema.Type != "object" || len(inlineSchema.Bases) > 0 { + if reference == nil || inline == nil || !slices.Equal(inline.Type, []string{"object"}) { return false } - schema.Type = "object" - schema.Bases = []*Schema{baseSchema} - schema.Properties = inlineSchema.Properties - schema.Required = inlineSchema.Required - schema.AdditionalProperties = inlineSchema.AdditionalProperties - if schema.Description == "" { - schema.Description = inlineSchema.Description + target := referenceSchemaShape(reference, make(map[*base.Schema]bool)) + if target == nil { + return false } - a.unsupported( - "schema-allof-flattened", - path+"/allOf", - "object inheritance is rendered with Extend and regenerated OpenAPI flattens the composition", - ) + *schema = *a.schemaObjectReference(inline, target, reference.GetReference(), inlinePath) return true } diff --git a/internal/openapiimport/analyze_test.go b/internal/openapiimport/analyze_test.go index e15c58024..253cc635b 100644 --- a/internal/openapiimport/analyze_test.go +++ b/internal/openapiimport/analyze_test.go @@ -588,12 +588,12 @@ components: require.NoError(t, err) requireDiagnosticCode(t, diagnostics, "duplicate-parameter") requireDiagnosticCode(t, diagnostics, "parameter-reference") - requireDiagnosticCode(t, diagnostics, "request-body-reference") - requireDiagnosticCode(t, diagnostics, "response-reference") - requireDiagnosticCode(t, diagnostics, "header-reference") + for _, diagnostic := range diagnostics { + require.NotContains(t, []string{"request-body-reference", "response-reference", "header-reference"}, diagnostic.Code) + } } -func TestAnalyzeRejectsUnrenderableComponentKinds(t *testing.T) { +func TestAnalyzeRejectsUnusedReusableComponents(t *testing.T) { source := []byte(`openapi: 3.1.1 info: {title: Component kinds, version: "1"} paths: diff --git a/internal/openapiimport/analyze_value_domain.go b/internal/openapiimport/analyze_value_domain.go index 0ef483c12..aafc546b7 100644 --- a/internal/openapiimport/analyze_value_domain.go +++ b/internal/openapiimport/analyze_value_domain.go @@ -80,7 +80,7 @@ func (a *analyzer) schemaValueDomain(schema *Schema, source *base.Schema, path s if keyword == "enum" { location += "/" + strconv.Itoa(index) } - value, included, err := a.valueDomainMember(schema, node, location) + value, included, err := a.valueDomainMember(schema, node) if err != nil { fail(location, err.Error()) return @@ -102,8 +102,8 @@ func (a *analyzer) schemaValueDomain(schema *Schema, source *base.Schema, path s // valueDomainMember first classifies the exact JSON value against the source // type. Representability failures for included values reject the whole domain. -func (a *analyzer) valueDomainMember(schema *Schema, node *yaml.Node, path string) (any, bool, error) { - if raw, present := a.numbers[path]; present { +func (a *analyzer) valueDomainMember(schema *Schema, node *yaml.Node) (any, bool, error) { + if raw, present := a.nodeNumericSource(node); present { if number, err := raw.number(); err == nil { if schema.Type != "number" && schema.Type != "integer" || schema.Type == "integer" && !number.IsInt() { return nil, false, nil diff --git a/internal/openapiimport/diagnostics.go b/internal/openapiimport/diagnostics.go index 442678da8..480840b95 100644 --- a/internal/openapiimport/diagnostics.go +++ b/internal/openapiimport/diagnostics.go @@ -28,7 +28,6 @@ var lossyDiagnosticCodes = map[string]struct{}{ "media-type-description": {}, "parameter-deprecated": {}, "path-metadata": {}, - "schema-allof-flattened": {}, "schema-format": {}, "schema-inline-array-item-promoted": {}, "vendor-extension": {}, diff --git a/internal/openapiimport/empty_object_test.go b/internal/openapiimport/empty_object_test.go index 3543f0e18..c17a7c82b 100644 --- a/internal/openapiimport/empty_object_test.go +++ b/internal/openapiimport/empty_object_test.go @@ -296,8 +296,7 @@ components: `) document, diagnostics, err := Analyze(source) require.NoError(t, err) - require.Len(t, diagnostics, 1) - require.Equal(t, "schema-allof-flattened", diagnostics[0].Code) + require.Empty(t, diagnostics) var operation *Operation for i := range document.Operations { if document.Operations[i].Path == "/value" { @@ -331,6 +330,11 @@ components: if ref, ok := schema["$ref"].(string); ok { schema = referencedSchema(t, contract, ref) } + if path == "/inherited" { + branches := schema["allOf"].([]any) + require.Equal(t, "#/components/schemas/Closed", branches[0].(map[string]any)["$ref"]) + schema = branches[1].(map[string]any) + } require.Equal(t, "object", schema["type"]) if path != "/inherited" { require.Equal(t, false, schema["additionalProperties"]) diff --git a/internal/openapiimport/grammar_coverage_test.go b/internal/openapiimport/grammar_coverage_test.go index af0035e5a..3681251bb 100644 --- a/internal/openapiimport/grammar_coverage_test.go +++ b/internal/openapiimport/grammar_coverage_test.go @@ -409,7 +409,7 @@ paths: } } -func TestSchemaReferenceSiblingsAreExplicitlyRejected(t *testing.T) { +func TestSchemaReferenceSiblingsArePreserved(t *testing.T) { source := []byte(`openapi: 3.1.1 info: {title: References, version: "1"} paths: @@ -431,9 +431,7 @@ components: _, diagnostics, err := Analyze(source) require.NoError(t, err) - requireDiagnosticCode(t, diagnostics, "schema-reference-siblings") - fatal, _ := diagnostics.Classify(true) - require.NotEmpty(t, fatal) + require.Empty(t, diagnostics) } func TestUnknownSchemaVocabularyIsExplicitlyRejected(t *testing.T) { @@ -553,7 +551,7 @@ func openAPIGrammarCoverage() []grammarObjectCoverage { model: v3.RequestBody{}, groups: []grammarFieldGroup{ group(grammarPreserved, "analyzer.requestBody", "Description", "Content", "Required", "Extensions"), - group(grammarRejected, "request-body-reference planner diagnostic", "Reference"), + group(grammarConditional, "resolveReusableDocument and TestReusableTransportComponentRoundTrip", "Reference"), }, }, { @@ -579,7 +577,8 @@ func openAPIGrammarCoverage() []grammarObjectCoverage { model: v3.Response{}, groups: []grammarFieldGroup{ group(grammarPreserved, "analyzer.response and TestImportPreservesExporterMetadataSymmetry", "Description", "Summary", "Headers", "Content", "Extensions"), - group(grammarRejected, "response reference and links diagnostics", "Reference", "Links"), + group(grammarConditional, "resolveReusableDocument", "Reference"), + group(grammarRejected, "response links diagnostic", "Links"), }, }, { @@ -589,7 +588,8 @@ func openAPIGrammarCoverage() []grammarObjectCoverage { group(grammarPreserved, "analyzer.header", "Description", "Required", "Schema"), group(grammarConditional, "analyzer.header and TestImportPreservesExporterMetadataSymmetry", "Style", "AllowReserved"), group(grammarLossy, "header-deprecated and examples diagnostics", "Deprecated", "Example", "Examples"), - group(grammarRejected, "header reference, serialization, content, and extension diagnostics", "Reference", "AllowEmptyValue", "Explode", "Content", "Extensions"), + group(grammarConditional, "resolveReusableDocument", "Reference"), + group(grammarRejected, "header serialization, content, and extension diagnostics", "AllowEmptyValue", "Explode", "Content", "Extensions"), }, }, { diff --git a/internal/openapiimport/issues_295_298_test.go b/internal/openapiimport/issues_295_298_test.go index 480717b09..ff5857278 100644 --- a/internal/openapiimport/issues_295_298_test.go +++ b/internal/openapiimport/issues_295_298_test.go @@ -125,7 +125,6 @@ components: document, diagnostics, err := Analyze(source) require.NoError(t, err) - requireDiagnosticCode(t, diagnostics, "schema-allof-flattened") requireDiagnosticCode(t, diagnostics, "schema-inline-array-item-promoted") fatal, warnings := diagnostics.Classify(true) require.Empty(t, fatal) @@ -140,7 +139,9 @@ components: operation := operationFromImportedSpec(t, contract, "/users", "get") items := operation["responses"].(map[string]any)["200"].(map[string]any)["content"].(map[string]any)["application/json"].(map[string]any)["schema"].(map[string]any)["items"].(map[string]any) itemSchema := referencedSchema(t, contract, items["$ref"].(string)) - properties := itemSchema["properties"].(map[string]any) + branches := itemSchema["allOf"].([]any) + require.Equal(t, "#/components/schemas/UserLite", branches[0].(map[string]any)["$ref"]) + properties := branches[1].(map[string]any)["properties"].(map[string]any) require.Contains(t, properties, "id") require.Contains(t, properties, "role") } diff --git a/internal/openapiimport/model.go b/internal/openapiimport/model.go index eca65434f..7e0a8ae25 100644 --- a/internal/openapiimport/model.go +++ b/internal/openapiimport/model.go @@ -5,6 +5,7 @@ package openapiimport import ( "github.com/CaliLuke/loom/internal/exactnum" "github.com/CaliLuke/loom/internal/httpencoding" + "github.com/CaliLuke/loom/internal/openapiref" ) // Document is the normalized subset of an OpenAPI document understood by the @@ -92,8 +93,6 @@ type SecurityScheme struct { OAuth2MetadataURL string // OAuthFlows contains supported OAuth2 flows in specification order. OAuthFlows []OAuthFlow - // Scopes contains the shared OAuth2 scope definitions. - Scopes []SecurityScope // Extensions contains scheme-level vendor extensions. Extensions map[string]any } @@ -108,6 +107,8 @@ type OAuthFlow struct { TokenURL string // RefreshURL is the optional flow refresh endpoint. RefreshURL string + // Scopes contains this flow's exact scope names and descriptions. + Scopes []SecurityScope } // SecurityScope is one OAuth2 scope definition. @@ -246,6 +247,11 @@ type Parameter struct { // RequestBody describes a request body. Ref is set instead of the remaining // fields for a component reference. type RequestBody struct { + // ReferenceDescription retains a present description override on Ref. + ReferenceDescription *string + // ReferenceSummary retains a present summary override on Ref. + ReferenceSummary *string + referenceChain []openapiref.Object // Ref is the authored local component reference, when retained as a reference. Ref string // Description is the request body description. @@ -275,6 +281,11 @@ type StatusResponse struct { // Response describes an HTTP response. Ref is set instead of the remaining // fields for a component reference. type Response struct { + // ReferenceDescription retains a present description override on Ref. + ReferenceDescription *string + // ReferenceSummary retains a present summary override on Ref. + ReferenceSummary *string + referenceChain []openapiref.Object // Ref is the authored local component reference, when retained as a reference. Ref string // Description is the response description. @@ -296,6 +307,11 @@ type Response struct { // Header describes a response header. Ref is set instead of the remaining // fields for a component reference. type Header struct { + // ReferenceDescription retains a present description override on Ref. + ReferenceDescription *string + // ReferenceSummary retains a present summary override on Ref. + ReferenceSummary *string + referenceChain []openapiref.Object // Ref is the authored local component reference, when retained as a reference. Ref string // Description is the response header description. @@ -311,8 +327,8 @@ type Header struct { } // Schema is the normalized JSON Schema subset expressible by the first import -// phase. Ref may be accompanied by constraints authored beside a supported -// single-reference allOf wrapper. +// phase. Ref may be accompanied by supported assertions and annotations +// authored beside $ref or a single-reference allOf wrapper. type Schema struct { // Ref is the authored local schema component reference. Ref string @@ -385,6 +401,9 @@ type Schema struct { Examples []Example // Extensions contains schema-level vendor extensions. Extensions map[string]any + referenceError *Diagnostic + referenceSiblings bool + inheritedFormat bool unsupportedComposition bool valueDomainKeyword string valueDomainError *Diagnostic diff --git a/internal/openapiimport/numeric_source.go b/internal/openapiimport/numeric_source.go index 7158a9ea1..5d1881f9b 100644 --- a/internal/openapiimport/numeric_source.go +++ b/internal/openapiimport/numeric_source.go @@ -10,6 +10,8 @@ import ( "strconv" "strings" + "github.com/CaliLuke/libopenapi/datamodel/high/base" + "github.com/CaliLuke/libopenapi/utils" "github.com/pb33f/go-yaml" "github.com/CaliLuke/loom/internal/exactnum" @@ -174,12 +176,35 @@ func yamlMergeSources(node *yaml.Node) ([]*yaml.Node, error) { return []*yaml.Node{node}, nil } -func (a *analyzer) schemaNumericBounds(schema *Schema, path string) { +func (a *analyzer) schemaNumericBounds(schema *Schema, authored *base.Schema, path string) { + var nodes map[string]*yaml.Node + if low := authored.GoLow(); low != nil { + nodes = map[string]*yaml.Node{ + "minimum": low.Minimum.ValueNode, + "maximum": low.Maximum.ValueNode, + "exclusiveMinimum": low.ExclusiveMinimum.ValueNode, + "exclusiveMaximum": low.ExclusiveMaximum.ValueNode, + } + // A non-finite carrier conversion can leave a low-level numeric field + // empty. Its authored keyword still owns a token and must be diagnosed. + root := utils.ResolveMergeNodes(low.RootNode) + if root != nil && root.Kind == yaml.MappingNode { + for index := 0; index+1 < len(root.Content); index += 2 { + name := root.Content[index].Value + if node, selected := nodes[name]; selected && node == nil { + nodes[name] = root.Content[index+1] + } + } + } + } for _, endpoint := range []struct { name string value **exactnum.Number }{{"minimum", &schema.Minimum}, {"maximum", &schema.Maximum}, {"exclusiveMinimum", &schema.ExclusiveMinimum}, {"exclusiveMaximum", &schema.ExclusiveMaximum}} { source, present := a.numbers[path+"/"+endpoint.name] + if nodes != nil { + source, present = a.nodeNumericSource(nodes[endpoint.name]) + } if !present { continue } diff --git a/internal/openapiimport/numeric_source_identity.go b/internal/openapiimport/numeric_source_identity.go new file mode 100644 index 000000000..d18688e0f --- /dev/null +++ b/internal/openapiimport/numeric_source_identity.go @@ -0,0 +1,64 @@ +package openapiimport + +import ( + "strconv" + + "github.com/pb33f/go-yaml" +) + +// numericPosition identifies an authored scalar in the single admitted source +// document. External references are rejected before normalization. +type numericPosition struct { + line, column int +} + +// numericSourcePositions binds pristine tokens before the parser constructs +// semantic reference views. Copied or inherited nodes retain their source +// coordinates even when their occurrence's JSON Pointer changes. +func numericSourcePositions(root *yaml.Node, sources map[string]numericSource) map[numericPosition]numericSource { + positions := make(map[numericPosition]numericSource) + var walk func(*yaml.Node, string) + walk = func(node *yaml.Node, path string) { + if node == nil { + return + } + switch node.Kind { + case yaml.DocumentNode: + if len(node.Content) == 1 { + walk(node.Content[0], path) + } + case yaml.MappingNode: + for index := 0; index+1 < len(node.Content); index += 2 { + walk(node.Content[index+1], path+"/"+escapeJSONPointer(node.Content[index].Value)) + } + case yaml.SequenceNode: + for index, child := range node.Content { + walk(child, path+"/"+strconv.Itoa(index)) + } + case yaml.ScalarNode: + if node.Line > 0 && node.Column > 0 { + source, found := sources[path] + if !found { + // Inline YAML merge sources have no effective JSON Pointer + // at their authored location. Capture their pristine scalar. + source = numericSource{text: node.Value, tag: node.Tag, style: node.Style} + } + positions[numericPosition{node.Line, node.Column}] = source + } + } + } + walk(root, "#") + return positions +} + +func (a *analyzer) nodeNumericSource(node *yaml.Node) (numericSource, bool) { + if node == nil { + return numericSource{}, false + } + if source, found := a.numericPositions[numericPosition{node.Line, node.Column}]; found { + return source, true + } + // Programmatic schemas have no original source coordinates. Their own + // node is authoritative; an occurrence path must never supply another value. + return numericSource{text: node.Value, tag: node.Tag, style: node.Style}, true +} diff --git a/internal/openapiimport/object_field_names_test.go b/internal/openapiimport/object_field_names_test.go index 98bae7351..151ea2469 100644 --- a/internal/openapiimport/object_field_names_test.go +++ b/internal/openapiimport/object_field_names_test.go @@ -79,7 +79,7 @@ components: `) document, diagnostics, err := Analyze(source) require.NoError(t, err) - requireDiagnosticCode(t, diagnostics, "schema-allof-flattened") + require.Empty(t, diagnostics) rendered, err := Render(document, Options{}) require.NoError(t, err) requireRenderedDesignGenerates(t, rendered) @@ -133,7 +133,14 @@ func TestImportedObjectFieldFamilyGenerates(t *testing.T) { {"Derived", []string{"type", "error", "@type", "type2"}}, {"Failure", []string{"type", "error", "@type", "errorField", "errorField2", "loomErrorName", "@loomErrorName"}}, } { - properties := schemas[tc.name].(map[string]any)["properties"].(map[string]any) + schema := schemas[tc.name].(map[string]any) + if tc.name == "Derived" || tc.name == "Failure" { + branches := schema["allOf"].([]any) + require.Len(t, branches, 2) + require.Equal(t, "#/components/schemas/Base", branches[0].(map[string]any)["$ref"]) + schema = branches[1].(map[string]any) + } + properties := schema["properties"].(map[string]any) require.ElementsMatch(t, tc.keys, objectPropertyKeys(properties)) } } diff --git a/internal/openapiimport/plan.go b/internal/openapiimport/plan.go index 89162cf96..bea72a4d5 100644 --- a/internal/openapiimport/plan.go +++ b/internal/openapiimport/plan.go @@ -18,6 +18,7 @@ const ( // documentPlan is the render-ready result of applying the importer subset. type documentPlan struct { + document *Document operations []operationPlan diagnostics Diagnostics schemas map[string]NamedSchema @@ -32,10 +33,13 @@ func planDocument(document *Document) documentPlan { if document == nil { return documentPlan{diagnostics: Diagnostics{{Code: "document", Path: "#", Message: "document is nil"}}} } + resolved, diagnostics := resolveReusableDocument(document) + document = resolved planner := documentPlanner{ - document: document, - schemas: make(map[string]NamedSchema, len(document.Components.Schemas)), - used: make(map[string]struct{}), + diagnostics: diagnostics, + document: document, + schemas: make(map[string]NamedSchema, len(document.Components.Schemas)), + used: make(map[string]struct{}), } planner.index() if document.SecurityDefined { @@ -63,15 +67,6 @@ func planDocument(document *Document) documentPlan { planner.unsupported("unused-component-parameter", path, "unreferenced parameter components cannot be rendered faithfully") } } - for _, named := range document.Components.RequestBodies { - planner.unsupported("component-request-body", "#/components/requestBodies/"+escapeJSONPointer(named.Name), "reusable request body components are not in the strict import subset") - } - for _, named := range document.Components.Responses { - planner.unsupported("component-response", "#/components/responses/"+escapeJSONPointer(named.Name), "reusable response components are not in the strict import subset") - } - for _, named := range document.Components.Headers { - planner.unsupported("component-header", "#/components/headers/"+escapeJSONPointer(named.Name), "reusable header components are not in the strict import subset") - } if len(planner.diagnostics) > 0 { return documentPlan{diagnostics: planner.diagnostics.sorted()} } @@ -82,7 +77,7 @@ func planDocument(document *Document) documentPlan { // Error clones depend on final operation names; reference occurrences depend on // those clones, so neither can be independently rediscovered during rendering. func (p *documentPlanner) renderPlan() documentPlan { - plan := documentPlan{} + plan := documentPlan{document: p.document} operationRenderer := renderer{document: p.document, schemas: p.schemas} for index := range p.document.Operations { operation, err := operationRenderer.planOperation(&p.document.Operations[index], fmt.Sprintf("#/operations/%d", index)) @@ -225,7 +220,7 @@ func (p *documentPlanner) requestBody(body *RequestBody, path string) { return } if body.Ref != "" { - p.unsupported("request-body-reference", path, "request body references are not renderable") + // resolveReusableDocument already reported this unresolved occurrence. return } if len(body.ContentTypes) == 0 { @@ -259,7 +254,7 @@ func (p *documentPlanner) requestBody(body *RequestBody, path string) { func (p *documentPlanner) response(response Response, path string, allowNonJSON bool) { if response.Ref != "" { - p.unsupported("response-reference", path, "response references are not renderable") + // resolveReusableDocument already reported this unresolved occurrence. return } p.content(response.ContentType, response.Schema, path, allowNonJSON, responseObjectRole(response, allowNonJSON)) @@ -269,7 +264,7 @@ func (p *documentPlanner) response(response Response, path string, allowNonJSON p.unsupported("wire-name", headerPath, "response header names containing ':' are not renderable") } if named.Header.Ref != "" { - p.unsupported("header-reference", headerPath, "header references are not renderable") + // Reference resolution owns this diagnostic. continue } if named.Header.Schema == nil { @@ -442,13 +437,11 @@ func (p *documentPlanner) schemaAdmission(schema *Schema, path string, role obje p.unsupported("schema-object-constructor", path, "inline object requires a named type in this constructor; analyze source schemas before rendering") admitted = false } - if diagnostic := schema.valueDomainError; diagnostic != nil { - p.unsupported(diagnostic.Code, diagnostic.Path, diagnostic.Message) - return false - } - if diagnostic := schema.patternError; diagnostic != nil { - p.unsupported(diagnostic.Code, diagnostic.Path, diagnostic.Message) - return false + for _, diagnostic := range []*Diagnostic{schema.valueDomainError, schema.referenceError, schema.patternError} { + if diagnostic != nil { + p.unsupported(diagnostic.Code, diagnostic.Path, diagnostic.Message) + return false + } } if schema.Boolean != nil && (!*schema.Boolean || strings.HasPrefix(p.document.OpenAPIVersion, "3.0.")) { p.unsupported("schema-boolean", path, "only true boolean schemas in OpenAPI 3.1 and 3.2 are in the strict import subset") diff --git a/internal/openapiimport/plan_reusable.go b/internal/openapiimport/plan_reusable.go new file mode 100644 index 000000000..3f5e20d97 --- /dev/null +++ b/internal/openapiimport/plan_reusable.go @@ -0,0 +1,137 @@ +package openapiimport + +import ( + "fmt" + "regexp" + "slices" + + "github.com/CaliLuke/loom/internal/openapiref" +) + +type reusableIndex[T any] struct { + kind string + values map[string]T + used map[string]bool + reference func(T) openapiref.Object +} + +type reusableIndexes struct { + bodies reusableIndex[RequestBody] + responses reusableIndex[Response] + headers reusableIndex[Header] +} + +var reusableComponentName = regexp.MustCompile(`^[a-zA-Z0-9._-]+$`) + +// resolveReusableDocument creates occurrence copies before admission. Shared +// definitions remain unchanged; reference edges retain their own annotations. +func resolveReusableDocument(document *Document) (*Document, Diagnostics) { + indexes := indexReusableComponents(document.Components) + result := *document + result.Operations = slices.Clone(document.Operations) + var diagnostics Diagnostics + fail := func(code, path string, err error) { + diagnostics = append(diagnostics, Diagnostic{Code: code, Path: path, Message: err.Error()}) + } + for index := range result.Operations { + operation := &result.Operations[index] + path := fmt.Sprintf("#/operations/%d", index) + if operation.RequestBody != nil { + body, chain, err := indexes.bodies.resolve(*operation.RequestBody) + if err != nil { + fail("request-body-reference", path+"/requestBody", err) + } else { + body.Ref, body.referenceChain = "", chain + operation.RequestBody = &body + } + } + operation.Responses = slices.Clone(operation.Responses) + for ri := range operation.Responses { + status := &operation.Responses[ri] + responsePath := path + "/responses/" + status.Status + response, chain, err := indexes.responses.resolve(status.Response) + if err != nil { + fail("response-reference", responsePath, err) + continue + } + response.Ref, response.referenceChain = "", chain + response.Headers = slices.Clone(response.Headers) + for hi := range response.Headers { + named := &response.Headers[hi] + header, chain, err := indexes.headers.resolve(named.Header) + if err != nil { + fail("header-reference", responsePath+"/headers/"+escapeJSONPointer(named.Name), err) + continue + } + header.Ref, header.referenceChain = "", chain + named.Header = header + } + status.Response = response + } + } + diagnostics = append(diagnostics, indexes.bodies.unused("component-request-body")...) + diagnostics = append(diagnostics, indexes.responses.unused("component-response")...) + diagnostics = append(diagnostics, indexes.headers.unused("component-header")...) + return &result, diagnostics.sorted() +} + +func (index reusableIndex[T]) resolve(value T) (T, []openapiref.Object, error) { + var chain []openapiref.Object + seen := make(map[string]bool) + for { + reference := index.reference(value) + if reference.Ref == "" { + return value, chain, nil + } + name, err := openapiref.Name(reference.Ref, "#/components/"+index.kind+"/") + if err != nil { + return value, nil, fmt.Errorf("%s reference %q %w", index.kind, reference.Ref, err) + } + if !reusableComponentName.MatchString(name) { + return value, nil, fmt.Errorf("invalid %s component name %q", index.kind, name) + } + if seen[name] { + return value, nil, fmt.Errorf("cyclic %s reference %q", index.kind, name) + } + seen[name], index.used[name] = true, true + target, exists := index.values[name] + if !exists { + return value, nil, fmt.Errorf("%s reference %q does not resolve", index.kind, reference.Ref) + } + reference.Ref = "#/components/" + index.kind + "/" + name + chain = append(chain, reference) + value = target + } +} + +func (index reusableIndex[T]) unused(code string) Diagnostics { + var diagnostics Diagnostics + for name := range index.values { + if !index.used[name] { + diagnostics = append(diagnostics, Diagnostic{Code: code, Path: "#/components/" + index.kind + "/" + escapeJSONPointer(name), Message: "unreferenced reusable components cannot be rendered faithfully"}) + } + } + return diagnostics +} + +func indexReusableComponents(components Components) reusableIndexes { + bodies := reusableIndex[RequestBody]{"requestBodies", map[string]RequestBody{}, map[string]bool{}, func(v RequestBody) openapiref.Object { + return openapiref.Object{Ref: v.Ref, Description: v.ReferenceDescription, Summary: v.ReferenceSummary} + }} + responses := reusableIndex[Response]{"responses", map[string]Response{}, map[string]bool{}, func(v Response) openapiref.Object { + return openapiref.Object{Ref: v.Ref, Description: v.ReferenceDescription, Summary: v.ReferenceSummary} + }} + headers := reusableIndex[Header]{"headers", map[string]Header{}, map[string]bool{}, func(v Header) openapiref.Object { + return openapiref.Object{Ref: v.Ref, Description: v.ReferenceDescription, Summary: v.ReferenceSummary} + }} + for _, named := range components.RequestBodies { + bodies.values[named.Name] = named.RequestBody + } + for _, named := range components.Responses { + responses.values[named.Name] = named.Response + } + for _, named := range components.Headers { + headers.values[named.Name] = named.Header + } + return reusableIndexes{bodies: bodies, responses: responses, headers: headers} +} diff --git a/internal/openapiimport/plan_reusable_test.go b/internal/openapiimport/plan_reusable_test.go new file mode 100644 index 000000000..7a287a2d9 --- /dev/null +++ b/internal/openapiimport/plan_reusable_test.go @@ -0,0 +1,79 @@ +package openapiimport + +import ( + "strings" + "testing" + + "github.com/CaliLuke/loom/internal/openapiref" + "github.com/stretchr/testify/require" +) + +func TestReusableReferencePlanning(t *testing.T) { + for _, kind := range []string{"requestBodies", "responses", "headers"} { + t.Run(kind, func(t *testing.T) { + prefix := "#/components/" + kind + "/" + description := "base" + values := map[string]openapiref.Object{ + "Alias": {Ref: prefix + "ZTarget", Summary: new("alias")}, + "ZTarget": {Description: &description}, + "Cycle": {Ref: prefix + "Cycle"}, + } + index := reusableIndex[openapiref.Object]{kind, values, map[string]bool{}, func(value openapiref.Object) openapiref.Object { + return value + }} + for _, tc := range []struct { + name, ref string + rejected bool + }{ + {"forward alias", prefix + "Alias", false}, + {"encoded target", prefix + "%5ATarget", false}, + {"missing", prefix + "Absent", true}, + {"cycle", prefix + "Cycle", true}, + {"wrong kind", "#/components/schemas/ZTarget", true}, + {"invalid key", prefix + "bad~1name", true}, + } { + t.Run(tc.name, func(t *testing.T) { + value, chain, err := index.resolve(openapiref.Object{Ref: tc.ref, Description: new("")}) + if tc.rejected { + require.Error(t, err) + return + } + require.NoError(t, err) + require.Equal(t, "base", *value.Description) + require.Equal(t, "", *chain[0].Description) + require.Equal(t, prefix+"ZTarget", chain[len(chain)-1].Ref) + require.Equal(t, "base", description) + require.Equal(t, prefix+"ZTarget", values["Alias"].Ref) + }) + } + }) + } +} + +func TestReusableReferenceAnnotationVersions(t *testing.T) { + for _, version := range []string{"3.0.4", "3.1.2", "3.2.0"} { + source := []byte("openapi: " + version + ` +info: {title: References, version: "1"} +paths: + /value: + get: + responses: + "204": {$ref: '#/components/responses/Done', description: '', summary: 'local'} +components: + responses: + Done: {description: 'base'} +`) + document, diagnostics, err := Analyze(source) + require.NoError(t, err) + require.Empty(t, diagnostics) + occurrence := document.Operations[0].Responses[0].Response + if strings.HasPrefix(version, "3.0.") { + require.Nil(t, occurrence.ReferenceDescription) + require.Nil(t, occurrence.ReferenceSummary) + } else { + require.Equal(t, new(""), occurrence.ReferenceDescription) + require.Equal(t, new("local"), occurrence.ReferenceSummary) + } + require.Equal(t, "base", document.Components.Responses[0].Response.Description) + } +} diff --git a/internal/openapiimport/reference_object_siblings_test.go b/internal/openapiimport/reference_object_siblings_test.go new file mode 100644 index 000000000..4c1d909c6 --- /dev/null +++ b/internal/openapiimport/reference_object_siblings_test.go @@ -0,0 +1,121 @@ +package openapiimport + +import ( + "encoding/json/v2" + "strings" + "testing" + + "github.com/stretchr/testify/require" +) + +func TestReferenceObjectSiblingProperties(t *testing.T) { + for _, name := range []string{"inline field", "named field"} { + t.Run(name, func(t *testing.T) { + source := []byte(`openapi: 3.2.0 +info: {title: Reference contracts, version: "1"} +paths: + /products: + post: + operationId: createProduct + requestBody: + required: true + content: + application/json: + schema: + type: object + properties: + original: {$ref: '#/components/schemas/Value'} + refined: + $ref: '#/components/schemas/Value' + properties: + name: {minLength: 3} + enabled: {type: boolean} + required: [name] + required: [original, refined] + responses: {"204": {description: Accepted}} +components: + schemas: + Value: + type: object + properties: + id: {type: string, minLength: 1} + name: {type: string, maxLength: 8} + required: [id] +`) + if name == "named field" { + source = []byte(strings.Replace(string(source), "name: {type: string, maxLength: 8}", "name: {$ref: '#/components/schemas/Name'}", 1) + " Name: {type: string, maxLength: 8}\n") + } + document, diagnostics, err := Analyze(source) + require.NoError(t, err) + require.Empty(t, diagnostics) + rendered, err := Render(document, Options{PackageName: "design"}) + require.NoError(t, err) + module := requireRenderedDesignGenerates(t, rendered) + validateReferenceSpec(t, module) + validateReferenceRuntime(t, module, "object properties") + contract := readGeneratedOpenAPIContract(t, module) + operation := operationFromImportedSpec(t, contract, "/products", "post") + body := operation["requestBody"].(map[string]any)["content"].(map[string]any)["application/json"].(map[string]any)["schema"].(map[string]any) + if ref, ok := body["$ref"].(string); ok { + body = referencedSchema(t, contract, ref) + } + refined := body["properties"].(map[string]any)["refined"].(map[string]any) + if ref, ok := refined["$ref"].(string); ok { + refined = referencedSchema(t, contract, ref) + } + branches := refined["allOf"].([]any) + require.Equal(t, "#/components/schemas/Value", branches[0].(map[string]any)["$ref"]) + base := referencedSchema(t, contract, "#/components/schemas/Value") + require.NotContains(t, base["properties"].(map[string]any)["name"], "minLength") + // These generated document-level fields have separate import tickets; + // the schema reference graph is retained intact for this round trip. + delete(contract, "$self") + delete(contract, "jsonSchemaDialect") + delete(contract, "servers") + wire, err := json.Marshal(contract, json.Deterministic(true)) + require.NoError(t, err) + reimported, diagnostics, err := Analyze(wire) + require.NoError(t, err) + require.Empty(t, diagnostics) + design, err := Render(reimported, Options{PackageName: "design"}) + require.NoError(t, err) + validateReferenceRuntime(t, requireRenderedDesignGenerates(t, design), "object properties") + }) + } +} + +func TestReferenceObjectAdmission(t *testing.T) { + for _, tc := range []struct { + name, base, siblings string + accepted bool + }{ + {"closed existing property", `{type: object, additionalProperties: false, properties: {name: {type: string}}}`, `properties: {name: {minLength: 2}}`, true}, + {"closed sibling complete keys", `{type: object, properties: {name: {type: string}}}`, `properties: {name: {minLength: 2}} + additionalProperties: false`, true}, + {"closed sibling loses key", `{type: object, properties: {id: {type: string}, name: {type: string}}}`, `properties: {name: {minLength: 2}} + additionalProperties: false`, false}, + {"free map", `{type: object}`, `properties: {name: {type: string}}`, false}, + {"changed field type", `{type: object, properties: {name: {type: string}}}`, `properties: {name: {type: integer}}`, false}, + } { + t.Run(tc.name, func(t *testing.T) { + source := []byte(`openapi: 3.2.0 +info: {title: References, version: "1"} +paths: {} +components: + schemas: + Base: ` + tc.base + ` + Refined: + $ref: '#/components/schemas/Base' + ` + tc.siblings + "\n") + document, diagnostics, err := Analyze(source) + require.NoError(t, err) + if tc.accepted { + require.Empty(t, diagnostics) + } else { + requireDiagnosticCode(t, diagnostics, "schema-reference-siblings") + } + _, err = Render(document, Options{}) + require.Equal(t, tc.accepted, err == nil, "%v", err) + }) + } +} diff --git a/internal/openapiimport/reference_siblings_test.go b/internal/openapiimport/reference_siblings_test.go new file mode 100644 index 000000000..a2b220664 --- /dev/null +++ b/internal/openapiimport/reference_siblings_test.go @@ -0,0 +1,176 @@ +package openapiimport + +import ( + "fmt" + "os" + "os/exec" + "path/filepath" + "testing" + + "github.com/CaliLuke/libopenapi" + + "github.com/stretchr/testify/require" +) + +func TestReferenceSiblingAssertionsAndAnnotations(t *testing.T) { + for _, tc := range []struct { + name, declaration, siblings, keyword string + want any + }{ + {"string bounds", `{type: string, minLength: 3, pattern: '^[a-z]+$'}`, `maxLength: 8`, "maxLength", float64(8)}, + {"numeric bounds", `{type: integer, minimum: 2}`, `maximum: 10`, "maximum", float64(10)}, + {"membership", `{type: string, enum: [draft, ready, shipped]}`, `enum: [ready, shipped]`, "enum", []any{"ready", "shipped"}}, + {"array bounds", `{type: array, items: {type: string}, minItems: 1}`, `maxItems: 4`, "maxItems", float64(4)}, + {"required properties", `{type: object, properties: {id: {type: string}, name: {type: string}}, required: [id]}`, `required: [name]`, "required", []any{"name"}}, + {"format", `{type: string}`, `format: uuid`, "format", "uuid"}, + {"default", `{type: integer, minimum: 2}`, `default: 3`, "default", float64(3)}, + {"nullable enum excludes null", `{type: [string, 'null'], enum: [a]}`, `minLength: 1`, "minLength", float64(1)}, + {"annotations", `{type: string, description: Original}`, `description: Occurrence`, "description", "Occurrence"}, + } { + t.Run(tc.name, func(t *testing.T) { + source := []byte(`openapi: 3.2.0 +info: {title: Reference contracts, version: "1"} +paths: + /products: + post: + operationId: createProduct + requestBody: + required: true + content: + application/json: + schema: + type: object + properties: + original: {$ref: '#/components/schemas/Value'} + refined: + $ref: '#/components/schemas/Value' + ` + tc.siblings + ` + required: [original, refined] + responses: {"204": {description: Accepted}} +components: + schemas: + Value: ` + tc.declaration + "\n") + document, diagnostics, err := Analyze(source) + require.NoError(t, err) + require.Empty(t, diagnostics) + rendered, err := Render(document, Options{PackageName: "design"}) + require.NoError(t, err) + module := requireRenderedDesignGenerates(t, rendered) + contract := readGeneratedOpenAPIContract(t, module) + validateReferenceSpec(t, module) + validateReferenceRuntime(t, module, tc.name) + op := operationFromImportedSpec(t, contract, "/products", "post") + body := op["requestBody"].(map[string]any) + if ref, ok := body["$ref"].(string); ok { + body = contract["components"].(map[string]any)["requestBodies"].(map[string]any)[ref[len("#/components/requestBodies/"):]].(map[string]any) + } + schema := body["content"].(map[string]any)["application/json"].(map[string]any)["schema"].(map[string]any) + if ref, ok := schema["$ref"].(string); ok { + schema = referencedSchema(t, contract, ref) + } + properties := schema["properties"].(map[string]any) + original := properties["original"].(map[string]any) + refined := properties["refined"].(map[string]any) + require.Equal(t, "#/components/schemas/Value", original["$ref"]) + require.Equal(t, original["$ref"], refined["$ref"]) + require.Equal(t, tc.want, refined[tc.keyword]) + base := referencedSchema(t, contract, original["$ref"].(string)) + require.NotEqual(t, tc.want, base[tc.keyword], "occurrence must not mutate its base") + }) + } +} + +func TestReferenceSiblingsUnsupportedIntersectionsFailClosed(t *testing.T) { + for _, tc := range []struct{ name, declaration, siblings string }{ + {"incompatible type", `{type: string}`, `type: integer`}, + {"nullable narrowing", `{type: [string, 'null']}`, `type: string`}, + {"nullable membership narrowing", `{type: [string, 'null']}`, `enum: [yes]`}, + {"closed object addition", `{type: object, additionalProperties: false, properties: {name: {type: string}}}`, `properties: {extra: {type: string}}`}, + {"array item composition", `{type: array, items: {type: string}}`, `items: {maxLength: 8}`}, + {"object closure composition", `{type: object, properties: {name: {type: string}}}`, `additionalProperties: false`}, + {"composition sibling", `{type: string}`, `allOf: [{maxLength: 8}]`}, + } { + t.Run(tc.name, func(t *testing.T) { + source := []byte(`openapi: 3.2.0 +info: {title: Reference contracts, version: "1"} +paths: {} +components: + schemas: + Base: ` + tc.declaration + ` + Refined: + $ref: '#/components/schemas/Base' + ` + tc.siblings + "\n") + document, diagnostics, err := Analyze(source) + require.NoError(t, err) + requireDiagnosticCode(t, diagnostics, "schema-reference-siblings") + fatal, _ := diagnostics.Classify(true) + require.NotEmpty(t, fatal) + _, err = Render(document, Options{PackageName: "design"}) + require.Error(t, err) + }) + } +} + +func validateReferenceSpec(t *testing.T, module string) { + t.Helper() + path := filepath.Join(module, "gen", "http", "openapi.json") + before, err := os.ReadFile(path) + require.NoError(t, err) + parsed, err := libopenapi.NewDocument(before) + require.NoError(t, err) + _, err = parsed.BuildV3Model() + require.NoError(t, err) + generate := exec.Command(filepath.Join(module, "loom"), "gen", "example.com/imported/design", "-o", ".") + generate.Dir = module + output, err := generate.CombinedOutput() + require.NoError(t, err, string(output)) + after, err := os.ReadFile(path) + require.NoError(t, err) + require.Equal(t, before, after, "independent generation preserves contract bytes") +} + +func validateReferenceRuntime(t *testing.T, module, specimen string) { + t.Helper() + cases := map[string][]string{ + "inherited integer items": {`{"original":{"ns":[1]},"refined":{"ns":[1,2]}}`, `{"original":{"ns":[3]},"refined":{"ns":[1]}}`, `{"original":{"ns":[1]},"refined":{"ns":[1,2,1]}}`, `{"original":{"ns":[1]},"refined":{"ns":[3]}}`}, + "object properties": {`{"original":{"id":"a"},"refined":{"id":"b","name":"abc","enabled":true}}`, `{"original":{"id":""},"refined":{"id":"b","name":"abc"}}`, `{"original":{"id":"a"},"refined":{"id":"b","name":"a"}}`, `{"original":{"id":"a"},"refined":{"id":"b","name":"abcdefghi"}}`}, + "nullable enum excludes null": {`{"original":"a","refined":"a"}`, `{"original":"a","refined":null}`, `{"original":null,"refined":"a"}`}, + "string bounds": {`{"original":"abc","refined":"abcd"}`, `{"original":"a","refined":"abcd"}`, `{"original":"abc","refined":"abcdefghi"}`}, + "numeric bounds": {`{"original":2,"refined":5}`, `{"original":1,"refined":5}`, `{"original":2,"refined":11}`}, + "membership": {`{"original":"draft","refined":"ready"}`, `{"original":"other","refined":"ready"}`, `{"original":"draft","refined":"draft"}`}, + "array bounds": {`{"original":["a"],"refined":["b"]}`, `{"original":[],"refined":["b"]}`, `{"original":["a"],"refined":["a","b","c","d","e"]}`}, + "required properties": {`{"original":{"id":"a"},"refined":{"id":"b","name":""}}`, `{"original":{},"refined":{"id":"b","name":""}}`, `{"original":{"id":"a"},"refined":{"id":"b"}}`}, + "format": {`{"original":"anything","refined":"123e4567-e89b-12d3-a456-426614174000"}`, `{"original":"anything","refined":"bad"}`}, + "default": {`{"original":2,"refined":3}`, `{"original":1,"refined":3}`, `{"original":2,"refined":1}`}, + "annotations": {`{"original":"base","refined":"local"}`}, + } + wires, ok := cases[specimen] + require.True(t, ok) + source := `package imported_test +import ( + "testing" + "encoding/json/v2" + server "example.com/imported/gen/http/reference_contracts/server" +) +func TestReferenceContract(t *testing.T) { +` + for index, wire := range wires { + source += fmt.Sprintf(`{ + var body server.CreateProductRequestBody + if err := json.Unmarshal([]byte(%q), &body); err != nil { + t.Fatal(err) + } + err := server.ValidateCreateProductRequestBody(&body) + if (err == nil) != %t { + t.Errorf("case %%d: unexpected validation %%v", %d, err) + } +} +`, wire, index == 0, index) + } + source += "}\n" + require.NoError(t, os.WriteFile(filepath.Join(module, "reference_test.go"), []byte(source), 0o600)) + command := exec.Command("go", "test", "-mod=mod", ".") + command.Dir = module + output, err := command.CombinedOutput() + require.NoError(t, err, string(output)) +} diff --git a/internal/openapiimport/reference_source_ownership_test.go b/internal/openapiimport/reference_source_ownership_test.go new file mode 100644 index 000000000..6bff9e17c --- /dev/null +++ b/internal/openapiimport/reference_source_ownership_test.go @@ -0,0 +1,115 @@ +package openapiimport + +import ( + "testing" + + "github.com/pb33f/go-yaml" + + "github.com/stretchr/testify/require" +) + +func TestReferenceAllOfBranchSiblingsFailClosed(t *testing.T) { + for _, extra := range []string{"", `, {type: object, properties: {enabled: {type: boolean}}}`} { + source := []byte(`openapi: 3.2.0 +info: {title: References, version: "1"} +paths: {} +components: + schemas: + Base: {type: object, properties: {name: {type: string, maxLength: 8}}} + Refined: + allOf: [{$ref: '#/components/schemas/Base', type: object, properties: {name: {type: string, minLength: 3}}, required: [name]}` + extra + "]\n") + document, diagnostics, err := Analyze(source) + require.NoError(t, err) + requireDiagnosticCode(t, diagnostics, "schema-composition") + _, err = Render(document, Options{}) + require.Error(t, err) + } +} + +func TestReferenceInheritedNumericNullMembership(t *testing.T) { + source := []byte(`openapi: 3.2.0 +info: {title: References, version: "1"} +paths: {} +components: + schemas: + Base: {type: object, properties: {n: {type: [integer, 'null'], enum: [null, 1]}}} + Refined: + $ref: '#/components/schemas/Base' + properties: + n: {enum: [1]} +`) + document, diagnostics, err := Analyze(source) + require.NoError(t, err) + requireDiagnosticCode(t, diagnostics, "schema-reference-siblings") + _, err = Render(document, Options{}) + require.Error(t, err) +} + +func TestReferenceInheritedArrayNumericSource(t *testing.T) { + source := []byte(`openapi: 3.2.0 +info: {title: Reference contracts, version: "1"} +paths: + /products: + post: + operationId: createProduct + requestBody: + required: true + content: + application/json: + schema: + type: object + properties: + original: {$ref: '#/components/schemas/Base'} + refined: + $ref: '#/components/schemas/Base' + properties: + ns: {maxItems: 2} + required: [original, refined] + responses: {"204": {description: Accepted}} +components: + schemas: + Base: + type: object + properties: + ns: {type: array, items: {type: integer, enum: [1, 2], minimum: 1}} +`) + document, diagnostics, err := Analyze(source) + require.NoError(t, err) + require.Empty(t, diagnostics) + design, err := Render(document, Options{}) + require.NoError(t, err) + require.Contains(t, string(design), "Enum(1, 2)") + require.Contains(t, string(design), `Minimum("1")`) + module := requireRenderedDesignGenerates(t, design) + validateReferenceSpec(t, module) + validateReferenceRuntime(t, module, "inherited integer items") +} + +func TestNumericSourceIdentitySurvivesRelocation(t *testing.T) { + for _, source := range []string{ + `{"value":9007199254740993,"quoted":"9007199254740993"}`, + "value: 9007199254740993\nquoted: \"9007199254740993\"\n", + "<<: {value: 9007199254740993, quoted: \"9007199254740993\"}\n", + } { + values, err := numericSources([]byte(source)) + require.NoError(t, err) + var root yaml.Node + require.NoError(t, yaml.Unmarshal([]byte(source), &root)) + owner := analyzer{numericPositions: numericSourcePositions(&root, values)} + members, err := effectiveYAMLMapping(root.Content[0], make(map[*yaml.Node]bool)) + require.NoError(t, err) + for _, key := range []string{"value", "quoted"} { + node := *members[key] + node.Value, node.Tag = "0", "!!int" + token, found := owner.nodeNumericSource(&node) + require.True(t, found) + number, err := token.number() + if key == "quoted" { + require.Error(t, err) + } else { + require.NoError(t, err) + require.Equal(t, "9007199254740993", number.String()) + } + } + } +} diff --git a/internal/openapiimport/render.go b/internal/openapiimport/render.go index 98aafc866..c4f4f3826 100644 --- a/internal/openapiimport/render.go +++ b/internal/openapiimport/render.go @@ -39,6 +39,7 @@ func Render(document *Document, options Options) ([]byte, error) { if len(plan.diagnostics) > 0 { return nil, fmt.Errorf("render OpenAPI design: cannot preserve the input contract:\n%s", plan.diagnostics.Error()) } + document = plan.document r := renderer{ document: document, schemas: plan.schemas, diff --git a/internal/openapiimport/render_composition_test.go b/internal/openapiimport/render_composition_test.go index b2797cd00..fc0d23bc5 100644 --- a/internal/openapiimport/render_composition_test.go +++ b/internal/openapiimport/render_composition_test.go @@ -40,12 +40,12 @@ components: document, diagnostics, err := Analyze(source) require.NoError(t, err) - requireDiagnosticCode(t, diagnostics, "schema-allof-flattened") + require.Empty(t, diagnostics) requireNoDiagnosticCode(t, diagnostics, "schema") requireNoDiagnosticCode(t, diagnostics, "schema-composition") fatal, warnings := diagnostics.Classify(true) require.Empty(t, fatal) - require.NotEmpty(t, warnings) + require.Empty(t, warnings) rendered, err := Render(document, Options{PackageName: "design"}) require.NoError(t, err) diff --git a/internal/openapiimport/render_recursive_test.go b/internal/openapiimport/render_recursive_test.go index 51ba7a3db..54a3a7806 100644 --- a/internal/openapiimport/render_recursive_test.go +++ b/internal/openapiimport/render_recursive_test.go @@ -54,14 +54,7 @@ components: ` + tc.schemas + "\n") doc, diagnostics, err := Analyze(source) require.NoError(t, err) - if tc.name == "mixed_inheritance" { - require.Len(t, diagnostics, 1) - requireDiagnosticCode(t, diagnostics, "schema-allof-flattened") - fatal, _ := diagnostics.Classify(true) - require.Empty(t, fatal) - } else { - require.Empty(t, diagnostics) - } + require.Empty(t, diagnostics) design, err := Render(doc, Options{PackageName: "design"}) require.NoError(t, err) requireRenderedDesignEvaluates(t, design, 1) diff --git a/internal/openapiimport/render_reference_contract_test.go b/internal/openapiimport/render_reference_contract_test.go index b6e9c189a..860138d46 100644 --- a/internal/openapiimport/render_reference_contract_test.go +++ b/internal/openapiimport/render_reference_contract_test.go @@ -13,12 +13,12 @@ import ( ) // These expected graphs include the existing export normalization of aliases -// and flattened Extend bases. Other named targets and occurrence constraints stay exact. +// and retained Extend references. Other named targets and occurrence constraints stay exact. var recursiveSchemaGraphs = map[string]string{ "acyclic": `{"Node":{"properties":{"name":{"type":"string"}},"type":"object"}}`, "self_array": `{"Node":{"properties":{"children":{"items":{"$ref":"#/components/schemas/Node"},"type":"array"},"name":{"type":"string"}},"type":"object"}}`, "mutual_map_array": `{"Branch":{"properties":{"children":{"items":{"$ref":"#/components/schemas/Node"},"type":"array"}},"type":"object"},"Node":{"properties":{"branches":{"additionalProperties":{"$ref":"#/components/schemas/Branch"},"type":"object"}},"type":"object"}}`, - "mixed_inheritance": `{"Node":{"properties":{"child":{"$ref":"#/components/schemas/Node"},"name":{"type":"string"}},"type":"object"}}`, + "mixed_inheritance": `{"Base":{"properties":{"child":{"$ref":"#/components/schemas/Node"}},"type":"object"},"Node":{"allOf":[{"$ref":"#/components/schemas/Base"},{"properties":{"child":{"$ref":"#/components/schemas/Node"},"name":{"type":"string"}},"type":"object"}]}}`, "top_array_object": `{"Branch":{"properties":{"child":{"$ref":"#/components/schemas/Node"}},"type":"object"},"Node":{"items":{"$ref":"#/components/schemas/Branch"},"type":"array"}}`, "alias_object": `{"Branch":{"properties":{"child":{"$ref":"#/components/schemas/Branch"}},"type":"object"}}`, "nested_promoted": `{"Node":{"properties":{"children":{"items":{"$ref":"#/components/schemas/NodeChildrenItem"},"type":"array"},"name":{"minLength":2,"type":"string"}},"required":["name"],"type":"object"},"NodeChildrenItem":{"additionalProperties":{"items":{"anyOf":[{"$ref":"#/components/schemas/Node"},{"type":"null"}]},"minItems":1,"type":"array"},"type":"object"}}`, diff --git a/internal/openapiimport/render_request_body.go b/internal/openapiimport/render_request_body.go index 405e02a63..cff2cfa58 100644 --- a/internal/openapiimport/render_request_body.go +++ b/internal/openapiimport/render_request_body.go @@ -30,16 +30,10 @@ func (r *renderer) openAPIRequestBody(body RequestBody, path string) error { for index, contentType := range body.ContentTypes { contentTypes[index] = strconv.Quote(contentType) } - metadata, err := renderedExtensions("requestBody", body.Extensions) + metadata, err := requestBodyMetadata(body) if err != nil { return err } - if body.Description != "" { - metadata = append(metadata, renderedMetadata{ - name: "openapi:description:requestBody", - value: body.Description, - }) - } needsBlock := (!object && r.hasSchemaBlock(schema)) || len(metadata) > 0 prefix := fmt.Sprintf( "OpenAPIRequestBodyTypes(%s, []string{%s}, %t", @@ -119,30 +113,25 @@ func (r *renderer) payloadBody(body *renderedBody, path string) (bool, error) { schema := schemaWithExamples(body.body.Schema, body.body.Examples) if body.mode != requestBodyJSON { if body.envelope { - metadata, err := renderedExtensions("requestBody", body.body.Extensions) + metadata, err := requestBodyMetadata(body.body) if err != nil { return false, err } - if body.body.Description != "" { - metadata = append(metadata, renderedMetadata{name: "openapi:description:requestBody", value: body.body.Description}) - } return body.body.Required, r.attribute(body.field, schema, nil, path, metadata...) } - if err := r.emitExtensions("requestBody", body.body.Extensions); err != nil { + metadata, err := requestBodyMetadata(body.body) + if err != nil { return false, err } - if body.body.Description != "" { - r.line("Meta(%q, %q)", "openapi:description:requestBody", body.body.Description) + for _, item := range metadata { + r.line("Meta(%q, %q)", item.name, item.value) } return false, r.renderRequestTransportBody(schema, path) } - metadata, err := renderedExtensions("requestBody", body.body.Extensions) + metadata, err := requestBodyMetadata(body.body) if err != nil { return false, err } - if body.body.Description != "" { - metadata = append(metadata, renderedMetadata{name: "openapi:description:requestBody", value: body.body.Description}) - } return body.body.Required, r.attribute(body.field, schema, nil, path, metadata...) } @@ -194,13 +183,10 @@ func (r *renderer) renderRequestTransportMapPayload(body *renderedBody, path str if object { return fmt.Errorf("render OpenAPI design: %s dynamic form map resolved to object properties", path) } - metadata, err := renderedExtensions("requestBody", body.body.Extensions) + metadata, err := requestBodyMetadata(body.body) if err != nil { return err } - if body.body.Description != "" { - metadata = append(metadata, renderedMetadata{name: "openapi:description:requestBody", value: body.body.Description}) - } if !r.hasSchemaBlock(schema) && len(metadata) == 0 { r.line("Payload(%s)", expression) return nil diff --git a/internal/openapiimport/render_required.go b/internal/openapiimport/render_required.go new file mode 100644 index 000000000..d0ec12e14 --- /dev/null +++ b/internal/openapiimport/render_required.go @@ -0,0 +1,40 @@ +package openapiimport + +import "fmt" + +func (r *renderer) schemaRequired(schema *Schema, path string) error { + if len(schema.Required) == 0 { + return nil + } + for _, name := range schema.Required { + if !r.schemaDefinesProperty(schema, name, make(map[*Schema]bool)) { + return fmt.Errorf("render OpenAPI design: %s required property %q is not defined", path, name) + } + } + r.quotedCall("Required", schema.Required) + return nil +} + +func (r *renderer) schemaDefinesProperty(schema *Schema, name string, seen map[*Schema]bool) bool { + if schema == nil || seen[schema] { + return false + } + seen[schema] = true + for _, property := range schema.Properties { + if property.Name == name { + return true + } + } + if schema.Ref != "" { + base, err := r.resolveSchemaReference(schema, "required") + if err == nil && r.schemaDefinesProperty(base, name, seen) { + return true + } + } + for _, base := range schema.Bases { + if r.schemaDefinesProperty(base, name, seen) { + return true + } + } + return false +} diff --git a/internal/openapiimport/render_response.go b/internal/openapiimport/render_response.go index a0a7cffa6..e56d604f3 100644 --- a/internal/openapiimport/render_response.go +++ b/internal/openapiimport/render_response.go @@ -5,16 +5,23 @@ import ( "strconv" ) -func headerMetadata(header Header) []renderedMetadata { - if !header.AllowReserved { - return nil +func headerMetadata(header Header) ([]renderedMetadata, error) { + metadata, err := reusableMetadata("header", header.referenceChain) + if err != nil { + return nil, err + } + if header.AllowReserved { + metadata = append(metadata, renderedMetadata{name: "openapi:allowReserved", value: "true"}) } - return []renderedMetadata{{name: "openapi:allowReserved", value: "true"}} + return metadata, nil } func (r *renderer) responseHeaderAttributes(headers []renderedHeader, path string) error { for _, header := range headers { - metadata := headerMetadata(header.header) + metadata, err := headerMetadata(header.header) + if err != nil { + return err + } if err := r.attribute( header.field, header.header.Schema, @@ -65,11 +72,19 @@ func responseMappingPrefix(response renderedResponse, failure bool, path string) } func responseMappingNeedsBlock(response renderedResponse, failure bool) bool { - return failure || response.response.Summary != "" || response.response.Description != "" || + return len(response.response.referenceChain) > 0 || failure || response.response.Summary != "" || response.response.Description != "" || response.response.ContentType != "" || len(response.headers) > 0 || len(response.response.Extensions) > 0 } func (r *renderer) responseMappingMetadata(response renderedResponse, failure bool) error { + metadata, err := reusableMetadata("response", response.response.referenceChain) + if err != nil { + return err + } + for _, item := range metadata { + r.line("Meta(%q, %q)", item.name, item.value) + } + if response.response.Summary != "" { r.line("Meta(%q, %q)", "openapi:summary", response.response.Summary) } diff --git a/internal/openapiimport/render_reusable.go b/internal/openapiimport/render_reusable.go new file mode 100644 index 000000000..afa099b12 --- /dev/null +++ b/internal/openapiimport/render_reusable.go @@ -0,0 +1,33 @@ +package openapiimport + +import ( + "encoding/json/v2" + + "github.com/CaliLuke/loom/internal/openapiref" +) + +func reusableMetadata(kind string, chain []openapiref.Object) ([]renderedMetadata, error) { + if len(chain) == 0 { + return nil, nil + } + value, err := json.Marshal(chain, json.Deterministic(true)) + if err != nil { + return nil, err + } + return []renderedMetadata{{name: "openapi:reference:" + kind, value: string(value)}}, nil +} + +func requestBodyMetadata(body RequestBody) ([]renderedMetadata, error) { + metadata, err := renderedExtensions("requestBody", body.Extensions) + if err != nil { + return nil, err + } + if body.Description != "" || len(body.referenceChain) > 0 { + metadata = append(metadata, renderedMetadata{name: "openapi:description:requestBody", value: body.Description}) + } + references, err := reusableMetadata("requestBody", body.referenceChain) + if err != nil { + return nil, err + } + return append(metadata, references...), nil +} diff --git a/internal/openapiimport/render_schema.go b/internal/openapiimport/render_schema.go index 26291ad66..d404309a1 100644 --- a/internal/openapiimport/render_schema.go +++ b/internal/openapiimport/render_schema.go @@ -273,31 +273,22 @@ func (r *renderer) schemaBlock(schema *Schema, path string, errorType bool, desc if err := r.objectFieldAttributes(schema, path, errorType); err != nil { return err } - if len(schema.Required) > 0 { - properties := make(map[string]struct{}, len(schema.Properties)) - for _, property := range schema.Properties { - properties[property.Name] = struct{}{} - } - for _, name := range schema.Required { - if _, ok := properties[name]; !ok { - return fmt.Errorf("render OpenAPI design: %s required property %q is not defined", path, name) - } - } - r.quotedCall("Required", schema.Required) - } return r.schemaAttributeBlock(schema, path, description) } func (r *renderer) schemaAttributeBlock(schema *Schema, path string, description *string) error { + if err := r.schemaRequired(schema, path); err != nil { + return err + } r.untaggedValidation(schema) r.referenceOverlayMetadata(schema) if err := r.schemaAnnotations(schema, description); err != nil { return err } - if schema.Format == "" && (schema.Type == "integer" || schema.Type == "number") { + if !schema.inheritedFormat && schema.Format == "" && (schema.Type == "integer" || schema.Type == "number") { r.line("Meta(%q, %q)", "openapi:format", "") } - if schema.Type == "string" { + if schema.Type == "string" && !schema.inheritedFormat { representation := stringRepresentation(schema.Format) if representation.raw { r.line("Meta(%q, %q)", "openapi:format", schema.Format) @@ -368,7 +359,8 @@ func (r *renderer) untaggedValidation(schema *Schema) { } func (r *renderer) referenceOverlayMetadata(schema *Schema) { - if schema.Ref != "" && !schema.Nullable && r.hasSchemaBlock(schema) { + if schema.referenceSiblings && len(schema.Bases) > 0 || + schema.Ref != "" && !schema.referenceSiblings && !schema.Nullable && r.hasSchemaBlock(schema) { r.line("Meta(%q, %q)", "openapi:allOf:reference", "true") } } @@ -419,13 +411,13 @@ func (r *renderer) hasSchemaBlock(schema *Schema) bool { schema.AdditionalProperties != nil && schema.AdditionalProperties.Allowed != nil && !*schema.AdditionalProperties.Allowed { return true } - if schema.Type == "string" { + if schema.Type == "string" && !schema.inheritedFormat { representation := stringRepresentation(schema.Format) if representation.raw || representation.formatName != "" { return true } } - if schema.Format == "" && (schema.Type == "integer" || schema.Type == "number") { + if !schema.inheritedFormat && schema.Format == "" && (schema.Type == "integer" || schema.Type == "number") { return true } return schema.Nullable || schema.Deprecated || schema.ReadOnly || schema.WriteOnly || schema.Default != nil || len(schema.Examples) > 0 || diff --git a/internal/openapiimport/render_test.go b/internal/openapiimport/render_test.go index 5c2b5d979..1165ccb95 100644 --- a/internal/openapiimport/render_test.go +++ b/internal/openapiimport/render_test.go @@ -977,9 +977,9 @@ func TestRenderRejectsUnrepresentableModels(t *testing.T) { {name: "unresolved schema reference", edit: func(d *Document) { d.Operations[0].Responses[0].Response.Schema = schemaRef("Missing") }, want: `schema reference "#/components/schemas/Missing" does not resolve`}, - {name: "wrong reference kind", edit: func(d *Document) { + {name: "unresolved request body reference", edit: func(d *Document) { d.Operations[0].RequestBody = &RequestBody{Ref: "#/components/requestBodies/Body"} - }, want: "request body references are not renderable"}, + }, want: `requestBodies reference "#/components/requestBodies/Body" does not resolve`}, {name: "mixed object and map", edit: func(d *Document) { d.Components.Schemas[0].Schema = &Schema{ Type: "object", Properties: []NamedProperty{{Name: "id", Schema: &Schema{Type: "string"}}}, @@ -998,7 +998,7 @@ func TestRenderRejectsUnrepresentableModels(t *testing.T) { }, want: `boolean format "custom" is not renderable`}, {name: "response reference", edit: func(d *Document) { d.Operations[0].Responses[0].Response = Response{Ref: "#/components/responses/OK"} - }, want: "response references are not renderable"}, + }, want: `responses reference "#/components/responses/OK" does not resolve`}, } for _, test := range tests { diff --git a/internal/openapiimport/reusable_components_test.go b/internal/openapiimport/reusable_components_test.go new file mode 100644 index 000000000..866eba930 --- /dev/null +++ b/internal/openapiimport/reusable_components_test.go @@ -0,0 +1,80 @@ +package openapiimport + +import ( + "encoding/json/v2" + "os" + "testing" + + "github.com/pb33f/go-yaml" + "github.com/stretchr/testify/require" +) + +func TestReusableTransportComponentRoundTrip(t *testing.T) { + source, err := os.ReadFile("testdata/supported.yaml") + require.NoError(t, err) + var input map[string]any + require.NoError(t, yaml.Unmarshal(source, &input)) + input["openapi"] = "3.2.0" + components := input["components"].(map[string]any) + paths := input["paths"].(map[string]any) + create := paths["/pets"].(map[string]any)["post"].(map[string]any) + get := paths["/pets/{pet_id}"].(map[string]any)["get"].(map[string]any) + response := get["responses"].(map[string]any)["200"].(map[string]any) + header := response["headers"].(map[string]any)["X-Trace-ID"] + components["headers"] = map[string]any{"Trace": map[string]any{"$ref": "#/components/headers/ZTrace"}, "ZTrace": header} + response["headers"] = map[string]any{"X-Trace-ID": map[string]any{"$ref": "#/components/headers/Trace", "description": ""}} + response["summary"] = "Base success" + components["responses"] = map[string]any{"PetResponse": map[string]any{"$ref": "#/components/responses/ZPetResponse", "summary": "Alias success"}, "ZPetResponse": response} + get["responses"].(map[string]any)["200"] = map[string]any{"$ref": "#/components/responses/PetResponse", "description": "Local success", "summary": ""} + components["requestBodies"] = map[string]any{"PetInput": map[string]any{"$ref": "#/components/requestBodies/ZPetInput"}, "ZPetInput": create["requestBody"]} + create["requestBody"] = map[string]any{"$ref": "#/components/requestBodies/PetInput", "description": ""} + + // A second occurrence inherits both components without local overrides. + otherWire, err := json.Marshal(paths["/pets/{pet_id}"], json.Deterministic(true)) + require.NoError(t, err) + var other map[string]any + require.NoError(t, json.Unmarshal(otherWire, &other)) + otherGet := other["get"].(map[string]any) + otherGet["operationId"] = "pets.getOther" + otherGet["responses"].(map[string]any)["200"] = map[string]any{"$ref": "#/components/responses/PetResponse"} + paths["/other/{pet_id}"] = other + wire, err := json.Marshal(input, json.Deterministic(true)) + require.NoError(t, err) + document, diagnostics, err := Analyze(wire) + require.NoError(t, err) + require.Empty(t, diagnostics) + before, err := json.Marshal(document, json.Deterministic(true)) + require.NoError(t, err) + design, err := Render(document, Options{}) + require.NoError(t, err) + after, err := json.Marshal(document, json.Deterministic(true)) + require.NoError(t, err) + require.Equal(t, before, after) + module := requireRenderedDesignGenerates(t, design) + validateReferenceSpec(t, module) + contract := readGeneratedOpenAPIContract(t, module) + generated := contract["components"].(map[string]any) + for _, tc := range []struct{ kind, alias, target string }{ + {"requestBodies", "PetInput", "ZPetInput"}, + {"responses", "PetResponse", "ZPetResponse"}, + {"headers", "Trace", "ZTrace"}, + } { + entries := generated[tc.kind].(map[string]any) + require.Contains(t, entries, tc.target) + require.Equal(t, "#/components/"+tc.kind+"/"+tc.target, entries[tc.alias].(map[string]any)["$ref"]) + } + bodies := generated["requestBodies"].(map[string]any) + require.Equal(t, "Pet to create.", bodies["ZPetInput"].(map[string]any)["description"]) + responses := generated["responses"].(map[string]any) + require.Equal(t, "Found pet.", responses["ZPetResponse"].(map[string]any)["description"]) + require.Equal(t, "Base success", responses["ZPetResponse"].(map[string]any)["summary"]) + require.Equal(t, "Alias success", responses["PetResponse"].(map[string]any)["summary"]) + generatedPaths := contract["paths"].(map[string]any) + actualCreate := generatedPaths["/pets"].(map[string]any)["post"].(map[string]any) + require.Equal(t, map[string]any{"$ref": "#/components/requestBodies/PetInput", "description": ""}, actualCreate["requestBody"]) + inheritedGet := generatedPaths["/other/{pet_id}"].(map[string]any)["get"].(map[string]any) + require.Equal(t, map[string]any{"$ref": "#/components/responses/PetResponse"}, inheritedGet["responses"].(map[string]any)["200"]) + require.Equal(t, "Request trace identifier.", generated["headers"].(map[string]any)["ZTrace"].(map[string]any)["description"]) + actualGet := generatedPaths["/pets/{pet_id}"].(map[string]any)["get"].(map[string]any) + require.Equal(t, map[string]any{"$ref": "#/components/responses/PetResponse", "description": "Local success", "summary": ""}, actualGet["responses"].(map[string]any)["200"]) +} diff --git a/internal/openapiimport/security.go b/internal/openapiimport/security.go index 450aeb2a1..6ace17173 100644 --- a/internal/openapiimport/security.go +++ b/internal/openapiimport/security.go @@ -98,13 +98,12 @@ func (a *analyzer) securityScheme( return SecurityScheme{}, false } case "oauth2": - flows, scopes, ok := a.oauthFlows(source.Flows, path+"/flows") + flows, ok := a.oauthFlows(source.Flows, path+"/flows") if !ok || source.Scheme != "" || source.BearerFormat != "" || source.In != "" || source.Name != "" || source.OpenIdConnectUrl != "" { a.unsupported("security-scheme", path, "OAuth2 scheme contains fields outside the strict import subset") return SecurityScheme{}, false } scheme.OAuthFlows = flows - scheme.Scopes = scopes default: a.unsupported("security-scheme", path, fmt.Sprintf("security scheme type %q is not in the strict import subset", source.Type)) return SecurityScheme{}, false @@ -149,18 +148,18 @@ func (a *analyzer) supportedHTTPScheme(source *v3.SecurityScheme, path string) b return true } -func (a *analyzer) oauthFlows(source *v3.OAuthFlows, path string) ([]OAuthFlow, []SecurityScope, bool) { +func (a *analyzer) oauthFlows(source *v3.OAuthFlows, path string) ([]OAuthFlow, bool) { if source == nil { a.unsupported("security-scheme", path, "OAuth2 security scheme must define flows") - return nil, nil, false + return nil, false } if source.Device != nil || oauthDeviceAuthorizationPresent(source) { a.unsupported("security-scheme", path+"/deviceAuthorization", "OAuth device authorization import is blocked until the parser exposes the official field") - return nil, nil, false + return nil, false } if orderedmap.Len(source.Extensions) > 0 { a.unsupported("security-scheme", path, "OAuth flow-container extensions are not in the strict import subset") - return nil, nil, false + return nil, false } types := []struct { kind string @@ -172,7 +171,6 @@ func (a *analyzer) oauthFlows(source *v3.OAuthFlows, path string) ([]OAuthFlow, {kind: "clientCredentials", flow: source.ClientCredentials}, } var result []OAuthFlow - var sharedScopes []SecurityScope for _, candidate := range types { if candidate.flow == nil { continue @@ -180,28 +178,23 @@ func (a *analyzer) oauthFlows(source *v3.OAuthFlows, path string) ([]OAuthFlow, flowPath := path + "/" + candidate.kind if orderedmap.Len(candidate.flow.Extensions) > 0 { a.unsupported("security-scheme", flowPath, "OAuth flow extensions are not in the strict import subset") - return nil, nil, false + return nil, false } if !a.validOAuthFlow(candidate.kind, candidate.flow, flowPath) { - return nil, nil, false - } - scopes := oauthScopes(candidate.flow.Scopes) - if sharedScopes == nil { - sharedScopes = scopes - } else if !equalSecurityScopes(sharedScopes, scopes) { - a.unsupported("security-scheme", flowPath+"/scopes", "all OAuth flows must define the same scopes for Loom to preserve them") - return nil, nil, false + return nil, false } + result = append(result, OAuthFlow{ Kind: candidate.kind, AuthorizationURL: candidate.flow.AuthorizationUrl, TokenURL: candidate.flow.TokenUrl, RefreshURL: candidate.flow.RefreshUrl, + Scopes: oauthScopes(candidate.flow.Scopes), }) } if len(result) == 0 { a.unsupported("security-scheme", path, "OAuth2 security scheme must define at least one supported flow") - return nil, nil, false + return nil, false } - return result, sharedScopes, true + return result, true } func oauthDeviceAuthorizationPresent(flows *v3.OAuthFlows) bool { @@ -249,18 +242,6 @@ func (a *analyzer) validOAuthFlow(kind string, flow *v3.OAuthFlow, path string) return true } -func equalSecurityScopes(left, right []SecurityScope) bool { - if len(left) != len(right) { - return false - } - for index := range left { - if left[index] != right[index] { - return false - } - } - return true -} - func (a *analyzer) securityRequirements(source []*base.SecurityRequirement, path string, components Components) SecurityRequirements { requirements := make(SecurityRequirements, 0, len(source)) for index, sourceRequirement := range source { @@ -277,16 +258,7 @@ func (a *analyzer) securityRequirements(source []*base.SecurityRequirement, path a.unsupported("security-requirement", schemePath, fmt.Sprintf("security scheme %q is not a supported component", name)) continue } - if scheme.Type != "oauth2" && len(scopes) > 0 { - a.unsupported("security-requirement", schemePath, "non-OAuth security requirement scopes must be empty") - } - if scheme.Type == "oauth2" { - for _, scope := range scopes { - if !securityScopeDefined(scheme.Scopes, scope) { - a.unsupported("security-requirement", schemePath, fmt.Sprintf("OAuth scope %q is not defined by scheme %q", scope, name)) - } - } - } + normalizedScopes := append([]string(nil), scopes...) sort.Strings(normalizedScopes) requirement.Schemes = append(requirement.Schemes, SecurityRequirementScheme{ @@ -299,26 +271,12 @@ func (a *analyzer) securityRequirements(source []*base.SecurityRequirement, path return requirements } -func equalStrings(left, right []string) bool { - if len(left) != len(right) { - return false - } - want := append([]string(nil), left...) - got := append([]string(nil), right...) - sort.Strings(want) - sort.Strings(got) - for index := range want { - if want[index] != got[index] { - return false - } - } - return true -} - -func securityScopeDefined(scopes []SecurityScope, name string) bool { - for _, scope := range scopes { - if scope.Name == name { - return true +func securityScopeDefined(scheme SecurityScheme, name string) bool { + for _, flow := range scheme.OAuthFlows { + for _, scope := range flow.Scopes { + if scope.Name == name { + return true + } } } return false @@ -363,19 +321,20 @@ func (r *renderer) securityScheme(scheme SecurityScheme) error { for _, flow := range scheme.OAuthFlows { switch flow.Kind { case "authorizationCode": - r.line("AuthorizationCodeFlow(%q, %q, %q)", flow.AuthorizationURL, flow.TokenURL, flow.RefreshURL) + r.open("AuthorizationCodeFlow(%q, %q, %q, func()", flow.AuthorizationURL, flow.TokenURL, flow.RefreshURL) case "implicit": - r.line("ImplicitFlow(%q, %q)", flow.AuthorizationURL, flow.RefreshURL) + r.open("ImplicitFlow(%q, %q, func()", flow.AuthorizationURL, flow.RefreshURL) case "password": - r.line("PasswordFlow(%q, %q)", flow.TokenURL, flow.RefreshURL) + r.open("PasswordFlow(%q, %q, func()", flow.TokenURL, flow.RefreshURL) case "clientCredentials": - r.line("ClientCredentialsFlow(%q, %q)", flow.TokenURL, flow.RefreshURL) + r.open("ClientCredentialsFlow(%q, %q, func()", flow.TokenURL, flow.RefreshURL) default: return fmt.Errorf("render OpenAPI design: security scheme %q has unsupported OAuth flow %q", scheme.Name, flow.Kind) } - } - for _, scope := range scheme.Scopes { - r.line("Scope(%q, %q)", scope.Name, scope.Description) + for _, scope := range flow.Scopes { + r.line("Scope(%q, %q)", scope.Name, scope.Description) + } + r.close() } if err := r.emitExtensions("", scheme.Extensions); err != nil { return err @@ -392,8 +351,7 @@ func (r *renderer) securityRequirements(requirements SecurityRequirements) error continue } arguments := make([]string, 0, len(requirement.Schemes)) - scopes, err := securityRequirementScopes(requirement) - if err != nil { + if err := validateSecurityRequirement(requirement, r.document.Components, r.document.OpenAPIVersion); err != nil { return fmt.Errorf("render OpenAPI design: %w", err) } for _, reference := range requirement.Schemes { @@ -403,37 +361,61 @@ func (r *renderer) securityRequirements(requirements SecurityRequirements) error } arguments = append(arguments, scheme.GoName) } - if len(scopes) == 0 { + hasNames := false + for _, reference := range requirement.Schemes { + hasNames = hasNames || len(reference.Scopes) > 0 + } + if !hasNames { r.line("Security(%s)", strings.Join(arguments, ", ")) continue } r.open("Security(%s, func()", strings.Join(arguments, ", ")) - for _, scope := range scopes { - r.line("Scope(%q)", scope) + for _, reference := range requirement.Schemes { + scheme := securitySchemeByName(r.document.Components.SecuritySchemes, reference.Name) + call := "RoleFor" + if scheme.Type == "oauth2" { + call = "ScopeFor" + } + for _, name := range reference.Scopes { + r.line("%s(%s, %q)", call, scheme.GoName, name) + } } r.close() } return nil } -// securityRequirementScopes admits only a source conjunction that one Loom -// requirement scope list can express without changing any scheme's checks. -func securityRequirementScopes(requirement SecurityRequirement) ([]string, error) { - if len(requirement.Schemes) == 0 { - return nil, nil - } - first := requirement.Schemes[0] - for _, reference := range requirement.Schemes[1:] { - if !equalStrings(first.Scopes, reference.Scopes) { - return nil, fmt.Errorf("security schemes %q and %q in one requirement use different scopes; Loom requires the same scopes for every scheme in a conjunction", first.Name, reference.Name) +// validateSecurityRequirement also checks normalized documents supplied directly +// to Render. Role names have no protocol-defined inventory. +func validateSecurityRequirement(requirement SecurityRequirement, components Components, version string) error { + seen := make(map[string]bool) + for _, reference := range requirement.Schemes { + scheme := securitySchemeByName(components.SecuritySchemes, reference.Name) + if scheme == nil { + return fmt.Errorf("security scheme %q is not defined", reference.Name) + } + if seen[reference.Name] { + return fmt.Errorf("security scheme %q occurs twice in one requirement", reference.Name) + } + seen[reference.Name] = true + if scheme.Type != "oauth2" { + if strings.HasPrefix(version, "3.0.") && len(reference.Scopes) > 0 { + return fmt.Errorf("non-OAuth security requirement arrays must be empty in OpenAPI 3.0") + } + continue + } + for _, scope := range reference.Scopes { + if !securityScopeDefined(*scheme, scope) { + return fmt.Errorf("OAuth scope %q is not defined by scheme %q", scope, reference.Name) + } } } - return first.Scopes, nil + return nil } func (p *documentPlanner) securityRequirements(requirements SecurityRequirements, path string) { for index, requirement := range requirements { - if _, err := securityRequirementScopes(requirement); err != nil { + if err := validateSecurityRequirement(requirement, p.document.Components, p.document.OpenAPIVersion); err != nil { p.unsupported("security-requirement", fmt.Sprintf("%s/%d", path, index), err.Error()) } } diff --git a/internal/openapiimport/security_credentials.go b/internal/openapiimport/security_credentials.go index b6fea7441..828cd2bb4 100644 --- a/internal/openapiimport/security_credentials.go +++ b/internal/openapiimport/security_credentials.go @@ -46,7 +46,7 @@ func (r *renderer) securityParameters(operation *Operation, parameters []rendere seen := make(map[string]bool) slots := make(map[credentialSlot]int) for requirementIndex, requirement := range requirements { - if _, err := securityRequirementScopes(requirement); err != nil { + if err := validateSecurityRequirement(requirement, r.document.Components, r.document.OpenAPIVersion); err != nil { return nil, fmt.Errorf("render OpenAPI design: %s/%d: %w", path, requirementIndex, err) } for schemeIndex, reference := range requirement.Schemes { diff --git a/internal/openapiimport/security_credentials_import_test.go b/internal/openapiimport/security_credentials_import_test.go index 6afb3d1c7..716469726 100644 --- a/internal/openapiimport/security_credentials_import_test.go +++ b/internal/openapiimport/security_credentials_import_test.go @@ -69,7 +69,7 @@ func TestImportedSecurityCredentialsGenerated(t *testing.T) { } } -func TestSecurityCredentialScopePartialImport(t *testing.T) { +func TestSecurityCredentialUnknownScopePartialImport(t *testing.T) { source, err := os.ReadFile("testdata/security_credentials/imported.json") require.NoError(t, err) document, diagnostics, err := Analyze(source) @@ -78,16 +78,16 @@ func TestSecurityCredentialScopePartialImport(t *testing.T) { // Render also rejects a normalized source rather than relying on the parser. document.Operations[0].SecurityDefined = true document.Operations[0].Security = SecurityRequirements{{Schemes: []SecurityRequirementScheme{ - {Name: "Bearer"}, {Name: "OAuth", Scopes: []string{"read"}}, + {Name: "Bearer"}, {Name: "OAuth", Scopes: []string{"unknown"}}, }}} _, err = Render(document, Options{}) - require.ErrorContains(t, err, "use different scopes") + require.ErrorContains(t, err, "not defined by scheme") var input map[string]any require.NoError(t, json.Unmarshal(source, &input)) input["paths"] = map[string]any{ "/bad": map[string]any{"get": map[string]any{ - "security": []any{map[string]any{"Bearer": []any{}, "OAuth": []any{"read"}}}, + "security": []any{map[string]any{"Bearer": []any{}, "OAuth": []any{"unknown"}}}, "responses": map[string]any{"204": map[string]any{"description": "Done"}}, }}, "/good": map[string]any{"get": map[string]any{ @@ -101,7 +101,7 @@ func TestSecurityCredentialScopePartialImport(t *testing.T) { require.NoError(t, err) require.Len(t, partial.Skipped, 1) require.Equal(t, "/bad", partial.Skipped[0].Path) - require.Equal(t, "#/paths/~1bad/get/security/0", partial.Skipped[0].Diagnostics[0].Path) + require.Contains(t, partial.Skipped[0].Diagnostics[0].Path, "#/paths/~1bad/get/security/0") require.Equal(t, "security-requirement", partial.Skipped[0].Diagnostics[0].Code) require.Len(t, partial.Document.Operations, 1) require.Equal(t, "/good", partial.Document.Operations[0].Path) diff --git a/internal/openapiimport/security_credentials_partial_test.go b/internal/openapiimport/security_credentials_partial_test.go index a0bfa29e7..4d512b352 100644 --- a/internal/openapiimport/security_credentials_partial_test.go +++ b/internal/openapiimport/security_credentials_partial_test.go @@ -22,7 +22,7 @@ func TestSecurityCredentialRootScopePartialImport(t *testing.T) { fatal, _ := diagnostics.Classify(lossy) require.NotEmpty(t, fatal, "strict import must reject unsupported root requirements") _, err = Render(document, Options{}) - require.ErrorContains(t, err, "use different scopes") + require.ErrorContains(t, err, "not defined by scheme") partial, _, err := AnalyzePartial(source, Selection{}, lossy) if err != nil { @@ -31,8 +31,8 @@ func TestSecurityCredentialRootScopePartialImport(t *testing.T) { } require.Len(t, partial.Skipped, 1) require.Equal(t, "/inherit", partial.Skipped[0].Path) - require.Len(t, partial.Skipped[0].Diagnostics, 1) - require.Equal(t, "#/security/0", partial.Skipped[0].Diagnostics[0].Path) + require.NotEmpty(t, partial.Skipped[0].Diagnostics) + require.Contains(t, partial.Skipped[0].Diagnostics[0].Path, "#/security/0") require.Equal(t, "security-requirement", partial.Skipped[0].Diagnostics[0].Code) require.Len(t, partial.Document.Operations, len(paths)) require.False(t, partial.Document.SecurityDefined) @@ -44,9 +44,8 @@ func TestSecurityCredentialRootScopePartialImport(t *testing.T) { } else { require.Len(t, operation.Security, 1) require.Len(t, operation.Security[0].Schemes, 2) - for _, scheme := range operation.Security[0].Schemes { - require.Equal(t, []string{"read"}, scheme.Scopes) - } + require.Equal(t, []string{"read"}, operation.Security[0].Schemes[0].Scopes) + require.Empty(t, operation.Security[0].Schemes[1].Scopes) } } _, err = Render(partial.Document, Options{PackageName: "design"}) @@ -96,7 +95,7 @@ func partialCredentialSource(t *testing.T, retained []string) []byte { require.NoError(t, err) var input map[string]any require.NoError(t, json.Unmarshal(source, &input)) - input["security"] = []any{map[string]any{"Bearer": []any{}, "OAuth": []any{"read"}}} + input["security"] = []any{map[string]any{"Bearer": []any{}, "OAuth": []any{"unknown"}}} paths := input["paths"].(map[string]any) selected := map[string]any{"/inherit": paths["/inherit"]} for _, path := range retained { diff --git a/internal/openapiimport/security_credentials_readiness_test.go b/internal/openapiimport/security_credentials_readiness_test.go index 5dc38bea4..f76651e13 100644 --- a/internal/openapiimport/security_credentials_readiness_test.go +++ b/internal/openapiimport/security_credentials_readiness_test.go @@ -127,13 +127,14 @@ func TestSecurityCredentialReadinessScopeAdmission(t *testing.T) { requirement string unsupported bool }{ - {"{Bearer: [], OAuth: [read]}", true}, - {"{OAuth: [read], Bearer: []}", true}, - {"{OAuth: [], Other: [read]}", true}, - {"{OAuth: [read], Other: []}", true}, + {"{Bearer: [], OAuth: [read]}", false}, + {"{OAuth: [read], Bearer: []}", false}, + {"{OAuth: [], Other: [read]}", false}, + {"{OAuth: [read], Other: []}", false}, {"{OAuth: [read], Other: [read]}", false}, {"{Bearer: [], OAuth: []}", false}, {"{Bearer: []}, {OAuth: [read]}", false}, + {"{Bearer: [admin], OAuth: [read]}", false}, } { t.Run(tc.requirement, func(t *testing.T) { source := []byte(`openapi: 3.2.0 @@ -183,7 +184,7 @@ func TestSecurityCredentialRoles(t *testing.T) { }, Components: Components{SecuritySchemes: []SecurityScheme{ {Name: "JWT", GoName: "JWT", Type: "http", Scheme: "bearer"}, - {Name: "OAuth", GoName: "OAuth", Type: "oauth2"}, + {Name: "OAuth", GoName: "OAuth", Type: "oauth2", OAuthFlows: []OAuthFlow{{Kind: "clientCredentials", TokenURL: "https://example.com/token", Scopes: []SecurityScope{{Name: "read"}}}}}, }}, } before, err := json.Marshal(document, json.Deterministic(true)) diff --git a/internal/openapiimport/security_flow_scopes_test.go b/internal/openapiimport/security_flow_scopes_test.go new file mode 100644 index 000000000..07d1ecad1 --- /dev/null +++ b/internal/openapiimport/security_flow_scopes_test.go @@ -0,0 +1,48 @@ +package openapiimport + +import ( + "testing" + + "github.com/stretchr/testify/require" +) + +func TestIndependentOAuthFlowScopes(t *testing.T) { + source := []byte(`openapi: 3.2.0 +info: {title: Flow inventories, version: "1"} +security: [{Auth: [read, write]}] +paths: + /value: + get: + responses: {"204": {description: done}} +components: + securitySchemes: + Auth: + type: oauth2 + flows: + authorizationCode: + authorizationUrl: https://example.com/authorize + tokenUrl: https://example.com/token + scopes: {read: Read with consent, shared: User grant} + clientCredentials: + tokenUrl: https://example.com/token + scopes: {write: Machine write, shared: Machine grant} + implicit: + authorizationUrl: https://example.com/authorize + scopes: {} +`) + document, diagnostics, err := Analyze(source) + require.NoError(t, err) + require.Empty(t, diagnostics) + rendered, err := Render(document, Options{PackageName: "design"}) + require.NoError(t, err) + require.Contains(t, string(rendered), `Scope("shared", "User grant")`) + require.Contains(t, string(rendered), `Scope("shared", "Machine grant")`) + contract := readGeneratedOpenAPIContract(t, requireRenderedDesignGenerates(t, rendered)) + want := map[string]any{ + "authorizationCode": map[string]any{"authorizationUrl": "https://example.com/authorize", "tokenUrl": "https://example.com/token", "scopes": map[string]any{"read": "Read with consent", "shared": "User grant"}}, + "clientCredentials": map[string]any{"tokenUrl": "https://example.com/token", "scopes": map[string]any{"write": "Machine write", "shared": "Machine grant"}}, + "implicit": map[string]any{"authorizationUrl": "https://example.com/authorize", "scopes": map[string]any{}}, + } + got := contract["components"].(map[string]any)["securitySchemes"].(map[string]any)["Auth"].(map[string]any)["flows"] + require.Equal(t, want, got) +} diff --git a/internal/openapiimport/security_per_scheme_test.go b/internal/openapiimport/security_per_scheme_test.go new file mode 100644 index 000000000..5e2c78068 --- /dev/null +++ b/internal/openapiimport/security_per_scheme_test.go @@ -0,0 +1,97 @@ +package openapiimport + +import ( + "fmt" + "strings" + "testing" + + "github.com/stretchr/testify/require" +) + +func TestSecurityPerSchemeRequirements(t *testing.T) { + source := []byte(`openapi: 3.2.0 +info: {title: Per-scheme requirements, version: "1"} +security: [{Read: [read], Write: [], Key: [admin]}, {}] +paths: + /secure: + get: + responses: {"204": {description: done}} +components: + securitySchemes: + Read: + type: oauth2 + flows: + clientCredentials: {tokenUrl: https://example.com/token, scopes: {read: Read, write: Write}} + Write: + type: oauth2 + flows: + clientCredentials: {tokenUrl: https://example.com/token, scopes: {read: Read, write: Write}} + Key: {type: apiKey, in: header, name: X-Key} +`) + document, diagnostics, err := Analyze(source) + require.NoError(t, err) + require.Empty(t, diagnostics) + rendered, err := Render(document, Options{PackageName: "design"}) + require.NoError(t, err) + require.Contains(t, string(rendered), `ScopeFor(ImportedReadSecurity, "read")`) + require.Contains(t, string(rendered), `RoleFor(ImportedKeySecurity, "admin")`) + require.NotContains(t, string(rendered), `ScopeFor(ImportedWriteSecurity, "read")`) + for _, tc := range []struct { + name string + refs []SecurityRequirementScheme + want string + }{ + {"unknown scheme", []SecurityRequirementScheme{{Name: "missing"}}, "not defined"}, + {"unknown scope", []SecurityRequirementScheme{{Name: "Read", Scopes: []string{"missing"}}}, "not defined by scheme"}, + {"duplicate scheme", []SecurityRequirementScheme{{Name: "Read"}, {Name: "Read"}}, "occurs twice"}, + } { + t.Run(tc.name, func(t *testing.T) { + copy := *document + copy.Security = SecurityRequirements{{Schemes: tc.refs}} + _, err := Render(©, Options{}) + require.ErrorContains(t, err, tc.want) + }) + } +} + +func TestSecurityRoleVersionBoundary(t *testing.T) { + for _, version := range []string{"3.0.4", "3.1.1", "3.2.0"} { + for _, roles := range []bool{false, true} { + t.Run(fmt.Sprintf("%s/roles=%t", version, roles), func(t *testing.T) { + keyNames := "[]" + if roles { + keyNames = "[admin]" + } + source := []byte(fmt.Sprintf(`openapi: %s +info: {title: Versioned requirements, version: "1"} +security: [{Read: [read], Write: [], Key: %s}] +paths: + /secure: + get: + responses: {"204": {description: done}} +components: + securitySchemes: + Read: + type: oauth2 + flows: + clientCredentials: {tokenUrl: https://example.com/token, scopes: {read: Read}} + Write: + type: oauth2 + flows: + clientCredentials: {tokenUrl: https://example.com/token, scopes: {read: Read}} + Key: {type: apiKey, in: header, name: X-Key} +`, version, keyNames)) + document, diagnostics, err := Analyze(source) + require.NoError(t, err) + _, renderErr := Render(document, Options{}) + if roles && strings.HasPrefix(version, "3.0.") { + require.Contains(t, diagnostics.Error(), "must be empty in OpenAPI 3.0") + require.ErrorContains(t, renderErr, "must be empty in OpenAPI 3.0") + } else { + require.Empty(t, diagnostics) + require.NoError(t, renderErr) + } + }) + } + } +} diff --git a/internal/openapiimport/symmetry_test.go b/internal/openapiimport/symmetry_test.go index b5bbafad9..72e7bcce0 100644 --- a/internal/openapiimport/symmetry_test.go +++ b/internal/openapiimport/symmetry_test.go @@ -167,11 +167,11 @@ components: `Meta("openapi:deprecated", "true")`, `var ImportedOAuthSecurity = OAuth2Security("OAuth", func() {`, `Meta("openapi:oauth2MetadataUrl", "https://auth.example.com/.well-known/oauth-authorization-server")`, - `AuthorizationCodeFlow("https://auth.example.com/authorize", "https://auth.example.com/token", "https://auth.example.com/refresh")`, - `ClientCredentialsFlow("https://auth.example.com/token", "")`, + `AuthorizationCodeFlow("https://auth.example.com/authorize", "https://auth.example.com/token", "https://auth.example.com/refresh", func() {`, + `ClientCredentialsFlow("https://auth.example.com/token", "", func() {`, `Scope("read", "Read access.")`, `Security(ImportedOAuthSecurity, func() {`, - `Scope("read")`, + `ScopeFor(ImportedOAuthSecurity, "read")`, `Username("basicUsername", String)`, `Password("basicPassword", String)`, `Token("bearerToken", String, func() {`, diff --git a/internal/openapiimport/testdata/security_credentials/imported.json b/internal/openapiimport/testdata/security_credentials/imported.json index 94fef064a..3cb2c01ba 100644 --- a/internal/openapiimport/testdata/security_credentials/imported.json +++ b/internal/openapiimport/testdata/security_credentials/imported.json @@ -43,8 +43,12 @@ }, "security": [ { - "Bearer": [], - "OAuth": [] + "Bearer": [ + "admin" + ], + "OAuth": [ + "read" + ] } ] } @@ -110,9 +114,7 @@ "OAuth": [ "read" ], - "OAuthSecond": [ - "read" - ] + "OAuthSecond": [] } ] } diff --git a/internal/openapiimport/testdata/security_credentials/imported_wire.go.txt b/internal/openapiimport/testdata/security_credentials/imported_wire.go.txt index 361eeb2c7..c45e19a6a 100644 --- a/internal/openapiimport/testdata/security_credentials/imported_wire.go.txt +++ b/internal/openapiimport/testdata/security_credentials/imported_wire.go.txt @@ -21,6 +21,7 @@ type implementation struct { auth []string failJWT bool denyOAuth bool + denyRole bool } func (s *implementation) Either(context.Context, *svc.EitherPayload) error { @@ -74,14 +75,17 @@ func (s *implementation) JWTAuth(ctx context.Context, token string, scheme *secu if s.failJWT || token != "token" { return ctx, fmt.Errorf("JWT denied") } - return ctx, nil + if s.denyRole { + return ctx, scheme.Validate(nil) + } + return ctx, scheme.Validate([]string{"admin"}) } func (s *implementation) OAuth2Auth(ctx context.Context, token string, scheme *security.OAuth2Scheme) (context.Context, error) { s.auth = append(s.auth, fmt.Sprintf("%s:%s:%v", scheme.Name, token, scheme.RequiredScopes)) if s.denyOAuth || token != "token" { return ctx, fmt.Errorf("OAuth denied") } - return ctx, nil + return ctx, scheme.Validate([]string{"read"}) } func (s *implementation) BasicAuth(ctx context.Context, user, pass string, scheme *security.BasicScheme) (context.Context, error) { s.auth = append(s.auth, fmt.Sprintf("%s:%s:%s", scheme.Name, user, pass)) @@ -104,11 +108,11 @@ func TestImportedCredentialAuthWire(t *testing.T) { {"basic", "either", "Basic dXNlcjpwYXNz", "", false, false, 1, []string{"Basic:user:pass"}}, {"JWT", "either", "Bearer token", "", false, false, 1, []string{"Basic::", "Bearer:token:[]"}}, {"OAuth fallback scopes", "either", "Bearer token", "", true, false, 1, []string{"Basic::", "Bearer:token:[]", "OAuth:token:[read]"}}, - {"AND both", "both", "Bearer token", "", false, false, 1, []string{"Bearer:token:[]", "OAuth:token:[]"}}, - {"AND denied", "both", "Bearer token", "", false, true, 0, []string{"Bearer:token:[]", "OAuth:token:[]"}}, + {"AND both", "both", "Bearer token", "", false, false, 1, []string{"Bearer:token:[admin]", "OAuth:token:[read]"}}, + {"AND denied", "both", "Bearer token", "", false, true, 0, []string{"Bearer:token:[admin]", "OAuth:token:[read]"}}, {"shared Basic roles", "basic_roles", "Basic dXNlcjpwYXNz", "", false, false, 1, []string{"Basic:user:pass", "BasicSecond:user:pass"}}, {"shared bearer roles", "bearer_roles", "Bearer token", "", false, false, 1, []string{"Bearer:token:[]", "BearerSecond:token:[]"}}, - {"OAuth scopes", "oauth_roles", "Bearer token", "", false, false, 1, []string{"OAuth:token:[read]", "OAuthSecond:token:[read]"}}, + {"OAuth scopes", "oauth_roles", "Bearer token", "", false, false, 1, []string{"OAuth:token:[read]", "OAuthSecond:token:[]"}}, {"inherited", "inherit", "Bearer token", "", false, false, 1, []string{"Bearer:token:[]"}}, {"public override", "public", "", "", false, false, 1, nil}, {"anonymous alternative", "optional", "", "", true, false, 1, []string{"Bearer::[]"}}, @@ -182,7 +186,7 @@ func TestImportedCredentialClientWire(t *testing.T) { }{ {"either", &svc.EitherPayload{BasicUsername: &user, BasicPassword: &pass}, client.EncodeEitherRequest(nil), []string{"Basic:user:pass"}}, {"either", &svc.EitherPayload{BearerToken: &token}, client.EncodeEitherRequest(nil), []string{"Basic::", "Bearer:token:[]"}}, - {"both", &svc.BothPayload{BearerToken: &token}, client.EncodeBothRequest(nil), []string{"Bearer:token:[]", "OAuth:token:[]"}}, + {"both", &svc.BothPayload{BearerToken: &token}, client.EncodeBothRequest(nil), []string{"Bearer:token:[admin]", "OAuth:token:[read]"}}, {"keys", &svc.KeysPayload{OneCredential: &key}, client.EncodeKeysRequest(nil), []string{"One:key", "Two:key"}}, } { t.Run(tc.path+tc.auth[0], func(t *testing.T) { @@ -204,3 +208,19 @@ func TestImportedCredentialClientWire(t *testing.T) { }) } } + +func TestImportedMissingRole(t *testing.T) { + s := &implementation{denyRole: true} + mux := loomhttp.NewMuxer() + server.Mount(mux, server.New(svc.NewEndpoints(s), mux, loomhttp.RequestDecoder, loomhttp.ResponseEncoder, nil, nil)) + req := httptest.NewRequest(http.MethodGet, "/both", nil) + req.Header.Set("Authorization", "Bearer token") + rec := httptest.NewRecorder() + mux.ServeHTTP(rec, req) + if s.calls != 0 || rec.Code < 400 { + t.Errorf("missing role allowed invocation: calls=%d status=%d", s.calls, rec.Code) + } + if !reflect.DeepEqual(s.auth, []string{"Bearer:token:[admin]"}) { + t.Errorf("unexpected callbacks after role denial: %v", s.auth) + } +} diff --git a/internal/openapiimport/testdata/security_credentials/partial_wire.go.txt b/internal/openapiimport/testdata/security_credentials/partial_wire.go.txt index b97b0abba..a05ab2ccc 100644 --- a/internal/openapiimport/testdata/security_credentials/partial_wire.go.txt +++ b/internal/openapiimport/testdata/security_credentials/partial_wire.go.txt @@ -48,9 +48,9 @@ func TestPartialCredentialAuthWire(t *testing.T) { }{ {name: "public", path: "/public", calls: 1, status: 204}, {name: "scoped conjunction", path: "/oauth_roles", token: "token", calls: 1, status: 204, - auth: []string{"OAuth:token:[read]", "OAuthSecond:token:[read]"}}, + auth: []string{"OAuth:token:[read]", "OAuthSecond:token:[]"}}, {name: "second role denies", path: "/oauth_roles", token: "token", denySecond: true, - auth: []string{"OAuth:token:[read]", "OAuthSecond:token:[read]"}}, + auth: []string{"OAuth:token:[read]", "OAuthSecond:token:[]"}}, {name: "missing credential", path: "/oauth_roles", auth: []string{"OAuth::[read]"}}, {name: "inherited unsupported omitted", path: "/inherit", status: 404}, } { diff --git a/internal/openapiimport/tla/reference_siblings/README.md b/internal/openapiimport/tla/reference_siblings/README.md new file mode 100644 index 000000000..a4ee3b29d --- /dev/null +++ b/internal/openapiimport/tla/reference_siblings/README.md @@ -0,0 +1,56 @@ +# Reference sibling conjunction and declaration storage + +`analyze_reference_siblings.go` retains a local reference and normalizes admitted +sibling assertions against its carrier type. The occurrence is the conjunction +of the referenced contract and its siblings. It never replaces or mutates the +base. Object-property intersections use Extend for composed field storage and +retain contributors through `expr.AttributeExpr.CompositionBases` and +`ir.analyzeCompositionReferences`. Unsupported type/key-set intersections remain +explicit admission errors. + +`expr.NewObjectFields` captures both semantic requiredness and declaration +requiredness. `codegen.AttributeContext.FieldPresence` and `expr.ServiceFieldLayout` +use the latter for physical storage. Requiring an optional field at a named +occurrence cannot turn an existing Optional field into a native scalar/pointer. +Generated validation must test its presence. + +Run TLC 1.7.4 explicitly (Java 25 was used): + +```sh +java -XX:+UseParallelGC -cp /path/to/tla2tools-1.7.4.jar tlc2.TLC \ + -workers 1 -metadir /tmp/loom-reference-siblings -config conjoin.cfg ReferenceSiblings.tla +``` + +The `conjoin` configuration checks 700 states over four representative values +and Boolean declaration/occurrence requiredness. The other configurations are +intentional counterexamples: `reject` violates SupportedInputAdmitted (the old +blanket rejection); `replace` violates ConjunctionPreserved; `mutate` violates +BaseUnchanged; `flatten` violates ReferencePreserved by dropping the named +contributor from the emitted graph; `nullable` violates ConjunctionPreserved by restoring null +excluded from the target domain; `storage` violates DeclaredStoragePreserved when an optional +physical field becomes required at an occurrence. Use a separate temporary +`-metadir` for each configuration. These are design checks, never CI gates. + +The domain assumes nonempty intersections and abstracts away schema syntax, +recursive references, nullable carriers, forced transport pointers and custom +codecs. It does not prove arbitrary JSON Schema composition or the importer. +`reference_siblings_test.go` checks actual import, generation, compilation, +parsed contracts and independent-process bytes, including rejected intersections. +`codegen.TestReferenceRequirementPreservesDeclaredJSONStorage` reproduced the +incorrect carrier and checks both pointer contexts; the generated required-object +case reproduced a Go compile failure before the storage correction. +The nullable-enum regression reproduced acceptance of a forbidden null before +normalizing target membership. Object cases check inline and named fields, +recursive contributor identity, generated validation and schema re-import; +document-level server/dialect import limits are excluded explicitly from that +schema round trip. + +The `source` configuration reproduces the numeric-token ownership defect: +reading the occurrence's enum instead of the inherited node's enum changes +`{null, valid}` into `{valid}`. `SourceOwnershipPreserved` checks this explicit +owner distinction; `conjoin` passes it. This abstracts token identity rather than +parsing. `numeric_source_identity.go` captures original tokens by source position +before semantic construction. `reference_source_ownership_test.go` checks inherited +nullable membership, inherited array item enums and bounds, JSON/YAML precision +and quoted kinds, and inline YAML merges. Transformed references inside general +allOf branches are rejected rather than losing their assertions. diff --git a/internal/openapiimport/tla/reference_siblings/ReferenceSiblings.tla b/internal/openapiimport/tla/reference_siblings/ReferenceSiblings.tla new file mode 100644 index 000000000..e38377dfb --- /dev/null +++ b/internal/openapiimport/tla/reference_siblings/ReferenceSiblings.tla @@ -0,0 +1,29 @@ +---------------------- MODULE ReferenceSiblings ---------------------- +EXTENDS TLC +CONSTANT Mode +VARIABLES base, sibling, declaredRequired, localRequired +SourceTokens == [owner |-> {"null", "valid"}, occurrence |-> {"valid"}] +ReadTarget == SourceTokens[IF Mode = "source" THEN "occurrence" ELSE "owner"] +SourceOwnershipPreserved == ReadTarget = SourceTokens.owner +Values == {"short", "valid", "long", "null"} +Init == /\ base \in SUBSET Values + /\ sibling \in SUBSET Values + /\ base \cap sibling # {} + /\ declaredRequired \in BOOLEAN + /\ localRequired \in BOOLEAN +Admitted == Mode # "reject" /\ base # {} +Occurrence == IF Mode = "replace" THEN sibling + ELSE IF Mode = "nullable" THEN (base \cap sibling) \cup {"null"} + ELSE base \cap sibling +PublishedBase == IF Mode = "mutate" THEN Occurrence ELSE base +SupportedInputAdmitted == Admitted +ConjunctionPreserved == Admitted => Occurrence = base \cap sibling +BaseUnchanged == PublishedBase = base +PublishedReference == IF Mode = "flatten" THEN {} ELSE base +ReferencePreserved == PublishedReference = base +Carrier(required) == IF required THEN "native" ELSE "optional" +RenderedCarrier == Carrier(IF Mode = "storage" THEN declaredRequired \/ localRequired ELSE declaredRequired) +DeclaredStoragePreserved == RenderedCarrier = Carrier(declaredRequired) +Next == UNCHANGED <> +Spec == Init /\ [][Next]_<> +====================================================================== diff --git a/internal/openapiimport/tla/reference_siblings/conjoin.cfg b/internal/openapiimport/tla/reference_siblings/conjoin.cfg new file mode 100644 index 000000000..ba614bfb6 --- /dev/null +++ b/internal/openapiimport/tla/reference_siblings/conjoin.cfg @@ -0,0 +1,8 @@ +CONSTANT Mode = "conjoin" +SPECIFICATION Spec +INVARIANT SupportedInputAdmitted +INVARIANT ConjunctionPreserved +INVARIANT BaseUnchanged +INVARIANT DeclaredStoragePreserved +INVARIANT ReferencePreserved +INVARIANT SourceOwnershipPreserved diff --git a/internal/openapiimport/tla/reference_siblings/flatten.cfg b/internal/openapiimport/tla/reference_siblings/flatten.cfg new file mode 100644 index 000000000..bfcafb6ef --- /dev/null +++ b/internal/openapiimport/tla/reference_siblings/flatten.cfg @@ -0,0 +1,7 @@ +CONSTANT Mode = "flatten" +SPECIFICATION Spec +INVARIANT SupportedInputAdmitted +INVARIANT ConjunctionPreserved +INVARIANT BaseUnchanged +INVARIANT DeclaredStoragePreserved +INVARIANT ReferencePreserved diff --git a/internal/openapiimport/tla/reference_siblings/mutate.cfg b/internal/openapiimport/tla/reference_siblings/mutate.cfg new file mode 100644 index 000000000..36df48e9f --- /dev/null +++ b/internal/openapiimport/tla/reference_siblings/mutate.cfg @@ -0,0 +1,7 @@ +CONSTANT Mode = "mutate" +SPECIFICATION Spec +INVARIANT SupportedInputAdmitted +INVARIANT ConjunctionPreserved +INVARIANT BaseUnchanged +INVARIANT DeclaredStoragePreserved +INVARIANT ReferencePreserved diff --git a/internal/openapiimport/tla/reference_siblings/nullable.cfg b/internal/openapiimport/tla/reference_siblings/nullable.cfg new file mode 100644 index 000000000..bfc88487c --- /dev/null +++ b/internal/openapiimport/tla/reference_siblings/nullable.cfg @@ -0,0 +1,7 @@ +CONSTANT Mode = "nullable" +SPECIFICATION Spec +INVARIANT SupportedInputAdmitted +INVARIANT ConjunctionPreserved +INVARIANT BaseUnchanged +INVARIANT DeclaredStoragePreserved +INVARIANT ReferencePreserved diff --git a/internal/openapiimport/tla/reference_siblings/reject.cfg b/internal/openapiimport/tla/reference_siblings/reject.cfg new file mode 100644 index 000000000..3d57ea38a --- /dev/null +++ b/internal/openapiimport/tla/reference_siblings/reject.cfg @@ -0,0 +1,7 @@ +CONSTANT Mode = "reject" +SPECIFICATION Spec +INVARIANT SupportedInputAdmitted +INVARIANT ConjunctionPreserved +INVARIANT BaseUnchanged +INVARIANT DeclaredStoragePreserved +INVARIANT ReferencePreserved diff --git a/internal/openapiimport/tla/reference_siblings/replace.cfg b/internal/openapiimport/tla/reference_siblings/replace.cfg new file mode 100644 index 000000000..88dbec412 --- /dev/null +++ b/internal/openapiimport/tla/reference_siblings/replace.cfg @@ -0,0 +1,7 @@ +CONSTANT Mode = "replace" +SPECIFICATION Spec +INVARIANT SupportedInputAdmitted +INVARIANT ConjunctionPreserved +INVARIANT BaseUnchanged +INVARIANT DeclaredStoragePreserved +INVARIANT ReferencePreserved diff --git a/internal/openapiimport/tla/reference_siblings/source.cfg b/internal/openapiimport/tla/reference_siblings/source.cfg new file mode 100644 index 000000000..b8f5b32d1 --- /dev/null +++ b/internal/openapiimport/tla/reference_siblings/source.cfg @@ -0,0 +1,3 @@ +CONSTANT Mode = "source" +SPECIFICATION Spec +INVARIANT SourceOwnershipPreserved diff --git a/internal/openapiimport/tla/reference_siblings/storage.cfg b/internal/openapiimport/tla/reference_siblings/storage.cfg new file mode 100644 index 000000000..19730725d --- /dev/null +++ b/internal/openapiimport/tla/reference_siblings/storage.cfg @@ -0,0 +1,7 @@ +CONSTANT Mode = "storage" +SPECIFICATION Spec +INVARIANT SupportedInputAdmitted +INVARIANT ConjunctionPreserved +INVARIANT BaseUnchanged +INVARIANT DeclaredStoragePreserved +INVARIANT ReferencePreserved diff --git a/internal/openapiimport/tla/reusable_references/README.md b/internal/openapiimport/tla/reusable_references/README.md new file mode 100644 index 000000000..b9001e291 --- /dev/null +++ b/internal/openapiimport/tla/reusable_references/README.md @@ -0,0 +1,28 @@ +# Reusable transport reference planning + +`plan_reusable.go` indexes all three component families before resolving +occurrences. It walks forward alias edges with a visited set, copies the terminal +value, and carries each edge's summary/description separately. `reusable_authored.go` +in the OpenAPI IR publishes canonical definitions and aliases before inferred +component allocation. `object_ref.go` preserves present empty annotations. + +Run TLC explicitly (Java 27, TLC 1.7.4 release jar): + +```sh +java -XX:+UseParallelGC -cp /path/to/tla2tools-1.7.4.jar tlc2.TLC \ + -workers 1 -metadir /tmp/loom-reusable-copy -config copy.cfg ReusableReferences.tla +``` + +`copy` checks 486 states across three kinds, two names, nine possible two-node +reference graphs, and absent/empty/nonempty local annotations. `reject` reproduces +the former blanket admission boundary. `inline` loses identity, `mutate` changes +shared definitions, and `cycle` admits cyclic aliases. Run each negative control +with its matching configuration and a separate temporary metadir. + +The model abstracts parsed content, naming collisions, schema generation, and +annotation applicability by target object kind. It does not prove arbitrary +OpenAPI parsing. The planner tests cover wrong kinds, missing targets and escaped +names. The Pet API round trip covers shared occurrences, generated compilation, +parsed contracts and process-isolated determinism; IR tests cover reserved names +and conflicts, and JSON/YAML tests cover empty annotation preservation. Model +checks are opt-in design tools, never ordinary build or CI gates. diff --git a/internal/openapiimport/tla/reusable_references/ReusableReferences.tla b/internal/openapiimport/tla/reusable_references/ReusableReferences.tla new file mode 100644 index 000000000..11ac44ba8 --- /dev/null +++ b/internal/openapiimport/tla/reusable_references/ReusableReferences.tla @@ -0,0 +1,32 @@ +---------------------- MODULE ReusableReferences ---------------------- +EXTENDS TLC +CONSTANT Mode +VARIABLES kind, name, localDescription, localSummary, edges +Kinds == {"requestBodies", "responses", "headers"} +Names == {"A", "B"} +Absent == "absent" +Texts == {Absent, "", "local"} +RECURSIVE Resolve(_, _) +Resolve(current, seen) == IF current \in seen THEN "cycle" + ELSE IF edges[current] = "terminal" THEN current + ELSE Resolve(edges[current], seen \cup {current}) +Resolved == Resolve(name, {}) +Base == [description |-> "base", summary |-> "base-summary", content |-> "wire"] +Init == /\ kind \in Kinds /\ name \in Names + /\ localDescription \in Texts /\ localSummary \in Texts + /\ edges \in [Names -> Names \cup {"terminal"}] +Description == IF localDescription = Absent THEN Base.description ELSE localDescription +Summary == IF localSummary = Absent THEN Base.summary ELSE localSummary +Admitted == Mode # "reject" /\ (Mode = "cycle" \/ Resolved # "cycle") +Reference == IF Mode = "inline" THEN <> ELSE <> +PublishedBase == IF Mode = "mutate" THEN [Base EXCEPT !.description = Description] ELSE Base +Occurrence == [description |-> Description, summary |-> Summary, content |-> Base.content] +SupportedInputAdmitted == Resolved # "cycle" => Admitted +CycleRejected == Resolved = "cycle" => ~Admitted +IdentityPreserved == Reference = <> +BaseUnchanged == PublishedBase = Base +OverlayApplied == Occurrence.description = Description /\ Occurrence.summary = Summary +ContentPreserved == Occurrence.content = Base.content +Next == UNCHANGED <> +Spec == Init /\ [][Next]_<> +====================================================================== diff --git a/internal/openapiimport/tla/reusable_references/copy.cfg b/internal/openapiimport/tla/reusable_references/copy.cfg new file mode 100644 index 000000000..1f3eb3663 --- /dev/null +++ b/internal/openapiimport/tla/reusable_references/copy.cfg @@ -0,0 +1,8 @@ +CONSTANT Mode = "copy" +SPECIFICATION Spec +INVARIANT SupportedInputAdmitted +INVARIANT IdentityPreserved +INVARIANT BaseUnchanged +INVARIANT OverlayApplied +INVARIANT ContentPreserved +INVARIANT CycleRejected diff --git a/internal/openapiimport/tla/reusable_references/cycle.cfg b/internal/openapiimport/tla/reusable_references/cycle.cfg new file mode 100644 index 000000000..24d7a7803 --- /dev/null +++ b/internal/openapiimport/tla/reusable_references/cycle.cfg @@ -0,0 +1,3 @@ +CONSTANT Mode = "cycle" +SPECIFICATION Spec +INVARIANT CycleRejected diff --git a/internal/openapiimport/tla/reusable_references/inline.cfg b/internal/openapiimport/tla/reusable_references/inline.cfg new file mode 100644 index 000000000..f68723dc2 --- /dev/null +++ b/internal/openapiimport/tla/reusable_references/inline.cfg @@ -0,0 +1,8 @@ +CONSTANT Mode = "inline" +SPECIFICATION Spec +INVARIANT SupportedInputAdmitted +INVARIANT IdentityPreserved +INVARIANT BaseUnchanged +INVARIANT OverlayApplied +INVARIANT ContentPreserved +INVARIANT CycleRejected diff --git a/internal/openapiimport/tla/reusable_references/mutate.cfg b/internal/openapiimport/tla/reusable_references/mutate.cfg new file mode 100644 index 000000000..b6f04b2fe --- /dev/null +++ b/internal/openapiimport/tla/reusable_references/mutate.cfg @@ -0,0 +1,8 @@ +CONSTANT Mode = "mutate" +SPECIFICATION Spec +INVARIANT SupportedInputAdmitted +INVARIANT IdentityPreserved +INVARIANT BaseUnchanged +INVARIANT OverlayApplied +INVARIANT ContentPreserved +INVARIANT CycleRejected diff --git a/internal/openapiimport/tla/reusable_references/reject.cfg b/internal/openapiimport/tla/reusable_references/reject.cfg new file mode 100644 index 000000000..c7692cbbc --- /dev/null +++ b/internal/openapiimport/tla/reusable_references/reject.cfg @@ -0,0 +1,8 @@ +CONSTANT Mode = "reject" +SPECIFICATION Spec +INVARIANT SupportedInputAdmitted +INVARIANT IdentityPreserved +INVARIANT BaseUnchanged +INVARIANT OverlayApplied +INVARIANT ContentPreserved +INVARIANT CycleRejected diff --git a/internal/openapiimport/tla/schema_references/README.md b/internal/openapiimport/tla/schema_references/README.md index 4b74952f0..dd065283c 100644 --- a/internal/openapiimport/tla/schema_references/README.md +++ b/internal/openapiimport/tla/schema_references/README.md @@ -138,3 +138,8 @@ model into a proof of arbitrary-size SCC implementation or all DSL constructors. Keep the constructor classification and occurrence paths aligned with renderer changes. Bounded generation and external lint results are recorded in the owning readiness document. Temporary outputs are removed after final independent review. + +The historical flattening warning above was removed by the reference-sibling +work. Current mixed inheritance preserves an allOf reference plus the composed +object. See [reference siblings](../reference_siblings/README.md) for the +identity/conjunction model; the initializer model here remains unchanged. diff --git a/internal/openapiimport/tla/security_credentials/README.md b/internal/openapiimport/tla/security_credentials/README.md index 186d6b82c..f1fa7b58c 100644 --- a/internal/openapiimport/tla/security_credentials/README.md +++ b/internal/openapiimport/tla/security_credentials/README.md @@ -62,12 +62,12 @@ scoped secured overrides, blocked inheritance, strict rejection, the empty subset and actual generated authentication. Valid inherited root requirements remain covered by the existing partial-import tests. -Scopes are intentionally outside this physical grouping model. The existing -Loom requirement has one scope list applied to every scheme; OpenAPI carries one -list per scheme. A generated counterexample records that disagreement and direct -import tests demand explicit unsupported diagnostics for differing per-scheme -lists. Do not infer scope unions, scope intersections or a public policy from -this model. Header parsing, presence/default semantics, generated identifiers, +Per-scheme names are outside this physical grouping model. The shared model in +`expr/tla/security_requirements` now checks that independent scope/role lists +preserve AND, OR and anonymous alternatives. Import retains those arrays; +`security_credentials_import_test.go` checks generated role denial and unequal +OAuth requirements. Invalid OAuth scopes still reject, including during partial +import. Header parsing, presence/default semantics, generated identifiers, custom mappings, arbitrary graph size, repeated query values and malformed credentials remain concrete-test obligations. Raw Authorization conflicts and Basic/Bearer conjunctions remain under the existing expr admission model. diff --git a/internal/openapiref/object.go b/internal/openapiref/object.go new file mode 100644 index 000000000..b1ab1ea94 --- /dev/null +++ b/internal/openapiref/object.go @@ -0,0 +1,12 @@ +package openapiref + +// Object identifies a reusable OpenAPI object and its occurrence annotations. +// Nil annotations inherit; a pointer to an empty string explicitly clears one. +type Object struct { + // Ref is a local reference to a request body, response, or header component. + Ref string `json:"$ref"` + // Description overrides the target description at this occurrence. + Description *string `json:"description,omitzero"` + // Summary overrides the target summary when that object supports summaries. + Summary *string `json:"summary,omitzero"` +} diff --git a/jsonrpc/codegen/security_scope_test.go b/jsonrpc/codegen/security_scope_test.go index 4e8c19257..91b733e4a 100644 --- a/jsonrpc/codegen/security_scope_test.go +++ b/jsonrpc/codegen/security_scope_test.go @@ -32,3 +32,14 @@ func TestJSONRPCSecurityScope(t *testing.T) { }) } } + +func TestJSONRPCPerSchemeSecurityProjection(t *testing.T) { + root := RunJSONRPCDSL(t, testdata.SecurityPerSchemeDSL("jsonrpc")) + endpoint := CreateJSONRPCServices(root).Get("per_scheme").Endpoint("read") + requirements := endpoint.Method.Requirements + require.Len(t, requirements, 3) + require.Equal(t, []string{"read"}, requirements[0].Schemes[0].RequiredScopes) + require.Equal(t, []string{"write"}, requirements[0].Schemes[1].RequiredScopes) + require.Equal(t, []string{"read"}, requirements[1].Schemes[0].RequiredScopes) + require.Empty(t, requirements[2].Schemes) +} diff --git a/jsonrpc/codegen/untagged_arrays_test.go b/jsonrpc/codegen/untagged_arrays_test.go index 053f39a6d..bf9ea21db 100644 --- a/jsonrpc/codegen/untagged_arrays_test.go +++ b/jsonrpc/codegen/untagged_arrays_test.go @@ -17,9 +17,25 @@ func TestJSONRPCUntaggedArrayObject(t *testing.T) { }) }) + first := Type("Person", func() { + Attribute("name", String) + Required("name") + }) + second := Type("Group", func() { + Attribute("group", String) + Required("group") + }) + entry := Type("Entry", OneOf(first, second), Untagged) items := Type("Items", ArrayOf(String)) page := Type("Page", func() { Attribute("total", Int) + Attribute("entries", ArrayOf(entry)) + Attribute("labels", MapOf(String, String), func() { + MinLength(1) + Elem(func() { + MinLength(2) + }) + }) Attribute("status", String, func() { Nullable() Enum("open", "closed") @@ -55,6 +71,9 @@ import ( "context" "encoding/json/v2" "net/http/httptest" + "net/http" + "io" + "sync/atomic" "strings" "testing" @@ -66,7 +85,9 @@ import ( ) func TestJSONRPCArrayObject(t *testing.T) { + var calls atomic.Int32 echo := func(_ context.Context, value any) (any, error) { + calls.Add(1) return value, nil } mux := loomhttp.NewMuxer() @@ -74,13 +95,36 @@ func TestJSONRPCArrayObject(t *testing.T) { host := httptest.NewServer(mux) defer host.Close() c := client.NewClient("http", strings.TrimPrefix(host.URL, "http://"), host.Client(), loomhttp.RequestEncoder, loomhttp.ResponseDecoder, false) - for _, wire := range []string{"[]", "[\"ok\"]", "{\"total\":0}", "{\"total\":0,\"status\":null}", "{\"total\":0,\"status\":\"open\"}"} { + for _, wire := range []string{"[]", "[\"ok\"]", "{\"total\":0}", "{\"total\":0,\"status\":null}", "{\"total\":0,\"status\":\"open\"}", "{\"total\":0,\"labels\":{\"a\":\"ok\"}}", "{\"total\":0,\"entries\":[]}", "{\"total\":2,\"entries\":[{\"name\":\"person\"},{\"group\":\"team\"}]}"} { var payload svc.Choice require.NoError(t, json.Unmarshal([]byte(wire), &payload)) result, err := c.Echo()(t.Context(), &payload) require.NoError(t, err) require.Equal(t, &payload, result) } + for _, wire := range []string{"{\"total\":0,\"labels\":{}}", "{\"total\":0,\"labels\":{\"a\":\"x\"}}", "{\"total\":0,\"labels\":{\"a\":null}}"} { + var value svc.Choice + require.Error(t, json.Unmarshal([]byte(wire), &value)) + } + + for _, wire := range []string{"{\"total\":0,\"entries\":[{}]}", "{\"total\":0,\"entries\":[{\"name\":\"person\",\"group\":\"team\"}]}", "{\"total\":0,\"entries\":[null]}"} { + before := calls.Load() + body := "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"echo\",\"params\":" + wire + "}" + response, err := host.Client().Post(host.URL+"/rpc", "application/json", strings.NewReader(body)) + require.NoError(t, err) + raw, err := io.ReadAll(response.Body) + require.NoError(t, err) + require.NoError(t, response.Body.Close()) + require.Equal(t, http.StatusOK, response.StatusCode, string(raw)) + var envelope map[string]any + require.NoError(t, json.Unmarshal(raw, &envelope)) + problem, ok := envelope["error"].(map[string]any) + require.True(t, ok, string(raw)) + require.Equal(t, float64(-32602), problem["code"]) + require.Equal(t, "invalid request body", problem["message"]) + require.Equal(t, before, calls.Load()) + } + var invalid svc.Choice require.Error(t, json.Unmarshal([]byte("{\"total\":0,\"status\":\"other\"}"), &invalid)) } diff --git a/pkg/json_shape.go b/pkg/json_shape.go index f2571684f..764a2c184 100644 --- a/pkg/json_shape.go +++ b/pkg/json_shape.go @@ -16,14 +16,16 @@ type ( // DSL references, schema resolution, coercions, defaults, or branch selection. // Callers must finish building the graph before sharing it and keep it immutable. JSONShape struct { - // Kind is any, boolean, string, bytes, integer, number, object, array, or alias. + // Kind is any, boolean, string, bytes, integer, number, object, array, map, union, or alias. Kind string // Nullable admits explicit JSON null at this occurrence. Nullable bool - // Child is the array element or underlying alias shape. + // Child is the array element, map value, or underlying alias shape. Child *JSONShape // Fields are the visible object members in declaration order. Fields []JSONShapeField + // Branches are the alternatives of an untagged union; exactly one must match. + Branches []*JSONShape // Closed rejects object members absent from Fields. Closed bool // NonNullableElements rejects null array elements independently of Child. @@ -46,7 +48,7 @@ type ( Minimum, Maximum string // ExclusiveMinimum and ExclusiveMaximum are decimal exclusive bounds. ExclusiveMinimum, ExclusiveMaximum string - // MinLength and MaxLength bound strings, decoded bytes, or arrays. + // MinLength and MaxLength bound strings, decoded bytes, arrays, or map entries. MinLength, MaxLength *int // Patterns are conjoined regular expressions. Patterns []string @@ -82,19 +84,27 @@ func (s *JSONShape) validate(seen map[*JSONShape]bool) error { seen[s] = true switch s.Kind { case "any", "boolean", "string", "bytes", "integer", "number": - case "array", "alias": + case "array", "map", "alias": if s.Kind == "alias" { - aliases := make(map[*JSONShape]bool) - for node := s; node != nil && node.Kind == "alias"; node = node.Child { - if aliases[node] { - return fmt.Errorf("cyclic JSON matching alias") - } - aliases[node] = true + if err := s.validateAlternatives(make(map[*JSONShape]bool)); err != nil { + return err } } if err := s.Child.validate(seen); err != nil { return err } + case "union": + if len(s.Branches) < 2 { + return fmt.Errorf("JSON matching union requires at least two branches") + } + if err := s.validateAlternatives(make(map[*JSONShape]bool)); err != nil { + return err + } + for _, branch := range s.Branches { + if err := branch.validate(seen); err != nil { + return err + } + } case "object": names := make(map[string]bool) for _, field := range s.Fields { @@ -112,6 +122,28 @@ func (s *JSONShape) validate(seen map[*JSONShape]bool) error { return s.Rules.validate() } +// validateAlternatives rejects cycles that revisit the same wire value. Object +// fields and collection values consume input and may contain recursive unions. +func (s *JSONShape) validateAlternatives(active map[*JSONShape]bool) error { + if s == nil || s.Kind != "alias" && s.Kind != "union" { + return nil + } + if active[s] { + return fmt.Errorf("cyclic JSON matching alternatives") + } + active[s] = true + defer delete(active, s) + if s.Kind == "alias" { + return s.Child.validateAlternatives(active) + } + for _, branch := range s.Branches { + if err := branch.validateAlternatives(active); err != nil { + return err + } + } + return nil +} + func (r JSONShapeRules) validate() error { for _, bound := range []string{r.Minimum, r.Maximum, r.ExclusiveMinimum, r.ExclusiveMaximum} { if bound != "" { @@ -185,6 +217,10 @@ func (s *JSONShape) match(wire jsontext.Value, depth int) (bool, error) { } case "array": return s.array(wire, depth) + case "map": + return s.mapping(wire, depth) + case "union": + return s.union(wire, depth) case "object": matched, err := s.object(wire, depth) if err != nil || !matched { @@ -196,6 +232,23 @@ func (s *JSONShape) match(wire jsontext.Value, depth int) (bool, error) { return s.rules(wire, length) } +func (s *JSONShape) union(wire jsontext.Value, depth int) (bool, error) { + matches := 0 + for _, branch := range s.Branches { + matched, err := branch.match(wire, depth+1) + if err != nil { + return false, err + } + if matched { + matches++ + } + } + if matches != 1 { + return false, nil + } + return s.rules(wire, -1) +} + func (s *JSONShape) object(wire jsontext.Value, depth int) (bool, error) { if wire.Kind() != '{' { return false, nil @@ -221,6 +274,23 @@ func (s *JSONShape) object(wire jsontext.Value, depth int) (bool, error) { return !s.Closed || len(members) == 0, nil } +func (s *JSONShape) mapping(wire jsontext.Value, depth int) (bool, error) { + if wire.Kind() != '{' { + return false, nil + } + var members map[string]jsontext.Value + if err := json.Unmarshal(wire, &members); err != nil { + return false, err + } + for _, child := range members { + matched, err := s.Child.match(child, depth+1) + if err != nil || !matched { + return matched, err + } + } + return s.rules(wire, len(members)) +} + func (s *JSONShape) length(wire jsontext.Value) int { underlying := s for underlying != nil && underlying.Kind == "alias" { @@ -229,6 +299,12 @@ func (s *JSONShape) length(wire jsontext.Value) int { if underlying == nil { return -1 } + if underlying.Kind == "map" && wire.Kind() == '{' { + var members map[string]jsontext.Value + if json.Unmarshal(wire, &members) == nil { + return len(members) + } + } if wire.Kind() == '[' { var values []jsontext.Value if json.Unmarshal(wire, &values) == nil { diff --git a/pkg/json_shape_map_test.go b/pkg/json_shape_map_test.go new file mode 100644 index 000000000..2756e1866 --- /dev/null +++ b/pkg/json_shape_map_test.go @@ -0,0 +1,78 @@ +package loom + +import ( + "encoding/json/jsontext" + "testing" + + "github.com/stretchr/testify/require" +) + +func TestJSONShapeMaps(t *testing.T) { + text := &JSONShape{Kind: "string", Rules: JSONShapeRules{MinLength: new(2)}} + dictionary := &JSONShape{Kind: "map", Child: text} + bounded := &JSONShape{Kind: "alias", Child: dictionary, Rules: JSONShapeRules{MinLength: new(1), MaxLength: new(2)}} + nullable := &JSONShape{Kind: "map", Nullable: true, Child: text, Rules: JSONShapeRules{Enums: [][]jsontext.Value{{[]byte(`{"a":"ok"}`)}}}} + for _, tc := range []struct { + name string + shape *JSONShape + wire string + want bool + }{ + {"empty", dictionary, `{}`, true}, + {"string keys", dictionary, `{"":"ok","escaped\"":"yes","雪":"ok"}`, true}, + {"invalid member", dictionary, `{"a":"ok","b":"x"}`, false}, + {"wrong member type", dictionary, `{"a":2}`, false}, + {"null member", dictionary, `{"a":null}`, false}, + {"null map", dictionary, `null`, false}, + {"wrong container", dictionary, `[]`, false}, + {"duplicate keys", dictionary, `{"a":"ok","a":"yes"}`, false}, + {"bounded valid", bounded, `{"a":"ok"}`, true}, + {"bounded empty", bounded, `{}`, false}, + {"bounded too long", bounded, `{"a":"ok","b":"ok","c":"ok"}`, false}, + {"nullable enum null", nullable, `null`, true}, + {"nullable enum member", nullable, `{"a":"ok"}`, true}, + {"nullable enum other", nullable, `{"b":"ok"}`, false}, + {"nested map", &JSONShape{Kind: "map", Child: dictionary}, `{"outer":{"a":"ok"}}`, true}, + {"nested invalid", &JSONShape{Kind: "map", Child: dictionary}, `{"outer":{"a":"x"}}`, false}, + {"array of maps", &JSONShape{Kind: "array", Child: dictionary}, `[{"a":"ok"},{}]`, true}, + } { + t.Run(tc.name, func(t *testing.T) { + matched, err := tc.shape.Match([]byte(tc.wire)) + require.NoError(t, err) + require.Equal(t, tc.want, matched) + }) + } +} + +func TestJSONShapeMapDescriptionsValidateBeforeValues(t *testing.T) { + for _, shape := range []*JSONShape{ + {Kind: "map"}, + {Kind: "map", Nullable: true, Child: &JSONShape{Kind: "unknown"}}, + } { + for _, wire := range []string{`{}`, `null`} { + _, err := shape.Match([]byte(wire)) + require.Error(t, err) + } + } +} + +func TestJSONShapeRecursiveMapValues(t *testing.T) { + node := &JSONShape{Kind: "object", Closed: true} + node.Fields = []JSONShapeField{ + {Name: "name", Required: true, Shape: &JSONShape{Kind: "string"}}, + {Name: "children", Shape: &JSONShape{Kind: "map", Child: node}}, + } + for _, tc := range []struct { + wire string + want bool + }{ + {`{"name":"root","children":{"leaf":{"name":"leaf"}}}`, true}, + {`{"name":"root","children":{}}`, true}, + {`{"name":"root","children":{"leaf":{}}}`, false}, + {`{"name":"root","children":{"leaf":null}}`, false}, + } { + matched, err := node.Match([]byte(tc.wire)) + require.NoError(t, err) + require.Equal(t, tc.want, matched, tc.wire) + } +} diff --git a/pkg/json_shape_union_test.go b/pkg/json_shape_union_test.go new file mode 100644 index 000000000..10fd1d43d --- /dev/null +++ b/pkg/json_shape_union_test.go @@ -0,0 +1,91 @@ +package loom + +import ( + "encoding/json/jsontext" + "testing" + + "github.com/stretchr/testify/require" +) + +func TestJSONShapeNestedUnions(t *testing.T) { + first := &JSONShape{Kind: "object", Fields: []JSONShapeField{{Name: "name", Required: true, Shape: &JSONShape{Kind: "string"}}}} + second := &JSONShape{Kind: "object", Fields: []JSONShapeField{{Name: "group", Required: true, Shape: &JSONShape{Kind: "string"}}}} + choice := &JSONShape{Kind: "union", Branches: []*JSONShape{first, second}} + for _, tc := range []struct { + name string + shape *JSONShape + wire string + want bool + }{ + {"first", choice, `{"name":"person"}`, true}, + {"second", choice, `{"group":"team"}`, true}, + {"zero", choice, `{}`, false}, + {"multiple", choice, `{"name":"person","group":"team"}`, false}, + {"null", choice, `null`, false}, + {"empty array", &JSONShape{Kind: "array", Child: choice}, `[]`, true}, + {"mixed rows", &JSONShape{Kind: "array", Child: choice}, `[{"name":"person"},{"group":"team"}]`, true}, + {"invalid row", &JSONShape{Kind: "array", Child: choice}, `[{"name":"person"},{}]`, false}, + {"ambiguous row", &JSONShape{Kind: "array", Child: choice}, `[{"name":"person","group":"team"}]`, false}, + {"map values", &JSONShape{Kind: "map", Child: choice}, `{"a":{"name":"person"},"b":{"group":"team"}}`, true}, + } { + t.Run(tc.name, func(t *testing.T) { + matched, err := tc.shape.Match([]byte(tc.wire)) + require.NoError(t, err) + require.Equal(t, tc.want, matched) + }) + } + choice.Nullable = true + matched, err := choice.Match([]byte(`null`)) + require.NoError(t, err) + require.True(t, matched) +} + +func TestJSONShapeUnionRejectsMalformedDescriptions(t *testing.T) { + cycle := &JSONShape{Kind: "union", Nullable: true} + cycle.Branches = []*JSONShape{{Kind: "alias", Child: cycle}, {Kind: "string"}} + for _, shape := range []*JSONShape{ + {Kind: "union"}, + {Kind: "union", Branches: []*JSONShape{{Kind: "string"}}}, + {Kind: "union", Branches: []*JSONShape{{Kind: "string"}, nil}}, + {Kind: "union", Branches: []*JSONShape{{Kind: "string"}, {Kind: "unknown"}}}, + cycle, + } { + shape.Nullable = true + for _, wire := range []string{`null`, `"text"`, `{}`} { + _, err := shape.Match([]byte(wire)) + require.Error(t, err) + } + } +} + +func TestJSONShapeUnionChecksEveryDescription(t *testing.T) { + failing := &JSONShape{Kind: "number", Rules: JSONShapeRules{Minimum: "invalid"}} + for _, branches := range [][]*JSONShape{ + {{Kind: "number"}, failing}, + {failing, {Kind: "number"}}, + } { + shape := &JSONShape{Kind: "union", Branches: branches} + _, err := shape.Match(jsontext.Value(`1`)) + require.Error(t, err) + } +} + +func TestJSONShapeRecursiveNestedUnion(t *testing.T) { + tree := &JSONShape{Kind: "union"} + tree.Branches = []*JSONShape{ + {Kind: "object", Closed: true, Fields: []JSONShapeField{{Name: "children", Required: true, Shape: &JSONShape{Kind: "array", Child: tree}}}}, + {Kind: "object", Closed: true, Fields: []JSONShapeField{{Name: "name", Required: true, Shape: &JSONShape{Kind: "string"}}}}, + } + for _, tc := range []struct { + wire string + want bool + }{ + {`{"children":[]}`, true}, + {`{"children":[{"name":"leaf"},{"children":[]}]}`, true}, + {`{"children":[{}]}`, false}, + } { + matched, err := tree.Match([]byte(tc.wire)) + require.NoError(t, err) + require.Equal(t, tc.want, matched) + } +} diff --git a/pkg/json_union.go b/pkg/json_union.go index 39c1deeba..ff69c700f 100644 --- a/pkg/json_union.go +++ b/pkg/json_union.go @@ -3,6 +3,7 @@ package loom import ( "encoding/json/jsontext" "encoding/json/v2" + "errors" "fmt" "github.com/CaliLuke/loom/internal/jsondecode" @@ -18,6 +19,11 @@ type ( // mismatch, or an error for a broken matcher or unsupported operation. Decode func(jsontext.Value) (bool, error) } + // jsonUnionMismatch marks an ordinary generated union contract mismatch. + // Evaluator/configuration and application codec failures retain their causes. + jsonUnionMismatch struct { + message string + } ) // MatchUntaggedJSON requires the same unique schema and typed decoder match. @@ -54,27 +60,34 @@ func MatchUntaggedJSON(name string, data []byte, branches []JSONUnionBranch) (in } } if schemaCount != 1 { - return -1, fmt.Errorf("decode %s: untagged union matched %d branches in schema", name, schemaCount) + return -1, &jsonUnionMismatch{message: fmt.Sprintf("decode %s: untagged union matched %d branches in schema", name, schemaCount)} } if decodedCount != 1 { - return -1, fmt.Errorf("decode %s: untagged union matched %d branches in decoder", name, decodedCount) + return -1, &jsonUnionMismatch{message: fmt.Sprintf("decode %s: untagged union matched %d branches in decoder", name, decodedCount)} } if schemaIndex != decodedIndex { - return -1, fmt.Errorf("decode %s: untagged union schema and decoder identities differ", name) + return -1, &jsonUnionMismatch{message: fmt.Sprintf("decode %s: untagged union schema and decoder identities differ", name)} } return decodedIndex, nil } // DecodeJSONCandidate decodes an already structurally checked candidate. -// JSON type/conversion failures are mismatches; unexpected errors propagate. +// JSON type/conversion and nested union contract failures are mismatches; +// unexpected errors propagate. // Generated callers must exclude opaque application codecs during analysis. func DecodeJSONCandidate[T any](wire jsontext.Value, destination *T) (bool, error) { err := json.Unmarshal(wire, destination, JSONOptions()) if err == nil { return true, nil } - if jsondecode.IsMismatch(err) { + var nested *jsonUnionMismatch + if jsondecode.IsMismatch(err) || errors.As(err, &nested) { return false, nil } return false, err } + +// Error returns the union contract mismatch without hiding its classification. +func (e *jsonUnionMismatch) Error() string { + return e.message +} diff --git a/pkg/json_union_nested_test.go b/pkg/json_union_nested_test.go new file mode 100644 index 000000000..8d2b98333 --- /dev/null +++ b/pkg/json_union_nested_test.go @@ -0,0 +1,66 @@ +package loom + +import ( + "encoding/json/jsontext" + "slices" + "testing" + + "github.com/stretchr/testify/require" +) + +type nestedJSONCandidate struct{} + +func (*nestedJSONCandidate) UnmarshalJSON(data []byte) error { + _, err := MatchUntaggedJSON("Inner", data, []JSONUnionBranch{ + {Schema: &JSONShape{Kind: "integer"}, Decode: func(jsontext.Value) (bool, error) { + return false, nil + }}, + {Schema: &JSONShape{Kind: "string"}, Decode: func(jsontext.Value) (bool, error) { + return false, nil + }}, + }) + return err +} + +func TestJSONCandidateNestedMismatch(t *testing.T) { + for _, wire := range []string{`{"value":1}`, `{"value":"text"}`, `{"value":{}}`} { + var value struct { + Value nestedJSONCandidate `json:"value"` + } + matched, err := DecodeJSONCandidate([]byte(wire), &value) + require.NoError(t, err, wire) + require.False(t, matched, wire) + } +} + +func TestJSONCandidateNestedFailureClassification(t *testing.T) { + var unexpected struct { + Value failingJSONCandidate `json:"value"` + } + matched, err := DecodeJSONCandidate([]byte(`{"value":{}}`), &unexpected) + require.False(t, matched) + require.ErrorIs(t, err, errJSONCandidate) + + for _, reversed := range []bool{false, true} { + var inner struct { + Value nestedJSONCandidate `json:"value"` + } + var other map[string]int + branches := []JSONUnionBranch{ + {Schema: &JSONShape{Kind: "string"}, Decode: func(wire jsontext.Value) (bool, error) { + return DecodeJSONCandidate(wire, &inner) + }}, + {Schema: &JSONShape{Kind: "map", Child: &JSONShape{Kind: "integer"}}, Decode: func(wire jsontext.Value) (bool, error) { + return DecodeJSONCandidate(wire, &other) + }}, + } + want := 1 + if reversed { + slices.Reverse(branches) + want = 0 + } + selected, err := MatchUntaggedJSON("Outer", []byte(`{"value":1}`), branches) + require.NoError(t, err) + require.Equal(t, want, selected) + } +} diff --git a/pkg/tla/untagged_json/MapChecked.cfg b/pkg/tla/untagged_json/MapChecked.cfg new file mode 100644 index 000000000..9a20c1e92 --- /dev/null +++ b/pkg/tla/untagged_json/MapChecked.cfg @@ -0,0 +1,4 @@ +CONSTANT Mode = "checked" +SPECIFICATION Spec +INVARIANT Agreement +CHECK_DEADLOCK FALSE diff --git a/pkg/tla/untagged_json/MapExistential.cfg b/pkg/tla/untagged_json/MapExistential.cfg new file mode 100644 index 000000000..9b4e1eb4a --- /dev/null +++ b/pkg/tla/untagged_json/MapExistential.cfg @@ -0,0 +1,4 @@ +CONSTANT Mode = "existential" +SPECIFICATION Spec +INVARIANT Agreement +CHECK_DEADLOCK FALSE diff --git a/pkg/tla/untagged_json/MapMembership.tla b/pkg/tla/untagged_json/MapMembership.tla new file mode 100644 index 000000000..ac754abcf --- /dev/null +++ b/pkg/tla/untagged_json/MapMembership.tla @@ -0,0 +1,30 @@ +------------------------ MODULE MapMembership ------------------------ +EXTENDS Naturals, FiniteSets +CONSTANT Mode +VARIABLES present, nullable, required, wire, entries, minimum, phase, matched +vars == <> +Init == /\ present \in BOOLEAN + /\ nullable \in BOOLEAN + /\ required \in BOOLEAN + /\ wire \in {"null", "map", "array"} + /\ entries \in SUBSET {"valid", "invalid"} + /\ minimum \in 0..1 + /\ phase = "input" + /\ matched = FALSE +EveryValue == "invalid" \notin entries +LengthOK == Cardinality(entries) >= minimum +Expected == IF ~present THEN ~required + ELSE IF wire = "null" THEN nullable + ELSE wire = "map" /\ LengthOK /\ EveryValue +ValueGate == IF Mode = "existential" THEN "valid" \in entries + ELSE EveryValue +Match == /\ phase = "input" + /\ matched' = IF ~present THEN ~required + ELSE IF wire = "null" THEN nullable + ELSE wire = "map" /\ LengthOK /\ ValueGate + /\ phase' = "matched" + /\ UNCHANGED <> +Next == Match +Spec == Init /\ [][Next]_vars +Agreement == phase = "matched" => matched = Expected +===================================================================== diff --git a/pkg/tla/untagged_json/NestedAny.cfg b/pkg/tla/untagged_json/NestedAny.cfg new file mode 100644 index 000000000..9fc4caab8 --- /dev/null +++ b/pkg/tla/untagged_json/NestedAny.cfg @@ -0,0 +1,4 @@ +CONSTANT Mode = "any" +SPECIFICATION Spec +INVARIANT Agreement +CHECK_DEADLOCK FALSE diff --git a/pkg/tla/untagged_json/NestedChecked.cfg b/pkg/tla/untagged_json/NestedChecked.cfg new file mode 100644 index 000000000..9a20c1e92 --- /dev/null +++ b/pkg/tla/untagged_json/NestedChecked.cfg @@ -0,0 +1,4 @@ +CONSTANT Mode = "checked" +SPECIFICATION Spec +INVARIANT Agreement +CHECK_DEADLOCK FALSE diff --git a/pkg/tla/untagged_json/NestedFailure.tla b/pkg/tla/untagged_json/NestedFailure.tla new file mode 100644 index 000000000..4a4cefb19 --- /dev/null +++ b/pkg/tla/untagged_json/NestedFailure.tla @@ -0,0 +1,17 @@ +------------------------ MODULE NestedFailure ------------------------ +EXTENDS Naturals +CONSTANT Mode +VARIABLES innerFailure, phase, outcome +vars == <> +Init == /\ innerFailure \in {"mismatch", "unexpected"} + /\ phase = "input" + /\ outcome = "unset" +Expected == IF innerFailure = "mismatch" THEN "other branch" ELSE "error" +Decode == /\ phase = "input" + /\ outcome' = IF Mode = "unclassified" THEN "error" ELSE Expected + /\ phase' = "decoded" + /\ UNCHANGED innerFailure +Next == Decode +Spec == Init /\ [][Next]_vars +Agreement == phase = "decoded" => outcome = Expected +===================================================================== diff --git a/pkg/tla/untagged_json/NestedFailureChecked.cfg b/pkg/tla/untagged_json/NestedFailureChecked.cfg new file mode 100644 index 000000000..9a20c1e92 --- /dev/null +++ b/pkg/tla/untagged_json/NestedFailureChecked.cfg @@ -0,0 +1,4 @@ +CONSTANT Mode = "checked" +SPECIFICATION Spec +INVARIANT Agreement +CHECK_DEADLOCK FALSE diff --git a/pkg/tla/untagged_json/NestedFailureUnclassified.cfg b/pkg/tla/untagged_json/NestedFailureUnclassified.cfg new file mode 100644 index 000000000..788dfd10f --- /dev/null +++ b/pkg/tla/untagged_json/NestedFailureUnclassified.cfg @@ -0,0 +1,4 @@ +CONSTANT Mode = "unclassified" +SPECIFICATION Spec +INVARIANT Agreement +CHECK_DEADLOCK FALSE diff --git a/pkg/tla/untagged_json/NestedMembership.tla b/pkg/tla/untagged_json/NestedMembership.tla new file mode 100644 index 000000000..40192c905 --- /dev/null +++ b/pkg/tla/untagged_json/NestedMembership.tla @@ -0,0 +1,20 @@ +---------------------- MODULE NestedMembership ---------------------- +EXTENDS Naturals, FiniteSets, Sequences +CONSTANT Mode +VARIABLES rows, phase, matched +vars == <> +Candidates == {1, 2} +Init == /\ rows \in UNION { [1..n -> SUBSET Candidates] : n \in 0..2 } + /\ phase = "input" + /\ matched = FALSE +Expected == \A i \in DOMAIN rows : Cardinality(rows[i]) = 1 +RowMatch(set) == IF Mode = "any" THEN Cardinality(set) > 0 + ELSE Cardinality(set) = 1 +Match == /\ phase = "input" + /\ matched' = \A i \in DOMAIN rows : RowMatch(rows[i]) + /\ phase' = "matched" + /\ UNCHANGED rows +Next == Match +Spec == Init /\ [][Next]_vars +Agreement == phase = "matched" => matched = Expected +===================================================================== diff --git a/pkg/tla/untagged_json/README.md b/pkg/tla/untagged_json/README.md index a7a048f6a..d570c8f9e 100644 --- a/pkg/tla/untagged_json/README.md +++ b/pkg/tla/untagged_json/README.md @@ -90,3 +90,84 @@ The model assumes the nullability query, JSON parser and enum evaluator; it does not prove arbitrary alias graphs, Go codecs or JSON Schema rendering. Those boundaries remain direct and generated-test obligations, and the original union uniqueness/transaction model remains separate. + +## Map field membership (#1041) + +`MapMembership.tla` checks universal value membership together with field +presence, map nullability and entry-count minimums. It compares universal +membership against an existential candidate: accepting any matching member +incorrectly admits a mixed valid/invalid map and rejects the unconstrained empty +map. Run TLC 1.7.4 with `MapChecked.cfg` and `MapExistential.cfg`, each with its +own temporary `-metadir`, using the Java invocation above and `MapMembership.tla`. +Checked passes 384 distinct states; existential fails `Agreement` after 297 +states. This is a capability extension; the negative candidate is a design +alternative, not a previously shipped implementation. + +The bounded abstraction assumes correct per-value matching and counts at most +one representative valid and invalid entry. It does not prove arbitrary +recursion, JSON parsing, aliases, Go validation or schema rendering. The existing +strict orchestration model still owns outer uniqueness, identity and transactional +assignment. Map traversal must not weaken those obligations. + +Correspondence tests include `pkg/json_shape_map_test.go` (empty, invalid member, +nullability, aliases, duplicates and recursive values), +`expr/value_plan_json_shape_test.go` (prepared/lowered predicate agreement), +`codegen/unionjson/union_test.go` (member policies beneath maps), and the existing +HTTP/JSON-RPC generated fixtures (typed round trips and malformed values). +The HTTP fixture additionally exercises named map validation, nested maps and +arrays, recursive values, retained destinations and ambiguous map-bearing objects. +OpenAPI 3.1/3.2 specimen checks retain value schemas and entry-count bounds. + +The before/after comparison uses base `c74af637189d1af43a5c352fb8ce0aee6af49225` +and Go 1.27.3. All six `generationDeterminismDSLs` with the original fixture +inputs remain byte-identical. With the extended map fixture, only +`UntaggedArraysDSL` changes: service types/validators/descriptors; HTTP client +and server body conversion helpers, validators and union descriptors; CLI +examples; and OpenAPI JSON/YAML map components/properties/examples. Those are +the intended differences. The five other designs remain exact-byte controls. +Five process-isolated generations of the extended inputs agree byte for byte. +No checked-in integration fixture declares `Untagged`; none needs regeneration. + +## Nested untagged unions (#1036) + +`NestedMembership.tla` models up to two rows, each with zero, one or two matching +inner branches. `NestedChecked.cfg` requires exactly one match per row and passes +42 states. `NestedAny.cfg` fails `Agreement` after 26 states because an ambiguous +row would be admitted. Empty rows are accepted by universal membership; collection +length constraints remain independent predicates. + +`NestedFailure.tla` models an inner failure while another outer branch is the +unique schema and Go match. `NestedFailureChecked.cfg` distinguishes an ordinary +contract mismatch from an unexpected failure and passes four states. +`NestedFailureUnclassified.cfg` fails after three states: treating all inner errors +as fatal blocks the valid alternative. These are design candidates for a new +capability; the previous DSL rejected nested unions before code generation. +The direct decoder regression nevertheless reproduces the unclassified error at +the runtime seam before the implementation change. + +Run TLC 1.7.4 with the invocation above, each corresponding `.tla`/`.cfg` pair +and a separate temporary `-metadir`. Both models assume accurate branch +predicates and error classification. They do not prove Go codecs, arbitrary +recursion, JSON syntax or policy extraction. Keep proof execution opt-in. + +`pkg/json_shape_union_test.go` covers exact-one nesting, empty and mixed arrays, +map values, malformed descriptors, recursive objects and unconsumed cycles. +`pkg/json_union_nested_test.go` covers wrapped mismatch classification, candidate +order and unexpected error preservation. Prepared/lowered agreement and member +policy projection are checked in `expr` and `codegen/unionjson`. The existing +HTTP specimen adds nested arrays/maps, nullable unions, recursive members, +ambiguous nested objects and the distinct valid outer alternative. JSON-RPC +checks typed round trips and invalid-row error envelopes without service calls. + +Verification uses Go 1.27.3 and base +`78236cdcad977f296c80d9eef9907f5f0d3b75ea`. The six existing determinism designs +with unchanged inputs remain byte-identical. The extended `UntaggedArraysDSL` +changes only its generated service/client/endpoints, HTTP routing and CLI files, +body conversion helpers/validators/union descriptors, and OpenAPI JSON/YAML. +Those changes add nested Entry alternatives, recursive Group members and the +Fallback endpoint used to check candidate rejection under HTTP closed-object +policy. The other five designs remain byte-identical. Five independent processes +produce identical raw sections and rendered files. Generated HTTP and JSON-RPC +modules compile and run; the 3.1/3.2 OpenAPI specimens pass parsing and pinned +Vacuum lint. Direct projection/matching checks pass with the race detector. +No checked-in integration fixture declares `Untagged`; none needs regeneration. diff --git a/security/scheme.go b/security/scheme.go index 39faad214..2c3823bf1 100644 --- a/security/scheme.go +++ b/security/scheme.go @@ -24,8 +24,9 @@ type ( Name string // Scopes holds a list of scopes for the scheme. Scopes []string - // RequiredScopes holds a list of scopes which are required - // by the scheme. It is a subset of Scopes field. + // RequiredScopes holds required application permission or role names. + // RoleFor names need not occur in Scopes. Validate checks all these + // names against the authenticated principal's application-owned grants. RequiredScopes []string } @@ -36,8 +37,9 @@ type ( Name string // Scopes holds a list of scopes for the scheme. Scopes []string - // RequiredScopes holds a list of scopes which are required - // by the scheme. It is a subset of Scopes field. + // RequiredScopes holds required application permission or role names. + // RoleFor names need not occur in Scopes. Validate checks all these + // names against the authenticated principal's application-owned grants. RequiredScopes []string } @@ -48,8 +50,9 @@ type ( Name string // Scopes holds a list of scopes for the scheme. Scopes []string - // RequiredScopes holds a list of scopes which are required - // by the scheme. It is a subset of Scopes field. + // RequiredScopes holds required application permission or role names. + // RoleFor names need not occur in Scopes. Validate checks all these + // names against the authenticated principal's application-owned grants. RequiredScopes []string } @@ -57,7 +60,8 @@ type ( OAuth2Scheme struct { // Name is the scheme name defined in the design. Name string - // Scopes holds a list of scopes for the scheme. + // Scopes lists distinct available names across the effective flow inventories. + // It describes availability, not grants held by an authenticated token. Scopes []string // RequiredScopes holds a list of scopes which are required // by the scheme. It is a subset of Scopes field. @@ -76,6 +80,10 @@ type ( TokenURL string // RefreshURL to be used for obtaining refresh token. RefreshURL string + // Scopes retains this flow's scope names and descriptions. Nil uses + // the enclosing scheme's shared Scopes names; a nonnil empty map + // declares an empty inventory. Inventories do not grant permissions. + Scopes map[string]string } // AuthBasicFunc is the function type that implements the basic auth