Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,20 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

## [Unreleased]

### Fixed

- Include form-body selectors and time bounds in POST read-cache identities.
Requests with different form parameters no longer share a URL-only memo or
metadata response. Matching GET/POST parameters can reuse metadata answers;
normalization preserves body precedence and the original request form.
Affected metadata cache namespaces are versioned so earlier ambiguous entries
cannot be reused from memory, disk or peers after upgrade.

- Serve `GET/POST /loki/api/v1/label` directly through the existing labels
handler, matching Loki's singular labels alias. Both paths share tenant/auth
admission, request semantics and labels metadata-cache answers; the alias no longer
redirects to an invalid label-values URL.

### Security

- Restrict a Gitleaks false-positive exception to one historical browser-proof
Expand Down
10 changes: 5 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -94,14 +94,14 @@ Loki profile shown: `loki-defaults`, Loki with its own defaults (align_queries_w

| | Now |
|---|---|
| Loki endpoints the proxy routes | **24 of 27** (89%) |
| Endpoints with tests that compare the proxy against Loki | **18 of 27** (67%) |
| Endpoints whose registry state is `proven` | **11 of 27** (41%) |
| Routed endpoints answered natively by VictoriaLogs | 5 of 24 |
| Loki endpoints the proxy routes | **25 of 27** (93%) |
| Endpoints with tests that compare the proxy against Loki | **19 of 27** (70%) |
| Endpoints whose registry state is `proven` | **12 of 27** (44%) |
| Routed endpoints answered natively by VictoriaLogs | 6 of 25 |
| Behaviour and feature items | 6 proven, 19 partial, 3 gaps |
| Known differences from Loki (ranked) | **87**: 67 open, 12 documented, 6 owner-kept, 2 upstream |

Read routes not implemented: `/loki/api/v1/cache/generation_numbers`, `/loki/api/v1/label`, `/loki/api/v1/index/shards` (optional planner gap, low priority).
Read routes not implemented: `/loki/api/v1/cache/generation_numbers`, `/loki/api/v1/index/shards` (optional planner gap, low priority).

These are registry evidence counts, not a full runtime compatibility percentage. Optional deletion management is excluded from the read totals; deprecated aliases and write-only APIs are outside this score.

Expand Down
103 changes: 103 additions & 0 deletions bench/read-api/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,103 @@
# Read API alias and POST cache proof

This proof compares a base proxy, the candidate proxy and Loki against one
dual-written fixture. It checks a real filtered labels response, GET/POST form
equivalence, direct singular-alias responses, isolation from a second empty
selector and proxy upstream select counters.
It does not measure general query performance or fleet behavior.

`compose.json` is an isolated four-service subset of the existing compatibility
stack. Loki 3.7.9 and VictoriaLogs 1.52.0 have separate named volumes from the
playground. Each proxy has its own disk cache; peers are disabled. Primary L1 is
256 MiB with 10,000 entries, the additional GET compatibility cache is 10%, and
the logical disk cache budget is 512 MiB per proxy. `GOMEMLIMIT=2GiB` is a Go
runtime target, not a cgroup limit or measured RSS. Loki has a 12 GiB container
limit and VictoriaLogs an 8 GiB limit with a 3 GiB internal memory allowance.

## Reproduce

Run from the repository root with Docker, Go and Python 3 available. Use the
base/candidate revisions in the retained result. Build the base image from a
clean worktree at the base revision, then build the candidate from its clean
worktree. The commands below assume the current checkout is the candidate.

```sh
git worktree add --detach /tmp/read-api-base <base-sha>
docker build --build-arg REVISION=<base-sha> -t loki-vl-proxy:read-api-base /tmp/read-api-base
docker build --build-arg REVISION=<candidate-sha> -t loki-vl-proxy:read-api-local .
docker compose -p read-api-proof -f bench/read-api/compose.json up -d --no-build
```

Wait until Loki `/ready` on port 34101 and both proxy `/ready` endpoints on
34100/34102 return 200. On a **new proof stack**, ingest once:

```sh
LOKI_URL=http://127.0.0.1:34101 PROXY_URL=http://127.0.0.1:34100 VL_URL=http://127.0.0.1:34428 \
go test -tags=e2e ./test/e2e-compat -run '^TestSetup_IngestLogs$' -count=1 -v
```

Do not ingest the fixture repeatedly into retained volumes: the backends have
different duplicate-ingestion behavior. Then run the CI-wired core regression
and the read-only comparison:

```sh
LOKI_URL=http://127.0.0.1:34101 PROXY_URL=http://127.0.0.1:34100 VL_URL=http://127.0.0.1:34428 \
go test -tags=e2e ./test/e2e-compat -run '^TestCompat_LabelsSingularAlias$' -count=1 -v
python3 bench/read-api/labels_alias_proof.py \
--base-sha <base-sha> --pr-sha <candidate-sha> --out /tmp/read-api-result.json
```

The harness uses one fixed one-hour window and `{app="e2e-test"}`. Canonical GET
precedes its alias request; POST follows GET. The stack can already be warm from
the endpoint test, so this is not a controlled cold-cache measurement. Responses must be
nonempty and the candidate must match Loki. The retained counter rows, rather
than the script exit status alone, establish whether alias reuse avoided a
backend request. Absent counters are `null`, not zero. Counter resets abort the
proof. Do not run unrelated traffic against these proof proxies during capture.
The next POST uses a missing-app selector before its GET control. It must not
reuse the preceding app's labels. Loki omits `data` for this empty response;
the proxy returns `data: []`. Both mean no labels; the raw shapes are retained.

## Historical retained comparison

[Measured JSON](results/2026-10-31-labels-alias.json) records base `28a5a2f`
and candidate `c320190`, the exact committed runtime sources used for the images.
This capture predates the merge of #947. The branch now includes that S2 change
after a rebase, so these rows are not measurements of the rebased head. The
original measured identities remain unchanged; they must not be relabelled as
the current main and candidate.

| Signal | Historical main runtime | Historical candidate | Loki reference |
|---|---|---|---|
| Singular alias, GET and POST | 307 redirect | 200, same filtered labels as plural | 200 |
| Filtered canonical POST immediately after GET, VL selects | 4 | 0 | Not collected |
| Missing-app POST after nonempty app POST | Wrong previous labels | Empty, matches its GET control | Empty |
| Missing-app POST, VL selects | 0, wrong cache reuse | 3, required new answer | Not collected |
| Alias requests after canonical priming, VL selects | 0, redirect only | 0, correct labels | Not collected |

These are one sequence's process-counter deltas. They include the cost of
rejecting unsafe reuse: the candidate reads VL for the genuinely new selector.
CPU, RSS, physical disk traffic, per-tier occupancy/evictions, p50/p95 and fleet
scaling are not measured. Budgets and node count are unchanged; no capacity or
general performance improvement is claimed.

## Interpretation and limits

- Primary labels metadata entries are shared by the two routes and equivalent
GET/POST forms. The optional outer GET compatibility cache is path-specific.
- Unit tests cover differing selectors/windows, body precedence, preserved raw
forms, malformed partial forms, tenant/auth isolation and migration away from
ambiguous old namespaces. HTTP migration tests seed old RAM, SSD and outer
compatibility entries. The same versioned keys address peers and stale reads;
this proof does not run a live peer-loss or mixed-version fleet experiment.
- Namespace fencing causes initial misses after an upgrade. Old records remain
subject to existing expiry and eviction; this change does not implement a
deletion-generation protocol or change configured cache capacity.
- Upstream select counters are logical proxy-to-VL requests. They are not SSD
reads, IOPS, CPU, RSS, backend-server resource use or per-tier hit ratios.
- Shifted windows, dashboard refresh, disk restart, pressure, peer loss, p50/p95,
throughput, allocation cost and process resource deltas are not measured here.
Existing admission, completeness and freshness rules continue to apply.

This is a manually reproducible retained proof. It adds no CI measurement job;
the Go endpoint regression runs in the existing `TestCompat_.*` core shard.
243 changes: 243 additions & 0 deletions bench/read-api/compose.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,243 @@
{
"name": "read-api-proof",
"services": {
"loki": {
"command": [
"-config.file=/etc/loki/local-config.yaml",
"-compactor.retention-enabled=true",
"-compactor.working-directory=/tmp/loki/compactor",
"-compactor.delete-request-store=filesystem",
"-compactor.compaction-interval=10m",
"-compactor.retention-delete-delay=1h",
"-store.retention=72h",
"-validation.reject-old-samples.max-age=72h"
],
"entrypoint": null,
"environment": {
"GOGC": "80",
"GOMEMLIMIT": "10GiB"
},
"healthcheck": {
"disable": true
},
"image": "grafana/loki:3.7.9",
"mem_limit": "12884901888",
"ports": [
{
"mode": "ingress",
"target": 3100,
"published": "34101",
"protocol": "tcp",
"host_ip": "127.0.0.1"
}
],
"restart": "unless-stopped",
"user": "0:0",
"volumes": [
{
"type": "bind",
"source": "../../test/e2e-compat/loki-local-config.yaml",
"target": "/etc/loki/local-config.yaml",
"read_only": true,
"bind": {}
},
{
"type": "volume",
"source": "e2e-loki-data",
"target": "/tmp/loki",
"volume": {}
}
]
},
"victorialogs": {
"command": [
"-storageDataPath=/vlogs",
"-retentionPeriod=3d",
"-httpListenAddr=:9428",
"-search.maxConcurrentRequests=16",
"-search.maxQueueDuration=60s",
"-search.maxQueryDuration=60s",
"-blockcache.missesBeforeCaching=1",
"-internStringCacheExpireDuration=15m",
"-memory.allowedBytes=3GiB",
"-inmemoryDataFlushInterval=1s"
],
"entrypoint": null,
"healthcheck": {
"test": [
"CMD",
"/victoria-logs-prod",
"-version"
],
"timeout": "2s",
"interval": "3s",
"retries": 10,
"start_period": "5s"
},
"image": "victoriametrics/victoria-logs:v1.52.0",
"mem_limit": "8589934592",
"ports": [
{
"mode": "ingress",
"target": 9428,
"published": "34428",
"protocol": "tcp",
"host_ip": "127.0.0.1"
}
],
"restart": "unless-stopped",
"volumes": [
{
"type": "volume",
"source": "vl-data",
"target": "/vlogs",
"volume": {}
}
]
},
"loki-vl-proxy": {
"command": [
"-listen=:3100",
"-backend=http://victorialogs:9428",
"-max-stats-query-series=1000000",
"-tenant-default-limits={\"max_entries_limit_per_query\":0,\"max_query_length\":\"721h\",\"query_timeout\":\"5m\",\"retention_period\":\"3d\"}",
"-backend-timeout=5m",
"-log-level=info",
"-cache-ttl=5s",
"-label-values-indexed-cache=true",
"-label-values-hot-limit=200",
"-label-values-index-max-entries=50000",
"-label-values-index-persist-path=/cache/label-values-index.json",
"-label-values-index-persist-interval=5s",
"-label-values-index-startup-stale-threshold=30s",
"-label-values-index-startup-peer-warm-timeout=2s",
"-disk-cache-path=/cache/proxy-l2.bolt",
"-disk-cache-max-bytes=536870912",
"-disk-cache-min-ttl=1s",
"-drilldown-scan-timeout=3s",
"-rate-limit-per-second=0",
"-rate-limit-burst=0",
"-max-concurrent=0",
"-metadata-default-lookback=1h",
"-label-style=underscores",
"-metadata-field-mode=translated",
"-emit-structured-metadata=true",
"-server.register-instrumentation=true"
],
"depends_on": {
"victorialogs": {
"condition": "service_healthy",
"required": true
}
},
"entrypoint": null,
"environment": {
"GOMEMLIMIT": "2GiB"
},
"healthcheck": {
"test": [
"CMD",
"/usr/local/bin/healthcheck"
],
"timeout": "2s",
"interval": "3s",
"retries": 10,
"start_period": "5s"
},
"image": "loki-vl-proxy:read-api-local",
"ports": [
{
"mode": "ingress",
"target": 3100,
"published": "34100",
"protocol": "tcp",
"host_ip": "127.0.0.1"
}
],
"user": "root",
"volumes": [
{
"type": "volume",
"source": "e2e-proxy-cache",
"target": "/cache",
"volume": {}
}
]
},
"loki-vl-proxy-base": {
"command": [
"-listen=:3100",
"-backend=http://victorialogs:9428",
"-max-stats-query-series=1000000",
"-tenant-default-limits={\"max_entries_limit_per_query\":0,\"max_query_length\":\"721h\",\"query_timeout\":\"5m\",\"retention_period\":\"3d\"}",
"-backend-timeout=5m",
"-log-level=info",
"-cache-ttl=5s",
"-label-values-indexed-cache=true",
"-label-values-hot-limit=200",
"-label-values-index-max-entries=50000",
"-label-values-index-persist-path=/cache/label-values-index.json",
"-label-values-index-persist-interval=5s",
"-label-values-index-startup-stale-threshold=30s",
"-label-values-index-startup-peer-warm-timeout=2s",
"-disk-cache-path=/cache/proxy-l2.bolt",
"-disk-cache-max-bytes=536870912",
"-disk-cache-min-ttl=1s",
"-drilldown-scan-timeout=3s",
"-rate-limit-per-second=0",
"-rate-limit-burst=0",
"-max-concurrent=0",
"-metadata-default-lookback=1h",
"-label-style=underscores",
"-metadata-field-mode=translated",
"-emit-structured-metadata=true",
"-server.register-instrumentation=true"
],
"depends_on": {
"victorialogs": {
"condition": "service_healthy",
"required": true
}
},
"entrypoint": null,
"environment": {
"GOMEMLIMIT": "2GiB"
},
"healthcheck": {
"test": [
"CMD",
"/usr/local/bin/healthcheck"
],
"timeout": "2s",
"interval": "3s",
"retries": 10,
"start_period": "5s"
},
"image": "loki-vl-proxy:read-api-base",
"ports": [
{
"mode": "ingress",
"target": 3100,
"published": "34102",
"protocol": "tcp",
"host_ip": "127.0.0.1"
}
],
"user": "root",
"volumes": [
{
"type": "volume",
"source": "e2e-proxy-base-cache",
"target": "/cache",
"volume": {}
}
]
}
},
"volumes": {
"vl-data": {},
"e2e-loki-data": {},
"e2e-proxy-base-cache": {},
"e2e-proxy-cache": {}
}
}
Loading
Loading