Skip to content

XDRIVE transport: Support two-step "template" operations (hrefPath) - #7188

Merged
RPRX merged 1 commit into
XTLS:mainfrom
genkauzen:xdrive-template-href
Oct 21, 2026
Merged

RPRX merged 1 commit into
XTLS:mainfrom
genkauzen:xdrive-template-href

Conversation

@genkauzen

@genkauzen genkauzen commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor

Why

Some storage APIs never take or return file content in the API call itself. The Yandex Disk REST API is one:
GET /v1/disk/resources/upload?path=… and GET /v1/disk/resources/download?path=… answer with JSON
{"href": "<presigned URL>"}, and the content goes to or comes from that URL. Today a template op is a single
request, so the template service cannot store segments on Yandex Disk through REST. The only one-request
alternative, WebDAV, is refused on free Disk accounts ("WebDAV is not available for the free tariff.", HTTP 402,
checked 2026-10-16 on a free and on a Yandex Plus account).

What

opTemplate gets two optional fields:

  • hrefPath — a JSON path (same syntax as tokenPath) in the first answer that holds the URL of a second request;
  • hrefMethod — the method of that request, GET by default.

With hrefPath set, the first request is sent without the payload, then the href is called with the payload and its
answer is what Put/Get/List/Delete see. The href is called as given — no {…} substitution, no auth, no template
headers — so credentials never reach the content host. A non-2xx first answer is returned unchanged, so a 404 from
/download still reads as a missing object. Retries and the concurrency limit apply to both requests.
Templates without hrefPath behave exactly as before.

Example: Yandex Disk REST

"template": {
  "flatten": true,
  "auth": {"type": "static", "header": {"Authorization": "OAuth {secret0}"}},
  "put":    {"url": "https://cloud-api.yandex.net/v1/disk/resources/upload?path=disk:/{folder}/{name}&overwrite=true",
             "hrefPath": "href", "hrefMethod": "PUT"},
  "get":    {"url": "https://cloud-api.yandex.net/v1/disk/resources/download?path=disk:/{folder}/{name}",
             "hrefPath": "href"},
  "delete": {"method": "DELETE",
             "url": "https://cloud-api.yandex.net/v1/disk/resources?path=disk:/{folder}/{name}&permanently=true"},
  "list":   {"url": "https://cloud-api.yandex.net/v1/disk/resources?path=disk:/{folder}&limit=10000&fields=_embedded.items.name",
             "namesRegex": "\"name\":\"([^\"]+)\""},
  "retry":  {"status": [423, 429, 500, 502, 503, 504]}
}

With an app:/ path and an OAuth app limited to cloud_api:disk.app_folder, the shared secret can only touch the
app folder, not the whole Disk.

Tests

  • TestTemplateTwoStep: put/get through a fake REST-style store, 404 → errNotFound, no credentials sent to the href.
  • TestTemplateTwoStepTransport: a full client/server stream over the two-step store.
  • Both fail without the change; go test ./transport/internet/xdrive/ passes.

Live (free Yandex Disk account, client and server on one macOS host, VLESS over xdrive, mux on):
first run HTTPS page 13 s cold / 11 s after reconnect, 1 MB download at ~66 KB/s, 300 KB upload at ~15 KB/s,
no errors, segments cleaned up (0 objects left). Later runs the same day were slower and flakier
(HTTPS 22–33 s, some requests timing out, 24–34 KB/s), equally with the change on 0888beb and rebased on
3834e50, so that is Yandex-side variance, not the patch. Every op checked on its own against Yandex after the
rebase: put/list/get/404/delete all fine.

Docs: hrefPath / hrefMethod would need a line in the XDRIVE template section of Xray-docs-next.

@RPRX

RPRX commented Oct 17, 2026

Copy link
Copy Markdown
Member

需要 rebase,另外可能会先发个版,@4RH1T3CT0R7 你看看

Some storage APIs never take or return file content in the API call
itself: the Yandex Disk REST API answers upload and download requests
with a JSON body whose "href" is a presigned URL for the content.

A template operation can now set "hrefPath" (and "hrefMethod", GET by
default): the first request is sent as before without the payload, the
URL at hrefPath in its JSON answer is then called with the payload and
its response is used. The presigned URL is called as given, with no
auth or template headers, so credentials never reach the content host.
A non-2xx first answer is returned unchanged, so a 404 still reads as
a missing object.
@genkauzen
genkauzen force-pushed the xdrive-template-href branch from 016f007 to cb8a52e Compare October 17, 2026 15:50
@RPRX

RPRX commented Oct 18, 2026

Copy link
Copy Markdown
Member

未深度优化版 XDRIVE 的速度太差了,@genkauzen 你通过 Yandex Disk API 下载个大文件是什么速度?#6748 (comment)

按我说的方案深度优化 XDRIVE 后它在俄罗斯、伊朗未尝不能取代 XHTTP,不打游戏的话

@genkauzen

Copy link
Copy Markdown
Contributor Author

Plain curl against the Yandex Disk REST API, free account, no xray in the path.

From a VPS in Russia:

  • download, 64 MiB, one connection: 55.7 MB/s. The download href redirects to *.storage.yandex.net, Range works (206). 4 parallel ranges gave 23.9 MB/s total, so one connection is already the fastest.
  • upload, one connection: 4 MiB at 6.8 MB/s, 32 MiB at 35 MB/s, 64 MiB at 40 MB/s. 4 parallel 4 MiB uploads: 7.1 MB/s total, href request included.
  • twice a 64 MiB upload was held at about 128 KiB/s for 8.5 minutes (once from the RU VPS, once from an EU host, same hour). Later the same size went at 40 MB/s, so it looks like a temporary throttle on Yandex's side.
  • latency: a 64 KiB put (href request + PUT) takes 1.4 to 2.6 s. The object is readable by name (download href + GET, no list) 0.7 to 0.8 s after the put returns. Each API call alone is about 0.4 s.

From an EU host behind a VPN the download was 7.5 MB/s, which is that link's limit.

So storage bandwidth is not what limits it. Through XDRIVE I got about 66 KB/s down, and that is per-segment latency: two requests per op plus list polling. Downloading the next segment by name instead of listing would save one 0.4 s call per segment here, the same as you described for Google Drive.

@4RH1T3CT0R7

Copy link
Copy Markdown
Contributor

@RPRX looked at it. it only acts with hrefPath set, templates without it go through do as before, and the href gets neither auth nor template headers, a 401 on it doesn't drop the token. tests pass here with -race, alone and merged with #7187, they don't conflict. one thing for the docs line: if streamSettings address or tlsSettings serverName is set, the href host is dialed at that address and with that sni too, so for yandex disk, whose hrefs point to other hosts than cloud-api, both have to stay unset

@RPRX

RPRX commented Oct 18, 2026

Copy link
Copy Markdown
Member

@genkauzen 也就是说 Yandex Disk 短时间内上传两个 64MB 就开始限速了?上传限速版百度网盘? 如果是一个 128MB 呢?

@genkauzen

Copy link
Copy Markdown
Contributor Author

Not after two. The two slow ones were each the first upload of a separate run (two runs back to back, one from the RU VPS and one from an EU host), so more likely a throttle that was on for that account or that hour than one triggered by volume.

Just now from the RU VPS, one after another:

  • 128 MiB: 54.7 MB/s
  • 64 MiB x3: 44.4, 36.1, 35.4 MB/s
  • 128 MiB: 56.2 MB/s
  • 512 MiB: 92.6 MB/s up, then 119 MB/s down (probably the VPS link)

That is 960 MiB up in about 20 s with no slowdown. I couldn't reproduce the 128 KiB/s again.

@RPRX
RPRX merged commit 48272e8 into XTLS:main Oct 21, 2026
40 checks passed
@RPRX RPRX changed the title XDRIVE transport: Support two-step template operations (hrefPath) XDRIVE transport: Support two-step "template" operations (hrefPath) Oct 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants