Repository navigation
XDRIVE transport: Support two-step "template" operations (hrefPath) - #7188
Conversation
|
需要 rebase,另外可能会先发个版,@4RH1T3CT0R7 你看看 |
Some storage APIs never take or return file content in the API call itself: the Yandex Disk REST API answers upload and download requests with a JSON body whose "href" is a presigned URL for the content. A template operation can now set "hrefPath" (and "hrefMethod", GET by default): the first request is sent as before without the payload, the URL at hrefPath in its JSON answer is then called with the payload and its response is used. The presigned URL is called as given, with no auth or template headers, so credentials never reach the content host. A non-2xx first answer is returned unchanged, so a 404 still reads as a missing object.
016f007 to
cb8a52e
Compare
|
未深度优化版 XDRIVE 的速度太差了,@genkauzen 你通过 Yandex Disk API 下载个大文件是什么速度?#6748 (comment) 按我说的方案深度优化 XDRIVE 后它在俄罗斯、伊朗未尝不能取代 XHTTP, |
|
Plain curl against the Yandex Disk REST API, free account, no xray in the path. From a VPS in Russia:
From an EU host behind a VPN the download was 7.5 MB/s, which is that link's limit. So storage bandwidth is not what limits it. Through XDRIVE I got about 66 KB/s down, and that is per-segment latency: two requests per op plus list polling. Downloading the next segment by name instead of listing would save one 0.4 s call per segment here, the same as you described for Google Drive. |
|
@RPRX looked at it. it only acts with hrefPath set, templates without it go through do as before, and the href gets neither auth nor template headers, a 401 on it doesn't drop the token. tests pass here with -race, alone and merged with #7187, they don't conflict. one thing for the docs line: if streamSettings address or tlsSettings serverName is set, the href host is dialed at that address and with that sni too, so for yandex disk, whose hrefs point to other hosts than cloud-api, both have to stay unset |
|
@genkauzen 也就是说 Yandex Disk 短时间内上传两个 64MB 就开始限速了? |
|
Not after two. The two slow ones were each the first upload of a separate run (two runs back to back, one from the RU VPS and one from an EU host), so more likely a throttle that was on for that account or that hour than one triggered by volume. Just now from the RU VPS, one after another:
That is 960 MiB up in about 20 s with no slowdown. I couldn't reproduce the 128 KiB/s again. |
Why
Some storage APIs never take or return file content in the API call itself. The Yandex Disk REST API is one:
GET /v1/disk/resources/upload?path=…andGET /v1/disk/resources/download?path=…answer with JSON{"href": "<presigned URL>"}, and the content goes to or comes from that URL. Today a template op is a singlerequest, so the template service cannot store segments on Yandex Disk through REST. The only one-request
alternative, WebDAV, is refused on free Disk accounts ("WebDAV is not available for the free tariff.", HTTP 402,
checked 2026-10-16 on a free and on a Yandex Plus account).
What
opTemplategets two optional fields:hrefPath— a JSON path (same syntax astokenPath) in the first answer that holds the URL of a second request;hrefMethod— the method of that request,GETby default.With
hrefPathset, the first request is sent without the payload, then the href is called with the payload and itsanswer is what Put/Get/List/Delete see. The href is called as given — no
{…}substitution, no auth, no templateheaders — so credentials never reach the content host. A non-2xx first answer is returned unchanged, so a 404 from
/downloadstill reads as a missing object. Retries and the concurrency limit apply to both requests.Templates without
hrefPathbehave exactly as before.Example: Yandex Disk REST
With an
app:/path and an OAuth app limited tocloud_api:disk.app_folder, the shared secret can only touch theapp folder, not the whole Disk.
Tests
TestTemplateTwoStep: put/get through a fake REST-style store, 404 →errNotFound, no credentials sent to the href.TestTemplateTwoStepTransport: a full client/server stream over the two-step store.go test ./transport/internet/xdrive/passes.Live (free Yandex Disk account, client and server on one macOS host, VLESS over xdrive, mux on):
first run HTTPS page 13 s cold / 11 s after reconnect, 1 MB download at ~66 KB/s, 300 KB upload at ~15 KB/s,
no errors, segments cleaned up (0 objects left). Later runs the same day were slower and flakier
(HTTPS 22–33 s, some requests timing out, 24–34 KB/s), equally with the change on 0888beb and rebased on
3834e50, so that is Yandex-side variance, not the patch. Every op checked on its own against Yandex after the
rebase: put/list/get/404/delete all fine.
Docs:
hrefPath/hrefMethodwould need a line in the XDRIVE template section of Xray-docs-next.