Repository navigation
feat(rsync): index files as directory pages, UTF-8 charset labels - #27
Merged
Merged
Conversation
INDEX_FILES (empty by default) names the upstream index files, in sequence. For a directory with such a file, pages writes the listing only to the slashless key and records "<source key> TAB <page key>" in .run/index-real.txt. index then copies the file to the page key with one server-side copy-object (--metadata-directive COPY), so /dir/ serves the same bytes, Content-Type and ETag as the index file. label adds "; charset=utf-8" to a text/* type when the whole file decodes as strict UTF-8. Generated pages upload as "text/html; charset=utf-8". smoke compares the full Content-Type value (CTYPE), not the first word, and for an index directory checks the copy against its source. index-real.txt starts each line with a full key: read with a TAB in IFS drops an empty first field, which the root directory would be.
The CLI percent-encodes --copy-source, so the encoded key went out encoded twice and the copy read a different key. The copy loop is now index-copy; offline runs it against a stub aws and compares the exact arguments for a key with a space and a plus. The run summary counts the copied page keys. The README says to set INDEX_FILES only where upstream serves the index file, and that ctan does not: CTAN asks its mirrors for DirectoryIndex disabled.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
label: atext/*type gets; charset=utf-8when the whole file decodes as strict UTF-8 (read 1 MiB at a time). Other text gets no charset. Generated pages upload astext/html; charset=utf-8.smoke: compares the full Content-Type value (newCTYPE), not the first word, so a type with a charset compares correctly.INDEX_FILES(new, empty by default): names of upstream index files, in sequence. For a directory with such a file,pageswrites the listing only to the slashless key and adds<source key> TAB <page key>to.run/index-real.txt;index-copythen copies the file to the page key with ones3api copy-object --metadata-directive COPY(same bytes, Content-Type and ETag)./dir/then serves the index file, as the upstream server does. The README says to set it only where upstream serves the index file at the directory URL; CTAN asks its mirrors not to (DirectoryIndex disabled).smokepage check: for a copied page key, size and type must equal the source; the slashless key is compared with the listing in the SLASH tree.With
INDEX_FILESunset (every mirror until its own PR), behaviour is unchanged apart from the charset on labels and pages.Test plan
task run -- task offline(new cases:pagespicksindex.htmloverindex.htm; label charset for ASCII, UTF-8, Latin-1, UTF-16, empty;smokeaccepts an equal copy and refuses one a byte longer;index-copypasses the raw key with a space and+to a stubaws)task check(render.txt updated)validate-vars.sh --check,chain-file.sh --check,gpg-gate-check.sh,op-retry-check.sh/timestampon ctan served astext/plain; charset=utf-8; smoke lines show charset types and pass