Skip to content

Commit d4c6e5d

Browse files
committed
test(web): serve.mjs --pages answers as Cloudflare Pages does, and e2e sees a terminal and no script answered with HTML
The first deploy to Cloudflare Pages (app.levelcode.ai, 2026-10-19) went live without its terminal: the client loads xterm, vscode-textmate and vscode-oniguruma from static/node_modules, wrangler never uploads a folder by that name, and Pages answered each of those scripts with index.html (200, text/html), so the browser refused to run them. Every check had passed against serve.mjs, a plain file server. web/lib/pages.mjs is what Pages does differently, and serve.mjs --dist <release> --pages answers that way: only what `wrangler pages deploy` uploads (its IGNORE_LIST: **/node_modules, **/.git, **/.DS_Store, its own files at the root; symbolic links skipped), _redirects first (200 rewrites, 3xx redirects, :name and * placeholders, first match wins), .html redirected to the address without it and served there, and index.html with 200 for an address that has no file. e2e.mjs, e2e-look.mjs and e2e-signin-tab.mjs pass --pages through. e2e.mjs gains two checks: in a release, LevelCode Shell: New Terminal opens a terminal (its library loaded); and no script, JSON or WebAssembly response was HTML (cdp.mjs now keeps each response's content type). Against the release that is live — same build id, 5bfb4df6caa1, rebuilt from the same client — `e2e.mjs --pages` fails both, as production did (30/32). The fix is the next commit. pages.mjs also holds the release's answer to the folder (NODE_MODULES_COPY, toRedirectsFile), which the next commit's build writes; pages.test.js runs that rule against every kind of path.
1 parent 3091c99 commit d4c6e5d

7 files changed

Lines changed: 200 additions & 7 deletions

File tree

‎web/lib/pages.mjs‎

Lines changed: 90 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,90 @@
1+
// How Cloudflare Pages answers a request for a release (scripts/build-web.mjs), for web/serve.mjs --pages. A release
2+
// that only ever ran under serve.mjs's plain file server went live on Pages without its terminal: Pages had never
3+
// been sent the client's node_modules, and answered every one of those scripts with the page. This is what Pages
4+
// does differently from a plain file server, so a release can be checked the way it will be served there.
5+
//
6+
// 1. What `wrangler pages deploy` does not upload is not there: any path with a segment named node_modules or .git,
7+
// .DS_Store, and _worker.js, _redirects, _headers, _routes.json and functions/ at the root (its IGNORE_LIST, in
8+
// workers-sdk packages/wrangler/src/pages/validate.ts). Symbolic links are skipped too.
9+
// 2. _redirects comes first, whatever the files: the first rule that matches applies. 200 serves another path of the
10+
// site in place (a rewrite); 301, 302, 303, 307 and 308 redirect. `:name` stands for one segment, `*` for the rest.
11+
// 3. An .html address is redirected to the same address without it (/a.html -> /a, /x/index.html -> /x/), and
12+
// /a serves a.html, /x/ serves x/index.html.
13+
// 4. An address that matches no file is answered with /index.html and 200 (what Pages does when there is no
14+
// 404.html): a missing script comes back as the page, which is the failure this is here to show.
15+
// Pure: no file system access. Callers say whether a path exists.
16+
17+
/**
18+
* The release's own answer to (1) for the client's node_modules (scripts/build-web.mjs): the same files are staged
19+
* again under this name, and _redirects serves them from there at the address the client asks for.
20+
*/
21+
export const NODE_MODULES_COPY = 'node-modules';
22+
23+
/** The release's _redirects. One rule for every build folder (:id), so a previous build kept online is served too. */
24+
export function toRedirectsFile() {
25+
return `/_/:id/static/node_modules/* /_/:id/static/${NODE_MODULES_COPY}/:splat 200\n`;
26+
}
27+
28+
/** Is this path (relative to the release, `/`-separated) one that `wrangler pages deploy` uploads? */
29+
export function uploaded(rel) {
30+
const parts = rel.split('/').filter(Boolean);
31+
if (parts.some((p) => p === 'node_modules' || p === '.git' || p === '.DS_Store')) { return false; }
32+
if (parts.length === 1 && ['_worker.js', '_redirects', '_headers', '_routes.json'].includes(parts[0])) { return false; }
33+
return parts[0] !== 'functions';
34+
}
35+
36+
/**
37+
* The rules of a _redirects file, in order: `from to [status]`, blank lines and `#` comments skipped.
38+
* @param {string} text
39+
* @returns {Array<{ from: string, to: string, status: number }>}
40+
*/
41+
export function parseRedirects(text) {
42+
const rules = [];
43+
for (const line of text.split('\n')) {
44+
const t = line.trim();
45+
if (!t || t.startsWith('#')) { continue; }
46+
const [from, to, status] = t.split(/\s+/);
47+
if (from && to) { rules.push({ from, to, status: status ? Number(status) : 302 }); }
48+
}
49+
return rules;
50+
}
51+
52+
/**
53+
* The first rule that matches `pathname`, with its destination filled in, or null.
54+
* @param {Array<{ from: string, to: string, status: number }>} rules
55+
* @param {string} pathname
56+
* @returns {{ to: string, status: number } | null}
57+
*/
58+
export function matchRedirect(rules, pathname) {
59+
for (const r of rules) {
60+
const names = [];
61+
const pattern = r.from.split('/').map((seg, i, all) => {
62+
if (seg === '*' && i === all.length - 1) { names.push('splat'); return '(.*)'; }
63+
if (seg.startsWith(':')) { names.push(seg.slice(1)); return '([^/]+)'; }
64+
return seg.replace(/[.*+?^${}()|[\]\\]/g, '\\$&');
65+
}).join('/');
66+
const m = new RegExp('^' + pattern + '$').exec(pathname);
67+
if (!m) { continue; }
68+
const values = Object.fromEntries(names.map((n, i) => [n, m[i + 1]]));
69+
const to = r.to.replace(/:([A-Za-z_][A-Za-z0-9_]*)/g, (all, n) => (n in values ? values[n] : all));
70+
return { to, status: r.status };
71+
}
72+
return null;
73+
}
74+
75+
/**
76+
* What Pages does with an address before it looks for a file: redirect an .html one, or name the file an
77+
* extensionless one stands for.
78+
* @param {string} pathname
79+
* @param {(rel: string) => boolean} isFile is there an uploaded file at this release-relative path?
80+
* @returns {{ redirect: string } | { file: string } | null} null: no such page; serve the fallback
81+
*/
82+
export function resolvePage(pathname, isFile) {
83+
if (pathname.endsWith('/index.html')) { return { redirect: pathname.slice(0, -'index.html'.length) }; }
84+
if (pathname.endsWith('.html') && isFile(pathname.slice(1))) { return { redirect: pathname.slice(0, -'.html'.length) }; }
85+
const rel = pathname.slice(1);
86+
if (pathname.endsWith('/')) { return isFile(rel + 'index.html') ? { file: rel + 'index.html' } : null; }
87+
if (isFile(rel)) { return { file: rel }; }
88+
if (isFile(rel + '.html')) { return { file: rel + '.html' }; }
89+
return null;
90+
}

‎web/serve.mjs‎

Lines changed: 19 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@
22
// Development / end-to-end server for LevelCode in the browser. Dependency-free.
33
//
44
// node web/serve.mjs --static <vscode-web dir> --extensions <dir of built extensions> [--port 8800]
5+
// node web/serve.mjs --dist <release> [--pages]
56
//
67
// It serves exactly what a static host serves in production (see scripts/build-web.mjs), with
78
// the Code-OSS web build mounted at /static and the LevelCode extensions at /extensions.
@@ -10,6 +11,7 @@ import fs from 'node:fs';
1011
import path from 'node:path';
1112
import { webConfig, renderIndex, WEB_DIR } from './lib/config.mjs';
1213
import { parseHeadersFile, headersFor } from './lib/headers.mjs';
14+
import { uploaded, parseRedirects, matchRedirect, resolvePage } from './lib/pages.mjs';
1315

1416
const args = Object.fromEntries(process.argv.slice(2).reduce((a, v, i, all) => {
1517
if (v.startsWith('--')) { a.push([v.slice(2), all[i + 1] && !all[i + 1].startsWith('--') ? all[i + 1] : 'true']); }
@@ -23,6 +25,16 @@ const STATIC_DIR = path.resolve(args.static || process.env.LEVELCODE_WEB_STATIC
2325
const EXT_DIR = path.resolve(args.extensions || process.env.LEVELCODE_WEB_EXTENSIONS || '');
2426
if (!DIST && !args.static && !process.env.LEVELCODE_WEB_STATIC) { console.error('missing --static <vscode-web dir> (or --dist <built release>)'); process.exit(2); }
2527
const RULES = DIST && fs.existsSync(path.join(DIST, '_headers')) ? parseHeadersFile(fs.readFileSync(path.join(DIST, '_headers'), 'utf8')) : { rules: [] };
28+
// --pages (with --dist): answer as Cloudflare Pages answers that release — only what wrangler uploads, its _redirects,
29+
// its .html redirects, and the page for an address that matches nothing (web/lib/pages.mjs).
30+
const PAGES = !!DIST && args.pages === 'true';
31+
const REDIRECTS = PAGES && fs.existsSync(path.join(DIST, '_redirects')) ? parseRedirects(fs.readFileSync(path.join(DIST, '_redirects'), 'utf8')) : [];
32+
/** A file Pages would have: uploaded, a regular file, not a symbolic link. */
33+
function pagesFile(rel) {
34+
if (!uploaded(rel)) { return false; }
35+
const p = within(DIST, rel);
36+
try { return !!p && fs.lstatSync(p).isFile(); } catch { return false; }
37+
}
2638

2739
const config = webConfig({
2840
account: args.account || process.env.LEVELCODE_ACCOUNT_ORIGIN,
@@ -51,7 +63,13 @@ const server = http.createServer((req, res) => {
5163
try { pathname = decodeURIComponent(url.pathname); } catch { res.writeHead(400).end('bad path'); return; }
5264

5365
let file = null;
54-
if (DIST) {
66+
if (PAGES) {
67+
const r = matchRedirect(REDIRECTS, pathname);
68+
if (r && r.status !== 200) { res.writeHead(r.status, { location: r.to + url.search }).end(); return; }
69+
const page = resolvePage(r ? r.to : pathname, pagesFile);
70+
if (page && 'redirect' in page) { res.writeHead(308, { location: page.redirect + url.search }).end(); return; }
71+
file = path.join(DIST, page ? page.file : 'index.html'); // no such file: the page, as Pages answers it
72+
} else if (DIST) {
5573
file = within(DIST, pathname === '/' ? 'index.html' : pathname.slice(1));
5674
if (file && fs.existsSync(file) && fs.statSync(file).isDirectory()) { file = path.join(file, 'index.html'); }
5775
} else {

‎web/test/cdp.mjs‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -62,7 +62,7 @@ export class Page {
6262
if (d.params.entry.level === 'error') { this.logs.push({ t: Date.now(), tag, type: 'log-error', text: `${d.params.entry.text} ${d.params.entry.url || ''}`.slice(0, 500) }); }
6363
break;
6464
case 'Network.responseReceived':
65-
this.requests.push({ url: d.params.response.url, status: d.params.response.status, type: d.params.type });
65+
this.requests.push({ url: d.params.response.url, status: d.params.response.status, type: d.params.type, mimeType: d.params.response.mimeType });
6666
if (d.params.response.status >= 400) { this.failed.push(`${d.params.response.status} ${d.params.response.url}`); }
6767
break;
6868
case 'Network.loadingFailed':

‎web/test/e2e-look.mjs‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,7 @@
1212
// The sign-in is the stand-in's (?signin=1), exactly as in e2e-signin-tab.mjs; the chat's empty state is what
1313
// a signed-in visitor sees.
1414
//
15-
// node web/test/e2e-look.mjs --dist <release dir> --stub-port <the account port it was built with> [--shots <dir>]
15+
// node web/test/e2e-look.mjs --dist <release dir> --stub-port <the account port it was built with> [--shots <dir>] [--pages]
1616
//
1717
// Release only: the look is a property of what the build ships (the theme package, chrome.css, the skinned chat
1818
// page), and the development server's pieces do not include them.
@@ -39,7 +39,7 @@ const results = [];
3939
const check = (name, ok, detail = '') => { results.push(ok); console.log(`${ok ? 'PASS' : 'FAIL'} ${name}${detail ? ' — ' + detail : ''}`); return ok; };
4040

4141
const stub = await startStub({ editorOrigin, port: args['stub-port'] ? Number(args['stub-port']) : 0 });
42-
const server = spawn(process.execPath, [path.join(HERE, '..', 'serve.mjs'), '--port', String(PORT), '--dist', DIST], { stdio: 'inherit' });
42+
const server = spawn(process.execPath, [path.join(HERE, '..', 'serve.mjs'), '--port', String(PORT), '--dist', DIST, ...(args.pages ? ['--pages'] : [])], { stdio: 'inherit' });
4343
let chrome;
4444
const finish = async (code) => { try { chrome && chrome.close(); } catch {} try { server.kill(); } catch {} try { await stub.close(); } catch {} process.exit(code); };
4545
process.on('SIGINT', () => finish(130));

‎web/test/e2e-signin-tab.mjs‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,7 @@
1717
// and an old one is not.
1818
//
1919
// node web/test/e2e-signin-tab.mjs --static <vscode-web dir> --extensions <built extensions dir> [--product <json>]
20-
// node web/test/e2e-signin-tab.mjs --dist <release dir> --stub-port <the account port it was built with>
20+
// node web/test/e2e-signin-tab.mjs --dist <release dir> --stub-port <the account port it was built with> [--pages]
2121
import { spawn } from 'node:child_process';
2222
import fs from 'node:fs';
2323
import path from 'node:path';
@@ -43,7 +43,7 @@ const check = (name, ok, detail = '') => { results.push(ok); console.log(`${ok ?
4343

4444
const stub = await startStub({ editorOrigin, port: args['stub-port'] ? Number(args['stub-port']) : 0 });
4545
const server = spawn(process.execPath, [path.join(HERE, '..', 'serve.mjs'), '--port', String(PORT), ...(DIST
46-
? ['--dist', DIST]
46+
? ['--dist', DIST, ...(args.pages ? ['--pages'] : [])]
4747
: ['--static', STATIC, '--extensions', EXTS, '--extension-names', 'levelcode-ai,levelcode-web', '--account', stub.origin,
4848
...(args.product ? ['--product', args.product] : [])])], { stdio: 'inherit' });
4949
// With the extension host on an origin of its own (--product with webEndpointUrlTemplate, or a release built with

‎web/test/e2e.mjs‎

Lines changed: 14 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@
88
// the real one must. It proves the editor's half of the contract; thin.ly's half is its own specs.
99
//
1010
// node web/test/e2e.mjs --static <vscode-web dir> --extensions <built extensions dir> [--shots <dir>]
11+
// node web/test/e2e.mjs --dist <release dir> --stub-port <port> [--pages] (--pages: served as Cloudflare Pages serves it)
1112
import { spawn } from 'node:child_process';
1213
import fs from 'node:fs';
1314
import path from 'node:path';
@@ -37,7 +38,7 @@ const shot = async (page, name) => { if (SHOTS) { await page.screenshot(path.joi
3738
// A release has its account origin baked into index.html, so the stand-in has to be where it was built to look.
3839
const stub = await startStub({ editorOrigin, port: args['stub-port'] ? Number(args['stub-port']) : 0 });
3940
const server = spawn(process.execPath, [path.join(HERE, '..', 'serve.mjs'), '--port', String(EDITOR_PORT), ...(DIST
40-
? ['--dist', DIST]
41+
? ['--dist', DIST, ...(args.pages ? ['--pages'] : [])]
4142
: ['--static', STATIC, '--extensions', EXTS, '--extension-names', args['extension-names'] || 'levelcode-ai,levelcode-web', '--account', stub.origin,
4243
...(args.product ? ['--product', args.product] : [])])], { stdio: 'inherit' });
4344
let chrome;
@@ -226,6 +227,18 @@ try {
226227
check('after a reload the file that was kept is still there', (await scratchFile('/scratch/notes.md')) === 'notes\n');
227228
await shot(page, '4-reload');
228229

230+
/* 4b. what the client loads from its own node_modules: the terminal's library (xterm), the syntax colouring
231+
(vscode-textmate, vscode-oniguruma). A host that does not have a file answers with something else — Cloudflare
232+
Pages, which is never sent a folder named node_modules, answers with the page — and the editor runs without it. */
233+
if (DIST) {
234+
await page.palette('LevelCode Shell: New Terminal');
235+
const term = await page.waitFor(() => page.eval(`[...document.querySelectorAll('.xterm .xterm-screen')].some((e) => e.getBoundingClientRect().width > 0)`), { ms: 30000, label: 'a terminal' }).catch(() => false);
236+
check('a terminal opens (LevelCode Shell: New Terminal): its library loaded', !!term);
237+
}
238+
const asPage = page.requests.filter((r) => /\.(m?js|json|wasm)(\?|$)/.test(r.url) && /text\/html/.test(r.mimeType || ''));
239+
check('no script, JSON or WebAssembly file came back as an HTML page (a file the host does not have)', asPage.length === 0,
240+
JSON.stringify(asPage.slice(0, 4).map((r) => r.url.replace(/^https?:\/\/[^/]+/, ''))));
241+
229242
/* 5. a folder from "the computer". The OS picker needs a person, so the page's File System Access entry point is
230243
replaced by one that hands back a real FileSystemDirectoryHandle (a folder of the origin-private file system);
231244
everything after the picker — the command, the workbench's file provider, the agent's reads and writes — is real. */

‎web/test/unit/pages.test.js‎

Lines changed: 72 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,72 @@
1+
/*---------------------------------------------------------------------------------------------
2+
* web/lib/pages.mjs — run: node web/test/unit/pages.test.js
3+
*
4+
* How Cloudflare Pages answers for a release, as web/serve.mjs --pages answers: what wrangler uploads, the _redirects
5+
* rules, the .html redirects and the page for anything else. And the release's own answer to the one that broke the
6+
* terminal: every file of the client's node_modules, which Pages is never sent, is served from the copy it is sent.
7+
*--------------------------------------------------------------------------------------------*/
8+
// @ts-check
9+
'use strict';
10+
11+
const assert = require('assert');
12+
const path = require('path');
13+
const { pathToFileURL } = require('url');
14+
15+
let n = 0;
16+
async function test(name, fn) { await fn(); n++; console.log(' ok - ' + name); }
17+
18+
(async () => {
19+
const { uploaded, parseRedirects, matchRedirect, resolvePage, toRedirectsFile, NODE_MODULES_COPY } =
20+
await import(pathToFileURL(path.join(__dirname, '..', '..', 'lib', 'pages.mjs')).href);
21+
22+
await test('what wrangler uploads: nothing in a folder named node_modules or .git, anywhere, and not its own files at the root', async () => {
23+
for (const rel of ['_/abc/static/node_modules/@xterm/xterm/lib/xterm.js', 'node_modules/x.js', '_/abc/extensions/x/node_modules/y.js',
24+
'_/abc/.git/HEAD', '_/abc/static/.DS_Store', '_headers', '_redirects', '_worker.js', '_routes.json', 'functions/api.js']) {
25+
assert.strictEqual(uploaded(rel), false, rel);
26+
}
27+
for (const rel of ['index.html', 'callback.html', '_/abc/static/node-modules/@xterm/xterm/lib/xterm.js', '_/abc/static/out/nls.messages.js',
28+
'_/abc/extensions/levelcode-ai/package.json', '_/abc/functions/x.js', '_/abc/static/node_modules_x/y.js']) {
29+
assert.strictEqual(uploaded(rel), true, rel);
30+
}
31+
});
32+
33+
await test('_redirects: rules in order, comments and blank lines skipped, 302 when no status is given', async () => {
34+
const rules = parseRedirects('# a comment\n\n/old /new 301\n/a/:x/* /b/:x/:splat 200\n/plain /there\n');
35+
assert.deepStrictEqual(rules, [{ from: '/old', to: '/new', status: 301 }, { from: '/a/:x/*', to: '/b/:x/:splat', status: 200 }, { from: '/plain', to: '/there', status: 302 }]);
36+
assert.deepStrictEqual(matchRedirect(rules, '/a/one/two/three.js'), { to: '/b/one/two/three.js', status: 200 });
37+
assert.strictEqual(matchRedirect(rules, '/a/one'), null, 'a splat needs the slash before it');
38+
assert.deepStrictEqual(matchRedirect(parseRedirects('/x /first 200\n/x /second 200\n'), '/x'), { to: '/first', status: 200 }, 'the first match wins');
39+
});
40+
41+
await test('the release\'s rule serves every node_modules file from the copy that was uploaded, in this build and in a kept one', async () => {
42+
const rules = parseRedirects(toRedirectsFile());
43+
for (const id of ['5bfb4df6caa1', 'ae34c91f6d37']) {
44+
for (const file of ['@xterm/xterm/lib/xterm.js', 'vscode-textmate/release/main.js', 'vscode-oniguruma/release/onig.wasm', '@vscode/codicons/dist/codicon.ttf']) {
45+
const asked = `/_/${id}/static/node_modules/${file}`;
46+
const r = matchRedirect(rules, asked);
47+
assert.ok(r && r.status === 200, asked + ' is rewritten, not redirected');
48+
assert.strictEqual(r.to, `/_/${id}/static/${NODE_MODULES_COPY}/${file}`);
49+
assert.ok(uploaded(r.to.slice(1)), r.to + ' is a path Pages is sent');
50+
}
51+
}
52+
assert.strictEqual(matchRedirect(rules, '/_/abc/static/out/vs/workbench/workbench.web.main.internal.js'), null, 'nothing else is rewritten');
53+
assert.strictEqual(matchRedirect(rules, '/_/abc/def/static/node_modules/x.js'), null, 'a build id is one segment');
54+
});
55+
56+
await test('.html addresses: redirected without it; the address without it serves the page', async () => {
57+
const files = new Set(['index.html', 'callback.html', 'callback.js', '_/abc/static/out/x/frame.html', '_/abc/static/out/x/index.html']);
58+
const isFile = (rel) => files.has(rel);
59+
assert.deepStrictEqual(resolvePage('/', isFile), { file: 'index.html' });
60+
assert.deepStrictEqual(resolvePage('/index.html', isFile), { redirect: '/' });
61+
assert.deepStrictEqual(resolvePage('/callback.html', isFile), { redirect: '/callback' });
62+
assert.deepStrictEqual(resolvePage('/callback', isFile), { file: 'callback.html' });
63+
assert.deepStrictEqual(resolvePage('/callback.js', isFile), { file: 'callback.js' });
64+
assert.deepStrictEqual(resolvePage('/_/abc/static/out/x/frame.html', isFile), { redirect: '/_/abc/static/out/x/frame' });
65+
assert.deepStrictEqual(resolvePage('/_/abc/static/out/x/frame', isFile), { file: '_/abc/static/out/x/frame.html' });
66+
assert.deepStrictEqual(resolvePage('/_/abc/static/out/x/index.html', isFile), { redirect: '/_/abc/static/out/x/' });
67+
assert.deepStrictEqual(resolvePage('/_/abc/static/out/x/', isFile), { file: '_/abc/static/out/x/index.html' });
68+
assert.strictEqual(resolvePage('/missing.js', isFile), null, 'no such file: the caller answers with the page');
69+
});
70+
71+
console.log(`\n${n} tests passed`);
72+
})().catch((e) => { console.error(e); process.exit(1); });

0 commit comments

Comments
 (0)