feat: support browser download mirrors

docs/dealing-with-browsers.md

@@ -2,30 +2,109 @@
 
 ## Dealing with Browsers
 
-All you need are browsers that Testplane could use for testing. To do this you need to install some browsers, such as [chrome](https://www.google.com/chrome/) (to automate this process you can use the [@testplane/headless-chrome](https://github.com/gemini-testing/testplane-headless-chrome) plugin).
+Testplane v9 runs browser sessions through WebDriver. You can connect to a remote WebDriver grid or let Testplane install and run supported browsers and drivers locally.
 
-Next, you have two ways to configure Testplane to work with browsers:
+### Local browsers and drivers
 
-* Using the devtools protocol (available only for `Chromium`-based browsers). This method does not need to be pre-configured. Just go to the [quick start](#quick-start).
-* Using the webdriver protocol. In this case you need to set up [Selenium](http://www.seleniumhq.org/) grid. The simplest way to get started is to use one of the NPM selenium standalone packages, such as [vvo/selenium-standalone](https://github.com/vvo/selenium-standalone). For more information about setting up, see [selenium-standalone](#selenium-standalone).
+Set `gridUrl` to `"local"` and describe the browsers in the usual `browsers` section:
 
-### Selenium-standalone
-Install `selenium-standalone` by command:
+```typescript
+export default {
+    gridUrl: "local",
 
-```
-npm i -g selenium-standalone
+    browsers: {
+        chrome: {
+            desiredCapabilities: {
+                browserName: "chrome",
+                browserVersion: "130",
+            },
+        },
+        firefox: {
+            desiredCapabilities: {
+                browserName: "firefox",
+            },
+        },
+    },
+} satisfies import("testplane").ConfigInput;
 ```
 
-Next you need to install browser drivers
+Install the configured browser binaries and drivers in advance:
 
+```bash
+npx testplane install-deps
 ```
-selenium-standalone install
+
+You can also request explicit versions:
+
+```bash
+npx testplane install-deps chrome@130 firefox@128
 ```
 
-and run your server by executing
+If `install-deps` is not run first, Testplane can download missing local dependencies when the browser session starts. For a remote grid, set `gridUrl` to its WebDriver endpoint instead of `"local"`.
+
+For local browser and driver installation, a `browserVersion` consisting only of digits is normalized by adding `.0`: `"139"` is treated as `"139.0"`. Versions containing dots or channel names are not rewritten by this normalization. The same rule applies to explicit `install-deps` versions such as `chrome@139`.
+
+### Browser download mirrors
+
+Configure mirrors once at the root of the Testplane config. The map is not a per-browser option:
 
+```typescript
+export default {
+    gridUrl: "local",
+
+    browserDownloadMirrors: {
+        chrome: "https://mirror.example/chrome-for-testing",
+        chromium: "https://mirror.example/chromium-browser-snapshots",
+        firefox: "https://mirror.example/firefox",
+    },
+
+    browsers: {
+        chrome: {
+            desiredCapabilities: {
+                browserName: "chrome",
+                browserVersion: "130",
+            },
+        },
+    },
+} satisfies import("testplane").ConfigInput;
 ```
-selenium-standalone start
+
+CI can supply or override individual mirrors with uppercase environment variables:
+
+```bash
+export TESTPLANE_BROWSER_DOWNLOAD_MIRRORS_CHROME=https://mirror.example/chrome-for-testing
+export TESTPLANE_BROWSER_DOWNLOAD_MIRRORS_CHROMIUM=https://mirror.example/chromium-browser-snapshots
+export TESTPLANE_BROWSER_DOWNLOAD_MIRRORS_FIREFOX=https://mirror.example/firefox
 ```
 
-:warning: If you will get error like `No Java runtime present, requesting install.` you should install [Java Development Kit (JDK)](https://www.oracle.com/technetwork/java/javase/downloads/index.html) for your OS.
+The uppercase variables take precedence over config values and compatibility variables with lowercase `testplane_` or `hermione_` prefixes. Leave an unused variable unset. An empty value is invalid.
+
+Mirror URLs must be absolute `http:` or `https:` URLs without credentials, a query string, or a fragment. Testplane trims surrounding whitespace and trailing slashes while preserving a pathname prefix.
+
+To see which mirror is used for each downloaded binary, enable `DEBUG=testplane:browser-installer` when running `testplane install-deps` or starting a local browser. The debug message includes the binary name, resolved version, and mirror URL. No mirror download message is emitted when an installed binary is reused.
+
+### Mirror layout
+
+A mirror must preserve the archive layout expected by `@puppeteer/browsers` for every target platform used in CI.
+
+- The Chrome mirror serves Chrome for Testing metadata at its root, including `LATEST_RELEASE_STABLE`, channel files such as `LATEST_RELEASE_BETA`, `latest-versions-per-milestone.json`, and `latest-patch-versions-per-build.json`.
+- Chrome, Chrome Headless Shell, and ChromeDriver archives use Chrome for Testing paths such as `<build-id>/<platform>/chrome-<platform>.zip`, `<build-id>/<platform>/chrome-headless-shell-<platform>.zip`, and `<build-id>/<platform>/chromedriver-<platform>.zip`.
+- The Chromium mirror serves snapshot archives under paths such as `<platform-folder>/<revision>/<archive>.zip`.
+- The Firefox mirror serves `firefox_versions.json` at its root. Release archives use paths such as `<version>/<platform>/en-US/<archive>`.
+
+For Chrome, an explicit `browserVersion: "latest"` follows `@puppeteer/browsers` and selects Canary (`LATEST_RELEASE_CANARY`), not the latest Stable release. Use `"stable"` (`LATEST_RELEASE_STABLE`) to check the current Stable release. If `browserVersion` is omitted, Testplane can reuse an already installed Chrome version without checking whether a newer Stable release exists; it queries Stable only when no suitable local browser is found.
+
+The actual platform directory and archive names vary between Linux, macOS, Windows, and architectures. Mirror the upstream paths rather than inventing a new layout.
+
+### Source exceptions and fallback behavior
+
+The mirror keys cover different download sources:
+
+- Chrome versions earlier than 113 are installed from Chromium snapshots and therefore use the `chromium` mirror.
+- ChromeDriver versions earlier than 115 continue to use the legacy `chromedriver.storage.googleapis.com` source.
+- GeckoDriver is not downloaded from the Firefox mirror. It continues to use its Mozilla/GitHub upstream source.
+
+When a mirror covers a requested metadata file or archive, Testplane does not fall back to the public upstream. A missing platform archive, unavailable metadata file, or network failure stops installation with an error.
+
+Download errors retain the underlying network diagnostics, including the mirror URL.
+If an artifact is unavailable during the pre-download check, the error includes the configured mirror URL.

src/browser-installer/chrome/browser.ts

@@ -14,11 +14,22 @@ import { normalizeChromeVersion } from "../utils";
 import { installUbuntuPackageDependencies } from "../ubuntu-packages";
 import { installChromeDriver } from "./driver";
 import { BrowserName } from "../../browser/types";
+import type { BrowserDownloadMirrors } from "../../config/types";
+import { getBrowserDownloadMirror } from "../mirrors";
+import { resolveChromeBuildIdFromMirror, type ChromeBuildIdResolver } from "./utils";
 
 const installChromeBrowser = async (
     browserName: typeof BrowserName.CHROME | typeof BrowserName.CHROMEHEADLESSSHELL,
     version: string,
-    { force = false } = {},
+    {
+        force = false,
+        browserDownloadMirrors,
+        resolveMirrorBuildId,
+    }: {
+        force?: boolean;
+        browserDownloadMirrors?: BrowserDownloadMirrors;
+        resolveMirrorBuildId?: ChromeBuildIdResolver;
+    } = {},
 ): Promise<string> => {
     const milestone = getMilestone(version);
 
@@ -27,7 +38,7 @@ const installChromeBrowser = async (
 
         const { installChromium } = await import("../chromium");
 
-        return installChromium(version, { force });
+        return installChromium(version, { force, browserDownloadMirrors });
     }
 
     const platform = getBrowserPlatform();
@@ -40,61 +51,102 @@ const installChromeBrowser = async (
     }
 
     const normalizedVersion = normalizeChromeVersion(version);
-    const buildId = await resolveBuildId(browserName, platform, normalizedVersion);
+    const mirror = getBrowserDownloadMirror(browserName, browserDownloadMirrors);
+    const buildId = mirror
+        ? await (resolveMirrorBuildId ? resolveMirrorBuildId() : resolveChromeBuildIdFromMirror(version, mirror))
+        : await resolveBuildId(browserName, platform, normalizedVersion);
 
     const cacheDir = getBrowsersDir();
-    const canBeInstalled = await canDownload({ browser: browserName, platform, buildId, cacheDir });
+    const canBeInstalled = await canDownload({ browser: browserName, platform, buildId, cacheDir, baseUrl: mirror });
 
     if (!canBeInstalled) {
         throw new Error(
-            [
-                `${browserName}@${version} can't be installed.`,
-                `Probably the version '${version}' is invalid, please try another version.`,
-                "Version examples: '120', '120.0'",
-            ].join("\n"),
+            mirror
+                ? `Couldn't download browser artifact from the configured mirror: ${mirror}`
+                : [
+                      `${browserName}@${version} can't be installed.`,
+                      `Probably the version '${version}' is invalid, please try another version.`,
+                      "Version examples: '120', '120.0'",
+                  ].join("\n"),
         );
     }
 
-    const installFn = (downloadProgressCallback: DownloadProgressCallback): Promise<string> =>
-        puppeteerInstall({
+    const installFn = (downloadProgressCallback: DownloadProgressCallback): Promise<string> => {
+        if (mirror) {
+            browserInstallerDebug(`downloading ${browserName}@${buildId} from mirror ${mirror}`);
+        }
+
+        return puppeteerInstall({
             platform,
             buildId,
             cacheDir,
             downloadProgressCallback,
             browser: browserName,
+            baseUrl: mirror,
             unpack: true,
         }).then(result => result.executablePath);
+    };
 
     return registry.installBinary(browserName, platform, buildId, installFn);
 };
 
 export const installChrome = async (
     browserName: typeof BrowserName.CHROME | typeof BrowserName.CHROMEHEADLESSSHELL,
     version: string,
-    { force = false, needWebDriver = false, needUbuntuPackages = false } = {},
+    {
+        force = false,
+        needWebDriver = false,
+        needUbuntuPackages = false,
+        browserDownloadMirrors,
+    }: {
+        force?: boolean;
+        needWebDriver?: boolean;
+        needUbuntuPackages?: boolean;
+        browserDownloadMirrors?: BrowserDownloadMirrors;
+    } = {},
 ): Promise<string> => {
+    const chromeMirror = getBrowserDownloadMirror(BrowserName.CHROME, browserDownloadMirrors);
+    const resolveMirrorBuildId = chromeMirror
+        ? _.once(() => resolveChromeBuildIdFromMirror(version, chromeMirror))
+        : undefined;
     const [browserPath] = await Promise.all([
-        installChromeBrowser(browserName, version, { force }),
-        needWebDriver && installChromeDriver(version, { force }),
+        installChromeBrowser(browserName, version, { force, browserDownloadMirrors, resolveMirrorBuildId }),
+        needWebDriver &&
+            installChromeDriver(version, {
+                force,
+                browserDownloadMirrors,
+                resolveMirrorBuildId,
+            }),
         needUbuntuPackages && installUbuntuPackageDependencies(),
     ]);
 
     return browserPath;
 };
 
-export const resolveLatestChromeVersion = _.memoize(async (force = false): Promise<string> => {
-    if (!force) {
-        const platform = getBrowserPlatform();
-        const existingLocallyBrowserVersion = registry.getMatchedBrowserVersion(BrowserName.CHROME, platform);
+export const resolveLatestChromeVersion = _.memoize(
+    async (force = false, browserDownloadMirrors?: BrowserDownloadMirrors): Promise<string> => {
+        if (!force) {
+            const platform = getBrowserPlatform();
+            const existingLocallyBrowserVersion = registry.getMatchedBrowserVersion(BrowserName.CHROME, platform);
 
-        if (existingLocallyBrowserVersion) {
-            return existingLocallyBrowserVersion;
+            if (existingLocallyBrowserVersion) {
+                return existingLocallyBrowserVersion;
+            }
+        }
+
+        const mirror = getBrowserDownloadMirror(BrowserName.CHROME, browserDownloadMirrors);
+
+        if (mirror) {
+            return resolveChromeBuildIdFromMirror("stable", mirror);
         }
-    }
 
-    return retryFetch(CHROME_FOR_TESTING_LATEST_STABLE_API_URL)
-        .then(res => res.text())
-        .catch(() => {
-            throw new Error("Couldn't resolve latest chrome version");
-        });
-});
+        return retryFetch(CHROME_FOR_TESTING_LATEST_STABLE_API_URL)
+            .then(res => res.text())
+            .then(version => version.trim())
+            .catch(() => {
+                throw new Error("Couldn't resolve latest chrome version");
+            });
+    },
+    (force, browserDownloadMirrors) =>
+        `${force}:${getBrowserDownloadMirror(BrowserName.CHROME, browserDownloadMirrors) ?? "default"}`,
+);

src/browser-installer/chrome/driver.ts

@@ -9,8 +9,23 @@ import {
     type DownloadProgressCallback,
 } from "../utils";
 import registry from "../registry";
+import { BrowserName } from "../../browser/types";
+import type { BrowserDownloadMirrors } from "../../config/types";
+import { getBrowserDownloadMirror } from "../mirrors";
+import { resolveChromeBuildIdFromMirror, type ChromeBuildIdResolver } from "./utils";
 
-export const installChromeDriver = async (chromeVersion: string, { force = false } = {}): Promise<string> => {
+export const installChromeDriver = async (
+    chromeVersion: string,
+    {
+        force = false,
+        browserDownloadMirrors,
+        resolveMirrorBuildId,
+    }: {
+        force?: boolean;
+        browserDownloadMirrors?: BrowserDownloadMirrors;
+        resolveMirrorBuildId?: ChromeBuildIdResolver;
+    } = {},
+): Promise<string> => {
     const platform = getBrowserPlatform();
     const existingLocallyDriverVersion = registry.getMatchedDriverVersion(
         DriverName.CHROMEDRIVER,
@@ -38,30 +53,47 @@ export const installChromeDriver = async (chromeVersion: string, { force = false
         return installChromeDriverManually(milestone);
     }
 
-    const buildId = await resolveBuildId(DriverName.CHROMEDRIVER, platform, milestone);
+    const mirror = getBrowserDownloadMirror(BrowserName.CHROME, browserDownloadMirrors);
+    const buildId = mirror
+        ? await (resolveMirrorBuildId ? resolveMirrorBuildId() : resolveChromeBuildIdFromMirror(chromeVersion, mirror))
+        : await resolveBuildId(DriverName.CHROMEDRIVER, platform, milestone);
 
     const cacheDir = getChromeDriverDir();
-    const canBeInstalled = await canDownload({ browser: DriverName.CHROMEDRIVER, platform, buildId, cacheDir });
+    const canBeInstalled = await canDownload({
+        browser: DriverName.CHROMEDRIVER,
+        platform,
+        buildId,
+        cacheDir,
+        baseUrl: mirror,
+    });
 
     if (!canBeInstalled) {
         throw new Error(
-            [
-                `chromedriver@${buildId} can't be installed.`,
-                `Probably the major browser version '${milestone}' is invalid`,
-                "Correct chrome version examples: '123', '124'",
-            ].join("\n"),
+            mirror
+                ? `Couldn't download browser artifact from the configured mirror: ${mirror}`
+                : [
+                      `chromedriver@${buildId} can't be installed.`,
+                      `Probably the major browser version '${milestone}' is invalid`,
+                      "Correct chrome version examples: '123', '124'",
+                  ].join("\n"),
         );
     }
 
-    const installFn = (downloadProgressCallback: DownloadProgressCallback): Promise<string> =>
-        puppeteerInstall({
+    const installFn = (downloadProgressCallback: DownloadProgressCallback): Promise<string> => {
+        if (mirror) {
+            browserInstallerDebug(`downloading ${DriverName.CHROMEDRIVER}@${buildId} from mirror ${mirror}`);
+        }
+
+        return puppeteerInstall({
             platform,
             buildId,
             cacheDir: getChromeDriverDir(),
             browser: DriverName.CHROMEDRIVER,
+            baseUrl: mirror,
             unpack: true,
             downloadProgressCallback,
         }).then(result => result.executablePath);
+    };
 
     return registry.installBinary(DriverName.CHROMEDRIVER, platform, buildId, installFn);
 };

src/browser-installer/chrome/index.ts

@@ -8,15 +8,19 @@ import { installChrome, resolveLatestChromeVersion } from "./browser";
 import { installChromeDriver } from "./driver";
 import { isUbuntu, getUbuntuLinkerEnv } from "../ubuntu-packages";
 import RuntimeConfig from "../../config/runtime-config";
+import type { BrowserDownloadMirrors } from "../../config/types";
 
 export { installChrome, resolveLatestChromeVersion, installChromeDriver };
 
 export const runChromeDriver = async (
     chromeVersion: string,
-    { debug = false } = {},
+    {
+        debug = false,
+        browserDownloadMirrors,
+    }: { debug?: boolean; browserDownloadMirrors?: BrowserDownloadMirrors } = {},
 ): Promise<{ gridUrl: string; process: ChildProcess; port: number; kill: () => void }> => {
     const [chromeDriverPath, randomPort, chromeDriverEnv] = await Promise.all([
-        installChromeDriver(chromeVersion),
+        installChromeDriver(chromeVersion, { browserDownloadMirrors }),
         getPort(),
         isUbuntu()
             .then(isUbuntu => (isUbuntu ? getUbuntuLinkerEnv() : null))