Skip to content

README.md

Latest commit

 

History

History
341 lines (264 loc) · 13.9 KB

README.md

File metadata and controls

341 lines (264 loc) · 13.9 KB
Raw

CF Clearance Scraper (Python)

Python port of ZFC-Digital/cf-clearance-scraper (v2.1.3) — same HTTP API, same four modes, same env vars. The browser layer is replaced: puppeteer-real-browser (Chromium) becomes invisible_playwright, an anti-detect headless Firefox with a seeded fingerprint. Everything else — route, request/response shapes, status codes, limits — is identical.

Created for testing and training purposes: retrieve the page source of websites, create Cloudflare Turnstile tokens and create Cloudflare WAF sessions.

Cloudflare protection not only checks cookies in the request. It also checks variables in the header. For this reason, it is recommended to use it with the sample code below.

Cookies with cf in the name belong to Cloudflare. You can find out what these cookies do and how long they are valid by Clicking Here.

Installation

Requires Python 3.11+.

git clone https://github.com/Duc-python/cf-scraper.git
cd cf-scraper
python -m venv .venv
.\.venv\Scripts\python.exe -m pip install -r requirements.txt
.\.venv\Scripts\python.exe -m invisible_playwright fetch   # one-time engine download
# linux / mac
python3 -m venv .venv
.venv/bin/python -m pip install -r requirements.txt
.venv/bin/python -m invisible_playwright fetch

Run:

python -m cf_scraper            # PORT=3000 by default

Create Cloudflare WAF Session

By creating a session as in the example, you can send multiple requests to the same site without being blocked. Since sites may have TLS protection, it is recommended to send requests with a TLS-fingerprinting client, reusing the returned headers and cookies.

import httpx

session = httpx.post("http://localhost:3000/cf-clearance-scraper", json={
    "url": "https://nopecha.com/demo/cloudflare",
    "mode": "waf-session",
    # "proxy": {"host": "127.0.0.1", "port": 3000,
    #           "username": "username", "password": "password"},
}).json()

assert session["code"] == 200
cookies = "; ".join(f"{c['name']}={c['value']}" for c in session["cookies"])
headers = {**session["headers"], "cookie": cookies}
print(headers["user-agent"])

Create Turnstile Token with Little Resource Consumption

This endpoint serves a minimal Turnstile render page instead of the full site, so no extra css/js is loaded. The siteKey must be sent along with the site. If this does not work, use turnstile-max (full page load) below.

import httpx

token = httpx.post("http://localhost:3000/cf-clearance-scraper", json={
    "url": "https://turnstile.zeroclover.io/",
    "siteKey": "0x4AAAAAAAEwzhD6pyKkgXC0",
    "mode": "turnstile-min",
}).json()

print(token)  # {"token": "...", "code": 200}

Creating Turnstile Token with Full Page Load

This request goes to the page at the given url address with a real browser, resolves the Turnstile and returns the token.

import httpx

token = httpx.post("http://localhost:3000/cf-clearance-scraper", json={
    "url": "https://turnstile.zeroclover.io/",
    "mode": "turnstile-max",
}).json()

print(token)  # {"token": "...", "code": 200}

Getting Page Source from a Site Protected with Cloudflare WAF

import httpx

page = httpx.post("http://localhost:3000/cf-clearance-scraper", json={
    "url": "https://nopecha.com/demo/cloudflare",
    "mode": "source",
}).json()

print(page["source"][:200])

Configuration

Variable Default Meaning
PORT 3000 HTTP port
browserLimit 20 max concurrent browser contexts (else HTTP 429)
timeOut 150000 per-request budget in ms (else "Timeout Error")
authToken — when set, requests need a matching authToken (else 401)
headless true hidden virtual display, headed rendering pipeline
SEED — fixed fingerprint seed (reproducible runs)
QUIET_CANVAS true disable canvas pixel substitution (matches real Firefox)
timezone — pin an IANA zone, e.g. Asia/Kolkata (see note)
locale — pin a BCP-47 tag, e.g. en-IN
SKIP_LAUNCH false true = serve the API without launching the browser

Timezone and proxy

Set per request with proxy. The browser derives its timezone from the egress IP, but that GeoIP lookup is only as good as its database: measured here, a proxy in India (<redacted-proxy-ip>) came back as Asia/Hong_Kong. A mismatched timezone next to a foreign IP is a plain tell, so pin it:

$env:timezone = "Asia/Kolkata"
$env:locale = "en-IN"

API

POST /cf-clearance-scraper with JSON body:

{
  "mode": "source | turnstile-min | turnstile-max | waf-session",
  "url": "https://...",
  "proxy": {"host": "...", "port": 8080, "username": "...", "password": "..."},
  "siteKey": "0x4AAAA...",
  "authToken": "..."
}

Responses mirror the original: {source, code: 200}, {token, code: 200}, {cookies, headers, code: 200}, errors as {code, message} with HTTP 400 / 401 / 404 / 429 / 500.

Tests

pytest tests/ -v                      # unit + engine pool, no browser needed
CF_LIVE=1 pytest tests/test_live.py -v   # launches the engine, uses network
CF_WIDGET=1 pytest tests/test_widget.py -v  # localhost Turnstile pages, no third party

tests/test_pool.py covers the engine pool's races and lifecycle with a fake launcher, so it runs in under two seconds and needs no browser. It guards three defects found while running proxies in parallel:

  • Engine.__aenter__ held the lock while calling start(), which takes the same lock - asyncio.Lock is not reentrant, so any context entering an engine deadlocked permanently.
  • _engine_for replaced a pooled engine that had not been launched yet, so concurrent callers on the same key each got their own browser to start.
  • close() tore engines down one after another, so the total grew with the pool size.

Turnstile token capture

turnstile-max reads the hidden cf-turnstile-response input that Cloudflare creates next to every widget it renders. Two approaches were tried first and are documented here because both are wrong in ways that look like success:

  • Polling [name="cf-response"] - only exists if the page's own callback creates one. A real page usually does not; its callback POSTs the token somewhere instead. On a real sitekey this failed 7 times out of 8.
  • Wrapping window.turnstile.render via a property setter to intercept the page's callback. It works in principle but breaks the widget: with the wrapper installed, nopecha.com/demo/turnstile renders nothing at all, because Cloudflare's api.js does not survive window.turnstile being an accessor before it loads.
  • window.turnstile.getResponse() with no widget argument returns the first widget's token. On a page carrying both a real sitekey and one of Cloudflare's test keys, that hands back XXXX.DUMMY.TOKEN.XXXX.

Cloudflare's documented test sitekeys (1x..., 3x...) return that placeholder instead of a token. Any captured placeholder is now rejected with an explanatory message rather than reported as success - a caller cannot tell a placeholder from a real token by length alone, and reporting one as a success is worse than reporting nothing.

Verified against a live page using a production-style sitekey (0x4..., read from the page rather than assumed), headless and headed, direct and through a proxy: 3/3 real tokens of 645-709 characters, against a placeholder length of 21. Tokens are read from the DOM and never submitted.

Verified behaviour

  • concurrency (browserLimit=1, 3 parallel): exactly 1x200 + 2x429
  • plain pages: no solve delay (race between the 200/302 gate and the solver)
  • timeOut overflow: 500 "Timeout Error"
  • dead proxy: fast 500 "Navigation failed: Problem loading page", server stays up
  • proxy chain: local CONNECT proxy logs the browser's CONNECT, job returns 200
  • source + waf-session live vs nopecha.com/demo/cloudflare, real cf_clearance
  • turnstile min/max vs localhost with Cloudflare test sitekeys (always-pass and forced-challenge, the latter proving the checkbox click path)

Wall clock, measured end to end through the API:

case time
turnstile, always-pass 4.5s
turnstile, forced challenge 5.0s
source, no proxy 8.4s
source, with proxy 17.1s
waf-session against a live challenge 10.0s
waf-session, with proxy 16.1s

The solve budget is deadline-aware rather than fixed: a fixed solve plus a fixed gate wait plus the settle reads can run well past what is left, which is how a job overran its budget. cf_scraper/jobs/_common.py carries the job deadline in a ContextVar and the solver sizes itself against what remains, reserving room for the steps after it. The solver returns the instant cf_clearance lands, so the budget only bounds the failing case - a healthy request costs the same regardless of what you set timeOut to. The default is 150s because engine launch (~5s) and geo resolution sit outside the job budget but still on the wall clock, and a slow proxy leaves little room for the solve.

Tune it per proxy if needed:

$env:timeOut = "90000"   # proxy chậm: vẫn dư thời gian cho challenge
$env:timeOut = "600000"  # proxy rất chậm hoặc nhiều challenge nối tiếp

test_live.py mirrors the original repo's tests byte-for-byte (same demo URLs). Note: turnstile.zeroclover.io returned Cloudflare 502 Host Error when this port was verified (origin server dead, not a bot block) — the min/max flows were instead verified locally against 127.0.0.1 pages using Cloudflare's documented test sitekeys (1x00000000000000000000AA always-pass, 3x00000000000000000000FF force-challenge, the latter proving the checkbox click path end to end). source and waf-session pass live against nopecha.com/demo/cloudflare, including a real cf_clearance cookie.

Quick Questions and Answers

Does It Open A New Browser On Every Request?

No, a new context is started with each request and closed when the job is finished. Jobs run in isolated contexts through a single browser — same as the original.

How Do I Limit the Browser Context to Open?

Set the browserLimit variable. The default is 20.

How Do I Add Authentication to Api?

Set the authToken variable. Requests whose body authToken does not match get HTTP 401.

How Do I Set The Timeout Time?

Set the timeOut variable in milliseconds. The default is 60000.

What changed vs the Node original

  • Browser: puppeteer-real-browser → invisible_playwright (async API), one context per request.
  • Cloudflare checkbox solving is explicit (cf_scraper/jobs/_solve.py): the Node original delegates it to puppeteer-real-browser's turnstile: true, invisible_playwright clicks nothing on its own.
  • Proxy: same {host, port, username, password} shape. The engine is launched with the proxy rather than given one per context - see "Fingerprint coherence" below.
  • Canvas: pixel substitution is off by default (QUIET_CANVAS=true) so readback matches a real Firefox; set QUIET_CANVAS=false for stock behaviour.
  • src/data/fakePage.html is reused byte-for-byte for turnstile-min.

Fingerprint coherence

A detector's cheapest signal is a pair of values that disagree, so this port fixes three measured contradictions rather than any single value.

Proxy is applied at launch, not per context. The engine decides its timezone, locale and WebRTC candidate when it starts. Measured on one proxy (exit <redacted-proxy-ip>):

proxy applied timezone WebRTC candidate
per context Asia/Bangkok <redacted-host-ip> — the host's real IP
at launch Asia/Hong_Kong <redacted-proxy-ip> — the proxy

Both problems are the same problem: with the proxy set afterwards, the fingerprint still describes the host. A leaked WebRTC candidate and a timezone that disagrees with the exit IP are each one RTCPeerConnection away from being read. Headless and headed behaved identically here, so this is about where the proxy is applied, not about rendering mode.

cf_scraper/browser_manager.py therefore keeps one engine per proxy, with timezone and locale derived from that proxy's exit IP (cf_scraper/geo.py). Engines are refcounted and dropped when idle; the direct-connection engine is kept because ready reports on it.

Canvas reads back like a real browser. With the stock engine setting, a simple canvas read back 9145 distinct colours where real Firefox gives 32, because every pixel is replaced with hash(seed, idx). That is a loud outlier next to any real device. QUIET_CANVAS=true (the default here) turns it off, measured at 27 colours on the same content.

Accept-Language follows the proxy's country. waf-session reads the header the browser actually sends, so it arrives as zh-TW,zh;q=0.9,en-US;q=0.8,en;q=0.7 for a Hong Kong exit rather than a fixed en-US,en.

Set AUTO_GEO=false to keep one engine per zone without the geo lookup, or pin timezone / locale explicitly to override the derivation.

Disclaimer of Liability

This repository was created purely for testing and training purposes. The user is responsible for any prohibited liability that may arise from its use. The library is not intended to harm any site or company. The user is responsible for any damage that may arise. Users of this repository are deemed to have accepted this disclaimer. Only use it on sites you own or are permitted to test.