fossick

fossick gets text off the web and into a program. You give it a search query, a URL, an arXiv id, a YouTube link, a GitHub repository or a local file. It gives back markdown that a person or a language model can read. It was written for AI agents that look things up. Its functions return plain data (strings, lists and dictionaries) that you can print or pass to a model as they are.

The library works in layers. search sends a query to several search engines at once and merges their answers into one ranked list. read takes a single target and returns its text. Under read, fetch downloads a page and to_md turns its HTML into markdown. Many sites refuse plain downloads from scripts. fetch can also load a page in a headless browser, in a stealth browser built to pass bot checks, or in a copy of Chrome you have logged in to. For pages that need clicking and typing, cdp_connect drives that Chrome directly. Everything here is also available from the shell as the fossick command and as a Model Context Protocol (MCP) server that agents can call.

Install

uv add fossick

Searching and plain downloads need nothing else. Rendering JavaScript, the stealth fetcher and google run a headless browser through scrapling. If those calls raise BrowserUnavailable, run scrapling install to download its browsers. The browser automation in fossick.cdp drives your installed Google Chrome.

Quick start

Three calls cover most uses. search returns a list of results, each a dictionary with the page’s title, its URL in href and a snippet in body. score is the merged ranking score, and engines names the search engines that returned that page. method='flashrank' reorders the results with a small relevance model, which needs pip install flashrank; without a method the merged order is kept.

results = search('koshas github fts semantic code graph', method='flashrank', n=5)
for r in results: print(r['score'], r['engines'], r['title'], r['href'])

0.030536 ['brave', 'yahoo'] github.com › codegraph-ai › CodeGraphGitHub - codegraph-ai/CodeGraph: CodeGraph builds a semantic... https://github.com/codegraph-ai/CodeGraph
0.016129 ['yahoo'] vedicreader.github.io › kosha › graphgraph – koshas - vedicreader.github.io https://vedicreader.github.io/kosha/graph.html
0.015873 ['brave'] fts · GitHub Topics · GitHub https://github.com/topics/fts?o=desc&s=updated
0.016393 ['yahoo'] pypi.org › project › koshaskoshas · PyPI https://pypi.org/project/koshas/
0.015625 ['brave'] GitHub - colbymchenry/codegraph: Pre-indexed code knowledge graph, auto syncs on code changes, for Claude Code, Codex, Gemini, Cursor, OpenCode, AntiGravity, Kiro, and Hermes Agent — fewer tokens, fewer tool calls, 100% local https://github.com/colbymchenry/codegraph

res = research('sqlite WAL mode vs journal mode', n=5)
print(res['digest'][:500])
print([s['href'] for s in res['sources']])

## Write-Ahead Logging
https://www.sqlite.org/wal.html

Write-Ahead Logging

Table Of Contents

# 1\. Overview

The default method by which SQLite implements atomic commit and rollback is a rollback journal. Beginning with version 3.7.0 (2010-07-21), a new "Write-Ahead Log" option (hereafter referred to as "WAL") is available.

There are advantages and disadvantages to using WAL instead of a rollback journal. Advantages include:

1. WAL is significantly faster in most scenarios. 
  2. WAL provid
['https://www.sqlite.org/wal.html', 'https://blog.sqlite.ai/journal-modes-in-sqlite', 'https://mohit-bhalla.medium.com/understanding-wal-mode-in-sqlite-boosting-performance-in-sql-crud-operations-for-ios-5a8bd8be93d2', 'https://til.simonwillison.net/sqlite/enabling-wal-mode', 'https://fly.io/blog/wal-mode-in-litefs/']

page = read('https://en.wikipedia.org/wiki/Web_scraping')
print(page.text[:400])

The legality of web scraping varies across the world. In general, web scraping may be against the terms of service of some websites, but the enforceability of these terms is unclear.[11]

In the United States, website owners can use three major legal claims to prevent undesired web scraping: (1) copyright infringement (compilation), (2) violation of the Computer Fraud and Abuse Act ("CFAA"), and (

One function for every source

Use read when you have a target and want its text. read first calls what_is to classify the target. The result names the reader: dir for a local folder, file for a local file, arxiv for an arXiv URL or id, youtube for a YouTube link, github for a repository, ghfile for a file on GitHub, pdf for a URL ending in .pdf or /pdf, and web for any other http or https URL. A path that exists on disk wins over every URL rule.

read returns the same seven fields for every supported target:

field	holds
ok	True when the reader produced content
kind	the reader used, one of the names above
title	the title, or the first heading of the text
source	the URL, or the absolute path of a local file
text	the content as markdown
skipped	why nothing was read, when ok is False
meta	details from that reader, such as the authors of a paper or the HTTP status of a page, and the time it was read

Here read opens this repository’s own README:

target = str(repo_root()/'README.md')
assert what_is(target) == 'file'
result = read(target)
assert sorted(result) == ['kind', 'meta', 'ok', 'skipped', 'source', 'text', 'title']
assert (result.ok, result.kind, result.title) == (True, 'file', 'fossick')
assert '# fossick' in result.text and result.skipped is None
result.kind, result.title, result.ok

A folder or a GitHub repository is a tree of files, not one document. read does not turn it into text. It returns ok=True with an empty text and the local path in meta['path']. The caller chooses which files to open. For a GitHub repository, that path is a fresh clone in fossick’s cache. With pages=True, a PDF returns a list of (page_index, text) pairs. Page indexes start at 0.

Sometimes a target is recognised but yields nothing: a site serves a bot check instead of the page, a YouTube video has no English captions, or a URL that looked like a PDF returns something else. read reports these with ok=False and a reason in skipped, and does not raise. It raises ValueError only when the target is none of the kinds above.

Modules

notebook	what it covers
00_core	downloading pages (fetch, crawl), HTML to markdown (to_md), the readers for arXiv, GitHub, YouTube and PDFs, finding the JSON API behind a page, and read
01_cdp	driving Chrome through the DevTools protocol: page snapshots for agents, filling forms, running a list of steps, capturing network requests
02_search	web search across several engines, merging and reranking results, google, and research
03_cli	the fossick shell command
04_mcp	the MCP server that offers fossick’s functions to coding agents
05_shop	adding products to a shopping cart and filling checkout forms on a live store
06_quality	choosing which search results are worth reading: removing copycat sites, preferring authoritative ones, and splitting a long question into several searches

Common calls

search('query', n=10, region='auto', timelimit='y')   # results from the past year
research('query', n=5)                    # search, read the top pages, return a digest that cites them
fetch(url, auto=True)                     # try stronger download methods until one is not blocked
crawl(url, max_pages=5, same_domain=True) # follow links from url, staying on its site
google('query', n=10)                     # Google's own ranking, through the stealth browser
cdp = await cdp_connect(); pg = await cdp.open_page(url); print(await pg.snapshot())

From the shell: fossick search "q", fossick fetch URL --auto, fossick research "q". For agents: uvx fossick-mcp, or fossick-mcp --http to serve over HTTP instead of standard input and output.

A logged-in Chrome

Some pages are only shown to a logged-in user. fossick reaches them through a separate copy of Chrome that runs with remote debugging switched on, on port 9223, with its own profile in ~/.cache/fastcdp/cdp-chrome. cdp_connect starts it the first time it is needed. Log in to a site once in that window. The cookies stay in the profile. Later calls such as fetch(url, session=True) then use the logged-in session. fetch(url, auto=True) and read fall back to this Chrome when every other method is blocked.

To start it by hand in a visible window, for logging in, run await cdp_setup(headless=False). The fastcdp-setup command, installed with fossick’s fastcdp dependency, adds a launcher for the same Chrome and profile to your applications.

Untrusted URLs

fetch(url, auto=True) and read, which uses auto=True by default, can end in the logged-in Chrome on port 9223. That step starts the Chrome if it is not running and sends the cookies it has for the site. Use it only for URLs that you chose. When a URL comes from a web page, a document or another person, pass auto=False:

fetch(url, auto=False, max_bytes=20_000_000)   # plain HTTP only; stop after 20 MB
read(url, auto=False, max_bytes=20_000_000)    # also for PDFs, arXiv papers and GitHub files

With auto=False, fossick does not start a browser or connect to port 9223, also for PDFs and the redirects that a page sends. The plain path refuses private and local addresses. It connects only to the address that it checked, so a second DNS answer cannot send it to 127.0.0.1. It checks each redirect in the same way. max_bytes and max_recv_speed limit the size and speed of a download. Both are off by default. Through a proxy, and in the browser paths, fossick checks only the first URL.

On a machine that inspects TLS traffic with its own root certificate, curl_cffi does not trust that root, because it does not use the system keychain. A download then fails with curl: (60) ... unable to get local issuer certificate. Set SSL_CERT_FILE to a CA bundle file that holds the normal roots and the inspection root, for example export SSL_CERT_FILE=~/Library/Java/macos-all-ca.pem. fossick also reads CURL_CA_BUNDLE and REQUESTS_CA_BUNDLE, and takes ca_bundle= on each call. Verification stays on.