Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
123 changes: 102 additions & 21 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,12 +2,10 @@

[![NPM version](<https://img.shields.io/npm/v/context.dev.svg?label=npm%20(stable)>)](https://npmjs.org/package/context.dev) ![npm bundle size](https://img.shields.io/bundlephobia/minzip/context.dev)

This library provides convenient access to the Context Dev REST API from server-side TypeScript or JavaScript.
Context.dev is a web scraping API for AI agents and LLMs. This SDK turns any URL into clean, LLM-ready markdown, crawls whole sites, searches the web, takes screenshots and extracts structured JSON against a schema you define, all with one API key. Proxies, JavaScript rendering and anti-bot handling run on Context.dev's side, so there is no headless browser to host.

The REST API documentation can be found on [docs.context.dev](https://docs.context.dev/). The full API of this library can be found in [api.md](api.md).

It is generated with [Stainless](https://www.stainless.com/).

## Installation

```sh
Expand All @@ -16,21 +14,103 @@ npm install context.dev

## Usage

Set `CONTEXT_DEV_API_KEY` to your API key; the client reads it automatically.

### Scrape markdown and HTML

The full API of this library can be found in [api.md](api.md).

<!-- prettier-ignore -->
```js
```ts
import ContextDev from 'context.dev';

const client = new ContextDev({
apiKey: process.env['CONTEXT_DEV_API_KEY'], // This is the default and can be omitted
const client = new ContextDev();

const page = await client.web.scrape({
url: 'https://example.com',
formats: { markdown: true, html: true },
});

console.log(page.markdown.data);
console.log(page.html.data);
```

### Extract structured JSON

`jsonParams.schema` accepts JSON Schema; install Zod 4 (`npm install zod@^4`) to build a schema with Zod and convert it with `z.toJSONSchema`.

```ts
import ContextDev from 'context.dev';
import { z } from 'zod';

const client = new ContextDev();
const pageSchema = z.object({
title: z.string().nullable(),
description: z.string().nullable(),
});

const page = await client.web.scrape({
url: 'https://example.com',
formats: { json: true },
jsonParams: { schema: z.toJSONSchema(pageSchema) },
});

console.log(page.json.data);
```

### Extract relevant highlights

Return the passages that answer a question about the page.

```ts
import ContextDev from 'context.dev';

const client = new ContextDev();

const page = await client.web.scrape({
url: 'https://example.com',
formats: { highlights: true },
highlightsParams: { query: 'What is this domain used for?' },
});

const brand = await client.brand.retrieve({ domain: 'REPLACE_ME', type: 'by_domain' });
console.log(page.highlights.data);
```

### Take a screenshot

The screenshot is returned as a base64 image data URL.

console.log(brand.request_id);
```ts
import ContextDev from 'context.dev';

const client = new ContextDev();

const page = await client.web.scrape({
url: 'https://example.com',
formats: { screenshot: true },
});

console.log(page.screenshot.data);
```

## What you can do

| Task | Method |
| ---------------------------------------------------------------- | ------------------------ |
| Scrape a URL to markdown, HTML, JSON, highlights or a screenshot | `client.web.scrape` |
| Crawl a site and get every page as markdown | `client.web.webCrawlMd` |
| Map every URL on a domain | `client.web.mapUrls` |
| Search the web | `client.web.search` |
| Take a screenshot of a page | `client.web.screenshot` |
| Parse PDFs and documents | `client.parse.handle` |
| Run thousands of URLs as a batch | `client.batch.submit` |
| Watch a page for changes | `client.monitors.create` |
| Look up a company's logo, colors and brand data | `client.brand.retrieve` |

## Use it from an AI agent

Context.dev also ships as a plugin for [Claude](https://github.com/context-dot-dev/claude-plugin), [Cursor](https://github.com/context-dot-dev/cursor-plugin) and [Gemini CLI](https://github.com/context-dot-dev/gemini-cli-context), and as tools for [LangChain](https://github.com/context-dot-dev/langchain-context) and [Haystack](https://github.com/context-dot-dev/context-haystack).

### Request & Response types

This library includes TypeScript definitions for all request params and response fields. You may import and use them like so:
Expand All @@ -43,8 +123,8 @@ const client = new ContextDev({
apiKey: process.env['CONTEXT_DEV_API_KEY'], // This is the default and can be omitted
});

const params: ContextDev.BrandRetrieveParams = { domain: 'REPLACE_ME', type: 'by_domain' };
const brand: ContextDev.BrandRetrieveResponse = await client.brand.retrieve(params);
const params: ContextDev.WebScrapeParams = { url: 'https://example.com', formats: { markdown: true } };
const page: ContextDev.WebScrapeResponse = await client.web.scrape(params);
```

Documentation for each method, request param, and response field are available in docstrings and will appear on hover in most modern editors.
Expand All @@ -57,8 +137,8 @@ a subclass of `APIError` will be thrown:

<!-- prettier-ignore -->
```ts
const brand = await client.brand
.retrieve({ domain: 'REPLACE_ME', type: 'by_domain' })
const page = await client.web
.scrape({ url: 'https://example.com', formats: { markdown: true } })
.catch(async (err) => {
if (err instanceof ContextDev.APIError) {
console.log(err.status); // 400
Expand Down Expand Up @@ -99,7 +179,7 @@ const client = new ContextDev({
});

// Or, configure per-request:
await client.brand.retrieve({ domain: 'REPLACE_ME', type: 'by_domain' }, {
await client.web.scrape({ url: 'https://example.com', formats: { markdown: true } }, {
maxRetries: 5,
});
```
Expand All @@ -116,7 +196,7 @@ const client = new ContextDev({
});

// Override per-request:
await client.brand.retrieve({ domain: 'REPLACE_ME', type: 'by_domain' }, {
await client.web.scrape({ url: 'https://example.com', formats: { markdown: true } }, {
timeout: 5 * 1000,
});
```
Expand All @@ -139,17 +219,17 @@ Unlike `.asResponse()` this method consumes the body, returning once it is parse
```ts
const client = new ContextDev();

const response = await client.brand
.retrieve({ domain: 'REPLACE_ME', type: 'by_domain' })
const response = await client.web
.scrape({ url: 'https://example.com', formats: { markdown: true } })
.asResponse();
console.log(response.headers.get('X-My-Header'));
console.log(response.statusText); // access the underlying Response object

const { data: brand, response: raw } = await client.brand
.retrieve({ domain: 'REPLACE_ME', type: 'by_domain' })
const { data: page, response: raw } = await client.web
.scrape({ url: 'https://example.com', formats: { markdown: true } })
.withResponse();
console.log(raw.headers.get('X-My-Header'));
console.log(brand.request_id);
console.log(page.markdown.data);
```

### Logging
Expand Down Expand Up @@ -229,8 +309,9 @@ parameter. This library doesn't validate at runtime that the request matches the
send will be sent as-is.

```ts
client.brand.retrieve({
// ...
client.web.scrape({
url: 'https://example.com',
formats: { markdown: true },
// @ts-expect-error baz is not yet public
baz: 'undocumented option',
});
Expand Down
Loading