Skip to content

README: lead with what the SDK does #48

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
YahiaBakour merged 3 commits into from
Oct 8, 2026
Merged

README: lead with what the SDK does #48

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
9fbd853
docs: lead README with Context.dev capabilities
YahiaBakour Oct 8, 2026
2b5f5d8
docs: remove Stainless README attribution
YahiaBakour Oct 8, 2026
2907ebc
docs: demonstrate scraping formats throughout README
YahiaBakour Oct 8, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
186 changes: 152 additions & 34 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,10 +6,9 @@

<!-- x-release-please-end -->

The Context Dev Go library provides convenient access to the [Context Dev REST API](https://docs.context.dev/)
from applications written in Go.
Context.dev is a web scraping API for AI agents and LLMs. This SDK turns any URL into clean, LLM-ready markdown, crawls whole sites, searches the web, takes screenshots and extracts structured JSON against a schema you define, all with one API key. Proxies, JavaScript rendering and anti-bot handling run on Context.dev's side, so there is no headless browser to host.

It is generated with [Stainless](https://www.stainless.com/).
The REST API documentation can be found on [docs.context.dev](https://docs.context.dev/).

## Installation

Expand Down Expand Up @@ -39,6 +38,10 @@ This library requires Go 1.22+.

## Usage

Set `CONTEXT_DEV_API_KEY` to your API key; the client reads it automatically.

### Scrape markdown and HTML

The full API of this library can be found in [api.md](api.md).

```go
Expand All @@ -49,26 +52,142 @@ import (
"fmt"

"github.com/context-dot-dev/context-go-sdk/v2"
"github.com/context-dot-dev/context-go-sdk/v2/option"
)

func main() {
client := contextdev.NewClient(
option.WithAPIKey("My API Key"), // defaults to os.LookupEnv("CONTEXT_DEV_API_KEY")
)
brand, err := client.Brand.Get(context.TODO(), contextdev.BrandGetParams{
OfByDomain: &contextdev.BrandGetParamsBodyByDomain{
Domain: "stripe.com",
client := contextdev.NewClient()
page, err := client.Web.Scrape(context.Background(), contextdev.WebScrapeParams{
URL: "https://example.com",
Formats: contextdev.WebScrapeParamsFormats{
Markdown: contextdev.Bool(true),
HTML: contextdev.Bool(true),
},
})
if err != nil {
panic(err)
}
fmt.Println(page.Markdown.Data)
fmt.Println(page.HTML.Data)
}
```

### Extract structured JSON

```go
package main

import (
"context"
"fmt"

"github.com/context-dot-dev/context-go-sdk/v2"
)

func main() {
client := contextdev.NewClient()
page, err := client.Web.Scrape(context.Background(), contextdev.WebScrapeParams{
URL: "https://example.com",
Formats: contextdev.WebScrapeParamsFormats{
Json: contextdev.Bool(true),
},
JsonParams: contextdev.WebScrapeParamsJsonParams{
Schema: map[string]any{
"type": "object",
"properties": map[string]any{
"title": map[string]any{"type": []string{"string", "null"}},
"description": map[string]any{"type": []string{"string", "null"}},
},
"required": []string{"title", "description"},
"additionalProperties": false,
},
},
})
if err != nil {
panic(err)
}
fmt.Printf("%+v\n", page.Json.Data)
}
```

### Extract relevant highlights

Return the passages that answer a question about the page.

```go
package main

import (
"context"
"fmt"

"github.com/context-dot-dev/context-go-sdk/v2"
)

func main() {
client := contextdev.NewClient()
page, err := client.Web.Scrape(context.Background(), contextdev.WebScrapeParams{
URL: "https://example.com",
Formats: contextdev.WebScrapeParamsFormats{
Highlights: contextdev.Bool(true),
},
HighlightsParams: contextdev.WebScrapeParamsHighlightsParams{
Query: "What is this domain used for?",
},
})
if err != nil {
panic(err.Error())
panic(err)
}
fmt.Printf("%+v\n", brand.RequestID)
fmt.Println(page.Highlights.Data)
}
```

### Take a screenshot

The screenshot is returned as a base64 image data URL.

```go
package main

import (
"context"
"fmt"

"github.com/context-dot-dev/context-go-sdk/v2"
)

func main() {
client := contextdev.NewClient()
page, err := client.Web.Scrape(context.Background(), contextdev.WebScrapeParams{
URL: "https://example.com",
Formats: contextdev.WebScrapeParamsFormats{
Screenshot: contextdev.Bool(true),
},
})
if err != nil {
panic(err)
}
fmt.Println(page.Screenshot.Data)
}
```

## What you can do

| Task | Method |
| --- | --- |
| Scrape a URL to markdown, HTML, JSON, highlights or a screenshot | `client.Web.Scrape` |
| Crawl a site and get every page as markdown | `client.Web.WebCrawlMd` |
| Map every URL on a domain | `client.Web.MapURLs` |
| Search the web | `client.Web.Search` |
| Take a screenshot of a page | `client.Web.Screenshot` |
| Parse PDFs and documents | `client.Parse.Handle` |
| Run thousands of URLs as a batch | `client.Batch.Submit` |
| Watch a page for changes | `client.Monitors.New` |
| Look up a company's logo, colors and brand data | `client.Brand.Get` |

## Use it from an AI agent

Context.dev also ships as a plugin for [Claude](https://github.com/context-dot-dev/claude-plugin), [Cursor](https://github.com/context-dot-dev/cursor-plugin) and [Gemini CLI](https://github.com/context-dot-dev/gemini-cli-context), and as tools for [LangChain](https://github.com/context-dot-dev/langchain-context) and [Haystack](https://github.com/context-dot-dev/context-haystack).

### Request fields

The contextdev library uses the [`omitzero`](https://tip.golang.org/doc/go1.24#encodingjsonpkgencodingjson)
Expand Down Expand Up @@ -270,7 +389,10 @@ client := contextdev.NewClient(
option.WithHeader("X-Some-Header", "custom_header_info"),
)

client.Brand.Get(context.TODO(), ...,
client.Web.Scrape(context.TODO(), contextdev.WebScrapeParams{
URL: "https://example.com",
Formats: contextdev.WebScrapeParamsFormats{Markdown: contextdev.Bool(true)},
},
// Override the header
option.WithHeader("X-Some-Header", "some_other_custom_header_info"),
// Add an undocumented field to the request body, using sjson syntax
Expand Down Expand Up @@ -301,18 +423,17 @@ When the API returns a non-success status code, we return an error with type
To handle errors, we recommend that you use the `errors.As` pattern:

```go
_, err := client.Brand.Get(context.TODO(), contextdev.BrandGetParams{
OfByDomain: &contextdev.BrandGetParamsBodyByDomain{
Domain: "stripe.com",
},
_, err := client.Web.Scrape(context.TODO(), contextdev.WebScrapeParams{
URL: "https://example.com",
Formats: contextdev.WebScrapeParamsFormats{Markdown: contextdev.Bool(true)},
})
if err != nil {
var apierr *contextdev.Error
if errors.As(err, &apierr) {
println(string(apierr.DumpRequest(true))) // Prints the serialized HTTP request
println(string(apierr.DumpResponse(true))) // Prints the serialized HTTP response
}
panic(err.Error()) // GET "/brand/retrieve": 400 Bad Request { ... }
panic(err.Error())
}
```

Expand All @@ -330,12 +451,11 @@ To set a per-retry timeout, use `option.WithRequestTimeout()`.
// This sets the timeout for the request, including all the retries.
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Minute)
defer cancel()
client.Brand.Get(
client.Web.Scrape(
ctx,
contextdev.BrandGetParams{
OfByDomain: &contextdev.BrandGetParamsBodyByDomain{
Domain: "stripe.com",
},
contextdev.WebScrapeParams{
URL: "https://example.com",
Formats: contextdev.WebScrapeParamsFormats{Markdown: contextdev.Bool(true)},
},
// This sets the per-retry timeout
option.WithRequestTimeout(20*time.Second),
Expand Down Expand Up @@ -370,12 +490,11 @@ client := contextdev.NewClient(
)

// Override per-request:
client.Brand.Get(
client.Web.Scrape(
context.TODO(),
contextdev.BrandGetParams{
OfByDomain: &contextdev.BrandGetParamsBodyByDomain{
Domain: "stripe.com",
},
contextdev.WebScrapeParams{
URL: "https://example.com",
Formats: contextdev.WebScrapeParamsFormats{Markdown: contextdev.Bool(true)},
},
option.WithMaxRetries(5),
)
Expand All @@ -389,19 +508,18 @@ you need to examine response headers, status codes, or other details.
```go
// Create a variable to store the HTTP response
var response *http.Response
brand, err := client.Brand.Get(
page, err := client.Web.Scrape(
context.TODO(),
contextdev.BrandGetParams{
OfByDomain: &contextdev.BrandGetParamsBodyByDomain{
Domain: "stripe.com",
},
contextdev.WebScrapeParams{
URL: "https://example.com",
Formats: contextdev.WebScrapeParamsFormats{Markdown: contextdev.Bool(true)},
},
option.WithResponseInto(&response),
)
if err != nil {
// handle error
}
fmt.Printf("%+v\n", brand)
fmt.Printf("%+v\n", page)

fmt.Printf("Status Code: %d\n", response.StatusCode)
fmt.Printf("Headers: %+#v\n", response.Header)
Expand Down
Loading