crawlfox / mcp
Sign up

MCP

Get Started

Set up the CrawlFox MCP server with Claude Desktop, Cursor, Claude Code, or any MCP-compatible host.

The server exposes CrawlFox's web scraping and search API as tools your model can call. It is a thin adapter over api.crawlfox.io/v1 — no browser to run, no proxies to manage. Connections authenticate with OAuth, so no API key is ever pasted into a client.

Connect your client

Drop the server URL into Claude Desktop, Cursor, or Claude Code. Copy-paste config for each, below.

Get the config

Sign in with OAuth

Your client opens a browser to sign in and approve access. No API key is pasted, stored, or shown.

Create an account

Browse the tools

Thirteen tools for scraping, search, batch fetches, structured extraction, and request logs.

See all tools

Connect

Point your MCP client at this URL. Authorization is handled by the client, not by you:

URL: https://mcp.crawlfox.io/mcp
Transport: Streamable HTTP
Auth: OAuth 2.1 — your client runs the flow, no key to paste

Your client registers itself dynamically (DCR) with PKCE and opens a browser for sign-in and approval. Usage bills to your account's single existing API key, which stays server-side — it is never shown, stored in a client, or sent to your machine.

This is a server URL for your MCP client, not a page to open in a browser. Opening /mcp directly returns 401 with a WWW-Authenticate header — that response is the auth challenge working, and it is what tells your client where to sign in.

Client setup

Pick your client. Each one runs the OAuth handshake for you.

Claude Desktop

Connectors

Remote servers are added through Connectors, not claude_desktop_config.json — that file only launches local stdio servers.

server url

https://mcp.crawlfox.io/mcp
  1. Open Settings → Connectors → Add custom connector.
  2. Paste the URL above, save, then click Connect to run the OAuth sign-in.

Cursor

mcp.json

Add to ~/.cursor/mcp.json for every project, or .cursor/mcp.json for just one.

~/.cursor/mcp.json

{
  "mcpServers": {
    "crawlfox": {
      "url": "https://mcp.crawlfox.io/mcp"
    }
  }
}
  1. Open Settings → MCP and sign in when Cursor prompts.

Claude Code

CLI

Add it from the terminal, or write the entry by hand. The type field is required — an entry with a url and no type is read as stdio and fails.

terminal

claude mcp add --transport http crawlfox https://mcp.crawlfox.io/mcp

.mcp.json

{
  "mcpServers": {
    "crawlfox": {
      "type": "http",
      "url": "https://mcp.crawlfox.io/mcp"
    }
  }
}
  1. Run /mcp, pick crawlfox, then Authenticate.
  2. A browser opens for sign-in; the session reconnects authenticated.

Any MCP host

Spec

A spec-compliant client discovers everything it needs from these two documents. Registration is dynamic (DCR) with PKCE, so there is no client ID to request from us.

discovery

GET /.well-known/oauth-protected-resource/mcp   # RFC 9728
GET /.well-known/oauth-authorization-server     # RFC 8414
  1. An unauthenticated call to /mcp returns 401 with a WWW-Authenticate header pointing at the metadata above — that response starts the flow.

Tools

Thirteen tools are available to every connected client once authenticated.

crawlfox_scrape

Fetch one URL as clean, LLM-ready content — Markdown, HTML, text, links, images or emails. Handles JS-rendered and anti-bot pages.

POST /v1/scrape

crawlfox_search

Web search via Google, Bing, or DuckDuckGo, returning ranked organic results.

POST /v1/serp/:engine

crawlfox_batch_scrape

Scrape up to 50 URLs in a single call, one result per page.

POST /v1/batch

crawlfox_extract

Pull named fields out of a page into JSON using CSS selectors — deterministic, no AI in the loop.

POST /v1/scrape

crawlfox_metadata

Return only page metadata: title, description, and status.

POST /v1/scrape

crawlfox_links

Extract every link on a page.

POST /v1/scrape

crawlfox_images

Extract every image URL on a page.

POST /v1/scrape

crawlfox_emails

Extract every email address on a page.

POST /v1/scrape

crawlfox_text

Return a page as plain text.

POST /v1/scrape

crawlfox_html

Return a page as clean HTML.

POST /v1/scrape

crawlfox_scrape_get

Lightweight GET scrape of a URL, for quick one-off fetches.

GET /v1/scrape/:url

crawlfox_get_log

Look up the status and timing of a past request by its id.

GET /v1/logs/:id

crawlfox_log_result

Retrieve the full stored result of a past scrape by id.

GET /v1/logs/:id/result

Next: read the CrawlFox API docs or learn more about MCP.