Agent Mag
SUBSCRIBE
Documentation
Web Intelligence

Scrape Endpoint

Fetch a single public URL and return clean content formats for AI agent workflows.

Status: Preview.

POST /v1/scrape is the first Web Intelligence endpoint. Use it when an agent needs a clean representation of one public page.

Request

code
curl https://api.agentmag.dev/v1/scrape \
  -H "Authorization: Bearer $AGENTMAG_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://docs.example.com",
    "formats": ["markdown", "links", "metadata"],
    "agentContext": true
  }'

Parameters

FieldTypeRequiredNotes
urlstringYesPublic URL to fetch.
formatsstring[]Nomarkdown, html, text, links, metadata, screenshot.
agentContextbooleanNoAdds summary and context hints for LLM workflows.
waitFornumberNoExtra browser wait time in milliseconds for JavaScript-heavy pages.
onlyMainContentbooleanNoPrefer article/docs body over nav, ads, and repeated chrome.

Response

code
{
  "id": "scrape_01h",
  "url": "https://docs.example.com",
  "status": "completed",
  "credits": 3,
  "data": {
    "markdown": "# Docs example...",
    "links": ["https://docs.example.com/api"],
    "metadata": {
      "title": "Docs example",
      "canonicalUrl": "https://docs.example.com"
    }
  }
}

Common Failures

CodeCauseHandling
400Missing or invalid URL.Validate URL before sending.
401Missing API key.Send bearer auth.
403Page blocks automated access or key lacks access.Try a different page or check workspace access.
408Page timed out.Lower requested formats or increase job timeout where allowed.
429Rate limit hit.Retry with exponential backoff.

Web Intelligence

Scrape, crawl, search, extract, and screenshot public web pages into agent-ready context.

Crawl Endpoint

Plan and inspect multi-page crawl jobs for docs sites, blogs, and product surfaces.

On this page

RequestParametersResponseCommon Failures