Scrape a URL

Fetch any public URL and receive its content as HTML, Markdown or plain text, together with cookies, response headers, the redirect chain, a block verdict, and optional CSS/XPath extraction. By default the endpoint tries a plain HTTP fetch first and escalates to a headless browser only when the target blocks it, billing only the winning attempt.

HTTP Request

1
POST /scrape

Send the parameters as a JSON body. The same parameters are also accepted as query-string parameters on a GET request: both verbs run the same handler, return the same response, and bill the same. Prefer POST for large body, extract_rules or cookie values, since URLs should stay under about 2 KB.

Parameters

Parameter	Type	Required	Default	Description
`url`	string	Yes	—	Absolute `http(s)` URL to fetch. Must point at a public address: private, loopback and internal addresses are rejected with a 400.
`render_js`	string	No	`auto`	`false`: plain HTTP fetch with a browser-grade TLS/header fingerprint. `true`: headless-browser rendering only. `auto`: HTTP first, escalating to the browser if the response is classified as blocked or the HTTP attempt errors. Only the winning attempt is billed.
`proxy_type`	string	No	`datacenter`	Proxy pool to exit from: `datacenter` or `isp`. `isp` uses static-residential addresses with higher trust on strict targets. Sessions are per pool: the same `session_id` on another `proxy_type` is a different exit.
`format`	string	No	`html`	Output format for `content`: `html`, `markdown` or `text`.
`main_content_only`	boolean	No	`false`	Only affects `markdown`/`text`: strips nav, footer and boilerplate and keeps the main article. The page’s `<h1>` is always kept.
`extras`	string	No	—	Comma-separated list of derived extras to compute, returned in `data.extras`. Allowed keys: `title`, `meta`, `links`, `images`, `emails`, `phone_numbers`, `tables`. Free. An unknown key returns a 400.
`extract_rules`	object	No	—	CSS/XPath extraction rules returned as JSON in `data.extract`. On `GET`, send it as a JSON-object string, e.g. `extract_rules={"title":"h1"}`. Free. Max 50 rules total, nesting depth 3.
`fields`	string	No	—	Allow-list of `data` keys to return, e.g. `content,http_status`. Trims the response to only the keys you name; useful because `content` is usually the largest field. An unknown name returns a 400 before anything is fetched or billed.
`session_id`	string	No	generated	Sticky-session handle, 1-64 chars `A-Za-z0-9_-`. The same value always maps to the same outbound IP, so follow-up requests come from the same address. Whatever was used is returned in `data.session_id`.
`cookies`	object, string or list	No	—	Cookies to send: an object `{"a": "1"}`, a cookie string `"a=1; b=2"`, or the `data.cookies` list exactly as `/unlock` returns it. Applied on the HTTP path only.
`user_agent`	string	No	—	Sent as-is on the HTTP path and used to select the matching TLS/header fingerprint. Ignored when the page is rendered in the browser, which uses its own User-Agent (returned in `data.user_agent`).
`headers`	object	No	`{}`	Extra request headers. On `GET`, send a JSON-object string. Reserved names are rejected (`Host`, `Cookie`, `User-Agent`, `Authorization`, `Content-Length` and others): use `cookies` and `user_agent` instead.
`method`	string	No	`GET`	HTTP method used against the target: `GET` or `POST`.
`body`	string	No	—	Request body sent to the target. Only allowed with `method=POST`, and only when `headers` carries a `Content-Type`.
`wait_for`	string	No	—	CSS selector to wait for after the page loads, before capturing. Browser path only (`render_js=true` or an `auto` escalation): useful for lazy-loaded or SPA content. At most 15 s and never past `timeout`. A selector that never appears is not an error.
`timeout`	integer	No	`25`	Total budget in seconds for the whole request, 1-27. The default is deliberately below the maximum so a browser render still fits inside the platform’s 30 s limit. Heavy pages should send `timeout=27` explicitly.

Credits

Credits depend on the proxy pool and on which path served the request:

Outcome	`proxy_type=datacenter`	`proxy_type=isp`
Served over plain HTTP	1	5
Served by the headless browser	5	20
Target blocked the request (`blocked: true`)	0	0

With render_js=auto, only the winning attempt is billed, never the sum of both. A blocked target costs 0 credits and the billing header is omitted entirely. data.credits, data.render_used and data.proxy_type always tell you exactly what you were billed. format=markdown, extras, extract_rules and fields cost nothing extra.

Response

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
{
  "status": true,
  "request_id": "a35488c9-6d45-3b35-e7a4-f9e75acd",
  "data": {
    "url": "https://httpbin.org/html",
    "http_status": 200,
    "redirect_chain": [{"url": "https://httpbin.org/html", "http_status": 200}],
    "blocked": false,
    "block_reason": null,
    "detected_protection": null,
    "format": "html",
    "content_type": "text/html",
    "charset": "utf-8",
    "source_length": 3739,
    "text_length": 3595,
    "extract": null,
    "extract_error": null,
    "extras": null,
    "convert_error": null,
    "content": "<!DOCTYPE html>\n<html>\n  <head>\n  </head>\n  <body>\n      <h1>Herman Melville - Moby-Dick</h1>…",
    "headers": {"content-type": "text/html; charset=utf-8"},
    "cookies": [],
    "user_agent": "Mozilla/5.0 (X11; Linux x86_64) … Chrome/146.0.0.0 Safari/537.36",
    "render_used": "http",
    "credits": 1,
    "elapsed_ms": 383,
    "session_id": "5e4cdb8ab2a90424",
    "proxy_type": "datacenter",
    "browser": null
  }
}

A blocked response is not an error: you still get the challenge page in content, the block_reason, and (when recognisable) detected_protection, all for 0 credits.

Response Fields

Field	Type	Description
`status`	boolean	Whether the request was successful
`request_id`	string	Unique identifier for the request
`data.url`	string	Final URL after redirects, on both paths
`data.http_status`	integer	The upstream HTTP status, never masked
`data.redirect_chain`	array or null	Every hop of the request in order, each `{url, http_status}`; the last entry is the final response. `null` only when the browser service did not report a chain
`data.redirect_chain[].url`	string	URL of this hop
`data.redirect_chain[].http_status`	integer	HTTP status of this hop
`data.blocked`	boolean	Whether the target served an anti-bot challenge or rejection instead of content
`data.block_reason`	string or null	Stable slug for why the page was classified as blocked, e.g. `rate_limited`, `forbidden`, `cloudflare_challenge`, `captcha`, `bot_detection`, `js_shell`, `auth_redirect`
`data.detected_protection`	string or null	Anti-bot vendor detected on the page: `cloudflare`, `akamai`, `datadome`, `perimeterx` or `incapsula`. Advisory: filled even when `blocked` is false
`data.format`	string	Echo of the requested format: `html`, `markdown` or `text`
`data.content_type`	string or null	Upstream media type without parameters, e.g. `text/html`
`data.charset`	string or null	Charset the body was decoded with. `null` when nothing could be determined, and on the browser path
`data.source_length`	integer	Character count of the source document, independent of `format`
`data.text_length`	integer or null	Character count of the visible text
`data.extract`	object or null	Result of `extract_rules`
`data.extract_error`	string or null	Set instead of `extract` when extraction failed at run time (timeout, caps exceeded). The rest of the response is still returned and billed
`data.extras`	object or null	Requested derived extras: `title`, `meta`, `links`, `images`, `emails`, `phone_numbers`, `tables`. Only the keys you requested are present; a requested key with nothing found is `null`
`data.extras.links`	array	List of `{href, text}` objects, absolutised and de-duplicated (first 500)
`data.convert_error`	string or null	Set when markdown/text conversion or the requested extras failed on this document. `content` then falls back to raw HTML and `format` to `html`
`data.content`	string or null	The document in the requested `format`
`data.headers`	object or null	Final-hop response headers with lowercased keys. `null` on the browser path
`data.cookies`	array	Cookies collected during the fetch, each `{name, value, domain, path, expires, http_only, secure}`. On the HTTP path these are the cookies the server set; cookies created client-side by JavaScript only appear on the browser path
`data.user_agent`	string or null	The User-Agent actually sent (HTTP) or used by the browser
`data.render_used`	string	Which path produced the answer: `http` or `browser`
`data.credits`	integer	Credits billed for this call. Same value as the `X-RapidAPI-Billing` header, which is omitted at 0
`data.elapsed_ms`	integer	Total server-side time for this call in milliseconds
`data.session_id`	string	The sticky-session handle used (yours, or a generated one). Reuse it to keep the same IP
`data.proxy_type`	string	Proxy pool this request went out of: `datacenter` or `isp`
`data.browser`	object or null	Browser path only: `{captchas_solved, elapsed_ms}`, the challenges solved and the browser-side render time

Error Responses

Code	Meaning
`400`	Invalid request: bad URL, reserved header name, malformed `extract_rules`, out-of-range `timeout`, unknown `extras` key, `body` without a `Content-Type`, or a URL that points at a non-public address. The error message names the offending parameter
`415`	The URL returned a non-textual document, e.g. `image/png`
`502`	The target could not be fetched. Body: `could not fetch target: <reason>` where `<reason>` is a fixed slug: `dns_error`, `connection_refused`, `connection_reset`, `tls_error`, `timeout`, `too_many_redirects`, `protocol_error`, `empty_response`, `target_error` or `fetch_error`
`503`	Our infrastructure is unavailable: the outbound proxy could not be used, the browser pool is exhausted, or a requested `proxy_type` pool is not configured. A target-side failure is always a 502, never a 503

Error responses are never billed and carry no billing header:

1
{"status": false, "request_id": "f9660af2", "error": "could not fetch target: dns_error"}

Code Examples

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
import requests

url = "https://ai-web-scraper-api1.p.rapidapi.com/scrape"

payload = {
    "url": "https://example-store.com/products",
    "render_js": "auto",
    "format": "markdown"
}

headers = {
    "X-RapidAPI-Key": "YOUR_API_KEY",
    "X-RapidAPI-Host": "ai-web-scraper-api1.p.rapidapi.com",
    "Content-Type": "application/json"
}

response = requests.post(url, headers=headers, json=payload)
data = response.json()["data"]

print(f"Status: {data['http_status']}, rendered via: {data['render_used']}")
print(f"Credits: {data['credits']}")
print(data["content"][:500])

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
const url = "https://ai-web-scraper-api1.p.rapidapi.com/scrape";

const response = await fetch(url, {
  method: "POST",
  headers: {
    "X-RapidAPI-Key": "YOUR_API_KEY",
    "X-RapidAPI-Host": "ai-web-scraper-api1.p.rapidapi.com",
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    url: "https://example-store.com/products",
    render_js: "auto",
    format: "markdown",
  }),
});

const { data } = await response.json();

console.log(`Status: ${data.http_status}, rendered via: ${data.render_used}`);
console.log(`Credits: ${data.credits}`);
console.log(data.content.slice(0, 500));

1
2
3
4
5
6
7
8
9
curl -X POST "https://ai-web-scraper-api1.p.rapidapi.com/scrape" \
  -H "Content-Type: application/json" \
  -H "X-RapidAPI-Key: YOUR_API_KEY" \
  -H "X-RapidAPI-Host: ai-web-scraper-api1.p.rapidapi.com" \
  -d '{
    "url": "https://example-store.com/products",
    "render_js": "auto",
    "format": "markdown"
  }'

Unlock — Clear anti-bot challenges in a headless browser once, then replay the returned cookies and session on /scrape with render_js=false at HTTP price
AI Extract — Fetch a URL and extract the fields you describe as JSON, no selectors needed
Generate Extraction Rules — Write a reusable extract_rules object from sample pages once, then run it here for fetch credits only