Unlock a Site

Open a URL in a headless browser, clear any anti-bot challenges the site throws, and receive the resulting session: the clearance cookies, the browser’s user_agent, and the sticky session_id. Replay all three on /scrape with render_js=false and the site keeps serving you over the 1-credit HTTP path, no JavaScript rendering needed.

The clearance is tied to the exact combination of User-Agent, IP, and proxy pool. Cookies are IP-bound, so you must send back the same session_id (which pins the same outbound IP) and the same proxy_type. Change any of the three and the cookies stop working.

HTTP Request

1
POST /unlock

Send the parameters as a JSON body. The same parameters are also accepted as query-string parameters on a GET request: both verbs run the same handler and bill the same.

Parameters

Parameter	Type	Required	Default	Description
`url`	string	Yes	—	Absolute `http(s)` URL to unlock. Must point at a public address.
`proxy_type`	string	No	`datacenter`	Proxy pool to exit from: `datacenter` or `isp` (static-residential, higher trust with strict targets). Use the same value on the follow-up `/scrape`, or the cookies come back from another pool’s IP.
`session_id`	string	No	generated	Sticky-session handle, 1-64 chars `A-Za-z0-9_-`. The same value always maps to the same outbound IP. Send the returned one back on `/scrape` so the cookies are replayed from the IP that minted them.
`headers`	object	No	`{}`	Extra request headers. Reserved names (`Host`, `Cookie`, `User-Agent`, `Content-Length`, and others) are rejected. As a `GET` param, pass a JSON object string.
`wait_for`	string	No	—	CSS selector to wait for after the page loads, before capturing the session. For sites whose clearance cookies are set by page scripts. At most 15 s and never past `timeout`; a selector that never appears is not an error. 1-500 chars.
`timeout`	integer	No	`25`	Total budget for the whole call in seconds, 1-27. The browser render is capped by whatever is left of it.
`fields`	string	No	—	Allow-list of `data` keys to return, e.g. `cookies,user_agent,session_id` (the replay bundle). Any unknown name returns a validation error before anything is fetched.

Credits

Outcome	`proxy_type=datacenter`	`proxy_type=isp`
Unlock completed	5	20
Target blocked the request	0	0

/unlock costs the same as a browser-rendered /scrape on the same pool. The point of the workflow is what comes after: every replayed /scrape with render_js=false costs only the HTTP price (1 credit on datacenter, 5 on isp).

Response

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
{
  "status": true,
  "request_id": "a35488c9-6d45-3b35-e7a4-f9e75acd",
  "data": {
    "url": "https://www.crunchbase.com/",
    "http_status": 200,
    "redirect_chain": [
      {"url": "https://crunchbase.com/", "http_status": 301},
      {"url": "https://www.crunchbase.com/", "http_status": 200}
    ],
    "cookies": [
      {
        "name": "cf_clearance",
        "value": "kX9f2...",
        "domain": ".crunchbase.com",
        "path": "/",
        "expires": 1787184162,
        "http_only": true,
        "secure": true
      },
      {
        "name": "__cf_bm",
        "value": "aQ7b1...",
        "domain": ".crunchbase.com",
        "path": "/",
        "expires": 1755649962,
        "http_only": true,
        "secure": true
      }
    ],
    "user_agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:152.0) Gecko/20100101 Firefox/152.0",
    "session_id": "5e4cdb8ab2a90424",
    "proxy_type": "datacenter",
    "credits": 5,
    "elapsed_ms": 8471,
    "browser": {
      "captchas_solved": [],
      "elapsed_ms": 8102
    }
  }
}

cookies is empty for a site that sets none. A challenged site returns the clearance cookies it issued, exactly in the shape shown above. There is deliberately no fingerprint hint in the response: /scrape derives the right TLS/header fingerprint from the user_agent you send back.

Response Fields

Field	Type	Description
`status`	boolean	Whether the request was successful
`request_id`	string	Unique identifier for the request
`data.url`	string	Final URL after redirects
`data.http_status`	integer	HTTP status of the final document, never masked
`data.redirect_chain`	array \| null	Every hop of the main-frame navigation, in order; the last entry is the final response. `null` if the browser did not report a chain
`data.redirect_chain[].url`	string	URL of this hop
`data.redirect_chain[].http_status`	integer	HTTP status of this hop
`data.cookies`	array	Cookies collected by the browser during the unlock, including cookies set client-side by page scripts
`data.cookies[].name`	string	Cookie name
`data.cookies[].value`	string	Cookie value
`data.cookies[].domain`	string	Domain the cookie is scoped to
`data.cookies[].path`	string	Path the cookie is scoped to
`data.cookies[].expires`	integer \| null	Expiry as Unix seconds, `null` for session cookies
`data.cookies[].http_only`	boolean	Whether the cookie carries the HttpOnly attribute
`data.cookies[].secure`	boolean	Whether the cookie carries the Secure attribute
`data.user_agent`	string	The User-Agent the browser used. Send it back verbatim on `/scrape`
`data.session_id`	string	The sticky-session handle used (yours, or a generated one). Send it back on `/scrape` to keep the same IP
`data.proxy_type`	string	Proxy pool this request went out of: `datacenter` or `isp`
`data.credits`	integer	Credits billed for this call: 5 on `datacenter`, 20 on `isp`, 0 when the target blocked the request
`data.elapsed_ms`	integer	Total server-side time for this call
`data.browser`	object	Browser-side details
`data.browser.captchas_solved`	array	Challenges that were solved during the render
`data.browser.elapsed_ms`	integer	Browser-side render time in milliseconds

Replay the Session

Take cookies, user_agent, and session_id from the response above and paste them into a /scrape call with render_js=false. The cookies field accepts the list of objects exactly as /unlock returned it (extra attributes like domain and expires are accepted and ignored; only name and value are sent).

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
curl -X POST "https://ai-web-scraper-api1.p.rapidapi.com/scrape" \
  -H "X-RapidAPI-Key: YOUR_API_KEY" \
  -H "X-RapidAPI-Host: ai-web-scraper-api1.p.rapidapi.com" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://www.crunchbase.com/organization/openai",
    "render_js": false,
    "session_id": "5e4cdb8ab2a90424",
    "user_agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:152.0) Gecko/20100101 Firefox/152.0",
    "cookies": [
      {"name": "cf_clearance", "value": "kX9f2..."},
      {"name": "__cf_bm", "value": "aQ7b1..."}
    ]
  }'

Each replayed request costs 1 credit (datacenter) instead of 5, and you can keep paginating the site from the same IP for as long as the clearance cookies stay valid.

Error Responses

Status	Meaning
`400`	Invalid parameter: bad URL, non-public target address, reserved header name, out-of-range `timeout`, unknown `fields` key. Nothing is fetched and nothing is billed.
`415`	The URL returned a non-textual document, e.g. `image/png`
`502`	The target could not be fetched. The error body is `could not fetch target: <reason>` where `<reason>` is a stable slug: `dns_error`, `connection_refused`, `connection_reset`, `tls_error`, `timeout`, `too_many_redirects`, `protocol_error`, `empty_response`, `target_error`, `unavailable`, or `fetch_error`. Not billed.
`503`	Our infrastructure is unavailable: the outbound proxy could not be used, the browser pool is exhausted, or a requested `proxy_type` pool is not configured. A target-side failure is always `502`, never `503`. Not billed.

Error bodies use the standard envelope and carry no billing header:

1
2
3
4
5
{
  "status": false,
  "request_id": "f9660af2",
  "error": "could not fetch target: dns_error"
}

A blocked target is not an error: you still get an HTTP 200 with the block verdict, billed at 0 credits.

Code Examples

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
import requests

base = "https://ai-web-scraper-api1.p.rapidapi.com"

headers = {
    "X-RapidAPI-Key": "YOUR_API_KEY",
    "X-RapidAPI-Host": "ai-web-scraper-api1.p.rapidapi.com",
    "Content-Type": "application/json"
}

# Step 1: unlock the site in a headless browser
unlock = requests.post(
    f"{base}/unlock",
    headers=headers,
    json={"url": "https://www.crunchbase.com/"}
).json()["data"]

# Step 2: replay the session on /scrape over plain HTTP (1 credit)
scrape = requests.post(
    f"{base}/scrape",
    headers=headers,
    json={
        "url": "https://www.crunchbase.com/organization/openai",
        "render_js": False,
        "session_id": unlock["session_id"],
        "user_agent": unlock["user_agent"],
        "cookies": unlock["cookies"]
    }
).json()["data"]

print(scrape["http_status"], scrape["render_used"], scrape["credits"])
print(scrape["content"][:500])

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
const base = "https://ai-web-scraper-api1.p.rapidapi.com";

const headers = {
  "X-RapidAPI-Key": "YOUR_API_KEY",
  "X-RapidAPI-Host": "ai-web-scraper-api1.p.rapidapi.com",
  "Content-Type": "application/json",
};

// Step 1: unlock the site in a headless browser
const unlockRes = await fetch(`${base}/unlock`, {
  method: "POST",
  headers,
  body: JSON.stringify({ url: "https://www.crunchbase.com/" }),
});
const { data: unlock } = await unlockRes.json();

// Step 2: replay the session on /scrape over plain HTTP (1 credit)
const scrapeRes = await fetch(`${base}/scrape`, {
  method: "POST",
  headers,
  body: JSON.stringify({
    url: "https://www.crunchbase.com/organization/openai",
    render_js: false,
    session_id: unlock.session_id,
    user_agent: unlock.user_agent,
    cookies: unlock.cookies,
  }),
});

const { data: scrape } = await scrapeRes.json();
console.log(scrape.http_status, scrape.render_used, scrape.credits);
console.log(scrape.content.slice(0, 500));

1
2
3
4
5
curl -X POST "https://ai-web-scraper-api1.p.rapidapi.com/unlock" \
  -H "Content-Type: application/json" \
  -H "X-RapidAPI-Key: YOUR_API_KEY" \
  -H "X-RapidAPI-Host: ai-web-scraper-api1.p.rapidapi.com" \
  -d '{"url": "https://www.crunchbase.com/"}'

Scrape — Fetch any URL and get its content as HTML, Markdown, or text
AI Extract — Describe the fields you want and get structured JSON back, no selectors