AI Web Scraper API
Complete documentation for the AI Web Scraper API. Scrape any URL as HTML, Markdown or JSON with anti-bot unlocking and AI extraction.
Web Scraping APIEndpoints
5 endpoints/scrapeFetch any URL and get its content as HTML, Markdown or text, with anti-bot detection, JS …
/unlockOpen a URL in a headless browser, clear anti-bot challenges, and get back the clearance …
/ai-extractFetch any URL and extract the fields you describe as structured JSON, no CSS selectors …
/ai-generate-extraction-rulesGenerate a reusable CSS/XPath extract_rules object from 1-3 sample URLs, then run it on …
/ai-summarizeFetch any URL and get back a prose summary as text or markdown, with key points, title, …
Overview
The AI Web Scraper API fetches any URL you give it and returns the content as HTML, Markdown or structured JSON. Every request runs through one of two fetch paths: a plain HTTP fetch with a real-browser TLS fingerprint, or a headless browser that renders JavaScript and solves anti-bot challenges. Two exit pools are available (datacenter and static-residential ISP), and three AI endpoints sit on top of the fetcher for extraction, rule generation and summarization.
Every endpoint accepts both GET and POST with identical parameters: send them as query-string parameters or as a JSON body, same handler, same billing, same errors.
Authentication
All requests require a valid RapidAPI key. Include the following headers with every request:
| Header | Value |
|---|---|
X-RapidAPI-Key | YOUR_API_KEY |
X-RapidAPI-Host | ai-web-scraper-api1.p.rapidapi.com |
You can obtain an API key by subscribing to the AI Web Scraper API on RapidAPI .
Base URL
| |
Endpoints
| Endpoint | Method | Path | Description |
|---|---|---|---|
| Scrape a URL | POST | /scrape | Fetch any URL and return its content as HTML, Markdown or text, plus cookies, headers, redirect chain and optional CSS/XPath extraction |
| Unlock a Site | POST | /unlock | Open a URL in a headless browser, clear anti-bot challenges and return the resulting cookies and User-Agent for cheap HTTP replay |
| AI Extract | POST | /ai-extract | Fetch a URL and extract the fields you describe with a schema or prompt, no selectors needed |
| Generate Extraction Rules | POST | /ai-generate-extraction-rules | Turn 1-3 sample URLs into a reusable extract_rules object you can run on /scrape at fetch price |
| AI Summarize | POST | /ai-summarize | Fetch a URL and return a summary as text or Markdown, with length, language and custom instructions |
The three render modes
The render_js parameter controls which fetch path a request uses:
render_js | What happens | Credits |
|---|---|---|
false | Plain HTTP fetch with a browser-grade TLS and header fingerprint | 1 |
auto (default) | HTTP first. If the response classifies as blocked, or the HTTP attempt errors, the request escalates to headless-browser rendering. Only the winning attempt is billed, never the sum | 1 or 5 |
true | Headless-browser rendering only | 5 |
The response field data.render_used ("http" or "browser") always tells you which path produced the answer.
Sessions
Every response carries a data.session_id. Send it back as session_id on your next request and it goes out through the same exit IP. Omit it and each request gets a fresh random exit.
Cookies and the User-Agent travel with the session: pass the cookies and user_agent values a response returned and they are replayed exactly. This is what makes the unlock-then-scrape pattern work. Anti-bot clearance cookies are bound to the IP that earned them, so calling /unlock once and then replaying its session bundle on /scrape with render_js=false and the same session_id lets you keep scraping a protected site over plain HTTP at 1 credit per request.
Credits & Billing
Each request costs credits depending on the fetch path and the exit pool (proxy_type):
| Outcome | proxy_type=datacenter (default) | proxy_type=isp |
|---|---|---|
/scrape served over HTTP | 1 | 5 |
/scrape served by the headless browser | 5 | 20 |
/unlock (always browser price) | 5 | 20 |
Target blocked the request (blocked: true) | 0 | 0 |
/ai-extract | fetch credits + 5 extraction | fetch credits + 5 extraction |
/ai-generate-extraction-rules | fetch credits + 15 generation | fetch credits + 15 generation |
/ai-summarize | fetch credits + 5 summary | fetch credits + 5 summary |
AI surcharges are billed only when output is actually produced: a blocked target or a failed extraction never adds the extra credits. A request the target blocked costs 0 credits.
On RapidAPI plans these credits appear as three metered objects: Requests, Requests JS (browser renders) and AI extraction or summarize. Using proxy_type=isp consumes the same meters at its higher credit rate. See the RapidAPI listing
for plan prices and quotas.
Rate Limits
Rate limits depend on your RapidAPI subscription plan. Check your plan details on the RapidAPI listing for specific limits.