AI Web Scraper API - Scrape Any Website to Markdown or JSON

What Is the AI Web Scraper API?

FlyByAPIs AI Web Scraper API is a web scraping API that fetches any URL and returns it as HTML, LLM-ready Markdown, or structured JSON. Under the hood there are two fetch paths: plain HTTP with a real-browser TLS fingerprint, and a headless browser that solves anti-bot challenges. On top sits an AI web scraper layer for schema-based extraction and summaries. And one rule we think the whole category should follow: if you didn’t get data, you don’t pay. Available on RapidAPI starting at $0/month with 100 free requests.

Need more than an API? We also build custom endpoints, run done-for-you extractions delivered as CSV or JSON, and help you set up automated pipelines. If your use case does not fit the endpoints below, tell us what you need:

Scrape Any Website, Pay Only for What Works

Every response tells you exactly what happened and exactly what you paid: the real upstream status code (never masked), whether the page was blocked and by which anti-bot system, which fetch path won, and the precise credits billed. Blocked requests cost 0 credits.

The default mode, render_js=auto, tries the cheap HTTP path first and escalates to the headless browser only when a site actually blocks it. Only the winning attempt is billed, never both. Output comes back the way you need it: raw HTML, clean Markdown with boilerplate stripped (main_content_only), plain text, CSS/XPath extraction rules, or free extras like title, links, emails, phone numbers, and tables. On top of that, three AI endpoints turn any page into structured JSON or a summary, without you writing a single selector.

Endpoints

All five endpoints accept both GET and POST with the same parameters. Full parameter reference in the Web Scraper docs . Just need summaries without the full scraping toolkit? The AI Article Extractor & Summarizer API is the standalone version of /ai-summarize.

Code Examples (Python, JavaScript, cURL)

Turn any URL into LLM-ready Markdown in one call:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

import requests

url = "https://ai-web-scraper-api1.p.rapidapi.com/scrape"

payload = {
    "url": "https://example.com/blog/some-article",
    "format": "markdown",
    "main_content_only": True,
    "render_js": "auto"
}

headers = {
    "X-RapidAPI-Key": "YOUR_API_KEY",
    "X-RapidAPI-Host": "ai-web-scraper-api1.p.rapidapi.com"
}

response = requests.post(url, json=payload, headers=headers)
data = response.json()["data"]

print(data["render_used"], data["credits"], data["http_status"])
print(data["content"])  # clean Markdown, boilerplate stripped

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

const url = "https://ai-web-scraper-api1.p.rapidapi.com/scrape";

const response = await fetch(url, {
  method: "POST",
  headers: {
    "X-RapidAPI-Key": "YOUR_API_KEY",
    "X-RapidAPI-Host": "ai-web-scraper-api1.p.rapidapi.com",
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    url: "https://example.com/blog/some-article",
    format: "markdown",
    main_content_only: true,
    render_js: "auto",
  }),
});

const { data } = await response.json();
console.log(data.render_used, data.credits, data.http_status);
console.log(data.content); // clean Markdown, boilerplate stripped

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

curl -X POST "https://ai-web-scraper-api1.p.rapidapi.com/scrape" \
  -H "X-RapidAPI-Key: YOUR_API_KEY" \
  -H "X-RapidAPI-Host: ai-web-scraper-api1.p.rapidapi.com" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/blog/some-article",
    "format": "markdown",
    "main_content_only": true,
    "render_js": "auto"
  }'

Example JSON Response

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

{
  "status": true,
  "request_id": "req_9f3a1c2e",
  "data": {
    "url": "https://example.com/blog/some-article",
    "http_status": 200,
    "blocked": false,
    "content_type": "text/html",
    "charset": "utf-8",
    "content": "# Some Article Title\n\nThe main content of the page as clean Markdown, with navigation, footer and boilerplate stripped out...",
    "render_used": "http",
    "credits": 1,
    "elapsed_ms": 1840
  }
}

That credits: 1 in the body is the whole philosophy in one field. If the browser had been needed, render_used would say "browser" and credits would say 5. If the site had blocked us, blocked would be true, block_reason would tell you why, and credits would be 0.

Features

• Two fetch paths, one decision made for you: render_js=auto tries plain HTTP with a real-browser TLS fingerprint first and escalates to a challenge-solving headless browser only when needed. You are billed for the winning attempt only.
• HTML, Markdown, or text output: format=markdown with main_content_only=true gives you clean, LLM-ready Markdown from any page in one call.
• Free extras: title, meta, links, images, emails, phone_numbers, and tables, parsed from the same fetch at no extra cost.
• CSS/XPath extract_rules: a ScrapingBee-compatible grammar, so existing extraction rules paste straight in. Hand-written or generated for you by AI.
• AI extraction that doesn’t guess: /ai-extract returns validated JSON matching your schema, with null for fields the page doesn’t have. Never a plausible invention.
• AI-generated rules, reusable forever: /ai-generate-extraction-rules runs the model once, writes validated selectors, and every future page of that layout costs fetch credits only. Nobody else in the category offers this.
• A real web unlocker: /unlock mints clearance cookies, the browser’s user_agent, and a session_id. Replay them on /scrape and keep scraping the cleared site at the 1-credit HTTP price.
• Honest diagnostics in every response: real upstream http_status (never masked), blocked with stable block_reason slugs, detected_protection naming Cloudflare, Akamai, DataDome, PerimeterX, or Imperva, render_used, and the exact credits billed.
• Sticky sessions: session_id keeps the same exit IP across requests, on both fetch paths and all five endpoints, so pagination and cookie replay actually work.
• Two exit pools per request: datacenter (1 credit HTTP / 5 browser) or isp static-residential trust for strict targets (5 / 20), each rate visible in the response.
• GET and POST everywhere: same parameters, same behavior, so it works from a browser bar, a cron job, or the RapidAPI playground.

Use Cases

Why Developers Switch

There is a cost baked into this category that nobody itemizes on their pricing page. We call it the scraping tax: you pay for attempts that returned nothing, for JS-render surcharges on pages that never needed rendering, for an LLM run on every single page of the same layout, and for responses that mask the real status code so you can’t even tell what went wrong. None of that is data. All of it is billed.

And the alternative, building it yourself, means fighting the anti-bot arms race alone: TLS fingerprints, challenge pages, rotating proxies, layouts that shift under your parser. We fight that fight every day. Every endpoint is monitored 24/7, and when a source changes we ship the fix within hours. Here is where the tax hides:

To be fair: some of these tools are genuinely good at things we don’t do. Firecrawl’s SDK ecosystem is mature and pleasant to work with. What we refuse to copy is the meter.

Comparison

Are we the cheapest at every volume? Depends on your workload, so we won’t claim that. Here is what we do claim: our meter never charges you for a page you didn’t get, a render that wasn’t used, or a model run that produced nothing. Check any response, the numbers are right there.

Get Started in Minutes

1. Sign up on RapidAPI , it’s free.
2. Subscribe to the Basic plan (no credit card required).
3. Copy your API key from the RapidAPI dashboard.
4. Make your first request using the code examples above.

Every plan meters JS renders at requests/5 and AI operations at requests/6, which is exactly the credit ladder above mapped to your quota: no hidden multipliers, no fine print. And every week you keep running a homemade scraper is another week of proxy bills and emergency patches while your actual product waits. Long story short: start on the free tier. 100 requests a month, hard limit, no credit card, no surprise bill. Your first Markdown response is usually back in under 5 minutes.