Generate Extraction Rules

Give this endpoint 1 to 3 sample URLs that share the same page layout, plus a schema or a plain-English prompt describing what you want. The model writes a reusable extract_rules object once, validates it across your sample pages (per-page validation report plus a matched_fields verdict), and returns it ready to paste into Scrape . From then on, /scrape runs those rules on every future page with that layout for fetch credits only: no more model calls, no extraction surcharge. This is the cheap path at volume.

HTTP Request

1
2

POST /ai-generate-extraction-rules
GET  /ai-generate-extraction-rules

Both verbs run the same handler and bill the same. On GET, pass urls comma-separated (up to 3, URL-encode each) and schema as a JSON-object string. Use POST for long URL lists or big schemas.

Parameters

* At least one of schema or prompt is required.

There is no url, fields, or source parameter on this endpoint. It always reads a cleaned-up copy of the page markup, because selectors need the underlying HTML, not just visible text.

The Reuse Workflow

1. POST /ai-generate-extraction-rules with 1 to 3 sample URLs of the same layout and a schema.
2. Take data.extract_rules from the response.
3. POST /scrape on any page with that layout, passing it as extract_rules. Fetch credits only, no further AI cost, no further model calls.

Credits

Sum of fetch credits for every non-blocked sample URL, plus 15 generation credits when extract_rules was produced and matched at least one field on at least one page (matched_fields is all or some). Rules that matched nothing anywhere (matched_fields: "none") are still returned so you can inspect what was tried, but the 15 generation credits are not billed. If every URL is blocked: 0 credits and ai_error: "blocked".

Fetch credits per URL follow the /scrape table: datacenter 1 (HTTP) / 5 (browser), isp 5 (HTTP) / 20 (browser). At most two model calls happen per request regardless of URL count, so 3 sample URLs do not cost 3x the generation price.

Response

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

{
  "status": true,
  "request_id": "a35488c9-6d45-3b35-e7a4-f9e75acd",
  "data": {
    "extract_rules": {
      "title": "h1.product-title",
      "price": {"selector": ".price", "output": "text"}
    },
    "sample": {"title": "14-inch Ultrabook", "price": "899.00"},
    "validation": [
      {"url": "https://shop.example/p/14-ultrabook", "http_status": 200, "render_used": "http", "blocked": false, "fetch_error": null, "empty_fields": [], "list_counts": {}},
      {"url": "https://shop.example/p/16-workstation", "http_status": 200, "render_used": "http", "blocked": false, "fetch_error": null, "empty_fields": [], "list_counts": {}}
    ],
    "matched_fields": "all",
    "warnings": [],
    "ai_error": null,
    "credits": 17,
    "elapsed_ms": 3820,
    "session_id": "5e4cdb8ab2a90424",
    "proxy_type": "datacenter"
  }
}

Response Fields

Error Responses

Generation failures (the pages fetched fine but the model produced no usable rules) are returned as HTTP 200 with status: false and data.ai_error set, never as an HTTP error status.

Code Examples

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

import requests

url = "https://ai-web-scraper-api1.p.rapidapi.com/ai-generate-extraction-rules"

payload = {
    "urls": [
        "https://shop.example/p/14-ultrabook",
        "https://shop.example/p/16-workstation"
    ],
    "schema": {"title": "string", "price": "number"},
    "timeout": 25
}

headers = {
    "Content-Type": "application/json",
    "X-RapidAPI-Key": "YOUR_API_KEY",
    "X-RapidAPI-Host": "ai-web-scraper-api1.p.rapidapi.com"
}

response = requests.post(url, json=payload, headers=headers)
data = response.json()["data"]

print("Verdict:", data["matched_fields"])
print("Rules:", data["extract_rules"])
print("Sample:", data["sample"])

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

const url = "https://ai-web-scraper-api1.p.rapidapi.com/ai-generate-extraction-rules";

const response = await fetch(url, {
  method: "POST",
  headers: {
    "Content-Type": "application/json",
    "X-RapidAPI-Key": "YOUR_API_KEY",
    "X-RapidAPI-Host": "ai-web-scraper-api1.p.rapidapi.com",
  },
  body: JSON.stringify({
    urls: [
      "https://shop.example/p/14-ultrabook",
      "https://shop.example/p/16-workstation",
    ],
    schema: { title: "string", price: "number" },
    timeout: 25,
  }),
});

const { data } = await response.json();

console.log("Verdict:", data.matched_fields);
console.log("Rules:", data.extract_rules);
console.log("Sample:", data.sample);

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

curl -X POST "https://ai-web-scraper-api1.p.rapidapi.com/ai-generate-extraction-rules" \
  -H "Content-Type: application/json" \
  -H "X-RapidAPI-Key: YOUR_API_KEY" \
  -H "X-RapidAPI-Host: ai-web-scraper-api1.p.rapidapi.com" \
  -d '{
    "urls": [
      "https://shop.example/p/14-ultrabook",
      "https://shop.example/p/16-workstation"
    ],
    "schema": {"title": "string", "price": "number"},
    "timeout": 25
  }'

Reusing the Rules on /scrape

Take data.extract_rules from the response above and pass it to /scrape on any page with the same layout. Fetch credits only, no model call:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

import requests

rules = data["extract_rules"]  # from the generation response above

response = requests.post(
    "https://ai-web-scraper-api1.p.rapidapi.com/scrape",
    json={"url": "https://shop.example/p/17-tablet", "extract_rules": rules},
    headers={
        "Content-Type": "application/json",
        "X-RapidAPI-Key": "YOUR_API_KEY",
        "X-RapidAPI-Host": "ai-web-scraper-api1.p.rapidapi.com",
    },
)

print(response.json()["data"]["extract"])
# {"title": "17-inch Tablet", "price": "499.00"}

Related Endpoints

• Scrape — Run the generated extract_rules on any page with the same layout, for fetch credits only
• AI Extract — One-off AI extraction from a single page, no rules needed