How to Use Scrapy: From Install to First Spider in 20 Minutes (2026)

There’s a specific moment where Scrapy sells itself. You write about fifteen lines of Python, run one command, and watch it crawl a whole website, follow links on its own, and drop a clean CSV on your disk. No threading code, no retry loops, no for url in urls babysitting.

That’s the pitch, and it’s real. Scrapy hands you concurrency, scheduling, retries, and data export for free, which is why it’s been the default Python scraping framework for over a decade. Getting the hang of Scrapy is mostly about trusting that machinery instead of rebuilding it by hand.

Now the honest catch, up front: Scrapy can’t run JavaScript, and it won’t win a fight against serious anti-bot systems. Learn where that wall is and you’ll know exactly when Scrapy is the right tool and when it’s the wrong one.

Scrapy hands you concurrency, retries, and data export for free, running 16 concurrent requests by default, so you can go from pip install to a working spider in about 20 minutes. What it can’t do is run JavaScript or beat serious anti-bot systems, and past that wall a managed extraction API like FlyByAPIs is the off-ramp.

We run web data extraction infrastructure at FlyByAPIs , pushing millions of scraping calls a month across Google, Amazon, and Maps. So I’ve watched Scrapy do beautiful work in production, and I’ve watched teams fight it for weeks on sites it was never going to beat. This guide is the version I’d give a teammate on their first day.

By the end you’ll have installed Scrapy, written a spider that follows links and exports data, seen four real use cases, and understood the exact point where a managed Python web scraping API becomes the smarter move.

What Scrapy actually is (and isn’t)

Before the first pip install, one idea saves you a lot of confusion: Scrapy is a framework, not a parser.

A parser like BeautifulSoup takes HTML you already have and helps you pull data out of it. It doesn’t fetch pages, doesn’t crawl, doesn’t manage anything. Scrapy is the whole machine around that job.

You’ll meet a few Scrapy nouns as we go, so here they are in plain terms:

• Spider: a Python class where you define where to start and how to parse each page.
• Selector: how you pull values out of a page, using CSS or XPath.
• Item: an optional structured shape for your scraped data (a dict works fine to start).
• Pipeline: an optional step that cleans, validates, or stores items after they’re scraped.
• The engine: the part you don’t write, running requests concurrently and handling retries.

That last one is the whole point. The engine is the reason a Scrapy job of 1,000 pages finishes far faster than a naive requests loop that crawls one page at a time.

For targets that block that kind of speed outright, a hosted Python scraping API for Google does the fetching for you, but we’ll get to that. If you want the full framework head-to-head with numbers, we measured it in BeautifulSoup vs Scrapy .

Install Scrapy the right way

Scrapy pulls in a handful of compiled dependencies (Twisted, lxml, cryptography), so do yourself a favor and install it in a virtual environment. Global installs are how you end up with three broken Python setups by Friday.

1
2
3
4
5
6

# Create and activate a virtualenv
python3 -m venv scrapy-env
source scrapy-env/bin/activate        # Windows: scrapy-env\Scripts\activate

# Install Scrapy
pip install scrapy

Confirm it landed:

1
2

scrapy version
# Scrapy 2.13.x

If that prints a version, you’re done. On older machines you might hit a compiler error from lxml or cryptography. When that happens, upgrade pip first with pip install --upgrade pip, and on Linux install your distro’s python3-dev and libxml2-dev packages before retrying.

The anatomy of a Scrapy spider

Every spider is a Python class with three things: a name, a list of start_urls, and a parse method. That’s the whole contract.

1
2
3
4
5
6
7
8
9

import scrapy

class MySpider(scrapy.Spider):
    name = "my_spider"                         # unique id for this spider
    start_urls = ["https://example.com"]       # where crawling begins

    def parse(self, response):                 # called for every downloaded page
        # pull data out of `response` here
        yield {"title": response.css("title::text").get()}

Three things are worth understanding here, because they’re the whole mental model.

• start_urls is where the engine begins. Scrapy fetches each URL and hands the result to parse.
• response is the downloaded page. It carries the HTML, the URL, the status code, and the selector methods you’ll use to extract data.
• yield is how you emit results. Yield a dict and it becomes a scraped item. Yield a request and Scrapy adds it to the crawl queue.

That yield doing double duty is the clever bit. The same keyword hands back data and schedules new pages to visit. Hold that thought, because it’s what makes following links feel effortless later.

Using Scrapy: your first spider in 20 minutes

Let’s make it real. We’ll scrape quotes.toscrape.com , a sandbox the Scrapy team built specifically for this. It’s static HTML, so it’s the honest way to learn the framework before real sites start fighting back.

Step 1: Explore in the Scrapy shell

The Scrapy shell is an interactive playground for testing selectors before you commit them to a spider. This one habit will save you hours, and the official Scrapy documentation goes deep on every command once you want more.

1

scrapy shell "https://quotes.toscrape.com"

Now poke at the page live:

1
2
3
4
5

>>> response.css("span.text::text").get()
'"The world as we have created it is a process of our thinking..."'

>>> response.css("small.author::text").getall()
['Albert Einstein', 'J.K. Rowling', 'Albert Einstein', ...]

Note the two methods: .get() returns the first match as a string, and .getall() returns every match as a list. Get these two right and Scrapy CSS selectors stop being mysterious.

Step 2: Write the spider

Save this as quotes_spider.py. It grabs the quote text, author, and tags from every quote on the page.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

import scrapy

class QuotesSpider(scrapy.Spider):
    name = "quotes"
    start_urls = ["https://quotes.toscrape.com"]

    def parse(self, response):
        for quote in response.css("div.quote"):
            yield {
                "text": quote.css("span.text::text").get(),
                "author": quote.css("small.author::text").get(),
                "tags": quote.css("div.tags a.tag::text").getall(),
            }

Look at the structure. We loop over each div.quote, then run selectors relative to that block. That scoping is why you get one clean record per quote instead of three tangled lists you have to zip back together by hand.

Step 3: Run it

1

scrapy runspider quotes_spider.py -o quotes.json

scrapy runspider executes a single spider file with zero project scaffolding, which is exactly what you want while learning. You’ll see Scrapy’s logs scroll past, then a quotes.json file appears with your data in it. That’s a working spider, in under twenty minutes with no framework ceremony.

Selecting data: CSS selectors vs XPath

Scrapy gives you two selector languages, and you can mix them freely in the same spider.

CSS selectors read like the styles you already know. div.quote span.text::text means “the text inside a span.text inside a div.quote.” Clean and readable for most jobs.

XPath is more powerful and more verbose. It can do things CSS can’t, like selecting an element by the text it contains, or walking back up to a parent node.

My rule of thumb: start with CSS because it’s easier to read, and drop into XPath only when CSS can’t express what you need. If you want the full cheat sheet with benchmarks, we wrote one in CSS selector vs XPath .

Following links and pagination

Here’s where Scrapy earns its keep. Real scraping is rarely one page. It’s a listing with 50 pages, each linking to detail pages, and you want all of it.

We’ll switch to books.toscrape.com , another practice sandbox, this one with a proper catalog. The spider crawls every listing page, follows each book to its detail page, and pulls the full record.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

import scrapy

class BooksSpider(scrapy.Spider):
    name = "books"
    start_urls = ["https://books.toscrape.com/catalogue/page-1.html"]

    def parse(self, response):
        # follow each book on this page to its detail page
        for href in response.css("article.product_pod h3 a::attr(href)").getall():
            yield response.follow(href, self.parse_book)

        # follow pagination to the next listing page
        next_page = response.css("li.next a::attr(href)").get()
        if next_page:
            yield response.follow(next_page, self.parse)

    def parse_book(self, response):
        yield {
            "title": response.css("div.product_main h1::text").get(),
            "price": response.css("p.price_color::text").get(),
            "stock": response.css("p.availability::text").getall()[-1].strip(),
        }

Read what’s happening, because this pattern is 80% of real-world scraping:

• response.follow(href, self.parse_book) yields a new request and says “when it’s downloaded, send it to parse_book.” Two parse methods, two page types.
• The pagination block finds the “next” link and yields a request back to parse itself. That recursion is what walks the whole catalog with no page counter.
• response.follow handles relative URLs for you, so you don’t hand-build absolute links.

One caveat on that parse_book code: the sandbox always has a p.availability element, so getall()[-1] is safe here. On a real site where the field can go missing, guard it or you’ll hit an IndexError.

You never wrote a queue. You never tracked which pages you’d visited. You yielded requests and the engine scheduled all of them, ran them concurrently (up to 8 per domain by default, 16 across domains), and retried the ones that failed.

How to export data with Scrapy

You’ve already seen the shortcut: the -o flag. Scrapy picks the format from the file extension, so you get four formats with zero extra code.

1
2
3
4

scrapy runspider books_spider.py -o books.csv     # CSV
scrapy runspider books_spider.py -o books.json    # JSON array
scrapy runspider books_spider.py -o books.jl      # JSON Lines (one object per line)
scrapy runspider books_spider.py -o books.xml     # XML

For anything beyond a quick dump, define a FEEDS setting inside the spider. It gives you control over encoding, field order, and whether to overwrite or append.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

class BooksSpider(scrapy.Spider):
    name = "books"
    custom_settings = {
        "FEEDS": {
            "books.csv": {
                "format": "csv",
                "encoding": "utf8",
                "overwrite": True,
            },
        },
    }
    # ... rest of the spider

One warning worth its own line: prefer JSON Lines (.jl) for large crawls. A regular .json file has to be one closed array, so if the crawl dies mid-run the array never closes and the file is invalid JSON. JSON Lines writes one independent record per line, so a crash still leaves you every row scraped up to that second.

Being polite: retries, delays, and not getting blocked

By default Scrapy is fast and a little rude. Eight concurrent requests per domain with a default Scrapy user agent is a great way to get rate-limited on a real site. A few settings fix that.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

class BooksSpider(scrapy.Spider):
    name = "books"
    custom_settings = {
        # identify yourself like a real browser
        "USER_AGENT": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) "
                      "AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0 Safari/537.36",
        "DOWNLOAD_DELAY": 1.0,          # wait 1s between requests to the same domain
        "AUTOTHROTTLE_ENABLED": True,   # auto-adjust speed to the server's health
        "RETRY_TIMES": 3,               # retry failed requests up to 3 times
        "ROBOTSTXT_OBEY": True,         # respect the site's robots.txt
    }

Each of these earns its place:

This is also where you should be honest with yourself about the target. Politeness settings keep you welcome on cooperative sites. They do nothing against a site that actively wants to keep scrapers out. When that’s your target, a managed SERP data API sidesteps the fight entirely.

We wrote the full field guide to blocking in why web scrapers get blocked , and it’s the natural next read after this one.

Four real use cases (and where each one breaks)

The quotes-and-books sandboxes are friendly on purpose. Real targets are messier. Here are four jobs people actually use Scrapy for, the pattern each one follows, and the exact wall each one runs into.

1. E-commerce product catalogs

The pattern you already learned: crawl category pages, follow each product to its detail page, pull price, title, rating, and stock. Scrapy’s concurrency makes short work of thousands of SKUs.

Where it breaks: big retailers render prices and reviews with JavaScript, rotate their HTML layout, and block datacenter IPs fast. The moment Amazon shows you a CAPTCHA instead of a price, your spider is done. That’s exactly the job our Amazon product data API exists for, and it’s why teams stop maintaining brittle store scrapers.

2. News and article archives

Crawl a section index, follow each headline to the article, extract title, author, date, and body. Great for media monitoring or building a training corpus. Static news sites are some of the friendliest targets around.

Where it breaks: paywalls, infinite-scroll feeds that load via JavaScript, and consent walls. A spider sees the raw HTML, so an article hidden behind a client-side render is invisible to it.

3. Job listings

Scrape a job board’s search results, follow each posting, pull title, company, location, and salary. The pagination pattern from the books spider maps directly onto this.

Where it breaks: the big boards are aggressively anti-bot and heavily JavaScript-driven. Rather than fight LinkedIn or Indeed’s defenses, most teams pull structured postings from a dedicated jobs search API and skip the blocking game entirely.

4. Search engine and maps data

People point Scrapy at Google results or Google Maps listings all the time. It technically starts working.

Where it breaks: faster than anywhere else. Google is a JavaScript-heavy, CAPTCHA-guarded, layout-shifting target that is engineered to stop exactly this. A raw Scrapy spider against Google gets blocked in minutes, which is why serious SERP work runs through a Google Search API and local data comes from a Google Maps scraper API instead of a homegrown crawler.

See the thread? Scrapy is excellent right up until a site renders in JavaScript or fights back. Every one of those walls is the same wall.

At FlyByAPIs we hit it across Google, Amazon, and Maps, which is exactly why we build data APIs for those targets instead of maintaining spiders against them.

When Scrapy is the wrong tool

I love Scrapy. At FlyByAPIs we still use it internally for the jobs it’s perfect for. But part of being good at this is knowing when to put it down.

Scrapy stops exactly where three problems begin: JavaScript rendering, proxy rotation, and anti-bot detection. You can bolt on fixes for all three.

scrapy-playwright adds a headless browser. A proxy provider adds rotating IPs. But now you’re maintaining a browser fleet and a proxy budget on top of your spider, and the thing you built to save time is eating it.

Here’s the math that decides it. A managed extraction API renders the page, rotates the IP, solves the anti-bot check, and hands you structured JSON. You call one endpoint and get the same data your spider was fighting for.

When staying unblocked costs more engineering time than parsing the data, the web scraping API is simply the cheaper line item.

The off-ramp code for a Google search scraper API is dead simple. No spider, no proxies, no headless browser:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

import os, requests

API_HOST = "google-serp-search-api.p.rapidapi.com"
resp = requests.get(
    f"https://{API_HOST}/search",
    headers={
        "X-RapidAPI-Key": os.environ["RAPIDAPI_KEY"],
        "X-RapidAPI-Host": API_HOST,
    },
    params={"q": "best python scraping library", "gl": "us", "num": 20},
    timeout=30,
)
for item in resp.json().get("organic_results", []):
    print(item["title"], "-", item["link"])

That’s the whole thing. The same idea covers other targets too: pull company records with the Crunchbase data API , or translate scraped multilingual content through the AI translation API . Different endpoints, same “call it and get clean data” pattern.

Where to go from here

Let’s close the loop. That moment at the start, fifteen lines of Python crawling a whole site, is completely real. You’ve now written it twice: a quotes spider and a books crawler that follows links, paginates, and exports clean data.

That’s the core of Scrapy, and you got there in an afternoon.

The rest of your Scrapy journey is just more of the same pattern applied to harder pages, until you hit JavaScript or an anti-bot wall. When that happens, you’ll know it isn’t a skill gap. It’s a tool boundary, and the smart move is a managed scraping API instead of a month of proxy debugging.

Grab the code from this post on GitHub , run the spiders, break them, fix them. That’s how this sticks.

P.S. My real advice: build the spider yourself first, even on a site you know an API could handle. You’ll understand what the API is doing for you, and you’ll respect the boundary when you hit it. Nobody appreciates a good tool until they’ve done the job the hard way once.

Oriol.