PHP Web Scraping Tutorial: From First Request to Full Crawler (2026)

Open any PHP web scraping tutorial and you will see the same promise: “build a full crawler.” Then you scroll, and what you actually get is a loop that clicks a “next” button five times. That is not a crawler. That is pagination with extra steps.

A real crawler discovers its own pages. It keeps a queue of URLs it has not visited, a record of the ones it has, and it does not fall over when a request times out at 2am. None of the top-ranking guides build that. They stop at the easy part and call it done.

So this time we are going to actually finish the job. We will start from a single cURL request and end with a crawler that follows links across a whole site, runs requests in parallel, retries failures with backoff, and writes everything to SQLite.

A real PHP crawler needs more than a pagination loop: a link frontier, deduplication, concurrency, retries, and SQLite storage. This tutorial builds all of it in runnable PHP 8, crawling a 1,000-book catalog with up to a 5× concurrency speedup, then shows when a structured API like FlyByAPIs beats maintaining your own scraper.

I run scraping infrastructure for a living. At FlyByAPIs we handle millions of search and product requests a month, and most of the support tickets we get start the same way: someone built a PHP scraper, it worked for a week, then it broke. Usually because they skipped exactly the parts this guide covers.

By the end you will have real, runnable PHP 8 code for every stage, plus an honest answer to the question nobody likes: when does it stop being worth maintaining your own scraper, and when should you just call an API?

Every snippet below is real and runnable. The complete, working code for this tutorial lives in the blog-web-scraping-code repo on GitHub , in the php-web-scraping folder, so you are never copying a half-finished example.

Let me show you.

Is PHP good for web scraping, honestly?

Short answer: yes, for most of what people actually need.

PHP gets dunked on a lot, but it has three things going for it here. cURL is built into the language. There is a real DOM parser in the standard library. And Composer puts battle-tested HTTP clients one command away. You are not fighting the language to get started.

Here is the realistic boundary, because no tutorial should pretend a language is good at everything.

If you are still deciding which language to commit to, I wrote a longer breakdown comparing the best language for web scraping across Python, Node, Go, and Rust. For this guide, the assumption is you already know PHP and want to use it. Good. Let us set it up properly.

Setting up: PHP 8, Composer, and the modern stack

You need three things: PHP 8.1 or newer, the cURL extension (almost always already on), and Composer. Check your version first.

1
2
3
4

php --version
# PHP 8.3.x is ideal. Anything 8.1+ works fine.
php -m | grep curl
# should print: curl

Now create a project and pull in the two libraries that do the heavy lifting.

1
2
3

mkdir php-crawler && cd php-crawler
composer init --no-interaction --name=me/php-crawler
composer require guzzlehttp/guzzle symfony/dom-crawler symfony/css-selector

That is the whole stack. Guzzle for HTTP, Symfony DomCrawler for reading the HTML, and CssSelector so you can use CSS selectors instead of raw XPath.

One more thing before we write code. Pick a practice target that wants to be scraped. We will use books.toscrape.com , a sandbox built specifically for this. It has 1,000 books across 50 pages, stable HTML, and nobody gets hurt. For the JavaScript section later, we will switch to its sibling, quotes.toscrape.com.

Never learn scraping on a site that matters. Break things on the sandbox first.

The libraries this PHP web scraping tutorial relies on

Before we write more code, a quick map of the territory. Most guides dump a list of twelve libraries on you and let you sort it out. You do not need twelve. You need to know which job each one does and which two you will actually reach for.

There are four categories, and a real scraper usually combines one from the first group with one from the second.

For this guide we pair Guzzle with DomCrawler, because that combination teaches you what is actually happening at each step. A framework like Roach PHP is excellent, but it hides the moving parts, and you learn more by assembling them yourself first.

One name you will see in every older guide is Simple HTML DOM Parser. It was useful a decade ago. It is slow, unmaintained, and chokes on malformed markup that DomCrawler handles fine. Leave it in the past.

Your first request: cURL, then why you leave it behind

Every scrape is the same two moves: fetch a page, then pull data out of it. Let us do the fetch the rawest way possible, with cURL, so you understand what is actually happening underneath every library.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

<?php

$ch = curl_init('https://books.toscrape.com/');

curl_setopt_array($ch, [
    CURLOPT_RETURNTRANSFER => true,                  // return the body as a string
    CURLOPT_FOLLOWLOCATION => true,                  // follow redirects
    CURLOPT_TIMEOUT        => 15,                     // give up after 15s
    CURLOPT_USERAGENT      => 'MyCrawler/1.0 (+https://example.com/bot)',
]);

$html = curl_exec($ch);

if (curl_errno($ch)) {
    throw new RuntimeException('Request failed: ' . curl_error($ch));
}

$status = curl_getinfo($ch, CURLINFO_HTTP_CODE);
curl_close($ch);

echo "Status: {$status}\n";
echo "Bytes: " . strlen($html) . "\n";

Run it. You get a 200 status and a few thousand bytes of HTML. That is web scraping at its most basic. No magic.

But look at how much ceremony that is for one request. Now imagine adding retries, headers, cookies, and parallel requests on top of raw cURL. You would be rebuilding a library by hand.

So we do not. We switch to Guzzle, which wraps cURL with a clean API and gives us everything we will need later for free.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

<?php
require 'vendor/autoload.php';

use GuzzleHttp\Client;

$client = new Client([
    'base_uri' => 'https://books.toscrape.com/',
    'timeout'  => 15,
    'headers'  => [
        'User-Agent' => 'MyCrawler/1.0 (+https://example.com/bot)',
    ],
]);

$response = $client->get('catalogue/page-1.html');

echo $response->getStatusCode() . "\n";        // 200
$html = (string) $response->getBody();
echo strlen($html) . " bytes\n";

Same result, a third of the noise. From here on, Guzzle is our HTTP layer.

Parsing HTML: regex is a trap, use the DOM

You have the HTML. Now you need the book titles and prices out of it. Here is where a lot of beginners take a wrong turn.

The wrong turn is regex. You see <h3> tags, you think “I’ll just match those with a pattern.” It works on the first page. Then the site nests a tag differently, or adds a class, and your pattern silently returns garbage. HTML is not a regular language. Parsing it with regex is a classic mistake, and there is a famous Stack Overflow answer about why.

Use the DOM instead. Symfony’s DomCrawler loads the HTML into a real tree and lets you query it with CSS selectors.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

<?php
require 'vendor/autoload.php';

use GuzzleHttp\Client;
use Symfony\Component\DomCrawler\Crawler;

$client = new Client(['base_uri' => 'https://books.toscrape.com/']);
$html   = (string) $client->get('catalogue/page-1.html')->getBody();

$crawler = new Crawler($html, 'https://books.toscrape.com/catalogue/page-1.html');

$books = $crawler->filter('article.product_pod')->each(function (Crawler $node) {
    return [
        'title' => $node->filter('h3 a')->attr('title'),
        'price' => $node->filter('.price_color')->text(),
        'stock' => trim($node->filter('.availability')->text()),
    ];
});

print_r($books);

That gives you a clean array of 20 books from one page, each with a title, price, and stock status. No fragile pattern matching. If the site adds a class to .product_pod, your selector still works.

If you prefer XPath, DomCrawler does that too. $crawler->filterXPath('//article[@class="product_pod"]') gets you the same nodes. CSS is friendlier for most jobs, so we will stick with it.

We now have a scraper: it fetches one page and extracts data. This is exactly where most guides stop. We are barely getting started.

From one page to many: building a real link frontier

Here is the difference between a scraper and a crawler, and it is the whole point of this guide.

A scraper reads a page you hand it. A crawler finds its own pages. It needs three things working together: a queue of URLs to visit (the frontier), a seen set of URLs so it never visits the same page twice, and a depth limit so it does not crawl the entire internet by accident.

Let us build it. DomCrawler has a lovely feature here: if you give it the page’s URL, ->links() returns Link objects that resolve relative URLs to absolute ones for you. No manual URL joining, which is the part everyone gets wrong.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58

<?php
require 'vendor/autoload.php';

use GuzzleHttp\Client;
use Symfony\Component\DomCrawler\Crawler;

$client = new Client([
    'timeout' => 15,
    'headers' => ['User-Agent' => 'MyCrawler/1.0 (+https://example.com/bot)'],
]);

$seed  = 'https://books.toscrape.com/catalogue/page-1.html';
$host  = parse_url($seed, PHP_URL_HOST);

$frontier = new SplQueue();          // URLs waiting to be visited
$frontier->enqueue($seed);

$seen    = [];                        // URLs we have already queued
$results = [];

while (!$frontier->isEmpty()) {
    $url = $frontier->dequeue();

    if (isset($seen[$url])) {
        continue;                     // dedup: never visit twice
    }
    $seen[$url] = true;

    $html    = (string) $client->get($url)->getBody();
    $crawler = new Crawler($html, $url);

    // 1. Extract the data on this page
    foreach ($crawler->filter('article.product_pod') as $node) {
        $book = new Crawler($node);
        $results[] = [
            'title' => $book->filter('h3 a')->attr('title'),
            'price' => $book->filter('.price_color')->text(),
        ];
    }

    // 2. Discover new pages and add them to the frontier
    foreach ($crawler->filter('a')->links() as $link) {
        $next = $link->getUri();

        // stay on the same host, drop fragments, skip what we have seen
        $next = strtok($next, '#');
        if (parse_url($next, PHP_URL_HOST) !== $host) {
            continue;
        }
        if (!isset($seen[$next])) {
            $frontier->enqueue($next);
        }
    }

    echo "Visited {$url} — " . count($results) . " books so far\n";
}

echo "Done. " . count($results) . " books from " . count($seen) . " pages.\n";

Read that loop carefully, because it is the spine of every crawler ever written. Pull a URL off the queue. Skip it if seen. Fetch it. Extract data. Find links. Queue the new ones. Repeat until the queue is empty.

Point it at books.toscrape.com and it will walk all 50 catalogue pages and every product page, deduplicating as it goes, until there is nothing left to visit. That is a crawler. A real one.

There is one problem, though. This crawler fetches pages one at a time. Page two does not start until page one finishes. For 50 pages that is fine. For 50,000, you will be waiting all afternoon. Time to fix that.

Crawling faster: concurrent requests with Guzzle

Scraping is mostly waiting. Your code fires a request, then sits there doing nothing while bytes cross the internet. If you can keep ten requests in flight at once, the whole job finishes roughly ten times faster.

Guzzle has this built in with Pool, which runs a stream of requests at a fixed concurrency. You give it a generator of requests and callbacks for success and failure.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48

<?php
require 'vendor/autoload.php';

use GuzzleHttp\Client;
use GuzzleHttp\Pool;
use GuzzleHttp\Psr7\Request;
use Psr\Http\Message\ResponseInterface;
use Symfony\Component\DomCrawler\Crawler;

$client = new Client([
    'timeout' => 15,
    'headers' => ['User-Agent' => 'MyCrawler/1.0 (+https://example.com/bot)'],
]);

// Build the list of pages up front (50 catalogue pages)
$urls = [];
for ($i = 1; $i <= 50; $i++) {
    $urls[] = "https://books.toscrape.com/catalogue/page-{$i}.html";
}

$results = [];

$requests = function (array $urls) {
    foreach ($urls as $url) {
        yield new Request('GET', $url);
    }
};

$pool = new Pool($client, $requests($urls), [
    'concurrency' => 5,                 // 5 requests in flight at once
    'fulfilled' => function (ResponseInterface $response, int $index) use ($urls, &$results) {
        $crawler = new Crawler((string) $response->getBody(), $urls[$index]);
        foreach ($crawler->filter('article.product_pod') as $node) {
            $book = new Crawler($node);
            $results[] = [
                'title' => $book->filter('h3 a')->attr('title'),
                'price' => $book->filter('.price_color')->text(),
            ];
        }
    },
    'rejected' => function ($reason, int $index) use ($urls) {
        fwrite(STDERR, "Failed {$urls[$index]}: {$reason->getMessage()}\n");
    },
]);

$pool->promise()->wait();

echo count($results) . " books scraped concurrently\n";

Set concurrency to 5 and you have five requests running at once instead of one. On a 50-page crawl that is the difference between a coffee break and a blink.

Now, the important part: do not crank concurrency to 100 because you can. That is how you get rate-limited or banned, and it is rude to the site you are scraping. Five to ten is polite and plenty for most jobs. The number should reflect what the target can handle, not what your laptop can fire off. Inside the pool, that concurrency cap is your throttle. The per-request usleep delay from earlier only applies to the sequential loop, so keep the pool small instead.

This is also where languages start to differ. If you genuinely need thousands of concurrent requests, a Go web scraper built with Colly will out-scale a PHP one, because goroutines make that kind of concurrency cheap. For most jobs, Guzzle’s pool is more than enough.

Going two levels deep: listing pages and detail pages

Real crawls are rarely flat. The listing page gives you a title and a price, but the good stuff (the full description, the UPC, the review count) lives on each item’s own detail page. So you crawl in two passes: collect links from the listing pages, then visit every detail page to extract the full record.

This is exactly where concurrency pays off, because you usually have hundreds of detail pages to fetch. First, gather the detail-page URLs from a listing.

1
2
3
4
5
6
7
8
9

<?php

$listing = (string) $client->get('https://books.toscrape.com/catalogue/page-1.html')->getBody();
$crawler = new Crawler($listing, 'https://books.toscrape.com/catalogue/page-1.html');

// Collect the link to each book's own page
$detailUrls = $crawler->filter('article.product_pod h3 a')->each(
    fn (Crawler $node) => $node->link()->getUri()
);

Now run those detail pages through the Guzzle pool from earlier and pull the richer fields each one exposes.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

<?php

function extractDetail(Crawler $page, string $url): array {
    return [
        'url'         => $url,
        'title'       => $page->filter('.product_main h1')->text(),
        'price'       => $page->filter('.price_color')->first()->text(),
        // The product table holds UPC, tax, availability count, and more
        'upc'         => $page->filter('table.table-striped td')->first()->text(),
        'description' => $page->filter('#product_description ~ p')->count()
            ? $page->filter('#product_description ~ p')->text() : '',
    ];
}

That two-level pattern (listing to detail) is how most serious crawls are shaped, and almost no beginner tutorial shows it. They scrape the listing, get a price, and stop. The detail pages are where the data you actually wanted has been hiding the whole time.

Making it survive: retries and exponential backoff

Here is a truth nobody puts in the intro: on a long crawl, requests fail. Connections drop. Servers return a 503 because they are briefly overloaded. The network hiccups. If one failure kills your whole crawl after two hours, you will lose your mind.

The fix is retries with backoff. Try again, but wait longer each time so you are not hammering a struggling server. Guzzle handles this with a middleware, so you set it up once and every request inherits it.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42

<?php
require 'vendor/autoload.php';

use GuzzleHttp\Client;
use GuzzleHttp\HandlerStack;
use GuzzleHttp\Middleware;
use GuzzleHttp\Psr7\Request;
use GuzzleHttp\Exception\ConnectException;
use Psr\Http\Message\RequestInterface;
use Psr\Http\Message\ResponseInterface;

$stack = HandlerStack::create();

$stack->push(Middleware::retry(
    // Decide whether to retry
    function (int $retries, RequestInterface $request, ?ResponseInterface $response = null, ?\Throwable $e = null): bool {
        if ($retries >= 3) {
            return false;                        // give up after 3 tries
        }
        if ($e instanceof ConnectException) {
            return true;                         // network error, retry
        }
        if ($response && $response->getStatusCode() >= 500) {
            return true;                         // server error, retry
        }
        return false;
    },
    // How long to wait, in milliseconds: 1s, 2s, 4s
    function (int $retries): int {
        return 1000 * (2 ** $retries);
    }
));

$client = new Client([
    'handler' => $stack,
    'timeout' => 15,
    'headers' => ['User-Agent' => 'MyCrawler/1.0 (+https://example.com/bot)'],
]);

// Every request through this client now retries automatically
$response = $client->send(new Request('GET', 'https://books.toscrape.com/'));
echo $response->getStatusCode() . "\n";

That waits one second after the first failure, two after the second, four after the third, then gives up. Exponential backoff. It is the single biggest reliability win you can add to a crawler, and almost no tutorial bothers to include it.

Wrap your extraction in a try/catch too, so one malformed page does not crash the run. Log the URL, skip it, keep going. A crawler that finishes with 998 of 1,000 pages beats one that dies at page 3.

Storing what you scrape: SQLite, not a pile of CSVs

Every other guide ends with fputcsv() and calls it storage. CSV is fine as a final export. It is a terrible working store for a crawler.

Why? Because a crawler that runs for hours needs to deduplicate, resume after a crash, and update records when it re-crawls. CSV does none of that. SQLite does all of it, it is built into PHP through PDO, and it is just a single file. No server to run.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

<?php

$pdo = new PDO('sqlite:books.db');
$pdo->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);

$pdo->exec('
    CREATE TABLE IF NOT EXISTS books (
        url        TEXT PRIMARY KEY,
        title      TEXT NOT NULL,
        price      REAL,
        scraped_at TEXT NOT NULL
    )
');

// INSERT OR REPLACE = upsert. Re-crawling updates the row instead of duplicating it.
$stmt = $pdo->prepare('
    INSERT OR REPLACE INTO books (url, title, price, scraped_at)
    VALUES (:url, :title, :price, :scraped_at)
');

function saveBook(PDOStatement $stmt, array $book): void {
    $stmt->execute([
        ':url'        => $book['url'],
        ':title'      => $book['title'],
        // "£51.77" -> 51.77
        ':price'      => (float) preg_replace('/[^0-9.]/', '', $book['price']),
        ':scraped_at' => date('c'),
    ]);
}

The url primary key plus INSERT OR REPLACE gives you deduplication for free. Scrape the same book twice and you get one up-to-date row, not two. Crash halfway and your data is already on disk, so you can resume from where you stopped.

When the crawl is done and you want a spreadsheet, export from SQLite to CSV in one query. If your end goal is Excel specifically, I walk through that path in scraping website data to Excel . The point stands: crawl into a database, export at the end.

Cleaning the data before you store it

Raw scraped data is messy. Prices arrive as "£51.77", titles come padded with newlines and tabs, and half the time a field you expected is just missing. Store that as-is and every query downstream becomes a fight. Clean it at the point of extraction instead.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

<?php

function clean(array $raw): array {
    return [
        // "£51.77" -> 51.77 (a real number you can sort and sum)
        'price' => (float) preg_replace('/[^0-9.]/', '', $raw['price'] ?? ''),

        // collapse runs of whitespace, trim the ends
        'title' => trim(preg_replace('/\s+/', ' ', $raw['title'] ?? '')),

        // a sane default beats a null that crashes a later query
        'stock' => str_contains($raw['stock'] ?? '', 'In stock') ? 'in_stock' : 'unknown',
    ];
}

Three small habits save you hours later. Cast numbers to actual numbers so you can do math on them. Normalize whitespace so "Book Title\n" and "Book Title" are not treated as different values. And give missing fields a defined default instead of letting a stray null blow up a report three steps downstream.

Running it on a schedule: cron and incremental crawls

A scraper you run by hand is a script. A scraper that runs itself on a schedule is a data pipeline. The jump between them is small, and it is where a crawler starts earning its keep.

On any Unix box, cron handles the scheduling. One line runs your crawler every morning at 6am.

1
2

# crontab -e
0 6 * * * /usr/bin/php /home/me/php-crawler/crawl.php >> /var/log/crawl.log 2>&1

The smarter move is an incremental crawl. On the first run you scrape everything. On every run after, you only fetch pages you have not seen recently, using the scraped_at column you already store. No point re-downloading 1,000 pages to find the 12 that changed.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

<?php

// Skip pages scraped in the last 24 hours
function needsRefresh(PDO $db, string $url): bool {
    $stmt = $db->prepare('SELECT scraped_at FROM pages WHERE url = ?');
    $stmt->execute([$url]);
    $last = $stmt->fetchColumn();

    if ($last === false) {
        return true;                       // never seen, crawl it
    }
    return strtotime($last) < strtotime('-24 hours');
}

That one check turns a brute-force re-crawl into a polite, fast update. Your crawler does less work, the target site sees fewer requests, and your data stays fresh. Everyone wins, which is rare.

Logging in: cookies, sessions, and forms

Plenty of the data worth scraping sits behind a login. A dashboard, a members-only listing, a price you only see when signed in. To reach it, your crawler has to log in like a browser does: post the form, receive a session cookie, then send that cookie on every request after.

Guzzle makes this painless with a cookie jar. Turn it on and Guzzle remembers cookies across requests automatically, the same way your browser does.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

<?php
require 'vendor/autoload.php';

use GuzzleHttp\Client;
use GuzzleHttp\Cookie\CookieJar;

$jar = new CookieJar();

$client = new Client([
    'base_uri' => 'https://quotes.toscrape.com/',
    'cookies'  => $jar,                 // store and resend cookies automatically
    'headers'  => ['User-Agent' => 'MyCrawler/1.0'],
]);

// 1. Many login forms carry a CSRF token. Grab it from the login page first.
use Symfony\Component\DomCrawler\Crawler;

$loginPage = (string) $client->get('login')->getBody();
$token = (new Crawler($loginPage))
    ->filter('input[name="csrf_token"]')
    ->attr('value');

// 2. Post the credentials. The session cookie lands in the jar.
$client->post('login', [
    'form_params' => [
        'csrf_token' => $token,
        'username'   => 'admin',
        'password'   => 'admin',
    ],
]);

// 3. Every request from here is authenticated, no extra work.
$dashboard = (string) $client->get('/')->getBody();
echo str_contains($dashboard, 'Logout') ? "Logged in\n" : "Login failed\n";

Three steps, and they map to what a human does. Load the form, read the hidden CSRF token, submit your credentials. After that the jar carries the session cookie on every call, so the rest of your crawler does not even know it is authenticated.

A word of caution, because this is where scraping gets legally and ethically grey. Scraping data behind a login you control is one thing. Scraping behind someone else’s login, in violation of their terms, is another. Stay on the right side of that line.

Scraping responsibly: robots.txt and rate limits

Before you crawl anything you do not own, check its robots.txt. It is the site telling you, in plain text, which paths it does and does not want bots touching. Ignoring it is both rude and, in some places, legally risky.

You do not need a library for the basics. PHP can read and respect it in a few lines.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

<?php

function isAllowed(string $url, string $userAgent = 'MyCrawler'): bool {
    $robots = @file_get_contents(
        parse_url($url, PHP_URL_SCHEME) . '://' .
        parse_url($url, PHP_URL_HOST) . '/robots.txt'
    );

    if ($robots === false) {
        return true;                    // no robots.txt = no restrictions
    }

    $path = parse_url($url, PHP_URL_PATH) ?: '/';

    foreach (preg_split('/\r?\n/', $robots) as $line) {
        if (stripos($line, 'Disallow:') === 0) {
            $rule = trim(substr($line, 9));
            if ($rule !== '' && str_starts_with($path, $rule)) {
                return false;           // this path is off-limits
            }
        }
    }
    return true;
}

if (!isAllowed('https://books.toscrape.com/catalogue/page-1.html')) {
    exit("robots.txt disallows this path\n");
}

That is a deliberately simple parser. For production crawls across many sites, reach for a proper one that understands user-agent groups and Allow rules, but the principle holds: ask before you take.

Politeness is also pure self-interest. A scraper that hammers a site gets noticed and blocked fast. A scraper that behaves like a considerate visitor often runs for months without anyone caring. Restraint is a feature.

Catching breakage before your users do: selector drift

Here is the failure mode that bites everyone who runs a scraper long enough. It works perfectly for weeks. Then one morning the data is empty, or wrong, and you have no idea since when. The site quietly changed its HTML and your selectors stopped matching. We call it selector drift, and it is the number one reason scrapers rot.

The fix is not cleverness. It is validation. After every extraction, assert that you got something sane. If you did not, fail loudly instead of writing garbage to your database.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

<?php

function extractBook(Crawler $node): array {
    $title = $node->filter('h3 a')->attr('title');
    $price = $node->filter('.price_color')->text();

    // Guardrails: if the shape is wrong, the page changed. Shout about it.
    if ($title === null || trim($title) === '') {
        throw new RuntimeException('Title selector returned nothing — markup may have changed');
    }
    if (!preg_match('/[0-9]/', $price)) {
        throw new RuntimeException("Price looks wrong: '{$price}'");
    }

    return ['title' => $title, 'price' => $price];
}

// At the end of a run, sanity-check the totals too.
$expected = 1000;
if (count($results) < $expected * 0.9) {
    fwrite(STDERR, "WARNING: scraped " . count($results) . ", expected ~{$expected}. Site may have changed.\n");
    // In production: send yourself an alert here.
}

Two layers of defense. Per-record validation catches a broken selector on the spot. A count check at the end catches the subtler case where everything “works” but you scraped half the data you should have.

This is also one of the quiet advantages of using an API for the hard targets. When Google or Amazon reshuffles their HTML, that is our problem to fix on our end, not a 6am alert in your inbox. At FlyByAPIs we run a whole monitoring process for exactly this kind of drift, so the JSON you get back stays the same shape even when the underlying page does not.

That is the full local crawler. Frontier, dedup, concurrency, retries, storage, logins, politeness, and breakage detection. Now we hit the two walls that send most people looking for help.

When the page is empty: scraping JavaScript with Panther

You point your shiny new crawler at a modern site and get back… nothing. Empty divs. No data. What happened?

The page renders its content with JavaScript after it loads. Guzzle, like any plain HTTP client, only sees the raw HTML the server sends. It does not run JavaScript, so all that client-rendered content is invisible to it. This is the single most common “my scraper sees no data” bug.

The fix in PHP is Symfony Panther. It drives a real Chrome instance, waits for the page to render, then hands you the same DomCrawler API you already know.

1
2

composer require symfony/panther
# You also need Chrome/Chromium and a matching chromedriver installed.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

<?php
require 'vendor/autoload.php';

use Symfony\Component\Panther\Client;

$client  = Client::createChromeClient();

// quotes.toscrape.com/js builds its content with JavaScript
$crawler = $client->request('GET', 'https://quotes.toscrape.com/js/');

// Wait until the rendered content actually exists
$client->waitFor('.quote');

$quotes = $crawler->filter('.quote')->each(function ($node) {
    return [
        'text'   => $node->filter('.text')->text(),
        'author' => $node->filter('.author')->text(),
    ];
});

print_r($quotes);
$client->quit();

The key line is waitFor('.quote'). Panther pauses until that selector appears, which is how you handle content that loads a beat after the page does. Without it, you would read the page too early and get nothing, same as Guzzle.

A quick way to check before you reach for Panther: open the page, view source (not “inspect,” actual view-source), and search for the data. If it is there, Guzzle can get it. If it is not, the page builds it with JavaScript and you need Panther or an API.

Common PHP scraping errors, and how to fix them

You will hit these. Everyone does. Knowing the cause turns an hour of confusion into a one-line fix, so here are the ones that come up again and again.

The encoding one trips up everyone scraping non-English sites. If a page declares a non-UTF-8 charset, your accents turn into question marks. Fix it before you parse, not after.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

<?php

$html = (string) $client->get($url)->getBody();

// Detect and normalise to UTF-8 so DomCrawler reads it correctly
$encoding = mb_detect_encoding($html, ['UTF-8', 'ISO-8859-1', 'Windows-1252'], true);
if ($encoding && $encoding !== 'UTF-8') {
    $html = mb_convert_encoding($html, 'UTF-8', $encoding);
}

$crawler = new Crawler($html, $url);

The memory error is the other classic, and it shows up exactly when your crawler gets serious. If you append every result to a PHP array, a big crawl will eventually blow past the memory limit and die. The cure is the SQLite store from earlier: write each record to disk the moment you have it, and keep almost nothing in memory.

Most of the time, though, the error you are fighting is not in your code at all. It is the site deciding it does not want you there. Which brings us to the wall.

Why you keep getting blocked, and the honest fix

So far we have practiced on sites that want to be scraped. The real web is different. The moment you point a crawler at a site that does not want you there, you meet its defenses.

The basic hygiene buys you a lot, and you should always do it: set a realistic User-Agent, add a small random delay between requests, respect robots.txt, and keep concurrency modest. Here is the polite-delay pattern, by the way.

1
2
3

// Between requests, pause a random fraction of a second.
// Less predictable than a fixed sleep, easier on the target.
usleep(random_int(500_000, 1_500_000)); // 0.5s to 1.5s

When one IP is not enough, the next step is proxies. The idea is simple: route requests through different IP addresses so no single one trips a rate limit. Guzzle takes a proxy per request, so rotating through a pool is a few lines.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

<?php

$proxies = [
    'http://user:pass@proxy1.example.com:8000',
    'http://user:pass@proxy2.example.com:8000',
    'http://user:pass@proxy3.example.com:8000',
];

$response = $client->get($url, [
    'proxy' => $proxies[array_rand($proxies)],   // pick a random proxy per request
]);

That works, on paper. In practice, good residential proxies cost real money, free ones are slow and often poisoned, and you now own a second system to monitor: which proxies are alive, which are banned, which are leaking your real IP. Rotating proxies is a project of its own.

But let us be honest about the ceiling. For small, friendly targets, the hygiene above is enough. For big sites that actively fight scrapers, especially the search engines and large marketplaces, you end up in an arms race: rotating proxies, header rotation, CAPTCHA solving, browser fingerprint spoofing. You can build all of that. The question is whether you should.

This is the wall everyone hits with a homemade crawler. And it is exactly where a different approach starts to make sense.

When to stop building and call an API instead

I am not going to pretend a DIY crawler is always the answer. I run an API company, sure, but I also maintained scrapers for years before that, and I know which jobs are worth the maintenance and which ones quietly eat your weeks.

Here is the honest split.

Take search results as the obvious case. The day your PHP crawler points at Google, you walk straight into CAPTCHAs, layout changes every few weeks, and IP bans. You can fight that, or you can call a Google Search API built for PHP developers that returns clean JSON and absorbs the blocking on its end.

Here is the same crawl-the-search-results job, but through our SERP API instead of a homemade Google scraper . Notice it is the same Guzzle you already learned.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

<?php
require 'vendor/autoload.php';

use GuzzleHttp\Client;

$client = new Client();

$response = $client->get('https://google-serp-search-api.p.rapidapi.com/search', [
    'query' => [
        'q'   => 'php web scraping',
        'num' => 10,
        'gl'  => 'us',
        'hl'  => 'en',
    ],
    'headers' => [
        'X-RapidAPI-Key'  => getenv('RAPIDAPI_KEY'),
        'X-RapidAPI-Host' => 'google-serp-search-api.p.rapidapi.com',
    ],
]);

$data = json_decode((string) $response->getBody(), true)['data'];

foreach ($data['organic_results'] as $result) {
    echo "{$result['position']}. {$result['title']}\n";
    echo "   {$result['link']}\n";
}

No HTML parsing. No proxy pool. No CAPTCHA solving. One request, structured results, position tracking included. The FlyByAPIs PHP-friendly SERP API handles the part that would otherwise consume your maintenance budget.

The same logic applies to other heavily defended targets. Scraping product pages is its own arms race, which is why a structured way to scrape Amazon data without getting blocked tends to beat a homemade product crawler once you care about reliability. Our Amazon scraping API returns prices, ratings, and reviews as JSON across 22 marketplaces, with the IP and anti-bot work handled for you.

And it goes well beyond search and shopping. Whatever data you are after, the pattern repeats: the harder the site fights, the more an API earns its keep.

Every one of those is the same one-request, JSON-back pattern you saw above. You keep the crawler skills from this guide for the friendly targets, and you hand the hostile ones to an API that already solved blocking. That is not giving up. That is spending your time on the part that is actually your product.

Putting the full crawler together

Let us recap what you built, because it is genuinely more than any of the top-ranking guides give you.

A fetch layer that started with raw cURL and graduated to Guzzle. A parser that reads the DOM with CSS selectors instead of fragile patterns. A frontier-based crawler that discovers its own pages and never visits one twice. Concurrency through Guzzle’s pool, retries with exponential backoff, and SQLite storage that survives a crash. Plus Panther for the JavaScript pages and a clear-eyed view of when to stop fighting and call an API.

Here is the local crawler pulled into one small class, so the pieces sit together instead of scattered across snippets. This is roughly what a clean version looks like before you start adding the production extras.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75

<?php
require 'vendor/autoload.php';

use GuzzleHttp\Client;
use GuzzleHttp\HandlerStack;
use GuzzleHttp\Middleware;
use Symfony\Component\DomCrawler\Crawler;

final class Crawler_
{
    private Client $client;
    private SplQueue $frontier;
    private array $seen = [];
    private PDO $db;

    public function __construct(private string $host, int $maxRetries = 3)
    {
        $stack = HandlerStack::create();
        $stack->push(Middleware::retry(
            fn ($r, $req, $res = null, $e = null) =>
                $r < $maxRetries && (($res && $res->getStatusCode() >= 500) || $e !== null),
            fn ($r) => 1000 * (2 ** $r)
        ));

        $this->client   = new Client(['handler' => $stack, 'timeout' => 15,
            'headers' => ['User-Agent' => 'MyCrawler/1.0 (+https://example.com/bot)']]);
        $this->frontier = new SplQueue();
        $this->db       = new PDO('sqlite:crawl.db');
        $this->db->exec('CREATE TABLE IF NOT EXISTS pages (
            url TEXT PRIMARY KEY, title TEXT, scraped_at TEXT)');
    }

    public function crawl(string $seed): void
    {
        $this->frontier->enqueue($seed);

        while (!$this->frontier->isEmpty()) {
            $url = $this->frontier->dequeue();
            if (isset($this->seen[$url])) {
                continue;
            }
            $this->seen[$url] = true;

            try {
                $html    = (string) $this->client->get($url)->getBody();
                $crawler = new Crawler($html, $url);
                $this->save($url, $crawler);
                $this->discover($crawler);
                usleep(random_int(500_000, 1_500_000));   // be polite
            } catch (\Throwable $e) {
                fwrite(STDERR, "Skip {$url}: {$e->getMessage()}\n");
            }
        }
    }

    private function save(string $url, Crawler $crawler): void
    {
        $title = $crawler->filter('title')->count()
            ? trim($crawler->filter('title')->text()) : '';
        $this->db->prepare('INSERT OR REPLACE INTO pages VALUES (?, ?, ?)')
            ->execute([$url, $title, date('c')]);
    }

    private function discover(Crawler $crawler): void
    {
        foreach ($crawler->filter('a')->links() as $link) {
            $next = strtok($link->getUri(), '#');
            if (parse_url($next, PHP_URL_HOST) === $this->host && !isset($this->seen[$next])) {
                $this->frontier->enqueue($next);
            }
        }
    }
}

(new Crawler_('books.toscrape.com'))->crawl('https://books.toscrape.com/');

That is a real crawler. Not a pagination loop wearing a costume. The full runnable version, including the concurrency pool and validation, lives in the blog-web-scraping-code repo so you are never copying half-finished snippets.

If you are weighing PHP against other languages for your next scraping project, the same crawler concepts carry over. I built the equivalent in Python and Node.js if you want to compare the ergonomics, and I benchmarked the managed options in the best web scraping APIs for when DIY stops making sense.

The crawler you built here will serve you well on every site that does not actively fight back. For the ones that do, you now know the line between a fun engineering challenge and a maintenance sinkhole. Knowing where that line sits is most of the skill.

Build the crawler. Learn how the web actually works under the hood. And when you find yourself writing your third proxy-rotation hack to keep up with Google, remember there is a Google search data API for PHP that already lost that fight so you do not have to.

All the code from this tutorial, runnable end to end, is in the blog-web-scraping-code repo under php-web-scraping. Clone it, run composer install, and point it at the sandbox site to watch the whole crawler work.

What are you crawling? If you build something with this, I would genuinely like to hear about it.

Oriol.