How to Scrape Data From Google: 3 Methods, Free to Production

Here is an uncomfortable truth: the moment you write a script to pull Google search results, Google starts trying to stop you. Not sometimes. Every time.

I have watched developers spend a full weekend on a scraper that works beautifully on their laptop, then dies in production three days later when Google serves it a CAPTCHA on request number 40. The code was never the hard part. The blocking is.

So this post is the honest version of how to scrape data from Google. Not one magic snippet, but three real methods, ranked from free to production, with the tradeoffs I would tell a friend over coffee.

The short version: there are three ways to scrape data from Google, and they ladder from free to production. A raw Python script blocks around request 40. A managed SERP API like FlyByAPIs skips that fight entirely, returning a parsed page of results as JSON, with a free tier of 100 requests a month.

We run search infrastructure for a living. Our Google Search API processes millions of queries a month, and almost all of that traffic comes from developers who tried the DIY route first and got tired of patching it.

So I am not going to pretend scraping Google yourself is impossible. It is not. I am going to show you exactly where it gets painful, and what each method actually costs, so you can pick the right one on purpose instead of by accident.

By the end you will know which of the three methods fits your project, and roughly how long each one keeps working before it fights back.

Why scrape Google search results at all

Before the how, a quick word on the why. Because the reason you are scraping shapes which method makes sense.

Google’s results page is the single richest map of the public internet. Every query returns ranked organic links, the questions real people ask, the related searches, and often a featured snippet. That is a goldmine if you can get it as structured data instead of a screenshot.

The common jobs I see:

Notice the pattern: none of these are one-time jobs. They all want fresh data, repeatedly. Hold that thought, because it is exactly where the free methods fall apart and the case for an API gets strong.

If your target is a different Google surface, the same logic applies. Local business data lives in Google Maps , and product listings live on Amazon. Pick the source that matches the job. For everything on the classic web results page, read on.

What data actually lives on a results page

Before you scrape anything, it helps to know what is on the page. A modern Google results page is not ten blue links anymore. It is a dozen different blocks, and which method you pick decides how many of them you can reach.

There is more depending on the query: a knowledge panel on the right, a local pack of three map results, shopping cards, and ads bracketing the whole thing. The local pack, worth noting, is powered by the same data you would pull from a dedicated Google Maps scraper , not the web index.

Here is the catch that decides your method. Some of these blocks are plain HTML you can parse. Others are injected by JavaScript after the page loads, which means a simple requests fetch never sees them.

That single fact is why a raw script gets you the organic links but misses half the good stuff, and why a rendered result matters. A structured Google search results API hands you the organic results, People Also Ask, and related searches already separated into fields, no matter how they were rendered.

What makes Google so hard to scrape

Here is the part the tutorials rush past. Google search is one of the most aggressively defended pages on the web. It has to be, because half the internet wants to scrape it.

When you send a request that does not look human, Google notices fast. Three signals give you away, and a raw script fails all three at once.

Beat all three and you still have a moving target. Google changes the HTML of its results page regularly, and it serves different layouts to different regions and devices. The CSS selector that grabbed the title yesterday can return None today.

I wrote a whole separate piece on the mechanics of this, because it deserves it. If you want the deep version, read why web scrapers get blocked . The short version: request rate, proxy quality, and fingerprint. Get any one wrong and you are done.

The thing nobody tells beginners: getting the data once is trivial. Getting it reliably, at volume, forever, is a full-time job. That gap is the whole story of this post.

With that context, let’s walk the three methods.

How to scrape data from Google: the three methods at a glance

Three routes, three very different profiles. Here is the whole decision on one screen before we go deep on each.

Read that table as a ladder, not a menu. Most projects start on rung one to prove the idea, then climb as the data need gets serious. Let’s take them in order.

Method 1: the free, no-code way

If you need a list of results for a dozen queries and you never want to write a line of code, start here. There are two zero-cost tools worth knowing, and both have a hard ceiling you should understand going in.

Browser extensions

Search Chrome or Firefox for a SERP scraper extension and you will find several free ones. You run your search in the browser, click the extension, and it reads the results already rendered on the page into a CSV.

Why it works: the results are loading in a real browser, with your real IP and a real fingerprint. Google sees a human, because there mostly is one. No blocking, no proxies, no code.

Google Sheets and IMPORTXML

Here is the trick most people do not know. Google Sheets can scrape a page with a built-in formula:

1

=IMPORTXML("https://www.google.com/search?q=your+query", "//div//h3")

It fetches the URL and pulls elements matching an XPath, straight into your cells. No script, no server. For a small, occasional job it feels like magic.

And then it stops feeling like magic. Google rate-limits the Sheets fetcher hard, so a batch of formulas quickly returns #N/A or Loading... forever. The XPath breaks whenever Google tweaks its markup. And there is no way to schedule or scale it cleanly.

This is exactly the rung the big tutorials skip. They jump straight to Python. But for a lot of readers, a free extension is genuinely the right answer, and I would rather tell you that than sell you a script you do not need.

Method 2: the DIY Python route (the real dirty job)

Now we roll up our sleeves. This is the method every Google-scraping tutorial teaches, and it is worth doing at least once so you understand what an API is actually saving you from.

If you are new to this stack, our Python web scraping guide covers requests and BeautifulSoup on any site from scratch. Here we go Google-specific, and Google-specific means Google-hard.

The part that looks easy

Fetching a results page starts out looking trivial. A few lines and you have HTML:

1
2
3
4
5

import requests

headers = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) Chrome/120.0"}
resp = requests.get("https://www.google.com/search?q=web+scraping", headers=headers)
print(resp.status_code)

Run it a few times and it works. You feel like a wizard. Then you loop it over 50 keywords, and around request 40 the status code turns into a 429, or the HTML comes back as a “before you continue” consent wall or a CAPTCHA page. Welcome to the real job.

Where it gets dirty

To keep going, you have to impersonate a human convincingly. That means layering on defenses, one painful lesson at a time.

Residential proxies are where the “free” method quietly stops being free. Quality rotating residential IPs run anywhere from $3 to $15 per GB, and a scraping job at real volume chews through gigabytes. Suddenly your zero-cost script has a monthly bill.

Parsing the mess

Say you get a clean response back. Now you parse it. With BeautifulSoup, you locate result blocks and dig out the title and link:

1
2
3
4
5
6
7
8
9

from bs4 import BeautifulSoup

soup = BeautifulSoup(resp.text, "html.parser")
results = []
for block in soup.select("div.g"):
    title = block.select_one("h3")
    link = block.select_one("a")
    if title and link:
        results.append({"title": title.get_text(), "link": link["href"]})

Looks fine. Here is the catch that will ruin your week: div.g and h3 are not stable. Google obfuscates and rotates its class names, and it ships layout changes without warning. The day it does, your selector returns nothing, your job silently collects empty rows, and you find out when someone asks why the dashboard is blank.

When you escalate to a headless browser

When requests keeps hitting CAPTCHAs and consent walls, the next thing most developers reach for is a real browser under automation: Selenium or Playwright. It runs actual Chrome, executes the page’s JavaScript, and looks far more human than a bare HTTP call.

1
2
3
4
5
6
7
8

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    page.goto("https://www.google.com/search?q=web+scraping")
    titles = page.locator("h3").all_inner_texts()
    browser.close()

It does help. A headless browser renders the JavaScript-injected blocks a plain fetch misses, and it carries a more believable fingerprint. But do not mistake it for a fix.

A headless browser moves the wall further back. It does not remove it. You have traded a fast, cheap script for a slow, heavy one that still gets blocked, just later. For a handful of tricky queries that is a fine trade. For thousands, it is a server bill and a babysitting rota.

Want to do the same thing in JavaScript instead? The tradeoffs are identical, just with Puppeteer doing the browser work. I walk through it in the Node.js scraping guide . Different language, same fight with the same three signals.

Field notes: staying unblocked at the DIY level

If you are committed to method 2, these are the habits that keep a Google scraper alive the longest. None of them are optional at real volume, and together they are basically a part-time job.

That last one matters more than it sounds. The worst failure mode is not a crash, it is a scraper that keeps running and quietly returns nothing while everyone assumes the data is fresh. If you would rather not build this whole apparatus, a managed SERP API for Google search results folds all four practices into the service itself. You send a query, it handles the rest.

When DIY breaks: the maintenance math nobody shows you

Let me put a number on the “your time is not free” line, because vague is not persuasive and I promised you honest.

A DIY Google scraper is not a one-time build. It is an ongoing commitment. Here is roughly where the hours and dollars go once you are running it for real.

Now compare that to a developer’s hourly rate. If you spend even four hours a month keeping the scraper alive, plus the proxy bill, you have quietly built something that costs more than a managed API and works less reliably. I did the full breakdown of this in the Google Search API cost piece, including the trap where Google’s own Custom Search bills 100 results as ten separate queries.

This is the moment most teams switch. Not because DIY is impossible, but because it stops being worth it. But before we get to third-party APIs, the obvious question: does Google not have its own?

What about Google’s own API?

Fair question, and worth answering before you reach for a third-party service. Google does offer the Programmable Search Engine and its Custom Search JSON API . On paper it sounds like the clean, official route. In practice it fits almost no real scraping job.

So the official API is real, but it is a narrow tool wearing an official badge, and Google has signaled changes to it that make betting a pipeline on it risky. I dug into the exact numbers, including the looming shutdown, in the Google Search API cost breakdown. For most people, “Google’s own API” is not the shortcut it appears to be, which is why a dedicated SERP service exists at all. Which brings us to the last rung.

Method 3: the production way, a SERP API

A SERP API flips the problem around. Instead of you pretending to be a browser, a service that already handles the proxy rotation, the fingerprinting, and the CAPTCHAs does it for you. You send a query. You get clean JSON. That is the whole idea.

This is what we built with the FlyByAPIs Google Search API . Here is the same job from method 2, minus the misery:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

import requests

url = "https://google-serp-search-api.p.rapidapi.com/search"
params = {"q": "web scraping", "num": 10, "gl": "us", "hl": "en"}
headers = {
    "X-RapidAPI-Key": "YOUR_KEY",
    "X-RapidAPI-Host": "google-serp-search-api.p.rapidapi.com",
}

data = requests.get(url, headers=headers, params=params).json()["data"]
for r in data["organic_results"]:
    print(r["position"], r["title"], r["link"])

No proxies. No header juggling. No selector that breaks next Tuesday. The response comes back structured, with the fields already named.

What you get back

One call returns the whole results page as data, not HTML you have to dissect:

The num parameter is the quiet win here. Set it to 100 and you get up to a hundred organic results back as a single billed request. Google’s own Custom Search charges that same depth as ten paginated queries.

Same data, a fraction of the cost. The full field list lives in the search endpoint docs if you want the reference.

Why country-pinning matters

One detail that trips up DIY scrapers and cheaper APIs alike: location. Google shows different results in Berlin than in Boston. If your proxy exits in the wrong country, your “rankings” are quietly wrong.

The gl parameter routes each request through an IP inside the country you asked for. Ask for German results, get a request from a German IP. That removes the location noise that causes most “why is this data inconsistent” complaints. There is also a separate autocomplete endpoint included in every FlyByAPIs plan, which is how a lot of our users do long-tail keyword discovery without a second tool.

On price, a SERP API is cheaper than the sticker suggests once you count your own hours. FlyByAPIs plans start free at 100 requests a month, then $19.99 for 15,000 requests on Pro, scaling down to $0.50 per 1,000 at volume. I compared that against nine other providers in the cheapest SERP API breakdown if you want to check my math against the field.

A worked example: tracking rankings in a loop

Theory is cheap, so here is the single most common job in practice. You have a list of keywords, and you want to know where your domain ranks for each. With the structured Google results in hand, that is a short loop:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

import requests

url = "https://google-serp-search-api.p.rapidapi.com/search"
headers = {
    "X-RapidAPI-Key": "YOUR_KEY",
    "X-RapidAPI-Host": "google-serp-search-api.p.rapidapi.com",
}
keywords = ["serp api", "scrape google", "rank tracker"]
my_domain = "flybyapis.com"

for kw in keywords:
    data = requests.get(url, headers=headers,
                        params={"q": kw, "num": 10, "gl": "us"}).json()["data"]
    rank = next((r["position"] for r in data["organic_results"]
                 if my_domain in r["domain"]), None)
    print(f"{kw}: {'#' + str(rank) if rank else 'not on page 1'}")

That is the entire thing. No proxy pool, no retry logic, no selector that rots. One keyword is one request and one billed unit, and page one is where almost all the traffic lives anyway. If you need positions 11 to 30, add a start parameter and loop, counting a request per page. Run it on a schedule, write the numbers to a database, and you have a rank tracker that would have taken days to build the DIY way.

Swap the loop body and the same pattern powers keyword research from the People Also Ask field, or competitor monitoring by domain. The Google search results API returns the same clean structure every time, which is exactly what makes it worth building on.

The three methods, side by side

You have seen each one in depth. Here is the full comparison, with the columns that actually decide it.

Look down the maintenance and blocking rows. That is the real difference. Methods 1 and 2 make you the anti-bot system. Method 3 makes that someone else’s problem.

Which method should you actually pick?

No universal answer. It depends on volume, skill, and how much you value your evenings. Here is how I would decide.

The same logic scales to the rest of your data needs, by the way. When the source is not the web results page, we run the same managed, country-pinned model on our other endpoints: Amazon product data , Google Maps business listings , company data from Crunchbase , job postings across boards , and even AI translation for scraped text . One less anti-bot system for you to maintain, whatever you are pulling.

A quick word on legality

I am not a lawyer, so take this as a practitioner’s read, not legal advice. Scraping publicly visible search results is generally treated as legal in the US. Courts have repeatedly found that collecting public data does not violate the Computer Fraud and Abuse Act, a line most clearly drawn in the hiQ v. LinkedIn litigation.

That said, two caveats matter. Scraping Google breaks its Terms of Service, which is a contract issue, not a criminal one, but it is why Google blocks you. And personal or copyrighted data brings its own rules under GDPR and similar laws.

Stick to public, non-personal data, respect rate limits, and you are on solid ground. A managed API also keeps you off Google’s infrastructure directly, which is a cleaner position to be in.

Wrapping up

Remember the developer who lost a weekend to a scraper that died on request 40? The mistake was not the code. It was picking the DIY route for a job that needed production reliability, without knowing that is what he was choosing.

So now you know the whole ladder. A free extension for a quick list. Python if you want to feel exactly why Google is hard, and you have the patience to maintain it. And a Google Search API when the data has to show up, fresh and structured, without you standing over it.

Pick the rung that matches the job, not the one that sounds most impressive. If that turns out to be method three, our real-time SERP API for Google search results has a free tier so you can test the exact code above against your own queries before spending a cent.

What are you building with Google data? If you get stuck choosing between the three, that decision usually comes down to one question: is this a one-time job, or does it need to keep running? Answer that honestly and the method picks itself.

Oriol.