How to scrape Google Search results in 2027

Extracting structured data from Google search engine results pages (SERPs) has never been more important for competitive research, search engine optimization (SEO) tracking, and powering artificial intelligence (AI) applications. It has also become more challenging to do reliably.

The classic "10 blue links" tell only part of the story. AI Overviews, local map packs, job cards, and Knowledge Graph panels can appear alongside organic results. Meanwhile, consent screens, traffic challenges, and changing page layouts can turn a working Google scraper into an empty dataset.

In this article, you will learn how to scrape Google Search results with Python and prepare your workflow for 2027. We will examine three traditional DIY methods, break down where they become difficult to maintain, and walk through extracting structured search data using Google Search Scraper on the Apify platform. You will also learn how to process rich results, call a Standby search endpoint, and compare pay-per-event costs with running your own infrastructure.

TL;DR

• Google SERPs contain more than organic links: AI Overviews, local results, and question panels can make search data useful for both rank tracking and content research. Which sections appear depends on the query and search context.
• DIY scraping has ongoing costs: Direct HTTP requests cannot execute JavaScript, while headless browsers add browser management, resource usage, and selector maintenance.
• Google Custom Search is a migration concern: The Custom Search JSON API is closed to new customers, and existing customers must transition by January 1, 2027. It is not a new integration option for a 2027 project. Google's service notice.
• A managed Actor simplifies delivery: Google Search Scraper turns supported SearchApi.io responses into structured JSON on the Apify platform, with batch and Standby interfaces.
• Understand the billing unit: The updated pricing proposal charges per saved search page, including available requested rich sections. Check the live Pricing tab for event rates, startup charges, and platform usage.

The state of Google SERPs in 2027

To understand why a simple Google results scraper can fail, first look at what you are asking it to collect.

An older tutorial might send an HTTP GET request to google.com/search?q=your+query, parse the HTML with BeautifulSoup or lxml, and select links using classes such as .r > a or div.g. That is a useful introduction to HTML parsing, but it leaves you responsible for recognizing the right page, handling different layouts, and finding the data your application needs.

For a workflow you want to keep using in 2027, three changes deserve particular attention.

1. The rise of AI Overviews and dynamic components

Google Search can present a generated answer alongside links to websites. For a content researcher, that introduces another question: is your competitor merely ranking in organic results, or is its content also cited in an AI Overview?

An AI Overview can include summary text and links to supporting sources. Other sections, such as People Also Ask (PAA), can reveal follow-up questions that a basic list of titles and URLs would miss. Google explains that AI Overviews appear when its systems determine they will be helpful; they are not guaranteed for every informational or commercial query. Google's explanation of AI features in Search.

Some content may be available in the initial HTML, while other details require rendering or interaction. Downloading a page successfully does not prove that you captured everything visible in a browser. Your extraction workflow needs to distinguish an absent section from one it could not collect.

2. Aggressive anti-bot and fingerprinting infrastructure

A search request also has a network and browser context. When an automated request receives different content from an interactive browser, investigate several layers:

• IP and traffic context: Shared networks, request frequency, and simultaneous requests can affect whether you receive results or a traffic challenge.
• Transport Layer Security (TLS) behavior: A Python HTTP client and a browser establish connections differently. Copying a browser's User-Agent header does not make their connection behavior identical.
• HTTP protocol behavior: Headers, redirects, cookies, and connection settings can change what a server returns. These are useful debugging clues, rather than proof of a particular private detection rule.
• Browser and session state: JavaScript execution, consent choices, and session cookies can affect the page you receive and the content available to extract.

Google documents that automated traffic can trigger its unusual-traffic warning. A successful HTTP status alone is therefore not enough: a response may contain a consent page or challenge instead of search results. Validate the page content before recording a scrape as successful, and handle access restrictions rather than repeatedly retrying them.

3. Rapidly shifting and obfuscated DOM structures

The document object model (DOM) describes a page's structure. A selector such as div.g depends on that structure staying consistent, but Google layouts can differ by device, query, region, and experiment.

You may encounter generated class names such as VwiC3b or different containers around a result heading. A selector that extracts the title from one layout can return nothing, duplicate links, or the wrong parent element in another.

This is a particularly frustrating failure mode because your script may finish without an exception. For an SEO pipeline, an empty list should trigger an investigation, not automatically become a report that a competitor has disappeared from Search.

What data can you extract from modern Google SERPs?

Before writing code, it is helpful to inventory the data surfaces available across a modern SERP. The fields you need depend on your business question. These are the eight result categories handled by the Actor, with fields returned where available:

The 3 traditional ways to scrape Google Search (and why they break)

Developers building a Google SERP scraper typically follow one of three paths. Let us analyze each option with Python code to examine where they break in production.

Option 1: Google Custom Search JSON API

Google's Custom Search JSON API returns structured web or image results from a configured Programmable Search Engine. It has often appeared in Python tutorials because it avoids writing an HTML parser.

There is an important availability change: the API is closed to new customers, and Google gives existing customers until January 1, 2027 to transition to an alternative. The following example helps explain an existing integration during that transition; do not use it as the starting point for a new 2027 service. Google's Custom Search documentation.

An existing customer with an API key and search engine ID can recognize the request pattern below. Install requests with pip install requests if needed:

import os
import requests

API_KEY = os.environ["GOOGLE_API_KEY"]
SEARCH_ENGINE_ID = os.environ["GOOGLE_CSE_ID"]

def query_google_custom_search(query: str, start_index: int = 1) -> dict:
    url = "https://www.googleapis.com/customsearch/v1"
    params = {
        "key": API_KEY,
        "cx": SEARCH_ENGINE_ID,
        "q": query,
        "start": start_index,
        "num": 10,
    }

    response = requests.get(url, params=params, timeout=10)
    response.raise_for_status()
    return response.json()

results = query_google_custom_search("best web scraping tools 2027")
for item in results.get("items", []):
    print(f"- {item['title']}: {item['link']}")

Why it breaks in production: Google Custom Search JSON API

The main issue for a new project is availability, followed by whether the returned data answers your question:

• A transition deadline: Existing integrations need a replacement by the announced date. New customers cannot obtain access to this API.
• Quotas during the transition: For eligible existing customers, Google documents 100 free queries per day, then $5 per 1,000 additional queries, up to 10,000 queries per day. Those are legacy service terms, not a new 2027 offer.
• A different search product: Programmable Search Engine results are not a capture of a google.com results page. Do not assume identical rankings or the same AI Overview, local pack, and question-panel coverage.

For an application that searches a defined collection of sites, evaluate a supported search service suited to that task. For an SEO tool observing Google SERPs, evaluate the actual result fields and localization support instead.

Option 2: Direct HTTP requests with BeautifulSoup or Parsel

The second common approach is sending direct HTTP GET requests to google.com/search using an HTTP client like httpx or requests and parsing the resulting HTML with BeautifulSoup or Parsel. To try this small parsing example, install its dependencies with pip install "httpx[http2]" parsel:

import httpx
from parsel import Selector

headers = {
    "User-Agent": (
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
        "AppleWebKit/537.36 (KHTML, like Gecko) "
        "Chrome/130.0.0.0 Safari/537.36"
    ),
    "Accept-Language": "en-US,en;q=0.9",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
}

def scrape_google_serp(query: str) -> list[dict]:
    url = "https://www.google.com/search"
    params = {"q": query, "hl": "en", "gl": "us"}

    with httpx.Client(http2=True, headers=headers, follow_redirects=True, timeout=20.0) as client:
        response = client.get(url, params=params)

        if "unusual traffic" in response.text.lower() or response.status_code == 429:
            raise RuntimeError("Request was blocked by Google anti-bot systems.")

        response.raise_for_status()
        selector = Selector(response.text)
        results = []

        for box in selector.xpath("//a[.//h3]"):
            title = "".join(box.xpath(".//h3//text()").getall()).strip()
            link = box.xpath("./@href").get() or box.xpath(".//a/@href").get()

            if title and link and link.startswith("http"):
                results.append({"title": title, "url": link})

        return results

try:
    data = scrape_google_serp("python web scraping tutorial")
    print(f"Extracted {len(data)} results")
except Exception as exc:
    print(f"Scraping failed: {exc}")

Why it breaks in production: direct HTTP requests

This example illustrates how to collect heading links from a returned document. It is not a complete Google SERP parser, and an empty result is not proof that Google has no matching pages.

• Blocked or unexpected responses: A request can receive a rate limit, challenge, consent screen, or other content. There is no universal number of requests after which every IP is blocked. Inspect the response and stop when access is restricted.
• No JavaScript execution: An HTTP client cannot render or interact with the page. Content loaded after the initial response may be missing, even when some rich results are already present in the HTML.
• Selector maintenance: A heading selector does not reliably distinguish organic results from every other linked heading. Snippets, result positions, local cards, and citation panels need additional parsing and validation.

Direct requests can be useful for learning or a small experiment. The workload grows when you need consistent coverage across hundreds of keywords, different devices, and repeated collection dates.

Option 3: Headless browsers with Playwright or Selenium

To overcome static HTML limitations, many teams turn to headless browser automation using Playwright, Puppeteer, or Selenium. By spinning up a full Chromium instance, the script executes Google's client-side JavaScript, renders fonts, and can interact with the page. Install Playwright with pip install playwright, then install its browser with playwright install chromium:

from urllib.parse import urlencode
from playwright.sync_api import sync_playwright

def scrape_with_playwright(query: str) -> list[dict]:
    with sync_playwright() as p:
        browser = p.chromium.launch(headless=True)
        try:
            context = browser.new_context(
                locale="en-US",
                viewport={"width": 1280, "height": 800},
            )
            page = context.new_page()
            params = urlencode({"q": query, "gl": "us", "hl": "en"})
            response = page.goto(
                "https://www.google.com/search?" + params,
                wait_until="domcontentloaded",
                timeout=30_000,
            )
            if response is None or not response.ok:
                raise RuntimeError("Search navigation did not succeed.")

            # A page can load successfully without containing search results.
            body = page.locator("body").inner_text(timeout=10_000).lower()
            if "sorry/index" in page.url or "unusual traffic" in body:
                raise RuntimeError("Encountered a Google traffic challenge.")
            if "consent.google.com" in page.url:
                raise RuntimeError("Consent is required before collecting results.")

            page.locator("a h3").first.wait_for(timeout=15_000)
            results = []
            for item in page.locator("a h3").all():
                title = item.inner_text().strip()
                link = item.locator("xpath=ancestor::a[1]").get_attribute("href")
                if title and link and link.startswith("http"):
                    results.append({"title": title, "url": link})
            return results
        finally:
            browser.close()

data = scrape_with_playwright("python web scraping tutorial")
print(f"Found {len(data)} heading links with Playwright")

For more background on browser automation, see my comparison of Playwright and WebdriverIO.

Why it breaks in production: headless browsers

A browser gives you JavaScript execution, but it also gives you a browser fleet to operate. The example above still collects heading links, rather than a complete structured SERP.

1. Resource consumption: Browser processes use CPU and memory. The amount depends on the page, browser settings, and concurrency, so measure your workload before choosing a server size.
2. Challenges and session handling: Opening Chromium does not guarantee access. Your workflow still needs to recognize consent screens, rate limits, and challenges, then handle them appropriately.
3. Network and operational costs: Depending on the source and permitted collection method, you may need network infrastructure, monitoring, and retry controls. Include those expenses and failed attempts in your estimate.
4. Selector upkeep: Rendering a widget does not automatically turn it into structured data. You still need extraction logic for AI Overviews, Knowledge Graphs, job listings, and their changing layouts.

If you enjoy building and maintaining that infrastructure, a browser-based approach gives you control. If your goal is to analyze search data or power an application, a managed search-data interface can reduce the work between a query and a useful dataset.

The modern solution: Google Search Scraper on the Apify platform

Instead of building, debugging, and hosting the collection infrastructure yourself, you can use a managed cloud Actor designed for search data. An Actor is an application that runs on the Apify platform and can be called from Apify Console, an API, or your own code.

Google Search Scraper bridges search queries and structured API delivery. The updated implementation calls SearchApi.io, normalizes the supported response fields, and saves batch results to an Apify dataset. It does not run its own Google HTML parser.

Version note: This walkthrough covers the updated Actor interface, which is in private testing at the time of writing. Use an authorized deployment to follow along until that version is available in the Store; older builds and prices can differ.

The updated request flow is:

flowchart TD
    App["Your application or pipeline"]
    Actor["Google Search Scraper on the Apify platform"]
    Provider["SearchApi.io"]
    Search["Google Search data"]
    Output["Batch dataset or Standby JSON response"]
    App --> Actor
    Actor --> Provider
    Provider --> Search
    Search --> Provider
    Provider --> Actor
    Actor --> Output

Why this approach wins in 2027

• Eight result categories in one response: Collect organic results and available AI Overviews, Knowledge Graphs, local results, jobs, events, People Also Ask questions, and related searches. Each rich section depends on what the provider returns for that query.
• No customer proxy setup: When you use the hosted Actor, you do not configure a browser fleet or a proxy rotation pool. SearchApi.io supplies the upstream search-data service.
• Location and device controls: Set a country (gl), interface language (hl), and device. The batch interface also supports city or regional targeting through location, helping you compare like-for-like search observations.
• A clear page-based event: The updated pay-per-event (PPE) proposal uses one saved search page as its custom billing unit. Available requested AI Overview and Knowledge Graph sections are included, rather than charged as separate custom add-ons. Startup and usage charges still depend on the live pricing configuration.
• Batch and Standby modes: Use batch runs for keyword collections and recurring audits, or request a JSON response through the Standby endpoint for an application or AI agent. Response time depends on the upstream request and deployment.

For the hosted Actor, the operator provisions the SearchApi.io credentials. If you deploy your own copy, configure SEARCHAPI_IO_KEY as an environment secret and budget for that provider separately. Your Apify token authenticates calls to Apify; it is not a replacement for the operator's upstream key.

Step-by-step tutorial: scraping Google Search with Python

Let us walk through a complete, hands-on tutorial using the official Apify Python client to extract structured SERP data.

Step 1: Install the Apify client

In your Python terminal or virtual environment, install the apify-client package:

pip install "apify-client>=3.3,<4"

Step 2: Retrieve your Apify API token

1. Log in to Apify Console. If you do not have an account, sign up and check the current free allowance and account pricing before running the example.
2. Open the API & Integrations page in Settings to find your API tokens.
3. Copy your personal API token and set it in your local environment:

export APIFY_TOKEN="your_apify_token_here"

If you use PowerShell on Windows, set it for the current session with:

$env:APIFY_TOKEN = "your_apify_token_here"

Keep the token out of source control and client-side application code.

Step 3: Run the Actor and process results

Create a file named scrape_google.py and add the following Python code. Start with two queries and one page each so you can inspect the output before increasing the volume. For a private deployment, set GOOGLE_SEARCH_ACTOR_ID to its Actor ID or username/actor-name; the default below is the Store identifier.

import os
from apify_client import ApifyClient

# Initialize the Apify client with your API token
client = ApifyClient(os.environ["APIFY_TOKEN"])
actor_id = os.environ.get("GOOGLE_SEARCH_ACTOR_ID", "eunit/google-search-scraper")

# Configure your search parameters
actor_input = {
    # Provide one or more search queries
    "queries": [
        "Apify documentation",
        "best telehealth platforms",
    ],
    # Number of result pages per query; the organic result count can vary
    "max_pages_per_query": 1,
    # Geographic country code and interface language
    "gl": "us",
    "hl": "en",
    # Request desktop, mobile, or tablet search results
    "device": "desktop",
    # SafeSearch filter: "active", "blur", or "off"
    "safe": "blur",
    # Select which rich data surfaces to extract
    "include_ai_overview": True,
    "include_knowledge_graph": True,
    "include_jobs": True,
    "include_events": True,
    "include_local_results": True,
    "include_related_questions": True,
    "include_related_searches": True,
    # Concurrency control
    "max_concurrency": 2,
}

print("Starting Google Search Scraper on the Apify platform...")

# Execute the Actor run and wait for it to complete
run = client.actor(actor_id).call(run_input=actor_input)
if run is None:
    raise RuntimeError("Actor run was not found.")

print(f"Run completed with status: {run.status}")
print(f"Dataset ID: {run.default_dataset_id}")

# Fetch and iterate through the structured items in the default dataset
if run.status != "SUCCEEDED":
    raise RuntimeError(f"Actor run did not succeed: {run.status}. Check its log.")

dataset = client.dataset(run.default_dataset_id)

for item in dataset.iterate_items():
    query_term = item["searchQuery"]["term"]
    if item.get("error"):
        print(f"Search failed for {query_term}: {item['error']}")
        continue
    total_results = item["searchInformation"].get("totalResults")
    if total_results is None:
        total_results = "N/A"

    print("\n" + "=" * 60)
    print(f"Search query: {query_term} (Reported result estimate: {total_results})")
    print("=" * 60)

    # 1. AI Overview
    if item.get("aiOverview"):
        print("\n[AI Overview]")
        ai_text = item["aiOverview"].get("text") or ""
        print(f"{ai_text[:300]}..." if len(ai_text) > 300 else ai_text)
        print("Citations:")
        for ref in item["aiOverview"].get("referenceLinks") or []:
            print(f"  - {ref['title']}: {ref['link']}")

    # 2. Knowledge Graph
    if item.get("knowledgeGraph"):
        kg = item["knowledgeGraph"]
        print(f"\n[Knowledge Graph: {kg.get('title')} ({kg.get('type')})]")
        if kg.get("description"):
            print(f"Description: {kg['description']}")
        if kg.get("website"):
            print(f"Website: {kg['website']}")

    # 3. Organic Results
    print(f"\n[Organic Results: {len(item.get('organicResults', []))} found]")
    for org in item.get("organicResults", []):
        pos = org["position"]
        title = org["title"]
        url = org["url"]
        print(f"  {pos}. {title}")
        print(f"     URL: {url}")
        if org.get("description"):
            print(f"     Snippet: {org['description'][:120]}...")

    # 4. People Also Ask
    if item.get("relatedQuestions"):
        print(f"\n[People Also Ask: {len(item['relatedQuestions'])} questions]")
        for q in item["relatedQuestions"][:3]:
            print(f"  ? {q['question']}")
            if q.get("answer"):
                print(f"    Ans: {q['answer'][:100]}...")

    # 5. Related Searches
    if item.get("relatedSearches"):
        print("\n[Related Searches]")
        related = [r["query"] for r in item["relatedSearches"][:5]]
        print("  " + " | ".join(related))

Step 4: Inspecting the output data

Run the script with python scrape_google.py. After the run succeeds, it reads the default dataset and prints the query, organic results, and any available rich sections.

Here is an excerpt from an actual test for Apify documentation on October 21, 2026. The arrays and organic result fields are shortened for readability; the values below come from that saved observation:

{
  "searchQuery": {
    "term": "Apify documentation",
    "url": "https://www.google.com/search?q=Apify documentation",
    "device": "desktop",
    "page": 1,
    "gl": "us",
    "hl": "en"
  },
  "searchInformation": {
    "queryDisplayed": "Apify documentation",
    "totalResults": 340000,
    "timeTakenDisplayed": 0.25,
    "didYouMean": null,
    "detectedLocation": "95240, California"
  },
  "organicResults": [
    {
      "position": 1,
      "title": "Apify Documentation",
      "url": "https://docs.apify.com/",
      "displayedUrl": "https://docs.apify.com",
      "description": "Apify is all about Actors\u2014a new way to package your code to make it easy to share, integrate, and build upon. Actors on the Apify ..."
    },
    {
      "position": 2,
      "title": "Apify: Marketplace of ready-to-run tools for AI",
      "url": "https://apify.com/",
      "displayedUrl": "https://apify.com",
      "description": "Thousands of tools to automate your business. Get real-time web data, track competitors, generate leads, and integrate your apps and AI agents."
    }
  ],
  "relatedQuestions": [
    {
      "question": "What is Apify mainly used for?",
      "answer": null,
      "link": null,
      "source": null,
      "thumbnail": null
    }
  ],
  "relatedSearches": [
    {
      "query": "Apify documentation examples",
      "link": "https://www.google.com/search?sca_esv=9dd08dafeeb6e81c&gl=us&hl=en&pws=0&sxsrf=APpeQnvn1FNcy-Pkd1swx5pHUqvUPm7Azg:1792623169968&q=Apify+documentation+examples&sa=X&ved=2ahUKEwjc1YvamcyXAxXZHoAAHVM9DpcQ1QJ6BAgqEAE"
    },
    {
      "query": "Apify login",
      "link": "https://www.google.com/search?sca_esv=9dd08dafeeb6e81c&gl=us&hl=en&pws=0&sxsrf=APpeQnvn1FNcy-Pkd1swx5pHUqvUPm7Azg:1792623169968&q=Apify+login&sa=X&ved=2ahUKEwjc1YvamcyXAxXZHoAAHVM9DpcQ1QJ6BAg-EAE"
    }
  ],
  "localResults": [],
  "jobs": [],
  "events": [],
  "aiOverview": null,
  "knowledgeGraph": null,
  "scrapedAt": "2026-10-21T22:52:51.501393+00:00",
  "page": 1,
  "error": null
}

Notice that aiOverview and knowledgeGraph are null in this example. Requesting those sections makes the Actor include them when available; it does not make Google generate them. Similarly, an empty jobs or events array can be normal for a query that is not about jobs or events.

The returned totalResults is a reported estimate, not a count of records in your dataset. Keep searchQuery, page, and scrapedAt with your results so you can compare observations collected under the same settings.

You can export a batch dataset from Apify Console in JSON, CSV, Excel, XML, or HTML table format. JSON is useful for nested citations and site links; a spreadsheet is convenient for sorting rankings or reviewing content ideas. Integrations and completion webhooks can connect the run to your wider workflow.

For recurring SEO monitoring, save the input as a task and schedule it. Each run creates its own dataset, so use that run's dataset ID when importing results into your database or spreadsheet. A saved task does not automatically append every observation to one permanent search-history table.

Standby mode: turning Google into a real-time HTTP search API

Batch scraping is ideal when you need to process hundreds or thousands of keywords for weekly SEO audits. Applications can also need search results on demand - for example, AI agents researching a question, live chat assistants retrieving citations, or retrieval-augmented generation (RAG) pipelines.

Starting a separate batch run for each question adds startup overhead. To avoid repeating that pattern, Google Search Scraper supports Standby mode.

How Standby mode works

In Standby mode, the Apify platform keeps the Actor ready to receive HTTP requests, allowing it to behave like an API server. Instead of starting a new batch job for every query, your application calls the /search endpoint and receives structured JSON.

Open the Actor's Endpoints tab and copy the deployed hostname. Set that base URL as SEARCH_STANDBY_URL in your environment, without appending /search. If the tab is absent, Standby is not enabled for that version. Authenticate with your Apify token in the Authorization: Bearer header, as described in the Apify Standby documentation.

Standby reduces the need to create individual batch runs, but the upstream search still takes time. Benchmark your own query mix before choosing application timeouts or promising a response-time target.

Standby API example in Python

You can call the endpoint with a standard HTTP client. Install httpx with pip install httpx if you did not install it for the earlier example. Keep the API token on your server, rather than exposing it in browser JavaScript.

import os
import httpx

# Copy the base hostname from the Actor's Endpoints tab.
endpoint = os.environ["SEARCH_STANDBY_URL"].rstrip("/") + "/search"
headers = {"Authorization": "Bearer " + os.environ["APIFY_TOKEN"]}

# Standby uses URL parameters, rather than the batch input's queries list.
params = {
    "q": "best telehealth startups Nigeria",
    "gl": "ng",
    "hl": "en",
    "device": "desktop",
    "page": 1,
    "include_ai_overview": "true",
}

with httpx.Client(timeout=45.0) as client:
    response = client.get(endpoint, headers=headers, params=params)
    response.raise_for_status()
    serp = response.json()

    if serp.get("error"):
        raise RuntimeError(f"Search failed: {serp['error']}")

    print(f"Scraped term: {serp['searchQuery']['term']}")
    print(f"Total organic results: {len(serp.get('organicResults', []))}")
    for item in serp.get("organicResults", [])[:3]:
        print(f"[{item['position']}] {item['title']} -> {item['url']}")

    if serp.get("aiOverview"):
        print("AI Overview:", serp["aiOverview"].get("text"))

This interface is useful when an AI agent needs a few current sources for a question, while batch mode is more convenient for a collection you want to export and compare later. Standby also incurs platform usage, so include it in the cost of an application that sends frequent requests.

Cost comparison: DIY infrastructure vs. pay-per-event pricing

When choosing how to scrape Google Search results, compare the cost of obtaining useful data, rather than only the price of one successful request. A DIY workflow has operating and maintenance expenses; a managed Actor has event charges and potentially separate usage charges.

The updated Actor's proposed custom event is search-page-scraped at $0.008 per saved search page, or $8 per 1,000 pages. A page can contain several organic links, so one event is not one individual website result. Available requested AI Overview and Knowledge Graph sections are included in that custom event.

For a workload of 50,000 successfully saved pages per month, compare these components:

The $400 calculation is a custom-event subtotal, not an all-inclusive invoice or an activated Store price. Apify supports PPE configurations that pass platform usage to the customer, and Standby usage is billed in addition to events. Review the actual configuration before running a large workload. Apify's PPE billing documentation.

For the hosted Actor, you do not separately configure a SearchApi.io subscription. If you deploy your own copy, you become the operator and must include that provider's plan, request allowance, and overage costs in your estimate. For an Actor business, event revenue is also different from profit: deduct the platform's applicable share and any seller-paid operating costs before comparing prices.

Start with a representative batch, inspect its output, and review the charges. That gives you a useful basis for forecasting a daily rank tracker, a weekly audit, or an on-demand search application without relying on unmeasured DIY cost ranges.

Frequently asked questions

Is scraping Google Search legal?

There is no single answer for every project or jurisdiction. Public visibility alone does not establish permission for every collection method or downstream use. Review Google's terms, the applicable service and provider terms, and the rules that apply to the data you collect.

For example, storing business rankings raises different questions from building a database of personal information found in search snippets. Define what you need, minimize unnecessary personal data, respect access restrictions, and seek qualified advice where the intended collection or reuse is uncertain.

How does the scraper extract Google AI Overviews?

The updated Actor calls SearchApi.io and normalizes any AI Overview returned in the provider response. It does not independently reconstruct an answer from the organic links.

When available, aiOverview.text holds the returned text, aiOverview.markdown holds formatted content, and aiOverview.referenceLinks contains citation information. Enable include_ai_overview to request the section, then check whether it is present before processing it. Under the updated pricing proposal, an available AI Overview is included in the saved-page event.

For citation research, save the query, language, country, device, and capture time alongside those references. A missing overview in one response does not prove that Google never shows one for that query.

Can I localize queries to specific countries, languages, and cities?

Yes. Use gl for the country, such as us, gb, de, or ng, and hl for the interface language, such as en, es, or fr. Choose desktop, mobile, or tablet through device.

The batch interface also accepts location for city or regional targeting, such as "New York, United States" or "Munich, Germany". Use it to compare a local business's visibility across markets. The Standby example above uses country and language parameters; its current handler does not accept the batch interface's location field.

These controls configure the search request, but they do not guarantee the exact results seen by a particular person. Personalization, timing, and source behavior can still differ. Keep the same settings when comparing rankings over time.

What data export formats are supported?

Batch results are stored in an Apify dataset, which you can export as JSON, CSV, Microsoft Excel (XLSX), XML, or an HTML table. Use JSON when you need nested AI citations, site links, or Knowledge Graph attributes; use a spreadsheet-friendly format for ranking reviews and content planning.

You can also retrieve dataset items through the Apify API or Python client, then load them into your own database. Standby returns JSON directly to the caller. If you need a permanent history of those responses, save them in your application.

Wrapping up

Scraping Google Search results for a 2027 workflow means looking beyond titles and links. Organic rankings, AI citations, local listings, and follow-up questions can answer different business questions, and a useful pipeline preserves both the data and the context in which it was collected.

The three DIY approaches explain the trade-offs: an official API has its own availability and coverage limits, direct HTTP requests leave you responsible for parsing and incomplete pages, and headless browsers add an infrastructure and maintenance workload.

Google Search Scraper on the Apify platform packages supported search data into structured JSON, with batch runs for recurring research and a Standby endpoint for application requests. Start with a small query set, check which rich sections are returned, and use the live pricing to plan a workload you can sustain.

Ready to turn search results into a useful dataset?

1. Create your Apify account and review the current allowance.
2. Open Google Search Scraper in Apify Store, then check the available version, input fields, and Pricing tab.
3. Once you have access to the updated version, enter a few target queries, click Start, and inspect the output before scheduling a larger run.