How to Scrape Product Prices from Websites Using Python: A Complete Guide

Master price monitoring strategies without the headaches of manual tracking. Learn practical web scraping techniques using Python, Scrapy, and BeautifulSoup to extract competitor pricing data at scale—enabling you to implement dynamic pricing strategies, conduct market research, and maintain competitive advantage through automated price intelligence.

So here's the thing about running an online business today—every morning, dozens of new stores launch, and they're all competing for the same customers you're after. You need competitive prices, but how are you supposed to keep track of everyone else's pricing when you've got hundreds of competitors?

You could spend hours manually visiting websites and jotting down prices in a spreadsheet. I've done it. It's soul-crushing work that eats up time you could spend actually growing your business. That's why I learned to automate the whole thing with competitor price scraping.

Let me walk you through how to scrape prices from websites using Python. We'll use Scrapy and BeautifulSoup to pull product prices from Zara, giving you a hands-on understanding of how web scraping works for price monitoring.

Why Even Bother Scraping Prices?

Look, keeping tabs on competitor pricing isn't just nice to have—it's essential for survival. When you automate the process of collecting and analyzing pricing data, you stay informed about what everyone else is doing and make better decisions for your own business.

Dynamic Pricing: Real-time competitor pricing data lets you adjust your prices based on market fluctuations, demand trends, and competitor moves. That's how you maximize profits without guessing.

Market Research and Insights: Price scraping isn't just about knowing what competitors charge. It helps you spot pricing trends in your industry, understand cost structures, and figure out buyer behavior patterns you'd never notice otherwise.

Agility and Adaptability: Markets shift fast, and prices follow. By scraping prices regularly, you stay updated and can react quickly—whether that means adjusting your own prices or pivoting your marketing message.

Of course, before you can analyze anything, you need data. Let's build that dataset.

Scraping Product Prices with Python

For this project, we're building a web scraper to extract price data from Zara's men's shirt collection. While we're focusing on Zara, these same concepts work for any e-commerce site out there.

Project Requirements

Here's what you'll need:

• Scrapy – An all-in-one suite for crawling the web, downloading documents, processing them, and storing data in an accessible format

• BeautifulSoup – A Python library that simplifies parsing HTML content from web pages

Install them using pip:

pip install scrapy beautifulsoup4

When it comes to handling the technical challenges of web scraping at scale—like bypassing anti-bot systems and managing JavaScript rendering—you'll want reliable infrastructure that just works. That's where professional scraping tools become invaluable for serious price monitoring operations.

👉 Get the infrastructure you need to scrape prices reliably without getting blocked

You'll also want to set up a ScraperAPI account to handle potential blocking issues. The free tier gives you 5,000 API credits to get started.

Step 1: Setting Up Scrapy

First, create a new Scrapy project:

scrapy startproject price_scraper

This creates a new directory with all the necessary files. Navigate into it and you'll see this structure:

$ cd price_scraper
$ tree
.
├── price_scraper
│ ├── init.py
│ ├── items.py
│ ├── middlewares.py
│ ├── pipelines.py
│ ├── settings.py
│ └── spiders
│ ├── init.py
└── scrapy.cfg

Your Scrapy project is ready to go.

Step 2: Understanding Zara's Website Structure

Before writing any code, you need to understand the structure of what you're scraping. Open Zara's men's shirts section in your browser, right-click, and select "Inspect" to open Developer Tools.

Look through the HTML source code to find the selectors you need. Specifically, find the elements containing product names and prices.

For Zara, product information sits inside a <div> tag with the class product-grid-product-info. The product name lives in an <h2> tag, and the price sits in a <span> tag with the class money-amount__main.

Step 3: Creating the Spider

Navigate to the spiders directory and create a new file called zara_spider.py:

cd spiders
touch zara_spider.py

Step 4: Defining the ScraperAPI Method

I'll use ScraperAPI to handle potential blocking and JavaScript rendering. At the top of zara_spider.py, import the required libraries:

import scrapy
from urllib.parse import urlencode
from bs4 import BeautifulSoup

Here's the ScraperAPI integration code:

APIKEY = "YOUR_SCRAPERAPI_KEY"

def get_scraperapi_url(url):
payload = {'api_key': APIKEY, 'url': url, 'render': True}
proxy_url = 'http://api.scraperapi.com/?' + urlencode(payload)
return proxy_url

This function takes a URL, adds your ScraperAPI key, and returns a modified URL that routes requests through ScraperAPI. This makes it much harder for websites to detect and block your scraping.

Replace "YOUR_SCRAPERAPI_KEY" with your actual API key.

Step 5: Writing the Main Spider Class

In Scrapy, you create different spider classes to scrape specific pages or groups of sites. Here's the code for our ZaraProductSpider class:

class ZaraProductSpider(scrapy.Spider):
name = "zara_products"

def start_requests(self):

    urls = [

        'https://www.zara.com/ww/en/man-shirts-l737.html?v1=2351464',

        'https://www.zara.com/ww/en/man-shirts-l737.html?v1=2351464&page=2',

        'https://www.zara.com/ww/en/man-shirts-l737.html?v1=2351464&page=3'

    ] 

    for url in urls:

        yield scrapy.Request(url=get_scraperapi_url(url), callback=self.parse)

This class inherits from Scrapy's base Spider class. The name defines what we'll call when running the spider. The start_requests method specifies which URLs to scrape—in this case, the first three pages of men's shirts.

The code iterates through each URL and uses Scrapy's Request object to fetch the HTML. The callback=self.parse argument tells Scrapy to call the parse method to handle the downloaded content.

Step 6: Parsing the HTML Response

Now define the parse method to process the HTML content:

def parse(self, response):
soup = BeautifulSoup(response.body, 'html.parser')
for product in soup.select('div.product-grid-product-info'):
product_name = product.select_one('h2').get_text(strip=True) if product.select_one('h2') else None
price = product.select_one('span.money-amount__main').get_text(strip=True) if product.select_one('span.money-amount__main') else None
yield {
'product_name': product_name,
'price': price,
}

The parse method receives the HTML as a response object. I create a BeautifulSoup object to parse it, then iterate through all <div> elements with the class product-grid-product-info.

For each product, I extract the name from the <h2> tag and the price from the <span> tag. The get_text(strip=True) removes extra whitespace. If a tag isn't found, I set the value to None.

Finally, I yield a dictionary with the product name and price, which lets Scrapy collect and process the data.

Step 7: Running the Spider

Run your spider with this command:

scrapy crawl zara_products

Scrapy will send requests to Zara's website, process the HTML, extract product data based on your logic, and output the data to the console.

Step 8: Saving the Price Data

Viewing data in the console is fine for debugging, but you'll want to save it. Scrapy makes this easy with built-in export support:

scrapy crawl zara_products -o zara_mens_prices.csv

This runs the spider and saves extracted data to zara_mens_prices.csv. The -o option creates a new file and inserts the scraped data.

The Complete Code

Here's everything together:

import scrapy
from urllib.parse import urlencode
from bs4 import BeautifulSoup

def get_scraperapi_url(url):
APIKEY = "YOUR_SCRAPERAPI_KEY"
payload = {'api_key': APIKEY, 'url': url, 'render': True}
proxy_url = 'http://api.scraperapi.com/?' + urlencode(payload)
return proxy_url

class ZaraProductSpider(scrapy.Spider):
name = "zara_products"

def start_requests(self):

    urls = [

        'https://www.zara.com/ww/en/man-shirts-l737.html?v1=2351464',

        'https://www.zara.com/ww/en/man-shirts-l737.html?v1=2351464&page=2',

        'https://www.zara.com/ww/en/man-shirts-l737.html?v1=2351464&page=3'

    ]  

    for url in urls:

        yield scrapy.Request(url=get_scraperapi_url(url), callback=self.parse)

def parse(self, response):

    soup = BeautifulSoup(response.body, 'html.parser')

    for product in soup.select('div.product-grid-product-info'):

        product_name = product.select_one('h2').get_text(strip=True) if product.select_one('h2') else None

        price = product.select_one('span.money-amount__main').get_text(strip=True) if product.select_one('span.money-amount__main') else None

        yield {

            'product_name': product_name,

            'price': price,

        }

Remember to replace YOUR_SCRAPERAPI_KEY with your actual API key.

Now you have a CSV file with product names and prices from three pages of Zara's men's shirts, ready for analysis.

Scraping Prices with Structured Data Endpoints

ScraperAPI's Structured Data Endpoints simplify the entire process by providing structured data in JSON format. Instead of wrestling with HTML parsing and website changes, SDEs deliver structured data that makes extraction faster and less error-prone.

Amazon Product Prices

Bypassing Amazon's anti-bot mechanisms at scale is no joke. You'll run into IP blocking, CAPTCHAs, and constant changes to their product pages. ScraperAPI's Amazon SDEs handle these challenges and deliver clean product data.

Here's how to extract product data from Amazon:

import requests
import json

APIKEY= "YOUR_SCRAPER_API_KEY"
QUERY = "Sauvage Dior"

payload = {'api_key': APIKEY, 'query': QUERY, 'country': 'us'}
r = requests.get('https://api.scraperapi.com/structured/amazon/search', params=payload)

data = r.json()

with open('amazon_results.json', 'w') as json_file:
json.dump(data, json_file, indent=4)

print("Results have been stored in amazon_results.json")

Define your API key and search query, construct the payload, send the request, and if successful, parse the response into a JSON object containing all product information including prices. Save it to amazon_results.json.

Walmart Product Prices

Like Amazon, Walmart presents challenges for traditional scraping because of bot detection and changing website structure. ScraperAPI's Walmart API bypasses these obstacles.

Here's a basic example:

import requests
import json

APIKEY= "YOUR_SCRAPER_API_KEY"
QUERY = "Sauvage Dior"

payload = {'api_key': APIKEY, 'query': QUERY, 'page': '2'}
r = requests.get('https://api.scraperapi.com/structured/walmart/search', params=payload)

data = r.json()

with open('walmart_results.json', 'w') as json_file:
json.dump(data, json_file, indent=4)

print("Results have been stored in walmart_results.json")

Google Shopping Product Prices

Google Shopping's dynamic content loading and anti-scraping mechanisms make it tricky with traditional methods. The Google Shopping API simplifies this:

import requests
import json

APIKEY= "YOUR_SCRAPER_API_KEY"
QUERY = "Chop sticks"

payload = {'api_key': APIKEY, 'query': QUERY, 'country_code': 'jp'}
r = requests.get('https://api.scraperapi.com/structured/google/shopping', params=payload)

data = r.json()

with open('google_results.json', 'w') as json_file:
json.dump(data, json_file, indent=4)

print("Results have been stored in google_results.json")

Automating Price Scraping with DataPipeline

If you want to automate the entire scraping process without writing or maintaining code, ScraperAPI offers DataPipeline—a hosted scraper with a visual interface.

Log into your ScraperAPI dashboard and click "Create a new DataPipeline project" at the top. Choose the Amazon Search template to get started.

Provide a list of search terms to scrape—up to 10,000 per project. Enter them directly, upload a text file, or use a Webhook for dynamic lists.

Customize your project by enabling different parameters for geotargeting, data delivery options (download or Webhook, JSON or CSV), scraping frequency (one-time or custom intervals), and notification preferences.

Click "Review & Start Scraping" when everything's set. The tool shows estimated credits used per run for transparency.

After clicking "Start Scraping," you'll land on the project dashboard where you can monitor performance, cancel running jobs, and review configurations. Download results after every run—previous data stays accessible.

That's it. You're now ready to scrape thousands of product prices automatically.

Wrapping Up

Throughout this guide, you've learned practical techniques for scraping product prices from e-commerce sites. Whether you're monitoring competitor pricing for dynamic pricing strategies, conducting market research, or staying agile in fast-moving markets, automated price scraping gives you the data advantage you need.

The methods covered here—from building custom Scrapy spiders to leveraging structured data endpoints—provide flexibility for any scale of operation. For businesses serious about price intelligence, combining these techniques with reliable infrastructure removes the technical headaches and lets you focus on strategic decisions.

👉 Start monitoring competitor prices at scale with tools built for reliable data extraction