How to Scrape Indeed.com: Complete 2025 Guide for Job Data Collection

Extract real-time job listings, market trends, and hiring insights from Indeed.com using Python. Learn efficient scraping techniques that parse embedded JSON data, handle search parameters, and avoid blocking—no headless browsers required.

I'm going to be honest with you: when I first looked at scraping Indeed, I thought it'd be complicated. You know, dealing with anti-bot systems, parsing messy HTML, maybe even renting a server farm. Turns out, Indeed made things surprisingly straightforward—almost like they're okay with you collecting their public data, as long as you're not being rude about it.

Why Bother Scraping Indeed?

Think about it. The# Best Web Scraping APIs for 2025: Speed, Structure, and Real Cost

Most scraping APIs promise unblocked access and CAPTCHA handling. What you actually get is raw HTML with a 200 status code—then you're stuck parsing, cleaning, and debugging the mess yourself.

We tested seven popular APIs against identical targets, measuring latency at scale, baseline costs, and the quality of what actually comes back. The winners didn't just scrape. They returned clean, structured data that worked the first time. In this field, boring reliability beats flashy promises.

---

## How We Tested

We fired 1,000 requests at each service during the same time window, hitting identical websites. Here's what we measured:

- **Latency:** P50, P75, and P95 response times—the actual round-trip for a working response

- **Response Quality:** Success rates were all 99-100%, so we focused on output structure (clean JSON vs. raw HTML dump)

- **Baseline CPM:** Cost per 1,000 standard requests, no JS rendering or premium proxies

- **Developer Experience:** Documentation clarity, auth setup, SDK availability

## What the Numbers Say

**HasData** offers AI-powered parsing that turns messy HTML into predictable JSON. It's built to scale from prototype to production pipeline without rewriting your code. **Bright Data** gives you surgical proxy control when you need it, but at enterprise pricing.

| API | CPM* | P50 (s) | P75 (s) | P95 (s) | Output | Notes |

|-----|------|---------|---------|---------|--------|-------|

| HasData | $0.08 | 2.873 | 3.785 | 4.34 | html, text, markdown, json | Clean JSON, retry-safe, LLM-ready, lowest CPM |

| Bright Data | $0.79 | 4.256 | 5.148 | 5.559 | html | Strong proxy infra, HTML-only output needs extra work |

| ScraperAPI | $0.1 | 9.066 | 10.574 | 12.57 | markdown, text | Slow with inconsistent latency |

| Apify | $7 | 16.788 | 30.537 | 34 | JSON, markdown, text | Actor ecosystem ultra-flexible but painfully slow |

| Oxylabs | $1.25 | 2.715 | 3.879 | 4.301 | json, text | Stable, flexible proxies, supports JSON, higher cost |

| ScrapingBee | $0.07 | 2.177 | 2.394 | 3.284 | html, json | Reliable, fast, works well with LLM pipelines |

| Zyte | $0.11 | 1.429 | 1.738 | 2.25 | html, json | Fastest tested, JSON+HTML, LLM-ready output |

*CPM = baseline cost for 1,000 standard requests. JS rendering, screenshots, or residential proxies cost extra.*

---

## HasData

HasData is a web scraping API that returns structured data instead of raw HTML. It uses AI to transform messy pages into clean JSON, eliminating the parsing step entirely. The API supports Python and Node.js SDKs, with parameters for extraction rules, JS scenarios, and CSS selectors. Responses include full metadata, request IDs, and optional screenshots.

Performance held steady in testing: P95 latency under 4.5 seconds. Output is LLM-friendly by default—clean JSON, markdown, or raw text and HTML.

Pricing uses a credit system starting at $49/month (roughly $0.25 CPM baseline). Enterprise plans drop that to $0.08 CPM at scale. The service has high user ratings on Trustpilot and offers support via chat, email, and Discord.

Best for developers who need fast, scalable data extraction with minimal post-processing for production apps or LLM pipelines.

---

## Bright Data

Bright Data built its scraping stack on top of its massive proxy network. Instead of a simple endpoint, you're using its Browser API—remote control for headless browsers like Playwright or Puppeteer. This gives you flexibility, but you're writing all the extraction and cleanup logic yourself. The API returns raw HTML with no parsing layer.

Performance is solid and built for scale. Our tests showed P50 latency at 4.25s, P95 at 5.55s. The enterprise focus shows in pricing: $499/month minimum for subscriptions. Support is responsive, and enterprise clients get dedicated account managers.

When you need deep infrastructure for complex scraping workflows and want to avoid managing proxy infrastructure yourself, 👉 [explore how modern APIs like ScraperAPI handle the heavy lifting with built-in rotation and retry logic](https://www.scraperapi.com/?fp_ref=coupons), or choose Bright Data if you need manual control and have the budget for it.

Best for teams that need granular control at enterprise scale, with the engineering capacity to build their own extraction layer.

---

## ScraperAPI

ScraperAPI offers straightforward API key access with official SDKs for Python, Node, Java, Ruby, and PHP. It includes standard features: JS rendering, geo-targeting, CAPTCHA handling. The platform supports scheduled jobs and webhooks, with screenshot capture available for an extra fee.

Output is raw HTML by default—no cleanup. You handle parsing and filtering of ads, base64 images, and other noise. Structured JSON exists but only via specialized endpoints. Performance was moderate in testing: median latency around 9s, P95 near 12.5s. Scalability depends on concurrency limits in your pricing tier.

Pricing starts at $49/month, credit-based. Cost per request varies by target complexity. The service has strong ratings on Trustpilot and Capterra, with support through in-dashboard chat.

Best for teams needing flexible endpoint control and global proxy support, who don't mind post-processing raw HTML.

---

## Apify

Apify runs serverless automation scripts called "Actors" instead of functioning as a direct scraping API. This Actor model offers deep control—concurrency, proxy rotation, custom code injection via hooks. The marketplace of pre-built Actors adds flexibility but introduces a learning curve steeper than a REST API.

Performance reflects this architecture: slow for single calls (P95 over 30s) because each Actor spins up an isolated container. Output is clean JSON by default, though other formats require using the platform UI.

Pricing combines subscriptions with usage-based costs for "Compute Units," proxy traffic, and storage. Our baseline test ran about $7 per 1,000 requests.

Best for workflows needing automation, task scheduling, or reusable components—not for high-speed, low-latency scraping.

---

## Oxylabs

Oxylabs operates as a direct scraping API using HTTP basic auth (username:password format). Official SDKs for Python and Go add convenience, but the core is a traditional REST API.

Performance was fast and consistent: median response at 2.71s, P95 at 4.3s. Responses come as clean JSON with full HTML under the content field. Custom parsing rules can be configured for structured extraction.

Pricing starts at $49/month with a straightforward subscription. Screenshot capture isn't supported, but the API offers deep customization—headers, sessions, base64 payloads, redirect behavior, and more.

Best for scraping workflows that need reliable JS rendering, proxy-level control, and stable performance at scale.

---

## ScrapingBee

ScrapingBee is a developer-focused API using simple key authentication with official Python and Node.js SDKs. Documentation is functional, with example queries and a basic API playground.

It supports JS rendering, CAPTCHA bypass, and various proxy options. Requests can be customized with user agents, cookies, headers, JS scenarios, resource blocking, and wait conditions. Screenshot capture and structured JSON output are available when enabled.

Performance was strong: median latency at 2.18s, P95 at 3.28s. Stability under load was solid, though concurrency limits vary by plan.

Pricing starts at $49/month, credit-based. Feature set is well-balanced for typical scraping needs.

Best for developers looking for a low-cost solution with enough flexibility for most use cases.

---

## Zyte

Zyte offers a job-based scraping API with key authentication and an official Python SDK. Documentation covers advanced use cases well but is less beginner-friendly than competitors. A built-in API Playground allows quick testing and response previews.

The API supports full JS rendering, CAPTCHA handling, proxy rotation, and screenshot capture. Request customization is extensive—headers, sessions, geolocation, device emulation, cookie control. Browser automation includes actions like scrolling and clicking. Screenshots are optional and priced separately.

Performance testing showed consistent results: median latency at 1.429s, P95 at 2.25s. The system scaled well under load.

Pricing begins at $100/month, billed per request.

Best for projects requiring structured data with minimal coding or parsing effort.

---

## What Actually Matters

- **Best Value & Scalability:** HasData—AI parsing, high concurrency, clean JSON output at the lowest CPM

- **Powerful Proxy Infrastructure:** Bright Data and Oxylabs—industry-leading networks, pricier, more setup required

- **Complex Automation:** Apify—highly flexible for chained jobs via Actors, but slow and expensive for direct API calls

---

## Choose Based on Your Bottleneck

Pick HasData if you need production-ready data at scale. You get enterprise-grade speed, high concurrency (up to 1500), and AI parsing without the enterprise price tag.

If raw speed is your only metric, pick Zyte. If you need complex multi-step automation instead of just scraping, look at Apify. Bright Data and Oxylabs are workhorses for projects requiring granular proxy control and substantial budgets.

Your core need defines the right tool. For real-time apps or LLM pipelines, reliable speed and structured JSON matter most. For large-scale analytics, cost at scale and data format become the deciding factors. When you're choosing a web scraping API, 👉 [consider how ScraperAPI's built-in features eliminate common bottlenecks without requiring custom infrastructure](https://www.scraperapi.com/?fp_ref=coupons) and compare that against what you actually need from your scraping stack. job market changes every single day. New positions pop up, old ones disappear, salary ranges shift. If you're tracking hiring trends or building a job aggregator, manually clicking through pages is like trying to empty an ocean with a teaspoon.

Scraping Indeed lets you:

• Get real-time updates on job postings across industries and locations

• Study market trends—which skills are hot, what companies are hiring

• Set up custom alerts without relying on Indeed's own notification system

Plus, let's be real: sometimes you just want the data in a format you can actually work with.

Getting Started (It's Easier Than You Think)

You'll need Python and an HTTP client. I recommend httpx because it supports HTTP/2 and won't get you blocked as easily as older libraries. Install it like this:

bash
pip install httpx

That's it. No fancy setup, no complicated dependencies.

How Indeed Search Actually Works

Here's where it gets interesting. Go to Indeed, type "Python" in the search box, add "Texas" as the location, hit search. Look at the URL—you'll see something like ?q=Python&l=Texas.

That's literally all there is to it. Indeed's search is just URL parameters:

• q = what you're searching for

• l = where you're looking

Want to find remote data science jobs? ?q=data%20science&l=remote. See? Not rocket science.

The Secret: Hidden JSON Data

Now here's the cool part that most people miss. You could parse the HTML with CSS selectors or XPath, spending hours figuring out which div contains what. Or—and this is way smarter—you can grab the JSON data that's already sitting right there in the page source.

Indeed embeds all their job listing data in a JavaScript variable called mosaic-provider-jobcards. It's the same data that powers their frontend, just before it gets rendered into HTML. We're basically intercepting it before it gets dressed up.

Here's how you extract it:

python
import httpx
import json
import re
from typing import List, Dict

async def scrape_search(
client: httpx.AsyncClient,
query: str,
location: str,
page: int = 0
) -> Dict:
"""Scrape Indeed job search results"""
url = "https://www.indeed.com/jobs"
params = {
"q": query,
"l": location,
"start": page * 10
}

response = await client.get(url, params=params)

# Extract the JSON data embedded in the page

match = re.search(

    r'window.mosaic.providerData\["mosaic-provider-jobcards"\]=(\{.+?\});',

    response.text

)

if not match:

    return {"results": [], "total_pages": 0}

data = json.loads(match.group(1))

results = data["metaData"]["mosaicProviderJobCardsModel"]["results"]

total_pages = data["metaData"]["mosaicProviderJobCardsModel"]["tierSummaries"][0]["jobCount"] // 10

return {

    "results": results,

    "total_pages": total_pages

}

When dealing with large-scale job data extraction, you'll want a reliable solution that handles anti-scraping measures automatically. 👉 Extract Indeed job listings effortlessly with enterprise-grade scraping infrastructure that scales with your needs. This approach saves development time and lets you focus on analyzing the data rather than fighting blocks.

Scraping Multiple Pages

One page gives you about 15 job listings. But what if you need hundreds? Thousands? The pagination is baked right into that URL parameter: start=0 for page 1, start=10 for page 2, and so on.

python
async def scrape_all_pages(
client: httpx.AsyncClient,
query: str,
location: str,
max_pages: int = None
) -> List[Dict]:
"""Scrape all search result pages"""

# Get first page to determine total pages

first_page = await scrape_search(client, query, location, 0)

total_pages = min(first_page["total_pages"], max_pages or 100)

all_results = first_page["results"]

# Scrape remaining pages

for page in range(1, total_pages):

    page_data = await scrape_search(client, query, location, page)

    all_results.extend(page_data["results"])

return all_results

The data you get back is surprisingly rich—company names, locations, job titles, salary info, even company ratings. All structured nicely in JSON format.

Getting Full Job Descriptions

Search results give you most of what you need, but sometimes you want the complete job description. Each job has a jobkey field—that's your ticket to the full details.

The full job page also uses embedded JSON, this time in a variable called _initialData:

python
async def scrape_jobs(
client: httpx.AsyncClient,
job_ids: List[str]
) -> List[Dict]:
"""Scrape full job details by job IDs"""
results = []

for job_id in job_ids:

    url = f"https://www.indeed.com/viewjob?jk={job_id}"

    response = await client.get(url)

    # Extract embedded job data

    match = re.search(

        r'window._initialData=(\{.+?\});',

        response.text

    )

    if match:

        data = json.loads(match.group(1))

        results.append(data)

return results

The Blocking Problem (And How to Solve It)

Here's the thing nobody tells you upfront: scrape too fast, and Indeed will notice. They'll start giving you 403 errors or CAPTCHAs. It's not personal—they just don't want their servers hammered.

You need realistic headers. Make your scraper look like a regular browser:

python
HEADERS = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.94 Safari/537.36",
"Accept-Encoding": "gzip, deflate, br",
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,/;q=0.8",
"Accept-Language": "en-US,en;q=0.9",
}

async with httpx.AsyncClient(headers=HEADERS) as client:
# Your scraping code here

For larger operations, consider adding delays between requests. Be respectful. The data's publicly available, but that doesn't mean you should act like you own the place.

Putting It All Together

Here's a complete example that searches for Python jobs in Texas:

python
import asyncio
import httpx
import json

async def main():
HEADERS = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36",
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8",
}

async with httpx.AsyncClient(headers=HEADERS) as client:

    results = await scrape_all_pages(

        client,

        query="python",

        location="texas",

        max_pages=5

    )

    print(f"Found {len(results)} job listings")

    print(json.dumps(results[0], indent=2))

asyncio.run(main())

The output is clean JSON with everything you need—job titles, companies, locations, salaries, requirements. No messy HTML parsing, no fragile CSS selectors that break when Indeed redesigns their site.

Common Questions

Is this legal? Yes. The data's publicly available. You're not bypassing logins or accessing private information. Just don't scrape PII protected by GDPR or hammer their servers like you're running a DDoS attack.

Do I need a headless browser? Nope. That's the beauty of this approach. Indeed's data is right there in the JavaScript variables. No need for Playwright or Selenium—those are overkill here.

Is there an official API? As of 2025, no. Which is why we're doing this in the first place.

Final Thoughts

Scraping Indeed isn't about being clever or sneaky. It's about understanding how modern websites work and using that knowledge efficiently. Indeed's already doing the hard work of structuring their data—we're just collecting it in a format that's useful for analysis.

Be respectful. Scrape at reasonable speeds. Don't republish entire datasets if that's illegal in your jurisdiction. And remember: the goal isn't to break things, it's to gather publicly available information that helps you make better decisions.

The job market moves fast. Now you can keep up with it.

Summary

Indeed.com job scraping is straightforward when you understand the underlying structure. Search via URL parameters (q and l), extract embedded JSON data from JavaScript variables (mosaic-provider-jobcards for searches, _initialData for job details), and parse the structured responses. Use realistic headers, respect rate limits, and 👉 leverage professional scraping infrastructure for scaling beyond basic implementations. This approach handles thousands of job listings efficiently while maintaining compliance with public data access standards.