Build a Web Scraping Tool with Real-Time Preview Using ToolJet and Scraper API

Ever tried scraping dynamic websites and wished you could see the results immediately instead of waiting to parse through logs? Here's the thing: most web scraping feels like sending letters in the mail—you write your code, run it, wait, and hope something useful comes back. But what if you could watch data flow in as it's being scraped, like a live feed?

That's exactly what happens when you combine ToolJet's visual builder with Scraper API. You click a button, and boom—products, prices, images all populate in a table right in front of you. No jumping between terminals and spreadsheets. No wondering if your script crashed somewhere on page 47.

Why This Approach Actually Works

Traditional scraping with Selenium or Google Colab gets the job done, sure. But you're basically working blind until the script finishes. You write code, run it, stare at your terminal, and cross your fingers.

With ToolJet, you're building a proper interface where data appears as it's collected. Think of it like the difference between batch-downloading files versus streaming a video—one makes you wait, the other shows you what's happening right now.

Here's what you need: ToolJet (it's open-source and low-code) and a Scraper API key. Sign up for ToolJet Cloud or run it locally with Docker. Easy enough.

Setting Up Your Interface

First, create a new app called "TJ Web Scraper" in ToolJet.

Now here's where it gets fun. ToolJet lets you drag and drop components like you're arranging furniture. No fighting with CSS, no div soup.

Drop a Container onto your canvas. Inside it, place an Icon on the left (your logo), then a Text component next to it that says "TJ Web Scraper." On the right side, add another Text component to show your product count as it climbs.

Below all that, add a Table component—this is where scraped products will appear. Finally, stick a Button in the bottom-right corner. Done.

We're using blue (#075ab9) as our primary color because, well, it looks sharp. Style everything accordingly.

The whole UI takes maybe two minutes. No exaggeration.

Writing the Scraping Logic

This is where the magic happens. We're targeting a sample eCommerce site that has products with images, titles, and prices, plus a "Load More" button that dynamically fetches additional content via AJAX.

Open the Query Panel at the bottom of ToolJet and create a new "Run JavaScript code" query. Name it scrapeData.

Start with your main function that orchestrates everything:

javascript
function runMainScript() {
const API_KEY = 'YOUR_SCRAPER_API_KEY';
//Logic goes here
}

Now build a helper function using axios to handle API requests:

javascript
async function makeRequest(url) {
try {
const response = await axios.get('https://api.scraperapi.com/', {
params: {
api_key: API_KEY,
url: url
}
});
return response.data;
} catch (error) {
if (error.response && error.response.status === 404) {
throw new Error('404NotFound');
}
console.error(Error making request to ${url}: ${error});
return null;
}
}

Next, write a function to extract product details from the HTML. This uses the DOM parser to grab titles, prices, and images:

javascript
function parseProducts(html) {
const parser = new DOMParser();
const doc = parser.parseFromString(html, 'text/html');
const items = doc.querySelectorAll('.product-item');
return Array.from(items).map(item => ({
title: item.querySelector('.product-name')?.textContent.trim() || '',
price: item.querySelector('.product-price')?.textContent.trim() || '',
image: item.querySelector('img')?.src || 'N/A',
url: item.querySelector('a')?.href || null
})).filter(item => item.title && item.price);
}

Here's where it gets interesting. The site loads content dynamically, so you need to handle both the initial page and subsequent AJAX requests:

javascript
async function fetchProducts(pageUrl, ajaxUrl) {
let products = [];
let offset = 0;
const initialPageHtml = await makeRequest(pageUrl);

if (!initialPageHtml) return products;

products = products.concat(parseProducts(initialPageHtml));

while (true) {

    const ajaxHtml = await makeRequest(`${ajaxUrl}?offset=${offset}`);

    if (!ajaxHtml) break;

    const newProducts = parseProducts(ajaxHtml);

    if (newProducts.length === 0) break;

    products = products.concat(newProducts);

    offset += 12;

    console.log(`Scraped ${products.length} products so far...`);

    actions.setVariable('totalProductsScraped', products.length);

}

return products;

}

Notice that actions.setVariable() line? That's ToolJet storing the count in real-time. Your UI updates as products roll in.

When you're dealing with complex scraping workflows that require handling dynamic content, rate limits, and rotating proxies automatically, 👉 managing web scraping infrastructure becomes dramatically simpler with the right API solution. Instead of maintaining proxy pools and handling CAPTCHAs yourself, you can focus on extracting the data you actually need.

Now trigger the scraping:

javascript
async function scrapeProducts() {
const pageUrl = "https://www.scrapingcourse.com/button-click";
const ajaxUrl = "https://www.scrapingcourse.com/ajax/products";
let products = await fetchProducts(pageUrl, ajaxUrl);
console.log(\nTotal products scraped: ${products.length});
return products;
}

Finally, run everything and store the results:

javascript
scrapeProducts().then(products => {
actions.setVariable('scrapedProducts', products);
console.log("\nScraped products stored in 'scrapedProducts' variable.");
console.log(Total products: ${products.length});
console.log("\nSample of scraped products:");
products.slice(0, 5).forEach(product => {
console.log(Title: ${product.title});
console.log(Price: ${product.price});
console.log(Image: ${product.image});
console.log(URL: ${product.url});
console.log("---");
});
}).catch(error => {
actions.setVariable('scrapingError', error.message);
console.error("An error occurred:", error);
});

Call runMainScript() at the end to kick off the whole process.

Note: You'll need to adjust those HTML selectors if you're scraping a different website. Each site structures its HTML differently.

Connecting Everything Together

Now for the satisfying part—making it all work.

Select your Button component and add an event handler. Set it to run the scrapeData query on click.

Select the Table component and set its Data property to {{variables.scrapedProducts}}.

For the product count Text component in the header, set its Data property to {{"Total Products Scraped: " + variables.totalProductsScraped || 0}}.

That's it. Click the button and watch the table fill up with scraped data in real-time. The count updates as products stream in. It's oddly satisfying.

Why This Setup Actually Matters

Scraping isn't just about getting data—it's about getting it efficiently and being able to see what's happening. Traditional methods leave you flying blind until everything completes (or crashes).

This ToolJet approach lets you build a proper interface in minutes, write clean JavaScript that handles pagination automatically, and watch results populate instantly. No more wondering if your script is stuck or if that AJAX endpoint changed.

For more complex scraping needs involving authentication, JavaScript rendering, or enterprise-scale data extraction, 👉 robust API infrastructure handles the heavy lifting while you focus on data analysis. The difference between maintaining your own scraping infrastructure and using a managed solution is like the difference between hosting your own email server versus using Gmail—both work, but one lets you actually get stuff done.