Puppeteer Infinite Scroller

Puppeteer-Infinite-Scroller provides a simple and efficient solution for scraping data loaded through infinite scrolling on web pages using Puppeteer.

Installation

You can install the package using npm:

1npm install puppeteer-infinite-scroller

Usage

Import the puppeteerInfiniteScroller function from the package and use it to scrape data from infinite scrolling web pages.

1const puppeteer = require("puppeteer");
2const puppeteerInfiniteScroller = require('puppeteer-infinite-scroller');
3
4(async () => {
5  const pageUrl = "https://infiniteajaxscroll.com/examples/blocks/";
6  const browser = await puppeteer.launch();
7  const page = await browser.newPage();
8  await page.setViewport({
9    width: 1200,
10    height: 800,
11  });
12
13  await page.goto(pageUrl);
14  await page.waitForSelector(".blocks .blocks__block");
15
16  const options = {
17    scrollDelay: 1000,      // Milliseconds between scrolls
18    itemCount: 50,          // Number of items to scrape
19    selector: '.blocks .blocks__block',      // CSS selector for items
20    // OR
21    // pageFunction: () => { /* Custom page function for scraping */ }
22  };
23
24  const scrapedData = await puppeteerInfiniteScroller(page, options);
25
26  console.log(scrapedData);
27
28  await browser.close();
29})();

Options

The following options can be configured when using the puppeteerInfiniteScroller function:

• scrollDelay (optional): Milliseconds between scrolls. Default is 1000ms.
• itemCount (optional): Number of items to scrape. Default is 10.
• selector (optional): CSS selector for the items. Either this or pageFunction must be provided.
• pageFunction (optional): Custom function for scraping data from the page. Either this or selector must be provided.

Example with pageFunction option

1const puppeteer = require("puppeteer");
2const puppeteerInfiniteScroller = require('puppeteer-infinite-scroller');
3
4(async () => {
5  const pageUrl = "https://infiniteajaxscroll.com/examples/blocks/";
6  const browser = await puppeteer.launch();
7  const page = await browser.newPage();
8  await page.setViewport({
9    width: 1200,
10    height: 800,
11  });
12
13  await page.goto(pageUrl);
14  await page.waitForSelector(".blocks .blocks__block");
15
16  function extractElements() {
17    const items = [];
18    const extractedElements = document.querySelectorAll(".blocks .blocks__block");
19    for (let element of extractedElements) {
20      items.push({
21        class: element.getAttribute("class"),
22        id: element.getAttribute("id"),
23        tagName: element.tagName,
24      });
25    }
26    return items;
27  }
28
29  const options = {
30    scrollDelay: 1000,      // Milliseconds between scrolls
31    itemCount: 50,          // Number of items to scrape
32    pageFunction: extractElements
33  };
34
35  const scrapedData = await puppeteerInfiniteScroller(page, options);
36
37  console.log(scrapedData);
38
39  await browser.close();
40})();

License This project is licensed under the MIT License - see the LICENSE file for details.