paged-request

Simplified requests for paged (paginated) content.

Please consider following this project’s author, Jon Schlinkert, and consider starring the project to show your and support.

Install

Install with npm:

$ npm install --save paged-request

Heads up!

See the release notes for information about changes made in v2.0.

Usage

This library recursively calls needle’s .get method as long as the user-provided next() function returns a string (the next url to get). See an example.

Example

const request = require('paged-request');
request(url, options, next)
  .then(acc => console.log(acc.pages.length))
  .catch(console.error);

Params

url {string} - (required) the initial url to get
options {object} - (optional) options object to pass to needle
next {function} - (required) function that returns the next url to get, a promise or undefined.

`next` function params

url {string} - the original (base) user-provided url
resp {object} - needle response object
acc {object} - accumulator object with the following properties:
- options {object} - user-provided options object
- pages {array} - array of responses
- urls {array} - array of requested urls

The next function should return a string (the next url to get), promise or undefined.

Example

The following example shows how to loop over pages of CSS posts on smashingmagazine.com (an arbitrary example, but they have great content!).

const request = require('paged-request');
async function next(url, resp, acc) {
  // do stuff to check response first if necessary
  const regex = /href="\/.*?\/(\d+)\/"/;
  const num = (regex.exec(resp.data) || [])[1];
  if (num && /^[0-9]+$/.test(num) && +num <= n) {
    // use the "original" url to avoid having to reparse
    // and recreate the url each time
    return `${acc.orig}/page/${num}/`;
  }
}
request('https://www.smashingmagazine.com/category/css', {}, next)
  .then(acc => console.log(acc.pages.length))
  .catch(console.error);

Release notes

v2.0

renamed .hrefs to .urls in response object
now using axios instead of needle. Please see the axios documentation for API information.

About

Contributing

Pull requests and stars are always welcome. For bugs and feature requests, please create an issue.

Please read the contributing guide for advice on opening issues, pull requests, and coding standards.

Running Tests

Running and reviewing unit tests is a great way to get familiarized with a library and its API. You can install dependencies and run tests with the following command:

sh
$ npm install && npm test

Building docs

(This project’s readme.md is generated by verb, please don’t edit the readme directly. Any changes to the readme must be made in the .verb.md readme template.)

To generate the readme, run the following command:

sh
$ npm install -g verbose/verb#dev verb-generate-readme && verb

You might also be interested in these projects:

gists: Methods for working with the GitHub Gist API. Node.js/JavaScript | homepage
github-base: Low-level methods for working with the GitHub API in node.js/JavaScript. | homepage
repos: Tiny wrapper around github-base for getting publicly available information for a repository, or all of… more | homepage

Contributors

Commits	Contributor
12	jonschlinkert
9	doowb
2	whitneyit

Author

Jon Schlinkert

License

This file was generated by verb-generate-readme, v0.8.0, on January 20, 2021.