Mangascraper is a package used to scrape mangas. It is a solution to retrieving mangas that do not offer an API. Mangascraper can run either asynchronously, returning a Promise, or synchronously if a callback function is provided.

Table of Contents

1. Installation
   • npm
2. Sources
3. Usage
4. Configuring puppeteer
   • Connecting to an endpoint
   • Using an existing puppeteer package
   • Overriding mangascraper’s puppeteer launch arguments
5. Examples
6. API Reference
7. License

Installation

npm

npm install @specify_/mangascraper

Sources

Currently, mangascraper supports 5 sources, but will support more in the future.

If a supported source uses axios, mangascraper will try to use axios as much as possible to save computer resources. If the network request is blocked by Cloudflare, mangascraper will resort to using puppeteer.

If a supported source uses both axios and puppeteer, it means one or more methods in the source use either axios or puppeteer. For example, Readmng uses puppeteer for search(), but uses axios for getMangaMeta() and getPages

Usage

To start using the package, import a class such as Mangakakalot from the package and use the methods to get mangas from that source.

Here’s an example:

import { Manganato } from '@specify_/mangascraper';
const manganato = new Manganato();
(async () => {
  const mangas = await manganato.search('One Piece');
  const meta = await manganato.getMangaMeta(mangas[0].url);
  console.log(meta.chapters);
})();

which outputs…

[
  {
    name: 'Chapter 1007',
    url: 'https://readmanganato.com/manga-aa951409/chapter-1007',
    views: '730,899',
    uploadDate: 2021-03-12T07:00:00.000Z
  },
  {
    name: 'Chapter 1006',
    url: 'https://readmanganato.com/manga-aa951409/chapter-1006',
    views: '364,964',
    uploadDate: 2021-03-05T07:00:00.000Z
  },
  ... and more items
]

Configuring puppeteer

Connecting to an endpoint

If you already have an existing puppeteer endpoint, mangascraper can connect to that endpoint instead and perform faster concurrent operations.

Mangascraper also includes its own puppeteer launch arguments, and it is recommended to use them for scraping to go smoothly.

import puppeteer from 'puppeteer';
import { initPuppeteer, MangaSee } from '@specify_/mangascraper';
(async () => {
  const browser = await puppeteer.launch({ ...initPuppeteer });
  const endpoint = browser.wsEndpoint();
  browser.disconnect();
  const mangasee = new MangaSee({ puppeteerInstance: { instance: 'endpoint', wsEndpoint: endpoint } });
  const mangas = await mangasee.search('Haikyu!');
})();

Since you are using your own puppeteer package, mangascraper cannot make any modificatins to the browser such as including a proxy.

const browser = await puppeteer.launch();
const mangapark = new MangaPark({
  proxy: { host: '127.0.0.1', port: 8080 },
  puppeteerInstance: { instance: 'custom', browser },
}); // ❌ Mangascraper cannot include proxy
const browser = await puppeteer.launch({ args: ['--proxy-server=127.0.0.1:8080'] });
const mangapark = new MangaPark({ puppeteerInstance: { instance: 'custom', browser } }); // ✔️ Our own browser instance will launch with a proxy

Because mangascraper is connecting to an existing endpoint, you must do all your browser arguments outside of mangascraper. See this for more on this.

Overriding mangascraper’s puppeteer launch arguments

If you want to override the launch arguments mangascraper uses, you can add this to any manga class such as MangaSee as long as you are using the default instance. Any other instance will require you to implement your own or inherit mangascraper’s puppeteer options with initPuppeteer

const mangasee = new MangaSee({ puppeteerInstance: { instance: 'default', launch: { ...myCustomLaunchOptions } } });

If you want to include a proxy, mangascraper will automatically put it into the launch arguments.

const manganato = new Mangahasu({
  proxy: { host: 'proxy_host', port: 8080 },
  puppeteerInstance: { instance: 'default' },
});

Using an existing puppeteer package

By using an existing puppeteer package in your app, this will enable mangascraper to use one browser instead of opening new browsers per operation. In addition, mangascraper will be able to scrape manga concurrently. With this approach, resources will be less intensive on chromium, and it can save you a lot of time if you are handling a lot of scraping operations. This is the best approach if you do not want to connect to an existing endpoint.

However, you must have puppeteer already installed.

This is the most basic setup:

import puppeteer from 'puppeteer';
import { MangaPark, initPuppeteer } from '@specify_/mangascraper';
(async () => {
  const browser = await puppeteer.launch(initPuppeteer);
  const mangapark = new MangaPark({ puppeteerInstance: { instance: 'custom', browser } });
})();

Since you are using your own puppeteer package, mangascraper cannot add any modifications to the browser such as including a proxy.

const browser = await puppeteer.launch();
const mangapark = new MangaPark({
  proxy: { host: '127.0.0.1', port: 8080 },
  puppeteerInstance: { instance: 'custom', browser },
}); // ❌ Mangascraper cannot include a proxy
const browser = await puppeteer.launch({ args: ['--proxy-server=127.0.0.1:8080'] });
const mangapark = new MangaPark({ puppeteerInstance: { instance: 'custom', browser } }); // ✔️ Our own browser instance will launch with a proxy

By default, mangascraper does not close the browser after the end of operation. If by any means you want to close the browser after an operation has finished. You can add the following to puppeteerInstance

puppeteerInstance: {
  instance: 'custom',
  browser: browser,
  options: {
    closeAfterOperation: true // After an operation is finished, close the browser
  }
}

However, this will prevent mangascraper from proceeding to another operation after one is finished such as this example:

const mangapark = new MangaPark({ puppeteerInstance: 'custom', browser, options: { closeAfterOperation: true } });
await mangapark
  .search('Naruto', { orderBy: 'latest_updates' })
  .then(async (mangas) => await Promise.all(mangas.map((manga) => mangapark.getMangaMeta(manga.url)))); // ❌ Browser will close after gathering results of mangas that match the title Naruto and will not gather metadata from each source.

Examples

API Reference

• Mangakakalot

• Manganato

• MangaSee

• Mangahasu

• MangaPark

• ReadMng

• MangaBox

License

Distributed under MIT © Joseph Marbella