URL READER

This project helps you to read the content of URLs, and return the title, length, html, text, markdown, excerpt.

"node": ">=20.11.0"

Installation

1yarn add url-reader
2# or npm install url-reader

Usage

1import URLReader from 'url-reader';
2
3const reader = new URLReader();
4await reader.init();
5
6const results = await reader.read({
7  urls: ['https://www.google.com'],
8  timeout: 10000, // ms, default: 60000
9  enableMarkdown: false, // default: true
10  runScripts: 'dangerously', // run the scripts included in the HTML and fetch remote resources, default is closed.
11});

Parsed Result:

1interface IReaderResult {
2  title: string;
3  length: number;
4  html: string;
5  text: string;
6  markdown?: string;
7  excerpt: string;
8}

Server

• start server

1git clone https://github.com/yokingma/url-reader.git
2cd url-reader
3
4# default listen on port 3030
5yarn install & yarn run start

• api

1GET /reader?url=https://www.google.com
2
3POST /reader
4Body:
5{
6  urls: ['https://www.google.com', 'https://www.bing.com']
7}

Docker

1docker build -t urlreader . # urlreader is your image's tag name

The service will listen on port 3030.

Tips

• puppeteer When you install Puppeteer, it will automatically downloads a recent version of Chrome for Testing (~170MB macOS, ~282MB Linux, ~280MB Windows) and a chrome-headless-shell binary.

Troubleshooting

• install error with puppeteer

1Error [ERR_TLS_CERT_ALTNAME_INVALID]: Hostname/IP does not match certificate's altnames...

remove .npmrc file and re-install.