url-metadata

Request a url and scrape the metadata from its HTML using Node.js or the browser. Has an optional mode that lets you pass in a string of html or a Response object as well (see Options section below).

Includes:

• meta tags
• favicons
• citations, per the Google Scholar spec
• Open Graph Protocol (og:) Tags
• Twitter Card Tags
• JSON-LD
• h1-h6 tags
• img tags
• relevant response headers
• automatic charset detection & decoding (optional)
• the full response body as a string of html (optional)

More details in the Returns section below.

v5.1.0+ Protects against:

• Infinite redirect loops
• SSRF attacks via request-filtering-agent in Node.js v18+ (custom options available)

To report a bug or request a feature please open an issue or pull request in GitHub. Please read the Troubleshooting section below before filing a bug.

Install

Works with Node.js versions >=6.0.0 or in the browser when bundled with Webpack (see /example-typescript) or Vite (see /example-vite). Use previous version 2.5.0 which uses the (now-deprecated) request module if you don't have access to node-fetch or window.fetch in your target environment.

npm install url-metadata --save

Usage

In your project file:

1const urlMetadata = require('url-metadata');
2
3(async function () {
4  try {
5    const url = 'https://www.npmjs.com/package/url-metadata';
6    const metadata = await urlMetadata(url);
7    console.log(metadata);
8  } catch (err) {
9    console.log(err);
10  }
11})();
12

Options & Defaults

To override the default options, pass in a second options argument. The default options are the values below.

1const options = {
2
3  // Customize the default request headers:
4  requestHeaders: {
5    'User-Agent': 'url-metadata (+https://www.npmjs.com/package/url-metadata)',
6    From: 'example@example.com'
7  },
8
9  // (Node.js v18+ only)
10  // To prevent SSRF attacks, the default option below blocks
11  // requests to private network & reserved IP addresses via:
12  // https://www.npmjs.com/package/request-filtering-agent
13  // Browser security policies prevent SSRF automatically.
14  requestFilteringAgentOptions: undefined,
15
16  // (Node.js v6+ only)
17  // Pass in your own custom `agent` to override the
18  // built-in request filtering agent above
19  // https://www.npmjs.com/package/node-fetch/v/2.7.0#custom-agent
20  agent: undefined,
21
22  // (Browser only) `fetch` API cache setting
23  cache: 'no-cache',
24
25  // (Browser only) `fetch` API mode (ex: 'cors', 'same-origin', etc)
26  mode: 'cors',
27
28  // Maximum redirects in request chain, defaults to 10
29  maxRedirects: 10,
30
31  // `fetch` timeout in milliseconds, default is 10 seconds
32  timeout: 10000,
33
34  // (Node.js v6+ only) max size of response in bytes (uncompressed)
35  // Default set to 0 to disable max size
36  size: 0,
37
38  // (Node.js v6+ only) compression defaults to true
39  // Support gzip/deflate content encoding, set `false` to disable
40  compress: true,
41
42  // Charset to decode response with (ex: 'auto', 'utf-8', 'EUC-JP')
43  // defaults to auto-detect in `Content-Type` header or meta tag
44  // if none found, default `auto` option falls back to `utf-8`
45  // override by passing in charset here (ex: 'windows-1251'):
46  decode: 'auto',
47
48  // Number of characters to truncate description to
49  descriptionLength: 750,
50
51  // Force image urls in selected tags to use https,
52  // valid for images & favicons with full paths
53  ensureSecureImageRequest: true,
54
55  // Include raw response body as string
56  includeResponseBody: false,
57
58  // Alternate use-case: pass in `Response` object here to be parsed
59  // see example below
60  parseResponseObject: undefined
61};
62
63// Basic options usage
64try {
65  const url = 'https://www.npmjs.com/package/url-metadata';
66  const metadata = await urlMetadata(url, options);
67  console.log(metadata);
68} catch (err) {
69  console.log(err);
70}
71
72// Alternate use-case: parse a Response object instead
73try {
74  // fetch the url in your own code
75  const response = await fetch('https://www.npmjs.com/package/url-metadata');
76  // ... do other stuff with it...
77  // pass the `response` object to be parsed for its metadata
78  const metadata = await urlMetadata(null, {
79    parseResponseObject: response
80  });
81  console.log(metadata);
82} catch (err) {
83  console.log(err);
84}
85
86// Similarly, if you have a string of html you can create
87// a response object and pass the html string into it.
88const html = `
89<!DOCTYPE html>
90<html lang="en">
91  <head>
92    <meta charset="utf-8">
93    <title>Metadata page</title>
94    <meta name="author" content="foobar">
95    <meta name="keywords" content="HTML, CSS, JavaScript">
96  </head>
97  <body>
98    <h1>Metadata page</h1>
99  </body>
100</html>
101`;
102const response = new Response(html, {
103  headers: {
104    'Content-Type': 'text/html'
105  }
106});
107const metadata = await urlMetadata(null, {
108  parseResponseObject: response
109});
110console.log(metadata);

Returns

Returns a promise resolved with an object. Note that the url field returned will be the last hop in the request chain. If you pass in a url from a url shortener you'll get back the final destination as the url.

A basic template for the returned metadata object can be found in lib/metadata-fields.js. Any additional meta tags found on the page are appended as new fields to the object.

The returned metadata object consists of key/value pairs as strings, with a few exceptions:

• favicons is an array of objects containing key/value pairs of strings
• jsonld is an array of objects
• responseHeaders is an object containing key/value pairs of strings
• all meta tags that begin with citation_ (ex: citation_author) return with keys as strings and values that are an array of strings to conform to the Google Scholar spec which allows for multiple citation meta tags with different content values. So if the html contains:

<meta name="citation_author" content="Arlitsch, Kenning">
<meta name="citation_author" content="OBrien, Patrick">

... it will return as:

'citation_author': ["Arlitsch, Kenning", "OBrien, Patrick"],

Troubleshooting

Issue: DNS Lookup errors. The SSRF filtering agent defaults on this package prevent calls to private ip addresses, link-local addresses and reserved ip addresses. To change or disable this feature you need to pass custom requestFilteringAgentOptions. More info here.

Issue: No fetch implementation available. You're in either an older browser that doesn't have the native fetch API or a Node.js environment that doesn't support node-fetch (Node.js < v6). Try dowgrading to url-metadata version 2.5.0 which uses the now-deprecated request module.

Issue: Response status code 0 or CORS errors. The fetch request failed at either the network or protocol level. Possible causes:

• CORS errors. Try changing the mode option (ex: cors, same-origin, etc) or setting the Access-Control-Allow-Origin header on the server response from the url you are requesting if you have access to it.

• Trying to access an https resource that has invalid certificate, or trying to access an http resource from a page with an https origin.

• A browser plugin such as an ad-blocker or privacy protector.

Issue: Request returns 404, 403 errors or a CAPTCHA form. Your request may have been blocked by the server because it suspects you are a bot or scraper. Check this list to ensure you're not triggering a block.