express-robots-txt

Express middleware for generating a robots.txt or responding with an existing file.

Usage

1// es6 usage requires a version of node which supports esm_conditional_exports, see
2// https://nodejs.org/api/esm.html#esm_conditional_exports
3import express from 'express'
4import robots from 'express-robots-txt'
5
6// commonjs
7const express = require('express')
8const robots = require('express-robots-txt')
9
10const app = express()
11const port = 3000
12
13app.use(robots({
14  UserAgent: '*',
15  Disallow: '/',
16  CrawlDelay: '5',
17  Sitemap: 'https://nowhere.com/sitemap.xml',
18}))
19
20app.listen(port)

Using a file

1app.use(robots(__dirname + '/robots.txt'));

Using an object

Basic object

1app.use(robots({
2  UserAgent: '*',
3  Disallow: '/'
4}))

Will produce:

User-agent: *
Disallow: /

Crawl Delay

You can optionally pass a CrawlDelay in just like passing in Disallow

1app.use(robots({
2  UserAgent: '*',
3  Disallow: '/',
4  CrawlDelay: '5'
5}))

Will produce:

User-agent: *
Disallow: /
Crawl-delay: 5

Sitemap

You can optionally pass a Sitemap in just like passing in CrawlDelay:

1app.use(robots({
2  UserAgent: '*',
3  Disallow: '/',
4  CrawlDelay: '5',
5  Sitemap: 'https://nowhere.com/sitemap.xml'
6}))

Will produce:

User-agent: *
Disallow: /
Crawl-delay: 5
Sitemap: https://nowhere.com/sitemap.xml

You can also pass in an array of Sitemaps:

1app.use(robots({
2  UserAgent: '*',
3  Disallow: '/',
4  CrawlDelay: '5',
5  Sitemap: [
6    'https://nowhere.com/sitemap.xml',
7    'https://nowhere.com/sitemap2.xml'
8  ]
9}))

Will produce:

User-agent: *
Disallow: /
Crawl-delay: 5
Sitemap: https://nowhere.com/sitemap.xml
Sitemap: https://nowhere.com/sitemap2.xml

Or an array of objects

1app.use(robots([
2  {
3    UserAgent: 'Googlebot',
4    Disallow: '/no-google'
5  },
6  {
7    UserAgent: 'Bingbot',
8    Disallow: '/no-bing'
9  }
10]));

Will produce:

User-agent: Googlebot
Disallow: /no-google
User-agent: Bingbot
Disallow: /no-bing

Or either property (UserAgent | Disallow) as an array

1app.use(robots([
2  {
3    UserAgent: [ 'Googlebot', 'Slurp' ],
4    Disallow: [ '/no-google', '/no-yahoo' ]
5  },
6  {
7    UserAgent: '*',
8    Disallow: [ '/no-bots', '/still-no-bots' ]
9  }
10]));

Will produce:

User-agent: Googlebot
User-agent: Slurp
Disallow: /no-google
Disallow: /no-yahoo
User-agent: *
Disallow: /no-bots
Disallow: /still-no-bots

Host

You can optionally pass a Host in just like passing in Disallow. Note that some crawlers may not support this drective.

1app.use(robots({
2  UserAgent: '*',
3  Disallow: '/',
4  CrawlDelay: '5',
5  Host: 'foobar.com'
6}))

Will produce:

User-agent: *
Disallow: /
Crawl-delay: 5
Host: foobar.com

Thanks

Originally forked from weo-edu/express-robots.