robotstxt-util

RFC 9309 spec compliant robots.txt builder and parser. 🦾 No dependencies, fully typed.

Before using this library, I recommend you to read the following guide by Google: https://developers.google.com/search/docs/crawling-indexing/robots/intro

Note to myself (and contributors): https://www.rfc-editor.org/rfc/rfc9309.html

Install

1npm i robotstxt-util

Use

Exports a parser parseRobotsTxt and an object RobotsTxt to create and manage robots.txt data.

Create robots.txt

1import { RobotsTxt } from 'robotstxt-util'
2
3const robotstxt = new RobotsTxt()
4
5const allBots = robotstxt.newGroup('*')
6allBots.disallow('/')
7
8const googleBot = robotstxt.newGroup('googlebot')
9googleBot.allow('/abc')
10googleBot.disallow('/def').disallow('/jkl')
11
12// specify multiple bots
13const otherBots = robotstxt.newGroup(['abot', 'bbot', 'cbot'])
14googleBot.allow('/qwe')
15// specify custom rules
16googleBot.addCustomRule('crawl-delay', 10)
17
18// add sitemaps
19robotstxt.add('sitemap', 'https://yoursite/sitemap.en.xml')
20robotstxt.add('sitemap', 'https://yoursite/sitemap.tr.xml')
21
22// and export
23const json = robotstxt.json()
24const txt = robotstxt.txt()

Parse robots.txt data

Parses the data and returns instance of RobotsTxt:

1import { parseRobotsTxt } from 'robotstxt-util'
2
3const data = `
4# hello robots
5
6User-Agent: *
7Disallow: *.gif$
8Disallow: /example/
9Allow: /publications/
10
11User-Agent: foobot
12Disallow:/
13crawl-delay: 10
14Allow:/example/page.html
15Allow:/example/allowed.gif
16
17# comments will be stripped out
18
19User-Agent: barbot
20User-Agent: bazbot
21Disallow: /example/page.html
22
23Sitemap: https://yoursite/sitemap.en.xml
24Sitemap: https://yoursite/sitemap.tr.xml
25`
26const robotstxt = parseRobotsTxt(data)
27
28// update something in some group
29robotstxt.findGroup('barbot').allow('/aaa').allow('/bbb')
30
31// store as json or do whatever you want
32const json = robotstxt.json()

Contributing

If you're interested in contributing, read the CONTRIBUTING.md first, please.

---

Thanks for watching 🐬