eleventy-plugin-robotstxt

[!NOTE] This plugin is only compatible with Eleventy versions >= 3.0.0-alpha.6.

Automatically generate a robots.txt file for your Eleventy site using front matter.

Getting Started

Install the plugin in your project using your preferred package manager:

npm install --save-dev eleventy-plugin-robotstxt

And update your Eleventy config to import and use the plugin:

1const EleventyPluginRobotsTxt = require("eleventy-plugin-robotstxt");
2
3module.exports = (eleventyConfig) => {
4  /** @type {import("eleventy-plugin-robotstxt/typedefs.js").EleventyPluginRobotsTxtOptions} */
5  const eleventyPluginRobotsTxtOptions = {};
6  eleventyConfig.addPlugin(
7    EleventyPluginRobotsTxt,
8    eleventyPluginRobotsTxtOptions,
9  );
10};

See the examples for how you might configure the plugin for different use cases.

API

The following plugin options are available for use in your .eleventy.js configuration:

Option	Type	Description	Example
sitemapURL	string | undefined	(Optional) The absolute location of a sitemap for this site. The sitemap URL must be a fully-qualified URL; Google doesn't assume or check http/https/www.non-www alternates. Sitemaps are a good way to indicate which content Google should crawl, as opposed to which content it can or cannot crawl. See also: Google Search Central - How to write robots.txt rules.	https://www.example.com/sitemap.xml
rules	Map<string | string[], ({ allow: string } | { disallow: string })[]> | undefined	(Optional) A map of robots.txt rules, grouped by user agent. Each key is an array of user agents in the group; the value is an array of allowed or disallowed paths.	See examples.
shouldBlockAIRobots	string | undefined	(Optional) Whether to soft-block a list of known AI robots (see the ai-robots GitHub repository for context).	true
frontMatterOverrides	Record<string, unknown> | undefined	Front matter overrides to apply to the robots.txt template. By default, the plugin automatically applies eleventyExcludeFromCollections: true and permalink: /robots.txt, so you do not need to set these yourself.	{ "frontMatterKey": "value" }

Examples

The following examples are direct translations of Google's guide on how to write and submit a robots.txt file.

Disallow crawling of the entire site

Input:

1const eleventyPluginRobotsTxtOptions = {
2  rules: new Map([["*", [{ disallow: "/" }]]]),
3};

Output:

1User-agent: *
2Disallow: /

Disallow crawling of a directory and its contents

Input:

1const eleventyPluginRobotsTxtOptions = {
2  rules: new Map([
3    [
4      "*",
5      [
6        { disallow: "/calendar/" },
7        { disallow: "/junk/" },
8        { disallow: "/books/fiction/contemporary/" },
9      ],
10    ],
11  ]),
12};

Output:

1User-agent: *
2Disallow: /calendar/
3Disallow: /junk/
4Disallow: /books/fiction/contemporary/

Allow access to a single crawler

Input:

1const eleventyPluginRobotsTxtOptions = {
2  rules: new Map([
3    ["Googlebot-news", [{ allow: "/" }]],
4    ["*", [{ disallow: "/" }]],
5  ]),
6};

Output:

1User-agent: Googlebot-news
2Allow: /
3
4User-agent: *
5Disallow: /

Allow access to all but a single crawler

Input:

1const eleventyPluginRobotsTxtOptions = {
2  rules: new Map([
3    ["Unnecessarybot", [{ disallow: "/" }]],
4    ["*", [{ allow: "/" }]],
5  ]),
6};

Output:

1User-agent: Unnecessarybot
2Disallow: /
3
4User-agent: *
5Allow: /

Disallow crawling of a single web page

For example, disallow the useless_file.html page located at https://example.com/useless_file.html, and other_useless_file.html in the junk directory.

Input:

1const eleventyPluginRobotsTxtOptions = {
2  rules: new Map([
3    [
4      "*",
5      [
6        { disallow: "/useless_file.html" },
7        { disallow: "/junk/other_useless_file.html" },
8      ],
9    ],
10  ]),
11};

Output:

1User-agent: *
2Disallow: /useless_file.html
3Disallow: /junk/other_useless_file.html

Disallow crawling of the whole site except a subdirectory

Input:

1const eleventyPluginRobotsTxtOptions = {
2  rules: new Map([["*", [{ disallow: "/" }, { allow: "/public/" }]]]),
3};

Output:

1User-agent: *
2Disallow: /
3Allow: /public/

Block a specific image from Google Images

Input:

1const eleventyPluginRobotsTxtOptions = {
2  rules: new Map([["Googlebot-Image", [{ disallow: "/images/dogs.jpg" }]]]),
3};

Output:

1User-agent: Googlebot-Image
2Disallow: /images/dogs.jpg

Block all images on your site from Google Images

Input:

1const eleventyPluginRobotsTxtOptions = {
2  rules: new Map([["Googlebot-Image", [{ disallow: "/" }]]]),
3};

Output:

1User-agent: Googlebot-Image
2Disallow: /

Disallow crawling of files of a specific file type

Input:

1const eleventyPluginRobotsTxtOptions = {
2  rules: new Map([["Googlebot", [{ disallow: "/*.gif$" }]]]),
3};

Output:

1User-agent: Googlebot
2Disallow: /*.gif$

Disallow crawling of an entire site, but allow Mediapartners-Google

Input:

1const eleventyPluginRobotsTxtOptions = {
2  rules: new Map([
3    ["*", [{ disallow: "*" }]],
4    ["Mediapartners-Google", [{ allow: "/" }]],
5  ]),
6};

Output:

1User-agent: *
2Disallow: /
3
4User-agent: Mediapartners-Google
5Allow: /

Disallow crawling for multiple user agents in one go

Input:

1const eleventyPluginRobotsTxtOptions = {
2  rules: new Map([[["agent1", "agent2", "agent3"], [{ disallow: "/" }]]]),
3};

Output:

1User-agent: agent1
2User-agent: agent2
3User-agent: agent3
4Disallow: /

Notes

• This plugin registers Liquid as a recognized template language, as suggested by Zach Leatherman here: https://github.com/11ty/eleventy/issues/1612#issuecomment-2027476340