robots.js

robots.js — is parser for robots.txt files for node.js.

Installation

It's recommended to install via npm:

1$ npm install -g robots

Usage

Here's an example of using robots.js:

1var robots = require('robots')
2  , parser = new robots.RobotsParser();
3
4parser.setUrl('http://nodeguide.ru/robots.txt', function(parser, success) {
5  if(success) {
6    parser.canFetch('*', '/doc/dailyjs-nodepad/', function (access) {
7      if (access) {
8        // parse url
9      }
10    });
11  }
12});

Default crawler user-agent is:

Mozilla/5.0 (X11; Linux i686; rv:5.0) Gecko/20100101 Firefox/5.0

Here's an example of using another user-agent and more detailed callback:

1var robots = require('robots')
2  , parser = new robots.RobotsParser(
3                'http://nodeguide.ru/robots.txt',
4                'Mozilla/5.0 (compatible; RobotTxtBot/1.0)',
5                after_parse
6            );
7            
8function after_parse(parser, success) {
9  if(success) {
10    parser.canFetch('*', '/doc/dailyjs-nodepad/', function (access, url, reason) {
11      if (access) {
12        console.log(' url: '+url+', access: '+access);
13        // parse url ...
14      }
15    });
16  }
17};

Here's an example of getting list of sitemaps:

1var robots = require('robots')
2  , parser = new robots.RobotsParser();
3
4parser.setUrl('http://nodeguide.ru/robots.txt', function(parser, success) {
5  if(success) {
6    parser.getSitemaps(function(sitemaps) {
7      // sitemaps — array
8    });
9  }
10});

Here's an example of getCrawlDelay usage:

1    var robots = require('robots')
2      , parser = new robots.RobotsParser();
3
4    // for example:
5    //
6    // $ curl -s http://nodeguide.ru/robots.txt
7    //
8    // User-agent: Google-bot
9    // Disallow: / 
10    // Crawl-delay: 2
11    //
12    // User-agent: *
13    // Disallow: /
14    // Crawl-delay: 2
15
16    parser.setUrl('http://nodeguide.ru/robots.txt', function(parser, success) {
17      if(success) {
18        var GoogleBotDelay = parser.getCrawlDelay("Google-bot");
19        // ...
20      }
21    });

An example of passing options to the HTTP request:

1var options = {
2  headers:{
3    Authorization:"Basic " + new Buffer("username:password").toString("base64")}
4}
5
6var robots = require('robots')
7  , parser = new robots.RobotsParser(null, options);
8
9parser.setUrl('http://nodeguide.ru/robots.txt', function(parser, success) {
10  ...
11});

API

RobotsParser — main class. This class provides a set of methods to read, parse and answer questions about a single robots.txt file.

• setUrl(url, read) — sets the URL referring to a robots.txt file. by default, invokes read() method. If read is a function, it is called once the remote file is downloaded and parsed, and it takes in two arguments: the first is the parser itself, and the second is a boolean which is True if the the remote file was successfully parsed.
• read(after_parse) — reads the robots.txt URL and feeds it to the parser
• parse(lines) — parse the input lines from a robots.txt file
• canFetch(userAgent, url, callback) — using the parsed robots.txt decide if userAgent can fetch url. Callback function: function callback(access, url, reason) { ... } where:
  • access — can this url be fetched. true/false.
  • url — target url
  • reason — reason for access. Object:
    • type — valid values: 'statusCode', 'entry', 'defaultEntry', 'noRule'
    • entry — an instance of lib/Entry.js:. Only for types: 'entry', 'defaultEntry'
    • statusCode — http response status code for url. Only for type 'statusCode'
• canFetchSync(userAgent, url) — using the parsed robots.txt decide if userAgent can fetch url. Return true/false.
• getCrawlDelay(userAgent) — returns Crawl-delay for the certain userAgent
• getSitemaps(sitemaps) — gets Sitemaps from parsed robots.txt
• getDisallowedPaths(userAgent) — gets paths explictly disallowed for the user agent specified AND *

License

See LICENSE file.

Resources

• Robots.txt Specifications by Google
• Robots.txt parser for python
• A Standard for Robot Exclusion