npmpackage.info

Gathering detailed insights and metrics for robots-txt-parse

Other packages similar to robots-txt-parse

robots-util

0.1.2

Parses the robots.txt file and other util

Gathering detailed insights and metrics for robots-txt-parse

robots-txt-parse - 2.0.1 | npmpackage.info

robots-txt-parse

Streaming parser for robots.txt files

2.0.1

MIT

JavaScript

2.3

Installations

npm install robots-txt-parse

Developer Guide

BETA

Typescript

No

Module System

CommonJS

Node Version

10.23.3

NPM Version

6.14.11 Pull Requests

Open

2

Total

9

Closed

1

Merged

6

Issues

Open

1

Total

3

Closed

2

Releases

Unable to fetch releases

Languages

JavaScript

JavaScript (100%)

Developer

Woorank

Download Statistics

Total Downloads

Last Day

Last Week

Last Month

Last Year

GitHub Statistics

MIT License

3 Stars

45 Commits

1 Forks

6 Watchers

4 Branches

6 Contributors

Updated on Apr 25, 2023

Maintainers

View All 6 Contributors

Package Meta Information

Latest Version

2.0.1

Package Id

robots-txt-parse@2.0.1

Size

4.30 kB

NPM Version

6.14.11

Node Version

10.23.3

Published on

Feb 23, 2021

Total Downloads

Cumulative downloads

Total Downloads

NaN

Last Day

NaN

Compared to previous day

Last Week

NaN

Compared to previous week

Last Month

NaN

Compared to previous month

Last Year

NaN

Compared to previous year

Weekly Downloads

Monthly Downloads

Yearly Downloads

Dependencies

split2

Dev Dependencies

@types/split2 chai mocha semistandard typescript

robots-txt-parse

Streaming robots.txt parser

usage

1const parse = require('robots-txt-parse');
2const fs = require('fs');
3
4const input = fs.createReadStream(__dirname + '/robots.txt');
5const result = await parse(input);

assuming this file

user-agent: *
user-agent: googlebot
disallow: /

user-agent: twitterbot
disallow: /
allow: /twitter

user-agent: mozilla
disallow: /path
noindex: /path

Sitemap: http://www.example.com/sitemap.xml

produces following output

1{
2  "groups": [{
3    "agents": [ "*", "googlebot" ],
4    "rules": [
5      { "rule": "disallow", "path": "/" }
6    ]
7  }, {
8    "agents": [ "twitterbot" ],
9    "rules": [
10      { "rule": "disallow", "path": "/" },
11      { "rule": "allow", "path": "/twitter" }
12    ]
13  }, {
14    "agents": [ "mozilla" ],
15    "rules": [
16      { "rule": "disallow", "path": "/path" },
17      { "rule": "noindex", "path": "/path" }
18    ]
19  }],
20  "extensions": [
21    { "extension": "sitemap", "value": "http://www.example.com/sitemap.xml" }
22  ]
23}

No vulnerabilities found.

10

Binary-Artifacts

Determines if the project has generated executable (binary) artifacts in the source repository.

10

License

Determines if the project has defined a license.

4

Code-Review

Determines if the project requires human code review before pull requests (aka merge requests) are merged.

0

Maintained

Determines if the project is "actively maintained".

0

CII-Best-Practices

Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.

0

Fuzzing

Determines if the project uses fuzzing.

0

Security-Policy

Determines if the project has published a security policy.

0

Branch-Protection

Determines if the default and release branches are protected with GitHub's branch protection settings.

0

SAST

Determines if the project uses static code analysis.

0

Vulnerabilities

Determines if the project has open, known unfixed vulnerabilities.

Score

2.3

/10

Last Scanned on 2025-07-14

The Open Source Security Foundation is a cross-industry collaboration to improve the security of open source software (OSS). The Scorecard provides security health metrics for open source projects.

Learn More