tokenize-comment

Uses snapdragon to tokenize a single JavaScript block comment into an object, with description, tags, and code example sections that can be passed to any other comment parsers for further parsing.

Install

Install with npm:

1$ npm install --save tokenize-comment

1/**
2 * foo bar baz
3 * 
4 * ```js
5 * var foo = "bar";
6 * ``` 
7 * @param {string} something
8 * @param {string} else
9 */

1{
2  description: 'foo bar baz',
3  footer: '',
4  examples: [
5    {
6      type: 'gfm',
7      val: '```js\nvar foo = "bar";\n```',
8      description: '',
9      language: 'js',
10      code: '\nvar foo = "bar";\n'
11    }
12  ],
13  tags: [
14    {
15      type: 'tag',
16      raw: '@param {string} something',
17      key: 'param',
18      val: '{string} something'
19    }, 
20    {
21      type: 'tag',
22      raw: '@param {string} else',
23      key: 'param',
24      val: '{string} else'
25    }
26  ]
27}

Usage

The main export is a function that takes a string with a single javascript comment only, no code.

1var tokenize = require('tokenize-comment');
2var token = tokenize(commentString);
3console.log(token);

The comment can be a "raw" comment with leading stars:

1/**
2 * foo bar baz
3 * @param {String} abc Some description
4 * @param {Object} def Another description
5 */

Or a comment with stars already stripped (with or without leading whitespace):

1 foo bar baz
2 @param {String} abc Some description
3 @param {Object} def Another description

Code examples

Recognizes gfm, javadoc and indented code examples. See the unit tests for a number of more complex examples.

GitHub Flavored Markdown

Supports GFM style code examples. The following comment:

1/**
2 * foo bar baz
3 * 
4 * ```js
5 * var foo = "bar";
6 * ``` 
7 * @param {string} something
8 * @param {string} else
9 */

Results in:

1{
2  description: 'foo bar baz',
3  footer: '',
4  examples: [
5    {
6      type: 'gfm',
7      val: '```js\nvar foo = "bar";\n```',
8      description: '',
9      language: 'js',
10      code: '\nvar foo = "bar";\n'
11    }
12  ],
13  tags: [
14    {
15      type: 'tag',
16      raw: '@param {string} something',
17      key: 'param',
18      val: '{string} something'
19    }, 
20    {
21      type: 'tag',
22      raw: '@param {string} else',
23      key: 'param',
24      val: '{string} else'
25    }
26  ]
27}

Indented code

Supports indented code examples:

1/**
2 * foo bar baz
3 * 
4 *     var foo = "bar";
5 *
6 * @param {string} something
7 * @param {string} else
8 */

javadoc (jsdoc)

Supports javadoc-style code examples:

1/**
2 * foo bar baz
3 * 
4 * @example
5 * var foo = "bar";
6 * var baz = "qux";
7 *
8 * @param {string} something
9 * @param {string} else
10 */

Results in:

1{
2  description: 'foo bar baz',
3  footer: '',
4  examples: [
5    {
6      type: 'javadoc',
7      language: '',
8      description: '',
9      raw: '@example\nvar foo = "bar";\nvar baz = "qux";\n',
10      val: '\nvar foo = "bar";\nvar baz = "qux";\n'
11    }
12  ],
13  tags: [
14    {
15      type: 'tag',
16      raw: '@param {string} something',
17      key: 'param',
18      val: '{string} something'
19    }, 
20    {
21      type: 'tag',
22      raw: '@param {string} else',
23      key: 'param',
24      val: '{string} else'
25    }
26  ]
27}

Mixture

It will also recognize a mixture of formats (javadoc-style examples must always be last):

1/**
2 * This is a comment with
3 * several lines of text.
4 *
5 * An example
6 *
7 * ```js
8 * var foo = bar;
9 * var foo = bar;
10 * var foo = bar;
11 * ```
12 *
13 * Another example
14 *
15 *     var baz = fez;
16 *     var baz = fez;
17 *     var baz = fez;
18 *
19 * Another example
20 *
21 *     var baz = fez;
22 *     var baz = fez;
23 *
24 *
25 *
26 * And another example
27 *
28 * ```js
29 * var foo = bar;
30 * var foo = bar;
31 * ```
32 *
33 * Another example
34 *
35 * @example
36 * var baz = fez;
37 *
38 * @example
39 * // this is a comment
40 * var alalla = zzzz;
41 *
42 * @param {String} foo bar
43 * @returns {Object} Instance of Foo
44 * @api public
45 */

Results in:

1{
2  description: 'This is a comment with\nseveral lines of text.',
3  footer: '',
4  examples: [
5    {
6      type: 'gfm',
7      language: 'js',
8      description: 'An example',
9      raw: '```js\nvar foo = bar;\nvar foo = bar;\nvar foo = bar;\n```',
10      val: '\nvar foo = bar;\nvar foo = bar;\nvar foo = bar;\n'
11    }, 
12    {
13      type: 'indented',
14      language: '',
15      description: 'Another example',
16      raw: '    var baz = fez;\n    var baz = fez;\n    var baz = fez;\n',
17      val: 'var baz = fez;\nvar baz = fez;\nvar baz = fez;\n'
18    }, 
19    {
20      type: 'indented',
21      language: '',
22      description: 'Another example',
23      raw: '    var baz = fez;\n    var baz = fez;\n',
24      val: 'var baz = fez;\nvar baz = fez;\n'
25    }, 
26    {
27      type: 'gfm',
28      language: 'js',
29      description: 'And another example',
30      raw: '```js\nvar foo = bar;\nvar foo = bar;\n```',
31      val: '\nvar foo = bar;\nvar foo = bar;\n'
32    }, 
33    {
34      type: 'javadoc',
35      language: '',
36      description: 'Another example',
37      raw: '@example\nvar baz = fez;\n',
38      val: '\nvar baz = fez;\n'
39    }, 
40    {
41      type: 'javadoc',
42      language: '',
43      description: '',
44      raw: '@example\n// this is a comment\nvar alalla = zzzz;\n',
45      val: '\n// this is a comment\nvar alalla = zzzz;\n'
46    }
47  ],
48  tags: [
49    {
50      type: 'tag',
51      raw: '@param {String} foo bar',
52      key: 'param',
53      val: '{String} foo bar'
54    }, 
55    {
56      type: 'tag',
57      raw: '@returns {Object} Instance of Foo',
58      key: 'returns',
59      val: '{Object} Instance of Foo'
60    }, 
61    {
62      type: 'tag',
63      raw: '@api public',
64      key: 'api',
65      val: 'public'
66    }
67  ]
68}

About

Related projects

• parse-comments: Parse code comments from JavaScript or any language that uses the same format. | homepage
• snapdragon: Easy-to-use plugin system for creating powerful, fast and versatile parsers and compilers, with built-in source-map… more | homepage
• strip-comments: Strip comments from code. Removes line comments, block comments, the first comment only, or all… more | homepage

Contributing

Pull requests and stars are always welcome. For bugs and feature requests, please create an issue.

Please read the contributing guide for advice on opening issues, pull requests, and coding standards.

Building docs

(This project's readme.md is generated by verb, please don't edit the readme directly. Any changes to the readme must be made in the .verb.md readme template.)

To generate the readme, run the following command:

1$ npm install -g verbose/verb#dev verb-generate-readme && verb

Running tests

Running and reviewing unit tests is a great way to get familiarized with a library and its API. You can install dependencies and run tests with the following command:

1$ npm install && npm test

Author

Jon Schlinkert

• github/jonschlinkert
• twitter/jonschlinkert

License

Copyright © 2017, Jon Schlinkert. Released under the MIT License.

This file was generated by verb-generate-readme, v0.4.3, on March 16, 2017.