Node.js - klaw

A Node.js file system walker extracted from fs-extra.

Install

npm i --save klaw

If you're using Typescript, we've got types:

npm i --save-dev @types/klaw

Name

klaw is walk backwards :p

Sync

If you need the same functionality but synchronous, you can use klaw-sync.

Usage

klaw(directory, [options])

Returns a Readable stream that iterates through every file and directory starting with dir as the root. Every read() or data event returns an object with two properties: path and stats. path is the full path of the file and stats is an instance of fs.Stats.

• directory: The directory to recursively walk. Type string or file URL.
• options: Readable stream options and the following:
  • queueMethod (string, default: 'shift'): Either 'shift' or 'pop'. On readdir() array, call either shift() or pop().
  • pathSorter (function, default: undefined): Sorting function for Arrays.
  • fs (object, default: graceful-fs): Use this to hook into the fs methods or to use mock-fs
  • filter (function, default: undefined): Filtering function for Arrays
  • depthLimit (number, default: undefined): The number of times to recurse before stopping. -1 for unlimited.
  • preserveSymlinks (boolean, default: false): Whether symlinks should be followed or treated as items themselves. If true, symlinks will be returned as items in their own right. If false, the linked item will be returned and potentially recursed into, in its stead.

Streams 1 (push) example:

1const klaw = require('klaw')
2
3const items = [] // files, directories, symlinks, etc
4klaw('/some/dir')
5  .on('data', item => items.push(item.path))
6  .on('end', () => console.dir(items)) // => [ ... array of files]

Streams 2 & 3 (pull) example:

1const klaw = require('klaw')
2
3const items = [] // files, directories, symlinks, etc
4klaw('/some/dir')
5  .on('readable', function () {
6    let item
7    while ((item = this.read())) {
8      items.push(item.path)
9    }
10  })
11  .on('end', () => console.dir(items)) // => [ ... array of files]

for-await-of example:

1for await (const file of klaw('/some/dir')) {
2  console.log(file)
3}

Error Handling

Listen for the error event.

Example:

1const klaw = require('klaw')
2
3klaw('/some/dir')
4  .on('readable', function () {
5    let item
6    while ((item = this.read())) {
7      // do something with the file
8    }
9  })
10  .on('error', (err, item) => {
11    console.log(err.message)
12    console.log(item.path) // the file the error occurred on
13  })
14  .on('end', () => console.dir(items)) // => [ ... array of files]

Aggregation / Filtering / Executing Actions (Through Streams)

On many occasions you may want to filter files based upon size, extension, etc. Or you may want to aggregate stats on certain file types. Or maybe you want to perform an action on certain file types.

You should use the module through2 to easily accomplish this.

Install through2:

npm i --save through2

Example (skipping directories):

1const klaw = require('klaw')
2const through2 = require('through2')
3
4const excludeDirFilter = through2.obj(function (item, enc, next) {
5  if (!item.stats.isDirectory()) this.push(item)
6  next()
7})
8
9const items = [] // files, directories, symlinks, etc
10klaw('/some/dir')
11  .pipe(excludeDirFilter)
12  .on('data', item => items.push(item.path))
13  .on('end', () => console.dir(items)) // => [ ... array of files without directories]

Example (ignore hidden directories):

1const klaw = require('klaw')
2const path = require('path')
3
4const filterFunc = item => {
5  const basename = path.basename(item)
6  return basename === '.' || basename[0] !== '.'
7}
8
9klaw('/some/dir', { filter: filterFunc })
10  .on('data', item => {
11    // only items of none hidden folders will reach here
12  })

Example (totaling size of PNG files):

1const klaw = require('klaw')
2const path = require('path')
3const through2 = require('through2')
4
5let totalPngsInBytes = 0
6const aggregatePngSize = through2.obj(function (item, enc, next) {
7  if (path.extname(item.path) === '.png') {
8    totalPngsInBytes += item.stats.size
9  }
10  this.push(item)
11  next()
12})
13
14klaw('/some/dir')
15  .pipe(aggregatePngSize)
16  .on('data', item => items.push(item.path))
17  .on('end', () => console.dir(totalPngsInBytes)) // => total of all pngs (bytes)

Example (deleting all .tmp files):

1const fs = require('fs')
2const klaw = require('klaw')
3const through2 = require('through2')
4
5const deleteAction = through2.obj(function (item, enc, next) {
6  this.push(item)
7
8  if (path.extname(item.path) === '.tmp') {
9    item.deleted = true
10    fs.unlink(item.path, next)
11  } else {
12    item.deleted = false
13    next()
14  }
15})
16
17const deletedFiles = []
18klaw('/some/dir')
19  .pipe(deleteAction)
20  .on('data', item => {
21    if (!item.deleted) return
22    deletedFiles.push(item.path)
23  })
24  .on('end', () => console.dir(deletedFiles)) // => all deleted files

You can even chain a bunch of these filters and aggregators together. By using multiple pipes.

Example (using multiple filters / aggregators):

1klaw('/some/dir')
2  .pipe(filterCertainFiles)
3  .pipe(deleteSomeOtherFiles)
4  .on('end', () => console.log('all done!'))

Example passing (piping) through errors:

Node.js does not pipe() errors. This means that the error on one stream, like klaw will not pipe through to the next. If you want to do this, do the following:

1const klaw = require('klaw')
2const through2 = require('through2')
3
4const excludeDirFilter = through2.obj(function (item, enc, next) {
5  if (!item.stats.isDirectory()) this.push(item)
6  next()
7})
8
9const items = [] // files, directories, symlinks, etc
10klaw('/some/dir')
11  .on('error', err => excludeDirFilter.emit('error', err)) // forward the error on
12  .pipe(excludeDirFilter)
13  .on('data', item => items.push(item.path))
14  .on('end', () => console.dir(items)) // => [ ... array of files without directories]

Searching Strategy

Pass in options for queueMethod, pathSorter, and depthLimit to affect how the file system is recursively iterated. See the code for more details, it's less than 50 lines :)

License

MIT

Copyright (c) 2015 JP Richardson