best-effort-concurrent-cache

Sometimes you want to cache things using the filesystem due to an expensive process, especially when developing build tools or other utilities. A perfect example is caching transpiled code. Another might simply be making no changes to the file contents, but instead using the cache to know if a file has changed between runs of a process.

NOTE: This package is not meant as a high-performance, bullet-proof caching utility. It is meant as a "best effort" to provide failsafe caching. In the best case, caching will happen. In the worst case, caching will not happen but your program won't fail.

Why?

Babel's require hook, for example, caches transpiled code, but does so as a monolithic JSON file. Another example is browserify-incremental. These monolithic JSON caches work well, but break down if multiple independent processes attempt to access the same cache. The cache is easily clobbered, corrupted, or made irrelevant by concurrent processes all reading an empty cache at startup, and writing multiple copies of the cache at process end.

When would you have multiple processes transpiling code? Perhaps when trying to speed up mocha unit tests by running them in parallel via separate processes. Or building multiple JS bundles from the same shared local libraries but different entry points.

Usage

1var fs = require('fs');
2var path = require('path');
3var Becc = require('best-effort-concurrent-cache');
4
5var becc = Becc(fs); // requires a file system implementation
6
7// Ensure cache folder exists
8var cachePath = path.join(process.cwd(), '.cache');
9try {
10  fs.mkdirSync(cachePath);
11} catch (e) {}
12
13var babel = require('babel-core');
14
15var f = path.join(process.cwd(), './myfile.es6');
16
17// Get a cached version
18// This checks the file's mtime compared to the cached hash.
19var retrieved = becc.retrieve(cachePath, f);
20
21if (!retrieved) {
22  // We didn't find a cached version, or the cached version has expired due
23  // to a differing mtime.
24  // Perform the expensive work and cache it!
25  retrieved = babel.transform(fs.readFileSync(f, 'utf8'), { presets: ['es2015'] }).code;
26  becc.cache(cachePath, f, retrieved);
27}
28
29// Do something with retrieved

API

Becc(fs, opt_statExtractor)

fs is an implementation of a node-compatible file system module. This is to facilitate testing and allow for in-memory or in-browser caching if needed. Generally the result of calling require('fs') will do.

opt_statExtractor is an optional function that is used to determine what properties of a file's fs.Stats object are used to cache the file. By default, only mtime is used. If one wanted to create a cache that also used file size:

1var fs = require('fs');
2var Becc = require('becc');
3
4var becc = Becc(fs, function (stat) {
5  return stat.mtime + stat.size;
6});

Note: the filename is always used within the cache key.

Examples

A shared Babel code cache can be used to speed up parallel (or subsequent) mocha runs. See examples/:

1$ rm -rf .cache
2
3$ time node examples/mocha-test.js
4...
5real  0m2.402s
6user  0m4.797s
7sys 0m0.354s
8
9# second run...
10$ time node examples/mocha-test.js
11...
12real  0m0.628s
13user  0m0.970s
14sys 0m0.161s

License

Apache 2.0