Redis Dataloader

Batching and Caching layer using Redis as the Caching layer. Redis Dataloader wraps Facebook Dataloader, adding Redis as a caching layer.

npm install redis-dataloader
or
yarn add redis-dataloader

Example

1const redisClient = require('redis').createClient();
2// the "ioredis" module is also supported
3// const Redis = require('ioredis');
4// const redisClient = new Redis();
5const DataLoader = require('dataloader');
6const RedisDataLoader = require('redis-dataloader')({ redis: redisClient });
7
8const loader = new RedisDataLoader(
9    // set a prefix for the keys stored in redis. This way you can avoid key
10    // collisions for different data-sets in your redis instance.
11    'redis-key-prefix',
12    // create a regular dataloader. This should always be set with caching disabled.
13    new DataLoader(myBatchLoadFunction, { cache: false }),
14    // The options here are the same as the regular dataloader options, with
15    // the additional option "expire"
16    {
17        // caching here is a local in memory cache. Caching is always done
18        // to redis.
19        cache: true,
20        // if set redis keys will be set to expire after this many seconds
21        // this may be useful as a fallback for a redis cache.
22        expire: 60,
23        // can include a custom serialization and deserialization for
24        // storage in redis.
25        serialize: date => date.getTime(),
26        deserialize: timestamp => new Date(timestamp),
27        // Set this to true to return Buffer objects to the deserialize function
28        // when using the ioredis driver.
29        buffer: false
30    }
31);
32
33// load an individual item by its key
34loader.load(5).then(resp => console.log(resp));
35
36//clear an individiaul item from the local and redis cache.
37loader.clear(5).then(() => {})

API Documentation

In general, RedisDataLoader has the same API as the Facebook Dataloader Api, with a few differences. Read through the Facebook Dataloader documentation and then note the differences mentioned here.

• clear returns a promise (waits until redis succeeds at deleting the key). Facebook Dataloader's clear method is synchronous.
• clearAll is not available (redis does not have an efficient way to do this?)
• prime will always overwrite the cache. Facebook Dataloader will only write to its cache if a value is not already present. Prime is asyncronous and returns a Promise.
• dataloader results must be either null or a JSON object.
• two functions: clearLocal(key) and clearAllLocal() allow you to clear the local cache only.

Instantiation

Dependency inject a Redis Connection

1const redis = require('redis').createClient();
2const RedisDataLoader = require('redis-dataloader')({ redis: redis });

Create a new Dataloader.

Each Dataloader holds its own local in memory cache (Same as Facebook Dataloader), and additionally caches to your Redis instance.

1const loader = new RedisDataLoader('<redis key prefix>', '<Facebook Dataloader>', '<Options>');

Redis Key Prefix

Specify a Prefix that will be appended to each key when storing in Redis.

So for example if your prefix is "bar" and you call loader.load('foo'), this key will be stored in Redis as bar:foo

Facebook Dataloader

A regular Facebook Dataloader is passed in as the second parameter. It will be used to fetch data from your underlying datastore (mongo, sql, whatever). It is very important to disable the cache on this dataloader. Redis dataloader will already do local in memory caching (unless you disable it).

Options

All the options available to Facebook Dataloader can be passed in here. An additional option called expire is also available, and will set a ttl in seconds on all keys set in redis if this option is passed.

The cacheKeyFn will default to serialize objects and arrays using json-stable-stringify and allow all other values to pass through unchanged.

buffer will pass a Buffer object to the deserialize function rather than a string. If not using the ioredis driver this will throw an error at instation.

Caching

The purpose of Redis Dataloader is to provide a caching layer in redis on top of the Facebook Dataloader. Facebook's Dataloader provides a local in memory cache. This may be ok for short lived per-request caches, but may not be sufficient if you need a long lived cache and/or you have multiple webservers that need to share data.

Redis Dataloader will additionally use the same local cache that Facebook Dataloader provides. It will first check the local cache, then check the redis cache, before finally checking your underlying datastore. This pattern may be desirable if for example you create a new DataLoader for each request. If your dataloader is long-lived you may want to disable to the local cache, and just rely on the redis cache instead

1const loader = new RedisDataLoader('prefix', new DataLoader(), { cache: false });

Development

1. Install Dependencies npm install
2. Start Redis docker-compose stop && docker-compose rm && docker-compose build && docker-compose up -d
3. Run Tests npm test