Redbase

A simple, fast, indexed, and type-safe database on top of Redis.

Built to answer the question, "how can I have a queryable, browsable db that also works well as a cache?"

Goals

• Simple: less than 500 lines. Only one dependency, ioredis. You can copy-paste the code instead if you want.
• Serverless-friendly: no Redis modules, only core Redis.
• Fast: Compared to optimized Postgres, 150% faster at paginating unindexed data. See all benchmarks below.
• Indexable: Supports hierarchical tags, a lightweight primitive for indexing your data.
• Browsable: browser-friendly API included for paginating and browsing by tag.

Non-goals

• No dependencies on Redis modules. Useful for deploying on platforms like Upstash.
• Never queries "KEYS" on your redis instance, which is expensive. Uses simple set theory and data structures to implement query logic.

In a few lines:

1import { Redbase } from 'redbase'
2const db = new Redbase<MyDataType>('my-project')
3const value = await db.get(id) // type: MyDataType

Exploration API: npm run server

• Redbase
  • Goals
  • Install
  • Usage
  • Core concepts
    • Entries
    • Tags
    • Example: Prompt Cache
    • Example: Browsing Your Data
  • Benchmarks
    • For the cache use-case
    • For the database use-case
    • Results
  • License

Install

1npm install redbase

Usage

1import { Redbase } from 'redbase'
2
3// Can use strings, numbers or buffers as well
4type MyValue = {
5  a: string
6  b?: {
7    c: string
8  }
9}
10
11// Options can also use your own ioredis instance if already defined,
12// as `redisInstance`
13const db = new Redbase<MyValue>('myProject', { redisUrl: 'redis://...' })
14
15const key = uuid()
16const value = { a: 'result' }
17
18await db.get(key) // undefined
19
20await db.set(key, value)
21
22await db.get(key) // value
23
24// Type safety!
25await db.set(key, { c: 'result2' }) // Type error on value
26
27// Browsing results
28let data = await db.filter()
29assertEqual(data, [{ id: key, value }])
30assertEqual(await db.count(), 1)
31
32// Hierarchical indexes, using a customizable tag separator (default: '/')
33await Promise.all([  // Redis auto-pipelines these calls into one fast request!  db.set(uuid(), { a: 'hi' }, { tags: ['user1/project1'] }),
34  db.set(uuid(), { a: 'there' }, { tags: ['user1/project2'] }),
35  db.set(uuid(), { a: 'bye' }, { tags: ['user2/project1'] })
36])
37
38data = await db.filter()
39assertEqual(data.length, 4)
40
41// WHERE queries, using tag names
42data = await db.filter({ where: 'user1'})
43assertEqual(data.length, 2)
44
45// AND and OR queries:
46data = await db.filter({ where: {OR: ['user1', 'user2']}})
47assertEqual(data.length, 3)
48
49const count = await db.count({ where: {OR: ['user1', 'user2']}})
50assertEqual(count, 3)
51
52// See all your indexes:
53const tags = await db.tags("user1")
54assertEqual(tags.length, 2)
55
56// Clear just parts of the database:
57const numberDeleted = await db.clear({ where: 'user2' })
58assertEqual(numberDeleted, 1)

For all functionality, see test/database.spec.ts.

Core concepts

There are two main concepts in Redbase: entries and tags. This section explains them and then provides example code combining both.

Entries

An entry is composed of an id and a value:

• Values are type-checked, but schemaless.
• IDs are strings. You are in charge of creating them, e.g. by hashing your value, making a UUID, using some external id, etc.
• If you have a non-string key that you want to use with your value (as is the case when you're storing prompts + completions), you can combine them into one value type. See the prompt cache example below.

Tags

Tags are a lightweight primitive for indexing your values. You attach them at insert-time, and they are schemaless. This makes them simple and flexible for many use cases. It also allows you to index values by external attributes (that aren't a part of the value itself: example).

Calling db.filter({ where: { AND: [...]}}) etc. allows you to compose them together as a list.

Tags are sort of self-cleaning: indexes delete themselves during bulk-delete operations, and they shrink when entries are deleted individually. However, when the last entry for an index is deleted, the index is not. This shouldn't cause a significant zombie index issue unless you're creating and wiping out an unbounded number of tags.

Tags can get unruly, you you can keep them organized by nesting them: parentindex/childindex. This effectively allows you to group your indexes, and makes browsing your data easier and more fun.

Call db.tags("parentindex") to get all the children tags.

As you might expect, when indexing an entry under parentindex/childindex the entry is automatically indexed under parentindex as well. This makes it easy to build a url-based cache exploration server. Call db.filter({ where: 'parentindex' }) to get all entries for all children tags.

Example: Prompt Cache

Sample code for setting up the prompt cache for a large language model (LLM), like OpenAI:

1import { CreateCompletionRequest, CreateCompletionResponse, OpenAIApi } from 'openai'
2
3export const CACHE_NAME = 'promptCache'
4const CACHE_EXPIRATION = 60 * 60 * 24 * 7 * 4 * 2 // 2 months
5
6export type PromptQuery = CreateCompletionRequest
7export type PromptResult = CreateCompletionResponse
8export type PromptEntry = { prompt: PromptQuery, completion: PromptResult }
9export const promptCache = new Redbase<PromptEntry>(CACHE_NAME, { defaultTTL: CACHE_EXPIRATION })

Now, you can cache and index your prompts by doing this:

1// Tags can include metadata that isn't present in the value itself.
2const tags = [
3  `user-${user || ""}`,
4  `url-${encodeURIComponent(url)}/leaf-${leafNumber}`
5]
6const payload = { ...requestData, user }
7const response = await openai.createCompletion(payload)
8await promptCache.save(id, {
9  prompt: requestData,
10  completion: response.data
11}, { tags })

Example: Browsing Your Data

To browse, paginate, filter, and delete your data directly from a browser, just fire up the API:

npm run server

Benchmarks

Note: I'm very new to benchmarking open-sourced code, and would appreciate pull requests here! One issue, for example, is that increasing the number of runs can cause the data to scale up (depending on which benchmarks you're running), which seems to e.g. make Redis win on pagination by a larger margin.

This project uses hyperfine to compare Redis in a persistent mode with Postgres in an optimized mode. Yes, this is comparing apples to oranges. I decided to do it anyway because:

1. A big question this project answers is "how can I have a queryable, browsable db that also works well as a cache?" Redis and Postgres are two backend choices that pop up frequently.

2. Not many people seem to know that Redis has an append-only file (AOF) persistence option. I heard about this and was curious to see how it performed.

3. I wanted to compare with an on-disk database solution to get a sense of where the advantage lies for using a persistent, in-memory database as a primary database.

All that said, the benchmarks put Redis and Postgres in roughly equivalent persistence settings, which you can play with:

• Redis is set up using SET appendfsync everysec, which issues fsync calls between 0 and 2 seconds.

• Postgres is set up using SET synchronous_commit=OFF, which delays writing to WAL by 0.6 seconds.

These settings were chosen because they somewhat balance the speed needs of a cache with the durability needs of a database. To use one or the other:

For the cache use-case

Comment-out the Redis config lines in setupRedis in /bench/redis. Similarly configure your project's Redis setup, of course, though it's the Redis default.

For the database use-case

Comment-in the Redis config line redis.config('SET', 'appendfsync', 'always') in /bench/redis.ts. Similarly configure your project's Redis setup, of course.

Comment-out the call to ALTER DATABASE ... SET synchronous_commit=OFF; in /bench/postgres.ts. This reverts to the default (fully-persistent) Postgres setting.

Results

• Inserting data: Tie
• Paginating unindexed data: Redis is ~150% faster
• Single-index pagination: Postgres is ~50% faster
• Joint-index pagination: Postgres is ~60% faster
• Inserting and deleting data: Redis is ~25% faster

Results on Apple M1 Max, 2021:

License

MIT