objection-cursor

An Objection.js plugin for cursor-based pagination, AKA keyset pagination.

Using offsets for pagination is a widely popular technique. Clients tell the number of results they want per page, and the page number they want to return results from. While easy to implement and use, offsets come with a drawback: when items are written to the database at a high frequency, offset based pagination becomes unreliable. For example, if we fetch a page with 10 rows, and then 10 rows are added, fetching the second page might contain the same rows as the first page.

Cursor-based pagination works by returning a pointer to a row in the database. Fetching the next/previous page will then return items after/before the given pointer. While reliable, this technique comes with a few drawbacks itself:

• The cursor must be based on a unique column (or columns)
• The concept of pages is lost, and thus you cannot jump to a specific one

Cursor pagination is used by companies such as Twitter, Facebook and Slack, and goes well with infinite scroll elements in general.

Installation

$ npm install objection-cursor

Usage

Mixin

1const Model = require('objection').Model;
2const cursorMixin = require('objection-cursor');
3
4// Set options
5const cursor = cursorMixin({limit: 10});
6
7class Movie extends cursor(Model) {
8  ...
9}
10
11// Options are not required
12class Car extends cursorMixin(Model) {
13  ...
14}

Quick Start

1const query = Movie.query()
2  // Strict ordering is required
3  .orderBy('title')
4  .orderBy('author')
5  .limit(10);
6
7query.clone().cursorPage()
8  .then(result => {
9    // Rows 1-10
10    console.log(result.results);
11    return query.clone().cursorPage(result.pageInfo.next);
12  })
13  .then(result => {
14    // Rows 11-20
15    console.log(result.results);
16    return query.clone().previousCursorPage(result.pageInfo.previous);
17  })
18  .then(result => {
19    // Rows 1-10
20    console.log(result.results);
21  });

You have the option of returning page results as plain database row objects (as in above example), or nodes where each result is associated with a cursor of its own, or both.

1const Model = require('objection').Model;
2const cursorMixin = require('objection-cursor');
3
4// Nodes are not returned by default, so you need to enable them
5const cursor = cursorMixin({nodes: true});
6
7class Movie extends cursor(Model) {
8  ...
9}
10
11const query = Movie.query()
12  .orderBy('title')
13  .orderBy('author')
14  .limit(10);
15
16query.clone().cursorPage()
17  .then(result => {
18    // Rows 1-10 with associated cursors
19    console.log(result.nodes);
20
21    // Let's take the second node
22    const node = result.nodes[1];
23
24    // result.nodes[1].data is equivalent to result.results[1]
25    console.log(result.nodes[1].data);
26
27    // You can get results before/after this row by using node.cursor
28    return query.clone().cursorPage(node.cursor);
29  });

Passing a reference builder to orderBy is supported. Raw queries, however, are not.

1const query = Movie.query()
2  .joinEager('director')
3  .orderBy(ref('director.name'))
4  // Order by a JSON field of an eagerly joined relation
5  .orderBy(ref('director.born:time').castText())
6  .orderBy('id')
7  ...

That doesn't mean raw queries aren't supported at all. You do need to use a special function for this though, called orderByExplicit (because orderByRaw was taken...)

1const {raw} = require('objection');
2
3const query = Movie.query()
4
5  // Coalesce null values into empty string
6  .orderByExplicit(raw('COALESCE(??, ?)', ['alt_title', '']))
7
8  // Same as above
9  .orderByExplicit(raw('COALESCE(??, ?)', ['alt_title', '']), 'asc')
10
11  // Works with reference builders and strings
12  .orderByExplicit(ref('details:completed').castText(), 'desc')
13
14   // Reference builders can be used as part of raw queries
15  .orderByExplicit(raw('COALESCE(??, ??, ?)', ['even_more_alt_title', ref('alt_title'), raw('?', '')]))
16
17   // Sometimes you need to go deeper...
18  .orderByExplicit(
19    raw('CASE WHEN ?? IS NULL THEN ? ELSE ?? END', ['alt_title', '', 'alt_title'])
20    'asc',
21
22    /* Since this is a cursor plugin, we need to compare actual values that are encoded in the cursor.
23     * `orderByExplicit` needs to know how to compare a column to a value, which isn't easy to guess
24     * when you're throwing raw queries at it. By default the callback's return value is similar to the
25     * column raw query, except the first binding is changed to the value. If this guess would be incorrect,
26     * you need to specify the compared value manually.
27     */
28    value => value || ''
29  )
30
31   // And deeper...
32  .orderByExplicit(
33    raw('CONCAT(??, ??)', ['id', 'title'])
34    'asc',
35
36    /* You can return a string, ReferenceBuilder, or a RawBuilder in the callback. This is useful
37     * when you need to use values from other columns.
38     */
39    value => raw('CONCAT(??, ?)', ['id', value]),
40
41    /* By default the first binding in the column raw query (after column name mappers) is used to
42     * access the relevant value from results. For example, in this case we say value = result['title']
43     * instead of value = result['id'].
44     */
45    'title'
46  )
47  .orderBy('id')
48  ...

Cursors ordered by nullable columns won't work out-of-the-box. For this reason the mixin also introduces an orderByCoalesce method, which you can use to treat nulls as some other value for the sake of comparisons. Same as orderBy, orderByCoalesce supports reference builders, but not raw queries.

Deprecated! Use orderByExplicit instead.

1const query = Movie.query()
2  .orderByCoalesce('alt_title', 'asc', '') // Coalesce null values into empty string
3  .orderByCoalesce('alt_title', 'asc') // Same as above
4  .orderByCoalesce('alt_title', 'asc', [null, 'hello']) // First non-null value will be used
5  .orderByCoalesce(ref('details:completed').castText(), 'desc') // Works with refs
6   // Reference builders and raw queries can be coalesced to
7  .orderByCoalesce('even_more_alt_title', 'asc', [ref('alt_title'), raw('?', '')])
8  .orderBy('id')
9  ...

API

Plugin

cursor(options | Model)

You can setup the mixin with or without options.

Example (with options):

1const Model = require('objection').Model;
2const cursorMixin = require('objection-cursor');
3
4const cursor = cursorMixin({
5  limit: 10,
6  pageInfo: {
7    total: true,
8    hasNext: true
9  }
10});
11
12class Movie extends cursor(Model) {
13  ...
14}
15
16Movie.query()
17  .orderBy('id')
18  .cursorPage()
19  .then(res => {
20    console.log(res.results.length) // 10
21    console.log(res.pageInfo.total) // Some number
22    console.log(res.pageInfo.hasNext) // true
23
24    console.log(res.pageInfo.remaining) // undefined
25    console.log(res.pageInfo.hasPrevious) // undefined
26  });

Example (without options):

1const Model = require('objection').Model;
2const cursorMixin = require('objection-cursor');
3
4class Movie extends cursorMixin(Model) {
5  ...
6}

CursorQueryBuilder

cursorPage([cursor, [before]])

• cursor - A URL-safe string used to determine after/before which element items should be returned.
• before - When true, return items before the one specified in the cursor. Use this to "go back".
  • Default: false.

Response format:

1{
2  results: // Page results
3  nodes: // Page results where each result also has an associated cursor
4  pageInfo: {
5    next: // Provide this in the next `cursorPage` call to fetch items after current results.
6    previous: // Provide this in the next `previousCursorPage` call to fetch items before current results.
7
8    hasMore: // If `options.pageInfo.hasMore` is true.
9    hasNext: // If `options.pageInfo.hasNext` is true.
10    hasPrevious: // If `options.pageInfo.hasPrevious` is true.
11    remaining: // If `options.pageInfo.remaining` is true. Number of items remaining (after or before `results`).
12    remainingBefore: // If `options.pageInfo.remainingBefore` is true. Number of items remaining before `results`.
13    remainingAfter: // If `options.pageInfo.remainingAfter` is true. Number of items remaining after `results`.
14    total: // If `options.pageInfo.total` is true. Total number of available rows (without limit).
15  }
16}

nextCursorPage([cursor])

Alias for cursorPage, with before: false.

previousCursorPage([cursor])

Alias for cursorPage, with before: true.

orderByCoalesce(column, [direction, [values]])

Deprecated: use orderByExplicit instead.

Use this if you want to sort by a nullable column.

• column - Column to sort by.
• direction - Sort direction.
  • Default: asc
• values - Values to coalesce to. If column has a null value, treat it as the first non-null value in values. Can be one or many of: string, number, ReferenceBuilder or RawQuery.
  • Default: ['']

orderByExplicit(column, [direction, [compareValue], [property]])

Use this if you want to sort by a RawBuilder.

• column - Column to sort by. If this is not a RawBuilder, compareValue and property will be ignored.
• direction - Sort direction.
  • Default: asc
• compareValue callback - Callback is called with a value, and should return one of string, number, ReferenceBuilder or RawQuery. The returned value will be compared against column when determining which row to show results before/after. See this code comment for more details.
• property - Values will be encoded inside cursors based on ordering, and for this reason orderByExplicit needs to know how to access the related value in the resulting objects. By default the first argument passed to the column raw builder will be used, but if for some reason this guess would be wrong, you need to specify here how to access the value.

When do I need to use compareValue?

Consider the following case, where we use a CASE statement instead of COALESCE to coalesce null values to empty strings

1Movie.query()
2  .orderByExplicit(
3    raw('CASE WHEN ?? IS NULL THEN ? ELSE ?? END', ['title', '', 'title']),
4    'desc',
5    value => value || ''
6  )
7  ...

In this case we have two reasons to use compareValue. One is that the column raw query uses the title column reference more than once. The other is that we would need to modify the statement slightly, at least in PostgreSQL's case (otherwise you would run into this).

When do I need to use property?

When the property name in your result is different than the first binding in your column raw query. For example, if your model's result structure is something like

1{
2  id: 1,
3  title: 'Hello there',
4  author: 'Somebody McSome'
5}

and your query looks like

1Movie.query()
2  .orderByExplicit(raw(`COALESCE(??, '')`, 'date'))
3  ...

you would need to use the property argument, because there is no date property in the result. This might happen if you use $parseDatabaseJson in your model, for example. Below is an example of using property argument together with $parseDatabaseJson.

1class Movie extends cursor(Model) {
2  $parseDatabaseJson(json) {
3    json = super.$parseDatabaseJson(json);
4
5    // Rename `title` to `newTitle`
6    json.newTitle = json.title;
7    delete json.title;
8
9    return json;
10  }
11}
12
13Movie.query()
14  .orderByExplicit(raw(`COALESCE(??, '')`, 'title'), 'asc', 'newTitle')
15  ....

When do I need to use both?

Basically when the column binding in your column raw query is not the first binding, or if criteria for needing to use both is met for some other reason (see the previous two subchapters). Consider the following example

1Movie.query()
2  .orderByExplicit(
3    raw('CONCAT(?::TEXT, ??)', ['the ', 'title']),
4    'asc',
5    val => raw('CONCAT(?::TEXT, ?::TEXT)', ['the ', val]),
6    'title'
7  )
8  ...

Here we are concatenating "the " in front of the movie title. Here we need both compareValue and property, because title is not the first binding in the column raw query (instead "the " is).

Options

Values shown are defaults.

1{
2  limit: 50, // Default limit in all queries
3  results: true, // Page results
4  nodes: true, // Page results where each result also has an associated cursor
5  pageInfo: {
6    // When true, these values will be added to `pageInfo` in query response
7    total: false, // Total amount of rows
8    remaining: false, // Remaining amount of rows in *this* direction
9    remainingBefore: false, // Remaining amount of rows before current results
10    remainingAfter: false, // Remaining amount of rows after current results
11    hasMore: false, // Are there more rows in this direction?
12    hasNext: false, // Are there rows after current results?
13    hasPrevious: false, // Are there rows before current results?
14  }
15}

Notes

• pageInfo.total requires additional query (A)
• pageInfo.remaining requires additional query (B)
• pageInfo.remainingBefore requires additional queries (A, B)
• pageInfo.remainingAfter requires additional queries (A, B)
• pageInfo.hasMore requires additional query (B)
• pageInfo.hasNext requires additional queries (A, B)
• pageInfo.hasPrevious requires additional queries (A, B)

remaining vs remainingBefore and remainingAfter:

remaining only tells you the remaining results in the current direction and is therefore less descriptive as remainingBefore and remainingAfter combined. However, in cases where it's enough to know if there are "more" results, using only the remaining information will use one less query than using either of remainingBefore or remainingAfter. Similarly hasMore uses one less query than hasPrevious, and hasNext.

However, if total is used, then using remaining no longer gives you the benefit of using one less query.