mongodb-collection-sample

Sample documents from a MongoDB collection.

Install

npm install --save mongodb-collection-sample

Example

npm install mongodb lodash mongodb-collection-sample

1const sample = require('mongodb-collection-sample');
2const { MongoClient } = require('mongodb');
3const _ = require('lodash');
4
5const client = new MongoClient();
6
7async function main() {
8  await client.connect('mongodb://localhost:27017');
9
10  // Generate 1000 documents
11  const docs = _range(0, 1000).map(function(i) {
12    return {
13      _id: 'needle_' + i,
14      is_even: i % 2
15    };
16  });
17
18  // Insert them into a collection.
19  await db.collection('haystack').insert(docs);
20
21  const options = {};
22  // Size of the sample to capture [default: `5`].
23  options.size = 5;
24
25  // Query to restrict sample source [default: `{}`]
26  options.query = {};
27
28  // Get a stream of sample documents from the collection.
29  const stream = sample(db, 'haystack', options);
30  stream.on('error', function(err){
31    console.error('Error in sample stream', err);
32    return process.exit(1);
33  });
34  stream.on('data', function(doc){
35    console.log('Got sampled document `%j`', doc);
36  });
37  stream.on('end', function(){
38    console.log('Sampling complete!  Goodbye!');
39    db.close();
40    process.exit(0);
41  });
42}
43
44main();

Options

Supported options that can be passed to sample(db, coll, options) are

• query: the filter to be used, default is {}
• size: the number of documents to sample, default is 5
• fields: the fields you want returned (projection object), default is null
• raw: boolean to return documents as raw BSON buffers, default is false
• sort: the sort field and direction, default is {_id: -1}
• maxTimeMS: the maxTimeMS value after which the operation is terminated, default is undefined
• promoteValues: boolean whether certain BSON values should be cast to native Javascript values or not. Default is true

How It Works

Native Sampler

MongoDB version 3.1.6 and above generally uses the $sample aggregation operator:

db.collectionName.aggregate([
  {$match: <query>},
  {$sample: {size: <size>}},
  {$project: <fields>},
  {$sort: <sort>}
])

However, if more documents are requested than are available, the $sample stage is omitted for performance optimization. If the sample size is above 5% of the result set count (but less than 100%), the algorithm falls back to the reservoir sampling, to avoid a blocking sort stage on the server.

Reservoir Sampling

For MongoDB version 3.1.5 and below we use a client-size reservoir sampling algorithm.

• Query for a stream of _id values, limit 10,000.
• Read stream of _ids and save sampleSize randomly chosen values.
• Then query selected random documents by _id.

The two modes, illustrated:

Performance Notes

For peak performance of the client-side reservoir sampler, keep the following guidelines in mind.

• The initial query for a stream of _id values must be limited to some finite value. (Default 10k)
• This query should be covered by an index
• Since there's a limit, you may wish to bias for recent documents via a sort. (Default: {_id: -1})
• Don't sort on {$natural: -1}: this forces a collection scan!

Queries that include a sort by $natural order do not use indexes to fulfill the query predicate

• When retrieving docs: batch using one $in to reduce network chattiness.

License

Apache 2