batched-items-accumulator

The BatchedAccumulator class is a lightweight utility for Node.js projects that accumulates items into fixed-size batches (number-of-items wise), preserving insertion order. It streams items directly into batches during runtime:ocean:, avoiding the overhead of post-processing a 1D array. This abstraction lets users focus on application logic without worrying about batch management.

Ideal for delayed processing tasks such as bulk write operations to databases, blob storage, and batched publishing of Kafka messages. Delayed execution helps optimize network efficiency by reducing the number of requests while increasing their size.

Example

Given a BatchedAccumulator instance with a batch size of 4, inserting the items 1, 2, 3, 4, 5, 6, 7 results in the following batches:

• [1, 2, 3, 4] (a full batch)
• [5, 6, 7] (a partial batch with the remaining items)

Note: If you need to group items into fixed-size batches per key - for example, Kafka messages per topic or MongoDB documents per collection - to accumulate them before periodic bulk publishing or writing, consider using the keyed version of this package for improved usability: keyed-batched-items-accumulator.

Table of Contents

• Key Features
• API
• Getter Methods
• Use Case Example: Batch Upsert for MongoDB Documents
• Design Decision: No Peeking
• Breaking Change in Version 2.0.0
• License

Key Features :sparkles:

• Designed for Efficient Bulk Data Preparation :package:: Applications often accumulate data from user interactions or message queues before persisting them in bulk to storage solutions like Amazon S3, Azure Blob Storage, or a database. To reduce network overhead, items are temporarily stored in memory and written in bulk, once a sufficient number has been collected or a timeout has been reached.
• Streaming-Friendly Accumulation :ocean:: Items are accumulated into batches during runtime, eliminating the need for a post-processing step that chunks a 1D array - a common approach in other packages. Post-processing chunking adds O(n) time and space complexity, which can degrade performance when batch processing is frequent or batch sizes are large. In contrast, this package’s extractAccumulatedBatches method operates in O(1) time and space, as items are stored in batches from the start.
• Fixed-Size Batches :straight_ruler:: The push method appends the item to the most recent batch if it has not yet reached the size threshold. Otherwise, it creates a new batch. Each batch contains the same number of items, except for the last batch, which may have fewer items.
• Zero Overhead :dart:: While simple in design, this class serves as a building block for more complex solutions. It abstracts batch management, allowing users to focus on their application logic while leveraging a well-tested, efficient batching mechanism.

• State Metrics :bar_chart:: The batchesCount, isEmpty and accumulatedItemsCount getters offer real-time insights into the accumulator's state, helping users make informed decisions, such as determining whether a minimum threshold of accumulated items has been reached before extracting batches.
• Comprehensive documentation :books:: Fully documented, enabling IDEs to provide intelligent tooltips for an enhanced development experience.
• Thoroughly Tested :test_tube:: Backed by extensive unit tests, to ensure reliability in production.
• Zero Runtime Dependencies :dove:: Only development dependencies are included.
• ES2020 Compatibility: The project targets ES2020 for modern JavaScript support.
• Full TypeScript Support: Designed for seamless TypeScript integration.

API :globe_with_meridians:

The BatchedAccumulator class provides the following methods:

• push: Adds an item to the accumulator, grouping it into a batch of fixed size. If the last batch is full or no batch exists, a new batch is created.
• extractAccumulatedBatches: Extracts and returns the accumulated batches as a 2D array, where each batch is a fixed-size array of items. The last batch may contain fewer elements if the total count is not a multiple of the batch size. Calling this method transfers ownership of the extracted batches to the caller, meaning the instance will no longer retain them. The accumulator is reset, clearing its internal storage to begin a new accumulation cycle.

If needed, refer to the code documentation for a more comprehensive description of each method.

Getter Methods :mag:

The BatchedAccumulator class provides the following getter methods to reflect the current state:

• batchesCount: Returns the number of batches currently held by this instance.
• isEmpty: Indicates whether this instance has accumulated any items.
• accumulatedItemsCount: Returns the total number of accumulated items across all batches. For example, if there are 5 full batches and the batch size is 100, the output will be 500. This method is useful for determining whether a minimum threshold of accumulated items has been reached before extracting batches, helping to avoid excessively small bulk operations.

Use Case Example: Batch Upsert for MongoDB Documents :package:

In many applications, MongoDB documents originate from sources such as message queues or user interactions. Instead of upserting each document individually - potentially causing excessive network load - it is common to accumulate them in memory before performing a periodic batch flush to the database.

To account for real-life complexities, this example triggers a batch flush when either of the following conditions is met:

• The number of accumulated documents exceeds a predefined threshold.
• The time since the last flush exceeds a specified maximum interval.

This example leverages the non-overlapping-recurring-task package to ensure that multiple batches are not upserted concurrently, helping to keep network bandwidth usage under control.

1import { BatchedAccumulator } from 'batched-items-accumulator';
2import {
3  NonOverlappingRecurringTask,
4  INonOverlappingRecurringTaskOptions
5} from 'non-overlapping-recurring-task';
6import { Collection, MongoError } from 'mongodb';
7
8const FLUSH_INTERVAL_MS = 5000;
9const BATCH_SIZE = 512;
10const MIN_BATCH_SIZE_BEFORE_FLUSH = 64;
11const MAX_FLUSH_INTERVAL_MS = 60 * 1000;
12
13class PeriodicDocumentFlusher<DocumentType> {
14  private readonly _documentsAccumulator = new BatchedAccumulator<DocumentType>(BATCH_SIZE);
15  private readonly _recurringFlush: NonOverlappingRecurringTask<MongoError>;
16
17  private _lastFlushTimestamp: number = 0;
18
19  /**
20   * Injects a collection and a logger instance.  
21   * Context-aware child loggers are commonly used,  
22   * especially in Nest.js apps (e.g., pino-http). 
23   */
24  constructor(
25    private readonly _collection: Collection<DocumentType>,
26    private readonly _logger: ILogger
27  ) {
28    const recurringFlushOptions: INonOverlappingRecurringTaskOptions = {
29      intervalMs: FLUSH_INTERVAL_MS,
30      immediateFirstRun: false
31    };
32    const forceFlush = false;
33    this._recurringFlush = new NonOverlappingRecurringTask<MongoError>(
34      () => this._flushAccumulatedBatches(forceFlush),
35      recurringFlushOptions,
36      this._onUpsertError.bind(this)
37    );
38  }
39
40  public async start(): Promise<void> {
41    // Initialize with the current timestamp to mark the start of the process.
42    this._lastFlushTimestamp = Date.now();
43    await this._recurringFlush.start();
44  }
45  
46  public async stop(): Promise<void> {
47    await this._recurringFlush.stop();
48    const forceFlush = true;
49    await this._flushAccumulatedBatches(forceFlush);
50  }
51
52  public add(doc: DocumentType): void {
53    // Accumulate documents in memory for batch processing.
54    this._documentsAccumulator.push(doc);
55  }
56
57  private async _bulkUpsert(batch: DocumentType[]): Promise<void> {
58    // Implementation: Upsert a batch of accumulated documents into MongoDB.
59  }
60
61  private get _elapsedTimeSinceLastFlushMs(): number {
62    return Date.now() - this._lastFlushTimestamp;
63  }
64
65  /**
66   * Extracts accumulated document batches and upserts them sequentially.  
67   * A production-ready implementation may include per-batch error handling
68   * or retries mechanism.
69   */
70  private async _flushAccumulatedBatches(forceFlush: boolean): Promise<void> {
71    const shouldSkip =
72      !forceFlush &&
73      this._documentsAccumulator.accumulatedItemsCount < MIN_BATCH_SIZE_BEFORE_FLUSH &&
74      this._elapsedTimeSinceLastFlushMs < MAX_FLUSH_INTERVAL_MS;
75    if (shouldSkip || this._documentsAccumulator.isEmpty) {
76      return;
77    }
78
79    this._lastFlushTimestamp = Date.now();
80    const batches: DocumentType[][] = this._documentsAccumulator.extractAccumulatedBatches();
81    for (const batch of batches) {
82      await this._bulkUpsert(batch);
83    }
84  }
85
86  private _onUpsertError(error: MongoError): void {
87    this._logger.error(
88      `Batch upload failed due to MongoDB error code ${error?.code}: ${error.message}`
89    );
90  }
91}

Design Decision: No Peeking :see_no_evil:

To maintain integrity, the class does not provide direct access to accumulated items or batches. Exposing internal references could allow unintended modifications, such as appending items to a full batch. Instead, the extractAccumulatedBatches method transfers ownership of all batches to the caller while resetting the instance to a clean state. This ensures the component's guarantees remain intact and prevents accidental modifications of extracted batches.

However, while direct peeking is not possible, users can utilize the getter methods batchesCount, isEmpty, and accumulatedItemsCount to assess whether extraction is needed.

Breaking Change in Version 2.0.0 :warning:

Method accumulateItem has been renamed to push to improve fluency and readability.

License :scroll:

Apache 2.0