batch-stream-csv

Process large CSV files in batches without backpressure

The CSV is streamed in batches and the stream is paused while you perform async operations on each batch to prevent backpressure building up

Uses csv-parser under the hood

Install

$ npm install batch-stream-csv

Usage:

1const batch = require('batch-stream-csv');
2
3batch('./file.csv', batchHandler, options).then(() => console.log('All done!'))
4
5async function batchHandler (batch, progress) => {
6  console.log(batch) // batch of rows parsed as JSON, e.g. [{ a: 1, b: 2 }, {  a: 2, b: 3 }]
7
8  await db.batchInsert('table', batch) // the stream is paused while you do your inserts
9
10  console.log(`${progress * 100}% complete`) // progress value can be fed into a progress bar
11}

API:

Options:

• batchSize: number of rows to process per batch (default = 5000)
• ...see https://github.com/mafintosh/ for other options