Fire-Read

Read multiple files line by line (to avoid chocking your RAM) and one after the other till all files are read!

This module automates the process of reading each line and automatically skipping to the next file using just one convenient method; read().

Example Use

1
2// require
3const fireRead = require('fire-read');
4
5// initialize
6let resp = new FireRead({
7
8    // we want to read these three log files
9	files: [
10		'/logs/info-2022-08-25.log',
11		'/logs/info-2022-08-24.log',
12		'/logs/info-2022-01-24.log',
13	],
14
15    // We also want to read one line at a time
16	lines: 1, 
17
18    // we want all line buffers decoded into ascii
19    encoding : 'ascii',
20
21    // optionally parse lines of each file read
22    // parser: JSON.parse
23
24});
25
26// loop through all lines and files
27while ((resp = resp.read())) {
28	console.log(resp);
29}
30
31console.log('Finished Reading');
32

This produces the following logs:

1{
2  files: {
3    current: '/logs/info-2022-01-24.log',
4    selected: [
5      '/logs/info-2022-08-25.log',
6      '/logs/info-2022-08-24.log',
7      '/logs/info-2022-01-24.log'
8    ]
9  },
10  fileNum: 2,
11  lineNum: 959,
12  lines: [
13    `{"level":"info","message":"hello world","timestamp":"2022-08-24 08:26:58.035 PM"}`
14  ],
15  read: [Function: bound read]
16}
17
18Finished Reading

What is Going on?

new FireRead(options}) returns an instance of the fire-read with the convenient read() method that you can use to read each line in every file given.

Every time you call read() it also wraps in an instance of itself so you can iterate through all reads till there are none to do.

So while ((resp = resp.read())) {} simply iterates through all line reads and files till there is nothing more to read, at which point it returns null.

Options

The following options are needed and expected by the module.

• files : an Array of files to read. Required.

• lines : the number (batch) of lines to read every time read() is called. Defaults to 20;

• encoding : the type encoding to use while converting the line Buffer to string. See Buffer.toString Documentation. Defaults to "ascii".

• parser : a function used to parse each line in your logs. Many file loggers like bunyan write each log line as JSON strings. Defaults to no parsing.

  So in the example above we could have enabled the JSON parser by passing JSON.parse to ensure lines are parsed as shown below.

  1{
  2    level: 'info',
  3    message: 'hello world',
  4    timestamp: '2022-08-24 09:53:53.373 PM'
  5}