Installations

npm install gulp-etl-tap-csv

Developer Guide

BETA

Typescript

No

Module System

CommonJS

Node Version

18.16.0

NPM Version

9.5.1 Score

28.5

Supply Chain

81.5

Quality

68.3

Maintenance

50

Vulnerability

97.3

License

Pull Requests

Open

1

Total

1

Closed

0

Issues

Open

0

Total

0

Closed

0

Releases

Unable to fetch releases

Contributors

1

Unable to fetch Contributors

View all 1 contributors

Languages

3

TypeScript

HTML

JavaScript

TypeScript (72.53%)

HTML (17.99%)

JavaScript (9.48%)

Developer

gulpetl

Download Statistics

Total Downloads

2,307

Last Day

1

Last Week

4

Last Month

29

Last Year

720

GitHub Statistics

31 Commits

2 Forks

2 Watching

2 Branches

1 Contributors

Maintainers

1

Package Meta Information

Latest Version

1.0.11

Package Id

gulp-etl-tap-csv@1.0.11

Unpacked Size

39.61 kB

Size

9.44 kB

File Count

8

NPM Version

9.5.1

Node Version

18.16.0

Publised On

26 Aug 2024

Total Downloads

Cumulative downloads

Total Downloads

2,307

Last day

0%

1

Compared to previous day

Last week

-33.3%

4

Compared to previous week

Last month

-29.3%

29

Compared to previous month

Last year

39.3%

720

Compared to previous year

Daily Downloads

Weekly Downloads

Monthly Downloads

Yearly Downloads

Dependencies

11

@gulpetl/node-red-core csv-parse from2 gulp-error-handle loglevel pkginfo plugin-error replace-ext stream-transform through2 vinyl

Dev Dependencies

11

@types/jest @types/node @types/replace-ext @types/vinyl @types/vinyl-paths gulp jest jest-spec-reporter ts-jest ts-node typescript

Versions

gulp-etl-tap-csv

This plugin converts CSV files to gulp-etl Message Stream files; originally adapted from the gulp-etl-handlelines model plugin. It is a gulp-etl wrapper for csv-parse.

This is a gulp-etl plugin, and as such it is a gulp plugin. gulp-etl plugins work with ndjson data streams/files which we call Message Streams and which are compliant with the Singer specification. In the gulp-etl ecosystem, taps tap into an outside format or system (in this case, a CSV file) and convert their contents/output to a Message Stream, and targets convert/output Message Streams to an outside format or system. In this way, these modules can be stacked to convert from one format or system to another, either directly or with tranformations or other parsing in between. Message Streams look like this:

1{"type": "SCHEMA", "stream": "users", "key_properties": ["id"], "schema": {"required": ["id"], "type": "object", "properties": {"id": {"type": "integer"}}}}
2{"type": "RECORD", "stream": "users", "record": {"id": 1, "name": "Chris"}}
3{"type": "RECORD", "stream": "users", "record": {"id": 2, "name": "Mike"}}
4{"type": "SCHEMA", "stream": "locations", "key_properties": ["id"], "schema": {"required": ["id"], "type": "object", "properties": {"id": {"type": "integer"}}}}
5{"type": "RECORD", "stream": "locations", "record": {"id": 1, "name": "Philadelphia"}}
6{"type": "STATE", "value": {"users": 2, "locations": 1}}

Usage

gulp-etl plugins accept a configObj as the first parameter; the configObj will contain any info the plugin needs. For this plugin the configObj is the "Options" object for csv-parse, described here; the only differences are around the columns option.

If its columns option is false csv-parse returns arrays for each row instead of objects, but the Singer spec specifies objects ("JSON maps") for the record property. So, we default to true (which tries to auto-discover column names from first line, and returns lines as objects), and if you set columns to false, the arrays returned will be converted to objects: e.g. ["valueA", "valueB"] becomes {"0":"valueA", "1":"valueB"}.

Sample gulpfile.js

1
2/* parse all .CSV files in a folder into Message Stream files in a different folder */
3
4let gulp = require('gulp')
5var rename = require('gulp-rename')
6var tapCsv = require('gulp-etl-tap-csv').tapCsv
7
8exports.default = function() {
9    return gulp.src('data/*.csv')
10    .pipe(tapCsv({ columns:true }))
11    .pipe(gulp.dest('output/'));
12}
13

gulp-data

This plugin supports the use of the gulp-data api for passing in its configObj parameter. This allows data/options from the pipeline to be used to create options passed to this plugin when it runs.

See the demonstration in debug/gulpfile.ts for usage examples.

Node-RED

Node-RED is a low-code, visual programming environment for event-driven applications. Install this node under Manage Palette, look for gulp-etl-tap-csv

Demo flow

This demo builds upon the more basic demo flow for gulp-etl-target-csv; here we'll use this plugin to parse a CSV file into gulp-etl's Message Stream format, and then use gulp-etl-target-csv (npm) to convert it right back; a nice round trip. As you see, all of the nodes except gulp-etl-tap-csv and gulp-etl-target-csv are standard nodes that ship with Node-RED.

CSV data

For our demo, we'll save this CSV file to the testdata subfolder of our starting folder--which is where we ran node-red. Or we could just use a full path, which is actually recommended...

1carModel,price,color
2"Audi",10000,"blue"
3"BMW",15000,"red"
4"Mercedes",20000,"yellow"
5"Porsche",30000,"green"

Set config

The config is also set in a Template node. Identical to the configObj above, it is the "Options" object for csv-parse, described here, and you can set any of the properties described.

Config template

Results

When we run, we'll watch the Debug console; we can see the data parsed into as Message Stream. This intermediate format is used by all gulp-etl plugins/nodes, and allows for smooth interactions between them all.

1{"type":"RECORD","stream":"cars","record":{"carModel":"Audi","price":"10000","color":"blue"}}
2{"type":"RECORD","stream":"cars","record":{"carModel":"BMW","price":"15000","color":"red"}}
3{"type":"RECORD","stream":"cars","record":{"carModel":"Mercedes","price":"20000","color":"yellow"}}
4{"type":"RECORD","stream":"cars","record":{"carModel":"Porsche","price":"30000","color":"green"}}

And when it finishes, the data should be back in CSV format again!

Flow code

Copy the demo flow for import in Node-RED under Import:

1[{"id":"bd15ad43f5227ed0","type":"file in","z":"cd67672e19ab739a","name":"","filename":"./testdata/cars.csv","filenameType":"str","format":"utf8","chunk":false,"sendError":false,"encoding":"none","allProps":false,"x":330,"y"
2:380,"wires":[["57240500756eadd1","bd4928a1fe33a509"]]},{"id":"5a22ef69893bbc79","type":"inject","z":"cd67672e19ab739a"
3,"name":"click to start","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false
4,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":130,"y":340,"wires":[["bd15ad43f5227ed0"]]},{"id"
5:"7e1608b8b0ea106d","type":"gulpetl-target-csv","z":"cd67672e19ab739a","name":"","x":330,"y":540,"wires"
6:[["bd4928a1fe33a509"]]},{"id":"16687292eb745dab","type":"template","z":"cd67672e19ab739a","name":"Set config","field"
7:"config","fieldType":"msg","format":"json","syntax":"plain","template":"{\n    \"quoted\":false,\n    \"header\":true\n}",
8"output":"json","x":120,"y":540,"wires":[["7e1608b8b0ea106d"]]},{"id":"5e5268849aacb118","type":"gulpetl-tap-csv","z"
9:"cd67672e19ab739a","name":"","x":320,"y":460,"wires":[["16687292eb745dab","bd4928a1fe33a509"]]},{"id"
10:"57240500756eadd1","type":"template","z":"cd67672e19ab739a","name":"Set config","field":"config","fieldType":"msg"
11,"format":"json","syntax":"plain","template":"{\n    \"columns\":true\n}","output":"json","x":120,"y":460,"wires"
12:[["5e5268849aacb118"]]},{"id":"bd4928a1fe33a509","type":"debug","z":"cd67672e19ab739a","name":"debug (msg object)"
13,"active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","statusVal":""
14,"statusType":"auto","x":570,"y":440,"wires":[]}]

Quick Start for Coding on This Plugin

Dependencies:
git
nodejs - At least v6.3 (6.9 for Windows) required for TypeScript debugging
npm (installs with Node)
typescript - installed as a development dependency
Clone this repo and run npm install to install npm packages
Debug: with VScode use Open Folder to open the project folder, then hit F5 to debug. This runs without compiling to javascript using ts-node

Compile to javascript: npm run build

Note: This document is written in Markdown. We like to use Typora and Markdown Preview Plus for our Markdown work..

No vulnerabilities found.

10

Binary-Artifacts

Determines if the project has generated executable (binary) artifacts in the source repository.

10

License

Determines if the project has defined a license.

0

Code-Review

Determines if the project requires human code review before pull requests (aka merge requests) are merged.

0

Maintained

Determines if the project is "actively maintained".

0

SAST

Determines if the project uses static code analysis.

0

CII-Best-Practices

Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.

0

Security-Policy

Determines if the project has published a security policy.

0

Fuzzing

Determines if the project uses fuzzing.

0

Branch-Protection

Determines if the default and release branches are protected with GitHub's branch protection settings.

0

Vulnerabilities

Determines if the project has open, known unfixed vulnerabilities.

Score

1.7

/10

Last Scanned on 2024-12-30

The Open Source Security Foundation is a cross-industry collaboration to improve the security of open source software (OSS). The Scorecard provides security health metrics for open source projects.

Learn More

Other packages similar to gulp-etl-tap-csv

gulp-etl-tap-csv

Installations

Developer Guide

No

CommonJS

18.16.0

9.5.1

Score

Pull Requests

1

1

0

Issues

0

0

0

Releases

Unable to fetch releases

Contributors

Unable to fetch Contributors

Languages

Developer

gulpetl

Download Statistics

GitHub Statistics

Maintainers

Package Meta Information

Total Downloads

2,307

Daily Downloads

Weekly Downloads

Monthly Downloads

Yearly Downloads

Dependencies

Dev Dependencies

gulp-etl-tap-csv

Usage

Sample gulpfile.js

gulp-data

Node-RED

Demo flow

CSV data

Set config

Results

Flow code

Quick Start for Coding on This Plugin

10

10

0

0

0

0

0

0

0

0

1.7

/10