serverless-offline-kinesis-streams

A Serverless Plugin that creates local Kinesis streams and then triggers your Serverless Lambda Functions upon receiving new records.

It works depends on serverless-offline.

It works with:

• serverless-webpack
• serverless-typescript
• serverless-parcel

Serverless Offline Kinesis Streams also works with multiple Kinesis streams in the same Serverless file. It respects batchWindow and batchSize. You can even listen to the same Kinesis Stream with multiple Lambda functions at once with different batchWindow and batchSize values.

Installation

Add the plugin to your project

1yarn add -D serverless-offline-kinesis-streams

Then, add serverless-offline-kinesis-streams to your plugins. Example:

plugins:
  - serverless-layers
  - serverless-plugin-typescript
  - serverless-offline-kinesis-streams
  - serverless-offline

Configuration

Simple setup

1custom:
2  offlineKinesisStreams:
3    port: 4567 # Optional; 4567 is the default port. The plugin launches a local Kinesis instance on this port
4    region: local # Optional; local is the default
5    streams: # Required; Define all streams and their streamNames with number of shards.
6      - streamName: myFirstKinesisStream
7        shards: 1
8
9functions:
10  myFirstKinesisStreamHandler:
11    handler: src/myFirstKinesisStreamHandler.handler
12    events:
13      - stream:
14          enabled: true
15          type: kinesis
16          arn: arn:aws:kinesis:${self:custom.region}:*:stream/myFirstKinesisStream # Same stream name from above.

Example with multiple stream:

1custom:
2  offlineKinesisStreams:
3    port: 4567
4    region: local
5    streams:
6      - streamName: myFirstKinesisStream
7        shards: 1
8      - streamName: mySecondKinesisStream
9        shards: 1
10
11functions:
12  myFirstKinesisStreamHandler:
13    handler: src/myFirstKinesisStreamHandler.handler
14    events:
15      - stream:
16          enabled: true
17          type: kinesis
18          arn: arn:aws:kinesis:${self:custom.region}:*:stream/myFirstKinesisStream
19  mySecondKinesisStreamHandler:
20    handler: src/mySecondKinesisStreamHandler.handler
21    events:
22      - stream:
23          enabled: true
24          type: kinesis
25          arn: arn:aws:kinesis:${self:custom.region}:*:stream/mySecondKinesisStream

Example with batchWindow and batchSize:

The example below will trigger both Lambda functions when the single Kinesis Stream has records. They are independently monitored and their batchWindow & batchSize events are individual respected.

So, when records are put into the Kinesis stream, the myFirstKinesisStreamHandler will most likely pick them up first (10 records at a time). And then, every 60 seconds, alsoListensToFirstStream will grab 1 record. (This example is contrived. You'd never want such a high batchWindow and low batchSize.)

1custom:
2  offlineKinesisStreams:
3    port: 4567
4    region: local
5    streams:
6      - streamName: myFirstKinesisStream
7        shards: 1
8
9functions:
10  myFirstKinesisStreamHandler:
11    handler: src/myFirstKinesisStreamHandler.handler
12    events:
13      - stream:
14          enabled: true
15          type: kinesis
16          arn: arn:aws:kinesis:${self:custom.region}:*:stream/myFirstKinesisStream
17  alsoListensToFirstStream:
18    handler: src/mySecondKinesisStreamHandler.handler
19    events:
20      - stream:
21          enabled: true
22          type: kinesis
23          batchWindow: 60 # Get records every 60 seconds
24          batchSize: 1 # Get one record at a time
25          arn: arn:aws:kinesis:${self:custom.region}:*:stream/myFirstKinesisStream

Thanks

Features were extended off of this implementation of a local Kinesis

Thanks to mhart for creating kinesalite