serverless-offline-streams

This plugin provides support for event driven systems using Serverless Offline

Supported Features

SQS

• Bind to both real AWS SQS queues and SQS compatible MQ instances at the same time. It is recommended to include an ElasticMQ instance as part of your test stack (see the example for a sample docker-compose file.
• Auto-create SQS queues from Resource definitions
• Create SQS queues via plugin configuration
• Auto-scan lambda function definitions and bind queues to appropriate handlers
• Resolve direct arn, Ref, Fn::Import and FN::GetAtt queue bindings
• Create, Delete and purge queues on stack start (configurable)
• Poll queues for new messages and invoke bound handlers
• Remove messages from queues on success

DynamoDB

• DynamoDB Streams events
• DynamoDB Streams event filtering

DynamoDB Streams

• Bridges the DynamoDB Local plugin and Serverless Offline
• Binds DynamoDB streams event mappings to the appropriate lambdas
• Supports Event Filtering

Installation

1npm install serverless-offline-streams

In your serverless.yml:

1plugins:
2  - serverless-offline-streams

Configuration

1custom:
2  serverless-offline-streams:
3    dynamodb:
4      endpoint: http://localhost:8000 # Required - dynamodb local endpoint
5      tableNames: # Optional. See below for explanation of table names mapping 
6        tableKey1: tableName1
7        tableKey2: tableName2
8    sqs:
9      enabled: false # Optional. Whether to activate SQS queue event mappings
10      host: http://127.0.0.1:8050 # Required. Host & port of elasticmq instance  
11      
12      localQueueManagement:
13          createFromResources: true # Optional. If true, will scan defined Resources for queues and create them according to the config
14          removeOnStart: true # Optional. If true, will remove all existing queues in elasticmq on startup
15          purgeOnStart: false # Optional. If true, will purge all existing queues in elasticmq on startup
16          
17      additionalQueues: # Optional. Additional queues to create on startup
18        -  name: 'Queue1', # Queue name to bind handlers to
19           aliases: # Aliases for the queue - Useful when binding to output variables in other stacks
20             - "Queue1Alias1"
21             - "Queue1Alias2"
22           visibilityTimeout: 5 # Optional. Queue VisibilityTimeout
23           delaySeconds: 5   # Optional. Queue DelaySeconds
24           remote: # Optional. If present, the plugin assumes the queue is a real AWS queue instead of a local elasticmq one
25              queueUrl: # Required. The full URL of the queue, in AWS (in the form http(s)://sqs.REGION.amazonaws.com/ACCOUNT/QUEUE-NAME)  
26           
27      pollConfig: # Optional. See below for an explanation of how polling works
28        strategy: backoff # Optional. Either backoff or fixed-interval
29        drainQueues: false # Optional. Whether to keep retrieving messages from a queue until there are no messages, if a message is found
30        messageBatchSize: 10 # Optional. How many messages to retrieve in each batch
31        fixedIntervalMs: 5000 # Optional. Only applicable to fixed-interval strategy
32        backOffType: double # Optional. Only applicable to backoff strategy. Either double or step
33        minIntervalMs: 100 # Optional. Only applicable to backoff strategy
34        maxIntervalMs: 5000 # Optional. Only applicable to backoff strategy
35        intervalStepMs: 100 # Optional. Only applicable to backoff strategy with step backoffType

SQS Configuration

Polling

The plugin will poll all bound queues at regular intervals, based on its polling strategy (see below), attempting to retrieve messageBatchSize messages on each poll.

If at least one message is found on a queue, and drainQueues is true, then the plugin will continue to retrieve messages until the queue is empty.

Warning: If a queue receives a lot of messages and drainQueues is set to true, this may cause starvation of handling to the other queues, as the thread will be blocked attempting to drain the busier queue. It is better to allow a round robin approach where each poll interval will attempt to retrieve messageBachSize messages from each queue in turn.

Polling Strategies

The plugin supports two polling strategies:

Backoff

This is the default strategy. The plugin will first poll every minIntervalMs milliseconds. If no messages are received on any queue, the polling interval will increase, up to maxIntervalMs. If a message is received, the polling will drop back down to minIntervalMs (on the basis that more messages are likely to follow)

The way in which the polling interval increases is determined by the backoffType property:

• step - The polling interval will increase by intervalStepMs each time
• double - The polling interval will increase by doubling each time (e.g. if minIntervalMs is 100, the next interval will be 200, then 400, then 800 etc.)

Fixed Interval

The plugin will poll every fixedIntervalMs, regardless of whether messages are found or not

DynamoDB Streams Configuration

Table Names

• If you are using only string ARNs, the plugin will be able to extract the table names from the ARN.
• If you are using a Ref function, the plugin will scan your resources to locate the table
• However, if you are using a cross stack reference, you must provide a mapping in the tableNames configuration block

The entry value should be the dynamo table name and the key should be the name of the cross stack reference after interpolation. For example, given the following event mapping and having stage set to 'dev':

1handler: src/handler.handler
2events:
3- stream:
4    arn: !ImportValue ${self:provider.stage}StreamArn
5    type: dynamodb
6    batchSize: 10
7    startingPosition: TRIM_HORIZON

The import value key would resolve to "devStreamArn" so the tableNames block would be as follows:

1custom:
2  serverless-offline-dynamodb-streams-handler:
3      tableNames: 
4        devStreamArn: my-dynamo-table

Usage with Typescript

In order for the plugin to correctly locate your transpiled handlers, set the 'location' property of serverless-offline This should be compatible with a number of transpilers that put transpiled code in a separate build directory (tested with the serverless-esbuild plugin)

1custom:
2  serverless-offline:
3    location: .esbuild/.build

Filter Patterns

See the Filter Patterns Guide

Future work:

• Supporting AWS EventBridge