@Gapi Advanced Serverless Sequelize

@Nginx, @Rabbitmq, @Postgres, @Sequelize, @Docker, @Graphql

This is advanced example serverless project related with GAPI to check basic example project go to basic-example

Included @gapi/sequelize module

To start developing clone repository

1git clone https://github.com/Stradivario/gapi-starter-serverless-sequelize

Better use command line utility(gapi) to install it type following command:

1npm i -g @gapi/cli

Type the following command to create new project from scratch via CLI

1gapi new my-project --serverless-sequelize

To start project for "development" type:

1npm start

To deploy to staging or production type:

This will run Parcel and will Bundle your lambdas to single file inside dist folder example: src/user/main.ts will be bundled inside dist/user/main.js with main.map example2 src/auth/main.ts will be bundled inside dist/auth/main.js with main.map Then the lambdas will be deployed based on your serverless.yml configuration

staging

1npm run deploy:serverless

production

1npm run deploy:serverless:prod

Adding microservices inside src/app/app-proxy.module.ts

1import { Module } from '@gapi/core';
2import { CoreModule } from './core/core.module';
3import { GapiMicroserviceModule } from '@gapi/microservices';
4
5@Module({
6    imports: [
7        CoreModule,
8        GapiMicroserviceModule.forRoot([
9            {
10                name: 'auth-microservice',
11                link: `${process.env.AWS_CLOUD_LINK}/auth/graphql`
12            },
13            {
14                name: 'user-microservice',
15                link: `${process.env.AWS_CLOUD_LINK}/user/graphql`
16            }
17        ])
18    ]
19})
20export class AppProxyModule { }

Environment AWS_CLOUD_LINK you can set inside gapi-cli.conf.yml for specific environment

1    lambda-proxy-production:
2      AWS_CLOUD_LINK: 'https://lpsm25eo1k.execute-api.us-east-2.amazonaws.com/prod'
3    
4    lambda-proxy-staging:
5      AWS_CLOUD_LINK: 'https://aiey6imalh.execute-api.us-east-2.amazonaws.com/staging'

To start @gapi as a proxy server using 2 created lambda microservices

More info inside gapi-cli.conf.yml

It will run environment lambda-proxy-staging from gapi-cli.conf.yml

1gapi proxy staging

It will run environment lambda-proxy-production from gapi-cli.conf.yml

1gapi proxy prod

Serverless.yml configuration

1service: gapi-sequelize-serverless
2provider:
3  name: aws
4  runtime: nodejs8.10
5  stage: staging
6  profile: default
7  region: us-east-2
8
9functions:
10  auth:
11    handler: src/app/auth/main.handler
12    events:
13      - http:
14          path: "/auth/{proxy+}"
15          method: any
16          cors: true
17          integration: lambda-proxy
18    memorySize: 128
19    timeout: 10
20  user:
21    handler: src/app/user/main.handler
22    events:
23      - http:
24          path: "/user/{proxy+}"
25          method: any
26          cors: true
27          integration: lambda-proxy
28    memorySize: 128
29    timeout: 10
30
31plugins:
32  - serverless-offline
33

Testing shows that Gapi is using 85 MB lambda memory so i decided to limit them to 128 mb if you use it well it will treat you well :)

Important!!! To deploy to AWS with Serverless you need to compile project to ES6 commonjs module everything is preconfigurated inside tsconfig.json

Compiling project to ES6 commonjs

1tsc

Simple as that :)

Bundle will be somewhere between 18-20 mb.The basic concept is that you can create many functions related with same project run them as SingleServer but when you deploy them every module is independent microservice.In the end you create PROXY server with Microservices which MERGES Both Microservices Lambdas as a Single Endpoint GraphQL

Njoy! :)

Testing

To start developing with testing GAPI uses JEST and gapi-cli is preconfigurated for your needs! :)

To run single test type:

1gapi test

Testing watch mode

Note: You need to start server before running tests

Note: Everytime you make change to server it will restart server and execute tests

Note: To add more tests just create e2e.spec.ts or unit.spec.ts somewhere inside the application

Start the application

1gapi start

Execute test with --watch argument

1gapi test --watch

You will end up with something like this

Custom logic before testing ( for example creating MOCK users to database before testing)

Create file test.ts inside root/src/test.ts with this content

Everytime you run test with --before argument it will set environment variable BEFORE_HOOK

1  if (process.env.BEFORE_HOOK) {
2    // do something here
3  }

Then execute tests with --before

1gapi test --before

This command will start root/src/test.ts file and will wait for process.exit(0) so you can customize your before logic check this link for reference

Following commands will start RabbitMQ, PostgreSQL, API, NGINX as a services you need DOCKER for them

API will be served on https://localhost:80 and https://localhost:80/subscriptions

Docker

To build project with Docker type:

1gapi app build

To start project with Docker type:

1gapi app start

To stop project type:

1gapi app stop

Workers

All workers will be mapped as Proxy and will be reverted to https://localhost:80 and https://localhost:80/subscriptions

So you don't have to worry about if some of your workers stopped responding

TODO: Create monitoring APP for all workers and main API

To start workers type:

1gapi workers start

To stop workers type:

1gapi workers stop

To add more workers

By default there are 4 workers with 4 processes with "exec_mode: cluster" of the original process inside single docker container

You can control Processes inside single docker container from "root/process.yml" file.

1apps:
2  - script   : './src/main.ts'
3    name     : 'APP'
4    exec_mode: 'cluster'
5    instances: 4
6

To map new worker as a stream open root/nginx/config/private/default

1upstream app_servers {
2    server 182.10.0.3:9000; # Main process
3    server 182.10.0.21:9000; # Worker 1
4    server 182.10.0.22:9000; # Worker 2
5    server 182.10.0.23:9000; # Worker 3
6    server 182.10.0.24:9000; # Worker 4
7
8    # Add more workers here
9    # server 182.10.0.25:9000; # Worker 5
10}
11
12server {
13    listen 80;
14    server_name api.yourdomain.com;
15    access_log api-yourdomain.access.log;
16
17    location / {
18       proxy_set_header Upgrade $http_upgrade;
19       proxy_set_header Connection "upgrade";
20       client_max_body_size 50M;
21       proxy_set_header Host $http_host;
22       proxy_set_header X-Real-IP $remote_addr;
23       proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
24       proxy_set_header X-Forwarded-Proto $scheme;
25       proxy_set_header X-Frame-Options SAMEORIGIN;
26       proxy_buffers 256 16k;
27	     proxy_buffering off;
28       proxy_buffer_size 16k;
29       proxy_read_timeout 600s;
30       proxy_pass http://app_servers;
31    }
32
33    location /subscriptions {
34         # prevents 502 bad gateway error
35        proxy_buffers 8 32k;
36        proxy_buffer_size 64k;
37
38        # redirect all HTTP traffic to localhost:9000;
39        proxy_pass http://app_servers/subscriptions;
40        proxy_set_header X-Real-IP $remote_addr;
41        proxy_set_header Host $http_host;
42        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
43        #proxy_set_header X-NginX-Proxy true;
44
45        # enables WS support
46        proxy_http_version 1.1;
47        proxy_set_header Upgrade $http_upgrade;
48        proxy_set_header Connection "upgrade";
49	      proxy_buffering off;
50        proxy_read_timeout 999999999;
51
52    }
53    if ($scheme = http) {
54       return 301 https://$server_name$request_uri;
55    }
56    listen 443;
57    ssl on;
58    ssl_certificate         /usr/share/certs/cert.pem;
59    ssl_certificate_key     /usr/share/certs/cert.key;
60}
61
62

When you add another worker it should be on different IP with same port 9000

Open root/gapi.conf.yml file you will find this file:

1config:
2# Application configuration
3  app: 
4    local:
5      API_PORT: 9000
6      API_CERT: ./cert.key
7      NODE_ENV: development
8      AMQP_HOST: 182.10.0.5
9      AMQP_PORT: 5672
10      DB_PORT: 5432
11      DB_NAME: postgres
12      DB_HOST: 182.10.0.4
13      DB_USERNAME: dbuser
14      DB_PASSWORD: dbuserpass
15      GRAPHIQL: true
16      ENDPOINT_TESTING: http://localhost:9000/graphql
17      TOKEN_TESTING: eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJlbWFpbCI6ImtyaXN0aXFuLnRhY2hldkBnbWFpbC5jb20iLCJzY29wZSI6WyJBRE1JTiJdLCJpZCI6MSwiaWF0IjoxNTE2OTk2MzYxfQ.7ANr5VHrViD3NkCaDr0nSWYwk46UAEbOwB52pqye4AM
18      GRAPHIQL_TOKEN: eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJlbWFpbCI6ImtyaXN0aXFuLnRhY2hldkBnbWFpbC5jb20iLCJpZCI6MSwic2NvcGUiOlsiQURNSU4iXSwiaWF0IjoxNTIwMjkxMzkyfQ.9hpIDPkSiGvjTmUEyg_R_izW-ra2RzzLbe3Uh3IFsZg
19    prod:
20      API_PORT: 9000
21      API_CERT: ./cert.key
22      NODE_ENV: production
23      AMQP_HOST: 182.10.0.5
24      AMQP_PORT: 5672
25      DB_PORT: 5432
26      DB_HOST: 182.10.0.4
27      DB_USERNAME: dbuser
28      DB_PASSWORD: dbuserpass
29      DB_NAME: postgres
30# Testing configuration for local(dev) or worker(running tests as a separate worker with separate database)
31  test: 
32    local: extends app/local
33    worker:
34      API_PORT: 9000
35      API_CERT: ./cert.key
36      NODE_ENV: production
37      DB_PORT: 5432
38      DB_HOST: 182.10.0.99
39      AMQP_HOST: 182.10.0.5
40      AMQP_PORT: 5672
41      DB_USERNAME: dbuser
42      DB_PASSWORD: dbuserpass
43      DB_NAME: postgres
44      ENDPOINT_TESTING: http://182.10.0.101:9000/graphql
45      TOKEN_TESTING: eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJlbWFpbCI6ImtyaXN0aXFuLnRhY2hldkBnbWFpbC5jb20iLCJzY29wZSI6WyJBRE1JTiJdLCJpZCI6MSwiaWF0IjoxNTE2OTk2MzYxfQ.7ANr5VHrViD3NkCaDr0nSWYwk46UAEbOwB52pqye4AM
46  schema:
47    introspectionEndpoint: http://localhost:9000/graphql
48    introspectionOutputFolder: ./src/app/core/api-introspection
49
50commands:
51  testing:
52    stop:
53      - docker rm -f gapi-api-prod-worker-tests-executor
54      - docker rm -f gapi-api-prod-worker-tests-provider
55    start:
56      - gapi testing start-provider
57      - sleep 10
58      - gapi testing start-executor
59      - echo Cleaning...
60      - gapi testing stop
61    start-executor:
62      - docker run -d --network=gapiapiprod_gapi --ip=182.10.0.100 --name gapi-api-prod-worker-tests-executor gapi/api/prod
63      - docker exec gapi-api-prod-worker-tests-provider npm -v
64      - gapi test --worker --before
65    start-provider: docker run -d --network=gapiapiprod_gapi --ip=182.10.0.101 --name gapi-api-prod-worker-tests-provider gapi/api/prod
66  workers:
67    start:
68      - gapi workers start-1
69      - gapi workers start-2
70      - gapi workers start-3
71      - gapi workers start-4
72    stop:
73      - docker rm -f gapi-api-prod-worker-1
74      - docker rm -f gapi-api-prod-worker-2
75      - docker rm -f gapi-api-prod-worker-3
76      - docker rm -f gapi-api-prod-worker-4
77    start-1: docker run -d --network=gapiapiprod_gapi --ip=182.10.0.21 --name gapi-api-prod-worker-1 gapi/api/prod
78    start-2: docker run -d --network=gapiapiprod_gapi --ip=182.10.0.22 --name gapi-api-prod-worker-2 gapi/api/prod
79    start-3: docker run -d --network=gapiapiprod_gapi --ip=182.10.0.23 --name gapi-api-prod-worker-3 gapi/api/prod
80    start-4: docker run -d --network=gapiapiprod_gapi --ip=182.10.0.24 --name gapi-api-prod-worker-4 gapi/api/prod
81    example-worker-with-port: docker run -d --network=gapiapiprod_gapi --ip=182.10.0.25 --name gapi-api-prod-worker-5 -p 9001:9000 gapi/api/prod
82  app:
83    start:
84      - docker-compose -p gapi-api-prod up --force-recreate -d
85      - gapi rabbitmq enable-dashboard
86    stop:
87      - gapi nginx stop
88      - gapi api stop
89      - gapi rabbitmq stop
90      - gapi postgres stop
91    build: docker build -t gapi/api/prod .
92  api:
93    stop: docker rm -f gapi-api-prod
94  nginx:
95    stop: docker rm -f gapi-api-nginx
96  postgres:
97    stop: docker rm -f gapi-api-postgres
98  rabbitmq:
99    stop: docker rm -f gapi-api-rabbitmq
100    restart: docker restart gapi-api-rabbitmq
101    enable-dashboard: docker exec gapi-api-rabbitmq rabbitmq-plugins enable rabbitmq_management
102
103# You can define your custom commands for example 
104# commands:
105#   your-cli:
106#     my-command: 'npm -v'
107# This command can be executed as "gapi your-cli my-command"

Adding one more worker:

1start-5: 'docker run -d --network=gapiapiprod_gapi --ip=182.10.0.25 --name gapi-api-prod-worker-5 -p 9005:9000 gapi/api/prod'

Then edit start task inside workers to start new worker 5

1start: 'gapi workers start-1 && gapi workers start-2 && gapi workers start-3 && gapi workers start-4 & gapi workers start-5'

Thats' it!! Now you have 4 processes like CLUSTERS inside 1 docker container with ip 182.10.0.25 and external port(optional) 9005;

You can specify worker also without port because all workers are inside internal network called "gapiapiprod_gapi"

182.10.0.21/22/23/24/25:9000

1start-5: 'docker run -d --network=gapiapiprod_gapi --ip=182.10.0.25 --name gapi-api-prod-worker-5 gapi/api/prod'

If you want to change port forwarding to another port you need to set just nginx configuration:

1  nginx:
2    image: sameersbn/nginx:1.10.1-5
3    ports:
4      - "81:80"
5      - "443:443"

Now you can find your API served onto https://localhost:81/ and https://localhost:81/subscriptions

All workers don't care about that because they will be served and mapped from nginx to port 80.

You can check docker-compose file to configurate environment variables

1version: '2'
2services:
3
4  nginx:
5    image: sameersbn/nginx:1.10.1-5
6    ports:
7      - "81:80"
8      - "444:443"
9    volumes:
10      - ./nginx/config:/etc/nginx
11      - ./nginx/html:/usr/share/nginx/html/
12      - ./nginx/certs:/usr/share/certs
13    restart: always
14    container_name: gapi-api-nginx
15    networks:
16      gapi:
17        ipv4_address: 182.10.0.2
18
19  api:
20    image: gapi/api/prod:latest
21    ports:
22      - "9000"
23    environment:
24      - NODE_ENV=production
25      - API_PORT=9000
26      - GRAPHIQL_TOKEN=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJlbWFpbCI6InRhNGVuc3RvcmVAZ21haWwuY29tIiwic2NvcGUiOlsiQURNSU4iXSwiaWQiOjIsImlhdCI6MTUxMTk3NDkzNX0.M9PnW2IrVp4XGRvbzFrl0tx6vqs6oXItFK-wF5roneI
27      - AMQP_HOST=182.10.0.5
28      - AMQP_PORT=5672
29      - CERT_PATH=
30
31      # Production db config
32      - DB_HOST=182.10.0.4
33      - DB_PORT=5432
34      - DB_NAME=postgres
35      - DB_USERNAME=dbuser
36      - DB_PASSWORD=dbuserpass
37
38    restart: always
39    mem_limit: 1000000000
40    cpu_shares: 73
41    container_name: gapi-api-prod
42    depends_on:
43      - nginx
44      - rabbitMq
45      - PostgreSQLDev
46    networks:
47      gapi:
48        ipv4_address: 182.10.0.3
49
50  PostgreSQLDev:
51    image: sameersbn/postgresql:9.5-3
52    ports:
53      - "5432"
54    environment:
55      - DEBUG=false
56      - TIMEZONE=Europe/Sofia
57      - LOCALE=bg_BG.UTF-8
58
59      - DB_USER=dbuser
60      - DB_PASS=dbuserpass
61      - DB_NAME=postgres
62      - DB_TEMPLATE=
63
64      - DB_EXTENSION=
65
66      - REPLICATION_MODE=
67      - REPLICATION_USER=
68      - REPLICATION_PASS=
69      - REPLICATION_SSLMODE=
70    restart: always
71    container_name: gapi-api-postgres
72    networks:
73      gapi:
74        ipv4_address: 182.10.0.4
75
76  PostgreSQLDevTesting:
77    image: sameersbn/postgresql:9.5-3
78    ports:
79      - "5432"
80    environment:
81      - DEBUG=false
82      - TIMEZONE=Europe/Sofia
83      - LOCALE=bg_BG.UTF-8
84
85      - DB_USER=dbuser
86      - DB_PASS=dbuserpass
87      - DB_NAME=postgres
88      - DB_TEMPLATE=
89
90      - DB_EXTENSION=
91
92      - REPLICATION_MODE=
93      - REPLICATION_USER=
94      - REPLICATION_PASS=
95      - REPLICATION_SSLMODE=
96    restart: always
97    container_name: gapi-api-postgres-testing
98    networks:
99      gapi:
100        ipv4_address: 182.10.0.99
101
102  rabbitMq:
103    image: rabbitmq:3.7.2
104    ports:
105      - "15672:15672"
106      - "5672:5672"
107      - "5671:5671"
108      - "4369:4369"
109    restart: always
110    container_name: gapi-api-rabbitmq
111    networks:
112      gapi:
113        ipv4_address: 182.10.0.5
114
115  pgadmin:
116    image: thajeztah/pgadmin4
117    ports:
118      - "5050"
119    volumes:
120      - /usr/bin/:/usr/pg
121      - /usr/database/:/usr/database
122      - /tmp/:/tmp
123    restart: always
124    container_name: gapi-api-pg-admin
125    networks:
126      gapi:
127        ipv4_address: 182.10.0.6
128
129networks:
130  gapi:
131    driver: bridge
132    ipam:
133     config:
134       - subnet: 182.10.0.0/16
135         gateway: 182.10.0.1
136
137

After successfully started project you can open your browser to localhost:80 or 182.10.0.3:9000 the api will be served there

TODO: Better documentation...

Enjoy ! :)