@Gapi Microservices Starter

@Nginx, @Rabbitmq, @Postgres, @Sequelize, @Docker, @Graphql

This is basic microservices example project related with GAPI

To check advanced example project go to advanced-example

To start developing clone repository

1git clone https://github.com/Stradivario/gapi-starter-microservices

Better use command line utility(gapi-cli) to install it type following command:

1npm i -g @gapi/cli

Type the following command to create new project from scratch via CLI

1gapi new my-project --microservices

To start project for "development" type:

1npm start

To start project for "production" type:

This will run following commands "pm2 process.yml --only APP" (DEVELPOMENT) or "pm2-docker process.yml --only APP"(PRODUCTION inside docker) (check process.yml inside root repository)

1npm run start:prod

To stop project for "production" type:

Following command will stop pm2 processes started

1npm run stop:prod

Testing

To start developing with testing GAPI uses JEST and gapi-cli is preconfigurated for your needs! :)

To run single test type:

1gapi test

Testing watch mode

Note: You need to start server before running tests

Note: Everytime you make change to server it will restart server and execute tests

Note: To add more tests just create e2e.spec.ts or unit.spec.ts somewhere inside the application

Start the application

1gapi start

Execute test with --watch argument

1gapi test --watch

You will end up with something like this

Custom logic before testing ( for example creating MOCK users to database before testing)

Create file test.ts inside root/src/test.ts with this content

Everytime you run test with --before argument it will set environment variable BEFORE_HOOK

1  if (process.env.BEFORE_HOOK) {
2    // do something here
3  }

Then execute tests with --before

1gapi test --before

This command will start root/src/test.ts file and will wait for process.exit(0) so you can customize your before logic check this link for reference

Docker

Following commands will start RabbitMQ, PostgreSQL, API, NGINX as a services you need DOCKER for them

API will be served on https://localhost:80 and https://localhost:80/subscriptions

To build project with Docker type:

1gapi app build

To start project with Docker type:

1gapi app start

To stop project type:

1gapi app stop

Workers

All workers will be mapped as Proxy and will be reverted to https://localhost:80 and https://localhost:80/subscriptions

So you don't have to worry about if some of your workers stopped responding

TODO: Create monitoring APP for all workers and main API

To start workers type

1gapi workers start

To stop workers type

1gapi workers stop

To add more workers

By default there are 4 workers with 4 processes with "exec_mode: cluster" of the original process inside single docker container

You can control Processes inside single docker container from "root/process.yml" file.

1apps:
2  - script   : './src/main.ts'
3    name     : 'APP'
4    exec_mode: 'cluster'
5    instances: 4
6

To map new worker as a stream open root/nginx/config/private/default

1upstream app_servers {
2    server 182.10.0.3:9000; # Main process
3    server 182.10.0.21:9000; # Worker 1
4    server 182.10.0.22:9000; # Worker 2
5    server 182.10.0.23:9000; # Worker 3
6    server 182.10.0.24:9000; # Worker 4
7
8    # Add more workers here
9    # server 182.10.0.25:9000; # Worker 5
10}
11
12server {
13    listen 80;
14    server_name api.yourdomain.com;
15    access_log api-yourdomain.access.log;
16
17    location / {
18       proxy_set_header Upgrade $http_upgrade;
19       proxy_set_header Connection "upgrade";
20       client_max_body_size 50M;
21       proxy_set_header Host $http_host;
22       proxy_set_header X-Real-IP $remote_addr;
23       proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
24       proxy_set_header X-Forwarded-Proto $scheme;
25       proxy_set_header X-Frame-Options SAMEORIGIN;
26       proxy_buffers 256 16k;
27	     proxy_buffering off;
28       proxy_buffer_size 16k;
29       proxy_read_timeout 600s;
30       proxy_pass http://app_servers;
31    }
32
33    location /subscriptions {
34         # prevents 502 bad gateway error
35        proxy_buffers 8 32k;
36        proxy_buffer_size 64k;
37
38        # redirect all HTTP traffic to localhost:9000;
39        proxy_pass http://app_servers/subscriptions;
40        proxy_set_header X-Real-IP $remote_addr;
41        proxy_set_header Host $http_host;
42        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
43        #proxy_set_header X-NginX-Proxy true;
44
45        # enables WS support
46        proxy_http_version 1.1;
47        proxy_set_header Upgrade $http_upgrade;
48        proxy_set_header Connection "upgrade";
49	      proxy_buffering off;
50        proxy_read_timeout 999999999;
51
52    }
53    if ($scheme = http) {
54       return 301 https://$server_name$request_uri;
55    }
56    listen 443;
57    ssl on;
58    ssl_certificate         /usr/share/certs/cert.pem;
59    ssl_certificate_key     /usr/share/certs/cert.key;
60}
61
62

When you add another worker it should be on different IP with same port 9000

Open root/gapi.conf.yml file you will find this file:

1config:
2# Application configuration
3  app: 
4    local:
5      API_PORT: 9000
6      API_CERT: ./cert.key
7      NODE_ENV: development
8      AMQP_HOST: 182.10.0.5
9      AMQP_PORT: 5672
10      GRAPHIQL: true
11      GRAPHIQL_TOKEN: eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJlbWFpbCI6ImtyaXN0aXFuLnRhY2hldkBnbWFpbC5jb20iLCJpZCI6MSwic2NvcGUiOlsiQURNSU4iXSwiaWF0IjoxNTIwMjkxMzkyfQ.9hpIDPkSiGvjTmUEyg_R_izW-ra2RzzLbe3Uh3IFsZg
12      ENDPOINT_TESTING: http://localhost:9000/graphql
13      TOKEN_TESTING: eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJlbWFpbCI6ImtyaXN0aXFuLnRhY2hldkBnbWFpbC5jb20iLCJzY29wZSI6WyJBRE1JTiJdLCJpZCI6MSwiaWF0IjoxNTE2OTk2MzYxfQ.7ANr5VHrViD3NkCaDr0nSWYwk46UAEbOwB52pqye4AM
14    prod:
15      API_PORT: 9000
16      API_CERT: ./cert.key
17      NODE_ENV: production
18      AMQP_HOST: 182.10.0.5
19      AMQP_PORT: 5672
20# Testing configuration for local(dev) or worker(running tests as a separate worker with separate environment)
21  test: 
22    local: extends app/local
23    worker:
24      API_PORT: 9000
25      API_CERT: ./cert.key
26      NODE_ENV: production
27      AMQP_HOST: 182.10.0.5
28      AMQP_PORT: 5672
29      ENDPOINT_TESTING: http://182.10.0.101:9000/graphql
30      TOKEN_TESTING: eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJlbWFpbCI6ImtyaXN0aXFuLnRhY2hldkBnbWFpbC5jb20iLCJzY29wZSI6WyJBRE1JTiJdLCJpZCI6MSwiaWF0IjoxNTE2OTk2MzYxfQ.7ANr5VHrViD3NkCaDr0nSWYwk46UAEbOwB52pqye4AM
31
32  schema:
33    introspectionEndpoint: http://localhost:9000/graphql
34    introspectionOutputFolder: ./src/app/core/api-introspection
35
36commands:
37
38  testing:
39    stop:
40      - docker rm -f gapi-api-prod-worker-tests-executor
41      - docker rm -f gapi-api-prod-worker-tests-provider
42    start:
43      - gapi testing start-provider
44      - sleep 10
45      - gapi testing start-executor
46      - echo Cleaning...
47      - gapi testing stop
48    start-provider: docker run -d --network=gapiapiprod_gapi --ip=182.10.0.101 --name gapi-api-prod-worker-tests-provider gapi/api/prod
49    start-executor:
50      - docker run -d --network=gapiapiprod_gapi --ip=182.10.0.100 --name gapi-api-prod-worker-tests-executor gapi/api/prod
51      - docker exec gapi-api-prod-worker-tests-provider npm -v
52      - gapi test --worker --before
53
54  workers:
55    start:
56      - gapi workers start-1
57      - gapi workers start-2
58      - gapi workers start-3
59      - gapi workers start-4
60    stop:
61      - docker rm -f gapi-api-prod-worker-1
62      - docker rm -f gapi-api-prod-worker-2
63      - docker rm -f gapi-api-prod-worker-3
64      - docker rm -f gapi-api-prod-worker-4
65    start-1: docker run -d --network=gapiapiprod_gapi --ip=182.10.0.21 --name gapi-api-prod-worker-1 gapi/api/prod
66    start-2: docker run -d --network=gapiapiprod_gapi --ip=182.10.0.22 --name gapi-api-prod-worker-2 gapi/api/prod
67    start-3: docker run -d --network=gapiapiprod_gapi --ip=182.10.0.23 --name gapi-api-prod-worker-3 gapi/api/prod
68    start-4: docker run -d --network=gapiapiprod_gapi --ip=182.10.0.24 --name gapi-api-prod-worker-4 gapi/api/prod
69    example-worker-with-port: docker run -d --network=gapiapiprod_gapi --ip=182.10.0.25 --name gapi-api-prod-worker-5 -p 9001:9000 gapi/api/prod
70
71  app:
72    start:
73      - docker-compose -p gapi-api-prod up --force-recreate -d
74      - gapi rabbitmq enable-dashboard
75    stop:
76      - gapi nginx stop
77      - gapi api stop
78      - gapi rabbitmq stop
79      - gapi postgres stop
80    build: docker build -t gapi/api/prod .
81
82  api:
83    stop: docker rm -f gapi-api-prod
84
85  nginx:
86    stop: docker rm -f gapi-api-nginx
87
88  postgres:
89    stop: docker rm -f gapi-api-postgres
90
91  rabbitmq:
92    stop: docker rm -f gapi-api-rabbitmq
93    restart: docker restart gapi-api-rabbitmq
94    enable-dashboard: docker exec gapi-api-rabbitmq rabbitmq-plugins enable rabbitmq_management
95
96# You can define your custom commands for example 
97# commands:
98#   your-cli:
99#     my-command: 'npm -v'
100# This command can be executed as "gapi your-cli my-command"

Adding one more worker:

1start-5: 'docker run -d --network=gapiapiprod_gapi --ip=182.10.0.25 --name gapi-api-prod-worker-5 -p 9005:9000 gapi/api/prod'

Then edit start task inside workers to start new worker 5

1start: 'gapi workers start-1 && gapi workers start-2 && gapi workers start-3 && gapi workers start-4 & gapi workers start-5'

Thats' it!! Now you have 4 processes like CLUSTERS inside 1 docker container with ip 182.10.0.25 and external port(optional) 9005;

You can specify worker also without port because all workers are inside internal network called "gapiapiprod_gapi"

182.10.0.21/22/23/24/25:9000

1start-5: 'docker run -d --network=gapiapiprod_gapi --ip=182.10.0.25 --name gapi-api-prod-worker-5 gapi/api/prod'

If you want to change port forwarding to another port you need to set just nginx configuration:

1  nginx:
2    image: sameersbn/nginx:1.10.1-5
3    ports:
4      - "81:80"
5      - "443:443"

Now you can find your API served onto https://localhost:81/ and https://localhost:81/subscriptions

All workers don't care about that because they will be served and mapped from nginx to port 80

You can check docker-compose file to configurate environment variables

1version: '2'
2services:
3
4  nginx:
5    image: sameersbn/nginx:1.10.1-5
6    ports:
7      - "80:80"
8      - "443:443"
9    volumes:
10      - ./nginx/config:/etc/nginx
11      - ./nginx/html:/usr/share/nginx/html/
12      - ./nginx/certs:/usr/share/certs
13    restart: always
14    container_name: gapi-api-nginx
15    networks:
16      gapi:
17        ipv4_address: 182.10.0.2
18
19  api:
20    image: gapi/api/prod:latest
21    ports:
22      - "9000"
23    restart: always
24    mem_limit: 1000000000
25    cpu_shares: 73
26    container_name: gapi-api-prod
27    depends_on:
28      - nginx
29      - rabbitMq
30    networks:
31      gapi:
32        ipv4_address: 182.10.0.3
33
34  rabbitMq:
35    image: rabbitmq:3.7.2
36    ports:
37      - "15672:15672"
38      - "5672:5672"
39      - "5671:5671"
40      - "4369:4369"
41    restart: always
42    container_name: gapi-api-rabbitmq
43    networks:
44      gapi:
45        ipv4_address: 182.10.0.5
46
47networks:
48  gapi:
49    driver: bridge
50    ipam:
51     config:
52       - subnet: 182.10.0.0/16
53         gateway: 182.10.0.1
54

After successfully started project you can open your browser to localhost:80 or 182.10.0.3:9000 the api will be served there

TODO: Better documentation...

Enjoy ! :)