Table of Contents

• Need
• Implementation
• Setup
• Usage
• Configuration
• Control

Need

In a clustered environment, a boot script would run on all instances of the application, as the cluster comes up. However, irrespective of the number of application-instances in the cluster, sometimes there is a need to execute an arbitrary function just once, on a single application instance, at the time of application-cluster startup; and later, if the application instance that executed the one-time function goes down for any reason, another running instance needs to take over and run the function.

An example would be the case of a function that sends "application is up" heartbeat event at regular intervals. Such a function, if started from a regular boot script, would start multiple timers sending extra/unwanted heartbeat events depending on the number of app-instances in the cluster. This calls for a module that can run the function only once on a single app-instance of the cluster. If the function-runner instance goes down, the function should be run again on another active app-instance.

Another example: A clustered application may be running 4 application instances, but a job scheduler function within this app may need to run once on boot, and only on a single app instance. The app instance which is elected to run the job-scheduler function is the job-scheduler-master instance. If the job-scheduler-master instance goes down, then another instance of the application should become the job-scheduler-master and run the job scheduler function.

Implementation

This module provides the infrastructure for catering to the above need. It is implemented as an app-list module for oe-Cloud based applications. It provides the ability to automatically elect one app-instance from among the cluster-instances and run a specified function once in the elected app-instance, which we call the <FUNCTION_>MASTER instance, for e.g., JOB_SCHEDULER_MASTER instance. If the master for a function goes down, a new master is elected for that function and the function is run again.

To achieve the aforementioned functionality, a database query/update-based locking is used, along with updates of a lock heartbeat timestamp at regular intervals by the master app-instance. All app-instances keep checking for missed master-heartbeats and are ready to take over as master by deleting the lock from the database and creating their own lock.

Setup

To get the oe-master-job-executor feature in the application, the oe-master-job-executor node module needs to be added as a package.json dependency in the application.

Also, the module needs be added to the server/app-list.json file in the app.

For e.g.,

package.json (only part of the file is shown here, with relevant section in bold):

...
   ...
   "dependencies": {
       ...
       ...
       ...
       "oe-workflow": "^2.0.0",
       "oe-master-job-executor": "^2.0.0",
       "passport": "0.2.2",
       ...
       ...

server/app-list.json (Relevant section in bold):

[
    {
        "path": "oe-cloud",
        "enabled": true
    },
    {
        "path": "oe-master-job-executor",
        "enabled": true
    },
	{
		"path" : "oe-workflow",
		"enabled" : true
	},
	{
        "path": "./",
        "enabled": true
    }
]

Usage

The oe-master-job-executor module can be used to start any number of masters, for performing various one-time-run functions/operations. Each such usage creates one master. Creation of a master involves providing a lockName for the master and a masterJob object which has the one-time-run function to execute, as shown below:

1
2function start() {
3    // Start the master function
4}
5
6function stop() {
7    // Stop/cleanup the master function
8}
9
10var masterJobExecutor = require('oe-master-job-executor');
11var options = {
12                  lockName: 'JOB-SCHEDULER',                  // Mandatory
13                  masterJob: { start: start, stop: stop },    // Mandatory
14                  checkMasterInterval: 60000                  // Optional. See 'Configuration' section below for details.
15              };
16masterJobExecutor.startMaster(options);
17
18

The above code can run, say, from a boot script in all app-instances of an application cluster, however, the function start() will be called only once on one of the app-instances (the master). If the master instance dies, start() will be called again once on another instance that is elected as master.

The stop() function is called whenever the master is stopped either due to manual disablement via HTTP API call (see Control section below), or self-stoppage due to heartbeat timestamp not getting updated for any reason.

Configuration

The oe-master-job-executor module can be configured via -

1. Default values in code (no configuration)
2. server/config.json
3. environment variables
4. startMaster options (see Usage section above)

with the following priority: 4 > 3 > 2 > 1

Priority is applicable on a per-parameter basis.

The following are the configuration parameters:

----------------------------------------------------------------------------------------------------------------------------------------
config.json setting                       startmaster option      Env Variable            type          default    Description
----------------------------------------------------------------------------------------------------------------------------------------
masterJobExecutor.initDelay               initDelay               INIT_DELAY              number (ms)   1000       This setting determines the delay
                                                                                                                   in milliseconds since boot, after
                                                                                                                   which the master is started

masterJobExecutor.checkMasterInterval     checkMasterInterval     CHECK_MASTER_INTERVAL   number (ms)   30000      This is the interval at which each
                                                                                                                   app-instance checks for the master heartbeat
                                                                                                                   in order to try and become master itself

masterJobExecutor.heartbeatInterval       heartbeatInterval       MASTER_JOB_HEARTBEAT_INTERVAL                    This is the interval at which heartbeat
                                                                                          number (ms)   8000       timestamp is updated by the master


masterJobExecutor.maxHeartbeatRetryCount  maxHeartbeatRetryCount  MASTER_JOB_MAX_HEARTBEAT_RETRY_COUNT             This is the number of times the master
                                                                                          number        3          heartbeat will be retried upon falure
                                                                                                                   to send heartbeat.

-----------------------------------------------------------------------------------------------------------------------------------------

An example of oe-master-job-executor configuration via server/config.json is shown below:

1
2{
3    . . .
4    . . .
5    "masterJobExecutor": {
6        "initDelay": 5000,
7        "checkMasterInterval": 20000
8        "heartbeatInterval": 15000,
9        "maxHeartbeatRetryCount": 10
10    },
11    . . .
12    . . .
13}

Control

The master can be stopped and started manually via HTTP API. Disabling the master causes the current master to call the stop() function of the masterJob and it will also prevent other app-instances from becoming a master.

The API to call for stopping/disabling the master is as follows:

POST /api/mastercontrols/disable

e.g., payload:

1{
2    "lockName": "JOB-SCHEDULER",
3    "reason": "testing"
4}
5

The API for restarting/enabling the master is as follows:

POST /api/mastercontrols/enable

e.g., payload:

1{
2    "lockName": "JOB-SCHEDULER"
3}
4

Upon restarting/re-enabling a master, one of the running app-instances will get elected as master and the masterJob's start() function is called once.