Traceable Nodejsagent

nodejsagent provides a set of instrumentation & blocking features for collecting relevant data to be processed by the Traceable.ai platform.

Prerequisites

The agent supports Node 8.17 and up.

The package comes with precompiled extensions that support the following operating systems & associated versions:

• Debian 10+
• Ubuntu 18.04+
• Centos 7+

If you are using the agent on an unsupported operating system, the observability features will work; however blocking functionality will not be loaded.

Installing

To install the package: npm install @traceable/nodejsagent

Configuration

The agent supports two methods of configuration.

• Environment variables
• Configuration file

For the agent to use a config file you must set a environment variable TA_CONFIG_FILE=./config.yaml, where the value is a path to the config.yaml file.

A minimal config.yaml is below:

1service_name: node_app
2reporting:
3  endpoint: http://agent.traceableai:4317
4opa:
5  endpoint: http://agent.traceableai:8181/
6blocking_config:
7  enabled: true
8  remote_config:
9    endpoint: agent.traceableai:5441

A complete config.yaml with default values is shown below:

1service_name: nodeagent
2reporting:
3  endpoint: http://localhost:4317
4  secure: false
5  trace_reporter_type: OTLP
6data_capture:
7  http_headers:
8    request: true
9    response: true
10  http_body:
11    request: true
12    response: true
13  rpc_metadata:
14    request: true
15    response: true
16  rpc_body:
17    request: true
18    response: true
19  body_max_size_bytes: 131072
20  body_max_processing_size_bytes: 131072
21propagation_formats: ["TRACECONTEXT"]
22enabled: true
23resource_attributes: { }
24opa:
25  enabled: true
26  endpoint: http://opa.traceableai:8181/
27  poll_period_seconds: 60
28blocking_config:
29  enabled: true
30  debug_log: true
31  evaluate_body: true
32  skip_internal_request: false
33  modsecurity:
34    enabled: true
35  regionBlocking:
36    enabled: true
37  remote_config:
38    enabled: true
39    endpoint: localhost:5441/
40    poll_period_seconds: 30

The full list of supported environment variables is listed below:

Name	Default	Description
TA_OPA_ENABLED	true	When true Open Policy Agent evaluation is enabled to block request
TA_OPA_ENDPOINT	http://opa.traceableai:8181/	Represents the endpoint for polling OPA config file e.g. http://opa.traceableai:8181/
TA_OPA_POLL_PERIOD_SECONDS	60	Poll period in seconds to query OPA service
TA_BLOCKING_CONFIG_ENABLED	false	The top level switch that enables/disables blocking functionality
TA_BLOCKING_CONFIG_DEBUG_LOG	false	Enables verbose logging of the native extensions
TA_BLOCKING_CONFIG_MODSECURITY_ENABLED	true
TA_BLOCKING_CONFIG_EVALUATE_BODY	true	Determines if the request body should be analyzed
TA_BLOCKING_CONFIG_REGION_BLOCKING_ENABLED	true	Enables IP based blocking
TA_BLOCKING_CONFIG_REMOTE_CONFIG_ENABLED	true	Denotes if config needs to be fetched from remote or not
TA_BLOCKING_CONFIG_REMOTE_CONFIG_ENDPOINT	localhost:5441	Denotes the agentmanager endpoint to connect to for config. eg: localhost:5441
TA_BLOCKING_CONFIG_REMOTE_CONFIG_POLL_PERIOD_SECONDS	30	Poll period in seconds to query for config updates
TA_BLOCKING_CONFIG_SKIP_INTERNAL_REQUEST	true	When true, blocking evaluation will be skipped for internal requests i.e. requests coming from private IPs
TA_SERVICE_NAME	nodeagent	Identifies the service/process running e.g. "my service"
TA_REPORTING_ENDPOINT	http://localhost:4317	Represents the endpoint for reporting the traces For ZIPKIN reporter type use http://api.traceable.ai:9411/api/v2/spans For OTLP reporter type use http://api.traceable.ai:4317
TA_REPORTING_SECURE	false	When true, connects to endpoints over TLS.
TA_DATA_CAPTURE_HTTP_HEADERS_REQUEST	true	When false it disables the capture for the request in a client/request operation
TA_DATA_CAPTURE_HTTP_HEADERS_RESPONSE	true	When false it disables the capture for the response in a client/request operation
TA_DATA_CAPTURE_HTTP_BODY_REQUEST	true	When false it disables the capture for the request in a client/request operation
TA_DATA_CAPTURE_HTTP_BODY_RESPONSE	true	When false it disables the capture for the response in a client/request operation
TA_DATA_CAPTURE_RPC_METADATA_REQUEST	true	When false it disables the capture for the request in a client/request operation
TA_DATA_CAPTURE_RPC_METADATA_RESPONSE	true	When false it disables the capture for the response in a client/request operation
TA_DATA_CAPTURE_RPC_BODY_REQUEST	true	When false it disables the capture for the request in a client/request operation
TA_DATA_CAPTURE_RPC_BODY_RESPONSE	true	When false it disables the capture for the response in a client/request operation
TA_DATA_CAPTURE_BODY_MAX_SIZE_BYTES	131072	Is the maximum size of captured body in bytes. Default should be 131_072 (128 KiB).
TA_DATA_CAPTURE_BODY_MAX_PROCESSING_SIZE_BYTES	131072	Is maximum size of body being processed by filters in bytes. For uncompressed bodies we capture all bytes up to body_max_processing_size_bytes in memory and pass that through the filter. For compressed and GRPC bodies, if the size of the body is larger than this, we ignore it entirely, otherwise we decompress/decode the body and then pass it to the filter.
TA_PROPAGATION_FORMATS	['TRACECONTEXT']	List the supported propagation formats e.g. TA_PROPAGATION_FORMATS="B3,TRACECONTEXT".
TA_ENABLED	true	When false, disables the agent
TA_LOG_LEVEL	info	Controls granularity of produced logs, one of: debug, info, warning, error

Instrumenting

The agent offers two methods for instrumenting an application: 1.) Auto instrumentation 2.) Manual instrumentation

Auto instrumentation

Add the -r @traceable/nodejsagent to your apps node startup command. Ex: TA_CONFIG_FILE=./config.yaml node -r @traceable/nodejsagent server-a.js

Note: If using ES6 import syntax in your application you must use auto instrumentation instead of manual instrumentation Ex: import express from 'express' instead of const express = require('express'). The former requires auto instrumentation.

Manual instrumentation

To manually instrument an application you will need to edit the apps main entrypoint file.

Ex:

1    const traceable = require('@traceable/nodejsagent')
2    const agent = new traceable.TraceableAgent()
3    agent.instrument() // Enables collection of observability data
4    agent.addTraceableFilter() // Enables blocking features

Development

Setup

• Initialize agent-config repo

  git submodule update --init --recursive
  

Updating Libtraceable

1. Replace the header file ./src/ext/lib/libtraceable.h
2. Update the LIBTRACEABLE_VERSION in `./prebuild-ci/docker-compose.yaml

Building and Packaging

1. Build:

   cd prebuild-ci;  ./generate.sh;  cd -
   

2. Package:

   npm run build
   

3. Artifact: traceable-nodejsagent-0.0.0.tgz

Testing

1. Copy the artifact to ./examples

   cp traceable-nodejsagent-0.0.0.tgz ./examples
   

2. Edit config.yaml to set up traceable agent. By default it looks at localhost for traceable agent
3. Build the test app docker image

   docker build -t nodejsagent-test-app-image .
   

   For Alpine use Dockerfile-alpine.

   docker build -t nodejsagent-test-app-image -f Dockerfile-alpine .
   

4. Run the test app. By default, the sample nodejs app listens on 8001.

   docker run --name nodejsagent-test-app --rm -p 8001:8001 nodejsagent-test-app-image
   

5. Send test requests

   curl localhost:8001/test
   curl -H "x-forwarded-for: 1.39.0.0" -XPOST "http://localhost:8001/test-post" -H 'Accept: application/json, */*;q=0.5' -H 'Content-type: application/json' -d @request500b.json
   

6. Example blocked requests

   curl -vvv -H "x-forwarded-for: 1.40.0.0" "localhost:8001/test"
   curl -vvv -H "x-forwarded-for: 8.8.8.8" "localhost:8001/test?a=/usr/bin/perl"
   curl -vvv -H "x-forwarded-for: 8.8.8.8" "localhost:8001/test?a=blockmetyrone"
   curl -vvv -H "x-forwarded-for: 8.8.8.8" -XPOST "http://localhost:8001/test-post" -H 'Accept: application/json, */*;q=0.5' -H 'Content-type: application/json' -d @requestBAD.json