NehonixURIProcessor

A comprehensive TypeScript library for detecting, decoding, and encoding various URI encoding schemes. This utility is designed for security testing, web application penetration testing, and analyzing potential attacks, with powerful auto-detection and decoding capabilities.

Version: 2.3.1 License: MIT
Documentation: lab.nehonix.space

Table of Contents

• Introduction
• Overview
• Installation
• Usage
• API Reference
  • Core Methods
    • checkUrl(url: string, options?: object)
    • asyncCheckUrl(url: string, options?: object)
    • isValidUri(url: string, options?: object)
    • asyncIsUrlValid(url: string, options?: object)
    • createUrl(uri: string)
    • detectEncoding(input: string, depth?: number)
    • autoDetectAndDecode(input: string, maxIterations?: number)
    • asyncAutoDetectAndDecode(input: string, maxIterations?: number, useWorker?: boolean)
    • scanUrl(url: string)
    • sanitizeInput(input: string, options?: object)
    • needsDeepScan(input: string)
    • detectMaliciousPatterns(input: string, options?: MaliciousPatternOptions)
  • Framework Integrations
    • Express Middleware
    • React Utils
• Supported Encoding Types
• Detection Capabilities
• Security Testing Features
• More Information
• License

Introduction

NehonixURIProcessor is a powerful TypeScript library for developers and security professionals. It provides advanced tools for URI validation, encoding/decoding, and security analysis. For convenience, you can import it as __processor__ to shorten the name (both are the same):

1import { NehonixURIProcessor } from "nehonix-uri-processor";
2` OR`;
3import { __processor__ } from "nehonix-uri-processor";

Overview

The NehonixURIProcessor class offers:

• URI Validation: Validate URIs with customizable rules and malicious pattern detection.
• Auto-Detection and Decoding: Decode complex URI encodings using autoDetectAndDecode or asyncAutoDetectAndDecode.
• Encoding/Decoding: Support for multiple encoding schemes (e.g., Base64, percent encoding, JWT).
• Security Analysis: Analyze URLs for vulnerabilities and generate WAF bypass variants.
• Framework Integration: Seamless integration with Express and React.
• Internationalized URIs: Handle non-ASCII characters with punycode support.

Installation

Install the library and its dependency:

1npm install nehonix-uri-processor punycode

Usage

Below are examples showcasing key features:

1import {
2  NehonixURIProcessor as __processor__,
3  MaliciousPatternType,
4} from "nehonix-uri-processor";
5
6async function main() {
7  // Validate a URI with malicious pattern detection
8  const result = await __processor__.asyncCheckUrl(
9    "https://example.com?user=admin' OR '1'='1",
10    {
11      detectMaliciousPatterns: true,
12      customMaliciousPatterns: [MaliciousPatternType.ANOMALY],
13      maliciousPatternSensitivity: 1.0,
14      maliciousPatternMinScore: 50,
15    }
16  );
17  console.log(result.isValid); // false (detects SQL injection attempt)
18
19  // Decode a complex URI
20  const decoded = await __processor__.asyncAutoDetectAndDecode(
21    "https://example.com?data=SGVsbG8gV29ybGQ="
22  );
23  console.log(decoded); // https://example.com?data=Hello World
24
25  // Check if deep scanning is needed
26  const needsScan = __processor__.needsDeepScan(
27    "https://example.com?user=<script>"
28  );
29  console.log(needsScan); // booelan
30}
31main();

API Reference

Core Methods

checkUrl

1static checkUrl(url: string, options?: UrlValidationOptions): UrlCheckResult

asyncCheckUrl

1static asyncCheckUrl(url: string, options?: AsyncUrlValidationOptions): Promise<AsyncUrlCheckResult>

Parameters

• url (string): The URI string to validate (e.g., https://example.com?test=true).

• options (UrlValidationOptions or AsyncUrlValidationOptions, optional): Configuration object to customize validation rules. Defaults to:

  1{
  2  strictMode: false,
  3  allowUnicodeEscapes: true,
  4  rejectDuplicateParams: true,
  5  httpsOnly: false,
  6  maxUrlLength: 2048,
  7  allowedTLDs: [],
  8  allowedProtocols: ["http", "https"],
  9  requireProtocol: false,
  10  requirePathOrQuery: false,
  11  strictParamEncoding: false,
  12  rejectDuplicatedValues: false,
  13  debug: false,
  14  allowInternationalChars: false,
  15  // AsyncUrlValidationOptions only (for asyncCheckUrl)
  16  detectMaliciousPatterns: false,
  17  customMaliciousPatterns: [],
  18  maliciousPatternSensitivity: 1.0,
  19  maliciousPatternMinScore: 50
  20}

UrlValidationOptions

| Option | Type | Default | Description | | ------------------------- | ---------------------- | ------------------- | ---------------------------------------------------------------------------- | ----------------------------------------------------------------------------- | -------------------------------------------------------------------------- | | strictMode | boolean | false | Requires a leading slash before query parameters (e.g., /path vs. path). | | allowUnicodeEscapes | boolean | true | Allows Unicode escape sequences (e.g., \uXXXX) in query parameters. | | rejectDuplicateParams | boolean | true | Rejects URIs with duplicate query parameter keys (e.g., ?test=1&test=2). | | rejectDuplicatedValues | boolean | false | Rejects URIs with duplicate query parameter values. | | httpsOnly | boolean | false | Restricts URIs to https:// protocol only. | | maxUrlLength | number | 2048 | Maximum URL length in characters (0 to disable). | | allowedTLDs | string[] | [] | Allowed top-level domains (empty for all). | | allowedProtocols | string[] | ["http", "https"] | Allowed protocols (e.g., http, https). | | requireProtocol | boolean | false | Requires an explicit protocol (e.g., http:// or https://). | | requirePathOrQuery | boolean | false | Requires a path or query string (e.g., /path or ?query=test). | | strictParamEncoding | boolean | false | Enforces strict URI encoding for query parameters. | | debug | boolean | false | Enables debug logging for custom validations, printing actual values. | | allowInternationalChars | boolean | false | Allows non-ASCII characters in URIs (normalized with punycode). | | customValidations | ComparisonRule[] | undefined | Array of custom validation rules for URL components or custom properties. | | literalValue | "@this" | string | number | "@this" | Value for literal rules in customValidations. Defaults to input url. | | fullCustomValidation | Record<string, string | number> | undefined | Defines custom properties for validation (e.g., { domain: "test_domain" }). |

AsyncUrlValidationOptions (extends UrlValidationOptions)

Option	Type	Default	Description
detectMaliciousPatterns	boolean	false	Enables detection of malicious patterns (e.g., SQL injection, XSS).
customMaliciousPatterns	MaliciousPatternType[]	[]	Specifies custom malicious patterns to detect.
maliciousPatternSensitivity	number	1.0	Sensitivity for malicious pattern detection (0.0 to 1.0).
maliciousPatternMinScore	number	50	Minimum score for malicious pattern detection.

ComparisonRule

A ComparisonRule defines a validation rule for a URL component or custom property:

1type ComparisonRule = [
2  ValidUrlComponents | custumValidUriComponent,
3  comparisonOperator,
4  string | number
5];

• ValidUrlComponents:

  1type ValidUrlComponents =
  2  | "href"
  3  | "origin"
  4  | "protocol"
  5  | "username"
  6  | "password"
  7  | "host"
  8  | "hostname"
  9  | "port"
  10  | "pathname"
  11  | "search"
  12  | "hash";

• custumValidUriComponent:

  1type custumValidUriComponent = "fullCustomValidation" | "literal";

• comparisonOperator:

  1type comparisonOperator =
  2  | "==="
  3  | "=="
  4  | "<="
  5  | ">="
  6  | "!="
  7  | "!=="
  8  | "<"
  9  | ">";

Rules can reference:

• Standard URL components (e.g., ["hostname", "===", "example.com"]).
• Literal values (e.g., ["literal", "===", "nehonix.space"] with literalValue set).
• Custom properties (e.g., ["fullCustomValidation.domain", "===", "test_domain"] or ["fcv.domain", "===", "test_domain"]).

Return Value

checkUrl

Returns a UrlCheckResult object:

1export interface UrlCheckResult {
2  /**
3   * Indicates whether the URL is valid based on all validation checks.
4   * `true` if all checks pass, `false` if any check fails.
5   */
6  isValid: boolean;
7
8  /**
9   * Return the reason of failing
10   */
11  cause?: string;
12
13  /**
14   * Contains detailed results for each validation check performed on the URL.
15   * Each property corresponds to a specific validation aspect and is optional,
16   * as not all validations may be relevant depending on the provided options.
17   */
18  validationDetails: {
19    customValidations?: {
20      isValid: boolean;
21      message: string;
22      results: {
23        isValid: boolean;
24        message: string;
25        rule: ComparisonRule;
26      }[];
27    };
28    length?: {
29      isValid: boolean;
30      message?: string;
31      actualLength?: number;
32      maxLength?: number | "NO_LIMIT";
33    };
34    emptyCheck?: {
35      isValid: boolean;
36      message?: string;
37    };
38    protocol?: {
39      isValid: boolean;
40      message?: string;
41      detectedProtocol?: string;
42      allowedProtocols?: string[];
43    };
44    httpsOnly?: {
45      isValid: boolean;
46      message?: string;
47    };
48    domain?: {
49      isValid: boolean;
50      message?: string;
51      hostname?: string;
52      error?: string;
53      type?: "INV_DOMAIN_ERR" | "INV_STRUCTURE" | "ERR_UNKNOWN";
54    };
55    tld?: {
56      isValid: boolean;
57      message?: string;
58      detectedTld?: string;
59      allowedTlds?: string[];
60    };
61    pathOrQuery?: {
62      isValid: boolean;
63      message?: string;
64    };
65    strictMode?: {
66      isValid: boolean;
67      message?: string;
68    };
69    querySpaces?: {
70      isValid: boolean;
71      message?: string;
72    };
73    paramEncoding?: {
74      isValid: boolean;
75      message?: string;
76      invalidParams?: string[];
77    };
78    duplicateParams?: {
79      isValid: boolean;
80      message?: string;
81      duplicatedKeys?: string[];
82    };
83    duplicateValues?: {
84      isValid: boolean;
85      message?: string;
86      duplicatedValues?: string[];
87    };
88    unicodeEscapes?: {
89      isValid: boolean;
90      message?: string;
91    };
92    parsing?: {
93      isValid: boolean;
94      message?: string;
95    };
96    internationalChars?: {
97      isValid: boolean;
98      message: string;
99      containsNonAscii?: boolean;
100      containsPunycode?: boolean;
101    };
102  };
103}

asyncCheckUrl

Returns a Promise<AsyncUrlCheckResult>, which extends UrlCheckResult with maliciousPatterns in validationDetails:

1interface DetectedPattern {
2  type: string; // e.g., "XSS", "SQL_INJECTION"
3  value: string; // The detected malicious content
4  score: number; // Severity score (0-100)
5}
6
7export type AsyncUrlCheckResult = Omit<UrlCheckResult, "validationDetails"> & {
8  validationDetails: UrlCheckResult["validationDetails"] & {
9    maliciousPatterns?: {
10      isValid?: boolean;
11      message?: string;
12      error?: string;
13      detectedPatterns?: DetectedPattern[];
14      score?: number;
15      confidence?: string;
16      recommendation?: string;
17    };
18  };
19};

• isValid: true if the URI passes all validation rules, false otherwise.
• validationDetails: Detailed results for each validation check.
• cause: Reason for failure (empty if isValid is true).
• maliciousPatterns (asyncCheckUrl only): Included in validationDetails, containing results of malicious pattern detection (e.g., XSS, SQL injection).

Custom Validation with fullCustomValidation

The fullCustomValidation option (aliased as fcv) allows defining custom properties for validation alongside standard URL components:

• Define Custom Properties: Provide a fullCustomValidation object (e.g., { domain: "test_domain", version: 1.2 }).
• Reference in Rules: Use fullCustomValidation.<property> or fcv.<property> in customValidations (e.g., ["fcv.domain", "===", "test_domain"]).
• Validate: Compares the property’s value against the rule’s value using the specified operator.

literalValue Usage

The literalValue option specifies the value for literal rules in customValidations. It defaults to "@this", which uses the input url. For specific comparisons (e.g., ["literal", "===", "nehonix.space"]), set literalValue explicitly.

Question: Synchronous (checkUrl) vs. Asynchronous (asyncCheckUrl) - Which is Best and When?

Question

When should you use checkUrl versus asyncCheckUrl? How do their performance characteristics and use cases differ, especially regarding execution time and resource usage?

Answer

Both checkUrl and asyncCheckUrl validate URIs, but their execution models and capabilities differ, impacting their suitability for various scenarios:

• checkUrl (Synchronous):

  • Pros:
    • Faster for simple validations, completing in microseconds, as it avoids Promise overhead.
    • Ideal for lightweight, single-threaded applications or quick checks in non-async contexts (e.g., CLI tools, synchronous middleware).
    • Lower memory overhead due to synchronous execution.
  • Cons:
    • Lacks malicious pattern detection, limiting its use in security-critical applications.
    • Can block the main thread, causing delays in event-driven environments (e.g., Node.js servers) for complex or long URLs.
  • Best Use Cases:
    • Quick validations in synchronous codebases.
    • Non-security-critical scenarios.
    • Example: Validating static URLs in a build script.
  • Performance: Microseconds for simple URLs, but may scale poorly with complex rules.

• asyncCheckUrl (Asynchronous):

  • Pros:
    • Includes malicious pattern detection, critical for security-focused applications (e.g., detecting XSS, SQL injection).
    • Non-blocking, ideal for event-driven environments like web servers or React apps.
    • Scales better for complex validations or long URLs by leveraging async processing.
  • Cons:
    • Slower due to Promise overhead, typically milliseconds.
    • Higher memory usage due to async context and pattern analysis.
  • Best Use Cases:
    • Security testing requiring malicious pattern detection.
    • Async workflows in Node.js or front-end apps.
    • Example: Validating user-submitted URLs in an Express API.
  • Performance: Milliseconds, but non-blocking, suitable for high-concurrency scenarios.

• When to Choose:

  • Use checkUrl for fast, non-security-critical validations in synchronous environments.
  • Use asyncCheckUrl for security-critical applications or async environments where non-blocking is essential.
  • Precious Time to Use:
    • checkUrl saves microseconds in low-latency, synchronous scenarios.
    • asyncCheckUrl is worth the millisecond overhead for security and non-blocking behavior.

• Recommendation:

  • In modern web applications, prefer asyncCheckUrl for its security features and non-blocking nature. Use checkUrl only in specific synchronous, non-security-critical cases.

Example Usage

Validating with checkUrl

1import { __processor__ } from "nehonix-uri-processor";
2
3const result = __processor__.checkUrl("https://google.com/api", {
4  literalValue: "nehonix.space",
5  debug: true,
6  fullCustomValidation: { domain: "test_domain", version: 1.2 },
7  customValidations: [
8    ["hostname", "===", "google.com"],
9    ["pathname", "===", "/api"],
10    ["literal", "===", "nehonix.space"],
11    ["fcv.domain", "===", "test_domain"],
12    ["fcv.version", ">=", 1.0],
13  ],
14});
15
16
17#### `isValidUri(url: string, options?: object)`
18
19Checks if a string is a valid URI with configurable rules and malicious pattern detection.
20
21- **Parameters**:
22
23  - `url` (`string`): The URI to validate.
24  - `options` (optional): Includes `detectMaliciousPatterns`, `allowInternationalChars`, etc.
25
26- **Returns**: `boolean`.
27
28- **Example**:
29
30```typescript
31const isValid = __processor__.isValidUri(
32  "https://xn--n3h.com?greeting=こんにちは",
33  {
34    allowInternationalChars: true,
35  }
36);
37console.log(isValid); // true

asyncIsUrlValid(url: string, options?: object)

Asynchronously validates a URI string, similar to isValidUri but designed for async workflows.

• Parameters:

  • url (string): The URI to validate.
  • options (optional): Same as isValidUri.

• Returns: Promise<boolean>.

• Example:

1const isValid = await __processor__.asyncIsUrlValid("https://example.com", {
2  httpsOnly: true,
3});
4console.log(isValid); // true

createUrl(uri: string)

Creates a native URL object from a URI string.

• Returns: URL.

• Example:

1const url = __processor__.createUrl("https://example.com/path");
2console.log(url.pathname); // /path

detectEncoding(input: string, depth?: number)

Detects encoding types in a URI string, with optional recursion for nested encodings.

• Returns: { mostLikely: string, confidence: number, nestedTypes: string[] }.

• Example:

1const detection = __processor__.detectEncoding("hello%20world");
2console.log(detection.mostLikely); // percentEncoding

autoDetectAndDecode(input: string, maxIterations?: number)

Recommended: Automatically detects and decodes a URI to plaintext.

• Parameters:

  • input (string): The URI to decode.
  • maxIterations (number, default: 10): Limits decoding iterations.

• Returns: string (decoded plaintext).

• Example:

1const decoded = __processor__.autoDetectAndDecode(
2  "https://example.com?test=dHJ1ZQ=="
3);
4console.log(decoded); // https://example.com?test=true

asyncAutoDetectAndDecode(input: string, maxIterations?: number, useWorker?: boolean)

Asynchronously decodes a URI to plaintext, suitable for complex URIs.

• Returns: Promise<string>.

• Example:

1const decoded = await __processor__.asyncAutoDetectAndDecode(
2  "https://example.com?data=SGVsbG8gV29ybGQ="
3);
4console.log(decoded); // https://example.com?data=Hello World

scanUrl(url: string)

Generates a security report for a URI, including vulnerability analysis and recommendations.

• Returns: { analysis, variants, recommendations }.

• Example:

1const report = __processor__.scanUrl(
2  "https://example.com?user=admin' OR '1'='1"
3);
4console.log(report.recommendations); // ["Sanitize parameter \"user\" to prevent SQL injection..."]

sanitizeInput(input: string, options?: object)

Sanitizes input by removing potentially malicious patterns. Note: This method is not stable and should be used cautiously.

• Parameters:

  • input (string): The string to sanitize.
  • options (optional): Additional sanitization options.

• Returns: string (sanitized string).

• Example:

1const sanitized = __processor__.sanitizeInput("<script>alert('xss')</script>");
2console.log(sanitized); // Sanitized string with malicious content removed

needsDeepScan(input: string)

Lightweight check to determine if a string requires deep scanning. Use as a pre-filter before full pattern detection.

• Parameters:

  • input (string): The string to check.

• Returns: boolean (whether deep scanning is needed).

• Example:

1const needsScan = __processor__.needsDeepScan(
2  "https://example.com?user=<script>"
3);
4console.log(needsScan); // true

detectMaliciousPatterns(input: string, options?: MaliciousPatternOptions)

Analyzes input for malicious patterns and returns detailed detection results.

• Parameters:

  • input (string): The string to analyze.
  • options (MaliciousPatternOptions, optional): Configuration for detection (e.g., sensitivity, patterns).

• Returns: Detailed analysis result (type depends on NSS.detectMaliciousPatterns).

• Example:

1const result = __processor__.detectMaliciousPatterns(
2  "https://example.com?user=admin' OR '1'='1",
3  { sensitivity: 1.0 }
4);
5console.log(result); // Detailed malicious pattern analysis

Framework Integrations

Express Middleware

Validate and decode URIs in Express applications.

• Setup:

1import express from "express";
2import { nehonixShieldMiddleware } from "nehonix-uri-processor";
3
4const app = express();
5app.use(nehonixShieldMiddleware({ blockOnMalicious: true }));
6app.get("/", (req, res) => res.send("Hello world"));
7app.listen(3000, () => console.log("Server running on port 3000"));

React Utils

Overview

The NSB DOM & Request Analysis feature enhances web application security by adding real-time scanning of DOM elements and network requests. This feature builds upon the existing Nehonix Security Booster framework to detect and block malicious content before it reaches the user.

Features

• DOM Analysis: Scan the document object model for malicious patterns
• Request Monitoring: Analyze network requests in real-time
• Automatic Protection: Components for easy integration of security features
• Blocking Capability: Optionally block and alert on malicious content
• Developer Controls: Toggle security features and access analysis results

Quick Start

Wrap your application in the NehonixShieldProvider to enable security features:

• Basic Usage:

1import { NehonixShieldProvider, useNehonixShield } from "nehonix-uri-processor";
2
3const App = () => (
4  <NehonixShieldProvider>
5    <SecurityDemo />
6  </NehonixShieldProvider>
7);
8
9const SecurityDemo = () => {
10  const { scanUrl } = useNehonixShield();
11  const handleAnalyze = async () => {
12    const result = await scanUrl("https://example.com?category=books");
13    console.log(result);
14  };
15  return <button onClick={handleAnalyze}>Analyze URL</button>;
16};

Core Components

NehonixShieldProvider

The main provider component that makes security features available to your application.

1<NehonixShieldProvider defaultOptions={{ debug: false }} autoBlocking={true}>
2  {children}
3</NehonixShieldProvider>

Props:

• defaultOptions: Default options for security analysis
• autoBlocking: Whether to block malicious content by default

NehonixProtector

All-in-one protection component that enables both DOM and request analysis.

1<NehonixProtector
2  domOptions={{ includeScripts: true, scanIframes: true }}
3  requestOptions={{ includeFetch: true, includeXHR: true }}
4  domInterval={60000} // Re-scan DOM every minute
5>
6  <UserGeneratedContent />
7</NehonixProtector>

Props:

• domOptions: Options for DOM analysis
• requestOptions: Options for request analysis
• domInterval: Interval in milliseconds for periodic DOM scanning (null for no periodic scanning)

Example of using:

1// Basic setup with automatic blocking
2<NehonixShieldProvider autoBlocking={true}>
3  <YourApp />
4</NehonixShieldProvider>
5
6// Add comprehensive protection to a specific component
7<NehonixProtector
8  domOptions={{ includeScripts: true, scanIframes: true }}
9  requestOptions={{ includeFetch: true, includeXHR: true }}
10  domInterval={30000} // Re-scan DOM every 30 seconds
11>
12  <UserContent />
13</NehonixProtector>
14
15// Use the hook for manual control
16function SecureComponent() {
17  const { analyzeDom, blockingEnabled, setBlockingEnabled } = useNehonixShield();
18
19  const handleUserContent = (content) => {
20    // Manually analyze content before rendering
21    analyzeDom({
22      targetSelector: "#user-content",
23      includeScripts: true
24    });
25  };
26
27  return (
28    <div>
29      <button onClick={() => setBlockingEnabled(!blockingEnabled)}>
30        {blockingEnabled ? "Disable" : "Enable"} Protection
31      </button>
32      <div id="user-content">{/* user content */}</div>
33    </div>
34  );
35}

Read more.

Supported Encoding Types

• percentEncoding / url
• doublepercent
• base64
• hex / hexadecimal
• unicode
• htmlEntity / html
• punycode
• asciihex
• asciioct
• rot13
• base32
• urlSafeBase64
• jsEscape
• cssEscape
• utf7
• quotedPrintable
• decimalHtmlEntity
• rawHexadecimal
• jwt

Detection Capabilities

The library detects all supported encoding types, including nested encodings, with high accuracy.

Security Testing Features

• Parameter Analysis: Detects SQL injection, XSS, and path traversal patterns.
• WAF Bypass: Generates encoded variants for testing.
• Malicious Pattern Detection: Configurable sensitivity for detecting attacks.
• Sanitization: Sanitizes harmful inputs (use sanitizeInput cautiously due to instability).

More Information

• Detailed checkUrl and asyncCheckUrl documentation: checkUrlMethod.md
• Full documentation: lab.nehonix.space
• Changelog: changelog.md
• Previous versions:
  • v2.2.0
  • v2.1.2
  • v2.0.9

License

MIT