near-duplicates

A TypeScript npm package to find near duplicate pairs in a string set.

Usage

1npm install near-duplicates

1import { findNearDuplicates } from "near-duplicates";
2
3findNearDuplicates(["hello", "hallo", "halo"]);
4
5// you can also specify a threshold for the maximum distance
6// if not specified, it will be a default value of 12
7findNearDuplicates(["hello", "hallo", "halo"], 2);

Benchmarks

Here's a benchmark of the performance of findNearDuplicates using the default max distance of 12.

1 DEV  v2.1.4 near-duplicates
2
3 ✓ test/index.bench.ts (5) 5211ms
4   ✓ findNearDuplicates (5) 5210ms
5     name                                       hz     min      max     mean      p75      p99     p995     p999     rme  samples
6   · small strings                      214,439.85  0.0043   0.2387   0.0047   0.0046   0.0053   0.0071   0.0592  ±0.38%   107220   fastest
7   · 10 random strings (20-500 chars)    29,533.57  0.0148   0.2318   0.0339   0.0369   0.0557   0.1355   0.1764  ±0.58%    14767
8   · 100 random strings (20-500 chars)    1,469.32  0.5852   1.3275   0.6806   0.6818   1.1802   1.1990   1.3275  ±1.05%      735
9   · 500 random strings (20-500 chars)     98.7281  9.1328  11.3120  10.1288  10.8487  11.3120  11.3120  11.3120  ±2.03%       50
10   · 2k random strings (20-500 chars)       6.2139  155.68   173.60   160.93   162.18   173.60   173.60   173.60  ±2.25%       10   slowest
11