Visitor IP Info (vipi)

Maxmind licenses should be properly considered in commercial and non-free use cases / environments.

:exclamation: :memo: This project uses Legacy Maxmind GeoIP .db databases that are now considered deprecated and no longer supported by Maxmind who only release v2 .mmdb files. :memo: :exclamation:
:clap: :star: Thanks to @terrablue for helping me port this code that had been broken for 3 years since node changes then - his a :crown: javascript legend :clap: :star:
:cupid: :+1: :v: :floppy_disk: DB files are provided curtesy of mailfud.org - MAXIMUM ONE DOWNLOAD AN IP PER HOUR. :floppy_disk: :v: :+1: :cupid:
:construction: if the linked mailfud.org mirror is down you can create your own DB using the script: GeoLite2xtables and the source CSV files from your account in Maxmind dashboard. Anther source is: miyuru.lk/geoiplegacy. :construction:
:heavy_check_mark: Versions tested: Node.js v16.16.0 & NPM 8.11.0 (August 2022) on :penguin: Linux :apple: macOS & :beginner: Windows :heavy_check_mark:

vipi is a Node.js utility using legacy Maxmind GeoIP DB as well legacy (v0.6.0) npm maxmind to provide known info of each IP:

• Location (Country, City, etc)
• Time Zone
• ASN

vipi is a standalone command-line-interface (CLI) utility and it may also act as a HTTP server / web daemon.

The intent with this utility is to provide dual log stores that are in Comman-Log-Format (CLF) and JSON contextual to application and user requirements. Session specific information can be stored and read to an automatically named file and or any other specified file.

Available functionality and features - by way of CLI arguments / parameters or query-strings (get-parameters) in daemon mode are documented herein.

Installation

npm

Using as a npm an OS with Node.js already installed.

1npm install -g vipi	# global install or with `sudo` for CLI use.
2npm install vipi	# for use in project or locally

Usage

Once installed vipi can be used as a standalone command or further extended as module in your Node.js script / project.

shell / Command Line Interface (CLI)

Once installed you may run in shell

1vipi		# // brief help & options
2man vipi	# // for manual if install globally
3man vipi.1.gz	# // manual file in repository / package root.

Example lookup uses:

1vipi 5.5.5.5 | jq '.'	# single IP lookup & `jq` pretty JSON response.

1[
2  {
3    "ip": "5.5.5.5",
4    "date": "2022-08-04T18:01:19.593Z",
5    "location": {
6        "countryCode": "DE",
7        "countryName": "Germany",
8        "region": "00",
9        "city": null,
10        "postalCode": null,
11        "latitude": 51.29929999999999,
12        "longitude": 9.49090000000001,
13        "dmaCode": 0,
14        "areaCode": 0,
15        "metroCode": 0,
16        "continentCode": "EU",
17        "regionName": null
18    },
19    "asn": "AS6805TelefonicaGermany",
20    "timeZone": "Europe/Berlin"
21  }
22]

1vipi 208.67.222.222 208.67.220.220	# multi IP lookups
2vipi 8.8.8.8 8.8.4.4 -cd		# include distance between pairs

Custom & automatic saving of JSON Object & String CLF key(s) for queried IP:

1vipi 208.67.222.222 -sa -skc='200 "GET / HTTP/1.1"'
2# Key string to save for IP entry in Auto-Save CLF logs

1vipi 208.67.222.222 -sa -sko='{"x":9}'
2# Key JSON Object string for IP entry in Auto-Save OBJ logs

1vipi 208.67.222.222 -sf=S3141_user -skc='200 "GET / HTTP/1.1" -sko='{"x":9}'
2# log OBJ & CLF to specified file 'S3141_user' session file(s)

Logs are saved to the specified absolute or relative path from where execution occurred (pwd / cwd).

daemon / HTTP Server

To execute vipi in web / daemon either -d, --daemon or -dp, --dipport options are required such as:

1vipi -d				# // default mode
2vipi -dp=127.0.0.1:65534	# // custom IP:PORT binding

Default daemon options allow for read, write and return to be freely available; post launch - visiting the address of the instance in a browser without any option ( or with !_help || !_h parameter) should yield a brief of usage and available parameters. Eg:

1curl 127.0.0.1:59999	# will output:

|\-------------------------/|
| ==== Visitor IP Info ==== |
|/_________________________\|
	vipi - v0.0.4

Usage (?!_=...):
	http://127.0.0.1:59999/?!_=ua		# use the inquiring address instead of ip.
	http://127.0.0.1:59999/?!_=208.67.222.222	# query & get results in text/plain

(&) Query String / Get-Parameters:
	!_as,	!_asnumber		Includes ASN number of IP queried.
	!_cd,	!_cdistance		Calculate & Include approximate distances in Kilometres between IPs pairs.
	!_df=,	!_deletefile=	Delete obj & clf named file(s).
	!_da,	!_deleteall		Deletes all clf & obj file(s) resetting to new.
	!_h,	!_help			Shows this help page.
	!_js,	!_json			Return content-type: 'application/json' instead of 'text/plain'.
	!_ls,	!_listsaves		Lists all available user obj json files.
	!_nu,	!_noua			Include requesting users agent in save.
	!_nr,	!_noreturn		No output or return.
	!_nc,	!_noclf			Include requesting users agent in save.
	!_ra,	!_readauto		Return content-type: 'application/json'
	!_rf=,	!_readfile=		Contents of log file to read & return.
	!_sua,	!_saveua		Include requesting users agent in save.
	!_tp,	!_textplain		Return content-type: 'text/plain'.
	!_sa,	!_saveauto		Save to file named using date: 'IP [DATE] KEY' (KEY optional) in .clf & .json file.
	!_sf=,	!_savefile=		Save like -sa: using specified file prefix name for .clf & .json saves.
	!_skc=,	!_skclf=		Save KEY clf string URL-Encoded string eg: '200 "GET / HTTP/1.1"' to save.
	!_sko=,	!_skobj=		Save KEY object json string URL-Encoded eg '{"eg":1}' to save.
	!_tz,	!_timezone		Includes time-zone of IP location queried.
	!_xi,	!_xinfo			Show OS / Node.js / Maxmind DB related info & exit.

Requests made to the server are by way of Query-String(s) / Get-Parameter(s) which are prefixed with !_. When in default non-referral mode - the !_= parameter is used to pass the IP being queried.

1curl '127.0.0.1:59999/?!_=208.67.222.222'	# will output:

1[{"ip":"208.67.222.222","date":"2016-02-04T11:40:01.984Z","location":{"countryCode":"US","countryName":"United States","region":"CA","city":"San Francisco","postalCode":"94107","latitude":37.7697,"longitude":-122.39330000000001,"dmaCode":807,"areaCode":415,"metroCode":807,"continentCode":"NA","regionName":"California"},"ua":"curl/7.38.0","timeZone":"America/Los_Angeles","asn":"AS36692 OpenDNS, LLC"}]

Multiple IP's may also be passed in sequence using multiple !_= values or using a single comma (,) separated value such as:

1curl '127.0.0.1:59999/?!_=208.67.222.222,8.8.8.8'	# two (2x) IPs comma separated
2curl '127.0.0.1:59999/?!_=208.67.222.222&!_=8.8.8.8'	# serialised.

In daemon mode GeoIP DB files are written at the path of execution in the directory: vipi_dbs.

A JSON Object key and CLF string key can be as passed as Percent-Encoded (URL-Encoded) strings which can be saved with each lookup ip where save features have been enabled. For example doing:

1curl '127.0.0.1:59999/?!_=208.67.222.222&!_skc=200%20%22GET%20%2F%20HTTP%2F1.1%22'
2# OBJ Key = '{"x":9}' - Object saved for this entry in JSON logs

1curl '127.0.0.1:59999/?!_=208.67.222.222&!_sko=%7B%22x%22%3A9%7D'
2# CLF Key = '200 "GET / HTTP/1.1"' - String saved for this entry in CLF logs

1curl '127.0.0.1:59999/?!_=208.67.222.222&!_sko=%7B%22x%22%3A9%7D&!_skc=200%20%22GET%20%2F%20HTTP%2F1.1%22'
2# Save both CLF & OBJ keys.

Collector / Silent daemon

Semi or fully silent daemon mode(s) are possible with the -dn... & -dx... set of parameters where they be act as collectors; a use case may be:.

1vipi -d -dno -dnr -dnw -dur -dxa -dsa	# // only collects auto saving & ignoring all !_ parameters.

Node.js & npm

Using vipi as an npm module is also possible though the original npm maxmind is recommended instead.
This module ships with the following available methods / functions:

function	parameters & description
.lookup(IPs, true, false)	(1st) String or Array of IP, (2nd) distance calculation of pairs, (3rd) benchmark mode.
.enableupdates(Hours)	Enables automatic update checks every 24 Hours where no value is passed

Simple example:

1var mVIPI = require("vipi");
2var aReturn = mVIPI.lookup(["208.67.222.222","208.67.220.220"]);
3mVIPI.enableupdates();
4console.log("IP Details:\n", aReturn);

Error Handling

Any error(s) incurred in the lookup process result in a String return containing ERROR: in CLI / Daemon modes or an array containing the same value when used as an array. Other queried requests which do not result in an error are not returned; this should NOT prevent or stop the logging of successful IP's in damon modes.

---

Notes

Performance

A performance of ~ 100,000 to 300,000 lookups per second can be expected in non-daemon / http modes with most modern (2015) X86 / AMD64 computers subject to available resources including the node.js version, hardware specification such as DDR as well as other architectural consideration. Daemon / http mode - are limited by the asynchronous and single process nature of Node.js; thus any concurrency beyond 100,000 is unlikely and not recommended; for greater concurrent volumes of lookups consider load-balancing / running multiple version of the service.

Benchmarking

The author of GoAccess (@allinurl) - had appropriately asked whether the acclaimed numbers herein have been tested; an example benchmark case has subsequently been included in vipi_benchmark.js.
The included benchmark performs a linear ordered lookup of over 100,000 IP which goes to demonstrate the expected number of looks that may be expected from the host computer on which vipi is executed.
Two example AMD64 architectures which were over a decade apart in age (2015 & older) are shown below:

Older Hardware 2004 Era

1cat /proc/cpuinfo && sudo dmidecode --type 17 # or 'hwinfo' 
2  # CPU - Model: Intel(R) Xeon(TM) CPU 3.60GHz
3  #  Clock: 3591 MHz
4  #  L1 Cache: 16 kb
5  #  L2 Cache: 1024 kb
6  # MEMORY - Type: DDR2 - ECC RAM - Speed: 400 MHz
7  # -------
8
9vipi_benchmark ;
10  # Doing lookups of:  100000  IPs ...
11  # Completed approximatly: 300000 operations (location, timezone & asn lookups)
12  # IP READ Time in Seconds: 0.16299986839294434
13  # LOOKUP Time in Seconds: 6.851000070571899
14  # TOTAL execution Time: 7.014999866485596
15  # EXPECTED lookups a sec: 14596.40913295923

Modern Hardware 2015 Era

1cat /proc/cpuinfo && sudo dmidecode --type 17 ;	# or 'hwinfo' 
2  # CPU - Model: Intel(R) Core(TM) i7-5820K CPU @ 3.30GHz
3  #  Clock: 2400 MHz
4  #  L1 Cache: 64 kb
5  #  L2 Cache: 256 kb
6  #  L3 Cache: 2560 kb
7  # MEMORY - Type: DDR4 - NON-ECC RAM - Speed: 2400 MHz
8  # -------
9
10vipi_benchmark ;
11  # Doing lookups of:  100000  IPs ...
12  # Completed approximately: 300000 operations (location, timezone & asn lookups) 
13  # IP READ Time in Seconds: 0.06599998474121094 
14  # LOOKUP Time in Seconds: 1.502000093460083 
15  # TOTAL execution Time: 1.569000005722046 
16  # EXPECTED lookups a sec: 66577.89199575545

Raspberry Pi 1 model B+

1cat /proc/cpuinfo && sudo dmidecode --type 17 ; # or 'hwinfo' 
2  # CPU - Model: 700 MHz single-core ARM1176JZF-S
3  #  Clock: 600 MHz
4  #  L1 Cache: 16 kb
5  #  L2 Cache: 128 kb
6  # MEMORY - Type: SDRAM - Speed: 400 MHz
7  # -------
8
9vipi_benchmark ;
10  # Doing lookups of:  100000  IPs ...
11  # Completed approximately: 300000 operations (location, timezone & asn lookups) 
12  # IP READ Time in Seconds: 3.819999933242798
13  # LOOKUP Time in Seconds: 510.35199999809265
14  # TOTAL execution Time: 514.1749999523163
15  # EXPECTED lookups a sec: 195.94319215046426

http / daemon testing

The daemon / http mode can be tested using ab tool. For example to perform 10,000 concurrent requests:

1ulimit -a ;	# check user limits
2ulimit -n 1000000 ; # set as high
3ab -n 10000 -c 10000 'http://127.0.0.1:59999/?!_=208.67.222.222&!_nr' ;
4  # ...
5  # Concurrency Level:      10000
6  # Time taken for tests:   2.651 seconds
7  # Complete requests:      10000
8  # Failed requests:        0
9  # Total transferred:      1590000 bytes
10  # HTML transferred:       50000 bytes
11  # Requests per second:    3772.14 [#/sec] (mean)
12  # Time per request:       2651.013 [ms] (mean)
13  # Time per request:       0.265 [ms] (mean, across all concurrent requests)
14  # Transfer rate:          585.71 [Kbytes/sec] received
15  # Connection Times (ms)
16  #              min  mean[+/-sd] median   max
17  # Connect:        0  277 400.0     87    1004
18  # Processing:    13  243 269.8    128    1617
19  # Waiting:       13  243 269.8    128    1617
20  # Total:         15  521 430.2    505    2617
21  # ...

This type of testing is limited by the servomechanism of loopback (127.0.0.1); use of (non-loopback) adaptors may yield greater performances.

This product includes GeoLite2 data created by MaxMind, available from Maxmind.

Version

0.0.4

License

GPLv3