urllib

Request HTTP URLs in a complex world — basic and digest authentication, redirections, timeout and more.

Install

1npm install urllib

Usage

TypeScript and ESM

1import { request } from 'urllib';
2
3const { data, res } = await request('http://cnodejs.org/');
4// result: { data: Buffer, res: Response }
5console.log('status: %s, body size: %d, headers: %j', res.status, data.length, res.headers);

CommonJS

1const { request } = require('urllib');
2
3const { data, res } = await request('http://cnodejs.org/');
4// result: { data: Buffer, res: Response }
5console.log('status: %s, body size: %d, headers: %j', res.status, data.length, res.headers);

API Doc

Method: async request(url[, options])

Arguments

• url String | Object - The URL to request, either a String or a Object that return by url.parse.
• options Object - Optional
  • method String - Request method, defaults to GET. Could be GET, POST, DELETE or PUT. Alias 'type'.
  • data Object - Data to be sent. Will be stringify automatically.
  • content String | Buffer - Manually set the content of payload. If set, data will be ignored.
  • stream stream.Readable - Stream to be pipe to the remote. If set, data and content will be ignored.
  • writeStream stream.Writable - A writable stream to be piped by the response stream. Responding data will be write to this stream and callback will be called with data set null after finished writing.
  • files {Array<ReadStream|Buffer|String> | Object | ReadStream | Buffer | String - The files will send with multipart/form-data format, base on formstream. If method not set, will use POST method by default.
  • contentType String - Type of request data. Could be json (Notes: not use application/json here). If it's json, will auto set Content-Type: application/json header.
  • dataType String - Type of response data. Could be text or json. If it's text, the callbacked data would be a String. If it's json, the data of callback would be a parsed JSON Object and will auto set Accept: application/json header. Default callbacked data would be a Buffer.
  • fixJSONCtlChars Boolean - Fix the control characters (U+0000 through U+001F) before JSON parse response. Default is false.
  • headers Object - Request headers.
  • timeout Number | Array - Request timeout in milliseconds for connecting phase and response receiving phase. Default is 5000. You can use timeout: 5000 to tell urllib use same timeout on two phase or set them seperately such as timeout: [3000, 5000], which will set connecting timeout to 3s and response 5s.
  • keepAliveTimeout number | null - Default is 4000, 4 seconds - The timeout after which a socket without active requests will time out. Monitors time between activity on a connected socket. This value may be overridden by keep-alive hints from the server. See MDN: HTTP - Headers - Keep-Alive directives for more details.
  • auth String - username:password used in HTTP Basic Authorization.
  • digestAuth String - username:password used in HTTP Digest Authorization.
  • followRedirect Boolean - follow HTTP 3xx responses as redirects. defaults to true.
  • maxRedirects Number - The maximum number of redirects to follow, defaults to 10.
  • formatRedirectUrl Function - Format the redirect url by your self. Default is url.resolve(from, to).
  • beforeRequest Function - Before request hook, you can change every thing here.
  • streaming Boolean - let you get the res object when request connected, default false. alias customResponse
  • compressed Boolean - Accept gzip, br response content and auto decode it, default is true.
  • timing Boolean - Enable timing or not, default is true.
  • socketPath String | null - request a unix socket service, default is null.
  • highWaterMark Number - default is 67108864, 64 KiB.

Options: options.data

When making a request:

1await request('https://example.com', {
2  method: 'GET',
3  data: {
4    'a': 'hello',
5    'b': 'world',
6  },
7});

For GET request, data will be stringify to query string, e.g. http://example.com/?a=hello&b=world.

For others like POST, PATCH or PUT request, in defaults, the data will be stringify into application/x-www-form-urlencoded format if content-type header is not set.

If content-type is application/json, the data will be JSON.stringify to JSON data format.

Options: options.content

options.content is useful when you wish to construct the request body by yourself, for example making a content-type: application/json request.

Notes that if you want to send a JSON body, you should stringify it yourself:

1await request('https://example.com', {
2  method: 'POST',
3  headers: {
4    'Content-Type': 'application/json',
5  },
6  content: JSON.stringify({
7    a: 'hello',
8    b: 'world',
9  }),
10});

It would make a HTTP request like:

1POST / HTTP/1.1
2host: example.com
3content-type: application/json
4
5{
6  "a": "hello",
7  "b": "world"
8}

This exmaple can use options.data with application/json content type:

1await request('https://example.com', {
2  method: 'POST',
3  headers: {
4    'content-type': 'application/json'
5  },
6  data: {
7    a: 'hello',
8    b: 'world',
9  }
10});

Options: options.files

Upload a file with a hello field.

1await request('https://example.com/upload', {
2  method: 'POST',
3  files: __filename,
4  data: {
5    hello: 'hello urllib',
6  },
7});

Upload multi files with a hello field.

1await request('https://example.com/upload', {
2  method: 'POST',
3  files: [
4    __filename,
5    fs.createReadStream(__filename),
6    Buffer.from('mock file content'),
7  ],
8  data: {
9    hello: 'hello urllib with multi files',
10  },
11});

Custom file field name with uploadfile.

1await request('https://example.com/upload', {
2  method: 'POST',
3  files: {
4    uploadfile: __filename,
5  },
6});

Response Object

Response is normal object, it contains:

• status or statusCode: response status code.
  • -1 meaning some network error like ENOTFOUND
  • -2 meaning ConnectionTimeoutError
• headers: response http headers, default is {}
• size: response size
• aborted: response was aborted or not
• rt: total request and response time in ms.
• timing: timing object if timing enable.
• socket: socket info

Run test with debug log

1NODE_DEBUG=urllib:* npm test

Mocking Request

export from undici

1import { strict as assert } from 'assert';
2import { MockAgent, setGlobalDispatcher, request } from 'urllib';
3
4const mockAgent = new MockAgent();
5setGlobalDispatcher(mockAgent);
6
7const mockPool = mockAgent.get('http://localhost:7001');
8
9mockPool.intercept({
10  path: '/foo',
11  method: 'POST',
12}).reply(400, {
13  message: 'mock 400 bad request',
14});
15
16const response = await request('http://localhost:7001/foo', {
17  method: 'POST',
18  dataType: 'json',
19});
20assert.equal(response.status, 400);
21assert.deepEqual(response.data, { message: 'mock 400 bad request' });

Request through a http proxy

export from undici

1import { ProxyAgent, request } from 'urllib';
2
3const proxyAgent = new ProxyAgent('http://my.proxy.com:8080');
4const response = await request('https://www.npmjs.com/package/urllib', {
5  dispatcher: proxyAgent,
6});
7console.log(response.status, response.headers);

Benchmarks

Fork undici benchmarks script

Connections 1

Tests	Samples	Result	Tolerance	Difference with slowest
http - no keepalive	15	6.38 req/sec	± 2.44 %	-
http - keepalive	10	6.77 req/sec	± 2.35 %	+ 6.13 %
urllib2 - request	45	40.13 req/sec	± 2.88 %	+ 528.66 %
urllib3 - request	10	58.51 req/sec	± 2.52 %	+ 816.64 %
undici - pipeline	5	59.12 req/sec	± 2.47 %	+ 826.18 %
undici - fetch	15	60.42 req/sec	± 3.00 %	+ 846.60 %
undici - dispatch	5	60.58 req/sec	± 1.39 %	+ 848.99 %
undici - stream	5	61.30 req/sec	± 1.31 %	+ 860.39 %
undici - request	5	61.74 req/sec	± 2.03 %	+ 867.20 %

Connections 50

Tests	Samples	Result	Tolerance	Difference with slowest
urllib2 - request	51	1465.40 req/sec	± 14.40 %	-
undici - fetch	40	3121.10 req/sec	± 2.82 %	+ 112.99 %
http - no keepalive	45	3355.42 req/sec	± 2.84 %	+ 128.98 %
http - keepalive	51	5179.55 req/sec	± 36.61 %	+ 253.46 %
urllib3 - request	30	7045.86 req/sec	± 2.93 %	+ 380.82 %
undici - pipeline	50	8306.92 req/sec	± 2.99 %	+ 466.87 %
undici - request	51	9552.59 req/sec	± 13.13 %	+ 551.88 %
undici - stream	45	12523.45 req/sec	± 2.97 %	+ 754.61 %
undici - dispatch	51	12970.18 req/sec	± 3.15 %	+ 785.10 %

Contributors

fengmk2	dead-horse	semantic-release-bot	xingrz	popomore	JacksonTian
ibigbug	greenkeeperio-bot	atian25	killagu	paambaati	tremby
denghongcai	gemwuu	XadillaX	alsotang	leoner	hyj1991
isayme	cyjake	whxaxes	chadxz	adapt0	danielwpz
danielsss	Jeff-Tian	nick-ng	rishavsharan	willizm	davidkhala
aleafs	Amunu	azure-pipelines[bot]	capsice	changzhiwin	yuzhigang33
elrrrrrrr	fishbar	gxcsoccer	mars-coder	rockdai	dickeylth
aladdin-add

This project follows the git-contributor spec, auto updated at Mon Dec 04 2023 00:13:39 GMT+0800.

License

MIT