@extra/proxy-router

A plugin for playwright-extra and puppeteer-extra to route proxies dynamically.

Install

1yarn add @extra/proxy-router
2# - or -
3npm install @extra/proxy-router

1yarn add playwright playwright-extra @extra/proxy-router
2# - or -
3npm install playwright playwright-extra @extra/proxy-router

1yarn add puppeteer puppeteer-extra @extra/proxy-router
2# - or -
3npm install puppeteer puppeteer-extra @extra/proxy-router

Compatibility

💫	Chromium	Chrome	Firefox	Webkit
playwright-extra	✅	✅	✅	✅
puppeteer-extra	✅	✅	🕒	-

Features

The plugin makes using proxies in the browser a lot more convenient:

• Handles proxy authentication
• Multiple proxies can be used
• Flexible proxy routing using the host/domain
• Change proxies dynamically after browser launch
• Collect traffic stats per proxy or host
• Uses native browser features, no performance loss

Usage

Using puppeteer? To use the following playwright examples simply change your imports

Simple

A single proxy for all browser connections

1// playwright-extra is a drop-in replacement for playwright,
2// it augments the installed playwright with plugin functionality
3// Note: Instead of chromium you can use firefox and webkit as well.
4const { chromium } = require('playwright-extra')
5
6// Configure and add the proxy router plugin with a default proxy
7const ProxyRouter = require('@extra/proxy-router')
8chromium.use(
9  ProxyRouter({
10    proxies: { DEFAULT: 'http://user:pass@proxyhost:port' },
11  })
12)
13
14// That's it, the default proxy will be used and proxy authentication handled automatically
15chromium.launch({ headless: false }).then(async (browser) => {
16  const page = await browser.newPage()
17  await page.goto('https://canhazip.com', { waitUntil: 'domcontentloaded' })
18  const ip = await page.evaluate('document.body.innerText')
19  console.log('Outbound IP:', ip)
20  await browser.close()
21})

Dynamic routing

Use multiple proxies and route connections flexibly

1// playwright-extra is a drop-in replacement for playwright,
2// it augments the installed playwright with plugin functionality
3// Note: Instead of chromium you can use firefox and webkit as well.
4const { chromium } = require('playwright-extra')
5
6// Configure the proxy router plugin
7const ProxyRouter = require('@extra/proxy-router')
8const proxyRouter = ProxyRouter({
9  // define the available proxies (replace this with your proxies)
10  proxies: {
11    // the default browser proxy, can be `null` as well for direct connections
12    DEFAULT: 'http://user:pass@proxyhost:port',
13    // optionally define more proxies you can use in `routeByHost`
14    // you can use whatever names you'd like for them
15    DATACENTER: 'http://user:pass@proxyhost2:port',
16    RESIDENTIAL_US: 'http://user:pass@proxyhost3:port',
17  },
18  // optional function for flexible proxy routing
19  // if this is not specified the `DEFAULT` proxy will be used for all connections
20  routeByHost: async ({ host }) => {
21    if (['pagead2.googlesyndication.com', 'fonts.gstatic.com'].includes(host)) {
22      return 'ABORT' // block connection to certain hosts
23    }
24    if (host.includes('google')) {
25      return 'DIRECT' // use a direct connection for all google domains
26    }
27    if (host.endsWith('.tile.openstreetmap.org')) {
28      return 'DATACENTER' // route heavy images through datacenter proxy
29    }
30    if (host === 'canhazip.com') {
31      return 'RESIDENTIAL_US' // special proxy for this domain
32    }
33    // everything else will use `DEFAULT` proxy
34  },
35})
36
37// Add the plugin
38chromium.use(proxyRouter)
39
40// Launch a browser and run some IP checks
41chromium.launch({ headless: true }).then(async (browser) => {
42  const page = await browser.newPage()
43
44  await page.goto('https://showmyip.com/', { waitUntil: 'domcontentloaded' })
45  const ip1 = await page.evaluate("document.querySelector('#ipv4').innerText")
46  console.log('Outbound IP #1:', ip1)
47  // => 77.191.128.0 (the DEFAULT proxy)
48
49  await page.goto('https://canhazip.com', { waitUntil: 'domcontentloaded' })
50  const ip2 = await page.evaluate('document.body.innerText')
51  console.log('Outbound IP #2:', ip2)
52  // => 104.179.129.27 (the RESIDENTIAL_US proxy)
53
54  console.log(proxyRouter.stats.connectionLog) // list of connections (host => proxy name)
55  // { id: 0, proxy: 'DIRECT', host: 'accounts.google.com' },
56  // { id: 1, proxy: 'DEFAULT', host: 'www.showmyip.com' },
57  // { id: 2, proxy: 'ABORT', host: 'pagead2.googlesyndication.com' },
58  // { id: 3, proxy: 'DEFAULT', host: 'unpkg.com' },
59  // ...
60
61  console.log(proxyRouter.stats.byProxy) // bytes used by proxy
62  // {
63  //   DATACENTER: 441734,
64  //   DEFAULT: 125823,
65  //   DIRECT: 100457,
66  //   RESIDENTIAL_US: 4764,
67  //   ABORT: 0
68  // }
69
70  console.log(proxyRouter.stats.byHost) // bytes used by host
71  // {
72  //   'a.tile.openstreetmap.org': 150685,
73  //   'c.tile.openstreetmap.org': 147054,
74  //   'b.tile.openstreetmap.org': 143995,
75  //   'unpkg.com': 57621,
76  //   'www.googletagmanager.com': 49572,
77  //   'www.showmyip.com': 40408,
78  // ...
79
80  await browser.close()
81})

Imports

1- const { chromium } = require('playwright-extra')
2+ const puppeteer = require('puppeteer-extra')
3// ...
4- chromium.use(proxyRouter)
5+ puppeteer.use(proxyRouter)
6// ...
7- chromium.launch()
8+ puppeteer.launch()
9// ...

1// You can use any browser: chromium, firefox, webkit
2import { firefox } from 'playwright-extra'
3import ProxyRouter from '@extra/proxy-router'
4// ...
5firefox.use(proxyRouter)

1import puppeteer from 'puppeteer-extra'
2import ProxyRouter from '@extra/proxy-router'
3// ...
4puppeteer.use(proxyRouter)

Debug logs

If you'd like to see debug output just run your script like so:

1# macOS/Linux (Bash)
2DEBUG=*proxy-router* node myscript.js
3
4# Windows (Powershell)
5$env:DEBUG='*proxy-router*';node myscript.js

How it works

The proxy router will launch a local proxy server and instruct the browser to use it. That local proxy server will in turn connect to the configured upstream proxy servers and relay connections depending on the optional user-defined routing function, while handling upstream proxy authentication and a few other things.

API

Options

1export interface ProxyRouterOpts {
2  /**
3   * A dictionary of proxies to be made available to the browser and router.
4   *
5   * An optional entry named `DEFAULT` will be used for all requests, unless overriden by `routeByHost`.
6   * If the `DEFAULT` entry is omitted no proxy will be used by default.
7   *
8   * The value of an entry can be a string (format: `http://user:pass@proxyhost:port`) or `null` (direct connection).
9   * Proxy authentication is handled automatically by the router.
10   *
11   * @example
12   * proxies: {
13   *   DEFAULT: "http://user:pass@proxyhost:port", // use this proxy by default
14   *   RESIDENTIAL_US: "http://user:pass@proxyhost2:port" // use this for specific hosts with `routeByHost`
15   * }
16   */
17  proxies?: {
18    /**
19     * The default proxy for the browser (format: `http://user:pass@proxyhost:port`),
20     * if omitted or `null` no proxy will be used by default
21     */
22    DEFAULT?: string | null
23    /**
24     * Any other custom proxy names which can be used for routing later
25     * (e.g. `'DATACENTER_US': 'http://user:pass@proxyhost:port'`)
26     */
27    [key: string]: string | null
28  }
29
30  /**
31   * An optional function to allow proxy routing based on the target host of the request.
32   *
33   * A return value of nothing, `null` or `DEFAULT` will result in the DEFAULT proxy being used as configured.
34   * A return value of `DIRECT` will result in no proxy being used.
35   * A return value of `ABORT` will cancel/block this request.
36   *
37   * Any other string as return value is assumed to be a reference to the configured `proxies` dict.
38   *
39   * @note The browser will most often establish only a single proxy connection per host.
40   *
41   * @example
42   * routeByHost: async ({ host }) => {
43   *   if (host.includes('google')) { return "DIRECT" }
44   *   return 'RESIDENTIAL_US'
45   * }
46   *
47   */
48  routeByHost?: RouteByHostFn
49  /** Collect traffic and connection stats, default: true */
50  collectStats?: boolean
51  /** Don't print any proxy connection errors to stderr, default: false */
52  muteProxyErrors?: boolean
53  /** Suppress proxy errors for specific hosts */
54  muteProxyErrorsForHost?: string[]
55  /** Options for the local proxy-chain server  */
56  proxyServerOpts?: ProxyServerOpts
57  /**
58   * Optionally exempt hosts from going through a proxy, even our internal routing proxy.
59   *
60   * Examples:
61   * `.com` or `chromium.org` or `.domain.com`
62   *
63   * @see
64   * https://chromium.googlesource.com/chromium/src/+/HEAD/net/docs/proxy.md#proxy-bypass-rules
65   * https://www-archive.mozilla.org/quality/networking/docs/aboutno_proxy_for.html
66   */
67  proxyBypassList?: string[]
68}

Alternatives

Proxy.pac files ^_Reference

• Only supported in chromium in headful mode
  • Despite the name (FindProxyForURL) can only route by host
• Firefox supports PAC files and including the path through a pref
• Only loaded once at browser launch, no dynamic proxies possible
• Does not handle authentication

Various "per-page proxy" plugins for puppeteer

• Advantage: Route proxies by page not host
• They rely on a massive hack: Using Node.js to send the requests instead of the browser
  • Will change the TLS fingerprint, error prone
• Uses CDP request interception which is chromium only
• Increased latency and resource overhead

License

Copyright © 2018 - 2023, berstend̡̲̫̹̠̖͚͓̔̄̓̐̄͛̀͘. Released under the MIT License.