The curl --trace parser

The story

Did you know that you can record raw HTTP communication of Curl command-line tool with the --trace and --trace-ascii option? It's the only way I know to get raw HTTP communication without using the tcpdump or wireshark. For example, this trick is very useful for the proper introspection into HTTP communication of an undocumented RESTful API.

The only glitch is that cURL --trace saves data in its custom format, far from human-friendly, saving chunks as they are being received and splitting them by packets. If you want a human readable form then this parser is what you need. Delivered as a Node.js package.

Usage

$ curl --trace - http://httpbin.org/ip | curl-trace-parser

Sample API

We will be using this sample API created with the Apiary.io mock server to demonstrate tracing an HTTP communication and the use of the cURL trace parser.

Install the cURL trace parser

1$ npm install -g curl-trace-parser

Record your first trace file

1$ curl --trace tracefile --header "Content-Type: application/json" \
2--request POST \
3--data-binary "{ \"product\":\"1AB23ORM\", \"quantity\": 2 }" \
4"http://curltraceparser.apiary.io/shopping-cart"

Note this cURL example is copied and pasted from Apiary interactive API documentation.

Examples

--raw format

The output is ASCII representation of a raw HTTP message with few modifications:

• Request line begins with >
• Response line begins with <
• Request and Response is delimited by CR+LF
• Both Request and Response are terminated by an extra trailing LF

Note: This is little bit tricky because HTTP RFC does not have declared delimiter for Request and Response, for obvious reasons.

1$ cat tracefile | curl-trace-parser --raw
2> POST /shopping-cart HTTP/1.1
3> User-Agent: curl/7.24.0 (x86_64-apple-darwin12.0) libcurl/7.24.0 OpenSSL/0.9.8x zlib/1.2.5
4> Host: curltraceparser.apiary.io
5> Accept: */*
6> Content-Type: application/json
7> Content-Length: 39
8>
9> { "product":"1AB23ORM", "quantity": 2 }
10
11< HTTP/1.1 201 Created
12< Content-Type: application/json
13< Date: Tue, 30 Jul 2013 11:32:30 GMT
14< X-Apiary-Ratelimit-Limit: 120
15< X-Apiary-Ratelimit-Remaining: 119
16< Content-Length: 50
17< Connection: keep-alive
18<
19< { "status": "created", "url": "/shopping-cart/2" }

--blueprint format

The output is HTTP Request and Response in the API blueprint format which is the superset of markdown.

$ cat tracefile | ./bin/curl-trace-parser --blueprint
# POST /shopping-cart
+ Request
    + Headers

            User-Agent:curl/7.24.0 (x86_64-apple-darwin12.0) libcurl/7.24.0 OpenSSL/0.9.8x zlib/1.2.5
            Host:curltraceparser.apiary.io
            Accept:*/*
            Content-Type:application/json
            Content-Length:39

    + Body

            { "product":"1AB23ORM", "quantity": 2 }

+ Response 201
    + Headers

            Content-Type:application/json
            Date:Tue, 30 Jul 2013 11:32:30 GMT
            X-Apiary-Ratelimit-Limit:120
            X-Apiary-Ratelimit-Remaining:119
            Content-Length:50
            Connection:keep-alive

    + Body

            { "status": "created", "url": "/shopping-cart/2" }

Note: This format does not expect any CR+LF in the message body

Parse the trace to raw HTTP file using Node.JS

1var fs = require('fs');
2var parser = require('curl-trace-parser');
3fs.readFile('./tracefile', 'utf8', function (err,trace) {
4  console.log(parser.parse(trace));
5})

Output format reverse parser

1var fs = require('fs');
2var parser = require('curl-trace-parser');
3fs.readFile('./tracefile', 'utf8', function (err,trace) {
4  console.log(parser.parseBack(trace));
5})

API Reference

parse(traceString) - parse string with trace to object with raw request and response

parseToString(traceString) - parse string with trace to output format

parseBack(outputString) - parse string with output format back to object with raw request an resposne