url-to-json-markdown

A TypeScript library that fetches URLs and converts them to structured JSON and Markdown format.

Built by 16x Writer and 16x Eval team.

Installation

1npm install url-to-json-markdown

Usage

1import { urlToJsonMarkdown } from 'url-to-json-markdown';
2
3// Reddit post (using fallback without credentials)
4const post = await urlToJsonMarkdown(
5  'https://www.reddit.com/r/example/comments/12345/title/'
6);
7console.log(post.title); // "Post Title"
8console.log(post.content); // "# Post Title\n\nPost content...\n\nby _username_ (↑ 123) 12/25/2024"
9console.log(post.type); // "reddit"
10
11// Reddit post with credentials (more reliable)
12const postWithCreds = await urlToJsonMarkdown(
13  'https://www.reddit.com/r/example/comments/12345/title/',
14  {
15    clientId: 'your_client_id',
16    clientSecret: 'your_client_secret',
17  }
18);
19
20// Reddit post with comments included
21const postWithComments = await urlToJsonMarkdown(
22  'https://www.reddit.com/r/example/comments/12345/title/',
23  {
24    clientId: 'your_client_id',
25    clientSecret: 'your_client_secret',
26    includeComments: true,
27  }
28);
29// Will include "## Comments" section with tree-structured comments
30
31// Reddit comment
32const comment = await urlToJsonMarkdown(
33  'https://www.reddit.com/r/example/comments/12345/comment/abc123/'
34);
35console.log(comment.title); // "First line of comment..."
36console.log(comment.content); // "# Comment by username\n\nComment text...\n\nby _username_ (↑ 45)"
37console.log(comment.type); // "reddit"
38
39// Reddit comment with child comments/replies
40const commentWithReplies = await urlToJsonMarkdown(
41  'https://www.reddit.com/r/example/comments/12345/comment/abc123/',
42  { includeComments: true }
43);
44// Will include "## Replies" section with tree-structured child comments
45
46// Generic web page
47const webpage = await urlToJsonMarkdown('https://example.com/article');
48console.log(webpage.title); // "Article Title"
49console.log(webpage.content); // "# Article Title\n\nMain content as markdown..."
50console.log(webpage.type); // "generic"

API

urlToJsonMarkdown(url: string, options?: RedditOptions): Promise<UrlToJsonResult>

Parameters:

• url - The URL to fetch and convert
• options - Optional Reddit configuration

Reddit Options:

1interface RedditOptions {
2  clientId?: string;
3  clientSecret?: string;
4  includeComments?: boolean;
5}

• clientId & clientSecret - Reddit API credentials for more reliable access
• includeComments - Include comments in a tree structure (Reddit posts) or child comments/replies (Reddit comments)

Return Type:

1interface UrlToJsonResult {
2  title: string;
3  content: string;
4  type: 'reddit' | 'generic';
5}

Reddit Access

For Reddit URLs, the library supports two modes:

1. Fallback mode (no credentials): Uses browser user agent to access Reddit's public JSON API. May be subject to rate limiting.

2. Authenticated mode (with credentials): Uses Reddit OAuth API for more reliable access. Requires Reddit app credentials.

To get Reddit credentials:

1. Go to https://www.reddit.com/prefs/apps
2. Create a new app (script type)
3. Use the client ID and secret

Supported URLs

• Reddit: Posts and comments from reddit.com
• Generic: Any website with automatic content extraction