args-tokenizer

args-tokenizer is a lightweight JavaScript library for parsing shell commands with arguments into an argv array. This makes it easy to work with command-line tools and libraries that expect an array format for arguments, such as tinyexec.

Features

• Simple and intuitive API.
• Handles quoted strings and escapes correctly.
• Supports multiline input.
• Ideal for parsing human-readable shell commands, especially curl-style commands.

---

  Made at Webstudio, open source website builder.

---

Installation

Install args-tokenizer:

1npm install args-tokenizer

Usage

Here's how you can use args-tokenizer to parse shell commands:

1import { tokenizeArgs } from "args-tokenizer";
2
3const args = tokenizeArgs(`ls -la "./src"`);
4console.log(args); // ["ls", "-la", "./src"]

Multiline Input Support

args-tokenizer also supports multiline commands, such as:

1const args = tokenizeArgs(`
2  curl \\
3    -X POST \\
4    "https://my-url.com"
5`);
6console.log(args); // ["curl", "-X", "POST", "https://my-url.com"]

Example with tinyexec

One common use case is passing more human-readable commands into the tinyexec library:

1import { tokenizeArgs } from "args-tokenizer";
2import { x } from "tinyexec";
3
4const [command, ...args] = tokenizeArgs("ls -la");
5const result = await x(command, args);
6console.log(result.stdout);

API

tokenizeArgs(command: string, options: Options): string[]

Parses a shell command string into an array of arguments. Properly handles:

• Quoted strings (e.g., '"./path/to/file"').
• Escaped characters (e.g., \").
• Multiline commands (e.g., lines ending with \\).

Options

• loose: If true, the tokenizer will not throw an error when closing quotes are missing. Default is false.

Examples

1// Without loose option (default behavior)
2// This will throw an error due to the missing closing quote
3tokenizeArgs('command "arg1 arg2');
4
5// With loose option enabled
6const args = tokenizeArgs('command "arg1 arg2', { loose: true });
7// ['command', 'arg1 arg2']

License

This project is licensed under the MIT License.

Contributing

Contributions are welcome! Feel free to open issues or submit pull requests to improve the library.