CSS Tokenizer

This tools lets you tokenize CSS according to the CSS Syntax Specification. Tokenizing CSS is separating a string of CSS into its smallest, semantic parts — otherwise known as tokens.

This tool is intended to be used in other tools on the front and back end. It seeks to maintain:

• 100% compliance with the CSS syntax specification. ✨
• 100% code coverage. 🦺
• 100% static typing. 💪
• 1kB maximum contribution size. 📦
• Superior quality over Shark P. 🦈

Usage

Add the CSS tokenizer to your project:

1npm install @csstools/tokenizer

Tokenize CSS in JavaScript:

1import { tokenize } from '@csstools/tokenizer'
2
3for (const token of tokenize(cssText)) {
4  console.log(token) // logs an individual CSSToken
5}

Tokenize CSS in classical NodeJS:

1const { tokenizer } = require('@csstools/tokenizer')
2
3let iterator = tokenizer(cssText), iteration
4
5while (!(iteration = iterator()).done) {
6  console.log(iteration.value) // logs an individual CSSToken
7}

Tokenize CSS in client-side scripts:

1<script type="module">
2
3import { tokenize } from 'https://unpkg.com/@csstools/tokenizer?module'
4
5for (const token of tokenize(cssText)) {
6  console.log(token) // logs an individual CSSToken
7}
8
9</script>

Tokenize CSS in classical client-side scripts:

1<script src="http://unpkg.com/@csstools/tokenizer"></script>
2<script>
3
4const tokens = Array.from(tokenizeCSS(cssText)) // an array of CSSTokens
5
6</script>

Serialize tokens

1import { tokenize } from '@csstools/tokenizer'
2
3let cssOutput = '';
4for (const token of tokenize(cssText)) {
5  // mutate some tokens
6
7  cssOutput += token.lead + token.data + token.tail
8}
9
10console.log(cssOutput) // logs the CSS string

How it works

The CSS tokenizer separates a string of CSS into tokens.

1interface CSSToken {
2  /** Position in the string at which the token was retrieved. */
3  tick: number
4
5  /** Number identifying the kind of token. */
6  type:
7    | 1 // Symbol
8    | 2 // Comment
9    | 3 // Space
10    | 4 // Word
11    | 5 // Function
12    | 6 // Atword
13    | 7 // Hash
14    | 8 // String
15    | 9 // Number
16  
17  /** Code, like the character code of a symbol, or the character code of the opening parenthesis of a function. */
18  code: number
19
20  /** Lead, like the opening of a comment, the quotation mark of a string, or the name of a function. */
21  lead: string,
22
23  /** Data, like the numbers before a unit, the word after an at-sign, or the opening parenthesis of a Function. */
24  data: string,
25
26  /** Tail, like the unit after a number, or the closing of a comment. */
27  tail: string,
28}

As an example, the CSS string @media would become a Atword token where @ and media are recognized as distinct parts of that token. As another example, the CSS string 5px would become a Number token where 5 and px are recognized as distinct parts of that token. As a final example, the string 5px 10px would become 3 tokens; the Number as mentioned before (5px), a Space token that represents a single space ( ), and then another Number token (10px).

Benchmarks

As of August 23, 2021, these benchmarks were averaged from my local machine:

Benchmark: Tailwind CSS
  ┌────────────────────────────────────────────────────┬───────┬────────┬────────┐
  │                      (index)                       │  ms   │ ms/50k │ tokens │
  ├────────────────────────────────────────────────────┼───────┼────────┼────────┤
  │ CSSTree 1 x 35.04 ops/sec ±6.55% (64 runs sampled) │ 28.54 │  1.51  │ 946205 │
  │ CSSTree 2 x 41.76 ops/sec ±7.57% (58 runs sampled) │ 23.95 │  1.27  │ 946205 │
  │ PostCSS 8 x 14.18 ops/sec ±3.31% (40 runs sampled) │ 70.54 │  3.77  │ 935282 │
  │ Tokenizer x 17.40 ops/sec ±0.98% (48 runs sampled) │ 57.48 │  3.04  │ 946206 │
  └────────────────────────────────────────────────────┴───────┴────────┴────────┘

Benchmark: Bootstrap
  ┌───────────────────────────────────────────────────┬──────┬────────┬────────┐
  │                      (index)                      │  ms  │ ms/50k │ tokens │
  ├───────────────────────────────────────────────────┼──────┼────────┼────────┤
  │ CSSTree 1 x 600 ops/sec ±0.87% (96 runs sampled)  │ 1.67 │  1.41  │ 59236  │
  │ CSSTree 2 x 695 ops/sec ±0.08% (100 runs sampled) │ 1.44 │  1.21  │ 59236  │
  │ PostCSS 8 x 432 ops/sec ±0.94% (94 runs sampled)  │ 2.31 │  2.26  │ 51170  │
  │ Tokenizer x 288 ops/sec ±0.40% (93 runs sampled)  │ 3.48 │  2.93  │ 59237  │
  └───────────────────────────────────────────────────┴──────┴────────┴────────┘

Development

You wanna take a deeper dive? Awesome! Here are a few useful development commands.

npm run build

The build command creates all the files needed to run this tool in many different JavaScript environments.

1npm run build

npm run benchmark

The benchmark command builds the project and then tests its performance as compared to PostCSS. These benchmarks are run against Boostrap and Tailwind CSS.

1npm run benchmark

npm run test

The test command tests the coverage and accuracy of the tokenizer.

As of September 26, 2020, this tokenizer has 100% test coverage:

1npm run test