QuickPickle

QuickPickle is a plugin for Vitest to run tests written in Gherkin Syntax. It seamlessly integrates Gherkin Feature files into your Vitest testing workflow by parsing them with the official Gherkin Parser and running them as Vitest tests.

Features

• Seamless integration of Gherkin Feature files into Vitest testing workflow
• Full support for Gherkin 6, using the official Gherkin Parser
• Full ESM, typescript and javascript support (because it's Vitest)
• Vitest-native test extensions concurrent, sequential, skip, todo, and fails
• Multiple test environments with vitest.workspace configurations
• Custom world constructors, similar to CucumberJS

CucumberJS Feature List

• Full Gherkin 6 support
• Step definitions using Cucumber Expressions or Regex
• setWorldConstructor: custom world constructor
• hooks: BeforeAll, Before, BeforeStep, AfterStep, After, AfterAll
• named hooks
• tagged hooks
• skipping in a before hook: (use world.context.skip())
• defineParameterType for custom parameter types in Cucumber Expressions
• setDefaultTimeout (use testTimeout in Vitest config)
• setDefinitionFunctionWrapper (use Before and After hooks)
• setParallelCanAssign (use @concurrent and @sequential special tags)
• attachments with world.attach, world.log, and world.link not supported, use Vitest reporters
• DataTables fully implemented, with full DataTable interface
• DocStrings fully implemented, with mediaType support
• debugging (use Vitest debugging)
• dry run not precisely supported, but you can use vitest list for something similar
• esm 100% support through Vite (the reason for this package)
• fail fast (use bail option in Vitest config or command run, e.g. vitest --bail=1)
• filtering partial support (use testNamePattern to run tests matching a regex pattern)
• formatters (use Vitest reporters)
• parallel (use @parellel and @sequential special tags)
• profiles (use Vitest workspaces)
• rerun (press "f" in watch mode terminal)
• retry (use Vitest retry option)
• snippets (only async/await javascript is supported)
• transpiling 100% support through Vite (the reason for this package)

Benefits of Gherkin

• Write tests in plain English (or your team's language)

  Because it uses natural language, Gherkin Syntax enables all stakeholders to understand and contribute to tests, and having a common vocabulary to describe functionality makes it easy to agree on and verify what the program does.

• Reuse test step definitions to minimize test code

  Bugs can happen even in test code, so it's best if the test code changes as seldom as possible, but functionality changes all the time. A small library of reusable step definitions is much easier to maintain than a large corpus of test code.

Goals of QuickPickle

1. Support the main technical ideas behind Gherkin / Cucumber

   • Natural language for features
   • Discrete, re-usable step definitions

2. Make Gherkin tests easy to set up and use in Javascript projects

   • Run tests with Vitest, to avoid painful snafus from js/ts/esm/commonjs issues.
   • Provide a plugin for testing web applications. (completed with packages/playwright)
   • Provide a plugin for testing components. (in progress with packages/components)

3. Experiment with means of supporting open-source projects

   • Provide a Svelte component exposing Gherkin steps, so that end users can potentially write bug reports and feature requests in the form of Gherkin scenarios. (in progress with packages/svelte)
   • Provide a GitHub actions and/or bots that interface with AI to:
     • write Gherkin scenarios based on issue reports
     • write code based on Gherkin scenarios

4. Experiment with Gherkin for unsanctioned testing purposes

   Note: These ideas do NOT align with Cucumber best practices; use at your own risk (of being ridiculed on reddit, etc.)

   • Develop a library of common step definitions that could be used across projects to test UI interaction: Gherkin scenarios and steps should generally be written with domain-specific language instead of focusing on the user interface (see the "Lots of user interface details" section of [Cucumber Anti-Patterns]). However:

     1. The browser is a specific domain that is nonetheless common across all web projects; most will need an "I navigate to {string}" step.
     2. There may be a benefit to having a well-written set of Gherkin steps for UI testing that is re-usable across a wide range of projects.
     3. This would make it really easy for people to get started, even without writing any step definitions.

     (in progress with packages/playwright)

   • Develop a library of common step definitions that could be used across projects for "unit testing" components: Gherkin is meant for testing the behavior of an application, not the code, and in most cases it is not a good tool for unit testing. However:

     1. In some ways components represent units of behavior, often containing minimal code.
     2. It might be beneficial to have a library of well-written Gherkin steps for testing every type of interaction that it supports.

     (in progress with packages/components)

Installation

1npm install --save-dev quickpickle vitest

Configuration

To get QuickPickle working you'll need to do three things:

1. configure Vitest to use QuickPickle, in vite.config.ts or vitest.workspace.ts
2. create a step definition file to
   • import or create your step definitions
   • set a different world variable constructor (optional)
3. install & configure VSCode Cucumber plugin (optional but highly recommended)

QuickPickle configuration in Vitest

Add the quickpickle plugin to your Vitest configuration in vite.config.ts (or .js, etc.). Also add the configuration to get the feature files, step definitions, world file, etc.

1// vite.config.ts
2import { quickpickle, type QuickPickleConfigSetting } from './src';
3const qpOptions:QuickPickleConfigSetting = {}
4
5export default {
6  plugins: [
7    quickpickle(qpOptions) // <-- Add the quickpickle plugin
8  ],
9  test: {
10    include : [
11      'features/*.feature', // <-- Add Gherkin feature files into "test" configuration
12      // (you'll probably want other test files too, for unit tests etc.)
13    ],
14    setupFiles: ['./tests/tests.steps.ts'] // <-- specify each setupfile here
15  },
16};

If you have multiple configurations, you can also add the test settings in vitest.workspace.ts. This is also where you could set up separate configurations for components vs. application, different browser environments, different world constructors, etc.

1// vitest.workspace.ts
2import { defineWorkspace } from 'vitest/config';
3import type { QuickPickleConfigSetting } from 'quickpickle';
4
5const options1:QuickPickleConfigSetting = {} // some settings for e.g. e2e tests
6const options2:QuickPickleConfigSetting = {} // other settings for e.g. component tests
7
8export default defineWorkspace([
9  { // configuration for feature files testing the application
10    extends: './vite.config.ts',
11    test: {
12      name: 'e2e',
13      include: [ 'tests/*.feature' ],
14      setupFiles: [ 'tests/tests.steps.ts' ],
15      quickpickle: options1,
16    },
17  },
18  { // a second configuration for feature files testing components
19    extends: './vite.config.ts',
20    test: {
21      name: 'components',
22      include: [ 'src/lib/*.feature' ],
23      setupFiles: [ 'tests/components.steps.ts' ],
24      quickpickle: options2,
25    },
26  },
27  { // configuration for unit tests
28    test: {
29      name: 'unit',
30      include: [ 'tests/*.test.ts' ],
31    }
32  }
33])

The QuickPickle configuration options can be seen in the quickpickle defaultConfig object:

1export const defaultConfig: QuickPickleConfig = {
2
3  /**
4   * The root directory for the tests to run, from vite or vitest config
5   */
6  root: '',
7
8  /**
9   * The maximum time in ms to wait for a step to complete.
10   */
11  stepTimeout: 3000,
12
13  /**
14   * Tags to mark as todo, using Vitest's `test.todo` implementation.
15   */
16  todoTags: ['@todo','@wip'],
17
18  /**
19   * Tags to skip, using Vitest's `test.skip` implementation.
20   */
21  skipTags: ['@skip'],
22
23  /**
24   * Tags to mark as failing, using Vitest's `test.failing` implementation.
25   */
26  failTags: ['@fails', '@failing'],
27
28  /**
29   * Tags to mark as soft failing, allowing further steps to run until the end of the scenario.
30   */
31  softFailTags: ['@soft', '@softfail'],
32
33  /**
34   * Tags to run in parallel, using Vitest's `test.concurrent` implementation.
35   */
36  concurrentTags: ['@concurrent'],
37
38  /**
39   * Tags to run sequentially, using Vitest's `test.sequential` implementation.
40   */
41  sequentialTags: ['@sequential'],
42
43  /**
44   * Explode tags into multiple tests, e.g. for different browsers.
45   */
46  explodeTags: [],
47
48  /**
49   * The config for the World class. Must be serializable with JSON.stringify.
50   * Not used by the default World class, but may be used by plugins or custom
51   * implementations, like @quickpickle/playwright.
52   */
53  worldConfig: {},
54
55}

Step definition file

You'll always need a step definition file, to set up the step defintions and potentially the world variable constructor. Here is an exmaple if you want to use @quickpickle/playwright to test web sites:

1// tests/example.steps.ts
2import '@quickpickle/playwright/actions' // <-- import action step definitions from @quickpickle/playwright
3import '@quickpickle/playwright/outcomes' // <-- import outcome step definitions from @quickpickle/playwright
4
5import '@quickpickle/playwright/world' // <-- use the playwright world variable (optional)
6
7import { Given, When, Then } from 'quickpickle' // <-- the functions to write step definitions
8
9// Custom step definitions
10Given('a/another number {int}', async (world) => {
11  if (!world.numbers) world.numbers = []
12  world.numbers.push(int)
13})

Cucumber plugin for VSCode

If you use VSCode, you'll want a cucumber plugin for code completion when writing gherkin features. Try the official "Cucumber" plugin, by "Cucumber". You'll also need to configure it so that it sees your step definitions.

1// VSCode settings.json
2"cucumber.glue": [
3    "**/*.steps.{ts,js,mjs}",
4    "**/steps/*.{ts,js,mjs}"
5],

Usage

Write Feature files

Write your feature files in the directory specified above. Common convention is to place feature files in a "features" directory, though some prefer the "tests" directory. You can put them anywhere as long as they're listed in the "include" configuration in vite.config.ts.

1# features/example.feature
2Feature: A simple example
3
4  Scenario: Adding numbers
5    Given a number 1
6    And a number 2
7    And a number 3
8    And another number 3
9    Then the sum should be 9

Run tests

1npx vitest --run

Write step definitions

Write your step definitions in a typescript or javascript file as configured in the "setupFiles" declaration in your vitest config.

These files will be imported into the Vitest test runner, and the code for Given, When, Then will register each of the step definitions with quickpickle. These step definitions should run immediately, i.e. at the top level of the script, not as exported functions like a normal node module would do.

1// features/example.steps.ts
2import { Given, Then } from 'quickpickle'
3
4Given('a/another number {int}', (world, int) => {
5  if (!world.numbers) world.numbers = [int]
6  else world.numbers.push(int)
7})
8
9Then('the sum should be {int}', (world, int) => {
10  expect(world.numbers.reduce((a,b) => a + b, 0)).toBe(int)
11})

Define a custom "world" variable constructor

To define a custom world variable constructor in QuickPickle, you can use the setWorldConstructor function exported from the package. This allows you to create a custom World class that extends the QuickPickleWorld interface, enabling you to add your own properties and methods to the world object. By setting up a custom world constructor, you can initialize specific data or services that will be available to all your step definitions throughout the test execution.

Each Scenario will receive a new instance of the world variable based on this class. If you need to write asynchronous code, you can write it inside an "init" function. Here is an example that should set up a sqlite database and initiate it with a "users" table:

1import { setWorldConstructor, QuickPickleWorld, QuickPickleWorldInterface } from 'quickpickle'
2import sqlite3 from 'sqlite3'
3import { Database, open } from 'sqlite'
4
5class CustomWorld extends QuickPickleWorld {
6  db?: Database;
7
8  constructor(context: TestContext, info?: QuickPickleWorldInterface['info']) {
9    super(context, info)
10  }
11
12  async init() {
13    await super.init()
14    this.db = await this.setupDatabase()
15  }
16
17  private async setupDatabase(): Promise<Database> {
18    const db = await open({
19      filename: ':memory:',
20      driver: sqlite3.Database
21    })
22
23    await db.exec(`CREATE TABLE IF NOT EXISTS users (id INTEGER PRIMARY KEY, name TEXT)`)
24
25    return db
26  }
27}
28
29setWorldConstructor(CustomWorld)

For a real world example of a custom world constructor, see PlaywrightWorld.ts.

Run Gherkin tests in your CI workflow

If Vite is properly configured, Gherkin tests should run the same on local environments as it does in your CI workflow. If you use browser testing tools like Playwright, you may need to take a step to set it up first.

See quickpickle's release.yml for an example.

Differences from CucumberJS

The main library for Gherkin in the JS ecosystem is CucumberJS, a test runner written explicitly for Gherkin tests. QuickPickle aims to be a complete replacement for that library, using Vite to handle bundling and test running while maintaining functional parity with the original. Nonetheless, there are differences. Here are the important ones that have come to notice:

world replaces this in step definitions

Each step definition in QuickPickle receives a "world" variable as its first parameter.

1// QuickPickle step definition
2Given('a number {int}', function(world:QuickPickleWorldInterface, int:number) {
3  if (!Array.isArray(world.numbers)) world.numbers = [int]
4  else world.numbers.push(int)
5})

In CucumberJS, you would write your step definitions using "this":

1// CucumberJS step definition
2Given('a number {int}', function(int:number) {
3  if (!Array.isArray(this.numbers)) this.numbers = [int]
4  else this.numbers.push(int)
5})

Aside from the fact that a passed variable is much easier to think about for a compiler than custom bindings, this led to some sub-optimal usage in modern Javascript, including:

• Arrow functions couldn't be used in step definitions or hooks, or this wouldn't work.
• When using a custom world, you would have to add (this:CustomWorldType, ...params) in typescript files or else you wouldn't get the right types.

Passing a variable is clearer and more intuitive, and provides more reliable support for modern JS.

Current step info in world variable

In CucumberJS, the default world variable contains information about the test suite, but not the current step. In QuickPickle, the "world" variable passed to each test step contains an "info" property with the data about the Scenario.

1export interface QuickPickleWorldInterface {
2  info: {
3    config: QuickPickleConfig   // the configuration for QuickPickle
4    feature: string             // the Feature name (not file name)
5    scenario: string            // the Scenario name
6    tags: string[]              // the tags for the Scenario, including tags for the Feature and/or Rule
7    steps: string[]             // an array of all Steps in the current Scenario
8    stepIdx?: number            // the index of the current Step, starting from 1 (not 0)
9    rule?: string               // the Rule name, if any
10    step?: string               // the current Step
11    line?: number               // the line number, in the file, of the current Step
12    explodedIdx?: number        // the index of the test case, if exploded, starting from 1 (not 0)
13    errors: any[]               // an array of errors that have occurred, if the Scenario is tagged for soft failure
14  }
15  context: TestContext,         // the Vitest context
16  isComplete: boolean           // (read only) whether the Scenario is on the last step
17  config: QuickPickleConfig                       // (read only) configuration for QuickPickle
18  worldConfig: QuickPickleConfig['worldConfig']   // (read only) configuration for the World
19  data: {[key:string]:any}      // Data limited to the current Scenario
20  common: Common                // Common data shared across ALL tests in one Feature file --- USE SPARINGLY
21  init: () => Promise<void>                       // function called by QuickPickle when the world is created
22  tagsMatch(tags: string[]): string[]|null        // function to check if the Scenario tags match the given tags
23  sanitizePath(path:string):string                // shim of npm module path-sanitizer
24  fullPath(relativePath:string):string            // function to return the full path, relative to project root
25}

Special Tags

In Gherkin, the meanings of tags are determined by the implementation, there are no defaults. Since quickpickle uses Vitest, some tags have been given default meanings:

• @todo / @wip: Marks scenarios as "todo" using Vitest's test.todo implementation
• @skip: Skips scenarios using Vitest's test.skip implementation
• @fails / @failing: Ensures that a scenario fails using Vitest's test.fails implementation
• @concurrent: Runs scenarios in parallel using Vitest's test.concurrent implementation
• @sequential: Runs scenarios sequentially using Vitest's test.sequential implementation

The relevant tags can be configured. Plugins may also have default tag implementations; for example, @quickpickle/playwright has @nojs to disable javascript, and @chromium, @firefox, and @webkit to run a scenario on a particular browser.

Unsupported CucumberJS Features

• return "skipped" in step definitions or hooks to skip a step definition or hook (use world.context.skip())
• setDefaultTimeout (use testTimeout in Vitest config)
• setDefinitionFunctionWrapper (use Before and After hooks)
• setParallelCanAssign (use @concurrent and @sequential special tags)
• attachments with world.attach, world.log, and world.link not supported, use Vitest reporters
• dry run not supported, but you can use vitest list which is similar
• [custom snippet formats] (only async/await javascript is supported)
• filtering CucumberJS tag matching with "and", "not", etc. In Vitest it is only possible to match the name of the test. With QuickPickle tests the name does include the tags, so running vitest -t "@some-tag" does work, but you can't specify that it should run all tests without a certain tag.

Acknowledgements

This project started out as a fork of vitest-cucumber-plugin by Sam Ziegler. It's been almost completely rewritten in the following ways:

• it has been converted to typescript
• a custom Gherkin parser has been replaced with the official Gherkin Parser
• the step definition format has been reverted to more closely match CucumberJS

Nonetheless, the brilliant ideas behind the original plugin are still present in the architecture of this project. Thanks Sam, your work blew my mind.

Media and Tutorials

• Behavioral testing with Gherkin and SvelteKit: The Svelte Summit presentation from 19 October 2024.

  • What is Gherkin / Cucumber and why should you use it
  • Comparing Gherkin to straight Playwright code
  • Testing the Svelte "Sverdle" app that ships with the demo site
  • Adding a new feature using Behavior Driven Development assisted by AI

• QuickPickle dev vlog 27 Oct. 2024: A near-real-time exploration of QuickPickle for beginners, highlighting Playwright functionality.

  • How to set up QuickPickle for testing websites with Playwright
  • Testing with and without Javascript, in multiple browsers, at multiple resolutions
  • Using QuickPickle's explodeTags to minimize test verbiage

• QuickPickle dev vlog 19 Nov. 2024: Migrating a complex CucumberJS implementation to QuickPickle

  tl;dr:

  • Set the proper configuration for vite / vitest
  • Change imports
  • Find and replace "this" to "world" in step definitions and hooks
  • Move your "setDefaultTimeout" to the vitest config "testTimeout"
  • Make any necessary changes in your custom world constructor, if applicable