workflow-builder

Graphical interface to build a workflow file

The haddock3-download application of the main branch is published at https://i-vresse-workflow-builder.netlify.app

The builder allows you to create a complex TOML formatted config file based on a set of JSON schemas.

The workflow builder is organized as a monorepo with packages and apps.

• haddock3-download app: To construct haddock3 workflow and download it for offline running
• haddock3-submit app: To construct haddock3 workflow and submit it for online running
• haddock3-galaxy app: To construct haddock3 workflow and submit inside a galaxy project instance.
• kitchensink app: Demonstration of features of workflow builder
• @i-vresse/wb-core package: React components, state management, input/output functions to create an application
• @i-vresse/wb-form package: Web Form based on JSON Schema
• haddock3_catalog package: Generate script and storage place for haddock3 catalogs

Develop

Requires NodeJS and yarn (tested with v3.2.1).

1# Install dependencies
2yarn
3
4# Run dev servers
5yarn dev

• Goto http://localhost:3000 for haddock3-download app
• Goto http://localhost:3001 for haddock3-submit app
• Goto http://localhost:3002 for kitchensink app

Unit tests

Tests (**/*.test.tsx?) written in vitetest can be run with:

1yarn test -- run

To run tests with code coverage use

1yarn  test -- run --coverage

Creates **/coverage/ directory with HTML and LCOV report.

Integration tests

The integration tests (**/integration-tests/**.spec.ts) are written in playwright.

Before running test ensure browsers are installed with

1cd apps/haddock3-download
2npx playwright install chromium

Tests can be run with

1yarn test:integration

To run a non-headless chromium browser use

1yarn test:headed

The browser will pause when a test calls await page.pause(), so you can investigate current state.

There is a VS code extension to run integration tests inside editor.

Linting

1yarn lint

To autofix lint errors use

1yarn lint -- --fix

To generate JSON report use

1yarn lint -- --report json > eslint.report.json

Build

To build production distribution run

1yarn build

Which will create apps/*/dist/ directories which should be hosted on the web somewhere. The build also creates packages/*/dist directories which should be published to npmjs.com.

Component development

Components can be developed/tested/documented using storybook.

The storybooks of the main branch are hosted at

• storybook for core package
• storybook for form package

Storybook can be started locally with

1yarn storybook

• Goto http://localhost:6008 for storybook of core package
• Goto http://localhost:6007 for storybook of form package

Format

Workflow archive

The workflow builder creates a zip file with a workflow configuration file called workflow.cfg in TOML format. The configuration file contains paths to input files which are included in the zip file.

Workflow configuration file

The workflow configuration file consists out of 2 parts:

1. Global parameters, which are available to engine and each node.
2. Tables with parameters for each node the workflow should run.

An uploaded workflow configuration file can contain tables with the same name (this is more lenient then the TOML format). A generated workflow configuration file with the same node twice will have a TOML string with [somenode] and ['somenode.2'] table respectively.

Catalog

The catalog is a YAML formatted file which tells the app what nodes are available. In has the following info:

1. global: Description of global parameters
   • schema: What parameters are valid. Formatted as JSON schema draft 7.
   • uiSchema: How the form for filling the parameters should be rendered.
   • tomlSchema: How toml keys are mapped to in-memory representation.
2. nodes: Description of available nodes.
   • id: Identifier of node, for computers
   • label: Label of node, for humans
   • category: Category to which node belongs
   • description: Text describing what node needs, does and produces.
   • schema: What parameters are valid. Formatted as JSON schema draft 7.
   • uiSchema: How the form for filling the parameters should be rendered.
   • tomlSchema: How toml keys are mapped to in-memory representation.
3. catagories: Descriptions of node categories
   • name: Name of category
   • description: Description of category
   • collapsed (optional): Whether category should be rendered collapsed initially
4. examples: Title and link to example workflows
   • map with title as key and link as value
5. title: Title of the catalog
6. nodeLegend: Legend of the node in the app. Default is 'Node'

schema

See docs/schema.md.

uiSchema

See docs/uiSchema.md.

tomlSchema

See docs/tomlSchema.md.

Catalog index

In the worklfow builder you can pick a catalog from a list. This list gets downloaded from public/catalog/index.json and is formatted like

1[
2    ["<title of catalog>", "<URL of catalog YAML file>"]
3]

The first catalog in the index.json file will be shown when you open the app.

The haddock3 catalogs can be generated by a Python script in packages/haddock3_catalog from the haddock3 library. The haddock3 catalogs and example are symbolicly linked to /app/*/public.