
What Is Evalite?
- Fully open source: No API Key required.
- Local-first: runs on your machine, your data never leaves your laptop.
- Based on Vitest, the best TypeScript test runner around.
- Terminal UI for quick prototyping.
- Supports tracing and custom scorers.
How Do I Learn More?
Evalite Is Experimental
Evalite is still an experimental project. I'm actively working on it, and for now am pushing breaking changes.
If you run into any unexpected behavior:
- Delete the
node_modules/.evalite
folder.
- Update
evalite
to the latest version.
- Rerun your evals.
If, after that, you run into unexpected behavior, report an issue.
Guides
Watch Mode
You can run Evalite in watch mode by running evalite watch
:
evalite watch
This will watch for changes to your .eval.ts
files and re-run the evals when they change.
[!IMPORTANT]
I strongly recommend implementing a caching layer in your LLM calls when using watch mode. This will keep your evals running fast and avoid burning through your API credits.
Running Specific Files
You can run specific files by passing them as arguments:
evalite my-eval.eval.ts
This also works for watch
mode:
evalite watch my-eval.eval.ts