Scorers
Scorers are used to score the output of your LLM call.
Autoevals is a great library of scorers to get you started.
You can create your own using createScorer
:
import { createScorer } from "evalite";
const containsParis = createScorer<string>({ name: "Contains Paris", description: "Checks if the output contains the word 'Paris'.", scorer: ({ output }) => { return output.includes("Paris") ? 1 : 0; },});
evalite("My Eval", { data: async () => { return [{ input: "Hello" }]; }, task: async (input) => { return input + " World!"; }, scorers: [containsParis],});
The name
and description
of the scorer will be displayed in the Evalite UI.
Score Properties
The score
function receives three properties on the object passed:
import { createScorer } from "evalite";
const containsParis = createScorer({ name: "Contains Paris", description: "Checks if the output contains the word 'Paris'.", scorer: ({ input, output, expected }) => { // input comes from `data` // expected also comes from `data` // output is the output of `task` return output.includes("Paris") ? 1 : 0; },});
Type Arguments
In TypeScript, createScorer
takes two type arguments, the input and output type:
import { createScorer } from "evalite";
const containsParis = createScorer< string, // Input type string // Output type>({ name: "Contains Paris", description: "Checks if the output contains the word 'Paris'.", scorer: (output) => { // output is typed as string! return output.includes("Paris") ? 1 : 0; },});
Scorer Metadata
You can provide metadata along with your custom scorer:
import { createScorer } from "evalite";
const containsParis = createScorer<string>({ name: "Contains Paris", description: "Checks if the output contains the word 'Paris'.", scorer: (output) => { return { score: output.includes("Paris") ? 1 : 0, metadata: { // Can be anything! }, }; },});
This will be visible along with the score in the Evalite UI.