Evaluation

Record what the workflow produced for a test case and how good it was. In a normal run this step does nothing and passes items on.

What it does

During a Test run (started from an Evaluation Trigger), the Operation decides what the step does:

• Set metrics: score this case. Add one card per metric, each with a Name and a Metric:
  • Exact match: Actual equals Expected (optionally Case-sensitive).
  • Contains: Actual contains Expected.
  • Similar text: how close two texts are, from 0 to 1; it passes at the Pass at (0–1) threshold.
  • Category: the predicted label equals the expected one.
  • Tools used: the agent called the Expected tools (read from Agent steps, for example {{ $("Agent").steps }}).
  • Correctness (AI judge, 1–5): a Judge model reads the answer and What a good answer does, and gives a score from 1 to 5. The answer is fenced, so instructions inside it are not followed.
  • Custom number: any number you compute.
• Set outputs: keep named values the workflow produced (the answer, the category), shown in each case of the test run.
• Is it an evaluation?: routes items to Evaluating during a test run and to Normal otherwise. Put emails, posts and other side effects on Normal so test runs don’t send anything.

Outside a test run, Set metrics and Set outputs pass the items through unchanged.

When to use it

• At the end of an AI workflow, to score its answer against the expected one.
• Before a step with side effects, to skip it during test runs.

Inputs and settings

Setting	Type	What it does
Operation	Choice	Options: Set metrics, Set outputs, Is it an evaluation?. Default: Set metrics
Outputs	List of entries
Outputs (each entry) › Name REQ(required)	Text
Outputs (each entry) › Value	Value
Metrics	List of entries
Metrics (each entry) › Name REQ(required)	Text
Metrics (each entry) › Metric	Choice	Options: Exact match, Contains, Similar text, Category, Tools used, Correctness (AI judge, 1–5), Custom number. Default: Exact match
Metrics (each entry) › Expected	Value
Metrics (each entry) › Actual	Value
Metrics (each entry) › Expected tools	Tags
Metrics (each entry) › Agent steps	Value
Metrics (each entry) › What a good answer does	Long text
Metrics (each entry) › Pass at (0–1)	Number
Metrics (each entry) › Case-sensitive	Toggle

The AI judge needs a chat model connected to the step’s Judge model slot. The other metrics need no model.

Outputs

• Output

More ports appear as you add them in the settings (one per rule).

Set metrics and Set outputs pass their input on. Is it an evaluation? has two outputs: Evaluating and Normal.

Dependencies and credentials

Slot	Accepts	Connections
Judge model	chat model	One node.

Example workflow

1. Evaluation Trigger with question and expected columns.
2. Basic LLM Chain answers the question.
3. Evaluation, Is it an evaluation?: Normal goes to Gmail, Evaluating goes on to step 4.
4. Evaluation, Set metrics: Contains with Actual {{ $input.text }} and Expected {{ $('Evaluation Trigger').expected }}.

Common issues

• Every case fails: check that Actual reads the field the AI step really returns. Open one case of the test run, then Open run.
• The judge scores vary: use a stronger judge model, and describe a good answer precisely.

Related nodes

• Evaluation Trigger: the test rows.
• Guardrails: check inputs and answers in normal runs.