Transformers Chat

Give an AI node a small chat model that runs inside your browser, even on a computer without WebGPU.

What it does

Transformers Chat is a model you plug into the Model slot of an AI node, such as the Tools Agent or Basic LLM Chain. It does not run on its own and has no output. When the AI node runs, a small instruct model (for example SmolLM2) answers on your device. It uses WebGPU when available and falls back to the processor otherwise. No API key, no cloud call.

• Temperature is 0.5 and Max new tokens is 256 by default, so replies are short unless you raise it.
• The model works with about 2,000 tokens of context, so a connected Chat Memory sends only the latest part of the conversation.
• It loads on first use (downloading it if needed), then stays ready for the session.

Using it with tools

It works with the Tools Agent. These models have no built-in tool feature, so the tools are described in the prompt and the model writes its tool calls in its reply. The common formats are all understood: <tool_call>…</tool_call>, [TOOL_CALLS] […], <function=name>…</function>, a fenced JSON block, or a reply that is only the JSON call. The agent runs the tool, sends back the result, and the model answers or calls another tool. Hidden reasoning (<think>…</think>) is removed from the reply.

When to use it

• You want local chat but your device has no WebGPU, so Web LLM can’t run.
• Quick, low-stakes replies with the smallest possible model.
• You already use on-device models for other tasks and want the same engine.

For better answers on a device with WebGPU, use Web LLM.

Inputs and settings

设置项	类型	作用
Model	Model picker	An installed transformers.js chat model. Install one from Local AI, or download it from the picker.
Options	Group
Options › Temperature	Number	Higher is more creative; 0 is deterministic. 默认值：0.5
Options › Max new tokens	Number	Cap on the reply length. 默认值：256

The Model picker lists only chat models for this engine. Install one from the Local Models page, or download it from the picker. When you publish or share the workflow, Model is cleared, so whoever installs it picks their own.

Outputs

This node has no output port. It plugs into a slot on another node instead of passing items along.

• Model slot of Tools Agent, Q&A Agent, RAG Agent, Basic LLM Chain, Indexer, Text Classifier, Sentiment Analysis, Summarization, Information Extractor, Guardrails, Auto-fixing Output Parser

The AI node you connect it to produces the output. Transformers Chat only decides which model answers.

Dependencies and credentials

None. This node needs no connection and has no slots for other nodes.

Example workflow

1. A Context Menu trigger on selected text.
2. Basic LLM Chain with Prompt Give this text a short title: {{ $input.selectionText }}.
3. Transformers Chat in its Model slot, with an installed SmolLM2 instruct model and Max new tokens 40.
4. Clipboard Write with Text {{ $input.response }}.

Common issues

• “Model parameter is required and cannot be empty”. Pick a model in Model.
• “… isn’t a transformers.js chat model. Pick one from the picker, or use the Web LLM node.” The model belongs to another engine (often after importing a workflow). Pick one from this node’s list, or use Web LLM.
• Replies are cut short. Raise Max new tokens. This also fixes tool calls that get cut off.
• The agent never calls a tool. The model answered in plain text. Use an instruct model trained for tool use, give tools clear descriptions, and keep the system message short. A call to a tool that isn’t connected, or with broken JSON, stays as plain text.

Related nodes

• Web LLM: larger in-browser models, needs WebGPU.
• Generate Text: on-device text generation as a standalone node.
• Tools Agent: an agent that can use tools and memory.