Remove Duplicates

Drop repeated items from a list, and optionally skip items you already handled on an earlier run.

What it does

Remove Duplicates looks at the whole incoming list and keeps the first item of each kind, dropping the later copies. The items that remain stay in their original order, so you can Sort first to choose which copy wins.

You decide what “the same” means with Compare. The order of keys inside an object does not matter: { "a": 1, "b": 2 } and { "b": 2, "a": 1 } are duplicates.

With Also Compare Against Previous Runs on, the node also remembers what it let through before and drops those items on later runs. That turns any repeated scrape into a “new items only” feed.

When to use it

• Clean a scraped list that shows the same row in several places on the page.
• Treat two records as the same even when a changing field, such as a timestamp, differs.
• Get notified only about new listings, posts or jobs, by turning on Also Compare Against Previous Runs.
• To see what changed between two lists (new, gone, edited), use Compare Datasets instead.

Inputs and settings

Einstellung	Typ	Was es bewirkt
Compare	Choice	Which part of each item decides whether two items are the same. Options: All fields, Selected fields, All fields except. Standard: All fields
Fields	Tags	Field names to compare on, or to ignore. Dots address nested fields.
Also Compare Against Previous Runs	Toggle	Skip items this node has already emitted on an earlier run. Standard: Off
Remember	Choice	How far back the history reaches. Options: Last N items, Time window. Standard: Last N items
History Size	Number	How many items to remember, or how many days to look back. Standard: 1000

• Compare: All fields compares the whole item. Selected fields compares only the names in Fields, and dots reach nested fields (seller.id). All fields except compares everything but the names in Fields; it can only leave out top-level fields, so seller.id would not work there.
• Remember and History Size only matter when Also Compare Against Previous Runs is on. With Last N items, History Size is a number of items (1000 by default, at most 2000). With Time window, it is a number of days (at most 365). Larger values are lowered to the maximum instead of failing.
• Each Remove Duplicates node keeps its own history, inside the workflow’s stored data.

Outputs

• Output

The same list with the copies removed, in the original order. Items are not changed. If every item was a duplicate, the output is an empty list.

Dependencies and credentials

None. This node needs no connection and has no slots for other nodes.

Example workflow

1. A Schedule trigger runs every hour.
2. Get HTML from Link and HTML Extract read the listings on a page, each with a url and a title.
3. Remove Duplicates with Compare Selected fields, Fields url, and Also Compare Against Previous Runs on.
4. A Browser Notification shows {{ $input.title }} for each listing that comes through.

The first run lets every listing through. After that, you only hear about listings you have not seen before.

Common issues

• “Compare mode “…” needs at least one field. Add a field, or switch to “All fields”.“ You chose Selected fields or All fields except without filling Fields.
• “Cannot compare these items: one of them contains a circular reference or a value that cannot be serialized.” An item holds something that cannot be written as plain data. Use Selected fields and compare on simple fields such as id or url.
• Nothing is removed. The items differ in a field you did not expect, such as a timestamp or a tracking parameter in a link. Compare only the fields that identify an item, or leave out the changing ones with All fields except.
• Items you already saw come back on every run. Remembering across runs only works when the node runs as part of a saved workflow; otherwise the run log warns “nothing was remembered for next time”. Save the workflow and run it again.
• You want old items to come through again. Open the Stored data tab in the canvas side panel and delete this node’s history entry (it starts with remove-duplicates-history).

Related nodes

• Sort: choose which copy is kept.
• Compare Datasets: sort two lists into new, gone and changed items.
• Data Store: keep your own values between runs.
• Limit: keep only the first or last few items.