Shape · Matching and dedupe
Flag Duplicates
Adds a true/false column marking rows that repeat on the key columns, and keeps every row for review. A fuzzy mode catches near-duplicates such as Jon and John.
Also known as: Find duplicates
How it works
Key columns define what makes two rows duplicates; empty means the whole row. What to mark decides which occurrence stays unflagged: Mark all but first (default), Mark all but last, or Mark all. The flag is appended as is_duplicate unless you set Output column, and the rows keep their input order. Fuzzy match compares the key columns by string similarity instead, with an Algorithm (Jaro-Winkler or Levenshtein) and a Threshold from 0 to 1, 0.85 by default; the similarity is averaged over the key columns. Fuzzy mode compares every row with every other, so it is slow on large tables, and it needs at least one key column.
Inputs and outputs
Inputs
data— Table
Outputs
data— Table
Settings
The Attributes panel of Flag Duplicates, as in the app: the controls work, but nothing runs — there is no data behind it.
The same settings, as the Assistant writes them
{
keyColumns: string[]
keep: "first"|"last"|"none"
outputColumn: string
fuzzy?: boolean
fuzzyAlgorithm?: "jaro_winkler"|"levenshtein"
fuzzyThreshold?: number
}empty keyColumns = all columns (exact only); fuzzy flags near-duplicates by string similarity over the key columns (O(n²))
Example
Spot repeated contacts before a mailing: key columns first_name and last_name, fuzzy match on with Jaro-Winkler at 0.9.
| first_name | last_name | is_duplicate |
|---|---|---|
| Jon | Kowalski | false |
| John | Kowalski | true |
| Anna | Nowak | false |