Shape · Data quality and flow control

Compare Tables

Lines up two versions of a table by key columns and puts each value side by side, flagging every cell that changed. A row on only one side shows blanks for the other.

Also known as: Diff, reconcile

How it works

Wire the baseline into left and the new version into right; both must have the same set of columns or the node stops with the differing names. Key columns identify a row on both sides. Value columns are the ones compared, all non-key columns by default. For each key the output has the key, then per value column c_left, c_right and a boolean c_changed; a row present on one side only has NULLs on the other. Output Differences only keeps rows with a change or present on one side; Full keeps every row. Treat two nulls as equal is on by default; off, a cell that is NULL on both sides counts as changed.

Inputs and outputs

Settings

{
  keyColumns: string[]
  valueColumns?: string[]
  outputMode: "diff_only"|"full"
  nullEqual: boolean
}

Example

Yesterday's price list against today's: key columns sku, value columns price, output Differences only.

More in Data quality and flow control

• Assert — Checks the data against rules, such as no blanks, unique keys, allowed values or freshness, and stops the run if one fails. Data that passes flows through unchanged.
• Expectations — Runs a set of quality rules and outputs a scorecard with one row per rule, its failed count and pass rate. Unlike Assert, it never stops the run.
• Gate — Passes data on only when a condition holds. The condition can test the data itself, a separate control table or a parameter. Otherwise everything after it is skipped and the run still succeeds. For example, publish only if the extract has rows.
• Wait For — Holds a branch back until other steps have finished, then passes its data on. For example, send a Slack message only once the database write is done.