Data
Reshape, split, compare, and validate values without leaving the process.
Work on the values a run already holds. Split a large CSV into chunks and reassemble it, shrink columns, compare or union two lists, and validate a string against a rule set.
The document components select values out of a JSON, XML, or assignment document by locator and patch them back without reformatting the rest of it, so a later patch step edits exactly the spans it was pointed at and leaves surrounding whitespace, comments, and ordering intact.
Components
Each component links to its own page with its full options and outputs.
| Name | Description |
|---|---|
data:csv:chunk | Splits a large CSV file into smaller chunk files of a fixed row count |
data:csv:merge | Merges columns from a secondary CSV file into a primary CSV file |
data:csv:shrink | Removes specified columns from a CSV file to reduce its width |
data:csv:unchunk | Combines multiple CSV chunk files back into a single output file |
data:document:extract | Walks a parsed document by segment pattern and emits keyed items. It reads the flat item list a document parse produces, keeps the items whose path matches the pattern, names the captured segments, and emits one output item per match. Repeated matches collapse into one item when group fields are configured, which is how plurals and arrays come back out of a document as a single entry. |
data:document:patch | Applies syntax-aware replacements without rebuilding the document. |
data:document:render | Writes a keyed list of items out as a document in one of the supported formats. Selection and patching are the pair for editing a document that already exists, while this one builds a document from data that never was one, which is what a workflow needs once it has assembled the entries itself. Every key and value is escaped for the format being written, so a value holding a separator, a quote, a backslash, or a line break survives a round trip. |
data:document:select | Reads syntax values without changing the source document. |
data:json:unchunk | Combines multiple JSON chunk files into a single array or map output file |
data:list:compare | Sorts two lists of items into buckets by comparing them key by key. Items pair up through the configured key paths, so the two lists may name the same thing differently. A pair whose compared values all agree is the same. A pair whose values disagree is classified against the base, which is git's three-way merge question and the only reason a base is accepted at all. The side that still equals the base did not move, so a pair lands in a_moved, b_moved or both_moved and a caller reads the merge policy straight off the bucket name instead of re-deriving it a step later. A pair with no base for its key is different, since nothing there says which side moved. Every bucket is emitted in key order and nothing is written anywhere. |
data:list:union | Coalesces two lists into one item per key, preferring the first list. Items pair up through the configured key paths, A's item is taken wherever a key appears in both, and B fills the keys A has nothing for. That is the link a precedence chain is built from, so a policy such as approved over published over base is a union of a union rather than a query that has to know the policy. Neither list may repeat a key, and nothing is written anywhere. |
data:shape:read | Reports the structure of a document a run is holding. |
data:validate | Checks that a content string is valid for a given format (csv, json, yaml) |