Utility Wrangles

Utility wrangles, with recipe examples, parameters, and behavior.

Accordion

Apply a series of wrangles to column(s) containing lists. The wrangles will be applied to each element in the list and the results will be returned back as a list.

Apply a series of wrangles to the individual elements of one or more lists.

Parameters

Name	Description	Accepted Values	Default	Required
I/O
input	The column(s) containing the list(s) that the wrangles will be applied to the elements of.	string, integer, array	—	Yes
output	Output of the wrangles to save back to the dataframe.	string, array, null	null	No
propagate	Limit the column(s) that will be available to the wrangles and replicated for each element. If not specified, all columns will be propogated. This may be useful to limit the memory use for large datasets.	string, array, null	null	No
Conditions
if	Condition that determines whether the wrangle runs as a whole. Recipe variables may be referenced with ${variable}.	string	—	No
where	Filter rows before applying the wrangle using SQL-like criteria, such as column1 = 123 OR column2 = 'abc'.	string	—	No
where_params	Values used with where for parameterized criteria. Uses SQLite placeholder syntax such as ? or :name.	array, object	—	No
Execution
wrangles	List of wrangles to apply.	array	—	Yes

Examples

This example applies convert.case to each string in a list, where the wrangle would not normally operate on the list as a whole.

wrangles:
  - accordion:
      input: list_column
      output: modified_lists
      wrangles:
        - convert.case:
            input: list_column
            output: modified_lists
            case: upper

list_column
["a", "b", "c"]
["e", "f", "g"]

modified_lists
["A", "B", "C"]
["E", "F", "G"]

Access

Requirement	Value
AI-powered	No
Requires WrangleWorks account	No
Requires subscription	No
Requires external API key	No

Technical details

Field	Value
Catalog ID	69
Catalog key	accordion
Recipe key	accordion
Catalog status	active
Lifecycle status	active
Recipe Writer eligible	No
Recipe Writer exclusion	Nested-list orchestration is not supported in the baseline.
Namespace	Root-level
Documentation group	utility
Aliases	None
Runtime symbol	wrangles.recipe_wrangles.main.accordion
Legacy UUID	None

Sources

• WranglesPY accordion implementation
• Archived accordion Markdown

---

Batch

Split the data into batches for executing a list of wrangles. Use this in situations such as where the intermediate data is too large to fit in memory.

Execute a series of wrangles in batches. Batches can run in parallel with threads and can provide fallback output when an error occurs.

Parameters

Name	Description	Accepted Values	Default	Required
Conditions
if	Condition that determines whether the wrangle runs as a whole. Recipe variables may be referenced with ${variable}.	string	—	No
where	Filter rows before applying the wrangle using SQL-like criteria, such as column1 = 123 OR column2 = 'abc'.	string	—	No
where_params	Values used with where for parameterized criteria. Uses SQLite placeholder syntax such as ? or :name.	array, object	—	No
Execution
wrangles	The wrangles to execute on the data. Each series of wrangles will be run against the data in batches of the size defined by batch_size.	array	—	Yes
batch_size	The number of rows to split each batch into.	integer	1000	No
threads	The number of threads to use for parallel processing. Default 1.	integer	1	No
timeout	The number of seconds to wait for a batch to complete before raising an error.	number, null	null	No
use_multiprocessing	Use process-based workers instead of threads. This is an advanced runtime option.	boolean	false	No
Errors
on_error	A dictionary of column_name: value to return if an error occurs while attempting to run a batch.	object, null	null	No

Examples

This example processes product descriptions in batches of two rows.

wrangles:
  - batch:
      batch_size: 2
      threads: 1
      wrangles:
        - extract.ai:
            api_key: Your OpenAI API key
            input: Product Description
            output:
              Title:
                type: string
                description: Title of the product

Product Description
Sleep better with our Memory Foam Pillow, designed to contour to your head and neck.
Stay comfortable and stylish with our Organic Cotton T-Shirt, made from soft, breathable fabric.
Keep drinks hot or cold with our Stainless Steel Water Bottle, featuring durable insulation.
Enjoy crisp sound and long battery life with our Wireless Bluetooth Earbuds.

Title
Memory Foam Pillow
Organic Cotton T-Shirt
Stainless Steel Water Bottle
Wireless Bluetooth Earbuds

Access

Requirement	Value
AI-powered	No
Requires WrangleWorks account	No
Requires subscription	No
Requires external API key	No

Technical details

Field	Value
Catalog ID	70
Catalog key	batch
Recipe key	batch
Catalog status	active
Lifecycle status	active
Recipe Writer eligible	Yes
Namespace	Root-level
Documentation group	utility
Aliases	None
Runtime symbol	wrangles.recipe_wrangles.main.batch
Legacy UUID	None

Sources

• WranglesPY batch implementation
• Archived batch Markdown

---

Concurrent

Run multiple wrangles concurrently rather than sequentially. Wrangles must specify output columns to be used concurrently. When using concurrent, Wrangles may not complete in a predictable order and it is not recommended to update overlapping columns with different wrangles.

Run multiple wrangles concurrently instead of sequentially. Concurrent wrangles must declare output columns, may finish in any order, and should not update overlapping columns.

See the Concurrent connector for the connector equivalent.

Parameters

Name	Description	Accepted Values	Default	Required
Conditions
if	Condition that determines whether the wrangle runs as a whole. Recipe variables may be referenced with ${variable}.	string	—	No
where	Filter rows before applying the wrangle using SQL-like criteria, such as column1 = 123 OR column2 = 'abc'.	string	—	No
where_params	Values used with where for parameterized criteria. Uses SQLite placeholder syntax such as ? or :name.	array, object	—	No
Execution
wrangles	The wrangles section of a recipe to execute for each combination of variables.	array	—	Yes
max_concurrency	The maximum number of wrangles to execute in parallel.	integer	10	No
use_multiprocessing	Use process-based workers instead of threads. This is an advanced runtime option.	boolean	false	No

Examples

wrangles:
  - concurrent:
      wrangles:
        - extract.codes:
            input: Products
            output: Part Codes
        - extract.attributes:
            input: Products
            output: Attributes

Products
SKF ball brg 2" od 6202
brg seal 1" id 5493
3lb hammer 87102

Part Codes	Attributes
6202	{"length":["2in"]}
5493	{"length":["1in"]}
87102	{"weight":["3lb"]}

Access

Requirement	Value
AI-powered	No
Requires WrangleWorks account	No
Requires subscription	No
Requires external API key	No

Technical details

Field	Value
Catalog ID	73
Catalog key	concurrent
Recipe key	concurrent
Catalog status	active
Lifecycle status	active
Recipe Writer eligible	No
Recipe Writer exclusion	Concurrent orchestration is not supported in the baseline.
Namespace	Root-level
Documentation group	utility
Aliases	None
Runtime symbol	wrangles.recipe_wrangles.main.concurrent
Legacy UUID	None

Sources

• WranglesPY concurrent implementation
• Archived concurrent Markdown

---

Log

Log the current status of the dataframe.

Print the current status of the dataframe. Only a sample of rows is logged.

Parameters

Name	Description	Accepted Values	Default	Required
I/O
columns	(Optional, default all columns) List of specific columns to log.	array, null	null	No
Options
write	(Optional) Allows for an intermediate output to a file/dataframe/database etc.	array, null	null	No
warning	Log a warning to the console.	string, null	null	No
info	Log info to the console.	string, null	null	No
log_data	Whether to log a sample of the contents of the dataframe. Default True if not logging to a write, error, warning or info. Default False otherwise.	boolean, null	null	No
error	Log an error to the console.	string	—	No
Conditions
if	Condition that determines whether the wrangle runs as a whole. Recipe variables may be referenced with ${variable}.	string	—	No
where	Filter rows before applying the wrangle using SQL-like criteria, such as column1 = 123 OR column2 = 'abc'.	string	—	No
where_params	Values used with where for parameterized criteria. Uses SQLite placeholder syntax such as ? or :name.	array, object	—	No

Examples

wrangles:
  - log: {}

Logging side-effect; no tabular input sample.

Logs a sample of rows or status information to the configured destination.

wrangles:
  - log:
      columns:
        - column1
        - column2

Logging side-effect; no tabular input sample.

Logs a sample of rows or status information to the configured destination.

wrangles:
  - log:
      write:
        - file:
            name: output/filepath
            columns:
              - column 1
              - column 2

Logging side-effect; no tabular input sample.

Logs a sample of rows or status information to the configured destination.

Variable	Function
${column_count}	Number of columns.
${columns}	List of all columns.
${df}	Current dataframe.
${row_count}	Number of visible rows processed in each batch.

Access

Requirement	Value
AI-powered	No
Requires WrangleWorks account	No
Requires subscription	No
Requires external API key	No

Technical details

Field	Value
Catalog ID	80
Catalog key	log
Recipe key	log
Catalog status	active
Lifecycle status	active
Recipe Writer eligible	No
Recipe Writer exclusion	Operational logging steps are not authored in the baseline.
Namespace	Root-level
Documentation group	utility
Aliases	None
Runtime symbol	wrangles.recipe_wrangles.main.log
Legacy UUID	6177808e-aa2d-4d0b-8385-858b16948a5d

Sources

• WranglesPY log implementation
• Archived log Markdown

---

Matrix

Apply a matrix of wrangles to the dataframe. This will run the wrangles for each combination of the variables.

Apply a matrix of wrangles to the dataframe. Each wrangle runs for the configured combinations of variables, including recipe variables and variables declared by the variables parameter.

See the Matrix connector for the connector equivalent.

Parameters

Name	Description	Accepted Values	Default	Required
Options
strategy	Determines how to combine variables when there are multiple. loop (default) iterates over each set of variables, repeating shorter lists until the longest is completed. permutations uses the combination of all variables against all other variables.	string; one of: permutations loop	"loop"	No
Conditions
if	Condition that determines whether the wrangle runs as a whole. Recipe variables may be referenced with ${variable}.	string	—	No
where	Filter rows before applying the wrangle using SQL-like criteria, such as column1 = 123 OR column2 = 'abc'.	string	—	No
where_params	Values used with where for parameterized criteria. Uses SQLite placeholder syntax such as ? or :name.	array, object	—	No
Execution
variables	A dictionary of variables to pass to the wrangle. The key is the variable name and the value is a list of values.	object	—	Yes
wrangles	The wrangles to apply to the dataframe. Each wrangle will be run for each combination of the variables.	array	—	Yes

Examples

Run a custom function once for each configured variable value.

wrangles:
  - matrix:
      variables:
        var: [A, B, C]
      wrangles:
        - custom.test_fn:
            input: Part Code
            output: Part Code ${var}
            value: ${var}

def test_fn(part_code, value):
    return part_code + value

Part Code
6202
br549
554-114
554-112

Part Code A	Part Code B	Part Code C
6202A	6202B	6202C
br549A	br549B	br549C
554-114A	554-114B	554-114C
554-112A	554-112B	554-112C

This example runs extract.custom once for each unique model ID.

wrangles:
  - matrix:
      variables:
        model_id: set(Model ID)
      wrangles:
        - extract.custom:
            input: Description
            output: Extracted Values
            model_id: ${model_id}
            where: "[Model ID] = ?"
            where_params:
              - ${model_id}

Description	Model ID
The SKF 6202 bearing is the best bearing in the world	xxxxxxxx-xxxx-xxxx
The Timken 6102 bearing is indestructible	yyyyyyyy-yyyy-yyyy
The Milwaukee impact has 1200ft-lbs of torque	zzzzzzzz-zzzz-zzzz

Extracted Values
6202
indestructible
1200ft-lbs

Variable	Function
${column_count}	Number of columns.
${columns}	List of all columns.
${df}	Current dataframe.
${row_count}	Number of visible rows processed in each batch.

Access

Requirement	Value
AI-powered	No
Requires WrangleWorks account	No
Requires subscription	No
Requires external API key	No

Technical details

Field	Value
Catalog ID	84
Catalog key	matrix
Recipe key	matrix
Catalog status	active
Lifecycle status	active
Recipe Writer eligible	No
Recipe Writer exclusion	Matrix orchestration is not supported in the baseline.
Namespace	Root-level
Documentation group	utility
Aliases	None
Runtime symbol	wrangles.recipe_wrangles.main.matrix
Legacy UUID	None

Sources

• WranglesPY matrix implementation
• Archived matrix Markdown

---

Recipe

Run another recipe as a wrangle against the current dataframe.

Behavior

Run another recipe as a wrangle against the current dataframe.

This guidance was derived from the callable signature and its embedded Python schema docstring.

Parameters

Name	Description	Accepted Values	Default	Required
I/O
input	Name, index, or list of input columns.	string, integer, array, null	null	No
output	Name or list of output columns.	string, array, null	null	No
Options
name	File name of the recipe.	string, null	null	No
Conditions
if	Condition that determines whether the wrangle runs as a whole. Recipe variables may be referenced with ${variable}.	string	—	No
where	Filter rows before applying the wrangle using SQL-like criteria, such as column1 = 123 OR column2 = 'abc'.	string	—	No
where_params	Values used with where for parameterized criteria. Uses SQLite placeholder syntax such as ? or :name.	array, object	—	No
Execution
variables	A dictionary of variables to pass to the recipe.	object, null	null	No

Access

Requirement	Value
AI-powered	No
Requires WrangleWorks account	No
Requires subscription	No
Requires external API key	No

Technical details

Field	Value
Catalog ID	86
Catalog key	recipe
Recipe key	recipe
Catalog status	active
Lifecycle status	active
Recipe Writer eligible	No
Recipe Writer exclusion	Nested recipe composition is outside the baseline.
Namespace	Root-level
Documentation group	utility
Aliases	None
Runtime symbol	wrangles.recipe_wrangles.main.recipe
Legacy UUID	None

Sources

• WranglesPY recipe implementation

---

Try

Try a list of wrangles and catch any errors that occur.

Run a list of wrangles and catch errors. When except is provided, its wrangles or fallback column values run after an error; otherwise, the error is logged and the recipe continues.

Parameters

Name	Description	Accepted Values	Default	Required
Conditions
if	Condition that determines whether the wrangle runs as a whole. Recipe variables may be referenced with ${variable}.	string	—	No
where	Filter rows before applying the wrangle using SQL-like criteria, such as column1 = 123 OR column2 = 'abc'.	string	—	No
where_params	Values used with where for parameterized criteria. Uses SQLite placeholder syntax such as ? or :name.	array, object	—	No
Execution
wrangles	List of wrangles to apply.	array	—	Yes
Errors
retries	Number of times to retry the wrangles if an error occurs. Default 0.	integer	0	No
except	An action to take if the wrangles encounter an error. This can contain a list of wrangles or a dictionary of column names and values. If except is not provided, the error will be logged and the recipe will continue.	object	—	No

Examples

wrangles:
  - try:
      wrangles:
        - risky_wrangle:
            input: column
      except:
        - backup_wrangle:
            input: column

The input dataframe is passed to the primary wrangles.

The primary result is returned on success; the except result is returned after an error.

Access

Requirement	Value
AI-powered	No
Requires WrangleWorks account	No
Requires subscription	No
Requires external API key	No

Technical details

Field	Value
Catalog ID	98
Catalog key	try
Recipe key	try
Catalog status	active
Lifecycle status	active
Recipe Writer eligible	No
Recipe Writer exclusion	Error-control orchestration is not supported in the baseline.
Namespace	Root-level
Documentation group	utility
Aliases	None
Runtime symbol	wrangles.recipe_wrangles.main.Try
Legacy UUID	None

Sources

• WranglesPY try implementation
• Archived try Markdown