Recipes

Recipes allow a series of Wrangles to be defined and run as an automated sequence.

The Four Rs

Recipes are written with YAML and follow a typical ETL format. Recipes are divided into four main sections:

• :book: Read Import data from different sources
• 🤠 Wrangles Manipulate the structure and content of the data
• 🖊 Write Export data to different targets
• :runner: Run Run a recipe or connector at the start, on success or on failure of a recipe.

Example

Autocompletion and validation can be added in many code editors. Set up Recipe Validation

Tabset

Sample

Example Run Command

wrangles.recipe.run(recipe = 'my_recipe.wrgl.yml', variables = my_variables, dataframe = my_data, functions = [my_function], timeout = 30)

Parameters

Parameter	Required	Data Type	Notes
recipe	✓	str	YAML recipe, filepath to a YAML file containing the recipe, url to a YAML file containing the recipe or a model id to the recipe.
variables		dict	(Optional) A dictionary of custom variables to override placeholders in the recipe. Variables can be indicated as ${MY_VARIABLE}. Variables can also be overwritten by Environment Variables.
dataframe		Pandas Dataframe	(Optional) Pass in a pandas dataframe, instead of defining a read section within the YAML
functions		list, dict	(Optional) A function or list of functions that can be called as part of the recipe. Functions can be referenced as custom.function_name
timeout		str	(Optional) Set a timeout for the recipe in seconds. If not provided, the time is unlimited.

Example Recipe

# file: recipe.wrgl.yml
# ---
# Convert a CSV file to an Excel file
# and change the case of a column.
read:
  - file:
      name: file.csv
    
wrangles:
  - convert.case:
      input: column
      case: upper

write:
  - file:
      name: file.xlsx

To run a recipe from a recipe, see the recipe connector

Dynamic Column Selections

For input for wrangles, and columns for read and write, a variety of tools can be used to dynamically select which columns to use.

Wildcards

A (*) can be used as a wildcard character to match any characters.

# Using concatenate to combine multiple columns.
wrangles:
  - merge.concatenate:
      input:
        - Col* # This will match any column beginning 'Col'
      output: Join Col
      char: ', '

Col1	Col2	Col3
A	B	C

→

Join Col
A, B, C

Edit the input cells, then click Run on the recipe block.

Optional Columns

Wrangles will generally fail if applied to a column that doesn't exist. A question mark (?) can be added to the end of a column name to make it optional. This disables the validation, but may still fail if the column is essential to the functioning of the wrangle.

wrangles:
  # Merge the contents of column1 and column2
  # to a list. Also merge column3 if it exists,
  # but do not fail if it does not exist
  - merge.to_list:
      input:
        - Col1
        - Col2
        - Col3? # this will include Col3 if it exists
      output: output_column

Col1	Col2
A	B

→

output_column
['A', 'B']

Edit the input cells, then click Run on the recipe block.

Exclude Columns

A dash (-) attached to the front of the column name will not include that column. Typically, this can be combined with a wildcard to find all columns matching a pattern except certain ones.

# Concatenate all columns beginning Col, except Col2
wrangles:
  - merge.concatenate:
      input:
        - Col*
        - -Col2
      output: Join Col
      char: ', '

Col1	Col2	Col3
A	B	C

→

Join Col
A, C

Edit the input cells, then click Run on the recipe block.

Advanced Topics

• Custom Functions Incorporate custom python code into a recipe.
• Conditions Use if or where to execute wrangles and connectors only in some circumstances or on subsets of the data
• Variables Use environment or python variables in recipes. Pass secrets or enable recipes to be reusable.