Batch
Split the data into batches for executing a list of wrangles. Use this in situations such as where the intermediate data is too large to fit in memory.
Execute a series of wrangles in batches. Batches can run in parallel with threads and can provide fallback output when an error occurs.
Parameters
| Name | Description | Accepted Values | Default | Required |
|---|---|---|---|---|
| Conditions | ||||
if | Condition that determines whether the wrangle runs as a whole. Recipe variables may be referenced with ${variable}. | string | — | No |
where | Filter rows before applying the wrangle using SQL-like criteria, such as column1 = 123 OR column2 = 'abc'. | string | — | No |
where_params | Values used with where for parameterized criteria. Uses SQLite placeholder syntax such as ? or :name. | array, object | — | No |
| Execution | ||||
wrangles | The wrangles to execute on the data. Each series of wrangles will be run against the data in batches of the size defined by batch_size. | array | — | Yes |
batch_size | The number of rows to split each batch into. | integer | 1000 | No |
threads | The number of threads to use for parallel processing. Default 1. | integer | 1 | No |
timeout | The number of seconds to wait for a batch to complete before raising an error. | number, null | null | No |
use_multiprocessing | Use process-based workers instead of threads. This is an advanced runtime option. | boolean | false | No |
| Errors | ||||
on_error | A dictionary of column_name: value to return if an error occurs while attempting to run a batch. | object, null | null | No |
Examples
This example processes product descriptions in batches of two rows.
wrangles:
- batch:
batch_size: 2
threads: 1
wrangles:
- extract.ai:
api_key: Your OpenAI API key
input: Product Description
output:
Title:
type: string
description: Title of the product
| Product Description |
|---|
| Sleep better with our Memory Foam Pillow, designed to contour to your head and neck. |
| Stay comfortable and stylish with our Organic Cotton T-Shirt, made from soft, breathable fabric. |
| Keep drinks hot or cold with our Stainless Steel Water Bottle, featuring durable insulation. |
| Enjoy crisp sound and long battery life with our Wireless Bluetooth Earbuds. |
| Title |
|---|
| Memory Foam Pillow |
| Organic Cotton T-Shirt |
| Stainless Steel Water Bottle |
| Wireless Bluetooth Earbuds |
Access
| Requirement | Value |
|---|---|
| AI-powered | No |
| Requires WrangleWorks account | No |
| Requires subscription | No |
| Requires external API key | No |
Technical details
| Field | Value |
|---|---|
| Catalog ID | 70 |
| Catalog key | batch |
| Recipe key | batch |
| Catalog status | active |
| Lifecycle status | active |
| Recipe Writer eligible | Yes |
| Namespace | Root-level |
| Documentation group | utility |
| Aliases | None |
| Runtime symbol | wrangles.recipe_wrangles.main.batch |
| Legacy UUID | None |
Sources