Tokenize
Split text into tokens. A variety of methods are available. The default method is to split on spaces.
Tokenize elements in a list or string into individual tokens.
Parameters
| Name | Description | Accepted Values | Default | Required |
|---|---|---|---|---|
| I/O | ||||
input | Column(s) to be split into tokens. | string, integer, array | — | Yes |
output | Name of the output column. | string, array, null | null | No |
| Options | ||||
method | Method to split the list. Options include space, boundary, boundary_ignore_space, custom functions as custom.<function>, or regex patterns as regex:<pattern>. | string; one of:
| "space" | No |
| Conditions | ||||
if | Condition that determines whether the wrangle runs as a whole. Recipe variables may be referenced with ${variable}. | string | — | No |
where | Filter rows before applying the wrangle using SQL-like criteria, such as column1 = 123 OR column2 = 'abc'. | string | — | No |
where_params | Values used with where for parameterized criteria. Uses SQLite placeholder syntax such as ? or :name. | array, object | — | No |
Examples
wrangles:
- split.tokenize:
input: Materials
output: Tokenized List
| Materials |
|---|
| Stainless Steel Oak Wood |
| Tokenized List |
|---|
| ['Stainless', 'Steel', 'Oak', 'Wood'] |
wrangles:
- split.tokenize:
input: Materials
output: Tokenized List
| Materials |
|---|
| ['Stainless Steel', 'Oak Wood'] |
| Tokenized List |
|---|
| ['Stainless', 'Steel', 'Oak', 'Wood'] |
Access
| Requirement | Value |
|---|---|
| AI-powered | No |
| Requires WrangleWorks account | No |
| Requires subscription | No |
| Requires external API key | No |
Technical details
| Field | Value |
|---|---|
| Catalog ID | 66 |
| Catalog key | split.tokenize |
| Recipe key | split.tokenize |
| Catalog status | active |
| Lifecycle status | active |
| Recipe Writer eligible | Yes |
| Namespace | split |
| Documentation group | split |
| Aliases | None |
| Runtime symbol | wrangles.recipe_wrangles.split.tokenize |
| Legacy UUID | 6cc88418-ae0c-43f6-84ee-31e0d5f838c3 |
Sources