Added v0.5
The S3 connector supports writing files to & from a bucket as well as uploading/downloading.
Tabset
Read
Recipe
read:
- s3:
bucket: bucket
key: key
access_key: access_key
secret_access_key: secret_access_key
Function
from wrangles.connectors import s3
df = s3.read(
bucket='bucket',
key='key',
access_key='access_key',
secret_access_key='secret_access_key'
)
Parameters
The S3 connector also supports the same optional parameters as the file connector. See the file connector for more details.
| Parameter | Required | Data Type | Notes |
|---|---|---|---|
| bucket | ✓ | str | The name of the bucket to download object from |
| key | ✓ | str | The name of the key to download from |
| access_code | ✓ | str | S3 access key |
| secret_access_code | ✓ | str | S3 secret access key |
| order_by | str | Uses SQL syntax to sort the input. | |
| if | str | A condition that will determine whether the action runs or not as a whole. |
Write
Recipe
write:
- s3:
bucket: bucket
key: key
access_key: access_key
secret_access_key: secret_access_key
Function
from wrangles.connectors import s3
df = Imagine a Dataframe
s3.write(
df,
bucket='bucket',
key='key',
access_key='access_key',
secret_access_key='secret_access_key'
)
Parameters
The S3 connector also supports the same optional parameters as the file connector. See the file connector for more details.
| Parameter | Required | Data Type | Notes |
|---|---|---|---|
| df | ✓ | DataFrame | Function only. DataFrame of contents to write to the S3 bucket |
| bucket | ✓ | str | The name of the bucket where file will be written |
| key | ✓ | str | The name of the key to download from |
| access_key | ✓ | str | S3 access key |
| secret_access_key | ✓ | str | S3 secret access key |
| order_by | str | Uses SQL syntax to sort the output. | |
| if | str | A condition that will determine whether the action runs or not as a whole. |
Upload
Recipe
run:
on_success:
- s3.upload_files:
bucket: s3-bucket
file: example.csv
Function
from wrangles.connectors import s3
s3.upload_files.run(
bucket='bucket',
file = 'example.csv',
key='key'
)
Parameters
| Parameter | Required | Data Type | Notes |
|---|---|---|---|
| bucket | ✓ | str | The name of the S3 bucket where file will be written |
| file | ✓ | str, list | File or list of files to upload |
| key | str, list | S3 file key or list of keys to upload as, defaults to None | |
| endpoint_url | str | Override the S3 host for alternative S3 storage providers | |
| aws_access_key_id | str | Set the access key. Can also be set as an environment variable | |
| aws_secret_access_key | str | Set the access secret. Can also be set as an environment variable | |
| if | str | A condition that will determine whether the action runs or not as a whole. |
Download
Recipe
run:
on_start:
- s3.download_files:
bucket: s3-bucket
key: example.csv
Function
from wrangles.connectors import s3
df = s3.download_files.run(
bucket='bucket',
key='key'
file = 'example.csv',
)
Parameters
| Parameter | Required | Data Type | Notes |
|---|---|---|---|
| bucket | ✓ | str | The name of the S3 bucket where file will be written |
| key | ✓ | str, list | S3 file key or list of keys to download |
| file | str, list | Local filename or list of filenames to save the downloaded files as, defaults to None | |
| endpoint_url | str | Override the S3 host for alternative S3 storage providers | |
| aws_access_key_id | str | Set the access key. Can also be set as an environment variable | |
| aws_secret_access_key | str | Set the access secret. Can also be set as an environment variable | |
| if | str | A condition that will determine whether the action runs or not as a whole. |