Python (boto3)

Before you start

Python 3.10 or later, with pip install boto3
Your access key ID, API key and bucket name from console.shelby.xyz

Export your credentials and bucket name from the console so the commands below can use them:

export SHELBY_ACCESS_KEY_ID="<your access key ID, SHLB…>"
export SHELBY_SECRET_ACCESS_KEY="<your API key>"
export SHELBY_BUCKET="<your bucket name>"

Create a client

shelby_s3.py

import os

import boto3
from botocore.config import Config

s3 = boto3.client(
    "s3",
    endpoint_url="https://s3.beta.shelby.xyz",
    aws_access_key_id=os.environ["SHELBY_ACCESS_KEY_ID"],
    aws_secret_access_key=os.environ["SHELBY_SECRET_ACCESS_KEY"],
    region_name="default",
    config=Config(
        s3={"addressing_style": "path"},
        request_checksum_calculation="when_required",
    ),
)
bucket = os.environ["SHELBY_BUCKET"]

request_checksum_calculation stops boto3 from sending uploads in aws-chunked framing, which it does by default (botocore 1.36 and later) and Shelby doesn't accept yet. Without it, uploads fail with 501 NotImplemented.

Upload, read back and delete

import hashlib

body = b"Hello, Shelby\n"
s3.put_object(Bucket=bucket, Key="examples/hello.txt", Body=body)

data = s3.get_object(Bucket=bucket, Key="examples/hello.txt")["Body"].read()
assert hashlib.sha256(data).digest() == hashlib.sha256(body).digest()

# A new object can take a few seconds to appear in listings.
for obj in s3.list_objects_v2(Bucket=bucket, Prefix="examples/").get("Contents", []):
    print(obj["Key"], obj["Size"])

s3.delete_object(Bucket=bucket, Key="examples/hello.txt")

Upload large files

upload_file switches to multipart automatically. Shelby allows 3,000 parts per upload, so set a part size that fits your largest file:

from boto3.s3.transfer import TransferConfig

s3.upload_file(
    "dataset.parquet",
    bucket,
    "data/dataset.parquet",
    Config=TransferConfig(multipart_chunksize=64 * 1024 * 1024),  # 64 MiB parts
)

At the default 8 MiB part size, files over about 23 GiB need a larger part size.

What's different with boto3

copy_object and s3.copy fail with 501. Shelby has no server-side copy. Download and upload again.
Paginators need PageSize of 1,000 or less. Larger values are rejected.

Compatibility matrix · Troubleshooting