Migrating from S3
What to change when you move code or pipelines from Amazon S3, Cloudflare R2 or MinIO to Shelby.
Most code that works with S3 works with Shelby after a configuration change. This page lists that change, then the S3 features worth checking your code for.
1. Change the client configuration
| Setting | Value |
|---|---|
| Endpoint | https://s3.beta.shelby.xyz |
| Access key ID | Your access key ID (starts with SHLB) |
| Secret access key | Your Shelby API key |
| Bucket | Your bucket name, for example shelby-managed-account-1 |
| Region | default |
| Addressing | Path-style |
Find your access key ID, API key and bucket name at console.shelby.xyz. Your bucket already exists.
For a boto3 client, the change looks like this:
s3 = boto3.client(
"s3",
endpoint_url="https://s3.beta.shelby.xyz",
aws_access_key_id=SHELBY_ACCESS_KEY_ID,
aws_secret_access_key=SHELBY_SECRET_ACCESS_KEY,
region_name="default",
config=Config(
s3={"addressing_style": "path"},
request_checksum_calculation="when_required",
),
)Each quickstart shows the same change for its client.
2. Check for features Shelby handles differently
Search your code for these calls and settings. Each one needs a change, or a decision.
| If your code uses | On Shelby | What to do |
|---|---|---|
CopyObject, aws s3 mv, s3.copy | Not supported | Download and upload again |
x-amz-meta-* user metadata over 2 KB | Rejected with MetadataTooLarge, as on S3 | Keep larger metadata in your application or database |
| Presigned URLs | Not supported | Proxy downloads and uploads through your server |
| Uploads from a browser | Not supported (no CORS) | Upload from your server |
If-Match for optimistic concurrency | Not enforced yet | Coordinate writers in your application. Create-only writes with If-None-Match: * do work. |
Conditional GET for caching (If-None-Match, If-Modified-Since) | Ignored | Each read downloads the full object |
| Lifecycle rules, object expiry | Not available | Delete objects yourself when you no longer need them |
| Versioning | Not available | Write each version to its own key, for example model/v12/weights.bin |
| Tags, storage classes | Ignored | Encode anything you need in the key |
x-amz-checksum-* | Accepted, not verified | Use Content-MD5, which Shelby verifies, or compare SHA-256 after download |
A paginator with PageSize over 1,000 | Rejected | Use 1,000 or less |
| Several buckets | One per account | Use key prefixes, or a separate account per bucket |
| Listing right after writing | A new object can take a few seconds to appear | Retry the listing, or track the keys you wrote |
DeleteObjects from the AWS SDK for JavaScript | Not recognised yet; nothing is deleted | Delete keys one at a time with DeleteObjectCommand |
3. Copy your existing data
rclone can copy straight from S3, R2 or MinIO into Shelby. Configure a remote for the source as usual, then:
rclone copy source:my-bucket "shelby:$SHELBY_BUCKET" --checksum --progressrclone streams each object through your machine, because Shelby can't copy server-side. Run it close to the source (for example, in the same cloud region) to keep transfer costs down.
4. Verify
After copying, compare the two sides:
rclone check source:my-bucket "shelby:$SHELBY_BUCKET" --one-wayrclone check compares sizes, and MD5 hashes wherever both sides have them. Large files
that rclone uploads in parts (over 200 MiB by default) have no whole-file MD5, so for those
it compares size only. For a byte-for-byte check, add --download.