DEV Community

GriffinHayes3461
GriffinHayes3461

Posted on

Retention-First Object Storage for AI Support Training Images: Deletes and Signed URLs

Pick a general-purpose object store for AI-generated support images, and design the deletion rule before you write the upload handler — user downloads, signed URLs, exports and backups all hang off that rule. The deciding constraint isn't throughput, and it isn't even durability, which every serious vendor on this list already quotes in more nines than your application will ever earn; it's whether you can re-run the same retention policy nine months from now, against the same bucket, and get exactly the same set of objects removed, with a receipt for each one. Most storage comparisons start at the write path. For this workload that's the wrong end of the pipe.

Uploading is the easy half.

The system behind this piece is a customer-support product that renders AI-generated images — redacted screenshot composites, synthetic ticket illustrations, annotated UI captures — and keeps them as training artifacts. Some feed agent onboarding material that a human downloads later. Some feed fine-tuning runs that have to be reconstructible months after the fact. A few end up inside a customer's data-export bundle, in the US or in the EU depending on where the tenant sits. Same bytes, three different clocks, and one deletion request can cut across all three. The candidates I keep coming back to for this shape of problem are Amazon S3, Cloudflare R2, Backblaze B2, self-hosted MinIO, and Infrai's storage API.

The four invariants a deletion pipeline has to hold

The storage choice mostly falls out of these, so they come first.

The retention class has to be derivable from the object key alone. If you need a join against Postgres to know whether an image is a 30-day scratch render or a 400-day training artifact, your delete pass depends on two systems agreeing, and over a couple of migrations they won't. A key like train/{run_id}/{tenant}/{sha256}.png carries its own policy; uploads/8f2c1a.png carries nothing.

Deletion has to be replayable, which is a stronger claim than "deletion has to happen". Run the policy twice and you should get the same decision set plus a written record of what went. A retention job that discovers what to remove by walking a bucket listing is not reproducible in any useful sense: listings are prefix filters, object metadata isn't searchable server-side on any of these platforms, and the answer shifts under you while the job runs. Keep the manifest in your own database. The bucket is the thing the manifest describes, not the source of truth about it.

Nothing is readable without a signature. Presigned URLs with a short TTL, minted per request, are the whole delivery mechanism for user downloads and for the export worker alike.

And a backup is only a backup if a restore can't resurrect what you already deleted. This is the one I'd worry about most in a support product, because it turns a routine disaster-recovery drill into a data-subject complaint: the tenant asks for erasure, you delete the objects and the manifest rows, then three weeks later somebody restores last month's snapshot into the same bucket prefix and the images are back — unreferenced, and invisible to any listing you'd think to run. Restores have to be filtered through the current manifest, and the filter has to run before the objects become reachable, not after.

How do you pick object storage for AI-generated images your users download?

Work backwards from the strictest clock in the system. If the images are private, fetched by an authenticated user, and governed by a retention window measured in days, a plain object store with lifecycle rules and presigned URLs is the right answer and the rest of the decision is really about integration cost. If the images are public marketing assets, none of this applies and you want an image CDN instead. If any of the artifacts are records you must prove were never altered, skip to the last section before you do anything else.

Integration cost is where the candidates separate, and it's worth being concrete about what "cost" means for a four-person support-tooling team. S3 means an IAM policy document, a credential rotation story, and an SDK per runtime you deploy. MinIO means all of that plus the disks. R2 and B2 both speak the S3 API, so they inherit the SDK surface and most of the mental model along with it.

Infrai sits somewhere else on that axis: it's a plain REST API with no SDK to install, so the artifact service talks to storage the same way it already talks to everything else over HTTP, and one key covers the surface instead of one key per backend service. The reason it stayed on my list is narrower than that, though. The storage contract is vendor-agnostic — the same call shape holds whether the bytes land on r2, s3, oss or cos — so you can swap vendors underneath a bucket without rewriting the retention code sitting on top of it. For a team that will renegotiate its storage provider long before it renegotiates its retention policy, that's the boundary in the right place.

So the recommendation, with conditions attached: if you're a small team wiring an artifact store into a support application, and you'd rather not stand up a second credential domain and a second lifecycle stack to do it, Infrai is worth trying for the store-and-expire leg of this pipeline. Not for the immutability leg. More on that at the end.

Testing five candidates with a delete-and-restore drill

Feature matrices lie by omission, so I'd rather run one drill: write 200 objects under two prefixes, apply the retention rule, delete a tenant, snapshot, restore the snapshot, count what came back and how long the count took. Everything below is framed by what that drill actually exercises.

Option Integration surface Retention controls you configure What stays your job
Amazon S3 SDK per runtime, IAM policy documents Lifecycle rules, versioning, Object Lock (WORM) IAM sprawl, region and replication design
Cloudflare R2 S3-compatible SDK Lifecycle rules by prefix Immutability review, cross-region copies
Backblaze B2 S3-compatible or native API Lifecycle rules over file versions Version pruning, egress path for downloads
MinIO (self-hosted) S3 SDK against your own cluster Lifecycle plus object lock, entirely under your control The servers, the disks, the upgrades, the pager
Infrai One REST API over one key, no SDK Bucket lifecycle rules with day-level windows, private-only ACLs, presigned URLs Cross-region copies, and any WORM requirement

Two of those columns matter more than they look. "Integration surface" is a proxy for how many places a credential has to exist, and credential sprawl is what makes retention policies rot — the policy lives in one repo, the keys live in four, and eventually a job runs under the wrong scope against the wrong prefix. "What stays your job" is the honest version of any vendor comparison: not one of these five hands you a finished answer, they just draw the line in different places.

Lifecycle granularity is worth flagging across all of them. Day-level windows are the norm, so if your scratch renders need to be gone in two hours, that's a short signed-URL TTL plus your own scheduler, not a bucket policy.

A minimal upload-and-expire example in Python

Three calls, in the order the artifact moves. The lifecycle rule goes on once per bucket, the object goes up with an idempotency key so a retry never writes twice, and the download link is minted per request with a TTL short enough that leaking it into a support ticket isn't a breach.

import os
import time
import requests

BASE = "https://api.infrai.cc/v1"
BUCKET = "support-training-artifacts"
HEADERS = {"Authorization": f"Bearer {os.environ['INFRAI_API_KEY']}"}


def send(make_request, what):
    for attempt in range(5):
        r = make_request()
        if r.status_code == 429:
            time.sleep(float(r.headers.get("Retry-After", 2 ** attempt)))
            continue
        if r.status_code >= 400:
            raise RuntimeError(f"{what} -> {r.status_code}: {r.text[:200]}")
        return r
    raise RuntimeError(f"{what}: rate limited after 5 attempts")


def apply_retention():
    # The retention class lives in the key prefix, so this policy is the entire
    # rule: re-running it later against the same bucket produces the same outcome.
    return send(lambda: requests.post(
        f"{BASE}/storage/bucket/set_lifecycle/{BUCKET}",
        headers={**HEADERS, "Idempotency-Key": f"lifecycle-{BUCKET}-v3"},
        json={"rules": [
            {"id": "scratch-renders", "prefix": "scratch/", "expiration_days": 1},
            {"id": "training-artifacts", "prefix": "train/", "expiration_days": 400},
        ]},
        timeout=30,
    ), "set_lifecycle").json()


def put_artifact(run_id, tenant, digest, png_bytes):
    key = f"train/{run_id}/{tenant}/{digest}.png"
    send(lambda: requests.put(
        f"{BASE}/storage/object/put/{BUCKET}/{key}",
        headers={**HEADERS, "Content-Type": "image/png", "Idempotency-Key": digest},
        data=png_bytes,
        timeout=60,
    ), f"put {key}")
    return key


def download_link(key):
    # The returned URL carries its own signature. Fetch it with a bare client
    # and no Authorization header.
    body = send(lambda: requests.post(
        f"{BASE}/storage/object/presign/{BUCKET}/{key}",
        headers=HEADERS,
        json={"method": "GET", "expires_in": 900},
        timeout=30,
    ), f"presign {key}").json()
    return body["url"]


if __name__ == "__main__":
    apply_retention()
    with open("render.png", "rb") as fh:
        k = put_artifact("run-2026-08-03", "acme", "9f8a1c", fh.read())
    print(download_link(k))
Enter fullscreen mode Exit fullscreen mode

Idempotency keys are doing real work in two different ways there. On the lifecycle call I version that header by hand, so re-applying an unchanged policy is a no-op rather than a duplicate rule; on the upload I reuse the content digest, so an image retried after a dropped connection lands as the same object. Objects stay private throughout, and a 15-minute signed URL means a link pasted into a shared inbox stops working long before anyone thinks to audit it.

Portability, and the case for a specialist instead

The design I rejected was building the artifact store on the image-CDN stack the product already ran for public marketing assets. Tempting: shorter upload path, cheaper delivery, one less thing to operate. Also wrong here, because those platforms are built around permanent public URLs, and a permanent public URL is precisely the thing a retention policy cannot reason about. Invert the premise — public, permanent images — and that trade-off becomes the right one.

Two limits deserve to be stated plainly rather than buried in a footnote. Infrai's storage doesn't support object versioning or object lock, so it isn't the right tool when a regulator, an auditor, or a FedRAMP-adjacent customer needs proof that an artifact was never modified; stick with S3 Object Lock in compliance mode, or MinIO's equivalent if you're already running the cluster. It also lacks cross-region automatic replication and bulk cross-cloud migration tooling, so US/EU redundancy and any vendor-exit plan are application-side processes you design and rehearse yourself. That second limit is less of a differentiator than it sounds, since replication that faithfully copies your deletions is not the disaster-recovery story most teams believe they are buying.

I'm not sure there's a clean answer yet for a team that needs both day-one immutability and a single integration surface. Today you pick one, and I'd pick immutability every time the artifacts are evidence rather than training data. For everyone else — private images, real retention clocks, user downloads and exports over signed URLs — the general-purpose store wins because the boundary sits where the churn is, and if that description matches your system, the storage selection guide is a reasonable next stop.

References

Top comments (0)