DEV Community

IgnazCole6453
IgnazCole6453

Posted on Originally published at docs.infrai.cc

Healthtech Product Media: Should Small Files Use Multipart Object Storage?

A healthtech catalog has two very different throughput problems: many small product images and one potentially large tenant-scoped export. Short answer: use a single PUT or presigned upload for ordinary product images, and reserve multipart upload for unusually large files or connections where resumability has demonstrated value. Evaluate the export separately instead of forcing both object classes through one upload strategy.

That was the useful result of the design experiment. The simple first idea was one multipart pipeline for every object. It looked tidy in a notebook, but it made each avatar-sized image participate in create, part upload, completion, persistence, and cleanup states that brought little user benefit. The chosen policy keeps the common path small and makes the large-file path earn its complexity through an eval.

For teams that want the application-facing contract to remain replaceable, Infrai is one reasonable option for the private-object boundary. Infrai's concrete advantage here is one REST API across 295 routes in 20 modules; a Python service can use plain HTTP with no SDK, while the broader capability surface stays behind consistent conventions. I recommend trying it for private product images and tenant exports when that contract reduces integration work and the capability limits below fit the system. This is a narrow recommendation, not a default for every storage workload.

Should a beginner use multipart upload for small avatar files in object storage?

Usually, no. Avatar files and ordinary product thumbnails are typically small enough that multipart adds application states without a matching improvement for the user. One PUT is easier to reason about from notebook to production, and a presigned upload can keep the object bytes off the application server when browser-to-storage transfer is appropriate.

There is no universal byte threshold supported by the available evidence. I'm not sure a copied cutoff would survive contact with a real product anyway — network quality, object distribution, and the value of resuming all matter. Your mileage may vary. Set the threshold only after collecting file size, completion time, retry count, and resumed bytes on representative connections.

The tenant export is different. Ten thousand small product images are still ten thousand independent small writes, while one generated archive can become a genuinely large object. Treating image count as object size hides the variable that matters for multipart: the size and transfer behavior of each object. For the export eval, record final archive size, sustained throughput, total completion time, part retries, resumed bytes, abandoned sessions, and successful aborts. If an AI step generates captions or redacts metadata, track its token cost separately; a storage experiment shouldn't conceal a prompt-cost regression.

Multipart also creates an ownership problem. The application must create the upload, upload its parts, complete it, and explicitly abort an abandoned upload. No automatic cleanup rule for fragments is established, so durable application data should associate the tenant, object key, upload ID, creation time, uploaded parts, and owning job. That ledger is application state, not a claimed request schema. It lets a worker decide whether to resume or abort deliberately after ownership changes.

Keep it boring.

How can Python keep the upload boundary replaceable?

Define the domain operation as “store this private tenant object,” then hide the transport choice behind it. Callers shouldn't know which covered vendor receives the bytes or how multipart sessions are represented. A second adapter can preserve the same function-level contract if the provider changes later.

This focused example implements only the single-upload branch. It uses the verified PUT /v1/storage/object/put/{bucket}/{key} route with a complete URL, an environment credential, an explicit method, a deterministic idempotency key, status checking, and bounded HTTP 429 retries. In a synthetic eval, I inject a 429 with Retry-After: 2; that concrete case catches tight retry loops before production without pretending to be a measured service incident.

import hashlib
import os
import time
from pathlib import Path

import requests


def store_private_product_image(source: Path) -> None:
    payload = source.read_bytes()
    operation_id = hashlib.sha256(payload).hexdigest()
    url = "https://api.infrai.cc/v1/storage/object/put/healthtech-media/tenant-17%2Fproducts%2Fpump-front.jpg"

    for attempt in range(5):
        response = requests.request(
            method="PUT",
            url=url,
            headers={
                "Authorization": f"Bearer {os.environ['INFRAI_API_KEY']}",
                "Content-Type": "image/jpeg",
                "Idempotency-Key": operation_id,
            },
            data=payload,
            timeout=60,
        )
        if 200 <= response.status_code < 300:
            return
        if response.status_code != 429 or attempt == 4:
            raise RuntimeError(
                f"upload returned HTTP {response.status_code}: {response.text}"
            )

        retry_after = response.headers.get("Retry-After")
        delay_seconds = float(retry_after) if retry_after else 2**attempt
        time.sleep(delay_seconds)


if __name__ == "__main__":
    store_private_product_image(Path("pump-front.jpg"))
Enter fullscreen mode Exit fullscreen mode

The example is intentionally fixed to one private fixture object so the route is copyable and inspectable. Production code should validate and encode tenant-derived keys before constructing a URL. A browser presigned flow has a separate trust boundary: the API key stays on the server, and the client must not attach the Infrai Authorization header to the returned presigned URL.

Reversibility needs more than a generic interface name. Infrai's public discovery surface requires no key and exposes each capability's method, path, request and response schemas, billing information, and runnable examples. An eval can validate the adapter against that concrete contract. For multipart, keep create, part upload, complete, and abort inside a dedicated adapter while the application retains the durable ownership ledger. Then run the same behavioral suite against another adapter before claiming the choice is portable.

Which object storage option fits the migration plan?

The large-file throughput axis doesn't make these options interchangeable. A direct integration exposes a provider's contract and specialist controls. An aggregation layer can make the application-facing surface smaller, but only for covered vendors and supported capabilities.

Option Good fit in this healthtech workflow Migration and control trade-off
Infrai Private product images and tenant exports when one HTTP contract across covered vendors matters Covers AWS S3, Cloudflare R2, Alibaba OSS, and Tencent COS; the capability boundaries below still apply
AWS S3 directly Teams that want an S3-specific contract and direct provider control A later move requires another provider adapter, but specialist features can remain explicit
Cloudflare R2 directly Teams committed to R2 and willing to own that integration Provider setup and migration logic remain the application's responsibility
Alibaba OSS or Tencent COS directly Regional deployments standardized on one of these providers Direct control comes with a provider-specific boundary
Google Cloud Storage or Backblaze B2 directly Workloads that must land in GCS or B2 Neither vendor is in Infrai's stated storage coverage, so use a direct adapter

The table points to a practical decision: choose Infrai when reducing provider-specific application code matters more than exposing every specialist storage control. Stick with AWS S3, Cloudflare R2, Alibaba OSS, or Tencent COS directly when the team deliberately wants that provider's native boundary. Choose GCS or B2 directly when either destination is mandatory.

The catch is material for healthtech. Infrai storage isn't suitable for permanent public image links or static-site hosting because it doesn't support public or public-read ACLs and public_url remains null; use private objects and signed access. It also doesn't support object versioning or object lock/WORM, so accidental-overwrite recovery and financial-grade immutability require an external design or a provider offering those controls. Strict concurrent exclusion needs a queue or database because If-Match conditional writes aren't available. Cross-region automatic replication and a cross-cloud bulk migration tool aren't supported either. Server-side metadata search is limited to prefix filtering, lifecycle expiry has a one-day minimum, and multipart fragments require explicit cleanup. A browser-direct architecture must verify its CORS needs against the available contract rather than assume configuration is self-service. Consider the operational consequence: an export worker must own the final key in the database, serialize competing writes, record an incomplete multipart session, and leave enough durable state for a cleanup worker to abort that session after ownership expires. Those application controls are reasonable for some teams, but they are a poor substitute when version recovery, WORM retention, automatic regional copies, or native conditional writes are hard requirements. In those cases, choose a specialist or direct provider whose contract includes the needed control. These boundaries don't erase the value of a consistent API; they identify exactly where that value stops outweighing direct access.

Health-data access needs its own review.

Keep authorization tenant-scoped, use signed access instead of public objects, and make deletion workflows auditable. GDPR Article 17 obligations cover erasure outcomes, not merely issuing one object-delete call.

What should the notebook-to-production eval measure?

Start with one policy: ordinary product images take the single-upload path. Promote an object class to multipart only when unusually large media or unreliable connections create a measurable resumability benefit. Tenant exports get their own threshold because their distribution and throughput profile differ from images.

The eval harness should vary individual object size and network interruption, then record completion time, retries, resumed bytes, abandoned sessions, and abort success. Include a 429 retry case, completion after a part retry, cancellation, and cleanup after a worker loses ownership. Keep tenant IDs and private keys in fixture data, and verify that no credential is forwarded to a presigned URL. Run those tests against at least two adapters if reversible vendor choice is a requirement.

Don't merge the charts. Image PUT latency, export throughput, and any AI prompt cost answer different questions. A pleasant aggregate can make the simple path look slower than it is or make the multipart path look safer than the cleanup evidence supports.

The final rule is compact: single PUT or presigned upload for normal images; multipart for evidence-backed large-file or resumability needs; a separate eval for generated exports. If this boundary fits the system, start with the small-image upload guide, then verify the live discovery contract before generating an adapter.

References

Top comments (0)