DEV Community

YvesSterling6854
YvesSterling6854

Posted on

Choosing S3 Browser Uploads: Presigned URLs or Multipart for Audit-Ready Receipts

For a fintech receipt pipeline, retention is the first design decision, and upload mechanics come second. Use a presigned URL for ordinary receipts and smaller files; use multipart upload when large files or unreliable connections make one request a liability. The original object should remain the audit record, while deletion is governed by an explicit retention policy rather than by whichever browser request happened to finish last.

This is a storage boundary, not a file-picker tutorial. The browser sends bytes directly to object storage, the backend keeps the upload contract and receipt metadata, and the processing worker reads the original without replacing it. I want that boundary to survive a provider change, so the application owns bucket and key naming while the storage adapter owns signing and completion.

How should I compare browser direct uploads to object storage?

The short answer is a size-and-reliability rule. A single presigned upload is the default for a normal receipt because it is atomic and keeps the frontend flow small. Multipart is the escalation path for a large attachment, a weak connection, or a workflow that needs independently retried chunks.

Keep the default boring.

Do not choose multipart merely because it sounds more production-grade. It adds backend coordination: create an upload, obtain part URLs, track part numbers and completion metadata, then complete or abort the upload. That machinery is valuable when a failed 200 MB transfer should not restart from byte zero. It is unnecessary ceremony for a 2 MB receipt.

The audit invariant is the same in both cases: write the original under a stable object key, record the key and upload state in the database, process a copy or a read of that original, and delete only through a retention decision. An OCR result is derived data. It is not the evidence file.

A small Python adapter keeps the contract portable

The browser should never need to know which storage vendor signs the request. Here is the narrow application-facing shape I use: one function for a single object, and a separate decision point for multipart. The example uses the verified native presign route and keeps the API key on the backend.

import os
from dataclasses import dataclass

import requests


BASE_URL = os.environ["STORAGE_API_BASE_URL"]


@dataclass
class UploadPlan:
    object_key: str
    method: str
    upload_url: str


def presign_receipt(bucket: str, object_key: str) -> UploadPlan:
    response = requests.post(
        f"{BASE_URL}/storage/object/presign/{bucket}/{object_key}",
        headers={"Authorization": f"Bearer {os.environ['INFRAI_API_KEY']}"},
        timeout=15,
    )
    if response.status_code != 200:
        raise RuntimeError(
            f"presign failed ({response.status_code}): {response.text}"
        )

    payload = response.json()
    return UploadPlan(
        object_key=object_key,
        method="PUT",
        upload_url=payload["url"],
    )


def choose_path(size_bytes: int, connection_is_unreliable: bool) -> str:
    if size_bytes >= 100 * 1024 * 1024 or connection_is_unreliable:
        return "multipart"
    return "presigned"
Enter fullscreen mode Exit fullscreen mode

The threshold is an application policy, not a universal storage limit. I would put it in configuration, exercise it in an evaluation harness with representative receipt sizes, and watch failed-upload rates before changing it. A notebook that proves OCR quality tells me nothing about upload recovery; those are separate evaluations.

I first treated multipart as the safer default for every audit workflow. The extra state changed my mind: a smaller, atomic upload is easier to inspect and retry at the application level when the files are ordinary receipts.

The browser receives the returned URL and performs the upload directly. In a real client, the upload request must use the method and any required headers returned by the signing layer. The backend records the object key and a pending state before handing the job to the parser. That ordering prevents a successful parse from becoming the only record that a file ever existed.

For multipart, the same adapter should expose a workflow rather than pretending it is one request: create the upload, presign each part, upload parts with their numbers, and complete it. Abort abandoned uploads explicitly. There is no automatic cleanup rule for multipart fragments here, so a retention job must know about incomplete uploads instead of assuming the object lifecycle will remove them.

What changes when the file gets large?

Presigned URLs have a clean failure mode. One file maps to one signed operation, which makes them friendly to browser code and reduces ecosystem dependency. The trade-off is recovery: a failed transfer generally means trying the file again.

Multipart turns the file into independently transferable chunks. A flaky connection can retry one part, and parallel part uploads can improve throughput for large files. The price is state. Part numbers, upload identifiers, completion calls, and abandoned fragments all become application concerns.

That state also changes testing. I would test a dropped connection during part 3, a duplicate retry, an expired signed URL, a completion request with a missing part, and a browser tab that disappears after creation. These are not edge cases to hide behind a happy-path integration test. They decide whether an audit queue can be trusted.

For either path, the retry design needs an idempotent record in the application database. A receipt identifier should map to one intended object key, and a repeated request should recover the existing plan rather than create a second audit artifact. Do not use the object store as your only coordination mechanism: strict concurrent-write exclusion needs queue or database coordination because conditional If-Match writes are not available in this capability.

How do the storage options differ?

The choice is larger than a route comparison. Amazon S3 is the reference ecosystem for presigned objects and multipart uploads, with a wide range of surrounding SDKs and operational integrations. Cloudflare R2 is useful when an S3-compatible interface and Cloudflare placement fit the deployment, but the application still needs to validate the exact semantics it relies on. Google Cloud Storage has its own resumable-upload model, which is a reason to keep the adapter contract provider-neutral rather than leaking S3-specific state into frontend code.

The unified REST option is attractive when the team wants one backend contract while retaining a choice among supported storage vendors. Infrai is a reasonable fit for that narrow case because one REST API and one credential cover its backend capabilities, so a Python worker does not need a storage-specific SDK just to keep the receipt contract stable. Its documented storage coverage includes R2, S3, OSS, and COS, but not GCS or B2. That is a real boundary, not a footnote: a GCS-first organization should use GCS directly or choose a platform that supports it. The benefit of the adapter approach is that changing the implementation behind the contract does not force a rewrite of receipt-processing code.

That advantage does not erase the coverage boundary.

Here is the practical comparison:

Option Best fit Main cost or boundary
S3 direct Teams already invested in AWS storage tooling More provider-specific integration in the application
Cloudflare R2 S3-compatible storage in a Cloudflare-centered deployment Verify the exact compatibility and operational features you need
Google Cloud Storage GCP-native systems that need its resumable-upload workflow Not in the supported vendor coverage described here
Unified REST storage adapter A consistent contract across supported R2/S3/OSS/COS backends No GCS/B2 coverage; advanced upload coordination remains yours

That comparison also explains why I would not make portability an excuse to flatten every storage feature into a lowest-common-denominator API. The application needs a stable receipt contract, but the adapter must still surface upload state, errors, and retention decisions. Hiding those details produces a prettier interface and a harder incident.

Retention is the audit policy, not a bucket setting

The original receipt may need to survive processing for a defined period. Set that policy before implementing deletion: identify the legal or internal retention owner, store the decision with the receipt record, and make deletion an explicit job with an auditable result. Lifecycle support has a minimum period of one day, so hour-level expiry cannot be delegated to lifecycle configuration.

There are sharper limits for audit-heavy systems. Public or public-read ACLs are not available, so this is not a fit for static-site hosting, permanent public links, or an image CDN pattern. Object versioning and WORM-style object lock are also unavailable; an accidental overwrite is not recoverable through versions, and financial-grade immutability needs an external control. CORS cannot be self-configured through an independent route, which matters for browser direct upload: confirm the deployment's CORS setup before promising a fully self-service frontend flow.

I would keep the deletion worker boring. It checks the database retention decision, verifies that processing state is no longer pending, deletes the object, and records the outcome. It does not infer deletion from a missing OCR result, and it does not overwrite the original with a normalized image. Boring is good here.

The decision I would ship

Start with presigned URLs for the common receipt path. They are atomic, frontend-friendly, and easier to reason about in an audit trail. Add multipart only when measured file sizes or failed-transfer behavior justify the extra coordination, then give incomplete uploads their own cleanup and monitoring path.

Before release, I would verify five things in prose and tests: the browser can perform the signed request with the deployment's CORS rules; the database maps a retry to the same object key; large-file recovery records each part and completion result; retention decisions cannot be bypassed by the parser; and the selected provider supports the required region and migration story. If any answer is vague, the upload choice is not finished.

That is the useful abstraction: the receipt service depends on a small upload-and-retain contract, while the storage implementation can move behind it. The contract stays put while the thing behind it moves. The file-size rule remains simple, and the audit boundary remains visible.

Sources

Top comments (0)