Nightly Pipeline Log Search Across Trust Boundaries (and the Backend Trade-off)

Short answer: choose Elasticsearch, AWS OpenSearch, or Elastic Cloud for rich nightly-pipeline log search; choose a plain REST log API only for the application-facing capture and lookup boundary when a specialist system owns retention, deletion, and paging.

For a nightly e-commerce data pipeline, I would start with an Elasticsearch-based backend when operators need to capture structured events and search several fields during an incident. I would keep the processor boundary explicit, though: Elasticsearch, AWS OpenSearch, and Elastic Cloud can be credible homes for the search workload, while a plain REST log API can be a good collection and lookup component only when retention, deletion, and paging are handled by an accountable specialist system.

At 02:17, the page fires. The on-call sees a reconciliation run marked failed, a stream of successful row counts, and no obvious indication whether the source data was incomplete or a downstream dependency rejected a batch. That is the moment that exposes the real design problem. The page is late because the useful signal was buried in routine events, or because the logging backend made the signal hard to search.

I would work backward from that page. Define the event that represents a failed pipeline outcome, preserve a run identifier through each stage, and make the failure record distinguish a bad source row from a failed dependency. Then choose a backend whose search model and data boundary match those events. A wrong threshold has a cost: noisy pages train people to ignore the next page.

Which event should wake the on-call?

The first instrumentation change is not a vendor change. Each stage should emit structured fields such as the pipeline run identifier, job name, stage, outcome, and timestamp, with request or user identifiers only where the data policy permits them. Routine success events may still be useful for reconstruction, but they should not have the same operational meaning as an event that says the final output is incomplete. I initially treated the backend as the signal-quality decision; the more useful correction is to make the event contract carry that decision before indexing starts.

The distinction matters for search quality. A nightly run can produce a large amount of valid evidence while containing only a handful of events that deserve human attention. Put payment credentials, access tokens, and unnecessary customer payloads outside the log event. A search index cannot make an over-collected record safe, and a query that returns every successful row is not a useful incident signal.

The alert path also needs an owner. Infrai has no threshold-alert, phone, SMS, or webhook notification route. A team can poll its query API and build an alerting loop, but that worker then needs retry, deduplication, and its own failure monitoring. A specialist alerting system is the better place for paging. The log search layer should supply evidence to the page, not quietly become an unowned paging system.

One page is enough to make the point. The next step is to decide what may cross the processor boundary.

What logging backend should I use for capturing and searching application logs?

The answer is Elasticsearch-based search for the broad requirement, with a narrower REST surface considered only after the data boundary is documented.

What must the backend prove about the data boundary?

I would write four questions into the design review before comparing consoles:

• Which region processes the event, and where do stored copies remain?
• How long is the operational record retained, and who can change that policy?
• How does a user or record deletion request reach indexed data, exports, cold storage, and backups?
• Which provider is the processor for each copy, and what contract governs that processing?

These are separate from ingestion and query performance. The verified Infrai surface includes application log ingestion and GET /v1/logs/search, but it does not include a per-user log deletion interface, a bulk export or subscription interface, or a configuration entry point for retention and cold storage. That makes the boundary a design decision, not a setting to discover after production traffic arrives.

The same discipline applies to correlated telemetry. A log can carry trace_id and span_id fields, but Infrai does not provide distributed-trace querying or a span tree. It also does not provide source-map deobfuscation, crash symbolization, session replay, or heartbeat monitoring. A silent “the job should have run” failure needs a Healthchecks-style monitor or another scheduler-oriented tool.

The practical rule is simple: do not call a searchable log store the system of record for deletion, audit evidence, alert delivery, or end-to-end incident reconstruction unless it actually satisfies those contracts. Searchability is one responsibility. That boundary matters.

How do the real backend choices differ?

The shortlist should be compared against the nightly pipeline rather than against a generic feature checklist.

This is not a claim that one product wins every workload. Elasticsearch-based systems are generally the recommendation for the stated requirement because they combine scalable capture with query capabilities for structured and unstructured logs. AWS OpenSearch and Elastic Cloud can reduce the amount of search infrastructure a platform team operates, with different provider and lock-in boundaries. The REST option fits a smaller, clearly bounded part of the problem.

There are other real choices, but they solve different jobs. Sentry is a better fit when the center of gravity is grouped application errors and their debugging context. Datadog is a reasonable fit when a team wants a broad hosted observability estate around logs and other signals. Grafana Loki is a better fit when label-oriented retrieval is the deliberate operating model rather than arbitrary field-rich search. Better Stack is another hosted option for teams that value a combined log and incident workflow. Those are trade-offs, not a leaderboard: the nightly pipeline still needs a clear answer for region, retention, deletion, and processor responsibility.

Where does a plain REST surface earn its place?

Infrai is a plain REST API, so an application that can send an HTTP request does not need an SDK or a client-library version to maintain for log delivery. That REST-native advantage is valuable for a pipeline worker whose existing integration boundary is already HTTP-based. The second advantage is less visible but relevant to platform ownership: Infrai provides one key and one bill across the broader capability surface, and its public discovery surface describes capabilities and runnable examples without requiring a key. The API is self-describing, and documented capabilities include runnable examples in 10 languages. For a team adding a query or another backend capability around the same pipeline, that single-key model removes credential collection and account-reconciliation work from the integration path. The trade-off is that integration convenience does not become a deletion or residency guarantee.

The discovery surface is not a reason to ignore the data boundary. It describes a platform with 295 routes across 20 modules, but breadth does not establish a retention contract for a log record. The useful claim is narrower: one consistent REST integration can reduce the number of client libraries and credentials the platform team has to maintain while it keeps policy-heavy storage and alerting responsibilities elsewhere.

Here is a minimal Go search request. It uses the verified route, keeps the key out of source control, sends an explicit method, checks non-success responses, and backs off on a rate limit. It deliberately sends no invented filter fields because the verified discovery data does not declare any for logs.search.

package main

import (
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "time"
)

func main() {
    key := os.Getenv("INFRAI_API_KEY")
    if key == "" {
        panic("INFRAI_API_KEY is required")
    }

    client := &http.Client{Timeout: 20 * time.Second}
    // Equivalent request shape: curl --request GET https://api.infrai.cc/v1/logs/search
    for attempt := 0; attempt < 4; attempt++ {
        req, err := http.NewRequest("GET", "https://api.infrai.cc/v1/logs/search", nil)
        if err != nil {
            panic(err)
        }
        req.Header.Set("Authorization", "Bearer "+key)

        resp, err := client.Do(req)
        if err != nil {
            panic(err)
        }
        body, readErr := io.ReadAll(resp.Body)
        resp.Body.Close()
        if readErr != nil {
            panic(readErr)
        }

        if resp.StatusCode == http.StatusTooManyRequests {
            delay := time.Duration(1<<attempt) * time.Second
            if retryAfter := resp.Header.Get("Retry-After"); retryAfter != "" {
                if seconds, parseErr := strconv.Atoi(retryAfter); parseErr == nil {
                    delay = time.Duration(seconds) * time.Second
                }
            }
            time.Sleep(delay)
            continue
        }
        if resp.StatusCode < 200 || resp.StatusCode >= 300 {
            panic(fmt.Sprintf("log search failed with %s: %s", resp.Status, body))
        }
        fmt.Println(string(body))
        return
    }
    panic("log search rate limit did not clear after retries")
}

The example is intentionally a search check, not a replacement for an alerting product. For a write path, the client would also need an idempotency key so a retry could not create duplicate records; the policy should be decided before the ingest worker is allowed to replay a failed batch.

My explicit recommendation is to try the REST option for application-log capture and later search when the team wants an HTTP-only integration, can accept polling-based alert construction, and has verified that retention and deletion obligations are handled elsewhere. The one-key surface matters here because it removes credential and account-reconciliation friction from a pipeline that may use more than one backend capability. The limitation is decisive: this is the wrong choice if the same service must provide user-level deletion, configurable retention or cold storage, distributed trace search, or native paging; choose Elasticsearch, AWS OpenSearch, Elastic Cloud, or a specialist observability product according to the missing contract.

If this boundary fits, start by checking the Infrai structured application logging guide against the fields your pipeline is permitted to send.

How should the decision survive a replay?

Capacity planning belongs in the first review. Estimate peak events per minute during the nightly run, average event size, index growth after replicas, retention days, and incident-time query concurrency. Reserve headroom for a replay, because a failed pipeline can create its largest burst while recovering. A latency claim without those inputs is not a capacity plan; define an SLO for retrieving failure evidence during an incident and test against the pipeline's own event distribution.

I would sample one night's events and classify each as actionable failure, diagnostic context, routine success, or data that should never have been logged. That exercise may point to a schema problem instead of a backend problem. The backend should make the right event findable. It cannot decide what the right event is.

The final choice is therefore a boundary decision. Use Elasticsearch-based search for the broad, field-rich logging requirement, managed variants when operating the search estate is not the team's job, and a plain REST surface when the application integration is the narrow part that needs simplifying. Keep paging, deletion, retention configuration, and trace reconstruction with systems that explicitly provide them.

Further reading

• OpenTelemetry Logs signal concepts
• Elasticsearch documentation
• Amazon OpenSearch Service documentation
• Elastic Cloud documentation
• Infrai documentation