Our air-quality heatmaps took about 5 seconds per image, on an hourly Python cron that used 4 GB of RAM and 2 CPUs and still wasn't enough. The reb...
For further actions, you may consider blocking this person and/or reporting abuse
Thanks for the correction on math.Pow — good point, precomputing the weights per request makes the power cost one-time, and every snapshot after that is multiply-and-add.
On WebP encode time: I haven't measured it in Go, so treat this as a hypothesis rather than a number. If you're on the pure-Go encoder (chai2010/webp — golang.org/x/image only decodes), you're paying several-fold over cgo libwebp, so the codec wrapper may matter more than the format choice. One structural saving: encode RGB-only and let the frontend mask own the transparency — the alpha channel is where lossy WebP spends extra time on smooth gradients, and you keep the polygon crop either way. Might be worth a quick A/B before committing to a format change.
@contentclips_st fair point on the wrapper. A pure-Go encoder against cgo libwebp could swamp the format difference, so an A/B has to name the library on both sides, not just the format.
The PNG cost only showed up because of a time log on every function, so the same harness would give the WebP number: same snapshot, JPEG vs WebP, encode time and bytes. RGB-only makes sense there, since the mask owns transparency anyway.
Would you judge it on encode time alone, or on bytes over the wire too?
Both, but in different columns. Encode time first — if the WebP encoder eats the same share of the pipeline the JPEG one did, that's the fix to make before anything else. Bytes over the wire is what the end user actually pays for on a heatmap endpoint, so once encoding isn't the bottleneck I'd report ms and KB side by side from the same harness (same snapshot, both formats, encoder library named on each side). One caveat: WebP only wins if the smaller file actually ships — with the mask forcing RGB anyway, keep an eye on whether the alpha handling erodes the gain. At 30 ms total you're already fast, so I'd treat wire size as the next metric to optimize and keep encode time as a regression guard.
@contentclips_st makes sense. Encode time answers whether the change is worth making at all, and bytes answer whether the user notices. One table from the same snapshot, ms and KB side by side, encoder named on each side.
Agreed on where the next cost is, too. At 30 ms for 20 images the server isn't the slow part any more. The images still have to reach old phones and corporate laptops, so wire size is the better next target. Thanks for staying with this thread.
The ms/KB side-by-side table is the right artifact — one measurement trap worth building into it: capture wire size (compressed, as served) alongside raw image bytes, because the CDN/transport layer silently re-encodes and the number a user's phone actually pulls can differ 20-40% from the encoder's output. That gap is where the 'encoder says 45KB, device downloads 70KB' surprises live. Same snapshot, three columns: ms, raw KB, wire KB — then the encoder comparison and the transport tax are both visible in one read.
@contentclips_st good catch. The encoder's byte count is not what the phone downloads, and a two-column table would hide that. So three columns from the same snapshot: encode ms, raw KB from the encoder, and wire KB read off the response as the browser received it. Then a format change and whatever the transport does to it show up separately. Where have you seen that 20-40% gap come from most often, a CDN re-compressing images or something closer to the client?
@panthpatel Exactly — that field makes the row self-validating. One companion column worth adding while you are there: the format actually served (response Content-Type), because the second mismatch source — the edge swapping AVIF/WebP for the PNG/JPEG the encoder measured — stays invisible if you only track URLs. And to make the rendition check explicit rather than inferred, log the srcset descriptor the browser picked (or the decoded naturalWidth) — then which mismatch it was becomes a glance instead of a guess. With both in, your 5-10% expectation should show up clean, or the table points straight at the culprit. Great thread — curious where the p95 lands once the baseline is clean.
@contentclips_st both make sense. Content-Type catches the edge swapping formats, and naturalWidth or the picked srcset descriptor catches the wrong rendition, so each bad row says which of the two it is instead of leaving me to guess. That puts the row at encode ms, raw KB, variant URL, Content-Type, Content-Length, decoded size and naturalWidth, all from one snapshot. When you have run this kind of table before, did the p95 move much once the mismatched rows were thrown out, or was it mostly the median that changed?
@panthpatel The honest shape I keep seeing: the mean barely budges, the p95 moves a lot — and almost all of the move shows up the moment swap rows get their own bucket, because you stop averaging two different failure signatures (encoder too slow vs delivery served the wrong bytes). Two things to watch in the re-run: the tail is usually a distribution problem — a handful of rows where the source was already small and the encoder still re-squeezed it; and once mismatches drain to ~0, any p95 gap left between weeks tends to track one or two routes re-generating variants inline. With an expected-vs-served column in the table, the comparison is honest from the first run — without it, a "worse" week may just be more AVIF traffic. Curious what your gap ends up being once the mismatch count clears.
Nice rebuild - the PNG encoding being the bottleneck is a classic trap. A few things that could push it further:
Your precomputed weight matrix only holds if wind speed/direction are baked into the weights. IDW with wind anisotropy usually means each known point's weight changes per snapshot - if wind varies per timestamp, keep the base distance weights precomputed and apply the wind term as a cheap per-snapshot correction instead of recomputing the full matrix.
Since you already quantize values to the 0-256 gradient integers, you can drop math.Pow: with p=2, IDW is 1/squared distance and weights normalize by their sum - one reciprocal per known point per pixel, no transcendentals.
For transparency you don't strictly need the frontend mask: WebP supports alpha and encodes these gradients in the same ballpark as JPEG. Frontend masking is still the right call for polygon cropping though.
JPEG at 100-150px will band on smooth gas gradients - encoding at 2x resolution and letting the frontend downsample gives smoother ramps for negligible cost.
@contentclips_st thanks, this is a useful list.
On math.Pow: distance power is a request option, not fixed at 2, so the power can't go away entirely. But the weights are precomputed once per request (step 6), so whatever the power costs is paid once per known point per pixel. Every snapshot after that is multiply-and-add. A p=2 fast path would still be cheap to add.
On wind: the post is thin on how wind enters the weights, that's a fair call. Keeping the distance weights fixed and applying wind as a per-snapshot correction is a clean way to split it.
On WebP: the frontend mask stays either way, because we crop to the GeoJSON polygon, so alpha in the image wouldn't save us a step.
Have you measured WebP encode time against JPEG in Go? The encoder was 80% of our time once, so that's the number I'd check first.
Glad the list was useful, Panth — and thanks for pushing on the details.
On math.Pow: agreed, distance power being a request option means it cannot disappear entirely — but since the weights are precomputed once per request anyway, a p=2 fast path should be nearly free. Worth adding.
On wind: per-snapshot correction on top of fixed distance weights is the clean split — we will model it that way.
On WebP: fair point on the GeoJSON mask making alpha moot. Honest answer on encode time: we have not measured WebP vs JPEG in Go yet, it is on the bench list. Your "encoder was 80% of our time" data point is exactly the number that decides it — we will profile before claiming anything.
Do not follow any external links! DEV.to uses Sloan for automated messages, this is likely phishing.