Our air-quality heatmaps took about 5 seconds per image, on an hourly Python cron that used 4 GB of RAM and 2 CPUs and still wasn't enough. The rebuild is a Go service that makes them on request: 20 images in 30 ms, for any past time range.
I lead the software team at Oizom, an air-quality monitoring company (the platform is Envizom). This is how the heatmap backend was rebuilt there, including the bottleneck I didn't expect: PNG encoding.
How the old version worked
We used the IDW (inverse distance weighting) algorithm with wind speed and direction to figure out the value for each spot.
- We set up a heatmap config in our database.
- A cron job ran every hour.
- We grabbed all the configs.
- For each config, we had boundaries, known devices, limits, and color codes.
- We pre-computed a grid of small 1km x 1km boxes and stored it in the CDN.
- We used the center of each box in the pre-computed grid as the unknown value.
- We knew each gas parameter for all devices, plus real-time geolocation, wind speeds, and directions.
- Using IDW with wind effects, we calculated the value for each box in the grid.
- Colors were added to the grid based on user-defined limits.
- A Base64 PNG image was created and stored in a time-series database.
- This image was served to the frontend when requested.
This was written in Python and used up a ton of resources (4GB RAM, 2 CPUs, 1-2 instances depending on the load), and it still wasn't enough. Each image took 10-20 seconds to process. It just wasn't fast. Back then, heatmap was a brand new feature, and we only had 3 configs, so it clearly wasn't scalable. Plus, on closer look, the wind effects weren't even used correctly.
A weekend proof of concept in the browser
When I found out about this, I suggested generating the heatmap on the frontend. My boss thought the idea was nuts. He didn't believe it was even possible. My argument was simple: today, there are tools like Canva and Figma, and many others that let you edit photos and videos right in the browser. There are even proper 2D games and simulations being played in the browser, and some resume websites are like 3D games. Technology has improved, and coding patterns and styles have changed a lot in the past 2 years, so I thought it was doable. He was convinced and decided to hire an intern to do the R&D. Typical corporate.
This really bugged me because I've always wanted to dive into image generation projects, and this seemed like my golden opportunity. So, I spent a weekend putting together a POC for a heatmap in the browser. It worked.
This was just the start. I still needed to add wind effects, map (long, lat) to (x, y) coordinates, include user-defined boundaries (from a GeoJson polygon), and finally, add gradient colors.
Boss saw it, boss liked it, and everyone was happy.
He still had a good point. We don't want to do this on the frontend because our frontend runs on old phones, corporate laptops, and sometimes even TVs—in other words, places without much computing power. So, maybe we can set up a proxy server to handle all the heatmaps and generate them on the fly.
Node.js, then Go, then the real bottleneck
So, I went ahead and set up a Node server and built the whole thing. It worked great. Turns out, when you use gradient colors, there's not much difference between a 150px resolution and a 1000px resolution, except for having sharper edges.
Test case: each request with 100px resolution, 10 known points with 40 images. Each load ran for 5 seconds.
In Node.js the results were already so much better than anything we had before. After this, I decided that to make it even faster, we should use a lower-level language. We also wanted to ensure the language is easy for other developers to understand. So, I chose GoLang. Go was faster and throttled less.
Something still felt off; it shouldn't take this long. After adding time logs for each function, I discovered the bottleneck was Base64 PNG encoding. Each image took 1-3ms, which was 80% of the time. Switching to JPEG boosted performance dramatically.
| Load (5 s) | Node.js | Go, Base64 PNG | Go, JPEG |
|---|---|---|---|
| 5 req/sec | 108ms-874ms (average 419ms) | 80ms-175ms (average 114ms) | 18ms-56ms (average 29ms) |
| 50 req/sec | 97ms-26,588ms (average 14,687ms, with 2 req/sec failing) | 157ms-265ms (average 458ms) | 29ms-132ms (average 88ms) |
| 500 req/sec | 172ms-37048ms (average 22,152ms, with 432 requests failing) | 241ms-10,891ms (average 4,405ms, 296 req/sec failing) | 44ms-929ms (average 496ms, 80 req/sec failing) |
The only downside to JPEG is that it cannot create transparent images, as it lacks an alpha channel. However, we have a straightforward solution: let the backend handle all the computational tasks, while the frontend focuses on one task—masking the image and selecting only the required portions. This can be accomplished in the frontend using JavaScript, within 10-30ms. If the client can render GeoJSON, maps, and images, it can also perform image masking.
This approach also offers an advantage: the difference between a 150px and a 1000px image is that the 1000px image has sharper edges. To achieve sharper edges for a 150px image, we generate an image with an additional 2px around the boundary and crop the necessary section in the frontend. This results in sharper edges.
What the new pipeline does
- The frontend creates a config from backend APIs and stores it in the database.
- The frontend requests images for a config, gas parameters, and given time bounds.
- The backend fetches all the resources and sets up a payload to send to the GoLang heatmap server.
- In the GoLang server: It takes all the boundary points in (long, lat), maps them to (x, y) within 0-1, creates a grid of boxes based on resolution, and finds the ones within the polygon.
- It converts all the known points/devices' locations from (long, lat) to (x, y) based on grid transformation.
- This grid context is used to precompute the weight of each known point on each target pixel.
- Based on the given colors, we create a gradient of values ranging from 0-256 integers.
- We set up the encoder and precompute the (x, y) to (pixel position) in the data image array.
- For each snapshot of values, we quickly compute the grid's value using precomputed weights.
- We magnify each value to bring it between 0-256.
- We fill the RGB values in the data image array.
- We use the encoder for the base64 image.
- These images are sent back to the core server.
- The core server sends them back to the client.
- The frontend masks the required portion using geo boundaries. Done!
Before and after
Images are generated on the fly, so we can now create past images too. We use fewer resources, which makes everything cheaper overall. Plus, it's finally scalable and versatile.
- Heatmaps were only available after they were created → Now we can get heatmaps for any time range.
- For a fixed 1-hour average → You can completely customize it.
- 5 seconds per image → Now it's 30 ms for 20 images.
- Python → GoLang.
- Complex scheduler-based solution → Simple request/response-based solution.
- Fixed options → Custom options for resolution, distance power, wind power, and wind effect.
Finally:
Of course, the data used to create this heatmap is fake and random. But this is how it would look.
The fix I didn't see coming wasn't the language or the algorithm. It was a time log on every function, which showed the image encoder taking 80% of the time. What was the last bottleneck you found that way, somewhere you weren't looking?
Top comments (22)
Thanks for the correction on math.Pow — good point, precomputing the weights per request makes the power cost one-time, and every snapshot after that is multiply-and-add.
On WebP encode time: I haven't measured it in Go, so treat this as a hypothesis rather than a number. If you're on the pure-Go encoder (chai2010/webp — golang.org/x/image only decodes), you're paying several-fold over cgo libwebp, so the codec wrapper may matter more than the format choice. One structural saving: encode RGB-only and let the frontend mask own the transparency — the alpha channel is where lossy WebP spends extra time on smooth gradients, and you keep the polygon crop either way. Might be worth a quick A/B before committing to a format change.
@contentclips_st fair point on the wrapper. A pure-Go encoder against cgo libwebp could swamp the format difference, so an A/B has to name the library on both sides, not just the format.
The PNG cost only showed up because of a time log on every function, so the same harness would give the WebP number: same snapshot, JPEG vs WebP, encode time and bytes. RGB-only makes sense there, since the mask owns transparency anyway.
Would you judge it on encode time alone, or on bytes over the wire too?
Both, but in different columns. Encode time first — if the WebP encoder eats the same share of the pipeline the JPEG one did, that's the fix to make before anything else. Bytes over the wire is what the end user actually pays for on a heatmap endpoint, so once encoding isn't the bottleneck I'd report ms and KB side by side from the same harness (same snapshot, both formats, encoder library named on each side). One caveat: WebP only wins if the smaller file actually ships — with the mask forcing RGB anyway, keep an eye on whether the alpha handling erodes the gain. At 30 ms total you're already fast, so I'd treat wire size as the next metric to optimize and keep encode time as a regression guard.
@contentclips_st makes sense. Encode time answers whether the change is worth making at all, and bytes answer whether the user notices. One table from the same snapshot, ms and KB side by side, encoder named on each side.
Agreed on where the next cost is, too. At 30 ms for 20 images the server isn't the slow part any more. The images still have to reach old phones and corporate laptops, so wire size is the better next target. Thanks for staying with this thread.
The ms/KB side-by-side table is the right artifact — one measurement trap worth building into it: capture wire size (compressed, as served) alongside raw image bytes, because the CDN/transport layer silently re-encodes and the number a user's phone actually pulls can differ 20-40% from the encoder's output. That gap is where the 'encoder says 45KB, device downloads 70KB' surprises live. Same snapshot, three columns: ms, raw KB, wire KB — then the encoder comparison and the transport tax are both visible in one read.
@contentclips_st good catch. The encoder's byte count is not what the phone downloads, and a two-column table would hide that. So three columns from the same snapshot: encode ms, raw KB from the encoder, and wire KB read off the response as the browser received it. Then a format change and whatever the transport does to it show up separately. Where have you seen that 20-40% gap come from most often, a CDN re-compressing images or something closer to the client?
The biggest contributor I've seen isn't CDN re-compression at all — it's rendition mismatch. The encoder measures the asset you generated, but the phone's srcset/picture resolution picks a different variant entirely at 2x or 3x DPR, so the two numbers describe different images and the gap is really a comparison bug. Second most common is content negotiation swapping format at the edge, AVIF or WebP served while the encoder measured PNG or JPEG — again different bytes, not overhead. Real transport overhead is smaller than people assume, maybe 5-10% once gzip or brotli and header framing are counted. A 20-40% delta usually means one of those two mismatches rather than the CDN inflating anything. Cheap way to settle it: log the response Content-Length together with the actual decoded body size and the requested variant URL — if the variant URL differs from what the encoder measured, you've found the gap without touching the CDN config.
@contentclips_st that turns the gap from a measurement into a bug report, which is more useful. So the row gets one more field: the variant URL the client actually requested, next to Content-Length and the decoded size. If that URL is not the file the encoder wrote, the row is comparing two images and gets thrown out before anyone reads the wire column. Thanks for the detail on the 5-10% too; it gives me a number to expect once the variants match.
On the variant URL field — one refinement before the two get compared literally: normalize them first, because param order and CDN signing rewrites make string equality noisy and healthy rows will read as mismatches. What earns its keep next to it is the request context: viewport, DPR, and the descriptor the page actually offered. With those three, 'wrong variant on the wire' splits into two different bugs — the page sized it wrong (selection bug, fix srcset/sizes) versus the edge served a sibling from the same cache key (config bug) — same row, two different fixes, and the URL alone can't say which. And once the mismatch count drains to near zero, check whether the stragglers cluster on a single edge or region: that pattern is almost always cache-key population rather than a rendering regression. Curious what your gap lands at once it clears.
@contentclips_st normalizing first is fair. Comparing raw URL strings would flag healthy rows over param order and signing, and bury the real mismatches among them. The split you describe is the part the URL alone can't give me: the page asking for the wrong size and the edge serving a sibling from the same cache key are fixed in different places, so the row should say which one it is. Honest answer on the gap: nothing measured yet. What I have is the column list this thread built, not a run, so I don't have a number to give you.
"Which one it is" comes almost free if the row records both ends literally: the candidate URL the page actually picked (currentSrc plus the descriptor that won) and the URL and headers the edge actually served. The classification then writes itself — normalized-equal with matching bytes is a pass; same request, different bytes or format is the edge swapping; different request is the page asking for the wrong thing. The honest "nothing measured yet" is the right answer, too: the column list is the deliverable, and a number before it would just be a number. If you want to de-risk before instrumenting everything, pilot twenty rows, classify them by hand, and watch which buckets feel ambiguous — that surfaces a missing column faster than any spec. One build note on currentSrc: it can be empty before the image loads, so capture it at load/settled time, not during a markup scan.
@contentclips_st recording both ends turns the classification into a lookup rather than a judgment call: same normalized request and same bytes is a pass, same request with different bytes or format is the edge, a different request is the page. The twenty-row pilot is a better first step than finishing the spec, since sorting a small batch by hand shows which bucket is missing a column. And the currentSrc note is useful. Read during a markup scan it would just come back as a column of blanks, so it gets captured once the image has loaded.
Two additions, both about keeping the result trustworthy once this scales. Store the raw captures and treat the label as a view over them (both URLs, bytes hash, format, capture timestamp — immutable per row), so when the first real mismatch forces a new bucket you re-classify the same twenty rows instead of re-measuring them. And if a capture never settles, keep the row as unresolved rather than dropping it: a dropped row moves the denominator silently, and then the run-over-run split stops being comparable. The hesitation cases from the hand-sort are worth keeping, too — they make a natural fixture set for whichever classifier eventually automates this.
@contentclips_st keeping the raw capture immutable and the label as a view over it is the right shape. A new bucket then costs a re-classify of rows already on disk, not another measuring run. The unresolved rows are the same point from the other side: drop them and a week with more slow loads reads cleaner than it was. And the rows I hesitate on in the hand-sort become the test set for whatever automates the sort later. Thanks for building this out with me.
Likewise, Panth — and thanks for pushing on the details that made the row format hold together. Your hesitation rows are queued as the classifier's first test: if it can't beat the hand-sort on those, it ships as advisory only. Good talking with you.
@panthpatel Exactly — that field makes the row self-validating. One companion column worth adding while you are there: the format actually served (response Content-Type), because the second mismatch source — the edge swapping AVIF/WebP for the PNG/JPEG the encoder measured — stays invisible if you only track URLs. And to make the rendition check explicit rather than inferred, log the srcset descriptor the browser picked (or the decoded naturalWidth) — then which mismatch it was becomes a glance instead of a guess. With both in, your 5-10% expectation should show up clean, or the table points straight at the culprit. Great thread — curious where the p95 lands once the baseline is clean.
@contentclips_st both make sense. Content-Type catches the edge swapping formats, and naturalWidth or the picked srcset descriptor catches the wrong rendition, so each bad row says which of the two it is instead of leaving me to guess. That puts the row at encode ms, raw KB, variant URL, Content-Type, Content-Length, decoded size and naturalWidth, all from one snapshot. When you have run this kind of table before, did the p95 move much once the mismatched rows were thrown out, or was it mostly the median that changed?
@panthpatel The honest shape I keep seeing: the mean barely budges, the p95 moves a lot — and almost all of the move shows up the moment swap rows get their own bucket, because you stop averaging two different failure signatures (encoder too slow vs delivery served the wrong bytes). Two things to watch in the re-run: the tail is usually a distribution problem — a handful of rows where the source was already small and the encoder still re-squeezed it; and once mismatches drain to ~0, any p95 gap left between weeks tends to track one or two routes re-generating variants inline. With an expected-vs-served column in the table, the comparison is honest from the first run — without it, a "worse" week may just be more AVIF traffic. Curious what your gap ends up being once the mismatch count clears.
Nice rebuild - the PNG encoding being the bottleneck is a classic trap. A few things that could push it further:
Your precomputed weight matrix only holds if wind speed/direction are baked into the weights. IDW with wind anisotropy usually means each known point's weight changes per snapshot - if wind varies per timestamp, keep the base distance weights precomputed and apply the wind term as a cheap per-snapshot correction instead of recomputing the full matrix.
Since you already quantize values to the 0-256 gradient integers, you can drop math.Pow: with p=2, IDW is 1/squared distance and weights normalize by their sum - one reciprocal per known point per pixel, no transcendentals.
For transparency you don't strictly need the frontend mask: WebP supports alpha and encodes these gradients in the same ballpark as JPEG. Frontend masking is still the right call for polygon cropping though.
JPEG at 100-150px will band on smooth gas gradients - encoding at 2x resolution and letting the frontend downsample gives smoother ramps for negligible cost.
@contentclips_st thanks, this is a useful list.
On math.Pow: distance power is a request option, not fixed at 2, so the power can't go away entirely. But the weights are precomputed once per request (step 6), so whatever the power costs is paid once per known point per pixel. Every snapshot after that is multiply-and-add. A p=2 fast path would still be cheap to add.
On wind: the post is thin on how wind enters the weights, that's a fair call. Keeping the distance weights fixed and applying wind as a per-snapshot correction is a clean way to split it.
On WebP: the frontend mask stays either way, because we crop to the GeoJSON polygon, so alpha in the image wouldn't save us a step.
Have you measured WebP encode time against JPEG in Go? The encoder was 80% of our time once, so that's the number I'd check first.
Glad the list was useful, Panth — and thanks for pushing on the details.
On math.Pow: agreed, distance power being a request option means it cannot disappear entirely — but since the weights are precomputed once per request anyway, a p=2 fast path should be nearly free. Worth adding.
On wind: per-snapshot correction on top of fixed distance weights is the clean split — we will model it that way.
On WebP: fair point on the GeoJSON mask making alpha moot. Honest answer on encode time: we have not measured WebP vs JPEG in Go yet, it is on the bench list. Your "encoder was 80% of our time" data point is exactly the number that decides it — we will profile before claiming anything.
Do not follow any external links! DEV.to uses Sloan for automated messages, this is likely phishing.