NuORDER · export worker, incident Nov 2024

Streaming the product export: from daily OOM kills to −50% memory

Export workers were OOM-killed every day and more RAM had bought only days. A root-cause analysis found the catalog held in memory three times; streaming the workbook to disk fixed it in five weeks.

MY ROLESecond engineer on the incident; wrote the RCA and implemented the fix

memory consumption for large data exports: −50%
OOM restarts: daily → 0
copies of the catalog in memory: 3 → 1
incident to fix: 5 weeks

THE SHORT VERSION

In November 2024, export worker pods were OOM-killed and restarted daily, and the export queue backed up for every brand. My RCA showed one export held the catalog in memory three times. My two-step fix (halve concurrency and free rows, then stream the workbook to a temp file) cut memory consumption for large data exports by 50% and brought OOM restarts to zero.

Context

Brands on NuORDER can export their full catalog as spreadsheets through 'Download your data'. In November 2024 the export worker pods started getting OOM-killed and restarted every day. Exports never finished and the queue backed up for all brands. The trigger was brands running full-catalog exports in parallel.

My role

I was the second engineer on the incident. A first fix, more RAM, had bought only days. I wrote the root-cause analysis and implemented both fix steps, including the stream option in the shared spreadsheet library.

Problem

The arithmetic ruled out more memory. The worker limit was 12 GB. Eight parallel jobs at 3–4 GB each need 24–32 GB.

SYSTEM NOTE / ENGINEERING DETAIL

Before: 8 jobs × 3–4 GB

24–32 GB demand

After the fix

parallel exports 8 → 4
catalog held once, streamed to a temp file on disk
memory for large data exports −50%; OOM restarts daily → 0

Before the fix: worst-case demand of 8 parallel jobs against the 12 GB worker limit. After: half the concurrency, one copy of the catalog, memory for large exports −50%.

The largest catalog had 37,535 products, up to 36 sizes and 143 schema fields: about 31M cells across 3 files of 186 columns. One XLSX export of it peaked at 3–4 GB. I reproduced this on staging with a copy of that catalog.

Root cause: data streamed in from the database, then was held three times over: as row arrays, as the full ExcelJS workbook model, and as the whole zipped file buffer before upload.

SYSTEM NOTE / ENGINEERING DETAIL

Before: catalog held 3 times

DB stream

1. row arrays

in memory

2. full ExcelJS workbook model

in memory

3. whole zipped file buffer

in memory

peak 3–4 GB per XLSX

Upload

After: catalog held once

DB stream

Streaming WorkbookWriter

commit every 1,000 rows; rows freed once written

Temp file on disk

memory for large exports −50%

Upload from file

temp file deleted in finally

Before: three full copies of the catalog in memory. After: rows flow through a streaming writer to a temp file and are uploaded from disk.

Constraints

Worker memory limit of 12 GB, with exports for all brands blocked behind the stuck jobs.
The spreadsheet code lives in a shared library, so the change had to ship as a library release, not a local patch.
Existing exports had to produce the same files, or fail visibly.

Decisions and tradeoffs

More RAM: rejected: Already tried; it bought days. 24–32 GB of demand against a 12 GB limit cannot be closed with headroom.
Ship a quick win first: One small change: parallel exports 8 → 4, and free rows once they are written. Tradeoff accepted: fewer concurrent exports.
Stream as an opt-in library option: A stream option in the shared spreadsheet library, released as a new minor version, instead of a fork inside the worker.
Fail loudly on what cannot stream: Merged cells, freeze panes and links cannot be written in streaming mode. The library throws instead of writing a broken file.
Split images out: Images do not stream, so they moved to a follow-up rather than blocking the fix.

Implementation

Quick win. Parallel exports 8 → 4; free row arrays as soon as they are written.
Streaming WorkbookWriter. Commit every 1,000 rows instead of building the whole workbook model.
Disk, not buffer. Write to a temp file, upload from the file, delete it in finally.
Library release. Stream option added to the shared spreadsheet library as a new minor version. The worker change itself was ≈25 lines.

Review and what bit

Images do not stream. Split into a follow-up.
Some features (merged cells, freeze panes, links) cannot stream. The library now fails loudly instead of writing a broken file.
An API misuse was caught in code review and pinned with a failing unit test.
A behaviour change (an empty export without a header row), which I flagged myself.

Results & evidence

Memory result confirmed for large data exports; qualitative rows measured on the largest catalog.
Metric	Before	After	Change
Memory for large data exports	baseline	half of baseline	−50%
Catalog copies in memory	3	1
OOM restarts	daily	0
Largest brand's XLSX export	never ends	completes
Concurrent exports (tradeoff)	8	4
Size of change	—	≈25 lines + library release

Engineering takeaways

Do the arithmetic before buying memory: 8 × 3–4 GB was never going to fit in 12 GB.
Ship the small, safe change first; it buys time for the structural fix.
When a mode cannot support a feature, throwing is kinder than writing a broken file.