Module streaming 

Streaming compilation for large (100K+ page) sites. Streaming compilation for large sites.

Processes content files in batches to cap peak memory usage, enabling compilation of 100K+ page sites within a configurable memory budget.

The streaming compiler divides content files into chunks based on the memory budget, compiles each chunk, then releases it before processing the next. After all chunks, a merge pass unifies cross-page artefacts (sitemap, search index, feeds).

Structs

MemoryBudget

Memory budget configuration for streaming compilation.

Constants

DEFAULT_MEMORY_BUDGET_MB

Default peak memory budget: 512 MB.

Functions

batched_content_files

Collects content files and returns them as batches.

compile_batch

Compiles a single batch of content files into the build directory.

should_stream

Determines whether streaming compilation should be used.