Skip to main content

Module io_pool

Module io_pool 

Source
Expand description

Bounded writer-thread pool that decouples disk writes from rayon CPU workers (issue #569, phase 1).

Rayon worker threads that call fs::write directly stall a CPU slot for the duration of the syscall. IoPool moves those writes onto a small dedicated pool of writer threads (2–4) fed by a bounded std::sync::mpsc channel: producers enqueue {path, bytes} jobs with IoPool::write and the bounded channel provides natural backpressure (a full queue blocks the sender instead of buffering unbounded memory).

§Design constraints

  • std-only, tokio-free — per ADR-0001, ssg runs one scheduler (rayon) plus plain OS threads; no async executor is introduced here.
  • io_uring is out of scope — that is phase 2 of issue #569 (v0.0.48+, Linux-only feature flag). This module is the thread-pool backend only.
  • No silent data loss — every write error is captured and surfaced by IoPool::flush. Dropping the pool without a final flush() still drains and joins the writers; any errors that were never observed via flush() are logged at error level from Drop.

§Flush semantics

IoPool::flush is a barrier, not a shutdown: it blocks until every job enqueued so far has been fully processed (written or failed), then reports the first captured error (logging any additional ones). The pool remains usable afterwards, so a build phase can flush() between batches and reuse the same threads.

§Examples

use ssg::io_pool::IoPool;
use tempfile::tempdir;

let dir = tempdir().unwrap();
let pool = IoPool::new();
pool.write(dir.path().join("a.html"), b"<p>a</p>".to_vec()).unwrap();
pool.write(dir.path().join("b.html"), b"<p>b</p>".to_vec()).unwrap();
pool.flush().unwrap(); // barrier: both files are durably on disk
assert_eq!(std::fs::read(dir.path().join("a.html")).unwrap(), b"<p>a</p>");

Structs§

IoPool
A small pool of dedicated writer threads fed by a bounded channel.