Share CPU/memory autoscaling budget across multiple crawler instances in one process

[MrGeorgen]

Feature

Expose a way to share CPU/memory snapshotting (or a combined concurrency/CPU budget) across multiple independent crawler instances (CheerioCrawler/HttpCrawler/etc.) running concurrently in the same Node process, so each instance's AutoscaledPool scales up/down based on combined load across all participating instances, not just its own isolated CPU snapshot.

Motivation

We run several independent crawler instances concurrently in one process (Promise.allSettled, one instance per target domain, each with its own isolated Configuration). We deliberately keep them separate rather than merging into one crawler: each domain needs its own maxRequestsPerMinute (some sites need a conservative rate to avoid bot-detection blocks, others tolerate a much higher rate — a single shared instance only exposes one global rate/concurrency, so merging would force a bad compromise). Running them in parallel this way is otherwise "free": one domain being rate-limited doesn't hold back another, so wall-clock time is roughly bounded by the slowest domain instead of the sum of all of them — no domain gets hit any faster than its own configured rate, so no additional block risk from the parallelism itself.

The problem: each instance's AutoscaledPool/Snapshotter only observes its own CPU usage and ramps concurrency up independently, unaware that N sibling instances in the same process are doing the same — collectively over-subscribing the CPU (Cheerio parsing is CPU-bound and blocking, cf. #282). Each pool looks "healthy" in isolation, but the sum across instances chokes the event loop, causing request timeouts.

We tried tuning each instance's maxRequestsPerMinute/maxConcurrency by hand so that, combined, they stay within our development machine's CPU headroom. That "worked" there, but it's tuning to one specific machine, not a real fix — the exact same config reintroduces timeouts on a weaker machine, because the actual constraint (total CPU across all instances) isn't something any single instance's AutoscaledPool knows about.

Ideal solution or implementation, and any additional constraints

An optional shared CPU/memory snapshotting mechanism (e.g. a shared Snapshotter, or a shared Configuration multiple crawler instances can opt into) so participating instances' AutoscaledPools base scale-up/scale-down decisions on total CPU/memory usage across all of them, not just their own. This needs to stay independent from each instance's own maxRequestsPerMinute/politeness limits — only the CPU/memory autoscaling signal should be shared, not the request rate itself.

Alternative solutions or implementations

• Manually tuning each instance's limits by hand: only works for one specific machine, breaks on weaker hardware.
• Splitting instances across OS processes: only pushes the ceiling since the number of usable cores is still finite, and reintroduces the same coordination problem one level up (now across processes instead of within one).

Other context

Related: #282 — Cheerio parsing being CPU-bound/blocking is what makes the combined-load choking noticeable in the first place.

[barjin]

Hello and thank you for your interest in this project @MrGeorgen !

Please share a minimal reproduction scenario showing the issue. Since this seems to be specifically a performance issue, adding brief comments on which parts of the code you'd expect to run faster would definitely help.

Cheers!

[MrGeorgen]

Hi @barjin, thanks for your patience — I wanted to actually verify this empirically before replying, and my first attempt at a repro was flawed (a synchronous busy-loop that defeated the exact mechanism I was trying to observe), so here's a corrected one with real async I/O.

Repro setup

A separate HTTP server process (so the crawler's own load can't starve the thing serving it) that responds after an async 80ms delay with a synthetic product-listing page, plus 10 isolated CheerioCrawler instances (own Configuration/MemoryStorage each, maxConcurrency: 30) vs. a single instance with one shared AutoscaledPool handling the same total request volume (maxConcurrency: 300) as a baseline.

// server.js
const http = require('http');
const PORT = 8976, DELAY_MS = 80, ELEMENTS = 150;
let body = '';
for (let i = 0; i < ELEMENTS; i++) {
  body += `<div class="product" data-sku="SKU-${i}"><h2 class="title">Product ${i}</h2><span class="price">${(i % 999) / 100}</span></div>`;
}
const html = `<!doctype html><html><body><main>${body}</main></body></html>`;
http.createServer((req, res) => {
  setTimeout(() => { res.writeHead(200, { 'content-type': 'text/html' }); res.end(html); }, DELAY_MS);
}).listen(PORT);

// repro.js  (node repro.js multi|single)
const { monitorEventLoopDelay } = require('perf_hooks');
const { CheerioCrawler, Configuration, MemoryStorage } = require('crawlee');

const mode = process.argv[2] || 'multi';
const elDelay = monitorEventLoopDelay({ resolution: 10 });
elDelay.enable();

function extract($) {
  const out = [];
  $('.product').each((_, el) => {
    const $el = $(el);
    out.push({ sku: $el.attr('data-sku'), title: $el.find('.title').text(), price: Number($el.find('.price').text()) });
  });
  return out;
}

function makeCrawler(config, maxConcurrency) {
  return new CheerioCrawler({ maxConcurrency, requestHandler: async ({ $ }) => { extract($); } }, config);
}

async function runMulti() {
  const crawlers = Array.from({ length: 10 }, (_, c) => {
    const config = new Configuration({ persistStorage: false });
    config.useStorageClient(new MemoryStorage({ persistStorage: false }));
    return { crawler: makeCrawler(config, 30), urls: Array.from({ length: 30 }, (_, i) => `http://localhost:8976/chain-${c}/${i}`) };
  });
  setInterval(() => {
    const sum = crawlers.reduce((a, { crawler }) => a + (crawler.autoscaledPool?.currentConcurrency ?? 0), 0);
    console.log('sumConcurrency', sum, 'elDelayMeanMs', elDelay.mean / 1e6, 'elDelayMaxMs', elDelay.max / 1e6);
  }, 500).unref();
  await Promise.allSettled(crawlers.map(({ crawler, urls }) => crawler.run(urls)));
}

async function runSingle() {
  const config = new Configuration({ persistStorage: false });
  config.useStorageClient(new MemoryStorage({ persistStorage: false }));
  const crawler = makeCrawler(config, 300);
  const urls = [];
  for (let c = 0; c < 10; c++) for (let i = 0; i < 30; i++) urls.push(`http://localhost:8976/chain-${c}/${i}`);
  setInterval(() => console.log('concurrency', crawler.autoscaledPool?.currentConcurrency, 'elDelayMeanMs', elDelay.mean / 1e6, 'elDelayMaxMs', elDelay.max / 1e6), 500).unref();
  await crawler.run(urls);
}

(mode === 'single' ? runSingle() : runMulti()).then(() => {
  console.log('final elDelayMeanMs', elDelay.mean / 1e6, 'elDelayMaxMs', elDelay.max / 1e6, 'p99', elDelay.percentile(99) / 1e6);
  process.exit(0);
});

Results (300 total requests either way, 4-core dev machine, no artificial CPU throttling):

	single (1 shared pool)	multi (10 isolated pools)
wall time	5675 ms	5978 ms
event-loop delay, mean	57.7 ms	161.2 ms
event-loop delay, max	265.6 ms	1146.1 ms
event-loop delay, p99	197.1 ms	1146.1 ms
failures	0	0

Throughput is basically unaffected here — the real cost is tail latency: 4-6x worse event-loop delay for the same total workload, just because it's split across 10 independently-autoscaling pools instead of one. With a heavier per-page parse cost (I also tried a 3000-element page), this got much worse: event-loop delay maxed at 6.2s and requests started failing with socket hang up.

Enabling AutoscaledPool's own debug logging shows why: each pool reports

"eventLoopInfo":{"isOverloaded":true,"limitRatio":0.7,"actualRatio":1}
"cpuInfo":{"isOverloaded":false,"limitRatio":0.4,"actualRatio":0}

at the same moment — cpuInfo never trips because LocalEventManager.getCurrentCpuTicks() averages os.cpus() across all cores, so one JS thread saturated on a 4-core box reads as ~0-25% aggregate. Only eventLoopInfo (real, thread-wide event-loop lag) ever detects the problem, and it does — but each of the 10 pools discovers it independently, after each has already ramped to its own "healthy" concurrency (~10), so the combined total (~100) already overshot the one thread's real capacity by the time any of them reacts.

To be precise about what this does and doesn't show: it's not that the pools "ramp up in an uncoordinated race" mid-run — each one converges quickly to roughly the same steady concurrency a single coordinated pool would settle at for its share of the work. The problem is that N pools independently converging to that same per-instance-healthy number still sums to N times what the process can actually sustain, and nothing tells any one of them how many siblings are doing the same. A shared event-loop-lag/CPU signal (as originally requested) would let them collectively back off instead of each acting on a purely local view — though it wouldn't by itself fix the separate minConcurrency-per-instance floor (10 pools × minConcurrency: 1 = a combined floor of 10 no matter what), which would need something more like a shared pool or coordinated minimum.

Happy to open this as a runnable repo/gist if that's easier to poke at than inline code.