[Feature]: Add Hybrid Crawler

[3nock]

Problem

Spider Suite already has a headless browser crawler that handles JavaScript-rendered pages well. However, in practice it is significantly slow. Spinning up and driving a real browser for each page is resource-intensive and creates a major bottleneck, especially on large sites.

The core inefficiency is that the headless browser is doing two jobs: rendering pages and crawling links. Most of the time, full browser-based crawling is unnecessary. The headless browser is only truly needed to execute JavaScript and expose the dynamically rendered DOM. The actual work of following links and fetching subsequent pages could be handled far more efficiently by the standard crawler.

There is currently no middle ground between the slow-but-thorough headless crawler and the fast-but-blind standard crawler. A hybrid approach that uses the headless browser only where it matters rendering and delegates everything else to the standard crawler would offer the best of both worlds.

Solution

Introduce a Hybrid Crawler mode that splits the crawling pipeline into two distinct, complementary stages:

Stage 1: Headless Rendering (Link Extraction Only)

• Load each page in a headless browser.
• Wait for the page's JavaScript to fully execute and the DOM to settle.
• Extract all links, navigation routes, API references, and dynamically injected hrefs from the fully rendered DOM.
• The headless browser's job ends here, it does not follow links or continue crawling. It is used purely as a rendering engine.

Stage 2: Standard Crawling (Traversal)

• Feed the extracted links from Stage 1 into Spider Suite's existing standard crawler.
• The standard crawler takes over and traverses the discovered URLs using its normal, lightweight HTTP-based engine.
• Any newly discovered pages during standard crawling that appear to be JavaScript-heavy are flagged and sent back to Stage 1 for rendering before their links are extracted.

Alternatives

No response

Additional context

No response