✨ A Kawaii Web Crawler with Real-Time Visualization ✨
A real-time web crawler with a Miku-themed UI and live visualization.
Watch pages get crawled in real-time, inspect essential page details, and export stored pages
— all wrapped in a cute interface.
🔄 Live SSE streaming · 📊 Content analysis · 💾 Persistent storage · 🎨 Miku-themed UI
Inspired by MikuMikuBeam by Sammwy 💕
| ||||||||||||||||||||
|
Every crawled page is reduced to the data used by crawling, recovery, search, and export:
| ||||||||||||||||||||
|
Requires Bun 1.3.14 — the repository's declared runtime.
git clone https://github.com/renbkna/mikumikucrawler
cd mikumikucrawler
bun install
bunx playwright install chromium
bun run devPlaywright's managed Chromium is preferred. If it is not installed, local development can
use chromium, chromium-browser, google-chrome, or google-chrome-stable from PATH.
On Linux systems missing browser libraries, run bunx playwright install --with-deps chromium.
| Service | URL | |
|---|---|---|
| 🎨 | Frontend | http://localhost:5173 |
| ⚙️ | Backend | http://localhost:3000 |
| 📋 | OpenAPI (development UI) | http://localhost:3000/openapi |
By default, the backend owns port 3000 exclusively and fails clearly if another process already owns it.
The checked-in default is owned by shared/deploymentDefaults.ts; the documented URLs are projections of that value.
Local Vite development proxies /api to PORT, so changing PORT needs no duplicate frontend setting.
Set VITE_BACKEND_URL only when the browser must connect directly to a separate backend origin.
It accepts an absolute HTTP(S) base URL (including a path prefix), without credentials, a query, or a fragment.
🔧 Environment Variables
Copy .env.example → .env. All variables have sensible defaults. Frontend vars need the VITE_ prefix.
PORT=3000
NODE_ENV=development
FRONTEND_URL=http://localhost:5173
# Optional direct-browser backend override; local Vite development uses PORT.
# VITE_BACKEND_URL=https://api.example.com
DB_PATH=./data/crawler.db
# SQLite allocation budget. Admission reserves an 8 MiB safety allowance per remaining page.
MAX_STORAGE_MB=2048
LOG_LEVEL=info
USER_AGENT=MikuCrawler/3.0.0
ROBOTS_PRODUCT_TOKEN=MikuCrawler
RENDER=false
# On Render this also trusts the platform's client-IP forwarding for rate limits.
# Browser rendering is skipped when process RSS exceeds this many MB.
# Defaults to 350 on Render and 600 elsewhere.One process exclusively owns DB_PATH while it is running. New and resumed
crawls reserve an 8 MiB safety allowance for each remaining page. When necessary,
the storage owner removes the oldest completed, stopped, or failed runs first;
active, paused, and interrupted checkpoints are never reclaimed automatically.
server/storage/schema.sql is the only supported schema. A database whose
schema differs is replaced at startup; releases do not migrate or preserve
incompatible stored data.
⚙️ Crawler Options
| Setting | Default | Range |
|---|---|---|
| Crawl Depth | 2 |
1–5 |
| Max Pages | 50 |
1–200 |
| Max Pages Per Domain | 0 |
0–1000 (0 = unlimited) |
| Page Crawl Delay | 1000ms |
200–10000ms |
| Method | full |
links / media / full |
| Concurrent Page Jobs | 5 |
1–10 |
| Retry Limit | 3 |
0–5 |
| Dynamic Content | true |
— |
| Respect Robots | true |
— |
| Content Only | false |
— |
Count Media (saveMedia) |
false |
— |
Dynamic pages use a separate fixed subrequest policy: at most four concurrent subrequests, at least 50 ms between same-host dispatches, and at most 100 requests or 20 MiB of response bodies per page.
The OpenAPI JSON specification is always available at
/openapi/json; the interactive/openapiUI is development-only.
| Method | Endpoint | Description | |
|---|---|---|---|
| 🆕 | POST |
/api/crawls |
Create a crawl run |
| 📋 | GET |
/api/crawls |
List crawl runs |
| ♻️ | GET |
/api/crawls/resumable?limit=25 |
List paused/interrupted runs (limit defaults to 25, maximum 100) |
| 🔍 | GET |
/api/crawls/:id |
Get crawl state & counters |
| ♻️ | GET |
/api/crawls/:id/snapshot |
Recover crawl state, latest stored pages, and total stored count in one response |
| ⏹️ | POST |
/api/crawls/:id/stop |
Request pause or force stop |
POST |
/api/crawls/:id/resume |
Resume a paused or interrupted crawl | |
| 📡 | GET |
/api/crawls/:id/events |
SSE event stream |
| 📄 | GET |
/api/crawls/:id/pages |
List the latest stored page summaries and total stored count |
| 📦 | GET |
/api/crawls/:id/export |
Export pages (JSON / CSV) |
| 🗑️ | DELETE |
/api/crawls/:id |
Delete a stored crawl |
| 📄 | GET |
/api/crawls/:id/pages/:pageId/content |
Fetch crawl-owned stored page content |
| 🔎 | GET |
/api/search?crawlId=:id&q=keyword |
Search one crawl's stored pages (FTS5) |
| 💚 | GET |
/health |
Health check |
const source = new EventSource(
"http://localhost:3000/api/crawls/<crawl-id>/events"
);
source.addEventListener("crawl.progress", (event) => {
const { sequence, payload } = JSON.parse(event.data);
console.log(payload.counters);
});| Event | When |
|---|---|
crawl.started |
Crawl begins processing |
crawl.progress |
Counter & queue stats update |
crawl.page |
A page was persisted, with its positive row ID and post-commit stored-page count |
crawl.log |
Runtime log message with explicit severity |
crawl.completed |
Crawl finished normally |
crawl.paused |
Paused by user and available to resume |
crawl.stopped |
Stopped by user |
crawl.failed |
Terminated due to error |
Events are sequenced. Last-Event-ID replays recent in-memory events; after a
restart or cleanup, recover from the backend-owned crawl snapshot, which contains
the persisted crawl summary, bounded latest-page window, and total stored count.
Settled streams close after their terminal frame; reconnects with no unseen
terminal event receive 204 so native EventSource clients stop reconnecting.
Search and export cover the full stored set.
|
|
📁 Project Structure
server/
├── api/ # Elysia route handlers
├── contracts/ # OpenAPI schemas + shared type re-exports
├── domain/crawl/ # Core crawl logic
│ ├── CrawlQueue.ts # Durable FIFO/delayed queue
│ ├── CrawlState.ts # Counters, visited URLs, stop logic
│ ├── DynamicRenderer.ts # Playwright lifecycle
│ ├── FetchService.ts # HTTP fetching with security checks
│ ├── PagePipeline.ts # Fetch → process → store pipeline
│ ├── RobotsService.ts # robots.txt evaluation
│ └── UrlPolicy.ts # URL filtering and normalization
├── runtime/ # Crawl execution layer
│ ├── CrawlRuntime.ts # Orchestrates a single crawl run
│ ├── CrawlManager.ts # Creates, stops, resumes, lists runs
│ └── EventStream.ts # Sequenced bounded live SSE publishing
├── processors/ # Content analysis
│ ├── ContentProcessor.ts # Dispatch by content type
│ ├── analysisUtils.ts # Word count, reading time, language
│ └── extractionUtils.ts # Main content, metadata, media count, links
├── storage/ # SQLite persistence
│ ├── schema.sql # Current schema; incompatible databases reset
│ └── repos/ # Query repositories
├── outbound/ # SSRF-safe DNS resolution and pinned HTTP
├── plugins/ # Elysia plugins (SSE, OpenAPI, static)
└── config/ # Env validation, logging setup
shared/ # Cross-boundary contracts and policy
├── contracts/ # Domain types (status, events, pages)
├── crawl.ts # Crawl option bounds
├── deploymentDefaults.ts # Deployment defaults
├── ipPolicy.ts # Public-address policy
├── text.ts # Text/byte conversion helpers
└── url.ts # URL validation & normalization
graph TD
A[🌐 Target URL] --> B[🎵 CrawlRuntime]
B --> C[📄 PagePipeline]
C --> D{Dynamic?}
D -->|Yes| E[🎭 Playwright]
D -->|No| F[⚡ Fetch + Cheerio]
E --> G[📊 ContentProcessor]
F --> G
G --> C
C --> B
B --> H[💾 SQLite]
B --> I[📡 EventStream]
I --> J[🎨 React UI]
- Client creates a crawl via
POST /api/crawls - CrawlManager spawns a CrawlRuntime with its own queue and state
- PagePipeline fetches each URL via FetchService (static) or Playwright (dynamic)
- ContentProcessor analyzes the page and PagePipeline admits discovered links
- CrawlRuntime commits each terminal result and its counters atomically, then EventStream publishes sequenced events ✨
bun run build
NODE_ENV=production \
FRONTEND_URL=https://crawler.example.com \
DB_PATH=./data/crawler.db \
bun startDevelopment mode permits localhost crawl targets and therefore binds only to
127.0.0.1. Production mode denies localhost targets and binds to 0.0.0.0.
Set FRONTEND_URL to the browser-visible production origin.
Render's public load balancer owns Brotli/gzip response compression. Any direct
self-host—including bun start and the container below—emits uncompressed
responses unless a compression-capable reverse proxy is placed in front.
🐳 Docker
docker build -t mikumikucrawler .
docker run --rm --init --ipc=host \
--security-opt seccomp=seccomp_profile.json \
-p 3000:3000 \
-v mikumikucrawler-data:/app/data \
-e FRONTEND_URL=http://localhost:3000 \
mikumikucrawlerThe named volume owns SQLite state across container replacement. The final image
runs as Playwright's unprivileged pwuser, explicitly enables Chromium's
sandbox, and uses the checked-in seccomp profile required for user namespaces.
That profile derives from Moby's seccomp/v0.2.1 default, adds Playwright's
clone/setns/unshare user-namespace allowance, and keeps socketcall
blocked. Its SHA-256 is
dfea086789bff2999aab1f950ffa6e50cf3d38492a3c7476aee637780e42c75b.
Both the Bun build image and Playwright runtime/browser image are pinned by OCI digest. Update those digests, the Playwright dependency, and the seccomp profile together as one reviewed browser-runtime migration.
VITE_BACKEND_URL is a frontend build-time setting, not a container runtime
variable. Same-origin deployments should omit it. To build a browser bundle
that talks directly to a separate backend, use:
docker build \
--build-arg VITE_BACKEND_URL=https://api.example.com \
-t mikumikucrawler .NODE_ENV=production
PORT=3000
FRONTEND_URL=https://your-domain.com
DB_PATH=/app/data/crawler.db
MAX_STORAGE_MB=2048bun run checkTypecheck (tsgo) → Format check and lint (biome ci) → Tests → Build
| ✅ | Get permission before crawling |
| ✅ | Respect robots.txt and rate limits |
| ✅ | Use reasonable delays |
| ❌ | Don't overload servers |
| ❌ | Don't scrape copyrighted content without authorization |
- Fork the repo
- Create a feature branch:
git checkout -b my-feature - Commit changes:
git commit -m 'Add feature' - Push:
git push origin my-feature - Open a Pull Request
