Releases · any4ai/AnyCrawl

Release list

v1.0.0-beta.36 Pre-release

Pre-release

QThans released this 11 Aug 18:32

v1.0.0-beta.36

689dc5b

Highlights

Template Runs (L3) — dedicated template endpoints + asynchronous orchestrated runs (revisions, config contracts, call-spec inputs/url_mode, output return_modes, OrchestratedRunner with max_concurrency, template-run worker, run-request/run-event ledgers).
Dataset platform — dedicated Dataset endpoints (L1+L2), jsonb-direct querying, async JSONL/CSV exports, run-item membership, RFC-6901 projection, crawl dataset_run finalize. Batch scrape now writes to a Dataset (parity with scrape/crawl/search).
Monitor — reliability + scheduled-execution lifecycle overhaul.
JS SDK — template client shipped in @anycrawl/js-sdk 0.0.8.

Fixed / Security

SQLite migrations: create missing scheduler/webhook tables (0020) + backfill user_id (0019) to match PostgreSQL.
Billing fail-closed invariant now throws outside production.
Template client validator blocks setTimeout/setInterval and caps code length.

Full changes: see CHANGELOG.md. Docker images: ghcr.io/any4ai/anycrawl:1.0.0-beta.36 (multi-arch).

Assets 2

v1.0.0-beta.35 Latest

Latest

QThans released this 05 Aug 18:33

v1.0.0-beta.35

c95a2a2

Added

Batch Scrape API — new asynchronous endpoint to scrape many known URLs in one job with a single shared set of scrape options (POST /v1/batch/scrape, GET /v1/batch/scrape/:jobId/status, GET /v1/batch/scrape/:jobId, DELETE /v1/batch/scrape/:jobId). Response shapes are separate from single POST /v1/scrape.
- Per-URL billing: credits charged per successfully scraped URL; failed URLs are not charged; pre-flight 402 when the account lacks estimated credits.
- Scrape template support: template_id + variables applied to every URL, with per-URL transforms, variable mapping/defaults, and domain restrictions enforced.
- Webhook events: batch_scrape.created, batch_scrape.started, batch_scrape.page, batch_scrape.completed, batch_scrape.failed, batch_scrape.cancelled.
- Configurable via ANYCRAWL_BATCH_SCRAPE_MAX_URLS (default 10000), ANYCRAWL_BATCH_SCRAPE_CONCURRENCY (default 10), ANYCRAWL_BATCH_SCRAPE_WAIT_BUFFER_MS (default 60000).
JS SDK batch scrape methods: batchScrape() (create + poll + aggregate), createBatchScrape(), getBatchScrapeStatus(), getBatchScrapeResults(), cancelBatchScrape() (published in @anycrawl/js-sdk 0.0.7).

Fixed

QueueManager: harden Redis error handling and clean up job polling to avoid leaked listeners/timers.

Assets 2

v1.0.0-beta.33

QThans released this 13 Jul 20:52

v1.0.0-beta.33

20a1798

What's Changed

merge latest updates. by @QThans in #1
Implement the File API for checking out files from S3 or local storage by @QThans in #2
✨ chore(release): update changelog for version 0.0.1-alpha.5. by @QThans in #3
merge v0.0.1-alpha.7 by @QThans in #4
Enhanced html-to-md, and improve all package.json by @QThans in #5
Dev by @QThans in #6
✨ feat(proxy): introduce advanced proxy configuration options, includ… by @QThans in #7
update docs by @QThans in #8
add AI model by @QThans in #9
enhance error handler and response, add json format by @QThans in #10
✨ fix(scrape): improve error logging in EngineConfigurator. by @QThans in #11
✨ chore: update project configuration and dependencies; modify .npmrc… by @QThans in #14
Dev by @QThans in #16
complete LLMExtract by @QThans in #18
Dev by @QThans in #27
✨ refactor(fix): reorganize exports in index.ts for clarity; updat… by @QThans in #29
Dev by @QThans in #30
Dev by @QThans in #31
✨ refactor(scrape, ai): add utility module exports in package.json; i… by @QThans in #32
Dev by @QThans in #33
✨ feat(api): prefix screenshot paths with public domain route in Craw… by @QThans in #34
Dev by @QThans in #35
🐞 fix(scrape): ensure credits are only deducted on successful scrapes… by @QThans in #36
Dev by @QThans in #37
Dev by @QThans in #38
✨ feat(api): add credit validation in CrawlController to ensure users… by @any4ai in #39
Dev by @QThans in #40
Dev by @QThans in #41
fix bug by @QThans in #42
✨ feat(scrape): enhance BaseEngine and ProgressManager to support enq… by @QThans in #44
Dev by @QThans in #45
fix by @QThans in #48
improve search by @QThans in #49
js-sdk and fixed by @any4ai in #50
update js-sdk by @QThans in #51
♻️ refactor(js-sdk): restructure crawl options to encapsulate scrape … by @QThans in #52
✨ chore(workflow): add workflow_dispatch trigger to publish-js-sdk.ym… by @QThans in #53
🔧 refactor(js-sdk): update README and type definitions to clarify scr… by @QThans in #55
fix js-sdk by @QThans in #56
release v1.0.0-beta.1 by @QThans in #62
✨ chore: update package versions to 1.0.0-beta.2 across multiple pack… by @QThans in #65
Dev by @QThans in #66
upgrade to v1.0.0-beta.4 by @QThans in #67
release v1.0.0-beta.5 by @QThans in #68
✨ feat: enhance search functionality by adding support for SearXNG an… by @any4ai in #70
Dev by @QThans in #71
✨ feat(crawl): introduce scrape_paths for selective content extract… by @QThans in #72
chore(deps): update Next.js and related dependencies to version 15.3.… by @QThans in #73
Dev by @QThans in #74
scheduled tasks & webhooks by @QThans in #75
✨ feat: enhance task scheduling by adding support for template task t… by @QThans in #76
v1.0.0-beta.12 by @QThans in #77
v1.0.0-beta.13 by @QThans in #78
new version v0.0.1-beta.19 by @QThans in #81
Dev by @QThans in #82
🔧 chore: update js-sdk publish workflow and package metadata by @QThans in #83
✨ feat: implement auto engine resolution and enhance crawl capabilities by @QThans in #84
✨ test: update default engine in AnyCrawlClient tests from "playwrigh… by @QThans in #85
✨ feat: unify configuration management across the platform by @QThans in #86
✨ feat: enhance search functionality with template support and credit… by @QThans in #87
feat: integrate Atlas Cloud provider by @lucaszhu-hue in #91
Codex/cloakbrowser browser engines by @QThans in #93

New Contributors

@QThans made their first contribution in #1
@any4ai made their first contribution in #39
@lucaszhu-hue made their first contribution in #91

Full Changelog: v1.0.0-beta.13...v1.0.0-beta.33

Contributors

QThans, any4ai, and lucaszhu-hue

Assets 2

v1.0.0-beta.32

QThans released this 30 Jun 12:25

v1.0.0-beta.32

7618d38

What's Changed

merge latest updates. by @QThans in #1
Implement the File API for checking out files from S3 or local storage by @QThans in #2
✨ chore(release): update changelog for version 0.0.1-alpha.5. by @QThans in #3
merge v0.0.1-alpha.7 by @QThans in #4
Enhanced html-to-md, and improve all package.json by @QThans in #5
Dev by @QThans in #6
✨ feat(proxy): introduce advanced proxy configuration options, includ… by @QThans in #7
update docs by @QThans in #8
add AI model by @QThans in #9
enhance error handler and response, add json format by @QThans in #10
✨ fix(scrape): improve error logging in EngineConfigurator. by @QThans in #11
✨ chore: update project configuration and dependencies; modify .npmrc… by @QThans in #14
Dev by @QThans in #16
complete LLMExtract by @QThans in #18
Dev by @QThans in #27
✨ refactor(fix): reorganize exports in index.ts for clarity; updat… by @QThans in #29
Dev by @QThans in #30
Dev by @QThans in #31
✨ refactor(scrape, ai): add utility module exports in package.json; i… by @QThans in #32
Dev by @QThans in #33
✨ feat(api): prefix screenshot paths with public domain route in Craw… by @QThans in #34
Dev by @QThans in #35
🐞 fix(scrape): ensure credits are only deducted on successful scrapes… by @QThans in #36
Dev by @QThans in #37
Dev by @QThans in #38
✨ feat(api): add credit validation in CrawlController to ensure users… by @any4ai in #39
Dev by @QThans in #40
Dev by @QThans in #41
fix bug by @QThans in #42
✨ feat(scrape): enhance BaseEngine and ProgressManager to support enq… by @QThans in #44
Dev by @QThans in #45
fix by @QThans in #48
improve search by @QThans in #49
js-sdk and fixed by @any4ai in #50
update js-sdk by @QThans in #51
♻️ refactor(js-sdk): restructure crawl options to encapsulate scrape … by @QThans in #52
✨ chore(workflow): add workflow_dispatch trigger to publish-js-sdk.ym… by @QThans in #53
🔧 refactor(js-sdk): update README and type definitions to clarify scr… by @QThans in #55
fix js-sdk by @QThans in #56
release v1.0.0-beta.1 by @QThans in #62
✨ chore: update package versions to 1.0.0-beta.2 across multiple pack… by @QThans in #65
Dev by @QThans in #66
upgrade to v1.0.0-beta.4 by @QThans in #67
release v1.0.0-beta.5 by @QThans in #68
✨ feat: enhance search functionality by adding support for SearXNG an… by @any4ai in #70
Dev by @QThans in #71
✨ feat(crawl): introduce scrape_paths for selective content extract… by @QThans in #72
chore(deps): update Next.js and related dependencies to version 15.3.… by @QThans in #73
Dev by @QThans in #74
scheduled tasks & webhooks by @QThans in #75
✨ feat: enhance task scheduling by adding support for template task t… by @QThans in #76
v1.0.0-beta.12 by @QThans in #77
v1.0.0-beta.13 by @QThans in #78
new version v0.0.1-beta.19 by @QThans in #81
Dev by @QThans in #82
🔧 chore: update js-sdk publish workflow and package metadata by @QThans in #83
✨ feat: implement auto engine resolution and enhance crawl capabilities by @QThans in #84
✨ test: update default engine in AnyCrawlClient tests from "playwrigh… by @QThans in #85
✨ feat: unify configuration management across the platform by @QThans in #86
✨ feat: enhance search functionality with template support and credit… by @QThans in #87
feat: integrate Atlas Cloud provider by @lucaszhu-hue in #91
Codex/cloakbrowser browser engines by @QThans in #93