FSCrawler 3.0 🌈
#2521
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
{{ message }}
Uh oh!
There was an error while loading. Please reload this page.
The FSCrawler team is pleased to announce the FSCrawler 3.0 release!
FSCrawler is a crawler for Elasticsearch that helps to index binary documents such as PDF, MS Office, and more.
Usage
Download FSCrawler 3.0:
wget https://repo1.maven.org/maven2/fr/pilato/elasticsearch/crawler/fscrawler-distribution/3.0/fscrawler-distribution-3.0.zip unzip fscrawler-distribution-3.0.zip cd fscrawler-distribution-3.0On first run, create the default job configuration:
Create a directory such as
/tmp/es, add files to index, then start FSCrawler:Or with Docker:
docker run -it --rm \ -v ~/.fscrawler:/root/.fscrawler \ -v ~/tmp:/tmp/es:ro \ dadoonet/fscrawler:3.0On first run with Docker, add
--setupto create the configuration.More details in the documentation.
Version 3.0
There is no in-place upgrade from FSCrawler 2.9. Install 3.0, recreate jobs with
--setup, and reindex. See {ref}upgrade-from-2.9.Breaking changes
For example, to exclude the folder
/tmp/foo, you need to use/tmp/foo/*. Thanks to dadoonet.So running
docker run -it -v ~/.fscrawler:/root/.fscrawler -v /documents:/tmp/es:ro dadoonet/fscrawler job_nameisenough. Thanks to dadoonet.
You need to use the
--listoption to list the jobs. Thanks to dadoonet.configuration folder with default settings. You need to use the
--setupoption to create the job settings.Thanks to dadoonet.
elasticsearch.nodes.urlsetting. You need to useelasticsearch.urlsinstead. Thanks to dadoonet.
_uploadREST endpoint has been removed. Please now use the_documentendpoint. Thanks to dadoonet.meta.raw.*metadata key names compared to FSCrawler 2.9.If you search, aggregate or map these fields, you will need to update your queries and index templates
accordingly:
img:. For exampleNumber of Tablesbecomesimg:Number of TablesandExif IFD0:Orientationbecomesimg:Exif IFD0:Orientation.icc:prefix instead ofICC:.access-permission:*andpdf:*keys now use hyphens instead of underscores or camelCase.For example
access_permission:assemble_documentbecomesaccess-permission:assemble-documentand
pdf:PDFVersionbecomespdf:pdf-version.resourceNameis renamed totk:resource-name.tk:prefix instead ofX-TIKA:. New keys includetk:content-type-magic-detected,tk:parsed-by-full-setand, for text documents, the encodingdetection keys
tk:detected-encoding,tk:encoding-detection-traceandtk:encoding-detector.Thanks to dadoonet.
fs.ocr.pdf_strategyis nowautoinstead ofocr_and_text. Withauto, OCR is skipped onPDF pages that already contain more than 10 characters of text. If you relied on the previous behaviour and want
OCR on every PDF page, explicitly set
fs.ocr.pdf_strategytoocr_and_text. Thanks to dadoonet.--setupsetfs.hash_algorithmtoSHA-256in the example settings. Existing jobs thatomit the setting keep
MD5so document_ids stay unchanged. Changing the algorithm later requires a fullreindex. See document ids. Closes #2425. Thanks to dadoonet.
New
fs.excludesnow also skips macOS Finder metadata files (.DS_Store), via thecase-insensitive pattern
*/.ds_store, in addition to*/~*. See includes_excludes.Thanks to dadoonet.
fs.providerinstead ofserver.protocol. Available providers arelocal(default),ftp, andssh.See crawler provider. Thanks to dadoonet.
next_checkfield tonullin the~/.fscrawler/{job_name}/_checkpoint.jsonfile and FSCrawler will starta new scan immediately.
jobs using multiple files in the
~/.fscrawler/job/_settingsdirectory. Also note that the system propertiesneed to be set in the
FS_JAVA_OPTSenvironment variable.license. See semantic_search. Warning: this might slow down the ingestion process. Thanks to dadoonet.
_document, you can now fetch a document from the local dir, from an http websiteor from an S3 bucket. See rest service. Thanks to dadoonet.
_documentendpoint. See rest service. Thanks to dadoonet.tika-vlmmodule: OCR can be delegated to a Vision LanguageModel through an OpenAI-compatible endpoint (vLLM, Ollama, Azure OpenAI…), Anthropic Claude or Google
Gemini, configured via a custom Tika configuration file. See vlm ocr. The default FSCrawler
parser chain is unchanged (Tesseract when available). Thanks to dadoonet.
XML implementation and to betofilippi for the switch to JSON.
If your JSON configuration uses
default-parser, exclude the VLM parser components unless youexplicitly enable one — see local fs settings and vlm ocr.
Note: since the Apache Tika 4 upgrade, the configuration file must be a Tika JSON configuration —
the XML-based configuration file mechanism was removed upstream. Existing XML configurations need to be
converted. See local fs settings.
externaldirectory from where jars can be loadedto the FSCrawler JVM. For example, you could provide your own Custom Tika Parser code. See layout. Thanks to dadoonet.
.meta.yml. See tags Thanks to dadoonet.fscrawlerby default. Thanks to dadoonet.without losing progress. It also automatically recovers from network errors with exponential backoff retry.
See rest service. Thanks to dadoonet.
elasticsearch.retry_max_duration,elasticsearch.retry_initial_delayandelasticsearch.retry_max_delay(defaults
5m/500ms/30s). The same budget applies to5xx,429, and acold-start
404onGET /. See http retry settings. Thanks to dadoonet.See kibana settings. Closes #2477. Thanks to dadoonet.
Fix
.key) files are now supported for content extraction and indexing. Closes #782.fs.ocr.enabledwas always false. Thanks to ywjung.d. Thanks to dadoonet._bulkHTTP calls now retry on429/5xxand no longer treat a failed bulk as success.Exhausted retries mark the crawl checkpoint as
ERRORand REST uploads returnok: false.Thanks to dadoonet.
org.apache.pdfboxtoerrorto avoid flooding logs withNo Unicode mapping for CID+…warnings from subset fonts. Thanks to dadoonet.org.apache.fontboxtoerrorto avoid flooding logs withNo PostScript name data is provided for the font …warnings. Thanks to dadoonet.logs/bulk-failures.log(reason-prefixed lines; truncatedpayloads at TRACE). Console /
fscrawler.logpoint to that file. Thanks to dadoonet.Deprecated
server.protocolsetting is deprecated. Usefs.providerinstead. Thanks to dadoonet.Updated
fscrawleranymore. Thanks to dadoonet./instead of/fscrawler/. Thanks to dadoonet.fields in alphabetical order. This affects the indexed documents, the generated
_settings.yamland_checkpoint.jsonfiles, and the REST API responses. This is purely cosmetic (field order is not significant inJSON), but users who version their configuration files may notice a one-time reordering. Thanks to dadoonet.
Removed
Thanks to
@dadoonet,@ywjung,@iadcode,@bdauvissat,@alexbluesteele,@betofilippi,@newschapmj1and all the contributors for this release!
What's Changed
fs.ocr.enabledis always false by @ywjung infs.ocr.enabledis always false #1358fileinstead ofFilefor facets by @dadoonet in Usefileinstead ofFilefor facets #1552--debugand--traceby @dadoonet in Deprecate--debugand--trace#1780elasticsearch.byte_sizeby @dadoonet in Supportelasticsearch.byte_size#1852filesizeis not provided by curl by @dadoonet infilesizeis not provided by curl #1871mvn testby @dadoonet in Unit tests not picked up bymvn test#1928build_flavorcheck. by @dadoonet in https://github.com/dadoonet/fscrawler/pull/2075next_checkto_status.jsonby @dadoonet in https://github.com/dadoonet/fscrawler/pull/2106path.*fields by @dadoonet in https://github.com/dadoonet/fscrawler/pull/2198deployphase by @dadoonet in https://github.com/dadoonet/fscrawler/pull/2203elasticsearch.nodes.urlsetting byelasticsearch.urlsby @dadoonet in https://github.com/dadoonet/fscrawler/pull/2206ThisDirHasSpaceAtEndby @dadoonet in https://github.com/dadoonet/fscrawler/pull/2226_iddeduplication in Tips and tricks by @dadoonet in https://github.com/dadoonet/fscrawler/pull/2461New Contributors
fs.ocr.enabledis always false #1358Full Changelog: fscrawler-2.9...fscrawler-3.0
This discussion was created from the release FSCrawler 3.0 🌈.
All reactions