ragflow-optimizer: Auto-tune chunking parameters with LLM-based evaluation

[stranger00135]

Hi RAGFlow community! 👋

I built an open-source tool that automatically finds the best chunking parameters for your RAGFlow knowledge bases.

The Problem

Chunking parameters (chunk_size, overlap, auto_questions, etc.) have a huge impact on retrieval quality, but most of us pick them by feel. There's no easy way to systematically compare configurations.

The Solution

ragflow-optimizer runs automated experiments on your actual documents:

1. Phase 1 (Preset Screening): Compares chunking presets (general, manual, QA, etc.)
2. Phase 2 (Fine-tuning): Tunes parameters for the winning preset
3. LLM Evaluation: Uses an LLM to judge retrieval relevance, computing Precision@3, MRR, fail rate
4. Robustness Testing: Injects distractor documents to ensure configs work in noisy KBs

Different document types get independently optimized — HR policies might need different chunking than technical SOPs.

Quick Start

git clone https://github.com/stranger00135/ragflow-optimizer.git
cd ragflow-optimizer
pip install -r requirements.txt
cp .env.example .env  # Add your RAGFlow URL + LLM API key
python main.py run

Works with OpenAI, DeepSeek, and DashScope for evaluation.

Links

• GitHub: https://github.com/stranger00135/ragflow-optimizer
• README includes full docs in English and Chinese

Would love feedback from the community! What parameters do you find hardest to tune? What metrics matter most to you?

⭐ If you find it useful, a star would really help!

[xXMrNidaXx]

This ragflow-optimizer for auto-tuning chunking is brilliant!

Why this matters:

• Manual parameter tuning is time-consuming
• Optimal settings depend on your docs
• LLM-based evaluation catches nuances

Key innovations:

1. Automated evaluation

   • No manual labeling needed
   • LLM judges quality
   • Scales to large doc sets

2. Parameter search space

params = {
    "chunk_size": [256, 512, 1024],
    "overlap": [0, 50, 100],
    "method": ["sentence", "token", "semantic"]
}

3. Metrics tracked
   • Answer relevance
   • Faithfulness
   • Context precision
   • Retrieval recall

Extensions I'd love:

• Bayesian optimization for faster search
• Multi-objective (quality vs cost)
• Continuous monitoring in production

Integration idea:

# Auto-tune on new doc types
optimizer.tune(
    docs=new_doc_set,
    eval_queries=sample_queries,
    budget=100  # Evaluation calls
)

We build adaptive RAG at RevolutionAI. This is exactly what production systems need!

Have you tested on domain-specific corpora (legal, medical, code)?

[stranger00135]

Yeah i am using this to chunk files for the whole corporate which entails fields like finance, procurement, hr - each of which has a different document type.