Add PageIndex Pipeline

[vkehfdl1]

https://github.com/VectifyAI/PageIndex

Can we add this as pipeline?
See the repository README, relevant paper, and the real implementation and finds out we can implement as the gewneration pipeline

[vkehfdl1]

AI-understood issue summary

Add a PageIndex-inspired generation pipeline to AutoRAG-Research. I interpret this conservatively as: the implementation agent should add a built-in generation pipeline that answers queries by reasoning over a document/page tree and selectively pulling page/section content, rather than adding a full ingestion system or replacing existing retrieval pipelines.

Why this issue is needed

PageIndex targets cases where vector/chunk similarity retrieval is weak for long structured documents. Its README describes a vectorless, reasoning-based RAG approach that builds a hierarchical tree index and performs tree-search retrieval before answering: PageIndex README - https://github.com/VectifyAI/PageIndex. This could broaden AutoRAG-Research beyond standard retrieve-then-generate pipelines, especially for long PDFs, reports, legal/financial docs, and section/page-grounded QA.

Why this issue may not be needed

AutoRAG-Research already has several generation pipelines that compose with existing retrieval pipelines, so PageIndex may overlap with existing “retrieve context then reason/generate” coverage unless we define a clear PageIndex-specific behavior. Relevant reusable code exists:

• autorag_research/pipelines/generation/base.py:21 defines the common BaseGenerationPipeline contract.
• autorag_research/pipelines/generation/base.py:91 requires subclasses to implement async _generate(query_id, top_k).
• autorag_research/pipelines/generation/basic_rag.py:154 shows the existing pattern: retrieve chunk IDs, fetch chunk contents, get query text, invoke the LLM, and return GenerationResult.
• autorag_research/orm/service/generation_pipeline.py:128 already handles batch execution, retries, persistence, and concurrency for generation pipelines.

Also, PageIndex’s open-source implementation includes indexing and tool-based retrieval concerns, not only final answer generation. Pulling the external package directly may introduce new dependencies (litellm, pymupdf, PyPDF2, python-dotenv) and a second LLM abstraction when this repo already standardizes on LangChain BaseLanguageModel (pyproject.toml:32).

Expected Outcome

After approval, the implementation agent should produce a reviewed, tested generation pipeline with roughly this user-facing shape:

• New pipeline module, e.g. autorag_research/pipelines/generation/pageindex.py.
• New config dataclass, e.g. PageIndexPipelineConfig(BaseGenerationPipelineConfig).
• New YAML config, e.g. configs/pipelines/generation/pageindex.yaml.
• Export from autorag_research/pipelines/generation/__init__.py.
• Tests under tests/autorag_research/pipelines/generation/test_pageindex_pipeline.py covering config, generation, metadata, no/empty context, and run/persistence.
• The pipeline should return GenerationResult text, token usage where available, and metadata that makes the reasoning path auditable (selected node/page ranges, retrieved chunk IDs, prompts/steps if safe to store).

Evidence-based implementation plan

Feasibility and evidence

In-repo reusable code found:

• autorag_research/config.py:133 defines BaseGenerationPipelineConfig; autorag_research/config.py:141 stores llm; autorag_research/config.py:142 requires retrieval_pipeline_name; autorag_research/config.py:155 provides common run kwargs.
• autorag_research/pipelines/generation/base.py:58 initializes generation pipelines with session_factory, name, llm, and retrieval_pipeline; autorag_research/pipelines/generation/base.py:138 delegates run() to the generation service.
• autorag_research/pipelines/generation/basic_rag.py:26 is the smallest config pattern; autorag_research/pipelines/generation/basic_rag.py:65 shows config-to-constructor kwargs; autorag_research/pipelines/generation/basic_rag.py:146 stores pipeline config; autorag_research/pipelines/generation/basic_rag.py:154 is the closest implementation template.
• autorag_research/orm/service/generation_pipeline.py:25 defines GenerationResult; autorag_research/orm/service/generation_pipeline.py:289 fetches chunk contents; autorag_research/orm/service/generation_pipeline.py:304 fetches query text.
• autorag_research/orm/schema_factory.py:102 defines Document; autorag_research/orm/schema_factory.py:118 defines Page; autorag_research/orm/schema_factory.py:143 defines text Chunk; autorag_research/orm/schema_factory.py:192 defines PageChunkRelation. These can support page/section grounding if the dataset ingestion populated pages and relations.
• configs/pipelines/generation/basic_rag.yaml:11 shows the built-in YAML _target_ pattern.
• tests/autorag_research/pipelines/generation/test_basic_rag_pipeline.py:15 provides a good unit/integration test structure for generation pipelines.

External sources/libraries found:

• PageIndex README - https://github.com/VectifyAI/PageIndex: describes PageIndex as vectorless, no-vector-DB/no-chunking, using a table-of-contents tree plus reasoning-based tree search.
• PageIndex: Next-Generation Vectorless, Reasoning-based RAG - https://pageindex.ai/blog/pageindex-intro: explains the framework as a tree-structured index with LLM agentic reasoning for context-aware retrieval.
• PageIndex Developer Docs - https://docs.pageindex.ai: documents Python SDK/API/MCP surfaces and cookbook workflows.
• Document Processing SDK - https://docs.pageindex.ai/sdk/tree and Retrieval SDK - https://docs.pageindex.ai/sdk/legacy/retrieval: indicate official SDK surfaces for document processing and retrieval.
• pageindex PyPI package - https://pypi.org/project/pageindex/: a Python SDK exists (pageindex, observed version 0.2.8 during research).
• VectifyAI/PageIndex client.py - https://github.com/VectifyAI/PageIndex/blob/main/pageindex/client.py: PageIndexClient.index() builds PDF/Markdown tree structures and stores document metadata/structure/page content.
• VectifyAI/PageIndex retrieve.py - https://github.com/VectifyAI/PageIndex/blob/main/pageindex/retrieve.py: provides get_document, get_document_structure, and get_page_content helpers used as agent tools.
• VectifyAI/PageIndex agentic demo - https://github.com/VectifyAI/PageIndex/blob/main/examples/agentic_vectorless_rag_demo.py: shows the intended QA loop: first inspect metadata, then structure, then fetch tight page ranges before answering.

Concrete plan for the implementation agent

1. Do not start by adding a hard dependency. First implement the AutoRAG-native pipeline against existing LangChain llm and existing DB/retrieval services. Reason: this repo already has the generation pipeline/service layer (autorag_research/pipelines/generation/base.py:21, autorag_research/orm/service/generation_pipeline.py:128) and does not currently depend on PageIndex’s litellm/PDF stack (pyproject.toml:24-46). Add pageindex only if the human explicitly approves dependency adoption.

2. Create PageIndexPipelineConfig and PageIndexPipeline following BasicRAGPipelineConfig/BasicRAGPipeline. Use the dataclass/config pattern from autorag_research/pipelines/generation/basic_rag.py:26 and autorag_research/pipelines/generation/basic_rag.py:65; subclass BaseGenerationPipeline as shown at autorag_research/pipelines/generation/basic_rag.py:77.

3. Define PageIndex-specific knobs in config. Suggested fields: max_tree_depth, max_nodes_in_prompt, max_page_ranges, page_window, tree_prompt_template, answer_prompt_template, and store_reasoning_metadata. Keep defaults conservative to avoid huge prompts. The YAML should mirror configs/pipelines/generation/basic_rag.yaml:11.

4. Build a lightweight tree view from retrieved context. Start from await self._retrieval_pipeline._retrieve_by_id(query_id, top_k) as in autorag_research/pipelines/generation/basic_rag.py:164. Fetch contents via GenerationPipelineService.get_chunk_contents() (autorag_research/orm/service/generation_pipeline.py:289). If page relations are available, enrich nodes with page/document metadata using the Page, Chunk, and PageChunkRelation schema (autorag_research/orm/schema_factory.py:118, :143, :192). If page relations are unavailable, fall back to chunk-level pseudo-nodes and state that fallback in metadata.

5. Implement an LLM tree-selection step, then final answer step. Use PageIndex’s documented workflow as the algorithmic guide: inspect structure, choose tight page/section ranges, fetch contents, then answer (see PageIndex README and the agentic demo URLs above). In AutoRAG terms, this can be two LangChain ainvoke() calls: one JSON selection call over the tree/pseudo-tree, then one answer call over selected node/page text. Follow existing async invocation and token tracking style from autorag_research/pipelines/generation/basic_rag.py:176 and :179.

6. Return auditable metadata. GenerationResult supports metadata (autorag_research/orm/service/generation_pipeline.py:25). Store selected node IDs/page ranges, fallback mode, retrieved chunk IDs/scores, and parsed selection JSON. This matters because PageIndex’s value proposition is traceability/explainability.

7. Wire exports and discovery. Add imports and __all__ entries in autorag_research/pipelines/generation/__init__.py:1 and :25. Add configs/pipelines/generation/pageindex.yaml so CLI discovery follows the existing config convention.

8. Add tests before or alongside implementation. Use tests/autorag_research/pipelines/generation/test_basic_rag_pipeline.py:15 as the template. Required tests: config creation/kwargs, _get_pipeline_config(), _generate() with mocked retrieval + mock LLM selection/final answer, empty retrieval fallback, malformed selection JSON fallback, metadata contains selection path, and run() persists results through GenerationPipelineService.run_pipeline() (autorag_research/orm/service/generation_pipeline.py:128). Also update any CLI discovery tests if they assert the exact built-in pipeline set.

9. Validation commands for the implementation agent: run the targeted new test file first, then the relevant generation/CLI tests, then make check. If Docker is available, run the full make test; otherwise document that the DB-backed full test could not run.

Risks and assumptions

• Assumption: because the issue says “implement as the generation pipeline,” the first implementation should not be a new ingestion pipeline or an external PageIndex service wrapper.
• Risk: AutoRAG data may not always contain page/document relations; the implementation should support a chunk-only fallback.
• Risk: PageIndex’s official implementation is broader than this generation pipeline: it includes PDF/Markdown indexing and tool-calling retrieval. A faithful full clone would require larger scope and likely new dependencies.
• Risk: if the desired behavior is to call the hosted PageIndex API/MCP rather than implement an AutoRAG-native pipeline, that changes credentials, dependencies, tests, and scope.

Open questions for the human (answer before /approve if these affect the desired scope)

1. Should this be AutoRAG-native and dependency-light (recommended), or should the implementation agent add/use the external pageindex PyPI package/API?
2. Should the first version support only existing text chunks, or must it require/use page/document relations when available?
3. Should this be named pageindex, page_index, or another convention in YAML/configs?

Approval reminder

Implementation starts only after a human follow-up comment containing /approve. The implementation agent will use this comment as the handoff context.

Agent Signature

Planning agent: dani issue-request reviewer for #349.