To increase the speed of this library

[takeruhukushima]

I apologize for bothering you during your busy schedule.

Today, Google introduced File search function. And I tried it with the PaperQA2 paper. It solved 6.5 out of 10 questions I create in under 6 seconds. Remarkably, this achieved the same score as when solving the same questions using PaperQA in about 40seconds.Of course, Google showed a tendency to correctly answer ambiguous questions like "What is pqa" while getting strict questions involving numbers wrong, but I'm amazed by this optimization of speed and accuracy.

So, how can we make paperqa's full-text search + similarity search via tantivy and RCS functionality via Gather evidence faster?

As a user, would probably set up something like these fast settings, but...

{
  "answer": {
    "evidence_k": 5,
    "evidence_summary_length": "25 to 50 words",
    "answer_max_sources": 3,
    "answer_length": "50 to 100 words",
    "max_concurrent_requests": 5
  },
  "parsing": {
    "use_doc_details": false
  },
  "prompts": {
    "use_json": false,
    "context_inner": "{name}: {text}"
  },
  "agent": {
    "agent_type": "fake"
  }
}

As a developer, how can I keep the speed under 10 seconds (ideally around 5 seconds) like Google does?

Is it impossible at the point of making the API call?
What is the difference between Google service and this library?
I'd like to hear a professional's detailed explanation.

[dosubot]

To get PaperQA's evidence gathering (full-text + similarity search) closer to Google's speed, focus on these areas:

• Use async methods and increase max_concurrent_requests to maximize parallel LLM calls, but stay within your API's rate limits to avoid throttling. For Gemini, set rate limits in tokens per minute, not requests per minute, and use a robust retry mechanism with exponential backoff for transient errors [thread].
• Disable multimodal parsing (parsing.multimodal: false) and set use_doc_details: false to reduce LLM and processing overhead, especially if you don't need images/tables [doc].
• Use a manifest file for document metadata to avoid slow LLM-powered metadata inference during indexing [thread].
• Tune chunk size, overlap, and batch size for your dataset. Larger chunks and higher batch sizes can speed up processing, but watch memory usage [thread].
• Use the "fake" agent type for a hard-coded, minimal call path (search → gather evidence → answer) to reduce token and LLM usage [README].
• For large datasets, consider using an external vector DB like Qdrant for faster similarity search [README].
• If you use agentic querying, set a max_timesteps to avoid agent loops and enable failover for timely answers [thread].
• Use a lightweight, fast local LLM for agent tasks and a stronger model for summarization/answering to avoid agent bottlenecks [thread].

The main bottleneck is often LLM/model API latency, especially since PaperQA calls the LLM multiple times (for summarization, re-ranking, and answer generation), while Google’s API benefits from massive infrastructure, hardware acceleration, and aggressive caching. PaperQA’s async and concurrency features help, but you may not reach Google's 5–6s latency without similar infrastructure and model speed. Still, with the above optimizations, sub-10s is realistic for many workloads [README].

If you want to go even faster, profile your pipeline to find the slowest step (retrieval, LLM, I/O), and consider precomputing or caching evidence for frequent queries.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}