Some doubt about metric chosen in paper

[Mobius0916]

In LoCoMo test，why choose F1 and BLEU (traditional metric) while in LongContext QA tasks, the authors choose QA accuracy?
I think F1 and BLEU may not a ideal choice.

[Cooki-e]

Thank you for your question.

For LoCoMo, our evaluation strictly follows the original dataset's metric and the metrics used by the baselines. Specifically, we report F1 as the primary metric, and additionally include F1 and BLEU-1 to ensure consistency with prior work. Among the baselines, only Mem0 adopts an LLM-as-a-judge metric.

To address this, we have added an LLM-as-a-judge evaluation script and validated GAM under the same setting. The script and results are provided in this issue: https://github.com/VectorSpaceLab/general-agentic-memory/issues/7. Under this evaluation protocol, GAM achieves state-of-the-art performance.

For RULER, we follow the official evaluation protocol defined in the original paper, which uses Accuracy as the metric. The reference paper is: https://arxiv.org/pdf/2404.06654.

We aim to ensure that all reported results are directly comparable to the original benchmarks and baselines, without introducing inconsistent or non-standard evaluation criteria.

If you have any other questions, feel free to let me know! Happy to help.

[Mobius0916]

For LoCoMo, our evaluation strictly follows the original dataset's metric and the metrics used by the baselines. Specifically, we report F1 as the primary metric, and additionally include F1 and BLEU-1 to ensure consistency with prior work. Among the baselines, only Mem0 adopts an LLM-as-a-judge metric.

To address this, we have added an LLM-as-a-judge evaluation script and validated GAM under the same setting. The script and results are provided in this issue: https://github.com/VectorSpaceLab/general-agentic-memory/issues/7. Under this evaluation protocol, GAM achieves state-of-the-art performance.

For RULER, we follow the official evaluation protocol defined in the original paper, which uses Accuracy as the metric. The reference paper is: https://arxiv.org/pdf/2404.06654.

We aim to ensure that all reported results are directly comparable to the original benchmarks and baselines, without introducing inconsistent or non-standard evaluation criteria.

If you have any other questions, feel free to let me know! Happy to help.

Thanks your reply！ I would like to ask another clarification question regarding the researcher module proposed in your paper.

Conceptually, how does your researcher module differ from existing iterative RAG approaches, such as MemSearcher (https://openreview.net/pdf?id=EWIAx3NgvA
) and Retrieve, Summarize, Plan: Advancing Multi-hop Question Answering with an Iterative Approach (https://arxiv.org/abs/2407.13101
)?

More specifically, from an algorithmic and modeling perspective, is there a fundamental difference between your researcher module and prior iterative RAG frameworks that repeatedly perform retrieval, intermediate reasoning/summarization, and refinement? Or should your researcher module be understood primarily as a particular instantiation or extension of iterative RAG, differing mainly in implementation details or task focus?

I would appreciate it if you could clarify the key conceptual distinctions (if any), especially in terms of memory management, retrieval control, and optimization objectives.

[545999961]

For LoCoMo, our evaluation strictly follows the original dataset's metric and the metrics used by the baselines. Specifically, we report F1 as the primary metric, and additionally include F1 and BLEU-1 to ensure consistency with prior work. Among the baselines, only Mem0 adopts an LLM-as-a-judge metric.
To address this, we have added an LLM-as-a-judge evaluation script and validated GAM under the same setting. The script and results are provided in this issue: https://github.com/VectorSpaceLab/general-agentic-memory/issues/7. Under this evaluation protocol, GAM achieves state-of-the-art performance.
For RULER, we follow the official evaluation protocol defined in the original paper, which uses Accuracy as the metric. The reference paper is: https://arxiv.org/pdf/2404.06654.
We aim to ensure that all reported results are directly comparable to the original benchmarks and baselines, without introducing inconsistent or non-standard evaluation criteria.
If you have any other questions, feel free to let me know! Happy to help.

Thanks your reply！ I would like to ask another clarification question regarding the researcher module proposed in your paper.

Conceptually, how does your researcher module differ from existing iterative RAG approaches, such as MemSearcher (https://openreview.net/pdf?id=EWIAx3NgvA ) and Retrieve, Summarize, Plan: Advancing Multi-hop Question Answering with an Iterative Approach (https://arxiv.org/abs/2407.13101 )?

More specifically, from an algorithmic and modeling perspective, is there a fundamental difference between your researcher module and prior iterative RAG frameworks that repeatedly perform retrieval, intermediate reasoning/summarization, and refinement? Or should your researcher module be understood primarily as a particular instantiation or extension of iterative RAG, differing mainly in implementation details or task focus?

I would appreciate it if you could clarify the key conceptual distinctions (if any), especially in terms of memory management, retrieval control, and optimization objectives.

GAM differs fundamentally from both through its Just-in-Time (JIT) Compilation architecture:

1. Vs. Standard Iterative RAG (e.g., Retrieve-Summarize-Plan):

• Memory-Guided Planning: Standard Iterative RAG typically plans retrieval based solely on the current query and intermediate results. In contrast, GAM’s Researcher utilizes the Lightweight Memory—a pre-computed "global map" generated offline—to guide its planning before executing search actions .
• Multi-Tool Retrieval: Unlike standard RAG which often relies solely on vector similarity, GAM equips the Researcher with a hybrid toolkit (Vector, BM25, and Page-ID).
• Direct Page Navigation: Uniquely, the Page-ID tool allows the Researcher to leverage the pre-constructed memory as a global index. Instead of guessing via similarity, it can pinpoint and directly retrieve specific historical pages identified in the memory, enabling precise access that semantic search alone cannot achieve.

2. Vs. MemSearcher:

• Lossless Storage vs. Rolling Compression: MemSearcher employs a "Rolling Memory" that iteratively compresses context to fit a fixed window, which is inherently lossy. GAM preserves the complete history in the Page-Store and retrieves details on demand, thus avoiding information loss.
• Decoupled Roles: MemSearcher relies on a single agent workflow to perform reasoning, acting, and memory updates sequentially. GAM decouples these roles into two distinct agents: a Memorizer (offline indexing) and a Researcher (online retrieval).
• Persistent vs. Per-Query Memory: Crucially, MemSearcher initializes an empty memory for each new question, treating memory as a transient working context for the current task. GAM maintains a persistent Page-Store and Memo, allowing the Researcher to serve multiple, distinct requests over time using a continuous, long-term history without rebuilding the index.