Agent Crash without any logs while trying to create Faiss VDB for kubernetes repository.

[zvigrinberg]

Input

Input Container image:

registry.redhat.io/openshift4/ose-hyperkube@sha256:8655c78e68f105ab32a4e281fd33b10ad776f68f7c5d85274e0196159bd8041e

CVE ID: CVE-2023-47108

Complete input example for re-producing:

payload-input-example.json

Description

When invoking the analysis on the above input, the Create VDB stage failing without any error log, the application is just crashing without any notification or alert...

We even freed a lot of memory in Our GPU 8x/A100 /40GB ( freed 5 out of 8 Replicas) and it still happened) so thus we eliminated that way a cause of "not enough memory"...

We were forced to change from vdb search to lexical search over code in order to be able to complete the analysis for the aforementioned input.

Seems to me like a bug in Faiss library , for this certain git repository, as we already processed and embedded bigger git repositories than that successfully.

[ashsong-nv]

Hi Zvi, thank you for raising this issue to our attention! I was able to reproduce the error. I ran the from_file.py configuration which did give me a traceback. As you suspected, the issue is during FAISS vdb creation, but from the traceback it looks like it's specifically during embedding generation. I will take a closer look and get back to you.

Traceback (most recent call last):
  File "/opt/conda/envs/morpheus-vuln-analysis/lib/python3.10/site-packages/morpheus/pipeline/pipeline.py", line 408, in post_start
    await executor.join_async()
  File "/workspace_nspect/external/vulnerability-analysis/src/cve/stages/build_vdb_stage.py", line 101, in _build_source_code_vdb_stage
    vdb_code_path, vdb_doc_path = self._build_vdb_fn(source_infos, self._ignore_code_embedding)
  File "/workspace_nspect/external/vulnerability-analysis/src/cve/utils/document_embedding.py", line 499, in build_vdbs
    vdb = self.create_vdb(source_infos=source_infos, output_path=vdb_output_dir)
  File "/workspace_nspect/external/vulnerability-analysis/src/cve/utils/document_embedding.py", line 440, in create_vdb
    db = FAISS.from_documents(chunk, self._embedding)
  File "/opt/conda/envs/morpheus-vuln-analysis/lib/python3.10/site-packages/langchain_core/vectorstores/base.py", line 843, in from_documents
    return cls.from_texts(texts, embedding, metadatas=metadatas, **kwargs)
  File "/opt/conda/envs/morpheus-vuln-analysis/lib/python3.10/site-packages/langchain_community/vectorstores/faiss.py", line 1043, in from_texts
    embeddings = embedding.embed_documents(texts)
  File "/opt/conda/envs/morpheus-vuln-analysis/lib/python3.10/site-packages/langchain_nvidia_ai_endpoints/embeddings.py", line 175, in embed_documents
    all_embeddings.extend(self._embed(batch, model_type="passage"))
  File "/opt/conda/envs/morpheus-vuln-analysis/lib/python3.10/site-packages/langchain_nvidia_ai_endpoints/embeddings.py", line 153, in _embed
    result = response.json()
  File "/opt/conda/envs/morpheus-vuln-analysis/lib/python3.10/site-packages/requests/models.py", line 978, in json
    raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)