Inquiry regarding Visualization Tools for Loudness Maps and Videos

[nahyunKim2221]

Hello,

First of all, thank you for your great work.

I am currently reviewing the results of your model. Since the outputs are in .npy and image formats, I am having some difficulty replicating the high-level visualizations. Could you provide guidance on the following?

• How to create the loudness map visualization (Figure 5 in the paper).
• The process for generating demo videos shown on the project page.

Any scripts or documentation you could share would be extremely helpful.
Thanks!

[AmandineBtto]

Hello,

Thank you for the kind words and your interest in NeRAF.
I’ve added a few examples to the "viz" folder to illustrate how the loudness map and video are generated. I hope this helps.

[nahyunKim2221]

Hello, Thanks for adding code. I'm currently testing it, but the model outputs don't look right. Specifically, the RGB outputs are completely grey, and the depth maps appear to contain random values. However, the trajectory plots look normal, suggesting the camera poses are being loaded correctly.

I am using the RAF "EmptyRoom" dataset and RAF/EmptyRoom_NeRAF/NeRAF/2024-10-10_041356/nerfstudio_models/step-000090000.ckpt checkpoint. When I run ns-render command, the message said "✅ Done loading checkpoint from /mypath/RAF/EmptyRoom_NeRAF/NeRAF/2024-10-10_041356/nerfstudio_models/step-000090000.ckpt". Is this a known issue, or could there be something I missed in the configuration?

My environment is as follows:

• Python: 3.8.10
• PyTorch: 2.1.2
• nerfstudio: 1.0.2
• librosa 0.10.1
• pyroomacoustics 0.7.3
• scipy 1.10.1

[AmandineBtto]

Hello,

This is not a known issue on my side: I tested the provided notebook using the same checkpoint, and everything worked as expected.
From your description, it looks like the problem may be related to Nerfstudio rather than the checkpoint itself. I verified that I’m using the same Nerfstudio version as you, so it’s unlikely to be a version mismatch.

To help narrow this down, could you please check the following:

• Before running ns-render, does the scene display correctly in ns-viewer? You should be able to see a reconstruction along with the training camera poses.
• Did you update the config.yml file that comes with the weights to point to your actual dataset path and output directory?

If I remember correctly, for both the viewer and rendering steps, it’s not only the checkpoint that matters, Nerfstudio also needs access to the training images and camera poses referenced in the config.

[nahyunKim2221]

Following your advice, I checked the ns-viewer. I found that the reconstruction does not appear according to the training camera poses; the screen remains entirely grey.

To investigate further, I added a print statement for the RGB tensor values in NeRF_model.py.

class NeRAFVisionModel(NerfactoModel):
    def get_outputs_for_camera(self, camera, obb_box, eval=False):
        if not eval: 
            output_audio = self.audio_model.get_outputs_for_camera(camera, obb_box)
        output_vision = super().get_outputs_for_camera(camera, obb_box)
        output_vision["rgb"] = torch.clip(output_vision["rgb"], 0, 1)        
        rgb = output_vision['rgb'] # added code for debugging
        print(f"out rgb: {rgb}")
        if not eval:
            output = {**output_audio, **output_vision}
        else: 
            output = output_vision
        return output

I confirmed that the values are consistently around 0.5, which explains the grey output. Here is sample print output:

out rgb: tensor([[[0.5209, 0.5039, 0.4684],
         [0.5209, 0.5039, 0.4684],
         [0.5209, 0.5040, 0.4684],
         ...,
         [0.5217, 0.5049, 0.4688],
         [0.5217, 0.5049, 0.4688],
         [0.5217, 0.5049, 0.4688]],

        [[0.5209, 0.5039, 0.4684],
         [0.5209, 0.5039, 0.4684],
         [0.5209, 0.5039, 0.4685],
         ...,
         [0.5217, 0.5049, 0.4688],
         [0.5217, 0.5049, 0.4688],
         [0.5217, 0.5049, 0.4688]],

Additionally, I updated the path in config.yml to match my checkpoint path. Since the message "✅ Done loading checkpoint" appeared in the logs, I assume the checkpoint is being loaded correctly without any issues.