Skip to content

README.md

Latest commit

 

History

History
55 lines (39 loc) · 2.42 KB

README.md

File metadata and controls

55 lines (39 loc) · 2.42 KB
Raw

RobustMVD Benchmark

RobustMVD Installation

If you have not installed with the "[all]" option, run the following command in the base directory of this repo:

pip install -e ".[rmvd]"

This will install the RobustMVD library for the benchmark.

Prepare Evaluation Data

Follow instructions by RMVD to prepare the datasets KITTI and Scannet. Place all the data under the external_benchmark_data_root_data_dir as specified in the machine config (for e.g., see any one of the configs/machine/*.yaml files).

Prepare MapAnything Checkpoint

The benchmarking system expects trained checkpoints in a specific format with a model (state_dict) key. Convert HuggingFace models to the required format:

# Convert default CC-BY-NC model
python scripts/convert_hf_to_benchmark_checkpoint.py \
    --output_path checkpoints/facebook_map-anything.pth

# Convert Apache 2.0 model for commercial use
python scripts/convert_hf_to_benchmark_checkpoint.py \
    --apache \
    --output_path checkpoints/facebook_map-anything-apache.pth

Generate Benchmark Scripts and Run the Benchmark

  1. Update machine configuration: Modify the machine variable in bash_scripts/benchmark/rmvd_mvs_benchmark/generate_benchmark_scripts.py to match your machine config name in configs/machine/.

  2. Update model checkpoint path: In the same file, update the model.pretrained entry in the get_model_settings() function to point to your converted checkpoint:

    def get_model_settings(model: str, dataset: str):
    if model == "mapanything":
        return {
            "model": "mapanything",
            "model.pretrained": "/path/to/your/converted/checkpoint.pth",  # Update this path (can be any checkpoint from training or the above generated checkpoints)
            "evaluation_resolution": "\\${dataset.resolution_options.518_1_33_ar}"
            if dataset != "kitti"
            else "\\${dataset.resolution_options.518_3_20_ar}",
        }

  3. Generate benchmark scripts: Run the script to generate all shell files:

    python bash_scripts/benchmark/rmvd_mvs_benchmark/generate_benchmark_scripts.py

  4. Run benchmarks: Execute the generated shell scripts. Each shell file corresponds to one number in the RobustMVD table of the paper. You will find the output in outputs/mapanything/benchmarking.