Skip to content

inference_diffusion_video2world_multiview.md

Latest commit

 

History

History
140 lines (121 loc) · 7.87 KB

inference_diffusion_video2world_multiview.md

File metadata and controls

140 lines (121 loc) · 7.87 KB
Raw

Inference with diffusion-based Video2World models (with multi-view data)

Environment setup

Please refer to the Inference section of INSTALL.md for instructions on environment setup.

Download checkpoints

  1. Generate a Hugging Face access token (if you haven't done so already). Set the access token to Read permission (default is Fine-grained).

  2. Log in to Hugging Face with the access token:

    huggingface-cli login

  3. Accept the Llama-Guard-3-8B terms

  4. Download the Cosmos model weights from Hugging Face:

    CUDA_HOME=$CONDA_PREFIX PYTHONPATH=$(pwd) python scripts/download_diffusion_checkpoints.py --model_sizes 7B --model_types Video2World-Multiview --checkpoint_dir checkpoints

Examples

There are one model available for diffusion multiview world generation from text and image/video input: Cosmos-Predict1-7B-Video2World-Sample-AV-Multiview.

The inference script is cosmos_predict1/diffusion/inference/video2world_multiview.py. It requires the input argument --input_image_or_video_path (image/video input); if the prompt upsampler is disabled, --prompt (text input) must also be provided. To see the complete list of available arguments, run

CUDA_HOME=$CONDA_PREFIX PYTHONPATH=$(pwd) python cosmos_predict1/diffusion/inference/video2world_multiview.py --help

We will set the prompt with an environment variable first.

PROMPT="The video is captured from a camera mounted on a car. The camera is facing forward. \
The video is taken from the perspective of a vehicle's dashboard camera, showing a straight road flanked by snow-covered trees and a clear sky. \
The road is mostly empty, with no visible traffic or pedestrians. \
The sun is setting, casting a warm glow on the horizon and creating long shadows on the snow. \
The trees are tall and leafless, with some coniferous trees interspersed among the bare deciduous trees. \
The snow on the ground appears undisturbed, suggesting a quiet and peaceful setting."

PROMPT_LEFT="The video is captured from a camera mounted on a car. The camera is facing to the left. \
The video captures a series of images from a moving vehicle, showcasing a winter scene with snow-covered ground and trees. \
The sky is a gradient of blue and orange hues, indicating either sunrise or sunset. \
The trees are tall and predominantly coniferous, with some deciduous trees as well. \
The snow appears undisturbed, suggesting a quiet, possibly early morning setting. \
There are no visible people or animals, and the road is clear of traffic. \
The video has a fisheye lens effect, which gives a wide-angle view of the surroundings."

PROMPT_RIGHT="The video is captured from a camera mounted on a car. The camera is facing to the right. \
The video captures a series of images taken from a moving vehicle, showcasing a winter scene with snow-covered ground and trees. \
The sky is a gradient of blue hues, indicating either dawn or dusk. \
The trees are predominantly coniferous, with some bare deciduous trees. \
The snow appears fresh and undisturbed, suggesting recent snowfall. \
There are no visible people or animals, and the environment is serene and untouched. \
The perspective changes as the vehicle moves, providing different angles of the same landscape."

PROMPT_BACK="The video is captured from a camera mounted on a car. The camera is facing backwards. \
The video captures a sequence of frames showing a road covered in snow, with tire tracks visible on the surface. \
The road is flanked by tall, leafless trees, and the sky is a gradient of pink and blue hues, indicating either sunrise or sunset. \
The lighting conditions suggest it is either early morning or late evening. \
There are no visible signs of people or animals, and the road appears to be in a rural or less populated area. \
The vehicles in the video are moving at a steady pace, and there are no visible traffic signs or markings that stand out."

PROMPT_BACK_LEFT="The video is captured from a camera mounted on a car. The camera is facing the rear left side."

PROMPT_BACK_RIGHT="The video is captured from a camera mounted on a car. The camera is facing the rear right side."

Example 1: single generation

This is the basic example for running inference on the 7B model with a single image. No text prompts are provided here.

CUDA_HOME=$CONDA_PREFIX PYTHONPATH=$(pwd) python cosmos_predict1/diffusion/inference/video2world_multiview.py \
    --checkpoint_dir checkpoints \
    --diffusion_transformer_dir Cosmos-Predict1-7B-Video2World-Sample-AV-Multiview \
    --input_image_or_video_path assets/diffusion/video2world_multiview_input1.mp4 \
    --num_input_frames 1 \
    --prompt "${PROMPT}" \
    --prompt_left "${PROMPT_LEFT}" \
    --prompt_right "${PROMPT_RIGHT}" \
    --prompt_back "${PROMPT_BACK}" \
    --prompt_back_left "${PROMPT_BACK_LEFT}" \
    --prompt_back_right "${PROMPT_BACK_RIGHT}" \
    --video_save_name diffusion-video2world-multiview-7b

Example 2: single generation with model offloading

We run inference with offloading flags enabled. This is suitable for low-memory GPUs.

CUDA_HOME=$CONDA_PREFIX PYTHONPATH=$(pwd) python cosmos_predict1/diffusion/inference/video2world_multiview.py \
    --checkpoint_dir checkpoints \
    --diffusion_transformer_dir Cosmos-Predict1-14B-Video2World_multiview \
    --input_image_or_video_path assets/diffusion/video2world_multiview_input1.mp4 \
    --num_input_frames 1 \
    --prompt "${PROMPT}" \
    --prompt_left "${PROMPT_LEFT}" \
    --prompt_right "${PROMPT_RIGHT}" \
    --prompt_back "${PROMPT_BACK}" \
    --prompt_back_left "${PROMPT_BACK_LEFT}" \
    --prompt_back_right "${PROMPT_BACK_RIGHT}" \
    --offload_tokenizer \
    --offload_diffusion_transformer \
    --offload_text_encoder_model \
    --video_save_name diffusion-video2world-multiview-7b

Example 3: single generation with multi-GPU inference

This example runs parallelized inference on a single prompt using 8 GPUs.

NUM_GPUS=8
CUDA_HOME=$CONDA_PREFIX PYTHONPATH=$(pwd) torchrun --nproc_per_node=${NUM_GPUS} cosmos_predict1/diffusion/inference/video2world_multiview.py \
    --num_gpus ${NUM_GPUS} \
    --checkpoint_dir checkpoints \
    --diffusion_transformer_dir Cosmos-Predict1-7B-Video2World-Sample-AV-Multiview \
    --input_image_or_video_path assets/diffusion/video2world_multiview_input1.mp4 \
    --num_input_frames 1 \
    --prompt "${PROMPT}" \
    --prompt_left "${PROMPT_LEFT}" \
    --prompt_right "${PROMPT_RIGHT}" \
    --prompt_back "${PROMPT_BACK}" \
    --prompt_back_left "${PROMPT_BACK_LEFT}" \
    --prompt_back_right "${PROMPT_BACK_RIGHT}" \
    --video_save_name diffusion-video2world-multiview-7b-8gpu

Example 4: batch generation

This example runs inference on a batch of prompts, provided through the --batch_input_path argument (path to a JSONL file).

{"prompt": "prompt1", "prompt_left": "prompt1_left", "prompt_right": "prompt1_right", "prompt_back": "prompt1_back", "prompt_back_left": "prompt1_back_left", "prompt_back_right": "prompt1_back_right", "visual_input": "path/to/video1.mp4"}
{"prompt": "prompt2", "prompt_left": "prompt2_left", "prompt_right": "prompt2_right", "prompt_back": "prompt2_back", "prompt_back_left": "prompt2_back_left", "prompt_back_right": "prompt2_back_right", "visual_input": "path/to/video2.mp4"}

Inference command (with 9 input frames):

CUDA_HOME=$CONDA_PREFIX PYTHONPATH=$(pwd) python cosmos_predict1/diffusion/inference/video2world_multiview.py \
    --checkpoint_dir checkpoints \
    --diffusion_transformer_dir Cosmos-Predict1-7B-Video2World-Sample-AV-Multiview \
    --batch_input_path assets/diffusion/batch_inputs/video2world_multiview.jsonl \
    --num_input_frames 9 \
    --video_save_folder diffusion-video2world-multiview-7b-batch