Captions and Preview

Prepare inputs, generate captions, optionally enhance them, and produce preview images.

Choosing a Captioning Model

The video captioning pipeline supports two model families. Pick a variant based on quality, GPU memory, and throughput:

VariantModelDefault Use Case
qwenQwen/Qwen2.5-VL-7B-InstructDefault — good quality/throughput balance
nemotron / nemotron-bf16Nemotron Nano 12B v2 VL (BF16)High-quality captions; auto-downloaded from Hugging Face
nemotron-fp8Nemotron Nano 12B v2 VL (FP8)Same model, FP8-quantized for lower memory
nemotron-nvfp4Nemotron Nano 12B v2 VL (NVFP4-QAD)NVFP4 quantization-aware-distilled checkpoint

Caption enhancement (the optional second-pass LLM rewrite) uses Qwen-LM (--enhance-captions-algorithm qwen_lm).

Quickstart

Use the pipeline stages or the example script flags to prepare captions and preview images.

Preparation and previews

  1. Prepare caption inputs from each clip window. This step splits clips into fixed windows, formats model‑ready inputs for the chosen VLM (Qwen‑VL or Nemotron), and optionally stores per‑window mp4 bytes for previews.

    from nemo_curator.stages.video.caption.caption_preparation import CaptionPreparationStage
from nemo_curator.stages.video.preview.preview import PreviewStage

prep = CaptionPreparationStage(
    model_variant="qwen",  # or "nemotron" / "nemotron-fp8" / ...
    prompt_variant="default",
    prompt_text=None,
    sampling_fps=2.0,
    window_size=256,
    remainder_threshold=128,
    preprocess_dtype="float16",
    model_does_preprocess=False,
    generate_previews=True,
    verbose=True,
)

  2. Optionally generate .webp previews from each window’s mp4 bytes for quick QA and review.

    preview = PreviewStage(
    target_fps=1.0,
    target_height=240,
    verbose=True,
)

Parameters

Caption generation and enhancement

  1. Generate window‑level captions with the chosen VLM (Qwen‑VL or Nemotron). This stage reads clip.windows[*].qwen_llm_input (created earlier) and writes window.caption["qwen"] (or window.caption["nemotron"], depending on the variant).

    from nemo_curator.stages.video.caption.caption_generation import CaptionGenerationStage
from nemo_curator.stages.video.caption.caption_enhancement import CaptionEnhancementStage

gen = CaptionGenerationStage(
    model_dir="/models",
    model_variant="qwen",  # or "nemotron" / "nemotron-fp8" / ...
    caption_batch_size=8,
    fp8=False,
    max_output_tokens=512,
    model_does_preprocess=False,
    generate_stage2_caption=False,
    stage2_prompt_text=None,
    disable_mmcache=True,
)

  2. Optionally enhance captions with a text‑based LLM (Qwen‑LM) to expand and refine descriptions. This stage reads window.caption["qwen"] and writes window.enhanced_caption["qwen_lm"].

    enh = CaptionEnhancementStage(
    model_dir="/models",
    model_variant="qwen",
    prompt_variant="default",
    prompt_text=None,
    model_batch_size=128,
    fp8=False,
    max_output_tokens=512,
    verbose=True,
)

Parameters

Preview Generation

Generate lightweight .webp previews for each caption window to support review and QA workflows. A dedicated PreviewStage reads per-window mp4 bytes and encodes WebP using ffmpeg.

Preview Parameters

  • target_fps (default 1.0): Target frames per second for preview generation.
  • target_height (default 240): Output height. Width auto-scales to preserve aspect ratio.
  • compression_level (range 0–6, default 6): WebP compression level. 0 is lossless; higher values reduce size with lower quality.
  • quality (range 0–100, default 50): WebP quality. Higher values increase quality and size.
  • num_cpus_per_worker (default 4.0): Number of CPU threads mapped to ffmpeg -threads.
  • verbose (default False): Emit more logs.

Behavior notes:

  • If the input frame rate is lower than target_fps or the input height is lower than target_height, the stage logs a warning and preview quality can degrade.
  • If ffmpeg fails, the stage logs the error and skips assigning preview bytes for that window.

Example: Configure PreviewStage

from nemo_curator.stages.video.preview.preview import PreviewStage

preview = PreviewStage(
    target_fps=1.0,
    target_height=240,
    compression_level=6,
    quality=50,
    num_cpus_per_worker=4.0,
    verbose=False,
)

Outputs

The stage writes .webp files under the previews/ directory that ClipWriterStage manages. Use the helper to resolve the path:

from nemo_curator.stages.video.io.clip_writer import ClipWriterStage
previews_dir = ClipWriterStage.get_output_path_previews("/outputs")

Refer to Save & Export for directory structure and file locations: Save & Export.

Requirements and Troubleshooting

  • ffmpeg with WebP (libwebp) support must be available in the environment.
  • If you observe warnings about low frame rate or height, consider lowering target_fps or target_height to better match inputs.
  • On encoding errors, check logs for the ffmpeg command and output to diagnose missing encoders.