Install (All Modalities)#

This guide covers installing NeMo Curator with support for all modalities and verifying your installation is working correctly. For a single-modality install or a 30-minute walkthrough, start with one of the modality quickstarts instead.

Before You Start#

System Requirements#

For comprehensive system requirements and production deployment specifications, refer to Production Deployment Requirements.

Quick Start Requirements:

• OS: Ubuntu 24.04/22.04/20.04 (recommended)
• Python: 3.10, 3.11, or 3.12
• Memory: 16GB+ RAM for basic text processing
• GPU (optional): NVIDIA GPU with 16GB+ VRAM for acceleration
• CUDA 12 (required for audio_cuda12, video_cuda12, image_cuda12, and text_cuda12 extras)

Development vs Production#

Installation Methods#

Choose one of the following installation methods based on your needs:

# Clone the repository
git clone https://github.com/NVIDIA-NeMo/Curator.git
cd Curator

# Install uv if not already available
curl -LsSf https://astral.sh/uv/install.sh | sh

# Install with all extras using uv
uv sync --all-extras --all-groups

# Pull the container from NGC
docker pull nvcr.io/nvidia/nemo-curator:{{ container_version }}

# Run the container with GPU support
docker run --gpus all -it --rm nvcr.io/nvidia/nemo-curator:{{ container_version }}

# Build the container locally
git clone https://github.com/NVIDIA-NeMo/Curator.git
cd Curator
docker build -t nemo-curator:latest -f docker/Dockerfile .

# Run the container with GPU support
docker run --gpus all -it --rm nemo-curator:latest

Install FFmpeg and Encoders (Required for Video)#

Curator’s video pipelines rely on FFmpeg for decoding and encoding. If you plan to encode clips (using --transcode-encoder h264_nvenc or --transcode-encoder libvpx-vp9), install FFmpeg with NVENC and libvpx-vp9 support. The maintained install script bundles both.

curl -fsSL https://raw.githubusercontent.com/NVIDIA-NeMo/Curator/main/docker/common/install_ffmpeg.sh -o install_ffmpeg.sh
chmod +x install_ffmpeg.sh
sudo bash install_ffmpeg.sh

ffmpeg -hide_banner -version | head -n 5
ffmpeg -encoders | grep -E "h264_nvenc|libvpx-vp9" | cat

bash /opt/Curator/docker/common/install_h264_support.sh

Software H.264/HEVC/AV1 Codec Support (Advanced)#

Curator’s default FFmpeg build deliberately ships NVDEC-only decoders for h264, hevc, and av1, and excludes software H.264 encoders (libopenh264, libx264, libx265). This keeps the codec footprint tight and routes every H.264/HEVC/AV1 decode through the GPU.

You may need to add software codec support in two cases:

• H.264 inputs in CPU-only pipeline stages. VideoReader and ClipWriter invoke ffprobe from CPU-only Ray actors that can’t see the GPU; they need a software h264/hevc/av1 decoder to extract metadata. Without it you’ll get a SoftwareCodecMissingError pointing back here.
• H.264 software encoding (for example, on GPUs without an NVENC encoder block such as A100 or H100, when VP9 isn’t acceptable).

Option 1: Run the bundled installer inside the container (Recommended)#

The repository ships a runtime opt-in script that recompiles FFmpeg with software h264/hevc/av1 decoders enabled, optionally including the libopenh264 encoder. It runs inside an existing container — no image rebuild required.

# Inside the container — adds h264/hevc/av1 software decoders only (LGPLv3):
bash /opt/Curator/docker/common/install_h264_support.sh

# Same plus the libopenh264 software h264 ENCODER, so --transcode-encoder=libopenh264 works:
bash /opt/Curator/docker/common/install_h264_support.sh --with-libopenh264

The build takes ~5–10 minutes, replaces /usr/local/bin/{ffmpeg,ffprobe} in place, and pins to the same FFmpeg tag as the image build. Script source: docker/common/install_h264_support.sh.

License notice: the default mode adds only FFmpeg-internal decoders (LGPL). With --with-libopenh264 the binary additionally links Cisco’s OpenH264 (BSD-2-Clause + Cisco-distributed binary license — see https://www.openh264.org/BINARY_LICENSE.txt). You are responsible for any license obligations the resulting binaries impose on your distribution.

Option 2: Use the System FFmpeg#

If you’re not using the Curator container, most Linux distributions ship FFmpeg with libx264 (and sometimes libopenh264) preinstalled:

sudo apt-get install -y ffmpeg
ffmpeg -hide_banner -encoders | grep -E "libx264|libopenh264"

Make sure the ffmpeg on your PATH is the one you want — it must shadow Curator’s bundled build.

Option 3: Edit install_ffmpeg.sh and Rebuild the Image#

For users distributing customized images, edit docker/common/install_ffmpeg.sh before building the container:

• For software h264/hevc/av1 decoders: append h264,hevc,av1 to the --enable-decoder=... list.
• For libopenh264 encoder: add libopenh264-dev to the apt list, libopenh264 to --enable-encoder=..., and --enable-libopenh264 to the configure flags.
• For libx264 encoder: add libx264-dev to the apt list and --enable-libx264 --enable-gpl to the configure flags. Note that --enable-gpl makes the resulting FFmpeg binary GPL-licensed.

Then rebuild your image.

Use the Encoder in ClipTranscodingStage#

libopenh264 is accepted by ClipTranscodingStage out of the box. At setup time, the stage probes the local FFmpeg build and raises a clear error pointing back to this section if the encoder is not actually compiled in. Once your FFmpeg build includes it, just pass:

python video_split_clip_example.py ... --transcode-encoder libopenh264

For other custom encoders not in SUPPORTED_ENCODERS (for example, libx264), edit nemo_curator/stages/video/clipping/clip_extraction_stages.py to extend the tuple, and add the encoder name to the --transcode-encoder argparse choices list in tutorials/video/getting-started/video_split_clip_example.py:

SUPPORTED_ENCODERS = ("h264_nvenc", "libvpx-vp9", "libopenh264", "libx264")  # add yours

Caveats#

• Default options for these encoders are not tuned. ClipTranscodingStage only sets quality presets for h264_nvenc and libvpx-vp9. Other encoders run with FFmpeg defaults, which may produce different quality/size trade-offs than you expect — see Configure encoders for how to pass an explicit bitrate.
• The NeMo Curator team does not test custom encoder configurations. Issues filed against custom encoder builds may be closed.

Package Extras#

NeMo Curator provides several installation extras to install only the components you need:

Installation Verification#

After installation, verify that NeMo Curator is working correctly:

1. Basic Import Test#

# Test basic imports
import nemo_curator
print(f"NeMo Curator version: {nemo_curator.__version__}")

# Test core modules
from nemo_curator.pipeline import Pipeline
from nemo_curator.tasks import DocumentBatch
print("✓ Core modules imported successfully")

2. GPU Availability Check#

If you installed GPU support, verify GPU access:

# Check GPU availability
try:
    import torch
    if torch.cuda.is_available():
        print(f"✓ GPU available: {torch.cuda.get_device_name(0)}")
        print(f"✓ GPU memory: {torch.cuda.get_device_properties(0).total_memory / 1e9:.1f} GB")
    else:
        print("⚠ No GPU detected")
    
    # Check cuDF for GPU deduplication
    import cudf
    print("✓ cuDF available for GPU-accelerated deduplication")
except ImportError as e:
    print(f"⚠ Some GPU modules not available: {e}")

3. Run a Quickstart Tutorial#

Try a modality-specific quickstart to see NeMo Curator in action:

• Text Curation Quickstart - Set up and run your first text curation pipeline
• Audio Curation Quickstart - Get started with audio dataset curation
• Image Curation Quickstart - Curate image-text datasets for generative models
• Video Curation Quickstart - Split, encode, and curate video clips at scale