Specialized Processing#

Domain-specific processing for code and advanced curation tasks using NeMo Curator’s specialized modules.

This section covers advanced processing techniques for specific data types and use cases that require specialized handling beyond general text processing. These tools are designed for specific domains like programming content.

How It Works#

Specialized processing modules in NeMo Curator are designed for specific data types and use cases:

• Code Processing: Handles programming languages with syntax-aware filtering

Each specialized processor understands the unique characteristics of its target domain and applies appropriate metrics and thresholds within the broader data processing framework .

Available Specialized Tools#

Usage#

Quick Examples#

from nemo_curator.pipeline import Pipeline
from nemo_curator.stages.text.filters import ScoreFilter
from nemo_curator.stages.text.filters.heuristic.code import PythonCommentToCodeFilter, NumberOfLinesOfCodeFilter
from nemo_curator.stages.text.io.reader import JsonlReader

# Filter Python code based on quality metrics
code_pipeline = Pipeline(
    name="code_processing_pipeline",
    stages=[
    JsonlReader(
        file_paths="code_data/*.jsonl",
        fields=["content"]
    ),
    ScoreFilter(
        filter_obj=PythonCommentToCodeFilter(
            min_comment_to_code_ratio=0.01,
            max_comment_to_code_ratio=0.8
        ),
        text_field="content",
        score_field="comment_ratio"
    ),
    ScoreFilter(
        filter_obj=NumberOfLinesOfCodeFilter(min_lines=5, max_lines=1000),
        text_field="content", 
        score_field="line_count"
    )
])

results = code_pipeline.run()

When to Use Specialized Processing#

• Code datasets: When working with programming content that needs syntax-aware filtering

Performance Considerations#

• Code processing: Fast heuristic-based filtering, suitable for large code repositories