Data Engineering

Data pipelines are the foundation of every ML system, responsible for collecting, transforming, and delivering data to training and serving infrastructure. A well-designed pipeline must handle diverse data sources, varying scales, and evolving schemas while maintaining data quality and freshness.

Batch vs. Stream Processing

Batch pipelines process large volumes of historical data on a schedule (hourly, daily) and are typically used for training data preparation. Stream pipelines process data in real-time as it arrives and are used for feature computation and online serving.

Pipeline Reliability

Data pipeline reliability is crucial because downstream model quality is directly dependent on data quality. Pipeline failures can silently corrupt training data and degrade model performance.

Scaling Data Pipelines

As data volumes grow, pipelines must scale horizontally across distributed processing frameworks. The choice of framework depends on factors like data volume, processing complexity, latency requirements, and the team's existing infrastructure.

• Apache Spark — Mature distributed processing engine, strong for batch ETL, wide ecosystem
• Apache Beam — Unified batch and stream programming model, runs on multiple backends
• Dask — Python-native parallel computing, great for teams already using pandas/NumPy
• Apache Flink — Stream-first processing with strong exactly-once semantics
• Ray Data — Distributed data processing integrated with Ray's ML ecosystem

Data labeling is often the most expensive and time-consuming part of ML development. Supervised learning requires labeled examples, and the quality of labels directly determines the upper bound on model performance. Labeling strategies must balance quality, cost, and throughput.

Human Labeling

Human labeling remains the gold standard for many tasks but faces challenges of scale, consistency, and cost. Labeling platforms provide workforce management, quality control, and tooling for annotation tasks.

Active Learning

Active learning reduces labeling costs by intelligently selecting the most informative samples for human annotation. Rather than labeling data randomly, active learning queries the model to identify examples where it is most uncertain.

Weak Supervision and Programmatic Labeling

Weak supervision and programmatic labeling offer alternatives to manual annotation. Frameworks like Snorkel enable engineers to write labeling functions that encode heuristics, patterns, and external knowledge sources.

class="tok-comment"># Example labeling functions in Snorkel
from snorkel.labeling import labeling_function

class="tok-decorator">@labeling_function()
def lf_keyword_spam(x):
    class="tok-string">class="tok-string">""class="tok-string">"Flag emails containing common spam keywords."class="tok-string">""
    spam_words = [&class="tok-comment">#class="tok-number">39;freeclass="tok-string">&#class="tok-number">39;, &#class="tok-number">39;winnerclass="tok-string">&#class="tok-number">39;, &#class="tok-number">39;click hereclass="tok-string">&#class="tok-number">39;, &#class="tok-number">39;act nowclass="tok-string">&#class="tok-number">39;]
    return SPAM if any(w in x.text.lower() for w in spam_words) else ABSTAIN

class="tok-decorator">@labeling_function()
def lf_short_email(x):
    class="tok-string">"""Very short emails with links are often spam."""
    return SPAM if len(x.text) < class="tok-number">50 and &class="tok-comment">#class="tok-number">39;http&#class="tok-number">39; in x.text else ABSTAIN

class="tok-comment"># Snorkel combines these noisy functions statistically
class="tok-comment"># to produce probabilistic training labels

Data versioning tracks changes to datasets over time, enabling reproducibility, debugging, and rollback. Just as code version control is fundamental to software engineering, data version control is essential for ML engineering. Without it, teams cannot reliably reproduce past training runs or diagnose model regressions.

Versioning Tools

Tools like DVC, LakeFS, and Delta Lake provide versioning capabilities for large datasets, each taking a different approach to the problem.

Data Lineage

Data lineage tracks the complete provenance of data through the pipeline, recording every transformation applied from source to model. This audit trail is essential for debugging data quality issues, understanding model behavior, and complying with regulations.

Storage and Format Considerations

Effective data versioning requires infrastructure decisions about storage, deduplication, and access patterns.

• Content-addressable storage — Files stored by their hash, so identical content is never duplicated across versions
• Columnar formats (Parquet, Arrow) — Enable reading specific columns without scanning the full dataset
• Partitioning — Organize data by date, category, or other keys for efficient incremental processing
• Compression — Snappy for speed, ZSTD for size, Gzip for compatibility

Data augmentation artificially increases the effective size and diversity of training data by applying transformations to existing examples. This is one of the most effective regularization techniques, improving model generalization while reducing the need for additional labeled data.

Image Augmentation

For image data, common augmentations include random cropping, flipping, rotation, color jittering, and cutout. More advanced techniques like MixUp and CutMix create synthetic examples by blending pairs of training images and their labels.

Text and Tabular Augmentation

For text data, augmentation techniques include synonym replacement, random insertion, back-translation, and paraphrase generation using language models. For tabular data, techniques like SMOTE address class imbalance by generating synthetic minority examples through interpolation.

Systems Considerations

From a systems perspective, data augmentation should be applied on-the-fly during training rather than pre-computed, to maximize diversity without increasing storage requirements.

class="tok-comment"># Efficient augmentation pipeline with Albumentations
import albumentations as A
from albumentations.pytorch import ToTensorV2

transform = A.Compose([
    A.RandomResizedCrop(class="tok-number">224, class="tok-number">224, scale=(class="tok-number">0.08, class="tok-number">1.0)),
    A.HorizontalFlip(p=class="tok-number">0.5),
    A.ColorJitter(brightness=class="tok-number">0.4, contrast=class="tok-number">0.4, saturation=class="tok-number">0.4),
    A.CoarseDropout(max_holes=class="tok-number">1, max_height=class="tok-number">56, max_width=class="tok-number">56, p=class="tok-number">0.5),
    A.Normalize(mean=[class="tok-number">0.485, class="tok-number">0.456, class="tok-number">0.406], std=[class="tok-number">0.229, class="tok-number">0.224, class="tok-number">0.225]),
    ToTensorV2(),
])
class="tok-comment"># Albumentations is class="tok-number">2-5x faster than torchvision.transforms

Data quality is the single most important factor in ML system performance. The adage "garbage in, garbage out" applies with particular force to ML, where models can silently learn patterns from data errors, duplicates, and label noise without any obvious symptoms until deployment.

Your model is only as good as your data. No amount of architectural innovation can compensate for fundamentally flawed training data.

Data Validation Frameworks

Data validation frameworks enable teams to define expectations about data distributions, schema constraints, and statistical properties. These checks run automatically as part of the pipeline, catching issues before they corrupt training data.

class="tok-comment"># Data validation with Great Expectations
import great_expectations as gx

context = gx.get_context()
validator = context.sources.pandas_default.read_csv(class="tok-string">"training_data.csv")

class="tok-comment"># Define expectations
validator.expect_column_values_to_not_be_null(class="tok-string">"user_id")
validator.expect_column_values_to_be_between(class="tok-string">"age", min_value=class="tok-number">0, max_value=class="tok-number">150)
validator.expect_column_mean_to_be_between(class="tok-string">"purchase_amount", min_value=class="tok-number">10, max_value=class="tok-number">500)
validator.expect_column_proportion_of_unique_values_to_be_between(class="tok-string">"email", class="tok-number">0.95, class="tok-number">1.0)

results = validator.validate()
if not results.success:
    raise ValueError(class="tok-string">"Data quality checks failed!")

Data Quality Dimensions

• Completeness — Are there missing values? What is the null rate per column?
• Consistency — Do records conflict with each other or with business rules?
• Accuracy — Are labels correct? What is the estimated label noise rate?
• Freshness — How old is the data? Is it current enough for the model's purpose?
• Representativeness — Does the data cover the target distribution? Are minority groups underrepresented?
• Uniqueness — Are there duplicate records that would bias the training distribution?

Proactive Data Quality Management

Proactive data quality management is more effective than reactive debugging. This includes establishing data contracts between producers and consumers, implementing continuous monitoring, and building automated alerts for drift detection.