AI Workflow

The ML development lifecycle provides a structured methodology for taking models from concept to production. Unlike traditional software development, ML development is fundamentally experimental: success requires systematic iteration across data preparation, model development, training, evaluation, and deployment stages.

Problem Formulation

The lifecycle begins with problem formulation, where engineers define the task, success metrics, and constraints. This stage is often underestimated but is critical for project success.

1. Define the business objective and how ML predictions will be used
2. Specify success metrics (accuracy, precision, recall, F1) with concrete thresholds
3. Establish latency budgets and throughput requirements for serving
4. Assess data availability, quality, and labeling needs
5. Identify potential biases and ethical considerations
6. Set a timeline and compute budget for experimentation

Iterative Development and Artifact Management

After problem formulation, the lifecycle moves through iterative cycles of data preparation, feature engineering, model selection, training, and evaluation. Each cycle produces artifacts: processed datasets, trained model weights, evaluation reports, and configuration files.

From Experimentation to Production

The deployment phase marks the transition from experimentation to production. This involves packaging the model, setting up serving infrastructure, implementing monitoring, and establishing feedback loops for continuous improvement. The lifecycle does not end at deployment; production models require ongoing monitoring, maintenance, and periodic retraining.

Models degrade over time as the data they encounter drifts away from the data they learned from.Deeper InsightUnlike traditional software that works the same way forever unless the code changes, ML models silently rot. The world changes -- user behavior shifts, new products launch, seasons change, language evolves -- but the model remains frozen in the snapshot of reality it was trained on. This silent degradation is more dangerous than a software crash because the model keeps producing outputs that look valid but are increasingly wrong.Think of it like...A guidebook written five years ago for a rapidly changing city. The restaurants have closed, the subway map is wrong, and new neighborhoods have appeared. The book still looks authoritative and gives confident recommendations -- they are just increasingly inaccurate. Without someone checking the guidebook against the current city, you would never know it had gone stale. Click to collapse

Experiment tracking is the systematic recording of all parameters, metrics, and artifacts associated with each training run. Without proper tracking, ML teams quickly lose the ability to reproduce results, compare approaches, or understand why certain configurations worked better than others.

Tracking Platforms

Modern experiment tracking platforms provide centralized dashboards for logging hyperparameters, training curves, evaluation metrics, and model artifacts. These tools enable teams to search, filter, and compare across hundreds of experiments.

What to Log

Effective experiment tracking requires discipline in logging practices. Every run should record the complete configuration to enable exact reproduction.

• Model architecture — Layer types, sizes, activation functions, number of parameters
• Hyperparameters — Learning rate, batch size, optimizer settings, regularization
• Data version — Hash or tag identifying the exact training and validation data
• Random seeds — All seeds (Python, NumPy, PyTorch/TensorFlow) for reproducibility
• Framework version — Exact versions of PyTorch/TensorFlow, CUDA, cuDNN
• Hardware environment — GPU type, number of devices, distributed training config
• Training metrics — Loss curves, gradient norms, learning rate schedule
• Evaluation metrics — Accuracy, F1, AUC, confusion matrix on validation set

class="tok-comment"># Example: Logging an experiment with MLflow
import mlflow

with mlflow.start_run(run_name=&class="tok-comment">#class="tok-number">39;resnet50-lr-sweepclass="tok-string">&#class="tok-number">39;):
    mlflow.log_params({
        &class="tok-comment">#class="tok-number">39;model&#class="tok-number">39;: class="tok-string">&#class="tok-number">39;resnet50&#class="tok-number">39;,
        &class="tok-comment">#class="tok-number">39;learning_rateclass="tok-string">&#class="tok-number">39;: class="tok-number">0.001,
        &class="tok-comment">#class="tok-number">39;batch_size&#class="tok-number">39;: class="tok-number">256,
        &class="tok-comment">#class="tok-number">39;optimizerclass="tok-string">&#class="tok-number">39;: &#class="tok-number">39;adamwclass="tok-string">&#class="tok-number">39;,
        &class="tok-comment">#class="tok-number">39;data_version&#class="tok-number">39;: class="tok-string">&#class="tok-number">39;v2.class="tok-number">3&#class="tok-number">39;,
        &class="tok-comment">#class="tok-number">39;seedclass="tok-string">&#class="tok-number">39;: class="tok-number">42,
    })
    class="tok-comment"># ... training loop ...
    mlflow.log_metrics({&class="tok-comment">#class="tok-number">39;val_accuracy&#class="tok-number">39;: class="tok-number">0.943, class="tok-string">&#class="tok-number">39;val_loss&#class="tok-number">39;: class="tok-number">0.187})
    mlflow.pytorch.log_model(model, &class="tok-comment">#class="tok-number">39;model&#class="tok-number">39;)

Collaborative Workflows

Beyond individual experiments, tracking systems support collaborative workflows where multiple team members contribute experiments toward shared objectives. Features like experiment groups, tags, and notes help organize work across team members.

Reproducibility is the ability to obtain consistent results when repeating an experiment under the same conditions. In ML, achieving reproducibility is surprisingly difficult due to sources of non-determinism in hardware, software, and the training process itself.

Sources of Non-Determinism

Sources of non-determinism include random weight initialization, data shuffling, GPU floating-point operation ordering, and framework-level optimizations that sacrifice determinism for performance.

Practical Reproducibility

class="tok-comment"># Setting seeds for practical reproducibility in PyTorch
import random
import numpy as np
import torch

def set_seed(seed: int = class="tok-number">42):
    random.seed(seed)
    np.random.seed(seed)
    torch.manual_seed(seed)
    torch.cuda.manual_seed_all(seed)
    class="tok-comment"># Optional: slower but deterministic
    torch.backends.cudnn.deterministic = True
    torch.backends.cudnn.benchmark = False

Version Control for ML

Version control for ML extends beyond code to include data, configurations, and model artifacts. Tools like DVC track large datasets and model files alongside code, enabling teams to switch between different versions of their entire ML pipeline.

MLOps brings DevOps principles to machine learning, establishing practices for continuous integration, delivery, and training of ML models. The goal is to reduce the time and friction involved in moving models from development to production while maintaining quality and reliability.

The Three Pillars of MLOps

A mature MLOps workflow automates the progression from code changes through testing, training, evaluation, and deployment.

Pipeline Orchestration

Pipeline orchestration is the backbone of MLOps workflows. Tools like Kubeflow Pipelines, Apache Airflow, and Prefect define ML workflows as composable steps with explicit dependencies.

• Apache Airflow — General-purpose workflow orchestration with a large ecosystem of integrations
• Kubeflow Pipelines — ML-specific orchestration on Kubernetes with native GPU support
• Prefect — Python-native orchestration with strong local development and cloud deployment
• Dagster — Data-aware orchestration with built-in testing and observability
• ZenML — MLOps framework that abstracts across infrastructure providers

Model Registries and Promotion

Model registries serve as the central hub for managing model versions and their lifecycle states. A model in the registry progresses through stages like "staging," "canary," and "production" based on validation gates.

Hyperparameter tuning is the process of finding optimal configuration values that are not learned during training, such as learning rate, batch size, model architecture choices, and regularization strength. The hyperparameter space is often high-dimensional and expensive to explore, making efficient search strategies essential.

Search Strategies

Grid search exhaustively evaluates all combinations of specified hyperparameter values, while random search samples randomly from the space. Research by Bergstra and Bengio showed that random search is more efficient than grid search for most problems because it covers more of the important dimensions.

Bayesian Optimization

Bayesian optimization methods like Tree-structured Parzen Estimators (TPE) and Gaussian Processes build a probabilistic model of the objective function and use it to select promising configurations to evaluate next. These methods are significantly more sample-efficient than random search, particularly for expensive training runs.

class="tok-comment"># Bayesian hyperparameter optimization with Optuna
import optuna

def objective(trial):
    lr = trial.suggest_float(&class="tok-comment">#class="tok-number">39;lrclass="tok-string">&#class="tok-number">39;, class="tok-number">1e-5, class="tok-number">1e-2, log=True)
    batch_size = trial.suggest_categorical(&class="tok-comment">#class="tok-number">39;batch_size&#class="tok-number">39;, [class="tok-number">32, class="tok-number">64, class="tok-number">128, class="tok-number">256])
    n_layers = trial.suggest_int(&class="tok-comment">#class="tok-number">39;n_layersclass="tok-string">&#class="tok-number">39;, class="tok-number">2, class="tok-number">8)
    dropout = trial.suggest_float(&class="tok-comment">#class="tok-number">39;dropout&#class="tok-number">39;, class="tok-number">0.0, class="tok-number">0.5)

    model = build_model(n_layers=n_layers, dropout=dropout)
    val_accuracy = train_and_evaluate(model, lr=lr, batch_size=batch_size)
    return val_accuracy

study = optuna.create_study(direction=&class="tok-comment">#class="tok-number">39;maximize&#class="tok-number">39;)
study.optimize(objective, n_trials=class="tok-number">100)

AutoML

AutoML extends hyperparameter tuning to automate broader aspects of the ML pipeline, including feature engineering, model selection, and architecture design.