Security & Privacy

Adversarial attacks exploit vulnerabilities in ML models to cause incorrect predictions through carefully crafted inputs. These attacks reveal fundamental limitations in how neural networks learn and generalize, with implications ranging from academic curiosity to serious safety and security concerns in deployed systems.

Evasion Attacks

Evasion attacks manipulate inputs at inference time to fool the model. The Fast Gradient Sign Method (FGSM) adds a small perturbation in the direction of the loss gradient, creating inputs that look identical to humans but cause misclassification.

Adversarial examples exploit the linearity of neural networks, not their nonlinearity.Deeper InsightIntuitively, you might expect that complex nonlinear models would be harder to fool. In reality, modern neural networks behave mostly linearly in high-dimensional space (ReLU is piecewise linear). A tiny perturbation applied to each of the thousands of input dimensions accumulates linearly, producing a massive shift in the output. FGSM works precisely because it exploits this linearity: each pixel gets a small nudge, but across 150,000+ pixels the cumulative effect overwhelms the model.Think of it like...Pushing a shopping cart. One person pushing with one finger barely moves it. But if 1,000 people each push with one finger in the same direction, the cart flies across the parking lot. Each push is negligible, but the linear accumulation across many dimensions creates a force the cart cannot resist. Neural networks are the cart. Click to collapse

Poisoning Attacks

Poisoning attacks corrupt the training data to implant backdoors or degrade model performance. An attacker who can influence the training dataset can insert poisoned examples that cause the model to learn malicious behaviors triggered by specific patterns.

Model Extraction Attacks

Model extraction attacks aim to steal a model's functionality by querying it and training a copy. An attacker with API access to a deployed model can systematically query it and use the input-output pairs to train a surrogate model.

• Rate limiting — Restrict the number of queries per user/IP to slow extraction.
• Output perturbation — Add small noise to probability outputs without affecting top-1 predictions.
• Watermarking — Embed verifiable signatures that transfer to extracted models.
• Prediction truncation — Return only the top label, not full probability distributions.

Adversarial Training

Adversarial training is the most well-studied defense, where models are trained on both clean and adversarially perturbed examples. During training, adversarial examples are generated on-the-fly and included in each batch.

Certified Defenses

Certified defenses provide provable guarantees that model predictions are robust within a specified perturbation radius. Randomized smoothing creates a smoothed classifier by averaging predictions over random noise perturbations.

Input Preprocessing Defenses

Input preprocessing defenses attempt to remove adversarial perturbations before they reach the model. These approaches are simple to deploy but can be bypassed by adaptive attacks that account for the preprocessing.

Defense in Depth

Defense in depth combines multiple protection layers rather than relying on any single defense. This layered approach makes attacks significantly harder while maintaining system usability.

1. Input validation — Reject out-of-distribution inputs that may indicate an attack.
2. Adversarial training — Harden the model against known perturbation types.
3. Confidence thresholds — Flag low-confidence predictions for human review.
4. Output monitoring — Detect unusual prediction patterns that suggest adversarial probing.
5. Rate limiting — Slow down potential model extraction attempts.

Federated learning enables collaborative model training across multiple data owners without sharing raw data. Each participant trains on their local data and shares only model updates, which are aggregated to improve a global model. This paradigm is particularly important for privacy-sensitive domains like healthcare, finance, and mobile applications.

Federated Averaging (FedAvg)

Federated Averaging is the foundational federated learning algorithm. Each participating device performs multiple local training steps on its data, and the resulting model updates are averaged at a central server.

Challenges in Federated Settings

Secure Aggregation and Privacy

Secure aggregation protocols ensure that the central server can compute the aggregate model update without seeing any individual participant's contribution. Combined with differential privacy, federated learning can provide strong privacy guarantees.

Model inversion attacks can reconstruct training data from a trained model, turning your model into a privacy liability.Deeper InsightA trained model memorizes more about its training data than its predictions reveal on the surface. Model inversion attacks systematically query a model and use gradient-based optimization to reconstruct inputs that maximize the model's confidence for a target class. Researchers have reconstructed recognizable faces from facial recognition models and extracted private text from language models. This means that deploying a model is, in a real sense, publishing a lossy copy of the training data.Think of it like...A sculptor who creates a bust from memory. Even though the original person is not present, studying the sculpture reveals details about the subject -- bone structure, facial features, unique characteristics. A trained neural network is the sculpture, and model inversion is the forensic technique that extracts details about the training data "subjects" the model memorized. Click to collapse

Differential privacy provides a mathematical framework for quantifying and limiting the privacy leakage of computations on sensitive data. Applied to ML, it ensures that the trained model does not reveal too much information about any individual training example.

DP-SGD: Private Gradient Descent

The core mechanism is adding calibrated noise to gradients during training (DP-SGD). Each per-example gradient is clipped to a maximum norm, the clipped gradients are averaged, and Gaussian noise proportional to the sensitivity and privacy budget is added.

class="tok-comment"># Simplified DP-SGD training step
def dp_sgd_step(model, batch, max_grad_norm, noise_multiplier):
    per_example_grads = compute_per_example_gradients(model, batch)
    class="tok-comment"># Step class="tok-number">1: Clip each gradient to max norm
    clipped_grads = [
        g * min(class="tok-number">1, max_grad_norm / grad_norm(g))
        for g in per_example_grads
    ]
    class="tok-comment"># Step class="tok-number">2: Average clipped gradients
    avg_grad = sum(clipped_grads) / len(clipped_grads)
    class="tok-comment"># Step class="tok-number">3: Add calibrated Gaussian noise
    noise_std = noise_multiplier * max_grad_norm / len(batch)
    noisy_grad = avg_grad + torch.randn_like(avg_grad) * noise_std
    class="tok-comment"># Step class="tok-number">4: Update model
    model.parameters -= learning_rate * noisy_grad

The Privacy-Utility Trade-off

The privacy-utility trade-off is the fundamental challenge of differential privacy. Stronger privacy requires more noise, which degrades model quality.

Privacy Budgeting

Privacy budgeting and composition are practical concerns for deploying differential privacy. Each training step consumes a portion of the total privacy budget.

Production ML systems face security threats beyond adversarial examples, including model theft, data extraction, and supply chain attacks. A comprehensive security strategy must address threats at every layer of the ML stack, from data collection through model serving.

Hardening Model Serving Infrastructure

Model serving infrastructure must be hardened against conventional security threats and ML-specific threats. Input validation, rate limiting, and prediction logging form the first line of defense.

• Input validation — Reject malformed, out-of-range, or out-of-distribution inputs before they reach the model.
• Authentication and authorization — Ensure only authorized users and services can access model endpoints.
• Rate limiting — Prevent systematic querying for model extraction or adversarial probing.
• Prediction logging — Record all inputs and outputs for post-hoc analysis of suspicious patterns.
• Encrypted transport — Use TLS for all model serving traffic to prevent interception.

Supply Chain Security

Supply chain security is increasingly important as ML systems depend on pre-trained models, datasets, and libraries from external sources. A compromised pre-trained model could contain backdoors that activate on specific triggers.

Model Watermarking

ML model watermarking embeds verifiable signatures into models that survive common modifications like fine-tuning and pruning. Watermarks can prove model ownership in intellectual property disputes.