DL Primer

Neural networks are the computational backbone of modern deep learning. At their core, they are compositions of linear transformations followed by nonlinear activation functions, organized into layers. Each layer transforms its input through a weight matrix multiplication, adds a bias vector, and applies an activation function to produce its output.

From Perceptrons to Deep Networks

The simplest neural network is the single-layer perceptron, which can only learn linearly separable functions. By stacking multiple layers, deep neural networks gain the ability to learn hierarchical representations of data. Early layers typically capture low-level features (edges, textures), while deeper layers compose these into increasingly abstract representations (objects, concepts).

Systems Implications of Network Architecture

From a systems perspective, understanding neural network architecture is essential for making informed decisions about memory requirements, computational complexity, and hardware utilization. The number of parameters directly determines memory consumption, while the number of floating-point operations (FLOPs) determines computational cost.

The Universal Approximation Theorem

The universal approximation theorem tells us that sufficiently wide neural networks can approximate any continuous function. However, this theoretical guarantee says nothing about the practical challenges of training such networks or deploying them efficiently. The gap between what is theoretically possible and what is practically achievable is where ML systems engineering lives.

Backpropagation is just the chain rule of calculus applied recursively through the network.Deeper InsightDespite its mystique, backpropagation is not a novel mathematical discovery. It is a systematic bookkeeping method for applying the chain rule at every layer. The brilliance lies in efficiency: rather than computing each gradient independently (which would be impossibly slow), backprop reuses intermediate results from deeper layers, turning an exponential problem into a linear one.Think of it like...A factory assembly line where each station gets a note saying "here is how much the final product was off -- your contribution to the error was this much." Each station passes the note backward to the previous station, adjusted for its own work. No station needs to understand the entire factory to know how to improve. Click to collapse

Backpropagation is the algorithm that makes deep learning possible. It efficiently computes the gradient of the loss function with respect to every parameter in the network by applying the chain rule of calculus from the output layer back through the network. This process enables gradient-based optimization of networks with millions or billions of parameters.

Forward and Backward Passes

The forward pass computes the network output by propagating inputs through each layer sequentially. The backward pass then propagates error gradients in reverse, computing how each parameter contributed to the total loss. This symmetry between forward and backward passes has profound implications for system design.

Memory Challenges and Gradient Checkpointing

Memory consumption during backpropagation is a critical systems concern. The standard implementation requires storing all intermediate activations from the forward pass to compute gradients during the backward pass. For deep networks, this can consume tens of gigabytes of GPU memory.

Automatic Differentiation Frameworks

Automatic differentiation, the generalization of backpropagation, is a core feature of modern ML frameworks. The design choice between static and dynamic graphs has significant implications for debugging, performance optimization, and deployment flexibility.

Activation functions introduce nonlinearity into neural networks, which is essential for learning complex patterns. Without nonlinear activations, a multi-layer network would reduce to a single linear transformation, regardless of its depth. The choice of activation function affects training dynamics, model expressiveness, and computational efficiency.

ReLU and the Vanishing Gradient Problem

The Rectified Linear Unit (ReLU) and its variants dominate modern deep learning due to their simplicity and favorable gradient properties. ReLU computes max(0, x), which is computationally cheap and avoids the vanishing gradient problem that plagued earlier activation functions like sigmoid and tanh.

Modern Activation Functions

GELU (Gaussian Error Linear Unit) has become the standard activation in Transformer models, providing smooth gradients that improve training stability. Swish (x * sigmoid(x)) offers similar benefits and is used in EfficientNet and other modern architectures.

Hardware Implications

From a systems perspective, activation function choice impacts quantization compatibility and hardware acceleration. ReLU is particularly hardware-friendly because it only requires a comparison operation. More complex activations like GELU require lookup tables or polynomial approximations on hardware without native support.

Loss functions quantify how far the model predictions are from the desired outputs, providing the signal that drives learning. The choice of loss function encodes assumptions about the problem and can significantly influence model behavior.

Stochastic Gradient Descent and Mini-Batches

Stochastic gradient descent (SGD) and its variants are the workhorses of deep learning optimization. Rather than computing gradients over the entire dataset (full batch gradient descent), SGD estimates gradients from small random subsets (mini-batches). This introduces noise that can actually help escape local minima and saddle points.

Modern Optimizers

Modern optimizers like Adam, AdamW, and LAMB combine momentum (exponential moving average of gradients) with adaptive learning rates (per-parameter scaling based on gradient history). Adam is the default choice for most applications due to its robustness to hyperparameter settings. AdamW adds proper weight decay decoupling, which improves generalization.

Learning Rate Scheduling

Learning rate scheduling is a critical component of training systems. The learning rate schedule can be more important than the optimizer choice for achieving good performance.

• Linear warmup + cosine decay — Start with a low learning rate, ramp up linearly, then decay following a cosine curve. Standard for Transformers.
• Step decay — Reduce learning rate by a factor (e.g., 0.1) at fixed epoch milestones. Classic choice for CNNs.
• One-cycle — Ramp learning rate up and then down over one full training run. Fast convergence with good generalization.
• Reduce on plateau — Monitor validation loss and reduce learning rate when progress stalls. Simple but effective.

Understanding the loss landscape is key to understanding why training neural networks is challenging. The loss surface of deep networks is high-dimensional and non-convex, containing many local minima, saddle points, and flat regions.

Batch Size: A Systems and Optimization Trade-off

Batch size is a critical hyperparameter that affects both training dynamics and systems performance. Larger batch sizes provide more accurate gradient estimates and enable better GPU utilization, but they can also lead to sharp minima that generalize poorly.

Gradient Clipping

Gradient clipping is a practical technique that prevents training instability by capping the magnitude of gradients. This is especially important for recurrent networks and Transformers, where gradient magnitudes can vary dramatically.

class="tok-comment"># Global gradient norm clipping in PyTorch
torch.nn.utils.clip_grad_norm_(
    model.parameters(),
    max_norm=class="tok-number">1.0  class="tok-comment"># Clip if global norm exceeds class="tok-number">1.0
)
optimizer.step()

Convergence Diagnostics

Convergence diagnostics are essential for efficient training. Monitoring key metrics over time helps identify issues before they waste expensive compute hours.

• Training loss — Should decrease steadily; plateaus suggest learning rate issues
• Validation loss — Divergence from training loss signals overfitting
• Gradient norms — Spikes indicate instability; consistently small values indicate vanishing gradients
• Learning rate — Verify the schedule is applied correctly
• GPU memory usage — Unexpected growth may indicate memory leaks in data loading

Batch normalization stabilizes training by normalizing layer inputs to zero mean and unit variance.Deeper InsightWithout batch normalization, each layer must constantly adapt to shifting input distributions caused by parameter updates in preceding layers (internal covariate shift). Batch norm eliminates this moving target, letting each layer learn independently. This is why adding batch norm often lets you use higher learning rates and train much faster -- each layer sees a stable input range.Think of it like...Trying to learn archery while someone keeps moving the target. Batch normalization pins the target in place for each layer, so every layer can focus on improving its own aim rather than constantly adjusting for the moving inputs from the layer before it. Click to collapse