A neural network transforms inputs into outputs through a series of layers. Each layer applies a linear transformation (weights and biases) followed by a non-linear activation function. Stacking layers allows the network to learn increasingly abstract representations.

Architecture

  • Input layer — one neuron per feature
  • Hidden layers — learned intermediate representations; more layers = deeper network
  • Output layer — one neuron per class (classification) or one neuron (regression)
  • Weights (W) and biases (b) — the learnable parameters
  • Activation function — introduces non-linearity (ReLU, sigmoid, tanh, softmax)

Backpropagation

Training minimises a loss function (e.g., cross-entropy for classification) by adjusting weights. Backpropagation computes the gradient of the loss with respect to each weight via the chain rule. Gradient descent then updates each weight in the direction that reduces the loss.

simple_nn.pypython
import torch
import torch.nn as nn

class TwoLayerNet(nn.Module):
    def __init__(self, input_dim, hidden_dim, output_dim):
        super().__init__()
        self.fc1 = nn.Linear(input_dim, hidden_dim)
        self.relu = nn.ReLU()
        self.fc2 = nn.Linear(hidden_dim, output_dim)

    def forward(self, x):
        return self.fc2(self.relu(self.fc1(x)))

model = TwoLayerNet(784, 256, 10)  # MNIST: 28x28 -> 10 classes
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)
loss_fn = nn.CrossEntropyLoss()

# Single training step
logits = model(x_batch)
loss = loss_fn(logits, y_batch)
optimizer.zero_grad()
loss.backward()   # backprop
optimizer.step()  # weight update