Neural Networks (NN)

What is Neural Networks (NN)?

Neural networks are computing systems loosely inspired by the biological neural networks in human brains. They consist of layers of interconnected nodes (called neurons or units) that process information by passing signals to each other, adjusting the strength of connections based on experience. A basic neural network has an input layer that receives data, one or more hidden layers that process it, and an output layer that produces a result. During training, the network adjusts millions of connection strengths (called weights) to minimize errors in its predictions. The simplest neural network, the perceptron, dates back to 1958, but neural networks only became truly powerful when researchers figured out how to train networks with many layers, a breakthrough that led to deep learning. Today, neural networks are the engine behind image recognition, language translation, voice synthesis, and most modern AI achievements.

Technical Deep Dive

Artificial neural networks (ANNs) are computational models composed of interconnected processing units (neurons) organized in layers that learn to approximate functions through iterative weight adjustment. The fundamental unit computes a weighted sum of inputs, applies a nonlinear activation function (sigmoid, tanh, ReLU, GELU), and passes the result forward. Training uses backpropagation to compute gradients of a loss function with respect to network weights, updating them via gradient descent variants (SGD, Adam, AdamW). Architecture evolution spans single-layer perceptrons (Rosenblatt, 1958), multi-layer perceptrons (MLPs), and specialized topologies including convolutional, recurrent, attention-based, and graph neural networks. Key concepts include universal approximation theorem, vanishing/exploding gradients, batch normalization, dropout regularization, skip connections, and learning rate scheduling. Modern neural networks scale to billions of parameters and form the basis of deep learning.

Why It Matters

Neural networks are the core technology behind virtually every modern AI breakthrough, from the image recognition in your phone's camera to the language understanding in ChatGPT and the game-playing AI that mastered Go and chess.

Examples

• Perceptron: The simplest neural network model (1958), consisting of a single layer of weights that learns to classify inputs into two categories through iterative training
• Multi-Layer Perceptron (MLP): Feedforward neural network with one or more hidden layers between input and output, capable of learning nonlinear decision boundaries for classification and regression
• Backpropagation: The fundamental training algorithm for neural networks that efficiently computes gradients by propagating error signals backward through the network layers
• Activation Functions: Nonlinear functions (ReLU, sigmoid, tanh, GELU) applied at each neuron that enable neural networks to learn complex, non-linear patterns in data
• Self-Organising Maps (SOM): Unsupervised neural network that produces a low-dimensional representation of high-dimensional input data while preserving topological relationships

Related Concepts