Sigmoid Function — Technology Wiki

Overview

Direct Answer

The sigmoid function is a mathematical activation function that transforms any input value into an output between 0 and 1 using the formula 1/(1+e^-x). It is particularly suited for binary classification tasks where outputs must represent probabilities.

How It Works

The function applies an exponential curve that produces smooth, differentiable outputs across its entire domain. As input values increase, the output asymptotically approaches 1; as they decrease, it approaches 0. This S-shaped curve enables neural networks to learn non-linear decision boundaries whilst maintaining gradient flow during backpropagation.

Why It Matters

Sigmoid enables binary classification outputs that directly correspond to probability estimates, critical for applications requiring calibrated confidence scores rather than arbitrary scaled values. Its mathematical properties support efficient training in shallow networks and remain standard in output layers for two-class prediction problems.

Common Applications

Common uses include medical diagnosis systems outputting disease probability, credit risk assessment producing default likelihood scores, and email spam detection yielding classification confidence. It remains the default activation for logistic regression implementations in enterprise analytics platforms.

Key Considerations

The function suffers from vanishing gradient problems in deep networks, making it less suitable for hidden layers in modern architectures. Its output range constraint can cause saturation, slowing convergence during training when gradients near 0 or 1.

Cross-References(1)

Deep Learning

Activation Function

Related in Training & Optimisation

Self-Attention

An attention mechanism where each element in a sequence attends to all other elements to compute its representation.

Multi-Head Attention

An attention mechanism that runs multiple attention operations in parallel, capturing different types of relationships.

Residual Network

A deep neural network architecture using skip connections that allow gradients to flow directly through layers, enabling very deep networks.

Layer Normalisation

A normalisation technique that normalises across the features of each individual sample rather than across the batch.

Dropout

A regularisation technique that randomly deactivates neurons during training to prevent co-adaptation and reduce overfitting.

Activation Function

A mathematical function applied to neural network outputs to introduce non-linearity, enabling the learning of complex patterns.

ReLU

Rectified Linear Unit — an activation function that outputs the input directly if positive, otherwise outputs zero.

Softmax Function

An activation function that converts a vector of numbers into a probability distribution, commonly used in multi-class classification.

Positional Encoding

A technique that injects information about the position of tokens in a sequence into transformer architectures.

Gradient Clipping

A technique that caps gradient values during training to prevent the exploding gradient problem.

Mixed Precision Training

Training neural networks using both 16-bit and 32-bit floating-point arithmetic to speed up computation while maintaining accuracy.

Rotary Positional Encoding

A position encoding method that encodes absolute position with a rotation matrix and naturally incorporates relative position information into attention computations.

More in Deep Learning

Flash Attention

Architectures

An IO-aware attention algorithm that reduces memory reads and writes by tiling the attention computation, enabling faster training of long-context transformer models.

Mixture of Experts

Architectures

An architecture where different specialised sub-networks (experts) are selectively activated based on the input.

Gradient Checkpointing

Architectures

A memory optimisation that trades computation for memory by recomputing intermediate activations during the backward pass instead of storing them all during the forward pass.

LoRA

Language Models

Low-Rank Adaptation — a parameter-efficient fine-tuning technique that adds trainable low-rank matrices to frozen pretrained weights.

Gated Recurrent Unit

Architectures

A simplified variant of LSTM that combines the forget and input gates into a single update gate.

Autoencoder

Architectures

A neural network trained to encode input data into a compressed representation and then decode it back to reconstruct the original.

Pipeline Parallelism

Architectures

A form of model parallelism that splits neural network layers across devices and pipelines micro-batches through stages, maximising hardware utilisation during training.

Representation Learning

Architectures

The automatic discovery of data representations needed for feature detection or classification from raw data.