Technical Glossary

A neural network trained on vast amounts of text data to understand and generate human language. Modern LLMs use the Transformer architecture with billions of parameters.

An architecture where multiple specialized sub-networks (experts) are conditionally activated based on input. A router network decides which experts to use for each token, enabling efficient scaling to much larger models.

A neural network architecture introduced in "Attention is All You Need" (2017). It processes sequences using self-attention mechanisms instead of recurrence, enabling parallel computation and capturing long-range dependencies.

An optimization of Multi-Head Attention where multiple query heads share the same key-value heads. This reduces KV cache memory while maintaining most of the model quality.

A method to encode positional information by rotating query and key vectors in complex space. Unlike absolute positional embeddings, RoPE encodes relative positions naturally through rotation, enabling better extrapolation to longer sequences.

A 4-bit quantization format from llama.cpp that achieves ~4.5 bits per weight. It uses super-blocks of 256 weights with hierarchical scaling for high quality compression.

A 6-bit quantization format offering higher precision than Q4_K. Each block stores 256 weights in 210 bytes, achieving ~6.5 bits per weight with excellent quality preservation.

A 16-bit floating-point format (IEEE 754-2008) commonly used in deep learning for its balance between precision and memory efficiency. Modern GPUs have dedicated FP16 tensor cores for accelerated matrix operations.

A 32-bit floating-point format providing higher precision than FP16. Used for accumulation in matrix operations to prevent precision loss during summation of many values.

A memory optimization that stores previously computed Key and Value tensors during autoregressive generation. This avoids recomputing attention for all previous tokens, reducing generation from O(n^2) to O(n).

A specialized cache system for MoE models that keeps frequently used expert weights in GPU memory. Essential for large models where all experts cannot fit in VRAM simultaneously.

NVIDIA's GPU-accelerated library for dense linear algebra operations. Provides highly optimized implementations of BLAS routines including GEMM (matrix multiplication).

The fundamental operation in neural network computation: C = alpha * A * B + beta * C. Modern GPUs are highly optimized for this operation, with tensor cores providing massive parallelism.

A compact AI supercomputer based on the Grace Blackwell architecture. Features unified CPU-GPU memory architecture with 128GB shared memory, ideal for running large LLM models locally.

A system call that maps a file into the process's virtual address space. Enables direct access to file contents as if they were in memory, with the OS handling paging. Essential for loading large model files efficiently.

A simplified normalization technique that only uses the RMS (root mean square) of activations, omitting the mean subtraction of LayerNorm. More efficient and works equally well for transformers.

An activation function that combines the Swish activation with a gating mechanism. Used in the FFN layers of modern transformers for improved expressiveness and training stability.

In MoE models, the process of selecting the K highest-scoring experts based on router logits. Typically K=2, meaning 2 experts are activated per token.

The difference between original floating-point weights and their quantized representation. Measured as RMSE (Root Mean Square Error) or perplexity degradation. Lower bit formats have higher error but smaller model size.

A memory management technique inspired by virtual memory paging. Stores KV cache in non-contiguous blocks, enabling efficient memory sharing between sequences and reducing fragmentation.