Infra & Hardware

AI inference is the runtime phase in which a trained model with fixed weights processes new inputs to produce prediction…

Batch inference is an offline prediction method that aggregates a large, fixed set of inputs and generates outputs in bu…

Amazon Bedrock is a fully managed AWS service that provides secure, enterprise-grade access to multiple foundation model…

CUDA is NVIDIA's programming model and runtime platform for running massively parallel computation on NVIDIA GPUs.

cuDNN is a GPU-accelerated library from NVIDIA that provides highly optimized implementations of core deep learning oper…

Edge deployment means running AI models or apps close to where data is created — for example on factory lines, inside re…

A GPU is an accelerator that executes uniform, matrix-heavy computations at high throughput via massive parallel threads…

A GPU cluster is a system where multiple GPUs (Graphics Processing Units) are networked together to function as a single…

Inference is the execution phase where a trained model receives new inputs and computes predictions, classifications, or…

Inference cost is the operational compute-and-infrastructure expense incurred each time a deployed LLM tokenizes a promp…

Inference latency is the actual time it takes for an AI model to process an input and return an output. It typically ref…

Inference scaling is the set of techniques that expand and adapt model serving capacity to meet latency and cost objecti…

A KV cache is the inference-time memory structure that stores previously computed attention key/value tensors in an auto…

KV offloading is an inference technique that tiers the self-attention Key/Value cache from GPU memory to CPU RAM or stor…

Model parallelism is a distributed technique that enables training or serving neural networks too large for a single GPU…

A model router is an orchestration layer that selects which model should handle a request based on difficulty, modality,…

Model serving is the operational system that deploys a trained model behind APIs, batch jobs, or streaming endpoints and…

NVIDIA provides an end-to-end AI software stack—NVIDIA AI Enterprise—spanning deployment microservices (NIM) and develop…

Observability is the operational capability to infer a distributed system’s internal state by collecting traces, metrics…

On-device AI means running artificial intelligence directly on your own device—like a phone, laptop, or tablet—instead o…

OpenTelemetry is a vendor-neutral open-source observability framework that instruments applications via language-specifi…

PagedAttention is an LLM-serving memory technique that partitions the attention key–value (KV) cache into fixed-size pag…

Quantization is a neural‑network compression‑and‑acceleration technique that represents weights and activations with low…

Real-time inference is a serving paradigm that exposes a trained model as an API to execute and respond immediately upon…

A Small Language Model (SLM) is a language model that performs natural-language understanding or generation with a small…

Speculative decoding is an inference acceleration method where a smaller drafter proposes multiple candidate tokens and …

Trainium is Amazon Web Services (AWS)’s custom AI training chip designed to train large deep learning models—especially …

vLLM is an open-source LLM serving engine that boosts throughput by managing the attention KV cache with PagedAttention—…