Tensor Cores

 

Tensor Cores are a key feature in NVIDIA’s GPU architecture that accelerates the GPUs’ computing performance for deep learning.  They are the processing subunits optimized for performing matrix multiplications and accumulations, essential operations in AI workloads.

Background

All semiconductor chips perform math operations (add, subtract, multiply, etc).  A tensor is a mathematical object that describes a multilinear relationship between sets of mathematical objects.  They are commonly shown as an array of numbers, where the dimension of the array is shown below.

                                0 dimension  = scalar

                               1 dimension  = vector

                               2 dimensions = matrix

Bonus:  Read here for excellent explanation of matrix multiplication.

Functionalities

Tensor Cores support mixed-precision computing, which involves using lower precision (e.g., FP16) for matrix multiplications and higher precision (e.g., FP32) for accumulation.  This approach balances performance and precision, allowing for faster computations without significant loss of accuracy.  

Tensor Cores can handle different data types, including FP16 (half-precision floating-point), BFLOAT16 (Brain Floating Point), INT8 (8-bit integer), and INT4 (4-bit integer).  This flexibility allows Tensor Cores to be used in a wide range of AI applications, from training large neural networks to performing efficient inference.

Key Features

AI models such as neural networks contain many layers and the computations running through these layers can be batched together to run in a single, large, matrix multiplication.  Tensor Cores significantly increase the throughput of these matrix operations.

Furthermore, by offloading intensive matrix operations to Tensor Cores, NVIDIA GPUs can achieve higher performance per watt, making them more energy-efficient compared to using general-purpose CUDA cores for the same tasks.

In summary, the specialized hardware Tensor Cores are a game-changing technology that provides unparalleled performance and efficiency for matrix operations.  They help to accelerate the generative AI evolution and thus, play a part in changing the future.