NVIDIA Resources

For the topics you wish to explore further, start by consulting the official documentation, such as the CUDA C++ Programming Guide if you're aiming to become a "CUDA Ninja". However, official documentation can sometimes be challenging to understand. In such cases, searching for "NVIDIA Blog" and "GTC Talks" on relevant topics through Google often leads to excellent presentations and discussions on the subject. Also make sure to prioritize blogs and talks from recent years, as they are more likely to cover the latest technologies.

The following are the reference materials for further study.

NVIDIA's Latest Hardware

• GB200 NVL72
  • [Blog] NVIDIA Contributes NVIDIA GB200 NVL72 Designs to Open Compute Project
  • [Blog] NVIDIA Ethernet Networking Accelerates World’s Largest AI Supercomputer, Built by xAI
• Grace Hopper & NVLink-C2C & Memory Coherent Architecture
  • [Blog] NVIDIA Grace Hopper Superchip Architecture In-Depth
  • [Blog] NVIDIA GH200 Superchip Accelerates Inference by 2x in Multiturn Interactions with Llama Models
  • [GitHub] Memory Coherent Architecture
• GPUDirect Storage & RDMA
  • NVIDIA GPUDirect
• DGX SuperPOD
  • NVIDIA DGX SuperPOD: Next Generation Scalable Infrastructure for AI Leadership Reference Architecture Featuring NVIDIA DGX H100
  • NVIDIA DGX SuperPOD FAQ

CUDA & Optimization

• CUDA C++ Programming Guide
• [GitHub] NVIDIA/cuda-samples
• Coalesced Memory Access
  • [Blog] How to Access Global Memory Efficiently in CUDA C/C++ Kernels
  • 5.3.2. Device Memory Accesses | CUDA C++ Programming Guide
• Shared Memory Bank Conflict
  • [Blog] Using Shared Memory in CUDA C/C++
  • [Blog] An Efficient Matrix Transpose in CUDA C/C++
  • 16.4.3. Shared Memory | CUDA C++ Programming Guide
• CUDA Streams
  • [Blog] How to Optimize Data Transfers in CUDA C/C++
  • [Blog] How to Overlap Data Transfers in CUDA C/C++
  • [Blog] GPU Pro Tip: CUDA 7 Streams Simplify Concurrency
  • [Blog] Using the NVIDIA CUDA Stream-Ordered Memory Allocator, Part 1
  • [Blog] Using the NVIDIA CUDA Stream-Ordered Memory Allocator, Part 2
  • 9.2.1.4. Streams and Events | CUDA C++ Programming Guide
  • 11. Stream Ordered Memory Allocator | CUDA C++ Programming Guide
• Atomics
  • [Blog] CUDA Pro Tip: Optimized Filtering with Warp-Aggregated Atomics
  • 7.14. Atomic Functions | CUDA C++ Programming Guide
• Vectorized Memory Access
  • [Blog] CUDA Pro Tip: Increase Performance with Vectorized Memory Access
• Warp-Level Primitives
  • [Blog] Using CUDA Warp-Level Primitives
  • 7.19. Warp Vote Functions | CUDA C++ Programming Guide
  • 7.20. Warp Match Functions | CUDA C++ Programming Guide
  • 7.21. Warp Reduce Functions | CUDA C++ Programming Guide
  • 7.22. Warp Shuffle Functions | CUDA C++ Programming Guide

Advanced CUDA

• Fat Binary & PTX (Parallel Thread Execution) & SASS (Streaming Assembly)
  • [Blog] CUDA Pro Tip: Understand Fat Binaries and JIT Caching
  • PTX and SASS Assembly Debugging
  • CUDA Binary Utilities
• Cooperative Groups (CG)
  • [Blog] Cooperative Groups: Flexible CUDA Thread Programming
  • 8. Cooperative Groups | CUDA C++ Programming Guide
• LDGSTS

  Require Ampere architecture or later.

  • [Blog] NVIDIA Ampere Architecture In-Depth
  • [Blog] Controlling Data Movement to Boost Performance on the NVIDIA Ampere Architecture
  • [Blog] Boosting Application Performance with GPU Memory Prefetching
  • 7.26. Asynchronous Barrier | CUDA C++ Programming Guide
  • 7.27. Asynchronous Data Copies | CUDA C++ Programming Guide
  • 7.28. Asynchronous Data Copies using cuda::pipeline
• Hopper Architecture & Thread Block Cluster & Distributed Shared Memory (DSMEM) & Tensor Memory Accelerator (TMA)

  Require Hopper architecture or later.

  • [Blog] NVIDIA Hopper Architecture In-Depth
  • 2.3. Memory Hierarchy | CUDA C++ Programming Guide
  • 3.2.5. Distributed Shared Memory | CUDA C++ Programming Guide
  • 7.29. Asynchronous Data Copies using the Tensor Memory Accelerator (TMA) | CUDA C++ Programming Guide
• CUDA Graphs
  • [Blog] Getting Started with CUDA Graphs
  • [Blog] Employing CUDA Graphs in a Dynamic Environment
  • [Blog] Constructing CUDA Graphs with Dynamic Parameters
  • [Blog] Enabling Dynamic Control Flow in CUDA Graphs with Device Graph Launch
  • [Blog] Dynamic Control Flow in CUDA Graphs with Conditional Nodes
  • [Blog] Optimizing llama.cpp AI Inference with CUDA Graphs
  • 3.2.8.7. CUDA Graphs | CUDA C++ Programming Guide

NVIDIA HPC SDK

• ISO C++/Fortran, OpenACC/OpenMP, CUDA
  • [GitHub] N-ways to GPU programming
• Python
  • [Blog] RAPIDS cuDF Accelerates pandas Nearly 150x with Zero Code Changes
  • [Blog] Faster HDBSCAN Soft Clustering with RAPIDS cuML
  • [Blog] Unifying the CUDA Python Ecosystem
  • [Blog] Effortlessly Scale NumPy from Laptops to Supercomputers with NVIDIA cuPyNumeric
  • [Blog] Creating Differentiable Graphics and Physics Simulation in Python with NVIDIA Warp
• NVIDIA CUDA-X Libraries
  • [Blog] Accelerating GPU Applications with NVIDIA Math Libraries
• CUDA C++ Core Libraries and stdexec
• NVSHMEM
  • [Blog] Scaling Scientific Computing with NVSHMEM
• NVIDIA Collective Communications Library (NCCL)
  • [Blog] Fast Multi-GPU collectives with NCCL
  • [Blog] Doubling all2all Performance with NVIDIA Collective Communication Library 2.12
• NVIDIA Scalable Hierarchical Aggregation and Reduction Protocol (SHARP)
  • [Blog] Advancing Performance with NVIDIA SHARP In-Network Computing
  • [YouTube] In-Network Computing with NVIDIA SHARP

Dev Tools

• Compute Sanitizer
  • [Blog] Efficient CUDA Debugging: Memory Initialization and Thread Synchronization with NVIDIA Compute Sanitizer
  • Compute Sanitizer
• Nsight Systems/Compute/Graphics & Nsight Visual Studio Code Edition
  • [GitHub] NVIDIA/nsight-vscode-edition
  • and more.
• Containerization
  • [GitHub] NVIDIA Docker
  • [GitHub] Singularity & [GitHub] Apptainer
  • [GitHub] ENROOT & [GitHub] Pyxis

Higher-Level SDKs

• NVIDIA NeMo
  • [GitHub] NVIDIA/NeMo

• NVIDIA Omniverse & Isaac
  • [GitHub] NVIDIA Isaac Summary

• NVIDIA PhysicsNeMo (formerly NVIDIA Modulus)
  • [GitHub] NVIDIA/PhysicsNeMo

Other GTC Talks

• GTC 24 - Advanced Performance Optimization in CUDA - S62192
• GTC 24 - CUDA, New Features and Beyond - S62400
• GTC 24 - How To Write A CUDA Program, The Ninja Edition - S62401
• GTC 24 - Introduction to CUDA Programming and Performance Optimization - S62191
• GTC 24 - Multi GPU Programming Models for HPC and AI - S61339
• GTC 24 - Warp, Advancing Simulation AI with Differentiable GPU Computing in Python - S63345
• GTC Spring 23 - Connect with the Experts, C++ Standard Parallelism and C++ Core Compute Libraries - CWES52064
• GTC Spring 23 - C++ Standard Parallelism - S51755
• GTC Spring 23 - CUDA, New Features and Beyond - S51225
• GTC Spring 23 - Robust and Efficient CUDA C++ Concurrency with Stream-Ordered Allocation - S51897
• GTC Spring 22 - C++ Standard Parallelism - S41960
• GTC Spring 22 - How CUDA Programming Works - S41487
• GTC Spring 22 - How to Understand and Optimize Shared Memory Accesses using Nsight Compute - S41723
• GTC Fall 21 - Accelerate Computing with CUDA Python - A31138
• GTC Fall 21 - GPU Acceleration in Python using CuPy and Numba - A31149
• GTC Fall 21 - Legate, Scaling the Python Ecosystem - A31168
• GTC Spring 21 - How GPU Computing Works - S31151

Epilogue

Last updated on 2024-12-08.