DEVOURED
The Modern CUDA Toolbox in Practice: A Step-by-Step Optimization Walkthrough
Optimizing image-processing code using modern CUDA tools and algorithms produced a 300x speedup.
What: Nicolas Blin demonstrates how moving from raw CUDA pointers to CCCL's launch API, CUB device algorithms, and pinned memory containers reduced pipeline latency from 6.8 seconds to 23 milliseconds.
Why it matters: This highlights the evolution of the CUDA ecosystem from manual memory management and custom kernels to high-level, performant abstractions that prioritize safety and concurrency.
Takeaway: Replace manual memory allocation with cuda::device_buffer and switch to CUB primitives for common algorithms to improve performance and safety.
Deep dive
- Use Compute Sanitizer to identify illegal memory access before debugging performance.
- Adopt CCCL's cuda::launch API to distinguish between global and block-level indexing.
- Utilize CUB device and block-level primitives instead of hand-written kernel loops.
- Replace raw pointers with non-owning cuda::std::span and mdspan for safer memory access.
- Use pooled cuda::device_memory_pool_ref to avoid frequent allocation overhead.
- Allocate host data with cuda::host_buffer for faster host-to-device memory transfers.
- Parallelize kernels across multiple cuda::streams to saturate the GPU.
Decoder
- CCCL: NVIDIA's C++ Core Compute Libraries, providing standard-library-like abstractions for CUDA.
- CUB: A library of cooperative block-level and device-level parallel primitives (e.g., sort, reduction).
- Pinned memory: Host memory that is page-locked, allowing the GPU to access it directly without driver-level buffering, increasing transfer speeds.
- NVTX: NVIDIA Tools Extension, an API used to annotate and visualize application performance in Nsight Systems.
Original article
Full article content is not available for inline reading.
Explore all 59 articles in today's digest
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi