DEVOURED
What happens when you run a CUDA kernel
A deep dive into the CUDA execution pipeline reveals how a kernel call travels from a C++ function stub down to raw GPU warp scheduling.
What: The analysis traces the compilation via nvcc, the host driver interaction, the pushbuffer method streaming, and how individual streaming multiprocessors (SMs) use hardware scoreboard barriers to hide memory latency.
Why it matters: Understanding these internals is crucial for developers optimizing high-performance compute kernels, as most bottlenecks are now found in memory latency hiding and instruction scheduling rather than just pure math throughput.
Takeaway: If you are optimizing CUDA, use `ncu` (NVIDIA Nsight Compute) to inspect `sm__warps_active` and `dram__throughput` to determine if your kernel is bound by memory access latency.
Deep dive
- Compilation: nvcc invokes cicc for PTX and ptxas for SASS, embedding both into a fatbin executable for compatibility.
- Launch Path: Host code uses a generated stub to register the kernel;
cuLaunchKerneltranslates high-level calls into device-native 'methods' in a pushbuffer. - Scheduling: Schedulers on each SM track warp eligibility via six physical scoreboard barriers per warp.
- Latency Hiding: Compilers pack control codes into the instruction stream to manage stall cycles and yield hints, allowing GPUs to switch between warps while memory loads are in flight.
- Coalescing: Consecutive memory accesses from 32-thread warps are coalesced into fewer, high-bandwidth transactions to the L2 cache or DRAM.
Decoder
- CUDA Kernel: A function that runs in parallel on many GPU threads.
- Warp: A group of 32 threads in NVIDIA GPUs that execute instructions in lock-step.
- PTX (Parallel Thread Execution): A low-level, device-agnostic intermediate representation for GPU code.
- SASS (Shader Assembly): The final, architecture-specific machine code that runs on NVIDIA hardware.
- QMD (Queue Meta Data): A launch descriptor structure that tells the GPU how to execute a parallel compute grid.
- Doorbell: An MMIO (Memory Mapped I/O) register that the CPU writes to in order to notify the GPU that new work is ready in the command stream.
Original article
Full article content is not available for inline reading.
Explore all 42 articles in today's digest
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi