Intro
You rented a GPU instance, kicked off a training or inference job, and now you're staring at nvidia-smi. The utilization number bounces between 15% and 90%. Is that normal? Is your GPU actually working, or are you burning money on idle silicon?
Some people assume a GPU is "working" just because a process is running. Then they check the numbers and realize half the time the GPU is waiting on data, or the batch size is too small to matter. I'll share my experience on how to improve GPU utilization benchmarking.
1. What "GPU Utilization" Actually Means
nvidia-smi reports a percentage called "GPU-Util." A lot of people assume this means "how busy the GPU is." It doesn't. It means the percentage of time at least one kernel was running on the GPU during the sample period.
That's a subtle but important distinction. A GPU can show 100% utilization while doing almost nothing useful—if the kernel it's running is tiny and the rest of the pipeline is stalled. Conversely, a GPU doing heavy matrix math might show 70% utilization because there are small gaps between kernels.
The lesson: utilization alone doesn't tell you if the GPU is working efficiently. You need to look at it alongside other metrics.
2. The Metrics That Actually Matter
Here's what to check, and what each one tells you:
Power draw is often the most honest signal. A GPU rated for 300W that's drawing 80W is not doing much work, no matter what utilization says.
3. How to Benchmark Properly
Step 1: Establish a baseline
Run nvidia-smi in loop mode before starting your workload:
nvidia-smi --query-gpu=utilization.gpu,memory.used,power.draw,temperature.gpu,clocks.sm --format=csv -l 1This gives you a per-second snapshot. Run it for a minute with no workload to see idle numbers.
Step 2: Run your actual workload
Start your training or inference job. Let it run for several minutes. Watch the numbers.
Step 3: Look at the pattern, not the peak
A healthy GPU workload looks like this:
Utilization: 70–95%, fairly stable
Power draw: 80–100% of TDP
Memory: stable, near expected usage
Temperature: stable, below throttle point
SM clock: stable, near boost clock
A problematic workload looks like this:
Utilization: spiky, oscillating between 10% and 90%
Power draw: low and fluctuating
Memory: bouncing or much lower than expected
Temperature: fine, but clock is dropping
SM clock: unstable
Step 4: Use a profiling tool
nvidia-smi gives you the surface. For deeper analysis, use:
nvidia-smi dmon— device monitoring, more metricsNsight Systems — timeline profiling, shows CPU/GPU gaps
PyTorch Profiler — framework-level breakdown
dcgm— data center GPU manager, production-grade monitoring
4. Common Reasons GPU Utilization Is Low
If your GPU is sitting at 20% utilization, here's what to check, in order:
1. Data loading bottleneck
The GPU is waiting for data. Check if your dataloader has enough workers, if the disk is slow, or if preprocessing is on the critical path. Fix: increase num_workers, preload data, use faster storage.
2. Batch size too small
A batch of 8 won't saturate a modern GPU. Fix: increase batch size until memory is nearly full, then tune from there.
3. CPU bottleneck
Preprocessing, tokenization, or Python overhead is eating time. Fix: move preprocessing to GPU, use faster libraries, or parallelize.
4. Kernel launch overhead
Many small kernels mean the GPU spends time waiting for the next launch. Fix: fuse operations, use larger kernels, or switch to a framework that does this automatically.
5. Synchronization points
Frequent .item(), .cpu(), or logging calls force syncs that stall the GPU. Fix: reduce sync points, log less frequently, use async logging.
6. Power or thermal throttling
The GPU is being held back by power limits or heat. Fix: check cooling, increase power limit if allowed, or move to a better-cooled environment.
7. Wrong precision
Running FP32 when FP16 or BF16 would work leaves performance on the table. Fix: enable mixed precision.
5. What Good Utilization Looks Like
For different workloads:
Note: single-request LLM inference will show low utilization because the GPU finishes the forward pass faster than the next request arrives. This isn't a problem—it's the nature of the workload. Batching fixes it.
6. Benchmarking Tools Worth Using
For most teams, nvidia-smi dmon plus Nsight Systems covers 90% of cases.
7. When It's Time to Change Hardware
If you've optimized everything and utilization is still low, the hardware may be the limit:
GPU too small: model doesn't fit, constant swapping
VRAM too small: batch size capped, can't saturate
PCIe bandwidth limited: data transfer is the bottleneck
CPU too weak: can't feed the GPU fast enough
Storage too slow: data loading stalls
This is where GPU bare metal often wins over shared cloud GPU. Dedicated VRAM, dedicated PCIe lanes, dedicated CPU—no noisy neighbors stealing bandwidth or compute.
8. Summary
GPU-Util alone doesn't tell you if the GPU is working efficiently.
Check power draw, memory, clocks, and temperature together.
Low utilization usually means data loading, batch size, CPU, or sync issues.
Benchmark with the right tools:
nvidia-smi dmon, Nsight, PyTorch Profiler.If optimization doesn't help, the hardware may be the bottleneck.
For production AI, GPU bare metal gives you predictable, dedicated resources.
Thank you for your patience in reading to the end!
