Skip to content
Novandri.

Why GPU Computing Is Becoming a Strategic Resource

GPUs matter because ML workloads are parallel and memory-bound, not because they are faster in general. Read the benchmark discipline, a budget example, and the limits.

Rachmadwipa Novandri7 min read

GPU computing is often described as a hardware trend. The more useful description is a mismatch: the dominant workloads in machine learning do thousands of similar arithmetic operations at once, while a CPU is designed to handle diverse tasks with low latency. A GPU is not a universally faster processor; it is a processor whose design fits a particular shape of work, and that shape has become economically important.

Direct answer:

GPUs became strategic because training and inference are dominated by parallel matrix operations and memory bandwidth, and because access to them now constrains what organisations can build.

MLCommons states that its benchmarks aim to "deliver a representative benchmark suite for AI/ML that fairly evaluates system performance", with goals including "enable fair comparison of competing systems" and keeping "benchmarking effort affordable so all can participate". The boundary: a benchmark result is a measurement under a defined workload, not proof that your workload will behave the same way.

35x to 49xreduction in storage requirements without affecting accuracy on the models studiedSong Han et al., 2015
9x to 13xreduction in the number of connections through pruningSong Han et al., 2015
from 32 to 5bits representing each connection after quantizationSong Han et al., 2015
A four-step flow: match the workload, benchmark properly, budget memory, then separate training from serving.
Conceptual: parallel matrix operations and memory bandwidth. Buying for the training profile and paying for it on every inference request is the common error.

What the hardware actually changes

Three differences matter:

Parallelism. A GPU has many cores designed to execute the same operation across many data elements. Matrix multiplication — the core of neural network training and inference — maps naturally onto this structure. A CPU with fewer, more complex cores handles branching and sequential logic better.

Memory bandwidth. Moving weights and activations in and out of memory often costs more time than the arithmetic itself. A GPU's high-bandwidth memory is therefore as important as its compute throughput, which is why a chip with impressive theoretical FLOPS can disappoint in practice.

Cost and availability. Accelerators are expensive, consume significant power, and are supplied by a concentrated market. Access becomes a planning constraint, not just a purchasing decision.

Benchmarks are a method, not a score

MLCommons' framing is careful: benchmarks should "fairly evaluate" systems and "enable fair comparison". That qualification matters because a number without a workload definition is not comparable.

Three questions make a benchmark result interpretable:

  1. What workload was measured? Training and inference differ, and so do model families and precision formats.
  2. What was held constant? Power, software versions, and batch size change results substantially.
  3. What was the metric? Throughput matters for training; latency and cost per request matter for serving.

A result that omits these is marketing, not evidence. The affordable-participation goal is also worth noting: a benchmark that only large labs can run produces a narrow evidence base.

Compression changes the requirement

Han et al.'s "Deep Compression" describes a three-stage pipeline — "pruning, trained quantization and Huffman coding" — that reduces storage requirements "by 35x to 49x without affecting their accuracy" on the models studied. Pruning reduces the number of connections by "9x to 13x", and quantization reduces the bits representing each connection from 32 to 5.

The related integer-arithmetic quantization work addresses efficient inference with integer operations. Together, these results show that the hardware requirement is partly a function of how the model is represented, not only of its architecture. A model that fits in memory after quantization can run on cheaper hardware, which changes the economics of deployment.

The important caveat is that these figures are results on specific models and tasks. They demonstrate the mechanism — compression reduces memory and compute needs — rather than guaranteeing a 35× reduction for your model.

A worked example of a training budget

The following numbers are hypothetical and illustrative.

A team wants to train a model for 10 days on 8 GPUs. Cloud GPU rental is 3 per hour per GPU.

  • Cost: 8 GPUs × 24 hours × 10 days × 3 = 5,760.
  • If a better-connected instance is 20% faster per GPU but costs 50% more, the run takes 8 days: 8 × 24 × 8 × 4.5 = 6,912. The faster option costs more overall despite finishing sooner.
  • If the team reduces the dataset to 70% and observes proportional training time, the first option becomes 7 days at 4,032 — but the model may be worse.

Two checks decide the choice:

  1. Is the job compute-bound or memory-bound? If the model does not fit in GPU memory, adding faster compute does nothing; the team needs more memory or a smaller model, which is where compression becomes relevant.
  2. What is the cost of delay? If a two-day earlier result lets the team ship a feature in a critical window, the faster option may be worth the premium. That is a business decision, not a hardware decision.

A third consideration is utilisation. GPUs billed by the hour but idle for half the day because of data-loading bottlenecks waste half the budget. Profiling the pipeline before buying more accelerators is usually the highest-return step.

Training and inference have different constraints

The word "GPU workload" hides two very different problems.

Training processes large batches repeatedly over many epochs. Throughput dominates, memory capacity determines the largest model or batch that fits, and a run can be scheduled around availability. Interruptions are costly but often recoverable from checkpoints.

Inference serves individual requests with tight latency expectations. A model may be small enough to fit on cheaper hardware, and the relevant metrics become time to first token, requests per second, and cost per million tokens. Compression techniques such as pruning and quantization matter more here, because they reduce memory bandwidth and allow deployment on smaller devices.

The implication is that an organisation's accelerator strategy cannot be a single number. A team may need high-memory accelerators for training and a different, cheaper profile for serving — or it may compress a model enough to serve on CPUs, which is where the Deep Compression results become commercially meaningful.

What the strategic framing misses

Three qualifications matter.

A benchmark is not your workload. A model with unusual sparsity, long sequences, or heavy preprocessing may rank differently from the benchmark leaderboard. Measure your own workload.

Supply and lock-in are real but variable. Access constraints and vendor ecosystems shape decisions, but the specific constraints change quickly. A claim about current availability should be dated, and an architecture that assumes a single supplier creates risk.

Compression has limits. Aggressive quantization can degrade accuracy on some tasks, and pruning changes the model in ways that may affect fairness or robustness. The compression papers report accuracy preservation on their evaluated models; that is a result, not a universal property.

The counterargument to buying into accelerators — that better algorithms reduce the need — is partly right. Algorithmic improvements have historically reduced compute requirements for a given capability.

But demand has grown faster than those savings, which is why the resource remains strategic.

A decision test

  1. Profile the workload before choosing hardware: compute-bound or memory-bound?
  2. Report benchmark context — workload, precision, software versions, power — with every performance claim.
  3. Measure utilisation, not just throughput, to find idle spend.
  4. Evaluate compression as a way to change the hardware requirement, with accuracy checks.
  5. Compute total cost including rental, power, and engineering time, as in the worked example.
  6. Price the cost of delay separately from the cost of compute.
  7. Avoid single-supplier assumptions in long-lived architecture.

For the deployment side, see Edge AI; for the economics, see Cloud Cost Optimization.

References & Further Reading