Quantization in Practice for Beginners
A beginner's guide to quantization: how integer arithmetic replaces floating point, what the two source methods add, and where accuracy can break.
A trained neural network usually stores weights and activations as floating-point numbers, which are expensive to move and compute. Quantization maps those values to lower-precision integers so inference can run faster and with a smaller memory footprint. The beginner's problem is knowing which method applies and what accuracy you might give up.
Direct answer: Quantization reduces the numerical precision of a model's weights and activations, with the Jacob et al. method training for integer-arithmetic-only inference and the Han et al. method combining pruning, trained quantization and coding. The boundary: smaller and faster does not mean free, because reduced precision can change outputs and must be validated on the target task.
Delta: This article adds a method-to-decision map beyond the two sources: it maps quantization choices to a deployment decision, with claim status explicit.
The mechanism
The Jacob et al. work shows that a neural network can be trained so that inference uses integer arithmetic only. In practice this means weights and activations are stored as integers and the multiply-accumulate operations are performed with integer math, with scaling factors chosen so the integer values approximate the original floating-point distribution. The practical consequence is that both model size and the cost of each operation fall, which matters most on hardware where integer units are cheaper or where floating-point support is limited.

The word trained in that method is important. Quantization is not just a post-hoc rounding of weights; the source describes training with quantization in the loop so the network adapts to reduced precision. The consequence is that accuracy loss can be much smaller than a naive conversion suggests, but it requires access to training data and a training pipeline.
The Han et al. work addresses compression more broadly. Deep Compression combines three steps: pruning connections that contribute little, quantizing the remaining weights so fewer bits represent them, and encoding the result with Huffman coding. The consequence is a much smaller model, often far below the original storage size. The method is distinct from integer-only inference: it targets size and storage, and its quantized weights may still be decoded before use.
Choosing between approaches depends on the deployment constraint. If the goal is fast inference on modest hardware, integer-arithmetic-only inference from the first method is the direct fit. If the goal is fitting a large model into limited storage or bandwidth, the compression pipeline is the relevant one.
Both trade precision for efficiency, and both require measuring the effect on the actual task rather than assuming that a smaller model is a harmless one.
A worked example
The following is hypothetical and illustrative. A team wants to run an image classifier on a low-power device with limited memory. Three checks guide the choice. First, constrain the target: if the device's accelerator supports integer math efficiently, a quantization-aware trained model from the Jacob et al. approach is the fit. Second, measure the size budget: if storage is the binding limit, add compression steps from the Han et al. pipeline and verify that decoding cost is acceptable. Third, validate on the real task by comparing accuracy before and after, using a held-out set. In this hypothetical case, the team keeps accuracy within an agreed margin and ships the integer model, while leaving the compression pipeline for a later storage-constrained variant.
Evidence status
| Claim | Status | Source | Verified | Recheck when |
|---|---|---|---|---|
| Integer-arithmetic-only inference can run a trained network efficiently | source-reported | Jacob et al. 2018 | 2026-11-18 | Hardware support or methods change |
| Deep Compression combines pruning, trained quantization and Huffman coding | source-reported | Han et al. 2015 | 2026-11-18 | Compression benchmarks are updated |
| Quantization error is task-dependent and needs validation | inferred | Derived from the sources | 2026-11-18 | Model family or task distribution changes |
The table marks the two method descriptions as reported from their papers and the validation claim as an inference drawn from them. Reported here means the sources state these results; it does not mean this article reproduced them. The inferred row depends on the model and task, so it should be rechecked whenever either changes.
What it cannot do
Quantization cannot preserve a model's behavior exactly. It reduces the number of bits available to represent a value, and that reduction introduces error. For many tasks the error is small enough to be harmless, but the method offers no guarantee that a given model will tolerate it. Sensitive outputs, such as those near a decision threshold, can flip when precision falls.
A beginner should treat accuracy retention as something to measure, not something the method promises.
The hidden cost is the work around the model, not the arithmetic. Quantization-aware training needs training data, a modified training loop, and validation, which can be more expensive than the inference savings in a small project. Compression adds pruning schedules and an encoding step, each with parameters to tune. The modest integer model may also require a runtime that supports the chosen format, which is a dependency that appears only at deployment time.
A common failure mode is quantizing the wrong parts of a network. Some layers are more sensitive than others, and applying a uniform scheme can damage accuracy even when the average looks fine. Another failure is a mismatch between the quantization format used in training and the one supported by the deployment runtime, which can force a fallback to slower code. A third is validating on the training distribution rather than the distribution the model will actually see, which hides the very errors that matter.
The boundary is the task, not the model. A quantized network may score well on a benchmark and still be unacceptable for a specific use, because the benchmark does not weight the errors that matter to that use. The two source methods describe techniques for efficiency; neither defines what accuracy is good enough. That decision belongs to the deployment context, which must set a tolerance and test against it.
Hardware changes the answer. A new accelerator with better integer support can make one method worthwhile where another was not, and a change in model architecture can shift which layers are sensitive. Teams that quantize once and ship indefinitely should recheck accuracy whenever the model, the runtime, or the target device changes. Keeping a small held-out set and a recorded tolerance makes that recheck cheap and turns quantization from a one-off trick into a maintained decision.
Counterargument: "The counterargument that quantization is a lossless step applied blindly at the end of training is fair for a tolerant model with a forgiving metric and incomplete when outputs sit near decision thresholds."
A decision test
- State the deployment constraint: speed, memory, storage, or hardware support.
- Identify which of the two source methods addresses that constraint.
- Check whether training access exists for quantization-aware training.
- Choose the precision and which layers to quantize.
- Validate on a held-out set from the real task distribution.
- Confirm the deployment runtime supports the chosen format.
- Record the accuracy tolerance and the condition that triggers revalidation.
These steps keep the efficiency gain honest by tying it to a measured tolerance. The point is not to avoid quantization but to know which errors it introduces and whether the target task can absorb them. For related deployment safety work, see AI Risk Assessment.
References & Further Reading
- Jacob et al., 2018. Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference — Integer-only inference and why it needs a quantisation-aware training procedure.
- Song Han et al., 2015. Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding — The pruning-and-quantisation pipeline behind most on-device compression.