Weight-Only Quantization - MachineLearningMastery.com
During autoregressive decoding, a transformer reads its weights for every new token. At small batch sizes, moving these weights can take more time than computation. Quantization reduces this cost by storing weights with fewer bits. For example, an INT8 weight uses half the raw storage of an FP16 or BF16 weight. Quantization can also make […]
MachineLearningMastery.com · Adrian Tam · https://www.facebook.com/MachineLearningMastery/