LLM Quantization from the Perspective of FP6: Where to Allocate Limited Bits

While researching quantization for LLMs, I found myself wondering in what situations FP8 would actually be useful. Further investigation revealed that FP6 and FP4 also exist. I became curious about how to store values in a 6‑bit format—an oddly sized width—and how much precision a floating‑point number with so few bits could actually represent.

When you list the values representable by FP6, the spacing between adjacent numbers becomes clearly visible—a detail that is usually overlooked with the floating‑point formats we use daily. Even with the same 6 bits, the spacing changes depending on how the bits are allocated between the exponent and the mantissa. That raises the question: could we choose the representation to match the distribution of the weights?

At first it was simply a question of “how do you store a 6‑bit value?” but as we examined the set of representable numbers, the issue shifted to which values should be preserved exactly. That led me to recall image compression, where a limited number of bits is allocated according to the bias in the data. I also thought of Zopfli, which spends more time on compression to achieve better results.

The question I want to explore in this article is: how should we allocate a limited number of storage bits, and in what representation, so that the model’s behavior is least likely to be disrupted? First, we examine the specific values that would be lost with FP6, then formulate hypotheses that we compare with the ideas behind JPEG and Zopfli and with existing quantization research. Finally, we consider what should be regarded as improvement and how to measure it.

This is not a presentation of implementation or evaluation results for a quantization scheme. Below, I separate the hypotheses that stem from observing numeric representations from the conclusions that can be drawn from prior work.

Background: Questions About 6‑bit Formats

Quantization and Storing 6 bits

LLM weights are often stored in formats such as FP16 or BF16. Quantization approximates these with fewer bits—e.g., INT8, INT4, FP8, FP6—to reduce model size and the memory bandwidth needed to fetch the weights. If suitable arithmetic units or kernels are available, this can also speed up computation, but merely reducing the bit‑width does not guarantee a performance gain.

If we stored the 30 billion weights of a 30‑billion‑parameter model entirely in 16 bits, the weights alone would occupy roughly 60 GB. Using 8 bits reduces that to about 30 GB, 6 bits to about 22.5 GB, and 4 bits to about 15 GB. Considering that a GPU must fetch massive amounts of weight data from memory for every computation, this difference isn’t just about storage capacity. However, in practice metadata such as scale factors are added, and at runtime memory is also required for activations, KV caches, and other non‑weight data, so these figures do not directly translate to the total memory needed.

In computers we commonly see widths of 8 bits, 16 bits, 32 bits, and 64 bits. I had imagined that a 6‑bit value would be placed in an 8‑bit slot with the remaining two bits discarded, but that would leave the storage footprint unchanged from 8 bits. In reality, multiple values can be packed at the bit level. Four 6‑bit values occupy 24 bits, which fits into three bytes.

6 bits × 4 values = 24 bits = 3 bytes

DRAM physically stores bits, while the CPU typically treats memory as an address space organized in bytes. There is no need to assign a separate address to each 6‑bit chunk; you simply extract the desired values from the read byte stream. For example, you can read three bytes at once and then pull out four 6‑bit values from them.

However, values that cross a byte boundary require shifts and masks when they are extracted. You also need to unpack the values into a computational format and apply a scale factor. Even when processing large numbers of weights in bulk with SIMD or GPU kernels, packing and unpacking are not free operations. Even if you reduce the amount transferred from memory by 25%, a slow unpacking step may not make the 6‑bit representation faster than 8 bits. The claim “6 bits uses 25 % less storage than 8 bits” is distinct from the claim “6 bits is faster than 8 bits.”

This distinction will later tie into the discussion of Zopfli. While it may be acceptable to make the model’s training computations more intensive, if a complex unpacking step is required for every inference, that cost will be incurred continuously.

Examining the Spacing Between Adjacent Values in FP6

FP8 is a choice that reduces data size and transfer volume while preserving a reasonable range and precision. So what happens with even smaller formats such as FP6 or FP4? The OCP MX format specification OCP MX format specification includes formats that use FP8, FP6, and FP4. Here we will look at the FP6 variants E3M2 and E2M3. The names indicate the number of exponent bits and mantissa bits, excluding the sign bit.

Looking only at positive values before applying a scale, the distances between adjacent representable values are as shown below.

For example, consider how the numbers from 8 to 14 are represented with E3M2. The exponent determines a base magnitude of 8, and the two‑bit mantissa yields four possible patterns—1.00, 1.01, 1.10, and 1.11—corresponding to 8, 10, 12, and 14. When the exponent moves to the next range, the base becomes 16, producing 16, 20, 24, and 28. Even though the mantissa’s step size stays the same, the absolute distance doubles each time the exponent increments.

With E2M3, the three‑bit mantissa allows eight distinct values within the same exponent range. However, because one fewer bit is allocated to the exponent, the overall range that can be represented before scaling is narrower. In this OCP FP6 format, encodings for Inf and NaN are not reserved, and subnormal numbers are supported. What we are comparing here are two concrete formats defined in the specification, not an informal 6‑bit version of a standard floating‑point type.

In E3M2, the smallest spacing and the largest spacing differ by a factor of 64. Between 8 and just below 16 the spacing is 8, 10, 12, 14, so the value 11 cannot be represented. For values 16 and above, the spacing becomes even coarser, 16, 20, 24, 28. Converting 11 to E3M2 does not preserve the original 11; it must be rounded to either 10 or 12. Which direction is chosen depends on the rounding rule, but either way introduces an absolute error of 1.

Near zero, the spacing is coarser, only 0.0625. For example, rounding 0.1 to the nearest representable value, 0.125, yields an absolute error of 0.025, which is far smaller than the error of 1 incurred when rounding 11 to 10 or 12. However, expressed as a proportion of the original value, the former error is 25 % while the latter is about 9 %. If you look only at absolute error, 11 appears worse; if you consider relative error, 0.1 appears worse. It is impossible to summarize “FP6 error” with a single number without taking into account the actual distribution of weight values and the scale applied to those values.

As the exponent grows, the absolute spacing widens. Because the mantissa is only 2 bits, each exponent range can hold only a few values. When the exponent changes, the spacing changes as well. This stair‑step pattern makes the characteristic behavior of floating‑point numbers—usually barely noticeable in FP32—obviously visible.

Even the familiar IEEE‑754 FP32 exhibits the same property: the distance between adjacent representable values varies with the exponent. However, with a 23‑bit mantissa the spacing is so fine that we rarely notice it. FP6 has far fewer patterns, so the mechanics of floating‑point arithmetic become visible in concrete numbers, which is intriguing.

Both E3M2 and E2M3 use 6 bits for storage, giving 64 possible bit patterns. The difference lies in where those 64 states are placed on the number line. According to the OCP specification, the largest normal value before scaling is 28 for E3M2 and 7.5 for E2M3. Allocating more bits to the exponent expands the range, while allocating more bits to the mantissa makes the spacing finer within that range.

In the actual MX format, a shared scale is applied to each block. For example, if the scale is 2, the value 8, 10, 12, 14 representable by E3M2 actually corresponds to 16, 20, 24, 28 after scaling, and the displayed interval is doubled. Conversely, using a smaller scale yields finer granularity, but it may cause larger values to exceed the representable range. Therefore, the choice of E3M2 versus E2M3 and the choice of scale are not independent.

In the OCP MX format, 32 values share a single scale. Making this unit smaller makes it easier to match the distribution of each block, but it increases the proportion of storage needed for scale metadata. Conversely, using larger blocks reduces the metadata overhead, yet a single outlier can pull the scale, causing the spacing for the remaining values to become coarser. Even if you look only at the 6‑bit values themselves, you cannot determine the actual storage efficiency.

In AMD’s MXFP4 and MXFP6 Quantization Verification , it is explained that because the shared scale covers most of the value range, MXFP6 tends to achieve better practical accuracy with E2M3 (which expands the mantissa) rather than E3M2 (which expands the exponent). Moreover, the rounding method used when converting the scale to the E8M0 format also impacts accuracy; omitting round‑to‑nearest‑even reduces precision. The quantization scheme includes not only allocating 6 bits per element but also deciding how the shared scale is set.

Rather than saying “they carry different amounts of information,” it is clearer to view the difference as how the same 6‑bit representation capacity is allocated between range and precision. Of course, this does not mean that the statistical information actually carried by the values is identical for both.

Hypothesis: Changing How Limited Bits are Used

What becomes clear from looking at FP6 is that it isn’t merely that “6 bits make the values coarse.” Even with the same 6 bits, you can choose which values to represent more finely. By making this choice on a per‑matrix or per‑block basis, and even by selecting the coordinate system and quantization procedure used to express the values, it may be possible to preserve the model’s behavior better while keeping the same storage capacity.

From here on, we consider the following steps in order: choosing a format that matches the data distribution, changing the coordinate system, allocating precision to important regions, and searching for candidate configurations during implementation. Similar prior works are referenced within each discussion, but I have not personally measured the effectiveness of combining these approaches.

Can the representation be chosen according to the value distribution?

Even with the same 6 bits, the arrangement of representable values changes. So, could we switch representations based on the distribution? Large language models contain massive weight matrices, and the distribution of values differs across matrices, channels, and even smaller blocks. Is it necessary to apply the same representation to all of them?

Suppose there are two blocks. In one, most values are tightly clustered within a narrow range; in the other, large values are interspersed at distant points. For the former, it may be more advantageous to allocate bits to the mantissa to distinguish nearby values rather than to the exponent to cover an unused wide range. For the latter, prioritizing fine granularity could cause the loss of large values, which would be problematic. From this perspective, selecting a scale and representation method on a per‑block basis becomes conceivable.

However, you cannot mechanically decide that E2M3 should be used for the first case and E3M2 for the second merely by looking at the distribution. The actual step size of each format changes depending on where the shared scale is placed, and there is also the option of storing outliers in a different format. Moreover, even with the same numerical error, its impact on inference results varies across blocks. At present, the idea is simply that there is room to experiment with different candidates and compare them.

Research in this direction already exists. MicroMix combines MXFP4, MXFP6, and MXFP8 on a per‑channel basis, assigning higher precision to locations where low‑precision quantization would incur large errors. AdaMX addresses the fact that appropriate precision‑recovery methods differ for each block. Both approaches are similar in that they avoid forcing the entire model into a single format, but they do not correspond to my own idea of selecting between E3M2 and E2M3 on a per‑block basis.

The mixed‑precision evaluation in the AMD article also combines MXFP4 and MXFP6, not the result of switching between E3M2 and E2M3 on a per‑block basis. That aspect needs to be considered separately.

Moreover, as the amount of metadata indicating which format is used grows, the inference side will require kernels that can distinguish those formats. MicroMix designs the quantization scheme together with the matrix‑multiply kernel precisely because this issue cannot be resolved by the storage format alone. In some situations, it is easier to keep the common INT4 format while varying the per‑block scale and rounding method.

Changing the coordinate system like in JPEG

When dealing with data whose information distribution varies by location, there is a similar concept in image compression. This brings JPEG to mind. The standard DCT‑based JPEG first transforms each 8 × 8 pixel block into frequency components and then quantizes them. In the JPEG specification T.81 specification, a different quantization value can be assigned to each DCT coefficient.

An 8 × 8 block yields 64 coefficients from the 64 pixels. The coefficient in the upper‑left corner represents the overall brightness of the block, while the other coefficients indicate how finely the pixel values vary horizontally and vertically. Rather than rounding all pixel values with the same step size, the quantization strength can be adjusted for each coefficient. Even if fine variations are heavily reduced, they are often less noticeable to the human eye. Conversely, damage to the overall brightness or shape is immediately apparent.

What makes this example interesting is that it does not designate any single pixel as “important.” The 64 pixels are rearranged into 64 separate components, and the tolerable error for each component can be varied. In other words, the transformation moves to a coordinate system where it becomes easier to choose which parts of the information to discard, rather than using the original pixel grid. The quantization table can be seen as embedding that decision into the storage format.

It seems there may be ways to quantize LLM weights other than simply rounding them in the original coordinate system. For instance, if a linear layer y=Wx uses an invertible transformation T, it can be expressed as follows:

$$ x’=Tx,\qquad W’=WT^{-1},\qquad W’x’=Wx $$

Here, T denotes a matrix that possesses an inverse. Even after applying the transformation, before quantization W'x' matches the original Wx. Replacing only the weights with T(W) would produce a different model. Therefore, either the input side must be adjusted accordingly, or the transformation must be absorbed into the neighboring layer so that the original computation remains unchanged.

At this point, no compression has been achieved. The compression benefit appears later, when W' is coarsely quantized; if this preserves the model’s output better than directly quantizing the original W, then a gain is realized. Simply making the transformed numbers appear “smaller” is insufficient; the inverse‑transform calculations and the absorption of the transformation into adjacent layers must also function correctly.

This line of research has already been pursued. QuaRot spreads outliers by rotating the model’s computations in a way that preserves accuracy, making low‑bit quantization easier. Furthermore, SpinQuant shows that not all rotations are equivalent; it learns rotations that improve post‑quantization accuracy. In other words, it goes as far as searching for a coordinate system that is more amenable to quantization.

The JPEG DCT and the rotations applied to models are not the same operation. JPEG separates an image into its frequency components and discards information according to human visual perception. In contrast, LLMs work to preserve the original computation as much as possible while seeking a distribution of values that is less prone to quantization error. This does not claim that the components imperceptible to the human eye correspond directly to any particular part of the LLM’s weight matrices.

Nevertheless, we have discovered that there is an alternative path beyond simply rounding the original numbers. Moreover, even with rotations that preserve the original computation, SpinQuant shows that post‑quantization accuracy can differ. If so, the transformation itself should not be chosen arbitrarily but can be explored as a candidate.

Distributing precision like JPEG’s quantization tables

In addition to the transformation, we also need to consider where to allocate the limited bits. Just as JPEG does not assign the same precision to all 64 coefficients, an LLM does not need to store every weight with the same number of bits. For example, one might allocate 8 bits to important channels, 4 bits to ordinary parts, 2–3 bits to regions with little impact, and keep outliers in a separate high‑precision format. This is less a concrete implementation proposal than an illustration of how precision can be traded within the same storage budget.

If, out of 100 weights, 20 are represented with 8 bits and the remaining 80 with 4 bits, the average bit‑width for the core becomes 4.8 bits—lower than the 6 bits required if all 100 were stored at 6 bits. Using this margin, it might be advantageous to protect the especially sensitive 20 weights. Conversely, if the errors accumulated in the other 80 weights degrade performance, this allocation would fail. Conclusions cannot be drawn solely from the average bit count.

However, a larger absolute weight value does not necessarily mean it is more important. Even for a single product y=wx, if the weight’s quantization error is Δw, the output error becomes Δy = Δw × x. Even with the same Δw, the output varies more where the input x is large. In an actual model, subsequent layers further propagate these errors, sometimes attenuating them and sometimes amplifying them.

Therefore, unless you examine the distribution of the activation inputs as well as the error that propagates to later layers, you cannot know which values need to be protected. AWQ identifies important channels from the activations and adjusts the weight scaling to protect them. Rather than mixing high‑precision values, the interesting aspect is that it reduces error while keeping the same low‑bit format. GPTQ uses approximate second‑order information to suppress quantization error more than simple rounding.

What I found intriguing is that, instead of uniformly choosing between 4‑bit and 6‑bit, you can vary the allocation while keeping the average number of bits constant. Bits become not just a storage width but a budget for preserving the model’s behavior, allowing you to decide where to reduce error.

Of course, mixing formats adds metadata and kernel branching overhead. The previously mentioned average of 4.8 bits was calculated without accounting for format‑identifying information, block‑level scales, alignment, or separately stored outliers. You need to assess whether the benefits outweigh those additional costs.

Can we try many candidates like Zopfli?

When deciding how to allocate representation and precision, you won’t know which choice to adopt until you actually try them. That reminded me of Zopfli . Zopfli spends time searching during compression to find a more compact representation. Its output is standard DEFLATE, so it can be read by existing decompressors.

Even for large language models, quantization is performed once before the model is distributed, and the same quantized weights are used repeatedly for inference. If that’s the case, couldn’t we try many candidates during the quantization stage and pick the one that best preserves the model’s behavior? Even if the search takes several hours—or in some cases, days—the cost may be recouped if the model is used for a long time.

If we look for the approach most similar to Zopfli, we fix the inference format up front. For example, we keep the already‑supported INT4 format, the group size, and the method for storing scales unchanged, while varying the scales, clipping thresholds, and rounding directions for each value during creation. Each candidate is evaluated on calibration data, and the best ones are retained. Ultimately, if we can export the weights and scales as the same INT4 format, the inference side does not need to know how many candidates were explored.

On the other hand, if we treat block size and bit allocation as variables as well, the final data layout and the required kernels can change. This moves beyond the Zopfli analogy of “compressing hard while the decompressor stays the same.” It seems better to separate the discussion of optimizing within an existing format from the design of a new format itself.

Zopfli performs lossless compression, whereas LLM quantization is usually lossy. Searching for the smallest file size is not the same as preserving model accuracy. What we want to borrow is the idea of increasing the number of trials during creation while keeping the inference format unchanged.

With a compressed file you can verify that it matches the original data exactly. However, there is no straightforward way to guarantee that a quantized model will produce the same output as the original model for every possible input. A candidate that looks good on calibration data may perform poorly on other tasks or on longer contexts. The claim that a candidate is “the most accurate” is only valid within the range of evaluated candidates and data. The more computation you devote to the search, the less you can afford to skimp on the evaluation side.

SignRound follows this direction. During the quantization stage it adjusts how weights are rounded and how values are truncated, without adding extra processing at inference time. Unlike Zopfli, which exhaustively tries every candidate, it uses gradients to search efficiently. If you increase the search effort, you need to compare not only brute‑force enumeration but also such approximate optimization methods.

Evaluation: What counts as “broken”

Before trying many candidates, you need to define what constitutes a good candidate. Even if the stored bit count is low, it is problematic if the model produces different answers from the original. Conversely, merely preserving the numerical values of the weights does not guarantee that the model’s behavior is preserved.

From weight error to model output

Let the weights be denoted as W, the choice of scale, rounding method, etc., be θ, and the quantized weights be (Q_{\theta}(W)). If we measure the difference in the weights themselves, we can write it as follows.

$$ \theta^*=\underset{\theta}{\operatorname{argmin}}\quad\lVert W-Q_\theta(W)\rVert_F^2 $$

This is easy to understand, but, as we saw with y=wx earlier, even a small difference in weights can lead to a large change in output depending on the input applied. Conversely, even a large weight error may have little impact on the model’s behavior if it occurs in a rarely used region. Relying solely on the Frobenius norm, which treats every element of the matrix equally, does not capture the true cost to the model.

One step further is to use the activations entering a layer and compare the layer’s output before and after quantization. If you want to evaluate the entire model, you can consider comparing the original model f_W with the quantized model on calibration data D.

$$ \theta^*=\underset{\theta}{\operatorname{argmin}}\quad \mathcal{L}\bigl(f_W(D),f_{Q_\theta(W)}(D)\bigr) $$

AQLM compresses the weights by taking into account both the inputs and outputs of the calibration data, and it also adjusts codebooks that span multiple layers. It is an example of a method that not only brings the weight values closer together but also strives to preserve the model’s computational results. However, because AQLM uses a proprietary representation that relies on codebooks, it does not directly satisfy the Zopfli‑style constraint of outputting to an existing INT4 format.

θ can encompass scale, block partitioning, bit allocation, transformation, handling of outliers, and so on. There are also choices for (\mathcal{L}). Metrics such as squared error of layer outputs, deviation of the next‑token distribution, perplexity, and accuracy on specific tasks do not measure the same thing. For example, a candidate that yields a probability distribution close to the original model may rank differently from one that achieves the same correct answers on only a few tasks.

The range of data used for evaluation also matters. A quantization selected based on calibration data heavy with code may not behave the same way on other languages or on longer inputs. If you overfit only to the calibration set, the model could break on unseen inputs, so the chosen method should be validated on independent data or real‑world tasks as well. The more candidates you explore, the greater the risk of over‑optimizing for the same evaluation set.

Comparing including storage size

If the only goal is to preserve the model’s behavior, you could simply retain high‑precision weights. However, when storage size is also part of the objective, the problem resembles rate‑distortion trade‑offs used in image compression. This is not merely a metaphor. Rate Distortion for Model Compression analyzes neural‑network compression within a rate‑distortion framework and examines objective functions that can be applied to quantization and similar techniques. However, it does not constitute a finalized quantization method tailored for large language models.

$$ \theta^*=\underset{\theta}{\operatorname{argmin}}\quad \left[\mathcal{L}\bigl(f_W(D),f_{Q_\theta(W)}(D)\bigr) +\lambda R\bigl(Q_\theta(W),\theta\bigr)\right] $$

R denotes the total number of bits required to store not only the weight values themselves but also the scale, format‑identifying information, and any separately stored outliers. λ determines the relative weighting of size and degradation. If all weights are fixed to the same INT4 format, the bit count for the core weights remains roughly constant, so the discussion then focuses on finding quality differences among the candidates. When mixing formats, R also changes.

The same idea can be expressed as a constrained optimization problem: limit the stored bit count to a certain maximum and minimize (\mathcal{L}) within that bound. If candidate A uses fewer bits but sacrifices a little accuracy, and candidate B uses more bits while preserving accuracy, which is preferable depends on the application and the memory budget. There is no single “best” quantization method; the trade‑off between storage size and error must be decided on a case‑by‑case basis.

Moreover, it’s not just the number of bits used for storage; inference speed also becomes a limiting factor. Even if R is small and the loss \(\mathcal{L}\) is low, a format that requires returning to the CPU each time the model is deployed—without a dedicated kernel—remains impractical. To make the Zopfli‑style approach of compressing a trained model once and reusing it many times work, it is essential to keep the cost of the “reuse” side low.

Entropy and Importance Are Separate

If the same quantized values appear frequently within a block, they might be stored more compactly through encoding. However, low entropy of a value and the fact that it is unnecessary for the model are distinct matters. Even if a block consists entirely of zeros, which can be described succinctly as “all zeros,” that does not mean those zeros can be replaced with other values. In fact, those zeros might be crucial for determining the model’s output. Low entropy should not be interpreted as “something that can be discarded,” but rather as “something that can potentially be represented correctly with fewer bits.”

Conversely, a block whose values are scattered and hard to encode may have little impact on the model’s output. In such cases, it is better to apply strong quantization rather than using many bits to preserve fine‑grained values. Mixing these two criteria and assuming that “places with more information are more important” can lead to a misallocation of bits.

What we need to consider is measuring separately the cost of storing the quantized value sequence and the impact of altering a block on the model’s output. For example, there are locations where even a small number of stored bits has a large effect on output, and other locations where using many bits yields only a small effect.

The entropy referred to here depends on the quantization representation and the encoding method used. If all weights are stored as fixed‑width INT4, each weight occupies four bits regardless of any bias in the values, unless additional compression is applied. Introducing variable‑length coding may shrink the file, but the GPU would need decoding and indexing operations to read specific locations of a matrix quickly. Low theoretical entropy is distinct from being able to store the data compactly in a format that is convenient for inference.

How to Treat the Search Space

Can Quantum Search Be Used When There Are Too Many Candidates?

Combining choices of transformation method, scaling, bit allocation, block size, and outlier handling rapidly multiplies the number of quantization schemes to try. Even assuming only four formats are selected for each of ten blocks, the total number of combinations becomes 4^10 = 1,048,576. In practice, each chosen format also brings its own set of scaling candidates, and errors between blocks can interact in later stages. Optimizing each block individually does not guarantee that the overall model achieves the best possible performance.

If there are N candidates and each is evaluated in the same way, a naïve exhaustive search would require O(N) evaluations. This raises the question of whether quantum computers could accelerate such a search. The “quantum search” mentioned here is a different concept from the “quantization” of LLMs. Although the names are similar, one refers to reducing the representation of weights, while the other denotes a search method based on quantum computation.

If we idealize an unstructured candidate search, algorithms such as Grover’s search (arXiv:quant-ph/9605043) and the minimum‑finding algorithm of Dürr and Høyer (arXiv:quant-ph/9607014) can locate a solution or a minimum with roughly O(√N) oracle queries. For example, if there are N = 2^40 candidates, a full exhaustive evaluation would require 2^40 calls, whereas the square‑root scaling would be on the order of 2^20.

Grover’s search algorithm

Dürr and Høyer’s minimum finding

However, this is merely a comparison of the number of queries. If a single query entails “choosing a bit allocation for a candidate, constructing a quantized model, and measuring loss on calibration data,” that operation itself is extremely costly. Simply substituting N into Grover’s formula does not even provide a realistic estimate of execution time.

To make this a genuine quantum search, the evaluation must be reformulated into a form that can be efficiently executed on a quantum circuit—essentially, turned into an oracle. Issues remain regarding how to handle a large model’s weights and calibration data, how to compute the evaluation within quantum states, and what the cost will be for actual hardware that includes error correction. Moreover, if the candidates possess structure, classical methods such as gradient‑based optimization, local search, or approximate importance estimation may be far more efficient than exhaustive search. While formulating the problem as a massive discrete optimization naturally evokes quantum search, it remains a theoretical question rather than a practical proposal at this stage.

Discussion: Constraints on the Inference Side and Untested Aspects

Ultimately, what we seek is a format that uses little storage, preserves the model’s behavior, and can be processed quickly during inference. The notion of first transforming the representation and then allocating precision—as in JPEG—and using computation for search during creation, as in Zopfli, connects here. However, we are not saying “use DCT because it’s JPEG‑like” or “perform exhaustive search because it’s Zopfli‑like.” We are borrowing the way to decompose the problem: what to compress, which errors are tolerable, and where computational effort can be applied.

On the other hand, if the transformation must be performed each time during inference, or if different formats need to be read for each block, the resulting overhead could erase any bandwidth savings. In addition to size and error, the speed on the target GPU and kernel must also be measured.

To verify these ideas, we should first standardize the comparison conditions. For example, fix the same model, the same calibration data, and an upper bound on the number of stored bits, and use a uniformly quantized version as the baseline. Then, introduce block‑wise variations—such as different scaling factors, format choices, or rounding methods—one at a time, and observe the changes in output on an independent evaluation dataset. When testing transformations or mixed formats, the comparison should include not only the weight files but also the storage required for metadata and the actual inference runtime. Otherwise, one might overlook outcomes like “the model’s accuracy improved, but it becomes harder to use during inference.”

The methods mentioned so far—MicroMix, QuaRot, SpinQuant, AWQ, and SignRound—each address different aspects of the problem. This article is not presenting a new technique based on these hypotheses; rather, it is a stage where we have organized which ideas drawn from JPEG and Zopfli overlap with existing research and where the constraints on inference formats and evaluation still remain. As for quantum search, there is no evidence from theoretical query counts that can substantiate a claim of practical speedup.

Summary

While investigating FP8, I became aware of FP6 and FP4, and seeing the coarseness of FP6 in concrete numbers led me to consider how to allocate a limited number of bits to different parts of a model. It is not enough to decide only how many bits to reduce to; one must also choose how to partition the data, which representation to convert to, and which errors the model will tolerate as “broken.” This can be framed as a search problem aimed at minimizing the required bit count while keeping the model’s behavior within an acceptable range of change.

There are prior works on representation transformation, precision allocation, quantization‑time optimization, and rate‑distortion. Thanks to those, it has become clear that “the same number of bits yields the same accuracy” is not a simplistic truth.

Even so, looking at FP6’s 8, 10, 12, 14 and asking, “If 11 cannot be represented, where should we allocate the available precision?” led surprisingly far. It pushed me to think beyond merely selecting an FP6 format—considering coordinate changes, protecting important regions, searching for candidates during quantization, and even defining what counts as “broken.” To compare approaches in practice, I would first gather candidates that produce the same inference format, then measure size, speed, and behavior on calibration data and an independent evaluation dataset.