Evaluation of FP6 Quantization in Small-Scale Image Super‑Resolution DNNs: Condition Exploration and Comparison of Residual and Direct Generation Architectures

Overview

We compared image quality and file size when using a small image‑upscaling DNN with FP16 weights converted to FP6. In the search for optimal quantization settings, the PSNR improved on four validation images but decreased on another set of four images. Afterwards, we evaluated using 174 images divided into five groups by acquisition series, as well as 12 images that had not been used until the FP16 baseline was fixed. Even with roughly the same number of parameters, the degradation caused by uniform conversion to FP6 differed markedly between residual‑type and direct‑generation‑type models. The comparison of the 12 final FP6 images was added after viewing the FP16 results, so it does not constitute a new blind evaluation.

This article records the verification of whether the improvements discovered during the search persist on other images, and how degradation varies between models under uniform FP6 quantization. It reports the results of a small image‑upscaling DNN and does not directly demonstrate LLM accuracy or the speed of FP6‑specific hardware.

Background and Questions

In the previous article (source), we examined FP6 formats E3M2 and E2M3 and considered how to allocate the limited bits to the weights. We also discussed whether we could, like JPEG, change the representation, or, like Zopfli, try many candidates during the quantization stage. However, at that point we had no results of quantizing a model ourselves and measuring the outcome.

Therefore, we created quant-search-exp . Rather than tackling an LLM right away, we started with an image‑upscaling DNN that has only a few hundred weights. The experimental results and procedure are also documented in Issue #9 verification report .

There are two points we want to investigate. When only the quantization parameters are searched while keeping the FP6 inference format unchanged, does image quality improve even on images that were not used for selection? And, even when using a different model with roughly the same number of parameters, is the image‑quality loss caused by uniform FP6 quantization about the same? Initially we measured things in an exploratory manner, and based on those results we restructured how the images were divided and how the models were compared.

Experimental Method

Small Image‑Upscaling Model

The first model we built takes a low‑resolution JPEG, upscales it by a factor of two in both dimensions using bicubic interpolation, and then corrects the result with a small DNN. It receives the RGB values of the 3 × 3 pixel region after upscaling—27 values in total—passes them through a ReLU hidden layer of 16 units, and outputs three RGB correction values to be added to the central pixel. The entire network contains only 499 parameters. It was not designed to compete with general super‑resolution models in terms of image quality, but rather to be small enough to allow repeated evaluation of different quantization settings.

For training, we extract crops of up to 512 × 512 pixels from the training images, downscale them by half, and save them as JPEGs with quality 70; the original crops serve as the ground truth for learning a two‑fold upscaling. The same procedure is used to generate artificial low‑resolution JPEGs for evaluation. The metric is the PSNR computed from the RGB pixels of the reconstructed image and the original image, with higher values indicating closer similarity to the original. Re‑compression error when saving the output as a JPEG is not included.

The file storing the model’s weights in FP16 is 1,006 bytes. In the FP6 version, the 480 connection weights are divided into blocks of 32, each block assigned a shared E8M0 scale. The 19 biases remain in FP16, reducing the total file size to 431 bytes. In other words, this does not mean that “all 499 parameters were converted to 6 bits.” Both versions use FP32 during computation, so this experiment cannot claim any speedup from FP6‑specific hardware.

Exploratory Comparison of Quantization Settings

First, we fixed the inference format to FP6 E2M3 and varied only the scale, clipping, and rounding method used during quantization. For the 15 blocks, we examined 26 candidates each—different from the default setting—for a total of 390 candidates, comparing them sequentially using the PSNR of the four validation images. We adopt only the blocks where a good candidate is found, rather than performing an exhaustive search of all combinations. The model’s file size and inference engine remain unchanged.

At this point we compared the four validation images with another set of four images. However, although the latter four images were not used in the selection of the 390 candidates, they had already been referenced during the earlier investigation of training conditions. This is not a final blind evaluation but an exploratory comparison.

Preliminary Comparison of Training Image Count

Next, we added 131 images to the original 18 training images and compared training lengths of 8, 32, and 64 epochs. The model remained the same 499‑parameter network, with three random seeds. The number of pixels extracted per epoch was fixed at 200,000, whether the training set consisted of 18 images or 149 images.

Follow‑up Test with Separated Acquisition Series

Subsequently, we increased the number of additional images to 148, combining them with the existing 26 images for a total of 174 images. These were grouped into 26 sets based on acquisition month and sequential camera IDs. To avoid any overlap between training and evaluation sets, the data were split into five folds, and we compared the results using seeds 42 and 43. The four images previously used for evaluation were also included in this set of 174, and each fold placed those images either in the training or evaluation portion.

We used two types of models. One is the previously described residual‑type model, where the DNN corrects an image that has been upscaled by bicubic interpolation. The other is a direct‑generation‑type model that directly produces high‑resolution 2 × 2 pixels from the 3 × 3 neighborhood of the low‑resolution image. The latter does not use a bicubic‑upsampled image during inference. We matched the parameter counts to roughly 500 and 650, respectively, and both models have only a single hidden layer. Training was performed for 24 epochs with JPEG quality 70 and a learning rate of 0.0003, using 800,000 teacher pixels per epoch and the same number of weight updates. However, because the input‑output tasks differ, the learning problems are not entirely identical. Because the follow‑up experiment recorded the model type and width in a separate file format, even the residual‑type model with the same 499 parameters has a slightly different file size compared with the original model.

The 12 images set aside were not used for training or cross‑validation until the conditions were fixed; after retraining the model on all 174 images, we evaluated it using these 12 images. No exact pixel matches or closely matching dHash values were found, though this does not rule out the possibility that the images share similar shooting scenes.

Starting from the saved FP16 model weights, we uniformly converted only the connection weights to MXFP6 E2M3 or E3M2, while leaving the biases in FP16. An E8M0 scale was applied to each block of 32 weights, and no block‑wise format selection or search among the 390 candidates was performed. The FP6 model is also expanded to FP32 at load time before computation.

Results

Exploration of Quantization Settings

On the validation set, the uniform E2M3 baseline improved by +0.031 dB. However, on the other four images it decreased by ‑0.018 dB. While increasing computation during generation can align the results with the images used for selection, this alone does not guarantee better performance on unseen images. The numerical results and trial logs are available at Search results JSON .

Preliminary Comparison of Training Image Count

For the original four evaluation images, averaged over three seeds, the FP16 model trained on 18 images for 8 epochs achieved 31.308 dB, while training for 64 epochs reached 31.438 dB. In contrast, training on 149 images for 64 epochs yielded 31.401 dB. When using the additional images and training for 32 epochs, the PSNR on the four validation images improved, but on the original four evaluation images it fell below the performance obtained with only the 18 images under the same conditions. At least under these conditions, we cannot claim that “adding more images leads to improvement.”

Because the total number of extracted pixels was fixed, increasing the number of images reduces the chance that any individual image is selected. The effect of adding data cannot be fully separated from the effect of how many times each image is observed. Moreover, the original four evaluation images had already been seen. We did not treat this preliminary comparison as conclusive and instead restructured how the images were divided for evaluation. The records, including seed‑specific values, are posted at Preliminary comparison of training amount and added images .

Evaluation with Separated Acquisition Series and Final 12 Images

The PSNR of the final 12 images using only bicubic upscaling is 33.405 dB. The next value is computed by summing the RGB‑pixel errors across all two seeds; it is not the average of the per‑image dB values.

Differences in FP16 models

At the FP16 stage, the residual type only slightly outperformed bicubic for narrow widths: +0.042 dB at width 16 and +0.031 dB at width 21. The direct‑generation type was lower than bicubic for both widths. When pairing FP16 models using the same image and seed, even in a 5‑fold split that separates the capture sequences, the direct‑generation type was on average about 500 parameter sets ‑0.329 dB and about 650 parameter sets ‑0.321 dB worse than the residual type. The 95 % confidence intervals obtained from bootstrap using 26 capture groups as units were also [‑0.395, ‑0.272] dB and [‑0.394, ‑0.256] dB, respectively. However, the definition of a capture group is based on inference from nearby images, so the groups are not necessarily completely independent samples. Even for the final 12 images, among the 24 cases (12 images × 2 seeds for each size), the direct‑generation type was superior in only one case. The PSNR differences obtained by summing all pixels in the table do not match the average of the per‑image paired differences because the two calculations use different methods.

Differences when converting to FP6

The impact of quantization differed further. In the 5‑fold split, the drop from FP16 using E2M3 was 0.015–0.024 dB for the residual type, whereas for the direct‑generation type it was 1.953–2.103 dB. Even for the final 12 images, the residual type with width 21 showed a decline of 0.028 dB, while the direct‑generation type fell by 1.699–2.111 dB. In other words, even when the number of parameters and hidden layers is similar, sensitivity to a uniform FP6 recipe varies greatly.

On the other hand, the residual type with width 16 dropped by 0.281 dB when converted to E2M3 for the final 12 images, while the E3M2 configuration yielded a higher value. E2M3 does not always win. Determining what caused these effects will require examining the weights and layers separately. The fact that the direct‑generation type’s E3M2 performance fell even further should not be taken as a generic superiority of one format over another; it should be interpreted as a result that depends on the model and the chosen scale.

The aggregated values and per‑image values can be found at 5‑fold analysis , Analysis of the final 12 images , and FP6 comparison results . The original images are not included in the repository.

Discussion

In the previous article we hypothesized that by selecting the format and scale and performing an exhaustive search during training, one could make the most of the limited bit budget. In practice, however, the PSNR gain observed on the four images used for selection did not guarantee improvement on other images. Even if we increase the number of search iterations, as is done with Zopfli, it is difficult to determine what constitutes a “good result” when quantizing a model. If we expand the set of candidates, we must separate the images used to select candidates from the evaluation images that are never touched.

Another insight we gained is that looking only at the post‑quantization numbers can lead to misidentifying the cause. The direct‑generation type was already lower than the residual type at the FP16 stage, and it was even more sensitive to uniform FP6 quantization. It is better to first assess the quality of the FP16 model itself, and then examine the difference when quantizing the same weights separately. One cannot simply claim that “having only one hidden layer makes it weak for FP6.” The residual type also has only one hidden layer, yet under certain conditions it suffered almost no degradation.

Limitations

There are also caveats concerning the unused 12 images. While the FP16 evaluation was performed with the FP16 conditions fixed, the FP6 comparison was added after looking at the FP16 results. Therefore, the 12 FP6 images should not be regarded as a new blind evaluation. The set of 12 images is small and may include images that are close in the capture sequence. The five‑fold split based on capture month and sequential numbering does not guarantee that the scenarios are completely independent.

The focus of this study is an image‑super‑resolution DNN with only a few hundred parameters, and it does not directly demonstrate the quantization accuracy of LLMs. The evaluation images are also artificially generated low‑resolution JPEGs. Moreover, because inference was not performed on an FP6‑specific accelerator, the results cannot be used for speed comparisons. Although the residual and direct‑generation types were matched in parameter count, they are not identical in how the output is produced. We have not yet identified which elements of the model architecture are responsible for the observed differences.

Conclusion

If we proceed further, we should isolate which layers or blocks of the direct‑generation type are most sensitive to quantization, and measure separately the FP16 image quality versus FP6 for the same image splits when the width or depth is varied. Then we can resume the search for optimal format and scale choices. At present, before deciding how to allocate bits, it is essential to firmly define what constitutes an improvement as the evaluation metric.