Previously, I removed whole MLP channels from Gemma 4 E2B and saved smaller matrices. At the nominal 10% pruning stage, total weight size fell about 3.5%. Generation became a little faster, but MLPs account for only about 34% of this model. Concentrating on them limits how much smaller the whole model can become.

This time I tried two approaches. One trims vocabulary around English and Japanese to shrink the large embedding tables. The other keeps the total MLP pruning budget fixed while changing how much each layer loses. Neither updates the weights through training.

With a 64k vocabulary, the text model went from 4.629B to 2.565B parameters, a 44.6% reduction. Its bf16 weights fell from 9.26 GB to 5.13 GB, while loss on the same text increased about 3.4% in English and 1.1% in Japanese. Layer-wise MLP allocation also improved on uniform pruning, although its size reduction remained modest. Where I pruned made a substantial difference to the storage savings.

The Large Tables Scaled With Vocabulary Size

The target was the text portion of Gemma 4 E2B, running on an Apple M3 Mac with 16 GB memory, MLX, and mlx-lm 0.32.2, as before. I keep the text model’s size separate from the original files that include image and audio functionality.

The original vocabulary has 262,144 entries. Input embeddings share weights with the output layer, and another embedding table supplies per-layer inputs. The implementation used contains embed_tokens, embed_tokens_per_layer, and the use of embedding weights for output. Together, the two tables accounted for roughly 2.75B parameters, about 60% of the text model counted in the experiment.

The report’s approximate parameter formula is about 1.88B + V × 10,496, where V is retained vocabulary size. Reducing these tables looked more promising for storage than pruning part of the MLPs. However, removing output tokens makes the cost depend on the languages and text being processed.

Selecting Vocabulary From English and Japanese

I counted about 2.46 million tokens from the WikiText-2 training split and 24.61 million from the Japanese validation split of Wiki40B. Tokens were prioritized by the average of their relative frequencies in the two languages. Adding raw counts would have favored the much larger Japanese sample.

Special tokens and all 256 byte tokens were retained. The experiment produced stages labeled 64k, 32k, and 16k; these labels are used below.

Slicing table rows is not sufficient. Old and new token IDs must agree across both embedding tables, the output, and the tokenizer. BPE also has merge rules: retaining a final token without the intermediate pieces needed to construct it can change tokenization.

The procedure tokenized each retained token’s string in isolation and also retained the intermediate tokens needed along that path. Merge rules survived in their original order when both components and their result were retained. Following every possible merge path was unsuitable here because about 160,000 tokens had multiple possible merges.

Text containing removed tokens can fall back to smaller fragments or bytes rather than immediately becoming unknown tokens. The number of tokens needed for the same text can still increase. Representability does not imply unchanged tokenization or quality.

Equal Logits Do Not Mean Equal Distributions

For inputs expressible entirely with retained tokens, the corresponding embeddings keep their original values. The report compared the original and sliced models on six unseen paragraphs, at 855 positions, and found zero difference in the retained-token logits.

Those are scores for corresponding output candidates. Softmax renormalizes over the surviving candidates, so removing candidates changes the probability distribution. Probability assigned to removed tokens and changes in input tokenization can affect quality. Equal retained logits do not establish identical generated text.

The expectation that more than 99.9% of tokens in unseen text would remain expressible with the original segmentation did not hold.

VocabularyEnglish OOV rateEnglish token increaseJapanese OOV rateJapanese token increase
64k1.45%+1.6%0.33%+0.4%
32k5.69%+7.0%1.12%+1.6%
16k12.86%+18.9%3.43%+4.9%

OOV here means original token occurrences belonging to the removed vocabulary, rather than text that could not be encoded. These measurements use the datasets’ test splits, which were not used to select vocabulary.

Of the English OOV tokens at 64k, 42% had never appeared in the vocabulary-selection text. They included fragments of proper names. The small English selection corpus is a possible explanation, but the experiment did not test a revised selection to establish that cause.

Paragraphs whose original segmentation used only retained vocabulary kept the same segmentation after trimming. At 64k, this held for 1,401/1,401 English paragraphs and 43,879/43,879 Japanese paragraphs. These denominators count paragraphs satisfying that condition; they do not mean every evaluation paragraph matched unconditionally.

Comparing Quality in Bits per Byte

Splitting text into smaller tokens changes the difficulty of predicting one token. Original and trimmed token-level perplexities therefore do not provide a comparison in the same unit.

Instead, I compared bits per byte, or BPB: negative log-likelihood on the same text, converted to bits and divided by byte count. Lower is better. Evaluation used text equivalent to about 66,000 tokens from each test split, with paragraph boundaries held fixed across vocabularies.

VocabularyEnglish BPBChangeJapanese BPBChange
Original 262,1440.801—0.780—
64k0.829+3.4%0.789+1.1%
32k0.933+16.5%0.818+4.9%
16k1.200+49.7%0.907+16.3%

At 64k, the English BPB difference was +0.0275, with a 95% confidence interval of [+0.0235, +0.0317]. Japanese increased +0.0087, with [+0.0060, +0.0116]. These came from 10,000 paired bootstrap resamples at paragraph level: small but significant degradations in both languages. Displayed values are rounded, so subtracting the table entries does not reproduce every digit of the reported differences.

The reported probability the original model assigned to removed tokens was 1.7% for English and 0.6% for Japanese at 64k. Both that quantity and BPB degradation increased at 32k and 16k. Being able to remove most vocabulary did not mean that further reductions were harmless.

The Saved 64k Model Was Substantially Smaller

I saved the 64k model and compared it with the original in bf16 and 8-bit quantization.

MetricOriginal bf1664k bf16Original 8-bit64k 8-bit
Parameters4.629B2.565B4.629B2.565B
Weight size9.26 GB5.13 GB4.92 GB2.73 GB
Peak generation memory9.61 GB5.48 GB5.47 GB3.28 GB
Generation throughput18.3 token/s21.0 token/s31.5 token/s37.4 token/s

Parameters fell 44.6%. Generation throughput increased about 14.3% in bf16 and 18.8% in 8-bit. Original and trimmed models were launched alternately twice each; the report compares medians of six measurements per configuration. The throughput ranges did not overlap in these trials.

However, the report does not attach input/output lengths or individual timing measurements. I do not assume the previous article’s 512-input/256-output condition applies here. Since tokenization changes the output unit, comparing the time to complete the same text would also benefit from characters or bytes per second.

Combined with 8-bit quantization, vocabulary trimming reduced weights from the original bf16 model’s 9.26 GB to 2.73 GB. But the BPB measurements above evaluate vocabulary trimming, not the combined trimmed-and-quantized model’s quality. Quantization alone barely changed perplexity in the earlier experiment; that does not establish the same behavior for this combination.

Redistributing a Fixed MLP Pruning Budget

The second experiment continued the earlier MLP work. Instead of removing the same fraction from all 35 layers, I assigned more pruning to layers where it caused less damage.

Within each layer, channels were still selected using act_err, the earlier measure of a channel’s effect on layer output. The change was how many channels each layer lost.

I measured mean negative log-likelihood, or NLL, after pruning 25% and 50% from each layer individually: 35 layers times two stages, or 70 evaluations. Allocation measurements used WikiText-2 test windows 65–96, a set of 32 windows. Comparison used the same first 64 windows as the previous experiment. These sets do not overlap, although both belong to the same test split.

The 0%, 25%, and 50% points for each layer formed a piecewise-linear lower convex envelope of loss increase. A greedy procedure allocated removals in blocks of 64 channels, with a 50% cap per layer for sensitivity-based allocation. Total removals matched the uniform 10%, 15%, 20%, and 30% stages.

The Last Layer Was Sensitive

Pruning 25% from the last layer, layer 34, increased mean NLL by +0.042. That was about five times the next-largest increase, layer 12’s +0.0090. Layers 0, 1, and 14 were around zero to +0.001. The same fraction removed had very different effects across layers.

The expectation that individual damage could simply be added was slightly outside its target range. Pruning 25% from every layer increased NLL by +0.281, versus +0.183 for the sum of individual measurements: about 1.53 times, just above the predefined upper bound of 1.5. Individual measurements could guide allocation without accurately describing independent, additive effects.

Nevertheless, measured-sensitivity allocation beat uniform pruning at every compared stage.

Total removals equivalent to uniform stageUniform perplexitySensitivity allocationStatistics-only allocation, as reported
10%10.2710.1811.38
15%10.6910.51—
20%11.4411.0112.60
30%14.1812.7114.61

Unpruned perplexity was 9.628. The report used 10,000 paired bootstrap resamples over the 64 windows and found sensitivity allocation significantly better than uniform pruning at all four stages. The gap grew with the pruning budget.

A cheaper approach comparing calibrated act_err scores across layers failed. Both layer-normalized and unnormalized scores were tried; rank correlations with measured allocation were between -0.30 and 0.14. The normalized approach attempted to remove 80–91% from the most sensitive layer, layer 34. This was a separate control from sensitivity allocation with its 50% cap.

The table reproduces the reported statistics-only aggregate. Separate numeric results for both normalized and unnormalized variants were not attached, so that column cannot resolve their detailed ranking. At least in this experiment, a useful within-layer measure did not transfer directly to allocating a budget across layers.

Saving the 15%-Equivalent Allocation

Among the tested stages, I saved the largest pruning budget within a 10% perplexity degradation: the 15%-equivalent allocation. Uniform 15% increased perplexity about 11.1%, while sensitivity allocation at that budget increased it about 9.2%.

MetricOriginalUniform 10%Sensitivity 15%
Parameters4.629B4.466B4.401B
Peak generation memory9.61 GB9.31 GB9.18 GB
Generation throughput18.5 token/s19.8 token/s20.3 token/s
Perplexity9.62810.26710.511

Sensitivity allocation at 15% reduced parameters about 4.9% and increased throughput about 9.8% over the original. Throughput figures are medians from separately launched processes, in a measurement series distinct from vocabulary trimming. Even the original model runs at 18.3 in one table and 18.5 in the other. Dividing across tables would not provide a controlled performance comparison between the methods.

Moving from uniform 10% to sensitivity 15% also raises perplexity from 10.267 to 10.511 in exchange for additional compression. Allocation improved quality at the same removal budget, but these two saved models do not have equal quality.

A model with different widths across layers cannot be described by the single intermediate_size used previously. Its saved files include an mlx-lm model definition and require trust_remote_code=True to load. The additional reduction gives up some convenience of loading through the standard configuration alone.

The Two Methods Have Not Yet Been Combined

Vocabulary trimming reduced large tables, while layer-wise allocation reduced damage at a fixed MLP pruning budget. The report does not evaluate a model applying both. Adding or multiplying their size and speed improvements would not establish a combined model’s performance.

Vocabulary evaluation covers BPB on English and Japanese Wikipedia-derived text. Conversation, code, extraction, other languages, and unusual symbols remain untested. MLP allocation uses only WikiText-2 perplexity, and the same evaluation results helped select the saved stage. Additional tests independent of that selection and stability on other text remain open.

The tables organize aggregate experiment reports. I also provide machine-readable transcriptions of the rounded values. The reported code locations are experiments/vocab_trim and experiments/gemma4_e2b_mlp_channels/layer_alloc.py. The code, issues, PRs, and unit tests were unavailable for inspection in this writing environment. This article does not represent a fresh execution or independent reproduction.

Vocabulary lists, ID mappings, BPE merges, model revisions, saved-model hashes, per-paragraph/window losses, and individual timing measurements are not attached either. The aggregate tables are insufficient to regenerate confidence intervals or saved models. Preserving those details would make it easier to test how much can be removed while retaining the desired behavior under other conditions.

After the previous MLP experiment reduced the whole model by only 3.5%, the result felt rather modest. Trimming vocabulary made a much larger difference to storage. Meanwhile, simply relaxing uniform pruning changed how much performance the MLP retained. Finding where size can be removed and measuring how much damage it causes both mattered.