[KongchangAI]
ConceptFuRA-S算法

FuRA-S

A forward-only optimization algorithm that trains only the spectral coordinates (column-block singular values) of pretrained weight blocks. It can approximate the weight changes induced by RLVR and enables efficient adaptation with an extremely small number of parameters.

Timeline (last 90 days)

Nov 4

MobileSpec's core algorithm FuRA-S trains only the block spectral coordinates of the pretrained weights (the singular values of column blocks), rotating the singular vectors by independently rescaling these blocks.

Unverified50%
Nov 4

The updates of FuRA-S can be rewritten as a transformation of the layer input, and thus can be adapted to INT4 quantized models without modifying their weights.

Unverified50%
Nov 4

The expert models trained by FuRA-S are only 1-5 MB in size, with the proportion of updated parameters only 0.015-0.05%.

Unverified50%
Nov 4

On RLVR benchmarks, FuRA-S is competitive with full-parameter evolution strategies (ES), and outperforms ES on LoRA adapters and quantized ES on INT4 models.

Unverified50%
Nov 4

FuRA-S can be viewed as performing evolution strategies (ES) on low-dimensional spectral coordinates, and thus is more efficient than full-parameter ES under the same forward-pass budget.

Unverified50%

All Facts (5)

Source Articles