Representation Flux
A local statistic characterizing the stability of neuron activations, measuring the degree of representation fluctuation of a neuron by computing the magnitude of change in the average activation value over several consecutive mini-batches.
Timeline (last 90 days)
FlowGenesis proposes "Representation Flux" as a local statistic to capture changes in a neuron's average activation on mini-batch data.
Unlike pruning criteria based on weight magnitude, Representation Flux directly observes forward behavior rather than parameter norms, and can therefore capture neurons that are "small in weight but fluctuate sharply in activation".
All Facts (2)
FlowGenesis proposes "Representation Flux" as a local statistic to capture changes in a neuron's average activation on mini-batch data.
50%UnverifiedUnlike pruning criteria based on weight magnitude, Representation Flux directly observes forward behavior rather than parameter norms, and can therefore capture neurons that are "small in weight but fluctuate sharply in activation".
50%