FlowGenesis: Let Neural Networks Decide How Big to Grow

Title

FlowGenesis: Let Neural Networks Decide How Big to Grow

Meta Description

FlowGenesis proposes a representation-flux-driven framework for neural network self-organization, using neuron splitting and pruning to let networks autonomously adjust capacity during training, validated on image/text classification and continual learning.

Article Content

Deep neural networks typically rely on an architecture fixed before training, yet their internal representations keep evolving throughout training. This tension between "fixed architecture and flowing representations" raises a new question: can network structure adjust itself in response to changes in representation? The FlowGenesis framework answers with a resounding yes. Through neuron splitting and pruning, it lets networks autonomously increase or decrease capacity during training, with no predefined target architecture or separate architecture search.

Core Mechanism: Driving Structural Evolution with Representation Flux

FlowGenesis's key innovation is the introduction of "Representation Flux," a local statistic that captures changes in a neuron's average activation across mini-batches. Simply put, it measures how much a neuron's output fluctuates.

A high-flux neuron means its representation keeps changing during training and may carry important learning signals, so it is split into two child neurons, with the splitting process designed to approximately preserve the network's existing function. Low-flux, low-activity neurons are pruned because they contribute little to the current task. This mechanism lets network capacity grow and shrink dynamically during training, driven entirely by data-driven representational change rather than hand-crafted rules.

This design breaks free from the traditional architecture search framework: no search space needs to be defined, and no large set of candidate architectures needs to be evaluated—the network completes structural adaptation during gradient descent itself.

Representation Flux is a local statistic characterizing the stability of a neuron's activation: over several consecutive mini-batches, record a neuron's average output activation, then compute how much these values change over time (e.g., variance or the mean difference between adjacent batches). The higher the value, the more unstable the neuron's response pattern during training, often corresponding to a feature direction that has not yet converged and is still carrying gradient signals. Unlike pruning criteria based on weight magnitude, representation flux directly observes forward behavior rather than parameter norms, so it can capture neurons that are "small in weight but violent in activation fluctuation."

Function preservation in neuron splitting: when duplicating a neuron into two, directly summing their outputs would double the network's total response and destroy the learned mapping. A common approach is to have each child neuron inherit half of the parent's output weights (e.g., multiplying outgoing edge weights by 0.5) and keep both parameters identical at the initial moment, so the network function is unchanged at the instant of splitting; subsequent gradient descent then gradually differentiates the two as they receive different gradients, giving each its own specialization. This technique aligns with the net-widening idea in network morphism methods such as Net2WiderNet, except that the timing of splitting is determined online by representation flux rather than specified in advance.

Experimental Validation: Different Initial Sizes Converge to Similar Ranges

The researchers conducted experiments on image and text classification tasks. Results show that even when networks start with different initial sizes, after FlowGenesis adjustment the final neuron count ranges on the same dataset tend to converge, while predictive performance remains comparable.

This means networks have a tendency to "self-regulate" to an appropriate capacity without needing to precisely set width before training. For practitioners, this lowers the trial-and-error cost of architecture selection—you can start from a rough initial size and let the network find the right complexity on its own.

Structural Adaptation in Continual Learning

FlowGenesis has also been extended to continual learning scenarios. In task-incremental learning settings, representation-guided structural adaptation can reduce the number of neurons and parameters allocated to subsequent tasks while maintaining accuracy comparable to fixed-width Progressive Neural Networks.

The core challenge of continual learning is catastrophic forgetting, and progressive networks avoid forgetting by allocating a new column for each task, at the cost of parameters growing linearly with the number of tasks. FlowGenesis dynamically adjusts structure based on representational feedback, curbing parameter bloat while maintaining performance, offering a more efficient capacity allocation approach for continual learning.

Progressive Neural Networks are a multi-task architecture proposed to mitigate catastrophic forgetting: each time a new task arrives, a new column of network layers is added, and features from old columns are connected to the new column via lateral connections (adapters), with old column parameters frozen to completely avoid forgetting. The cost is that parameter count grows linearly with the number of tasks—the more tasks, the more bloated the model. FlowGenesis performs structural adaptation on top of this, essentially using representation flux to determine which neurons truly need expansion for a new task, thereby compressing newly added capacity while preserving performance on old tasks—an approach of "growing on demand" rather than "adding a column for every task."

Insights and Limitations

The value of this work lies in proposing a new paradigm: architecture is not specified before training but is a product of the feedback loop between representation learning and structural change. It suggests that network capacity can self-organize through activity-dependent mechanisms, much like biological nervous systems.

Of course, current experimental scale is limited, covering only image and text classification, and the design that approximately preserves function during splitting may introduce extra computational overhead. But the direction is worth watching: as models grow larger, letting structure "grow" itself may be more sustainable than manual design.

(The End)

One-line Summary

FlowGenesis uses representation flux to drive neuron splitting and pruning, letting networks self-organize their architecture during training.

Paragraph Summary

Addressing the tension between "fixed architecture and flowing representations" in deep networks, FlowGenesis proposes using representation flux (the magnitude of change in a neuron's average activation across mini-batches) as a local statistic to guide structural evolution: high-flux neurons split into two child neurons that approximately preserve the original function, while low-flux, low-activity neurons are pruned, so network capacity dynamically grows and shrinks during gradient descent in response to data-driven representational change, without a predefined search space or separate architecture search. Image and text classification experiments show that networks of different initial sizes converge to similar neuron count ranges with comparable performance; in task-incremental continual learning, the method reduces parameter allocation for subsequent tasks, with accuracy comparable to fixed-width progressive neural networks, curbing parameter bloat.