Single Graph Hypernetwork Zero-Shot Generates Weights for Multiple Architectures, Open-Source Implementation Achieves Cross-Modal Generalization

New Possibilities for Hypernetworks from Zero-Shot Weight Generation

Recently, a developer on Reddit open-sourced a Graph HyperNetwork (GHN) project that can predict all weights for unseen ResNet, MLP-Mixer, MobileNet, and even Transformer models in a single forward pass. This means that for entirely new architectures, a usable initial set of weights can be obtained without any fine-tuning, significantly reducing the number of subsequent training steps required.

The project demonstrates zero-shot performance on multiple datasets, covering three modalities: vision, text, and 1D waveforms. Although some results still have room for improvement, its cross-architecture and cross-modal generalization capability is noteworthy.

How Hypernetworks Work

The core idea of a Graph HyperNetwork is to take the computational graph of a target architecture as input, combine it with a dataset embedding, and directly output all weights for that architecture. The developer trained a single GHN that can handle computational graphs of different structures, thus possessing generalization capability to unseen architectures.

During training, the GHN learns the mapping from computational graphs to weights. At inference time, one simply provides the computational graph of a new architecture and the dataset embedding to generate all weights in one shot, without iterative optimization. This "zero-shot" capability allows the GHN to serve as a warm start for training, reducing the number of subsequent gradient descent steps.

HyperNetwork is a class of meta-learning models whose core idea is to have one network (the hypernetwork) generate weights for another network (the target network). Traditional hypernetworks are typically designed for fixed architectures, taking task or dataset information as input and outputting weights for that architecture. In contrast, a Graph HyperNetwork (GHN) takes the computational graph of the target architecture (nodes represent operators, edges represent data flow) as input, encodes the structural information of the computational graph via a graph neural network, and then combines it with a dataset embedding to output the weights corresponding to each node in the graph. This design enables the GHN to handle computational graphs of arbitrary structures, thereby possessing generalization capability to unseen architectures. The computational graph representation is key to the GHN's architecture-agnostic nature, as it avoids the need to manually design hypernetwork structures for specific architectures.

Zero-Shot Performance Benchmarks

The developer provided zero-shot accuracy on multiple datasets (table below, data from the original post):

Model	ResMLP	ResNet2D	MLP-Mixer	MobileNet2D	MLP	CNN1D	Transformer
MNIST	93.96%	93.66%	89.58%	74.53%	/	/	/
FashionMNIST	83.99%	75.88%	78.59%	/	/	/	/
KMNIST	78.10%	62.15%	/	/	62.15%	/	/
Waveforms1D	/	/	/	/	/	99.84%	89.38%
20Newsgroups	/	/	/	/	/	71.30%	48.34%

As can be seen from the table, on image datasets such as MNIST and FashionMNIST, ResMLP and ResNet2D achieve relatively high zero-shot accuracy (e.g., ResMLP reaches 93.96% on MNIST), while MobileNet2D only achieves 74.53% on MNIST, indicating that zero-shot transfer for complex architectures remains challenging. On the 1D waveform dataset, CNN1D performs almost perfectly (99.84%), and Transformer achieves 89.38%. On the text dataset 20Newsgroups, CNN1D and Transformer achieve zero-shot accuracy of 71.30% and 48.34%, respectively, indicating that the text modality is more difficult.

Interestingly, the GHN had never seen these architectures during training, so these results are true zero-shot evaluations. The developer mentions that these weights can serve as training starting points, reducing the number of iterations required for convergence.

Zero-shot in the context of hypernetworks refers to the hypernetwork directly generating weights for a target architecture (or dataset) that it has never seen during training, without any fine-tuning or gradient updates. This is similar to zero-shot learning in traditional transfer learning, but the object is network weights rather than class labels. The GHN's zero-shot capability relies on the diverse computational graph structures seen during training, enabling it to learn a general mapping from graph structures to weights. When evaluating zero-shot performance, the generated weights are typically used directly in the target architecture for testing without any training. Therefore, zero-shot accuracy reflects the degree of generalization of the hypernetwork to architectures and tasks.

Cross-Architecture and Cross-Modal Generalization

The highlight of this work is that a single model simultaneously handles multiple architecture families (ResNet, MLP-Mixer, MobileNet, Transformer) and multiple modalities (vision, text, waveforms). Traditional hypernetworks are often designed for specific architectures or tasks, whereas the GHN achieves architecture-agnosticism through computational graph representation.

However, the performance differences across architectures and modalities also reveal uneven generalization. For example, on image tasks, simple MLP-like architectures (ResMLP, MLP) perform well, while architectures with depthwise separable convolutions such as MobileNet2D have lower zero-shot accuracy. On text tasks, Transformer's zero-shot accuracy is lower than CNN1D, possibly related to the training data distribution.

Dataset Embedding is part of the hypernetwork's input, used to encode the characteristics of the target dataset (such as number of classes, image resolution, modality, etc.), enabling the hypernetwork to adjust the generated weights according to the dataset. In the GHN, the dataset embedding is typically a learnable vector optimized for each dataset during training. At inference time, for a new dataset, a suitable embedding needs to be provided, which limits the hypernetwork's generalization to unseen datasets. The developer plans to use an encoder to encode random datasets to replace the dataset embedding, aiming to enable the hypernetwork to handle arbitrary datasets without learning a separate embedding for each dataset.

Developer's Next Steps

The developer states that the next goal is to use an encoder to encode random datasets to replace the dataset embedding, thereby handling more unseen datasets. Additionally, he hopes to scale up the model, but is currently limited by computational resources (using only Kaggle's free resources).

The project is open-sourced, and the developer will share the Kaggle link in the comments. For researchers with limited resources, this provides a reproducible starting point.

Limitations and Outlook

Although the zero-shot results are impressive, the GHN is currently validated only on smaller datasets, and accuracy for some architectures is low. In the future, if trained on larger-scale datasets and more architectures, its generalization capability is expected to improve further. Moreover, how to apply the GHN to more complex tasks (such as object detection and segmentation) remains an open question.

Overall, this open-source work demonstrates the potential of Graph HyperNetworks in cross-architecture weight generation, providing new ideas for automated machine learning and few-shot scenarios.

Note: This article is based on the content of the original Reddit post and does not add data not mentioned in the original post.