Collaborative Learning

The researchers propose Corroborative Learning (CL), which achieves mutual enhancement among models through task-response reconstruction without eliminating disagreement.

In CL, each model generates task responses and reconstructs response increments using collaborators' representations, but the reconstruction process excludes source information, separating fitting and evaluation on disjoint groups.

When linear reconstruction is used and collaborators' representations are fixed, CL's loss function is equivalent to a positive semi-definite directional regularizer that imposes stronger constraints on directions with lower reconstructability.

CL was experimentally validated on six language tasks, 24 models, and 36 pair combinations.

Experimental results show that CL outperforms capacity-matched independent adaptation on all tasks and yields positive gains in most directed model combinations.

Pairwise comparisons show that CL has broad advantages over transfer learning and mutual learning methods, and that different collaboration rules obtain different gains from complementarity.

Results on low-label language and vision tasks support CL's cross-domain applicability.

CL's core design adds no extra inference dependencies and requires no direct output or feature matching.

The key difference between CL and transfer learning and mutual learning is that CL does not require consistency of predictions or features among models, but instead exploits complementarity through task-response reconstruction.

Future CL may be applied to scenarios such as multimodal learning and federated learning, where complementarity among models is particularly valuable.