Context Encoders
A seminal paper proposing self-supervised feature learning through an image inpainting task. The core idea is to train a convolutional encoder-decoder network to fill in missing regions of an image, compelling the network to learn semantic image representations. It is considered a foundational work in self-supervised learning.
Timeline (last 90 days)
The paper "Context Encoders: Feature Learning by Inpainting" proposes self-supervised feature learning through an image inpainting task.
A Context Encoder is essentially a convolutional encoder-decoder structure, where the encoder compresses a corrupted image into a compact latent feature representation and the decoder regenerates the missing content.
The encoder in the original paper is derived from the first five convolutional layers plus a pooling layer of AlexNet, turning a 227×227 input into a 6×6×256 representation.
The Context Encoder's encoder is trained from scratch with random weights, trained solely on the inpainting task rather than for ImageNet classification.
The channel-wise fully connected layer is the most core architectural innovation of Context Encoders, used to address the problem that convolutional layers cannot connect distant positions within a single feature map.
Connecting a 256-channel, n×n feature map to the decoder with a full FC layer requires m²n⁴ parameters, which the paper notes would exceed 100 million parameters.
The grouped fully connected layer gives each feature map its own n²×n² weight matrix with no connections between different feature maps, reducing the parameter count from m²n⁴ to m·n⁴.
Because the channel-wise FC layer does not mix channels, the paper subsequently adds a stride-1 convolution to propagate information across channels.
Context Encoders use a decoupled joint loss function comprising a reconstruction loss (normalized masked L2 distance) and an adversarial loss, handling continuity with context and the multimodality of output separately.
The reconstruction loss only compares predictions with ground truth inside the hole; the mask M zeros out pixels the network has already seen, preventing the model from scoring by copying the context.
3 more timeline events
All Facts (13)
The paper "Context Encoders: Feature Learning by Inpainting" proposes self-supervised feature learning through an image inpainting task.
50%UnverifiedA Context Encoder is essentially a convolutional encoder-decoder structure, where the encoder compresses a corrupted image into a compact latent feature representation and the decoder regenerates the missing content.
50%UnverifiedThe encoder in the original paper is derived from the first five convolutional layers plus a pooling layer of AlexNet, turning a 227×227 input into a 6×6×256 representation.
50%UnverifiedThe Context Encoder's encoder is trained from scratch with random weights, trained solely on the inpainting task rather than for ImageNet classification.
50%UnverifiedThe channel-wise fully connected layer is the most core architectural innovation of Context Encoders, used to address the problem that convolutional layers cannot connect distant positions within a single feature map.
50%UnverifiedConnecting a 256-channel, n×n feature map to the decoder with a full FC layer requires m²n⁴ parameters, which the paper notes would exceed 100 million parameters.
50%UnverifiedThe grouped fully connected layer gives each feature map its own n²×n² weight matrix with no connections between different feature maps, reducing the parameter count from m²n⁴ to m·n⁴.
50%UnverifiedBecause the channel-wise FC layer does not mix channels, the paper subsequently adds a stride-1 convolution to propagate information across channels.
50%UnverifiedContext Encoders use a decoupled joint loss function comprising a reconstruction loss (normalized masked L2 distance) and an adversarial loss, handling continuity with context and the multimodality of output separately.
50%UnverifiedThe reconstruction loss only compares predictions with ground truth inside the hole; the mask M zeros out pixels the network has already seen, preventing the model from scoring by copying the context.
50%UnverifiedThe paper experimented with both L1 and L2 reconstruction losses and found the difference to be insignificant.
50%UnverifiedMAE (Masked Autoencoders) replaces the bottleneck constraint with a high masking ratio to force the model to learn semantic representations, continuing the design philosophy of Context Encoders.
50%UnverifiedFor an 8×8×256 latent feature, the grouped fully connected layer requires approximately 1.05 million weights, whereas a full FC layer would require approximately 268 million weights.
50%