Homogeneous Model
A mathematical model satisfying a scaling transformation property, namely f(cw) = c^L f(w), commonly found in deep linear networks and ReLU networks with biases omitted.
Timeline (last 90 days)
Prior work has shown that on homogeneous models, when the learning rate schedule is of the strongly decaying form η(t) = o(t^{1/L-1}), Adam is proven to converge to the solution that maximizes the ℓ∞ margin.
A homogeneous model is one satisfying f(cw) = c^L f(w) for any scalar c>0, where L is called the degree of homogeneity.
All Facts (2)
Prior work has shown that on homogeneous models, when the learning rate schedule is of the strongly decaying form η(t) = o(t^{1/L-1}), Adam is proven to converge to the solution that maximizes the ℓ∞ margin.
50%UnverifiedA homogeneous model is one satisfying f(cw) = c^L f(w) for any scalar c>0, where L is called the degree of homogeneity.
50%