Poster
SAS: Structured Activation Sparsification
Yusuke Sekikawa · Shingo Yashima
Abstract:
Wide networks usually yield better accuracy than their narrower counterpart at the expense of the massive cost.To break this tradeoff, we advocate a novel concept of , dubbed SAS, which boosts accuracy without increasing computation by utilizing the projected sparsity in activation maps with a specific structure. Concretely, the projected sparse activation is allowed to have N nonzero value among M consecutive activations.Owing to the local structure in sparsity, the wide between a dense weight and the sparse activation is executed as an equivalent narrow between a dense weight and dense activation, which is compatible with NVIDIA's developed for the N:M structured sparse weight.In extensive experiments, we demonstrate that increasing sparsity monotonically improves accuracy (up to 7% on CIFAR10) without increasing the count.Furthermore, we show that structured sparsification of scales better than that of given the same computational budget.
Chat is not available.