Poster
Dynamic Layer Tying for Parameter-Efficient Transformers
Tamir David-Hay · Lior Wolf
Halle B #104
Abstract:
In the pursuit of reducing the number of trainable parameters in deep transformer networks, we employ Reinforcement Learning to dynamically select layers during training and tie them together. Every few iterations, the RL agent is asked whether to train each layer $i$ independently or to copy the weights of a previous layer $j
Chat is not available.
Successful Page Load