Skip to yearly menu bar Skip to main content


Workshop

Bridging the Gap Between Practice and Theory in Deep Learning

Wei Chen ⋅ Christa Cuchiero ⋅ Hadi Daneshmand ⋅ Stefanie Jegelka ⋅ Zelda Mariet ⋅ Andre Niyongabo Rubungo ⋅ Jiaye Teng ⋅ Bohan Wang ⋅ Bohang Zhang ⋅ Jingzhao Zhang
[ Contact: zhangbohang@pku.edu.cn ]

Abstract

Video

Chat is not available.

Schedule

Timezone: America/Los_Angeles
2:00 AM
Contributed Talk 2: Catapults in SGD: spikes in the training loss and their impact on generalization through feature learning
Libin Zhu ⋅ Chaoyue Liu ⋅ Adityanarayanan Radhakrishnan ⋅ Misha Belkin
2:30 AM
Contributed Talk 4: Promises and Pitfalls of Generative Masked Language Modeling: Theoretical Framework and Practical Guidelines
Yuchen Li ⋅ Alexandre Kirchmeyer ⋅ Aashay Mehta ⋅ Yilong Qin ⋅ Boris Dadachev ⋅ Kishore Papineni ⋅ Sanjiv Kumar ⋅ Andrej Risteski
2:45 AM
Contributed Talk 5: Interpreting Grokked Transformers in Complex Modular Arithmetic
Hiroki Furuta ⋅ Gouki Minegishi ⋅ Yusuke Iwasawa ⋅ Yutaka Matsuo
A resource model for neural scaling laws
Jinyeop Song ⋅ Ziming Liu ⋅ Max Tegmark ⋅ Jeff Gore
Implicit Bias and Fast Convergence Rates for Self-attention
Bhavya Vasudeva ⋅ Puneesh Deora ⋅ Christos Thrampoulidis
Generalization Bounds for Magnitude Based Pruning
Etash Guha ⋅ Prasanjit Dubey ⋅ Xiaoming Huo
Fundamental Benefit of Alternating Updates in Minimax Optimization
Jaewook Lee ⋅ Hanseul Cho ⋅ Chulhee Yun
GNN-VPA: A Variance-Preserving Aggregation Strategy for Graph Neural Networks
Lisa Schneckenreiter ⋅ Richard Freinschlag ⋅ Florian Sestak ⋅ Johannes Brandstetter ⋅ Günter Klambauer ⋅ Andreas Mayr
Long-Range Synthetic Knowledge Graph Benchmarks for Double-Equivariant Models
Bruna Jasinowodolinski ⋅ Yucheng Zhang ⋅ Jincheng Zhou ⋅ Beatrice Bevilacqua ⋅ Bruno Ribeiro
Meta Prompting for AGI Systems
Yifan Zhang ⋅ Yang Yuan ⋅ Andrew Yao
Momentum Gradient Descent over single-neuron linear network: rich behaviors of limiting Sharpness
WenJie Zhou ⋅ Bohan Wang ⋅ Wei Chen ⋅ Zhi-Ming Ma ⋅ Xueqi Cheng
Conformal Prediction Sets Improve Human Decision Making
Jesse Cresswell ⋅ Yi Sui ⋅ Bhargava Kumar ⋅ Noël Vouitsis
Anchor Function for Studying Language Models
Zhongwang Zhang ⋅ Zhiwei Wang ⋅ Zhiqin Xu
Understanding multimodal contrastive learning through pointwise mutual information
Toshimitsu Uesaka ⋅ Taiji Suzuki ⋅ Yuhta Takida ⋅ Chieh-Hsin Lai ⋅ Naoki Murata ⋅ Yuki Mitsufuji
Optimizing for ROC Curves on Class-Imbalanced Data by Training over a Family of Loss Functions
Kelsey Lieberman ⋅ Shuai Yuan ⋅ Swarna Ravindran ⋅ Carlo Tomasi
The best algorithm for adversarial training
Nikolaos Tsilivis ⋅ Natalie Frank ⋅ Julia Kempe
Understanding Sub-domain Alignment for Domain Adaptation
Yiling Liu ⋅ Juncheng Dong ⋅ Ziyang Jiang ⋅ Ahmed Aloui ⋅ Keyu Li ⋅ Michael Klein ⋅ VAHID TAROKH ⋅ David Carlson
Binding Dynamics in Rotating Features
Sindy Löwe ⋅ Francesco Locatello ⋅ Max Welling
Robust Sparse Mean Estimation via Incremental Learning
Jianhao Ma ⋅ Rui Chen ⋅ Yinghui He ⋅ Salar Fattahi ⋅ Wei Hu
Towards a Theoretical Understanding of Model Collapse
Elvis Dohmatob ⋅ Yunzhen Feng ⋅ Julia Kempe
Simplicity Bias of Transformers to Learn Low Sensitivity Functions
Bhavya Vasudeva ⋅ Deqing Fu ⋅ Tianyi Zhou ⋅ Elliott Kau ⋅ Youqi Huang ⋅ Vatsal Sharan
Weisfeiler–Leman at the margin: When more expressivity matters
Billy Franks ⋅ Christopher Morris ⋅ Ameya Velingker ⋅ Floris Geerts
On the Relationship Between Small Initialization and Flatness in Deep Networks
Soo Min Kwon ⋅ Lijun Ding ⋅ Laura Balzano ⋅ Qing Qu
Cumulative Reasoning with Large Language Models
Yifan Zhang ⋅ Jingqin Yang ⋅ Yang Yuan ⋅ Andrew Yao
Measuring Sharpness in Grokking
Jack Miller ⋅ Patrick Gleeson ⋅ Noam Levi ⋅ Charles O'Neill ⋅ Thang Bui
Low-Rank Robust Graph Contrastive Learning
Yancheng Wang ⋅ Yingzhen Yang
Active Few-Shot Fine-Tuning
Jonas Hübotter ⋅ Bhavya ⋅ Lenart Treven ⋅ Yarden As ⋅ Andreas Krause
On Different Faces of Model Scaling in Supervised and Self-Supervised Learning
Matteo Gamba ⋅ Arna Ghosh ⋅ Kumar Agrawal ⋅ Blake A Richards ⋅ Hossein Azizpour ⋅ Mårten Björkman
Breaking the Dimension Dependence in Sketching for Distributed Learning
Berivan Isik ⋅ Qiaobo Li ⋅ Mayank Shrivastava ⋅ Arindam Banerjee ⋅ Sanmi Koyejo
OMPO: A Unified Framework for Reinforcement Learning under Policy and Dynamics Shifts
Yu Luo ⋅ Tianying Ji ⋅ Fuchun Sun ⋅ Jianwei Zhang ⋅ Huazhe Xu ⋅ Xianyuan Zhan
Towards an Improved Understanding and Utilization of Maximum Manifold Capacity Representations
Rylan Schaeffer ⋅ Berivan Isik ⋅ Dhruv Pai ⋅ Andres Carranza ⋅ Victor Lecomte ⋅ Alyssa Unell ⋅ Mikail Khona ⋅ Thomas Yerxa ⋅ Yann LeCun ⋅ SueYeon Chung ⋅ Andrey Gromov ⋅ Ravid Shwartz-Ziv ⋅ Sanmi Koyejo
On Learning Modular Polynomials
Darshil Doshi ⋅ Tianyu He ⋅ Aritra Das ⋅ Andrey Gromov
Towards Principled Graph Transformers
Luis Müller ⋅ Daniel Kusuma ⋅ Christopher Morris
How Uniform Random Weights Induce Non- uniform Bias: Typical Interpolating Neural Networks Generalize with Narrow Teachers
Gon Buzaglo ⋅ Itamar Harel ⋅ Mor Shpigel Nacson ⋅ Alon Brutzkus ⋅ Nathan Srebro ⋅ Daniel Soudry
ADOPT: Modified Adam Can Converge with the Optimal Rate with Any Hyperparameters
Shohei Taniguchi ⋅ Masahiro Suzuki ⋅ Yusuke Iwasawa ⋅ Yutaka Matsuo
Training Dynamics of Multi-Head Softmax Attention: Emergence, Convergence, and Optimality
Siyu Chen ⋅ Heejune Sheen ⋅ Zhuoran Yang ⋅ Tianhao Wang
Variational Linearized Laplace Approximation for Bayesian Deep Learning
Luis A. Ortega ⋅ Simon Rodriguez Santana ⋅ Daniel Hernández-Lobato
What Causes Polysemanticity? An Alternative Origin Story of Mixed Selectivity from Incidental Causes
Victor Lecomte ⋅ Kushal Thaman ⋅ Rylan Schaeffer ⋅ Naomi Bashkansky ⋅ Trevor Chow ⋅ Sanmi Koyejo
Addressing Sample Inefficiency in Multi-View Representation Learning
Arna Ghosh ⋅ Kumar Agrawal ⋅ Adam Oberman ⋅ Blake A Richards
Simulating the Implicit Effect of Learning Rates in Gradient Descent
Adrian Goldwaser ⋅ Bruno Mlodozeniec ⋅ Hong Ge
Provably Efficient Maximum Entropy Pure Exploration in Reinforcement Learning
Hongyi Guo ⋅ Yufeng Zhang ⋅ Zhuoran Yang ⋅ Zhaoran Wang
Weisfeiler and Leman Go Loopy: A New Hierarchy for Graph Representational Learning
Raffaele Paolino ⋅ Sohir Maskey ⋅ Pascal Welke ⋅ Gitta Kutyniok
Why do Learning Rates Transfer? Reconciling Optimization and Scaling Limits for Deep Learning
Alexandru Meterez ⋅ Lorenzo Noci ⋅ Thomas Hofmann ⋅ Antonio Orvieto
TRAVERSING CHEMICAL SPACE WITH LATENT POTENTIAL FLOWS
Guanghao Wei ⋅ Yining Huang ⋅ Chenru Duan ⋅ Yue Song ⋅ Yuanqi Du
Scaling laws and Zipf's law in AlphaZero
Oren Neumann ⋅ Claudius Gros
On the Theory of Skill-based Reinforcement Learning: Distribution Recovery and Generalization
Hongyi Guo ⋅ Xiaoyu Chen ⋅ Sirui Zheng ⋅ Zhuoran Yang ⋅ Zhaoran Wang
A Coefficient Makes SVRG Effective
Yida Yin ⋅ Zhiqiu Xu ⋅ Zhiyuan Li ⋅ trevor darrell ⋅ Zhuang Liu
On the Theory of Cross-Modality Distillation with Contrastive Learning
Hangyu Lin ⋅ Chen Liu ⋅ Chengming Xu ⋅ Zhengqi Gao ⋅ Yanwei Fu ⋅ Yuan Yao
Implicit Regularization of Gradient Flow for One-layer Softmax Attention
Heejune Sheen ⋅ Siyu Chen ⋅ Tianhao Wang ⋅ Huibin Zhou
In-context Newton’s method for regression: Transformers can provably converge
Angeliki Giannou ⋅ Liu Yang ⋅ Tianhao Wang ⋅ Dimitris Papailiopoulos ⋅ Jason Lee
Is robust overfitting inevitable? - An approximation viewpoint
Zhongjie Shi ⋅ Fanghui Liu ⋅ Yuan Cao ⋅ Johan Suykens
Stochastic restarting to overcome overfitting in neural networks
Yeongwoo Song ⋅ Youngkyoung Bae ⋅ Hawoong Jeong