Skip to yearly menu bar Skip to main content


Workshop

2nd Workshop on Mathematical and Empirical Understanding of Foundation Models

Sang Michael Xie ⋅ Ananya Kumar ⋅ Sewon Min ⋅ Sadhika Malladi ⋅ Lucio Dery ⋅ Aditi Raghunathan ⋅ Tengyu Ma ⋅ Percy Liang
[ Contact: smalladi@princeton.edu ]

Abstract

Video

Chat is not available.

Schedule

Timezone: America/Los_Angeles
11:50 PM
Opening remarks
12:00 AM
Invited Talk (Sasha Rush)
Sasha Rush
12:40 AM
Invited Talk (Yuandong Tian)
Yuandong Tian
1:20 AM
Invited Talk (Hannaneh Hajishirzi)
Hannaneh Hajishirzi
2:00 AM
Spotlight Talks
4:00 AM
Poster Session
5:00 AM
Panel Discussion (Sasha Rush, Yuandong Tian, Hannaneh Hajishirzi, Jacob Steinhardt, Aditi Raghunathan)
Aditi Raghunathan ⋅ Sasha Rush ⋅ Yuandong Tian ⋅ Hannaneh Hajishirzi ⋅ Jacob Steinhardt
6:00 AM
Invited Talk (Jacob Steinhardt)
Jacob Steinhardt
6:40 AM
Invited Talk (Amir Globerson)
Amir Globerson
Prompting a Pretrained Transformer Can Be a Universal Approximator
Aleksandar Petrov ⋅ Adel Bibi ⋅ Philip Torr
Can Mamba Learn How To Learn? A Comparative Study on In-Context Learning Tasks
Jongho Park ⋅ Jaden Park ⋅ Zheyang Xiong ⋅ Nayoung Lee ⋅ Jaewoong Cho ⋅ Samet Oymak ⋅ Kangwook Lee ⋅ Dimitris Papailiopoulos
Robust CLIP: Unsupervised Adversarial Fine-tuning of Vision Embeddings for Robust Large Vision-Language Models
Christian Schlarmann ⋅ Naman Singh ⋅ francesco croce ⋅ Matthias Hein
Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-constraint
Wei Xiong ⋅ Hanze Dong ⋅ Chenlu Ye ⋅ Ziqi Wang ⋅ Han Zhong ⋅ Heng Ji ⋅ Nan Jiang ⋅ Tong Zhang
Editing Large Language Models: Problems, Methods, and Opportunities
Yunzhi Yao ⋅ Peng Wang ⋅ Bozhong Tian ⋅ Siyuan Cheng ⋅ Zhoubo Li ⋅ Shumin Deng ⋅ Huajun Chen ⋅ Ningyu Zhang
Pre-training and In-context Learning IS Bayesian Inference a la De Finetti
Naimeng Ye ⋅ Hanming Yang ⋅ Andrew Siah ⋅ Hongseok Namkoong
Linear Alignment of Vision-language Models for Image Captioning
Fabian Paischer ⋅ Markus Hofmarcher ⋅ Sepp Hochreiter ⋅ Thomas Adler
Self-Supervised Open-Ended Classification with Small Visual Language Models
Mohammad Mahdi Derakhshani ⋅ Ivona Najdenkoska ⋅ Cees G Snoek ⋅ Marcel Worring ⋅ Yuki Asano
Simple linear attention language models balance the recall-throughput tradeoff
Simran Arora ⋅ Sabri Eyuboglu ⋅ Michael Zhang ⋅ Aman Timalsina ⋅ Silas Alberti ⋅ James Y Zou ⋅ Atri Rudra ⋅ Christopher Re
Best Arm Identification for Prompt Learning under a Limited Budget
Chengshuai Shi ⋅ Kun Yang ⋅ Jing Yang ⋅ Cong Shen
Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modifications
Boyi Wei ⋅ Kaixuan Huang ⋅ Yangsibo Huang ⋅ Tinghao Xie ⋅ Xiangyu Qi ⋅ Mengzhou Xia ⋅ Prateek Mittal ⋅ Mengdi Wang ⋅ Peter Henderson
Does Data Contamination Make a Difference? Insights from Intentionally Contamination Pre-training Data For Language Models
Minhao Jiang ⋅ Ken Liu ⋅ Ming Zhong ⋅ Rylan Schaeffer ⋅ Siru Ouyang ⋅ Jiawei Han ⋅ Sanmi Koyejo
Multi-Task Inference: Can Large Language Models Follow Multiple Instructions at Once?
Guijin Son ⋅ Sangwon Baek ⋅ Sangdae Nam ⋅ Ilgyun Jeong ⋅ Seungone Kim
QuRating: Selecting High-Quality Data for Training Language Models
Alexander Wettig ⋅ Aatmik Gupta ⋅ Saumya Malik ⋅ Danqi Chen
TOWARDS AN EMPIRICAL UNDERSTANDING OF MOE DESIGN CHOICES
Dongyang Fan ⋅ Bettina Messmer ⋅ Martin Jaggi
What makes vision transformers robust towards bit-flip attack?
Xuan Zhou ⋅ Souvik Kundu ⋅ Peter Beerel
The Effect of Model Capacity on the Emergence of In-Context Learning
Berkan Ottlik ⋅ Narutatsu Ri ⋅ Daniel Hsu ⋅ Clayton Sanford
MathSensei: Mathematical Reasoning with a Tool-Augmented Large Language Model
DEBRUP DAS ⋅ Debopriyo Banerjee ⋅ Somak Aditya ⋅ Ashish Kulkarni
Preserving Principal Subspaces to Reduce Catastrophic Forgetting in Fine-tuning
Jörg Franke ⋅ Michael Hefenbrock ⋅ Frank Hutter
Towards an Improved Understanding and Utilization of Maximum Manifold Capacity Representations
Rylan Schaeffer ⋅ Berivan Isik ⋅ Dhruv Pai ⋅ Andres Carranza ⋅ Victor Lecomte ⋅ Alyssa Unell ⋅ Mikail Khona ⋅ Thomas Yerxa ⋅ Yann LeCun ⋅ SueYeon Chung ⋅ Andrey Gromov ⋅ Ravid Shwartz-Ziv ⋅ Sanmi Koyejo
Backward Chaining Circuits in a Transformer Trained on a Symbolic Reasoning Task
Jannik Brinkmann ⋅ Abhay Sheshadri ⋅ Victor Levoso ⋅ Paul Swoboda ⋅ Christian Bartelt
Selecting Large Language Model to Fine-tune via Rectified Scaling Law
Haowei Lin ⋅ Baizhou Huang ⋅ Haotian Ye ⋅ Qinyu Chen ⋅ Zihao Wang ⋅ Sujian Li ⋅ Jianzhu Ma ⋅ Xiaojun Wan ⋅ James Y Zou ⋅ Yitao Liang
Zero-Shot Recognition with Guided Cropping
Piyapat Saranrittichai ⋅ Mauricio Munoz ⋅ Volker Fischer ⋅ Chaithanya Kumar Mummadi
Few-Shot Adaptation of Vision-Language Foundation Models via Dual-Path Inference
Ce Zhang ⋅ Simon Stepputtis ⋅ Katia Sycara ⋅ Yaqi Xie
SparQ Attention: Bandwidth-Efficient LLM Inference
Luka Ribar ⋅ Ivan Chelombiev ⋅ Luke Hudlass-Galley ⋅ Charles Blake ⋅ Carlo Luschi ⋅ Douglas Orr
Understanding and Improving In-Context Learning on Vision-language Models
Shuo Chen ⋅ Zhen Han ⋅ Bailan He ⋅ Mark Buckley ⋅ Philip Torr ⋅ Volker Tresp ⋅ Jindong Gu
Scaling Laws for Downstream Task Performance of Large Language Models
Berivan Isik ⋅ NATALIA PONOMAREVA ⋅ Hussein Hazimeh ⋅ Dimitris Paparas ⋅ Sergei Vassilvitskii ⋅ Sanmi Koyejo
Transformers' Spectral Bias and The Symmetric Group
Itay Lavie ⋅ Guy Gur-Ari ⋅ Zohar Ringel
Asymmetry in Low-Rank Adapters of Foundation Models
Jiacheng Zhu ⋅ Kristjan Greenewald ⋅ Kimia Nadjahi ⋅ Haitz Sáez de Ocáriz Borde ⋅ Rickard Gabrielsson ⋅ Leshem Choshen ⋅ Marzyeh Ghassemi ⋅ Mikhail Yurochkin ⋅ Justin Solomon
Benchmarking and Building Long-Context Retrieval Models with LoCo and M2-BERT
Jon Saad-Falcon ⋅ Dan Fu ⋅ Simran Arora ⋅ Neel Guha ⋅ Christopher Re
MoE-Mamba: Efficient Selective State Space Models with Mixture of Experts
Maciej Pióro ⋅ Kamil Ciebiera ⋅ Krystian Król ⋅ Jan Ludziejewski ⋅ Michał Krutul ⋅ Jakub Krajewski ⋅ Szymon Antoniak ⋅ Piotr Miłoś ⋅ Marek Cygan ⋅ Sebastian Jaszczur
Concept-aware Data Construction Improves In-context Learning of Language Models
Michal Štefánik ⋅ Marek Kadlčík ⋅ Petr Sojka
Uncovering Mesa-Optimization Algorithms in Transformers
Johannes von Oswald ⋅ Eyvind Niklasson ⋅ Maximilian Schlegel ⋅ Alexander Meulemans ⋅ Seijin Kobayashi ⋅ Nicolas Zucchet ⋅ Nino Scherrer ⋅ Nolan Miller ⋅ Mark Sandler ⋅ Blaise Aguera y Arcas ⋅ Max Vladymyrov ⋅ Razvan Pascanu ⋅ Joao Sacramento
How Well Does GPT-4V(ision) Adapt to Distribution Shifts? A Preliminary Investigation
Zhongyi Han ⋅ Guanglin Zhou ⋅ Rundong He ⋅ Jindong Wang ⋅ Tailin Wu ⋅ Yilong Yin ⋅ Salman Khan ⋅ Lina Yao ⋅ Tongliang Liu ⋅ Kun Zhang
Scaling Laws for Fine-Grained Mixture of Experts
Jan Ludziejewski ⋅ Jakub Krajewski ⋅ Kamil Adamczewski ⋅ Maciej Pióro ⋅ Michał Krutul ⋅ Szymon Antoniak ⋅ Kamil Ciebiera ⋅ Krystian Król ⋅ Tomasz Odrzygóźdź ⋅ Piotr Sankowski ⋅ Marek Cygan ⋅ Sebastian Jaszczur
Is Mamba Capable of In-Context Learning?
Riccardo Grazzi ⋅ Julien Siems ⋅ Simon Schrodi ⋅ Thomas Brox ⋅ Frank Hutter
Investigating Uncertainty Calibration of Aligned Language Models under the Multiple-Choice Setting
Guande He ⋅ Peng Cui ⋅ Jianfei Chen ⋅ Wenbo Hu ⋅ Jun Zhu
Fourier Circuits in Neural Networks: Unlocking the Potential of Large Language Models in Math Reasoning and Modular Arithmetic
Jiuxiang Gu ⋅ Chenyang Li ⋅ Yingyu Liang ⋅ Zhenmei Shi ⋅ Zhao Song ⋅ Tianyi Zhou
Lessons from Identifiability for Understanding Large Language Models
Patrik Reizinger ⋅ Szilvia Ujváry ⋅ Anna Mészáros ⋅ Anna Kerekes ⋅ Wieland Brendel ⋅ Ferenc Huszar
Prometheus-Vision: Vision-Language Model as a Judge for Fine-Grained Evaluation
Seongyun Lee ⋅ Seungone Kim ⋅ Sue Park ⋅ Geewook Kim ⋅ Minjoon Seo
Provably Robust DPO: Aligning Language Models with Noisy Feedback
Sayak Ray Chowdhury ⋅ Anush Kini ⋅ Nagarajan Natarajan
In-Context Data Distillation with TabPFN
Junwei (Jeremy) Ma ⋅ Valentin Thomas ⋅ Guangwei Yu ⋅ Anthony Caterini
Transformers Can Achieve Length Generalization But Not Robustly
Yongchao Zhou ⋅ Uri Alon ⋅ Xinyun Chen ⋅ Xuezhi Wang ⋅ Rishabh Agarwal ⋅ Denny Zhou
Scalable Ensembling For Mitigating Reward Overoptimisation
Ahmed Ahmed ⋅ Rafael Rafailov ⋅ Stepan Sharkov ⋅ Xuechen Li ⋅ Sanmi Koyejo
tinyBenchmarks: evaluating LLMs with fewer examples
Felipe Polo ⋅ Lucas Weber ⋅ Leshem Choshen ⋅ Yuekai Sun ⋅ Gongjun Xu ⋅ Mikhail Yurochkin
ShERPA: Leveraging Neuron Alignment for Knowledge-preserving Fine-tuning
Dongkyu Cho ⋅ Jinseok Yang ⋅ Jun Seo ⋅ Seohui Bae ⋅ Dongwan Kang ⋅ Soyeon Park ⋅ Hyeokjun Choe ⋅ Woohyung Lim
Eureka-Moments in Transformers: Multi-Step Tasks Reveal Softmax Induced Optimization Problems
David Hoffmann ⋅ Simon Schrodi ⋅ Jelena Bratulić ⋅ Nadine Behrmann ⋅ Volker Fischer ⋅ Thomas Brox
Predicting Grokking Long Before it Happens: A look into the loss landscape of models which grok
Tikeng Notsawo Pascal Junior ⋅ Hattie Zhou ⋅ Mohammad Pezeshki ⋅ Irina Rish ⋅ Guillaume Dumas
LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression
Huiqiang Jiang ⋅ Qianhui Wu ⋅ Xufang Luo ⋅ Dongsheng Li ⋅ Chin-Yew Lin ⋅ Yuqing Yang ⋅ Lili Qiu
"I'm not Racist but…": Discovering Bias in the Internal Knowledge of Large Language Models
Abel Salinas ⋅ Louis Penafiel ⋅ Robert McCormack ⋅ Fred Morstatter
LangBridge: Multilingual Reasoning Without Multilingual Supervision
Dongkeun Yoon ⋅ Joel Jang ⋅ Sungdong Kim ⋅ Seungone Kim ⋅ SHEIKH SHAFAYAT ⋅ Minjoon Seo
On Fairness Implications and Evaluations of Low-Rank Adaptation of Large Models
Ken Liu ⋅ Zhoujie Ding ⋅ Berivan Isik ⋅ Sanmi Koyejo
Dual Operating Modes of In-Context Learning
Ziqian Lin ⋅ Kangwook Lee
Can Generative Multimodal Models Count to Ten?
Sunayana Rane ⋅ Alexander Ku ⋅ Jason Baldridge ⋅ Ian Tenney ⋅ Thomas L. Griffiths ⋅ Been Kim
Attributing Mode Collapse in the fine-tuning of Large Language Models
Laura O'Mahony ⋅ Leo Grinsztajn ⋅ Hailey Schoelkopf ⋅ Stella R Biderman
Massive Activations in Large Language Models
Mingjie Sun ⋅ Xinlei Chen ⋅ J Kolter ⋅ Zhuang Liu
Shortened LLaMA: A Simple Depth Pruning for Large Language Models
Bo-Kyeong Kim ⋅ Geonmin Kim ⋅ Tae-Ho Kim ⋅ Thibault Castells ⋅ Shinkook Choi ⋅ Junho Shin ⋅ Hyoung-Kyu Song
Direct Alignment of Draft Model for Speculative Decoding with Chat-Fine-tuned LLMs
Raghavv Goel ⋅ Mukul Gagrani ⋅ Wonseok Jeon ⋅ Junyoung Park ⋅ Mingu Lee ⋅ Christopher Lott
BlackMamba: Mixture of Experts for State-Space Models
Quentin Anthony ⋅ Yury Tokpanov ⋅ Paolo Glorioso ⋅ Beren Millidge
Perplexed by Perplexity: Perplexity-Based Pruning with Small Reference Models
Zachary Ankner ⋅ Cody Blakeney ⋅ Kartik Sreenivasan ⋅ Max M Marion ⋅ Matthew Leavitt ⋅ Mansheej Paul
Prompting a Pretrained Transformer Can Be a Universal Approximator
Aleksandar Petrov ⋅ Adel Bibi ⋅ Philip Torr
Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-constraint
Wei Xiong ⋅ Hanze Dong ⋅ Chenlu Ye ⋅ Ziqi Wang ⋅ Han Zhong ⋅ Heng Ji ⋅ Nan Jiang ⋅ Tong Zhang
Selecting Large Language Model to Fine-tune via Rectified Scaling Law
Haowei Lin ⋅ Baizhou Huang ⋅ Haotian Ye ⋅ Qinyu Chen ⋅ Zihao Wang ⋅ Sujian Li ⋅ Jianzhu Ma ⋅ Xiaojun Wan ⋅ James Y Zou ⋅ Yitao Liang
Uncovering Mesa-Optimization Algorithms in Transformers
Johannes von Oswald ⋅ Eyvind Niklasson ⋅ Maximilian Schlegel ⋅ Alexander Meulemans ⋅ Seijin Kobayashi ⋅ Nicolas Zucchet ⋅ Nino Scherrer ⋅ Nolan Miller ⋅ Mark Sandler ⋅ Blaise Aguera y Arcas ⋅ Max Vladymyrov ⋅ Razvan Pascanu ⋅ Joao Sacramento
Scaling Laws for Fine-Grained Mixture of Experts
Jan Ludziejewski ⋅ Jakub Krajewski ⋅ Kamil Adamczewski ⋅ Maciej Pióro ⋅ Michał Krutul ⋅ Szymon Antoniak ⋅ Kamil Ciebiera ⋅ Krystian Król ⋅ Tomasz Odrzygóźdź ⋅ Piotr Sankowski ⋅ Marek Cygan ⋅ Sebastian Jaszczur
Massive Activations in Large Language Models
Mingjie Sun ⋅ Xinlei Chen ⋅ J Kolter ⋅ Zhuang Liu