Skip to yearly menu bar Skip to main content


Workshop

Building Trust in LLMs and LLM Applications: From Guardrails to Explainability to Regulation

Micah Goldblum ⋅ Ramasuri Narayanam ⋅ Bang An ⋅ Soumyabrata Pal ⋅ Martin Pawelczyk ⋅ Hima Lakkaraju ⋅ Shiv Saini
Project Page

Abstract

Video

Chat is not available.

Schedule

Timezone: America/Los_Angeles
7:20 PM
Break
9:15 PM
Lunch Break
11:50 PM
Panel Discussion
12:25 AM
Break
On the Role of Prompt Multiplicity in LLM Hallucination Evaluation
Prakhar Ganesh ⋅ Reza Shokri ⋅ Golnoosh Farnadi
UNLEARNING GEO-CULTURAL STEREOTYPES IN MULTILINGUAL LLMS
Alireza Dehghanpour Farashah ⋅ Aditi Khandelwal ⋅ Negar Rostamzadeh ⋅ Golnoosh Farnadi
Measuring In-Context Computation Complexity via Hidden State Prediction
Vincent Herrmann ⋅ Róbert Csordás ⋅ Jürgen Schmidhuber
Prune 'n Predict: Optimizing LLM Decision-making with Conformal Prediction
Harit Vishwakarma ⋅ Thomas Cook ⋅ Alan Mishler ⋅ Niccolo Dalmasso ⋅ Natraj Raman ⋅ Sumitra Ganesh
Why Do Multiagent Systems Fail?
Melissa Pan ⋅ Mert Cemri ⋅ Lakshya A Agrawal ⋅ Shuyi Yang ⋅ Bhavya Chopra ⋅ Rishabh Tiwari ⋅ Kurt Keutzer ⋅ Aditya Parameswaran ⋅ Kannan Ramchandran ⋅ Dan Klein ⋅ Joseph E Gonzalez ⋅ Matei Zaharia ⋅ Ion Stoica
Enhancing CBMs Through Binary Distillation with Applications to Test-Time Intervention
Matthew Shen ⋅ Aliyah Hsu ⋅ Abhineet Agarwal ⋅ Bin Yu
Fast Proxies for LLM Robustness Evaluation
Tim Beyer ⋅ Jan Schuchardt ⋅ Leo Schwinn ⋅ Stephan Günnemann
On-Premises LLM Deployment Demands a Middle Path: Preserving Privacy Without Sacrificing Model Confidentiality
Hanbo Huang ⋅ Yihan Li ⋅ Bowen Jiang ⋅ Lin Liu ⋅ Bo Jiang ⋅ Ruoyu Sun ⋅ Zhuotao Liu ⋅ Shiyu Liang
Token-Level Adversarial Prompt Detection Based on Perplexity Measures and Contextual Information
Zhengmian Hu ⋅ Gang Wu ⋅ Saayan Mitra ⋅ Ruiyi Zhang ⋅ Tong Sun ⋅ Heng Huang ⋅ Viswanathan Swaminathan
Interpretable Steering of Large Language Models with Feature Guided Activation Additions
Samuel Soo ⋅ Wesley Teng ⋅ Balaganesh Chandrasekaran ⋅ Guoxian TAN ⋅ Ming YAN
Towards Effective Discrimination Testing for Generative AI
Thomas Zollo ⋅ Nikita Rajaneesh ⋅ Richard Zemel ⋅ Talia Gillis ⋅ Emily Black
Automated Capability Discovery via Model Self-Exploration
Cong Lu ⋅ Shengran Hu ⋅ Jeff Clune
A False Sense of Privacy: Evaluating Textual Data Sanitization Beyond Surface-level Privacy Leakage
Rui Xin ⋅ Niloofar Mireshghallah ⋅ Shuyue Stella Li ⋅ Michael Duan ⋅ Hyunwoo Kim ⋅ Yejin Choi ⋅ Yulia Tsvetkov ⋅ Sewoong Oh ⋅ Pang Wei Koh
Finding Sparse Autoencoder Representations Of Errors In CoT Prompting
Justin Theodorus ⋅ V Swaytha ⋅ Shivani Gautam ⋅ Adam Ward ⋅ Mahir Shah ⋅ Cole Blondin ⋅ Kevin Zhu
Hidden No More: Attack and Defending Private Third-Party LLM Inference
Arka Pal ⋅ Rahul Thomas ⋅ Louai Zahran ⋅ Erica Choi ⋅ Akilesh Potti ⋅ Micah Goldblum
Conformal Structured Prediction
Botong Zhang ⋅ Shuo Li ⋅ Osbert Bastani
An Empirical Study on Prompt Compression for Large Language Models
Zheng Zhang ⋅ Jinyi Li ⋅ Yihuai Lan ⋅ Xiang Wang ⋅ Hao Wang
Evaluating and Mitigating the Safety Awareness-Execution Gaps of LM Agents
Yuzhi Tang ⋅ Tianxiao Li ⋅ Elizabeth Li ⋅ Chris Maddison ⋅ Honghua Dong ⋅ Yangjun Ruan
SPEX: Scaling Feature Interaction Explanations for LLMs
Justin Kang ⋅ Landon Butler ⋅ Abhineet Agarwal ⋅ Yigit Efe Erginbas ⋅ Ramtin Pedarsani ⋅ Bin Yu ⋅ Kannan Ramchandran
ExpProof : Operationalizing Explanations for Confidential Models with ZKPs
Chhavi Yadav ⋅ Evan Laufer ⋅ Dan Boneh ⋅ Kamalika Chaudhuri
Learning Automata from Demonstrations, Examples, and Natural Language
Karim Elmaaroufi ⋅ Marcell Vazquez-Chanlatte ⋅ Stefan Witwicki ⋅ Matei Zaharia ⋅ Sanjit Seshia
A Missing Testbed for LLM Pre-Training Membership Inference Attacks
Mingjian Jiang ⋅ Ken Liu ⋅ Sanmi Koyejo
MALIBU Benchmark: Multi-Agent LLM Implicit Bias Uncovered
Ishwara Vasista ⋅ Imran Mirza ⋅ Cole Huang ⋅ Rohan Patil ⋅ Aslihan Akalin ⋅ Kevin Zhu ⋅ Sean OBrien
Maybe I Should Not Answer That, but... Do LLMs Understand The Safety of Their Inputs?
Maciej Chrabaszcz ⋅ Filip Szatkowski ⋅ Bartosz Wójcik ⋅ Jan Dubiński ⋅ Tomasz Trzcinski
BaxBench: Can LLMs Generate Correct and Secure Backends?
Mark Vero ⋅ Niels Mündler ⋅ Victor Chibotaru ⋅ Veselin Raychev ⋅ Maximilian Baader ⋅ Nikola Jovanović ⋅ Jingxuan He ⋅ Martin Vechev
SafeMERGE: Preserving Safety Alignment in Fine-Tuned Large Language Models via Selective Layer-Wise Model Merging
Aladin Djuhera ⋅ Swanand Kadhe ⋅ Farhan Ahmed ⋅ Syed Zawad ⋅ Holger Boche
LLM Neurosurgeon: Targeted Knowledge Removal in LLMs using Sparse Autoencoders
Dylan Zhou ⋅ Kunal Patil ⋅ Yifan Sun ⋅ Karthik lakshmanan ⋅ Senthooran Rajamanoharan ⋅ Arthur Conmy
Enhancing LLM Robustness to Perturbed Instructions: An Empirical Study
Aryan Agrawal ⋅ Lisa Alazraki ⋅ Shahin Honarvar ⋅ Thomas Mensink ⋅ Marek Rei
A Generative Approach to LLM Harmfulness Detection with Red Flag Tokens
Sophie Xhonneux ⋅ David Dobre ⋅ Mehrnaz Mofakhami ⋅ Leo Schwinn ⋅ Gauthier Gidel
MFC-Bench: Benchmarking Multimodal Fact-Checking with Large Vision-Language Models
Shengkang Wang ⋅ Hongzhan Lin ⋅ Ziyang Luo ⋅ Zhen Ye ⋅ Guang Chen ⋅ Jing Ma
Antipodal Pairing and Mechanistic Signals in Dense SAE Latents
Alessandro Stolfo ⋅ Ben Wu ⋅ Mrinmaya Sachan
FiDeLiS: Faithful Reasoning in Large Language Models for Knowledge Graph Question Answering
Yuan Sui ⋅ Yufei He ⋅ Nian Liu ⋅ Xiaoxin He ⋅ Kun Wang ⋅ Bryan Hooi
Towards Understanding Fine-Tuning Mechanisms of LLMs via Circuit Analysis
Xu Wang ⋅ Yan Hu ⋅ Wenyu Du ⋅ Reynold Cheng ⋅ Wang Benyou ⋅ Difan Zou
Mind the Gap: A Practical Attack on GGUF Quantization
Kazuki Egashira ⋅ Robin Staab ⋅ Mark Vero ⋅ Jingxuan He ⋅ Martin Vechev
Model Evaluations Need Rigorous and Transparent Human Baselines
Kevin Wei ⋅ Patricia Paskov ⋅ Sunishchal Dev ⋅ Michael Byun ⋅ Anka Reuel ⋅ Xavier Roberts-Gaal ⋅ Rachel Calcott ⋅ Evie Coxon ⋅ Chinmay Deshpande
The Jailbreak Tax: How Useful are Your Jailbreak Outputs?
Kristina Nikolić ⋅ Luze Sun ⋅ Jie Zhang ⋅ Florian Tramer
AntifakePrompt: Prompt-Tuned Vision-Language Models are Fake Image Detectors
You Ming Chang ⋅ Chen Yeh ⋅ Wei-Chen Chiu ⋅ Ning Yu
Reliable and Efficient Amortized Model-based Evaluation
Sang Truong ⋅ Yuheng Tu ⋅ Percy Liang ⋅ Bo Li ⋅ Sanmi Koyejo
Integrated Gradients Provides Faithful Language Model Attributions for In-Context Learning
Theo Datta ⋅ Erik Wang ⋅ Kayla Huang ⋅ Finale Doshi-Velez
Red Teaming for Trust: Evaluating Multicultural and Multilingual AI Systems in Asia-Pacific
Akash Kundu ⋅ Adrianna Tan ⋅ Theodora Skeadas ⋅ Rumman Chowdhury ⋅ Sarah Amos
Top of the CLASS: Benchmarking LLM Agents on Real-World Enterprise Tasks
Michael Wornow ⋅ Vaishnav Garodia ⋅ Vasilis Vassalos ⋅ Utkarsh Contractor
Self-Ablating Transformers: More Interpretability, Less Sparsity
Jeremias Ferrao ⋅ Luhan Mikaelson ⋅ Keenan Pepper ⋅ Natalia Perez-Campanero
PATTERNS AND MECHANISMS OF CONTRASTIVE ACTIVATION ENGINEERING
Yixiong Hao ⋅ Ayush Panda ⋅ Stepan Shabalin ⋅ Sheikh Abdur Raheem Ali
StochasTok: Improving Fine-Grained Subword Understanding in LLMs
Anya Sims ⋅ Cong Lu ⋅ Klara Kaleb ⋅ Jakob Foerster ⋅ Yee Whye Teh
LLMS LOST IN TRANSLATION: M-ALERT UNCOVERS CROSS-LINGUISTIC SAFETY GAPS
Felix Friedrich ⋅ Simone Tedeschi ⋅ Patrick Schramowski ⋅ Manuel Brack ⋅ Roberto Navigli ⋅ Huu Nguyen ⋅ Bo Li ⋅ Kristian Kersting
No, Of Course I Can! Refusal Mechanisms Can Be Exploited Using Harmless Data
Joshua Kazdan ⋅ Lisa Yu ⋅ Rylan Schaeffer ⋅ Chris Cundy ⋅ Sanmi Koyejo ⋅ Krishnamurthy Dvijotham
Black-Box Adversarial Attacks on LLM-Based Code Completion
Slobodan Jenko ⋅ Niels Mündler ⋅ Jingxuan He ⋅ Mark Vero ⋅ Martin Vechev
Has My System Prompt Been Used? Large Language Model Prompt Membership Inference
Roman Levin ⋅ Valeriia Cherepanova ⋅ Abhimanyu Hans ⋅ Avi Schwarzschild ⋅ Tom Goldstein
Unnatural Languages Are Not Bugs but Features for LLMs
Keyu Duan ⋅ Yiran Zhao ⋅ Zhili Feng ⋅ Jinjie Ni ⋅ Tianyu Pang ⋅ Qian Liu ⋅ Tianle Cai ⋅ Longxu Dou ⋅ Kenji Kawaguchi ⋅ Anirudh Goyal ⋅ Zico Kolter ⋅ Michael Qizhe Shieh
MASAN: Enhancing Attack Stealth and Efficacy on Vision-Language Models via Smart Noise
Shuaiqi Wang ⋅ Sayali Deshpande ⋅ Rajesh Kudupudi ⋅ Alireza Mehrtash ⋅ Danial Dashti
In-Context Meta Learning Induces Multi-Phase Circuit Emergence
Gouki Gouki ⋅ Hiroki Furuta ⋅ Shohei Taniguchi ⋅ Yusuke Iwasawa ⋅ Yutaka Matsuo
Monitoring LLM Agents for Sequentially Contextual Harm
Chen Yueh-Han ⋅ Nitish Joshi ⋅ Yulin Chen ⋅ He He ⋅ Rico Angell
Mechanistic Anomaly Detection for "Quirky'' Language Models
David Johnston ⋅ Arkajyoti Chakraborty ⋅ Nora Belrose
The Steganographic Potentials of Language Models
Artem Karpov ⋅ Tinuade Adeleke ⋅ Seong Hah Cho ⋅ Natalia Perez-Campanero
How Does Entropy Influence Modern Text-to-SQL Systems?
Varun Kausika ⋅ chris lazar ⋅ Satya Mishra ⋅ Saurabh Jha ⋅ Priyanka Pathak
Understanding (Un)Reliability of Steering Vectors in Language Models
Joschka Braun ⋅ Carsten Eickhoff ⋅ David Krueger ⋅ Seyed Ali Bahrainian ⋅ Dmitrii Krasheninnikov
Working Memory Attack on LLMs
Bibek Upadhayay ⋅ Vahid Behzadan ⋅ Amin Karbasi
Do Multilingual LLMs Think In English?
Lisa Schut ⋅ Yarin Gal ⋅ Sebastian Farquhar
ASIDE: Architectural Separation of Instructions and Data in Language Models
Egor Zverev ⋅ Evgenii Kortukov ⋅ Alexander Panfilov ⋅ Soroush Tabesh ⋅ Sebastian Lapuschkin ⋅ Wojciech Samek ⋅ Christoph Lampert
Latent Adversarial Training Improves the Representation of Refusal
Alexandra Abbas ⋅ Nora Petrova ⋅ Hélios Lyons ⋅ Natalia Perez-Campanero
Evaluation of Large Language Models via Coupled Token Generation
Nina Corvelo Benz ⋅ Stratis Tsirtsis ⋅ Eleni Straitouri ⋅ Ivi Chatzi ⋅ Ander Artola Velasco ⋅ Suhas Thejaswi ⋅ Manuel Gomez Rodriguez
The Differences Between Direct Alignment Algorithms are a Blur
Alexey Gorbatovski ⋅ Boris Shaposhnikov ⋅ Viacheslav Sinii ⋅ Alexey Malakhov ⋅ Daniil Gavrilov
Diagnostic Uncertainty: Teaching Language Models to Describe Open-Ended Uncertainty
Brian Sui ⋅ Jessy Lin ⋅ Michelle Li ⋅ Anca Dragan ⋅ Dan Klein ⋅ Jacob Steinhardt
Language Models Use Trigonometry to Do Addition
Subhash Kantamneni ⋅ Max Tegmark
Systematic Evaluation of LLM-as-a-Judge in LLM Alignment Tasks: Explainable Metrics and Diverse Prompt Templates
Hui Wei ⋅ Shenghua He ⋅ Tian Xia ⋅ Fei Liu ⋅ Andy Wong ⋅ Jingyang Lin ⋅ Mei Han
Endive: A Cross-Dialect Benchmark for Fairness and Performance in Large Language Models
Abhay Gupta ⋅ Jacob Cheung ⋅ Philip Meng ⋅ Shayan Sayyed ⋅ Austen Liao ⋅ Kevin Zhu ⋅ Sean OBrien
Rethinking LLM Bias Probing Using Lessons from the Social Sciences
Kirsten Morehouse ⋅ Siddharth Swaroop ⋅ Weiwei Pan
AdvBDGen: A Robust Framework for Generating Adaptive and Stealthy Backdoors in LLM Alignment Attacks
Pankayaraj Pathmanathan ⋅ Udari Sehwag ⋅ Michael-Andrei Panaitescu-Liess ⋅ Furong Huang
Automated Red Teaming with GOAT: the Generative Offensive Agent Tester
Maya Pavlova ⋅ Erik Brinkman ⋅ Krithika Iyer ⋅ Vítor Albiero ⋅ Joanna Bitton ⋅ Hailey Nguyen ⋅ Cristian Ferrer ⋅ Ivan Evtimov ⋅ Aaron Grattafiori
PRUNING AS A DEFENSE: REDUCING MEMORIZATION IN LARGE LANGUAGE MODELS
Mansi Gupta ⋅ Nikhar Waghela ⋅ Sarthak Gupta ⋅ Shourya Goel ⋅ Sanjif Shanmugavelu
Harmful Helper: Perform malicious tasks? Web AI agents might help
Yang Fan Chiang ⋅ Seungjae (Jay) Lee ⋅ Jia-Bin Huang ⋅ Furong Huang ⋅ Yizheng Chen
CleanGen: Mitigating Backdoor Attacks for Generation Tasks in Large Language Models
Yuetai Li ⋅ Zhangchen Xu ⋅ Fengqing Jiang ⋅ Luyao Niu ⋅ Dinuka Sahabandu ⋅ Bhaskar Ramasubramanian ⋅ Radha Poovendran
Steering Fine-Tuning Generalization with Targeted Concept Ablation
Helena Casademunt ⋅ Caden Juang ⋅ Samuel Marks ⋅ Senthooran Rajamanoharan ⋅ Neel Nanda
Disentangling Sequence Memorization and General Capability in Large Language Models
Gaurav Ghosal ⋅ Pratyush Maini ⋅ Aditi Raghunathan
Scalable Fingerprinting of Large Language Models
Anshul Hemant Nasery ⋅ Jonathan Hayase ⋅ Creston Brooks ⋅ Peiyao Sheng ⋅ Himanshu Tyagi ⋅ Pramod Viswanath ⋅ Sewoong Oh
ToolScan: A Benchmark For Characterizing Errors In Tool-Use LLMs
Shirley Kokane ⋅ Ming Zhu ⋅ Tulika Awalgaonkar ⋅ Jianguo Zhang ⋅ Akshara Prabhakar ⋅ Thai Hoang ⋅ Zuxin Liu ⋅ Rithesh Ramapura Narasimha Murthy ⋅ Liangwei Yang ⋅ Weiran Yao ⋅ Juntao Tan ⋅ Zhiwei Liu ⋅ Shelby Heinecke ⋅ Huan Wang ⋅ Juan Carlos Niebles ⋅ Caiming Xiong ⋅ Silvio Savarese
AegisLLM: Scaling Agentic Systems for Self-Reflective Defense in LLM Security
Zikui Cai ⋅ Shayan Shabihi ⋅ Bang An ⋅ Zora Che ⋅ Brian Bartoldson ⋅ Bhavya Kailkhura ⋅ Tom Goldstein ⋅ Furong Huang