Language Models today provide a high accuracy across a large number of downstream tasks. However, they remain susceptible to adversarial attacks, particularly against those where the adversarial examples maintain considerable similarity to the original text. Given the multilingual nature of text, the effectiveness of adversarial examples across translations and how machine translations can improve the robustness of adversarial examples remain largely unexplored. In this paper, we present a comprehensive study on the robustness of current text adversarial attacks to round-trip translation. We demonstrate that 6 state-of-the-art text-based adversarial attacks do not maintain their efficacy after round-trip translation. Furthermore, we introduce an intervention-based solution to this problem, by integrating Machine Translation into the process of adversarial example generation and demonstrating an increased robustness to round-trip translation. Our results indicate that finding adversarial examples robust to round-trip translation can help identify insufficiency of language models that is common across languages, and motivate further research into multilingual adversarial attacks.

Due to growing concerns about demographic disparities and discrimination resulting from algorithmic and model-based decision-making, recent research has focused on mitigating biases against already disadvantaged or marginalised groups in classification models. From the perspective of classification parity, the two commonest metrics for assessing fairness are statistical parity and equality of opportunity. Current approaches to debiasing in classification either require the knowledge of the protected attribute before or during training or are entirely agnostic to the model class and parameters. This work considers differentiable proxy functions for statistical parity and equality of opportunity and introduces two novel debiasing techniques for neural network classifiers based on fine-tuning and pruning an already-trained network. As opposed to the prior work leveraging adversarial training, the proposed methods are simple yet effective and can be readily applied post hoc. Our experimental results encouragingly suggest that these approaches successfully debias fully connected neural networks trained on tabular data and often outperform model-agnostic post-processing methods.

In this work, we propose FedER, a federated learning method that is both efficient and robust. Our key idea is to reduce the communication cost of the state-of-the-art robust FL method via pruning the model updates. Specifically, the server collects a small clean dataset, which is split into a training set and a validation set. In each round of FL, the clients prune their model updates before sending them to the server. The server also derives a server model update based on the training set and prunes it. The server determines the pruning fraction via evaluating the model accuracy on the validation set. We further propose mutual masking for each client, which computes the parameters in the overlapping area of pruned client model update and server model update. The mutual mask is used to filter out the parameters of unusual dimensions in malicious updates. We also occasionally normalize the masked client model updates to limit the impact of attacks. Our extensive experiments show that FedER 1) significantly reduces the communication cost for clients in adversarial settings and 2) achieves comparable or even better robustness compared to the state-of-the-art Byzantine-robust method.

In recent years, Machine-Learning (ML)-driven approaches have been widely used in scientific discovery domains. Among them, the Fourier Neural Operator (FNO) \citep{zongyi} was the first to simulate turbulent flow with zero-shot super-resolution and superior accuracy, which significantly improves the speed when compared to traditional partial differential equation (PDE) solvers. To inspect the trustworthiness, we provide the first study on the adversarial robustness of scientific discovery models by generating adversarial examples for FNO, based on norm-bounded data input perturbations. Evaluated on the mean squared error between the FNO model's output and the PDE solver's output, our results show that the model's robustness degrades rapidly with increasing perturbation levels, particularly in non-simplistic cases like the 2D Darcy and the Navier cases. Our research provides a sensitivity analysis tool and evaluation principles for assessing the adversarial robustness of ML-based scientific discovery models.

Randomized Smoothing (RS) is considered the state-of-the-art approach to obtain certifiably robust models for challenging tasks. However, current RS approaches drastically decrease standard accuracy on unperturbed data, severely limiting their real-world utility. To address this limitation, we propose a compositional architecture, ACES, which certifiably decides on a per-sample basis whether to use a smoothed model yielding predictions with guarantees or a more accurate standard model without guarantees. This, in contrast to prior approaches, enables both high standard accuracies and significant provable robustness. On challenging tasks such as ImageNet, we obtain, e.g., 80.0% natural accuracy and 28.2% certifiable accuracy against l2 perturbations with r = 1.0. We release our code and models at https://github.com/eth-sri/aces.

While differentially private query release has been well-studied, research in this area is commonly restricted to data that do not exhibit hierarchical structure. However, in many real-world scenarios, individual data points can be grouped together (e.g., people within households, taxi trips per driver, etc.), begging the question---what statistical properties (or queries) are important when considering data of this form? In addition, although synthetic data generation approaches for private query release have grown increasingly popular, it is unclear how one can generate synthetic data at both the group and individual-level while capturing such statistical properties. In light of these challenges, we formalize the problem of hierarchical query release and provide a set of statistical queries that capture relationships between attributes at both the group and individual-level. Furthermore, we propose and implement a novel synthetic data generation algorithm, H-GEM, which outputs hierarchical data subject to differential privacy to answer such statistical queries. Finally, using the American Community Survey, we evaluate H-GEM, establishing a benchmark for future work to measure against.

We study the design of a class of incentive mechanisms that can effectively improve algorithm robustness in strategic learning. A conventional strategic learning problem is modeled as a Stackelberg game between an algorithm designer (a principal, or decision maker) and individual agents subject to the algorithm's decisions, potentially from different demographic groups. While the former benefits from the decision accuracy, the latter may have an incentive to game the algorithm into making favorable but erroneous decisions by merely changing their observable features without affecting their true labels. While prior works tend to focus on how to design decision rules robust to such strategic maneuvering, this study focuses on an alternative, which is to design incentive mechanisms to shape the utilities of the agents and induce improvement actions that genuinely improve their skills and true labels and thus, in turn, benefit both parties in the Stackelberg game. Specifically, the principal and the mechanism provider (could be the principal itself) move together in the first stage, publishing and committing to a classifier and an incentive mechanism. The agents are second movers and best respond to the published classifier and incentive mechanism. We study how the mechanism can induce improvement actions, positively impact a number of social well-being metrics, such as the overall skill levels of the agents (efficiency) and positive or true positive rate differences between different demographic groups (fairness).

As we increasingly rely on artificially intelligent algorithms to aid or automate decision making, we face the challenge of ensuring that these algorithms do not exhibit or amplify our existing social biases. An issue complicating the design of such fair AI is that algorithms are trained on datasets that can themselves be tainted due to the social biases of prior (human or AI) decision makers. In this paper, we investigate the robustness of existing (group) fairness criteria when an algorithm is trained on data that is biased due to errors by prior decision makers in identifying qualified individuals from a disadvantaged group. This can be viewed as labeling bias in the data. We first analytically show that some constraints such as Demographic Parity remain robust when facing such statistical biases, while others like Equalized Odds are violated if trained on biased data. We also analyze the sensitivity of the firm's utility to these biases under each constraint. Finally, we provide numerical experiments on three real-world datasets (the FICO, Adult, and German credit score datasets) supporting our analytical findings.

Deep networks have been found to be highly susceptible to adversarial attacks. One fundamental challenge is that it is typically possible for small input perturbations to produce large movements in the final-layer feature space of these networks. In this work, we define an attack model that abstracts this challenge, to help understand its intrinsic properties. In our model, the adversary may move data an arbitrary distance in feature space but only in random low-dimensional subspaces. We prove that such adversaries can be quite powerful: defeating any classifier that must output a class prediction on any input it is given. However, by giving the algorithm the ability to abstain, we show that such an adversary can be overcome when classes are reasonably well-separated in feature space and the dimension of the feature space is high, by an algorithm that examines distances of test points to training data in feature space. We further show how data-driven methods can be used to set algorithm parameters to optimize over the accuracy vs. abstention trade-off with strong theoretical guarantees. Our theory can also be viewed as providing new robustness guarantees for nearest-neighbor style algorithms, and has direct applications to the technique of contrastive learning, where we empirically demonstrate the ability of such algorithms to obtain high robust accuracy with only small amounts of abstention. Overall, our results provide insight into the intrinsic vulnerabilities of non-Lipschitz networks and the ways these may be addressed.

In fair machine learning, the goal is to train models that exhibit low bias while maintaining the utility. Most fair learning approaches assume the existence of demographic attributes on all of the data, which limits their usability. In contrast, some recent works introduce algorithms that can function without any demographic labels. In this work, we show these approaches tend to exhibit relatively high bias. Given that, we develop fair learning algorithms that can function with only a small number of demographically labeled data. Our experiments illustrate that our approaches train models better fairness-utility trade-offs.

The Shapley value (SV) and Least core (LC) are classic methods in cooperative game theory for cost/profit sharing problems. Both methods have recently been proposed as a principled solution for data valuation tasks, i.e., quantifying the contribution of individual datum in machine learning. However, both SV and LC suffer computational challenges due to the need for retraining models on combinatorially many data subsets. In this work, we propose to boost the efficiency in computing Shapley value or Least core by learning to estimate the performance of a learning algorithm on unseen data combinations. Theoretically, we derive bounds relating the error in the predicted learning performance to the approximation error in SV and LC. Empirically, we show that the proposed method can significantly improve the accuracy of SV and LC estimation with negligible additional runtime.

In federated learning, fair prediction across various protected groups (e.g., gender, race) is an important constraint for many applications. Unfortunately, prior work studying group fair federated learning lacks formal convergence or fairness guarantees. Our work provides a new definition for group fairness in federated learning based on the notion of Bounded Group Loss (BGL), which can be easily applied to common federated learning objectives. Based on our definition, we propose a scalable algorithm that optimizes the empirical risk and global fairness constraints, which we evaluate across a number of common fairness and federated learning benchmarks. Our resulting method and analysis are the first we are aware of to provide formal theoretical guarantees for training a fair federated learning model.

Secure aggregation is a popular protocol for privacy-aware model aggregation in federated learning. However, due to its large communication overhead, users with scarce wireless resources are unable to participate in the protocol as much as users with better wireless conditions, which can lead to significant bias against users from underserved communities. Towards addressing this challenge, in this work we propose a communication-efficient gradient sparsification technique for secure aggregation, where the server learns the aggregate of sparsified local gradients from a large number of users, without having access to the individual local gradients. Through large-scale distributed experiments with up to 100 users, we demonstrate up to 27x reduction in the communication overhead, and up to 8x speed up in the wall clock training time, compared to conventional secure aggregation.

An extractive rationale explains a language model's (LM's) prediction on a given task instance by highlighting the text inputs that most influenced the prediction. Ideally, rationale extraction should be faithful (reflective of LM's actual behavior) and plausible (convincing to humans), without compromising the LM's (i.e., task model's) task performance. Although attribution algorithms and select-predict pipelines are common in rationale extraction, they both rely on certain heuristics that hinder them from satisfying all three desiderata. In light of this, we propose UNIREX, a flexible learning framework which generalizes rationale extractor optimization as follows: (1) specify architecture for a learned rationale extractor; (2) select explainability objectives (i.e., faithfulness and plausibility criteria); and (3) jointly the train task model and rationale extractor on the task using selected objectives. UNIREX enables replacing prior works' heuristic design choices with a generic learned rationale extractor in (1) and optimizing it for all three desiderata in (2)-(3). To facilitate comparison between methods w.r.t. multiple desiderata, we introduce the Normalized Relative Gain (NRG) metric. Across five text classification datasets, our best UNIREX configuration outperforms baselines by an average of 32.9% NRG.Plus, we find that UNIREX-trained rationale extractors can even generalize to unseen datasets and tasks.

We propose a framework for sequential decision-making aimed at dynamically influencing long-term societal fairness, illustrated via the problem of selecting applicants from a pool consisting of two groups, one of which is under-represented. We consider a dynamic model for the composition of the applicant pool, in which admission of more applicants from a group in a given selection round positively reinforces more candidates from the group to participate in future selection rounds. Under such a model, we show the efficacy of the proposed Fair-Greedy selection policy which systematically trades the sum of the scores of the selected applicants (greedy'') against the deviation of the proportion of selected applicants belonging to a given group from a target proportion (fair''). In addition to experimenting on synthetic data, we adapt static real-world datasets on law school candidates and credit lending to simulate the dynamics of the composition of the applicant pool. We prove that the applicant pool composition converges to a target proportion set by the decision-maker when score distributions across the groups are identical.

Deep neural networks on 3D point cloud data have been widely used in the real world, especially in safety-critical applications. However, their robustness against corruptions is less studied. In this paper, we present ModelNet40-C, the first comprehensive benchmark on 3D point cloud \textit{corruption robustness}, consisting of 15 common and realistic corruptions. Our evaluation shows a significant gap between the performances on ModelNet40 and ModelNet40-C for state-of-the-art (SOTA) models. We also demonstrate the effectiveness of different data augmentation strategies in enhancing robustness for different corruption types. We hope our in-depth analysis will motivate the development of robust training strategies or architecture designs in the 3D point cloud domain. Our codebase and dataset will be made available upon acceptance.

Differential privacy provides a theoretical framework for processing a dataset about n users, in a way that the output reveals a minimal information about anysingle user. Such notion of privacy is usually ensured by noise-adding mechanisms and amplified by several processes, including subsampling, shuffling, iteration, mixing and diffusion. In this work, we provide privacy amplification bounds for quantum and quantum-inspired algorithms. In particular, we show for the firsttime, that algorithms running on quantum encoding of a classical dataset or the outcomes of quantum-inspired classical sampling, amplify differential privacy.Moreover, we prove that a quantum version of differential privacy is amplified by the composition of quantum channels, provided that they satisfy some mixing conditions.

In this work, we address the problem of learning provably stable neural network policies for stochastic control systems.While recent work has demonstrated the feasibility of certifying given policies using martingale theory, the problem of how to learn such policies is little explored.Here, we study the effectiveness of jointly learning a policy together with a martingale certificate that proves its stability using a single learning algorithm.We observe that the joint optimization problem becomes easily stuck in local minima when starting from a randomly initialized policy. Our results suggest that some form of pre-training of the policy is required for the joint optimization to repair and verify the policy successfully.

The goal of algorithmic recourse is to reverse unfavorable decisions under automated decision making by suggesting actionable changes (e.g., reduce the number of credit cards). Such changes allow individuals to achieve favorable outcomes (e.g., loan approval) under low costs for the affected individual. To suggest low cost recourse, several recourse methods have been proposed in recent literature. These techniques usually generate recourses under the assumption that the features are independently manipulable. This, however, can be misleading since the omission of feature dependencies comes with the risk that some required recourse changes are overlooked. In this work, we propose a novel, theory-driven framework, DisEntangling Algorithmic Recourse (DEAR), that suggests a solution to this problem by leveraging disentangled representations to find interpretable, small-cost recourse actions under input dependencies. Our framework addresses the independently manipulable feature (IMF) assumption by dissecting recourse actions into direct and indirect actionable changes.

As machine learning systems are increasingly employed in high-stakes tasks, algorithmic fairness has become an essential requirement for deep learning models. In this paper, we study how to transfer fairness under distribution shifts, a crucial issue in real-world applications. We first derive a sufficient condition for transferring group fairness. Guided by it, we propose a practical algorithm with a fair consistency regularization as the key component. Experiments on synthetic and real datasets demonstrate that our approach can effectively transfer fairness as well as accuracy under distribution shifts, especially under domain shift which is a more challenging but practical scenario.

Concept bottleneck models perform classification by first predicting which of a list of human provided concepts are true about a datapoint. Then a downstream model uses these predicted concept labels to predict the target label.The predicted concepts act as a rationale for the target prediction.Model trust issues emerge in this paradigm when soft concept labels are used: it has previously been observed that extra information about the data distribution leaks into the concept predictions.In this work we show how Monte-Carlo Dropout can be used to attain soft concept predictions that do not contain leaked information.

We consider the problem of machine unlearning to erase a target data, which is used in training but incorrect or sensitive, from a trained model while the training dataset is inaccessible. Previous works have assumed that the target data completely represent all the data to be erased. However, it is often infeasible to indicate all the data to be erased. We hence address a practical scenario of unlearning from a few samples of target data, so-called few-shot unlearning. To this end, we devise a few-shot unlearning method. We demonstrate that our method using only a subset of target data can outperform the state-of-the-art methods with a full indication of target data.

Machine learning models deployed as a service(MLaaS) are often susceptible tomodel stealing attacks. While existing works demonstrate near-perfect performance using softmax predictions of the classification network, most of the APIs allow access to only the top-1 labels. In this work, we show that it is indeed possible to steal Machine Learning models by accessing only top-1 predictions (Hard Label setting), without access to model gradients (Black-Box setting) and even the training dataset (Data-Free setting) within a low query budget. We propose a novel GAN-based framework that trains the student and generator in tandem to steal the model effectively while utilizing gradients of the clone network as a proxy to the victim’s gradients. We overcome the large query costs by utilizing publicly available (potentially unrelated) datasets as a weak image prior. We additionally show that even in the absence of such data, it is possible to achieve state-of-the-art results within a low query budget using synthetically crafted samples. We are the first to demonstrate the scalability of Model Stealing on a 100 class dataset.

As machine learning models are increasingly being deployed in high-stakes applications, there has been growing interest in providing recourse to individuals adversely impacted by model predictions (e.g., an applicant whose loan has been denied). To this end, several post hoc techniques have been proposed in recent literature. These techniques generate recourses under the assumption that the affected individuals will implement the prescribed recourses exactly. However, recent studies suggest that individuals often implement recourses in a noisy and inconsistent manner - e.g., raising their salary by $505 if the prescribed recourse suggested an increase of $500. Motivated by this, we introduce and study the problem of recourse invalidation in the face of noisy human responses. We propose a novel framework, EXPECTing noisy responses (EXPECT), which addresses the aforementioned problem by explicitly minimizing the probability of recourse invalidation in the face of noisy responses. Experimental evaluation with multiple real-world datasets demonstrates the efficacy of the proposed framework, and supports our theoretical findings.

Supervised machine learning is widely used for selection problems where an individual is selected among a pool of applicants. This problem can be applied in loan lending, hiring, and university admission. Machine learning models often suffer from a bias towards specific sensitive attributes (e.g., race, gender) as a reflection of pre-existing discrimination in the dataset. On top of unfairness, privacy concerns may incur when models are trained on sensitive personal information. In this work, we study the possibility of using a differentially private Laplace mechanism to enhance fairness and privacy in the selection problem. Also, we defined a condition that can make the selection problem fair and private.

Models that generate extractive rationales (ERs) (i.e., subsets of features) or natural language explanations (NLEs) for their predictions are important for explainable AI. While an ER provides a quick view of the features most responsible for a prediction, an NLE allows for a comprehensive description of the decision-making process behind a prediction. However, current models that generate the best ERs or NLEs often fall behind the state-of-the-art (SOTA) in terms of task performance. In this work, we bridge this gap by introducing RExC, a self-rationalizing framework that grounds its predictions and two complementary types of explanations (NLEs and ERs) in background knowledge. RExC improves over previous methods by: (i) reaching SOTA task performance while also providing explanations, (ii) providing two types of explanations while existing models usually provide only one, and (iii) beating by a large margin the previous SOTA in terms of quality of explanations. Furthermore, a perturbation analysis in RExC shows a high degree of association between explanations and predictions, a necessary property of faithful explanations.

Supervised learning methods that directly optimize the cross entropy loss on training data often overfit. This overfitting is typically mitigated through regularizing the loss function (e.g., label smoothing) or by minimizing the same loss on new examples (e.g., data augmentation and adversarial training). In this work, we propose a complementary regularization strategy: Maximum Predictive Entropy (MPE) forcing the model to be uncertain on new, algorithmically-generated inputs. Across a range of tasks, we demonstrate that our computationally-efficient method improves test accuracy, and the benefits are complementary to methods such as label smoothing and data augmentation.

There has been an increased interest in applying deep neural networks to automatically interpret and analyze the 12-lead electrocardiogram (ECG). However, the imbalance and heterogeneity of real-world datasets place obstacles to the efficient training of neural networks. Moreover, deep learning classifiers could be vulnerable to adversarial examples and perturbations and could lead to catastrophic outcomes for clinical trials and insurance claims.In this paper, we propose a physiologically-inspired data augmentation to improve the performance, generalization, and to increase the robustness of ECG prediction models. We obtain augmented samples by perturbing the data distribution towards other classes along the geodesic in Wasserstein space. To better utilize the domain knowledge, we design a ground metric that recognizes the difference between ECG signals based on physiological features. Learning from 12-lead ECG signals, our model is able to distinguish five categories of cardiac conditions. Our results demonstrate improvements in accuracy and robustness reflecting the effectiveness of our data augmentation method.