Abstract: Recent lines of work have proposed learning disentangled representations using observed auxiliary variables by $\textit{mechanism sparsity regularization}$. These works assume that the pairing between the auxiliary variables and samples is known. Inspired by biological problems in controllable counterfactual generation and mechanism transportability for genomics explorations, this work combines mechanism sparsity regularization and methods from Continual Learning to introduce a representation learning method which applies when the auxiliary variables are not directly observed and the assignment between the latent auxiliary variables and samples is not known. Rather than requiring observed auxiliary variables for disentanglement, we propose to use realizations of the auxiliary variables of interest. We propose an estimation procedure based on variational autoencoders and demonstrate it on various synthetic and biological data in generating counterfactual instances of cell states or transcriptional signatures to achieve desired cell state shifts.