Fast Updating Truncated SVD for Representation Learning with Sparse Matrices

Updating truncated Singular Value Decomposition (SVD) has extensive applications in representation learning.The continuous evolution of massive-scaled data matrices in practical scenarios highlights the importance of aligning SVD-based models with fast-paced updates.Recent methods for updating truncated SVD can be recognized as Rayleigh-Ritz projection procedures where their projection matrices are augmented based on the original singular vectors.However, the updating process in these methods densifies the update matrix and applies the projection to all singular vectors, resulting in inefficiency.This paper presents a novel method for dynamically approximating the truncated SVD of a sparse and temporally evolving matrix.The proposed method takes advantage of sparsity in the orthogonalization process of the augment matrices and employs an extended decomposition to store projections in the column space of singular vectors independently.Numerical experimental results on updating truncated SVD for evolving sparse matrices show an order of magnitude improvement in the efficiency of our proposed method while maintaining precision comparing to previous methods.