To reduce open space risk of hypotheses, we reexamine the 'simplest' hypothesis class, binary linear classifiers, geometrically.Providing a generalized formulation,we establish a surprising fact: linear classifiers can have arbitrarily high VC dimension, stemming from increasing the number of partitions in input space.Hence, linear classifiers with multiple margins are more expressive than single-margin classifiers.Despite a higher VC dimension, such classifiers have less open space risk than halfspace separators.These geometric insights are useful to detect unseen classes, while probabilistic modeling of risk minimization helps with seen classes. In supervised anomaly detection, we show that a classifier that combines a probabilistic and geometric lens can detect both seen and unseen anomalies well.