The application of large deep learning models in weather forecasting has led tosignificant advancements in the field, including higher-resolution forecasting andextended prediction periods exemplified by models such as Pangu and Fuxi. Despitethese successes, previous research has largely been characterized by the neglectof extreme weather events, and the availability of datasets specifically curated forsuch events remains limited. Given the critical importance of accurately forecastingextreme weather, this study introduces a comprehensive dataset that incorporateshigh-resolution extreme weather cases derived from the High-Resolution RapidRefresh (HRRR) data, a 3-km real-time dataset provided by NOAA. We alsoevaluate the current state-of-the-art deep learning models and Numerical WeatherPrediction (NWP) systems on HR-Extreme, and provide a improved baselinedeep learning model called HR-Heim which has superior performance on bothgeneral loss and HR-Extreme compared to others. Our results reveal that theerrors of extreme weather cases are significantly larger than overall forecast error,highlighting them as an crucial source of loss in weather prediction. These findingsunderscore the necessity for future research to focus on improving the accuracy ofextreme weather forecasts to enhance their practical utility