Abstract: Standard benchmarks often suggest that tasks like image classification are largely "solved". However, practitioners deploying models in industry frequently encounter scenarios where these solutions frequently fail, especially in critical fields like online safety and content moderation, where there is a long tail of special cases. This talk examines why performance frequently falls short of benchmark expectations in practice. We'll investigate key challenges such as: 1) efficiently obtaining nuanced specialized data; 2) understanding critical evaluation limitations; 3) efficiently training specialized models. Finally, we'll assess how Large Language Models (LLMs) can potentially assist, while also highlighting their limitations. Attendees will gain insights into diagnosing failures and developing robust strategies for tackling niche classification tasks in the real world.