Action recognition models have shown promising results in understanding consecutive human actions in instructional videos. However, they often rely on dominant action patterns in datasets rather than achieving true video comprehension. We define this as ordinal bias, a systematic reliance on dataset-specific action sequences. To mitigate this, we introduce two simple yet effective video manipulation techniques: action masking and sequence shuffling, where the latter action in dominant pairs is masked, or the sequence is randomized. Our findings reveal that existing models still tend to rely on dominant action pairs and struggle to adapt, highlighting their overestimated performance and lack of robustness.