AD3: Implicit Action is the Key for World Models to Distinguish the Diverse Visual Distractors