It Was Never Really About Prompts or Context