What Layers Do for Your Style