Our work on search strategy improvement centers around a core idea: Through learnable modeling of the search space (i.e., a performance predictor of architectures), we can know which search space regions are worth exploring, and thus accelerate the exploration process. This is the underlying idea of predictor-based NAS methods, including parametric or non-parametric ones (e.g., bayesian optimization). Here, we present our studies along this direction.

1. Following the intuition that "an architecture describes how the data flow and get processed", we propose to mimic the data flow and processing process to encode the architecture. The proposed GATES is a more suitable encoding method for data-processing directed acyclic graphs (DAGs) which matches the nature of this type of data, and surely, it can encode equivalent / isomorphic architectures to the same embedding intrinsically. We also propose to use the ranking loss to train the predictor as providing correct ranking is usually more important than predicting absolutely close performances in NAS.

2. TA-GATES is an improvement over GATES that can get contextualized embeddings for different operations (even those of the same type). Actually, we people surely know that even if two operations are of the same type, they have different functionalities according to their architectural context. However, plain GATES does not give contextualized embeddings for different operations of the same type. To get a more discriminative encoding, the intuition behind the principled design of TA-GATES is "an architecture not only describes how the data flow and get processed in the forward propagation, but it also decides the learning dynamics of the model". Accordingly, we propose to mimic the training process of an architecture to encode it, by conducting several forward and backward passes on the architecture DAG, and updating the operation embedding in each backward pass. TA-GATES can also enable many interesting applications: (1) Any-time performance prediction, which could be handy in early-stop / multi-fidelity NAS. (2) A natural joint encoding of other factors in the deep learning pipeline (e.g., loss design, hyper-parameters) for joint AutoDL. This is because the encoding process of TA-GATES explicitly mimics the model training process, it is straightforward to find the corresponding counterparts of those training-time factors in the encoding process.

3. Based on the intuition that "low-fidelity information can be beneficial for learning the modeling", we propose a dynamic mixture-of-expert predictor framework, DELE, to fuse beneficial knowledge from different low-fidelity experts. Specifically, each low-fidelity expert is trained with the aid of one type of low-fidelity information (e.g., zero-shot evaluation scores, complexity scores, and so on), and then a dynamic ensemble of these experts is trained using only a small set of ground-truth performance data. DELE is orthogonal to GATES and TA-GATES (specialized encoder designs for neural architectures according to their characteristics) in that every expert in DELE can be a GATES and TA-GATES.