Neural Architecture Search — a compact, categorized walkthrough

Neural Architecture Search (NAS) automates design of neural network architectures, trading manual trial-and-error for systematic search. Below is a curated, categorized list of influential papers, frameworks, benchmarks, and practical notes — each linked to a stable online source where possible. Use this as a jumping-off point for survey reading, trying out code, or building hardware-aware NAS flows.

Surveys & Reviews

A Survey on Neural Architecture Search — arXiv

A broad overview of search spaces, search strategies (RL, evolution, gradient, Bayesian), and performance estimation techniques.

Core NAS Methods & Milestones

Neural Architecture Search with Reinforcement Learning — arXiv (Zoph & Le)

One of the first influential papers applying reinforcement learning to automatically generate high-performing CNN cells.

ENAS: Efficient Neural Architecture Search via Parameter Sharing — arXiv

Introduced parameter sharing to drastically lower NAS compute cost by reusing weights across sampled architectures.

DARTS: Differentiable Architecture Search — arXiv

Formulated NAS as a differentiable optimization problem, enabling gradient-based search over continuous architecture parameters.

ProxylessNAS — arXiv

Performs NAS directly on target task and hardware (no proxy tasks), focusing on latency-aware architectures for mobile devices.

Once-for-All (OFA) — arXiv

Trains a single, elastic supernet that supports many subnetworks; enables instant specialization for hardware/accuracy trade-offs at deployment time.

Benchmarks & Reproducibility

NAS-Bench-101: Towards Reproducible NAS — arXiv

A tabular benchmark exposing the full search space and exact performance for many architectures to enable fair comparison of NAS algorithms.

Hardware-Aware & Resource-Constrained NAS

FBNet: Hardware-Aware NAS — arXiv

Jointly optimizes accuracy and latency using a differentiable search guided by measured hardware cost; targets mobile inference.

Frameworks & Tooling

AutoKeras — Project

An accessible AutoML library that includes NAS primitives and high-level APIs for tabular, image and text tasks.

NNI (Neural Network Intelligence) — Microsoft

Offers a broad suite of NAS algorithms, hyperparameter tuning, and built-in support for hardware-aware experiments and distributed search.

Quick Takeaways

Choose your search strategy by compute budget: RL/evolution often produce strong results but can be expensive; parameter sharing, surrogate models, and differentiable methods reduce cost.
Benchmarks matter: Use NAS-Bench datasets and standardized pipelines for reproducible comparisons and ablations.
Hardware must be in the loop: For deployment, include latency/power/size as explicit objectives or use hardware-aware layers like ProxylessNAS/FBNet/OFA flows.
One-shot & supernet approaches: Offer dramatic speed-ups (train once, derive many subnets) but require careful fairness and evaluation strategies to avoid bias.
Practical tip: start with constrained search spaces (cell-based, channel-level) and cheap proxies (smaller datasets, early stopping) before scaling up.