Bachelor's Thesis

Search Algorithms in Automated Design of Convolutional Neural Networks

Author of thesis: Neonila Mashlai

Acad. year: 2025/2026

Supervisor: prof. Ing. Lukáš Sekanina, Ph.D.

Abstract:

Manual convolutional neural network design is a time-consuming process that requires deep domain expertise, making automated architecture search a practical and valuable alternative. This thesis implements and evaluates four search algorithms for automated architecture design on the NAS-Bench-101 benchmark, which contains 423,624 unique architectures evaluated on the CIFAR-10 dataset: random search, hill climbing with memory, a genetic algorithm, and a non-dominated sorting genetic algorithm. All algorithms are implemented in Python and query the NAS-Bench-101 API to obtain precomputed validation accuracy and parameter counts, eliminating the need for network training. The genetic algorithm converges the fastest and produces CNNs with the highest accuracy (94.5% mean validation accuracy), while random search reaches 94.3% under the same evaluation budget and hill climbing reaches approximately 94.0% with notably higher run-to-run variance, where the patience hyperparameter proves critical. NSGA-II produces a Pareto front covering the trade-off between accuracy and parameter count, spanning architectures from 10^4 to 10^7 trainable parameters with error rates ranging from approximately 5% to 15%. All results are statistically supported by multiple independent runs.

Keywords:

neural architecture search, convolutional neural networks, genetic algorithm, NSGA-II, NAS-Bench-101, Pareto front, hill climbing, multi-objective optimization

Date of defence

15.06.2026

Result of the defence

Defended (thesis was successfully defended)

znamkaAznamka

Grading

Process of defence

Studentka nejprve prezentovala výsledky, kterých dosáhla v rámci své práce. Komise se poté seznámila s hodnocením vedoucího a posudkem oponenta práce. Studentka následně odpověděla na otázky oponenta a na další otázky přítomných. Komise se na základě posudku oponenta, hodnocení vedoucího, přednesené prezentace a odpovědí studentky na položené otázky rozhodla práci hodnotit stupněm A.

Topics for thesis defence

You note that the NAS-Bench-101 accuracy distribution is skewed and that the gap between a poor and an excellent architecture is under 2 percentage points (Section 4.2.4). This property strongly favors random search as a baseline. How do you expect the relative ranking of your four algorithms to change on a benchmark with a more uniform or wider accuracy distribution, such as NAS-Bench-201?
Your comparison in Section 6.2.1 uses an equal evaluation budget across all algorithms. However, the genetic algorithm and NSGA-II carry additional overhead that random search and hill climbing do not. Did you also measure and compare the wall-clock runtimes, and would the convergence ranking still hold if the algorithms were compared on equal runtime rather than equal number of evaluations?
Proč jste použila algortimus Hill-Climbing?

Language of thesis

English

Faculty

Fakulta informačních technologií

Department

Department of Computer Systems

Study programme

Information Technology (BIT)

Composition of Committee

doc. Ing. František Zbořil, Ph.D. (předseda)
doc. Mgr. Kamil Malinka, Ph.D. (místopředseda)
Ing. Jiří Matoušek, Ph.D. (člen)
Ing. Vladimír Veselý, Ph.D. (člen)
doc. Ing. Vítězslav Beran, Ph.D. (člen)

Supervisor’s report
prof. Ing. Lukáš Sekanina, Ph.D.

Studentka splnila zadání -- implementovala a porovnala vybrané stochastické optimalizační algoritmy v úloze návrhu architektury CNN. I když práce nepřináší nové poznatky, domnívám se, že byla pro studentku přínosná, protože ji naučila používat evoluční algoritmy, neuronové sítě, automatizovaně ladit architektury CNN a porovnávat výsledky optimalizačních algoritmů. Výsledky dosažené jednotlivými algoritmy jsou dost podobné, což plyne ze způsobu konstrukce prostoru CNN a zadání problému.

Evaluation criteria	Verbal classification
Informace k zadání	Jedná se o průměrně obtížné zadání, jehož cílem bylo implementovat a porovnat stochastické optimalizační algoritmy při návrhu konvolučních neuronových sítí (CNN) pro klasifikaci obrazu. Všechny algoritmy se pohybují v předem stanoveném prostoru architektur CNN. Pro ohodnocení architektury CNN byla využita databáze natrénovaných CNN, a to konkrétně NASBench101. Vyhodnocení proběhlo pomocí krabicových grafů sestavených z nezávislých běhů. Nad rámec zadání byla implementována vícekriteriální varianta evolučního algoritmu. Zadání bylo splněno.
Práce s literaturou	Studentka samostatně vyhledávala odbornou literaturu a využívala ji.
Aktivita během řešení, konzultace, komunikace	Studentka byla během řešení projektu aktivní. Konzultace probíhaly dle potřeby studentky. Na konzultace byla studentka dobře připravena a samostatně navrhovala pokračování řešení projektu.
Aktivita při dokončování	Práce byla dokončena v předstihu. Předfinální text práce byl konzultován, mé připomínky byly vesměs zapracovány.
Publikační činnost, ocenění	Není mi známo.

Points proposed by supervisor: 90

Grade proposed by supervisor: A

Reviewer’s report
Ing. Jan Klhůfek

Given the thorough and methodologically sound treatment of the topic, the extension of the assignment with multi-objective search, the careful statistical evaluation, and the proper discussion of results, I propose an overall classification of A (excellent).

Evaluation criteria	Verbal classification	Points
Náročnost zadání	Evaluation level: průměrně obtížné zadání Standard difficulty for a bachelor's thesis in this area. The assignment requires familiarity with CNNs, evolutionary algorithms, and expects working with an established benchmark (NAS-Bench-101) rather than training networks from scratch. The use of a precomputed benchmark reduces the engineering burden, but the multi-objective extension (NSGA-II) and the breadth of the experimental evaluation add appropriate complexity.
Prezentační úroveň technické zprávy	The structure is logical and the chapters are well-balanced. The reader is guided from CNN fundamentals through search algorithms and benchmarks to implementation and results. Chapters are clearly defined and build on one another. Minor remarks: the introduction to NAS (Section 2.6) sits inside the CNN chapter, where it does not quite belong topically – it would fit more naturally at the start of Chapter 3 discussing the search algorithms. Section 4.2.2 ("Search Space") slightly drifts into dataset comparison rather than describing the search space itself. However, these are just minor organizational details and do not impair readability.	90
Formální úprava technické zprávy	The writing is clear and the typography is consistent. A few minor issues: a stray "s" at the end of "(1 − validation accuracy)s" in Section 6.3.2, and inconsistent number formatting ("2" vs "three") in Section 2.4.1. The opening sentence of Section 6.3.2 states "Figure 6.8 compares the Pareto fronts obtained from 50 independent NSGA-II runs", but Figure 6.8 shows only 5 fronts – this should be clarified as "5 example fronts from 50 independent runs". Within Chapter 5, the "patience" parameter name is reused for two unrelated mechanisms (Loader retry limit and hill-climbing restart) with the same default value, which may confuse readers.	90
Realizační výstup	The implementation is clean, modular, and well-documented. Validation is solid. Each configuration is run 30 times for statistical reliability, and the visualization pipeline produces the figures used in the report directly from the recorded JSON outputs.	95
Využitelnost výsledků	The work is largely of a compilation and verification character. It implements and statistically evaluates well-established search algorithms on an established benchmark (NAS-Bench-101). The contribution lies in the systematic, statistically grounded comparison of four algorithms under matched evaluation budgets, the hyperparameter sensitivity analysis, and the multi-run Pareto front accumulation strategy for NSGA-II. The results confirm known findings (random search as a strong baseline on NAS-Bench-101, GA convergence advantages, the criticality of the patience parameter for hill climbing) and provide a clean reference implementation that can serve as a starting point for further work – for example, extension to NAS-Bench-201, addition of further objectives such as training time, or comparison against newer NAS methods.
Rozsah splnění požadavků zadání	Evaluation level: zadání splněno a práce obsahuje podstatná rozšíření The student became familiar with CNNs and NAS methods, and designed and implemented four search algorithms (random search, hill climbing with memory, a single-objective genetic algorithm, and NSGA-II), exceeding the required minimum of two. The inclusion of NSGA-II and the corresponding multi-objective treatment of the search problem goes beyond the assignment's scope, which mentions only search algorithms without requiring multi-objective methods. The experimental evaluation in Chapter 6 is thorough, and the results are properly evaluated and discussed.
Rozsah technické zprávy	Evaluation level: je v obvyklém rozmezí The technical report fits comfortably within the expected range for a bachelor's thesis. The chapters are information-dense and all parts contribute to the work's scope.
Práce s literaturou	The report contains 28 cited works. The literature is appropriately selected, covering both foundational sources and the relevant literature on NAS. Citations are correctly placed and the student clearly distinguishes adopted figures and code from their own contributions.	100

Topics for thesis defence:

You note that the NAS-Bench-101 accuracy distribution is skewed and that the gap between a poor and an excellent architecture is under 2 percentage points (Section 4.2.4). This property strongly favors random search as a baseline. How do you expect the relative ranking of your four algorithms to change on a benchmark with a more uniform or wider accuracy distribution, such as NAS-Bench-201?
Your comparison in Section 6.2.1 uses an equal evaluation budget across all algorithms. However, the genetic algorithm and NSGA-II carry additional overhead that random search and hill climbing do not. Did you also measure and compare the wall-clock runtimes, and would the convergence ranking still hold if the algorithms were compared on equal runtime rather than equal number of evaluations?

Points proposed by reviewer: 90

Grade proposed by reviewer: A

Responsibility: Mgr. et Mgr. Hana Odstrčilová

VUT

Faculties and university institutes

Parts

Search Algorithms in Automated Design of Convolutional Neural Networks