Přístupnostní navigace
E-application
Search Search Close
Master's Thesis
Author of thesis: Ing. Lucia Makaiová
Acad. year: 2025/2026
Supervisor: Ing. Martin Fajčík, Ph.D.
Reviewer: Ing. Michal Hradiš, Ph.D.
This work investigates vocabulary adaptation – the process of tailoring a model’s tokenizer and embeddings to a specific language – as a means of specializing a multilingual LLM for monolingual use targeting Czech. We benchmark existing methods, and analyze how quality and quantity of the alignments between source and target tokenizer vocabularies affect embedding initialization. We also evaluate the impact on tokenization efficiency, language modeling, and downstream task performance. Building on these findings, we propose a novel method combining Trans-Tokenization-based initialization with a low-rank vocabulary adapter. After training, the adapter is merged directly into the embedding matrix, introducing no computational overhead at inference time. Yet, our approach outperforms any individual method in terms of language modeling performance, under the same computational budget, as measured by word-level perplexity. On downstream tasks from BenCzechMark, the adapted model reaches performance competitive with or better than the original model on 70% of benchmarks. The new tokenizer reduces fertility by approximately 32% which leads to improved inference speed by around 25%. The resulting Czech-adapted Llama 3.2 3B model will be released publicly to support further research and applications in Czech NLP.
large language models, multilingual models, tokenization, vocabulary transfer, embedding initialization
Date of defence
23.06.2026
Result of the defence
Defended (thesis was successfully defended)
Grading
A
Process of defence
Studentka nejprve prezentovala výsledky, kterých dosáhla v rámci své práce. Komise se poté seznámila s hodnocením vedoucího a posudkem oponenta práce. Studentka následně odpověděla na otázky oponenta a na další otázky přítomných. Komise se na základě posudku oponenta, hodnocení vedoucího, přednesené prezentace a odpovědí studentky na položené otázky rozhodla práci hodnotit stupněm A.
Topics for thesis defence
Language of thesis
English
Faculty
Fakulta informačních technologií
Department
Department of Computer Graphics and Multimedia
Study programme
Information Technology and Artificial Intelligence (MITAI)
Specialization
Machine Learning (NMAL)
Composition of Committee
doc. Ing. Vítězslav Beran, Ph.D. (předseda) prof. Ing. Hynek Heřmanský, Dr. Eng. (místopředseda) doc. Ing. Ondřej Lengál, Ph.D. (člen) doc. Ing. František Zbořil, Ph.D. (člen) doc. Ing. Michal Bidlo, Ph.D. (člen) RNDr. Marek Rychlý, Ph.D. (člen)
Supervisor’s reportIng. Martin Fajčík, Ph.D.
The work has all the merits to become a novel contribution to the state of the art. The student worked actively on the thesis, studied recent literature, and actively discussed the achieved results. Therefore, I recommend an evaluation with grade A.
The assignment was of standard difficulty. Vocabulary swap follows our previous work on Czech Large Language Models, and studies an efficient way for "vocabulary swap" methodology that enables new generation of language-specialized models. I am fully satisfied with the results of the work.
Work was completed sufficiently ahead. The content of the work was consulted and reviewed.
The work still requires more evidence for achieving high-quality publication. However,
Student actively researched related literature.
The consultations were regular, and deadlines were met as agreed.
Grade proposed by supervisor: A
Reviewer’s reportIng. Michal Hradiš, Ph.D.
The student addressed an interesting and demanding topic. She became well acquainted with the state of the art in the field, proposed novel approaches, and evaluated them appropriately. Overall, the thesis extends the state of the art.
Evaluation level: assignment fulfilled
Evaluation level: is within the usual extent
The thesis presents rather complex ideas, and the reader can generally understand the information presented; however, in some respects, the text does not make this easy. Some ideas are distributed across several sections, and the terminology is sometimes not entirely precise or consistent, including the notation used in equations, tables, and graphs. At times, the text also omits some information.
In terms of presentation and typography, the thesis is of good quality, apart from unnecessary minor errors such as very occasional missing words, missing spaces, and missing punctuation after equations. I would also prefer figures and equations to be placed at the top of the page. Equations should be numbered.
The thesis refers to a larger number of high-quality and relevant sources, which provide good coverage of the current state of knowledge in the field. The sources are used well throughout the thesis, and overall, the thesis builds effectively on the current state of knowledge. Table 4.1 should contain citations.
The thesis proposes novel methods and examines their properties. Overall, the thesis brings new findings. The experiments are meaningful, and the results are well interpreted.
The thesis contributes to the state of the art in the field.
Evaluation level: more difficult assignment
The topic focuses on a more specialized aspect of large language models. It is also more challenging due to the computational demands of the experiments and the interpretation of the results.
Grade proposed by reviewer: A
Responsibility: Mgr. et Mgr. Hana Odstrčilová