Přístupnostní navigace
E-application
Search Search Close
Master's Thesis
Author of thesis: Ing. Petr Šilling
Acad. year: 2022/2023
Supervisor: doc. Ing. Michal Španěl, Ph.D.
Reviewer: doc. Ing. Vítězslav Beran, Ph.D.
Image stitching is an essential technique for reconstructing volumes of biological samples from overlapping tiles of electron microscopy (EM) images. Current volume EM stitching methods generally rely on handcrafted features, such as those produced by SIFT. However, recent developments indicate that convolutional neural networks (CNNs) can improve stitching accuracy by learning discriminative features directly from training images. Taking into account the potential of CNNs, this thesis proposes DEMIS, a novel EM image stitching tool based on LoFTR, an attention-based feature matching network. The thesis also proposes a novel dataset generated by splitting high-resolution EM images into grids of overlapping image tiles. The dataset is used to fine-tune LoFTR and to evaluate the DEMIS tool. Experiments on the synthetic dataset reveal higher feature matching accuracy compared to SIFT. Moreover, experiments on challenging images with small overlap regions and high resolution demonstrate significantly higher stitching robustness than SIFT. Overall, the results suggest that deep learning methods could be beneficial for EM imaging, for example, by allowing the use of smaller tile overlaps.
image stitching, volume electron microscopy, deep learning, convolutional neural networks, feature matching, SLAM optimisation, dataset of electron microscopy images, dataset synthesis, Python, DEMIS, LoFTR, SIFT
Date of defence
19.06.2023
Result of the defence
Defended (thesis was successfully defended)
Grading
A
Process of defence
Student nejprve prezentoval výsledky, kterých dosáhl v rámci své práce. Komise se poté seznámila s hodnocením vedoucího a posudkem oponenta práce. Student následně odpověděl na otázky oponenta a na další otázky přítomných. Komise se na základě posudku oponenta, hodnocení vedoucího, přednesené prezentace a odpovědí studenta na položené otázky rozhodla práci hodnotit stupněm A.
Topics for thesis defence
Language of thesis
English
Faculty
Fakulta informačních technologií
Department
Department of Computer Graphics and Multimedia
Study programme
Information Technology and Artificial Intelligence (MITAI)
Specialization
Machine Learning (NMAL)
Composition of Committee
prof. Dr. Ing. Jan Černocký (předseda) doc. Ing. Lukáš Burget, Ph.D. (člen) doc. RNDr. Milan Češka, Ph.D. (člen) Ing. Michal Hradiš, Ph.D. (člen) Ing. Jaroslav Rozman, Ph.D. (člen) Ing. František Grézl, Ph.D. (člen)
Supervisor’s reportdoc. Ing. Michal Španěl, Ph.D.
Jako vedoucí práce nemám co vytknout. Pan Šilling proniknul do problematiky, výborně zadání uchopil a komplexně zpracoval, odvedl velké množství svědomité a promyšlené práce.
Zadání vzniklo na základě partnerství FIT s firmou, pro kterou jde o velmi aktuální téma.
Zadání vyžadovalo nastudovat nejnovější metody hlubokého učení vhodné pro úlohu zarovnání obrazu, navrhnout kompletní postup sešití většího počtu obrázků a prozkoumat vlastnosti metod založených na hlubokém učení v porovnání s postupy využívajícími klasické detektory klíčových bodů jako SIFT.
Jako problematická se ukázala dostupnost vhodných trénovacích a validačních dat. Firma pro testování poskytla pouze dva ukázkové datasety z elektronového mikroskopu, což je při variabilitě dat žalostně málo. Student tak značné úsilí věnoval vytvoření vlastního datasetu.
Všechny požadované body zadání byly splněny.
Student strávil hodně času detailní analýzou existujících přístupů a základními experimenty pro ověření jejich vlastností. Vyhledal a prostudoval množství literatury, většinou ve formě publikovaných vědeckých článků.
Student byl aktivní a iniciativní. Pravidelné konzultace často sloužily jen pro ujištění, že další navrhovaný postup je vhodný. A nutno podotknout, že byl. Student sám postupoval při řešení velmi promyšleně a systematicky.
Práce byla dokončena včas, případné připomínky k technické zprávě byly průběžně zapracovávány.
Práce byla prezentována na studentské konferenci Excel@FIT.
Grade proposed by supervisor: A
Reviewer’s reportdoc. Ing. Vítězslav Beran, Ph.D.
Based on a very careful study of relevant scientific papers, the author has selected suitable existing modern methods for image analysis and adapted them for stitching of images taken by electron microscope. From the existing datasets, he created a new dataset that reflects the needs of the image stitching task in the electron microscopy domain. He has developed a high-quality software solution integrating existing libraries and his own methods, including a number of necessary tools for dataset generation and experiment execution. The technical report, written in English, has an excellent presentation and technical level. The resulting work contains new insights.
Evaluation level: obtížnější zadání
Although the assignment does not explicitly require a solution in the electron microscopy domain, the work focuses on this domain. The development of a method for image fusion in a less common domain is novel and may present unexpected problems.
Evaluation level: zadání splněno
Evaluation level: je v obvyklém rozmezí
The technical report has a clear and logical structure and the text follows each other well. The author expresses himself factually, clearly and professionally, the individual parts of the text are well-justified and relevant to the solution of the assignment. The chapter ranges or structure could be slightly revised to make the chapter scope a little more balanced.
There are a few minor flaws in the text. In the introduction to the solution proposal (Ch. 6), the term OpenCV is used somewhat inappropriately to refer to a method that computes a transformation between two images. Although the author refers to the literature on the use of metrics to evaluate the quality of stitched images (Ch. 8.2), at least a brief description of these metrics and the essence of these metrics (what they measure and what conclusions can be drawn from these values) is missing. The author mentions the size of the CNN architecture of the LoFTR module for the first time only at the end, which has not been discussed anywhere before. Also, the conclusion about the need for a larger dataset is not well justified and does not make much sense. It can be assumed that more data may slightly improve the quality of the LoFTR network, but it is not clear how it will help to improve the final stitched images, since the author does not know the reason for the worse final results even with this better new architecture.
The description of the implemented modules describes their functionality and integration, but not more detailed technical aspects such as the data structures and formats used, the integration of external tools and how to implement their specific use, etc.
Despite the minor shortcomings, the overall presentation level of the technical report is very good.
The formal and linguistic level of the technical report is excellent. The text is written in good professional English, is perfectly understandable and free of errors. All formatting is precise and the figures are in relevant formats and are of high quality.
The selection of study sources is of high quality, extremely extensive and relevant. The summaries of the different approaches show a careful study of the various articles, and the author summarizes the key aspects of the different approaches in a concise, factual and erudite manner.
The author has selected appropriate actual methods for solving the sub-steps on the basis of a good study. The selection of these methods is well described, as well as the overall structure of the SW solution. A new dataset is of high quality and relevance.
However, the key parts of the solution are not described in sufficient detail either in the design or in the implementation. The specific parameters and the procedure for retraining the CNN architecture of LoFTR on EM data are not known. For the global optimization, where it uses the SLAM method, again neither the theoretical part nor the implementation explains the optimization principle - in short, it somehow optimizes the local transformations to the global objective.
The experiments in Ch. 8.3 are presented only by images (Figs. 8.1 and 8.2), where the behaviour of the different methods can be clearly seen, but of course, quantitative results cannot be read from them. Further, it is not clear what the properties of the used datasets are. The experiment on the robustness of different variants of the methods is very relevant and crucial, but the results presented in this way are not very useful.
Technically, the software is of very good quality, the source files contain authorship information, are logically structured and well annotated. The solution contains all the necessary tools for working with datasets, retraining the LoFTR model, running experiments, etc.
The work brings new insights by applying relevant and modern methods in a new domain. It appropriately selects modern techniques from the field of image stitching and adapts them to the electron microscopy domain. Thus, the partial results of the work can be evaluated as new. The use of the results is still questionable, as experiments have shown improvements in sub-steps, but have not yet been able to improve the overall result over classical methods.
Grade proposed by reviewer: A
Responsibility: Mgr. et Mgr. Hana Odstrčilová