\chapter{Light Field}
\label{chap:lf}
This chapter contains general and in-depth explanations on the topic of light field.
The basic principles of light field rendering and related topics, such as acquisition of the data and their compression, are explained in this chapter.
The description of light field state-of-the-art methods serves as a theoretical background for the novel proposals in the later parts of this thesis.

\section{Overview of Light Field}
\label{sec:lfOverview}
As the name \textit{light field} suggests, a field of light can be defined for a given scene.
Such a field would contain information about the incoming light at any point in the scene; see Fig.~\ref{fig:rayTrace}.
This universal definition does not involve working with materials, surfaces, or geometry.
However, it is difficult and impractical to work with such general definitions.
In computer graphics, the light field is defined only on a subset of space.
Also, instead of an ideal theoretical continuous definition, a discrete approximation is used.
The scene is described as static, and the light is defined only for specific wavelengths corresponding to the human visual system.
In practice, a light field can be used as an alternate representation of a 3D scene, different from the classic structural boundary or volumetric representation; see Fig.~\ref{fig:lfVSGeom}.

\begin{figure}[!htbp]\centering
  \includegraphics[width=0.6\linewidth]{obrazky-figures/rayTrace.pdf}
  \caption{\label{fig:rayTrace}
Ray coming from a source changes its color according to the materials contacted. 
}
\end{figure}

\begin{figure}[!htbp]\centering
  \subfloat[\centering 3D editor Blender shows a geometry visualization, material editor, rendered view, etc.]{{\includegraphics[width=0.95\textwidth]{obrazky-figures/blenderRepre.jpg} }} \\
  \subfloat[\centering One \textcolor{red}{sampling point} with all possible \textcolor{mainOrange}{rays} are indicated, and four \textcolor{mainBlue}{example rays} with incoming light colors in the circles are highlighted in the theoretical light field representation.]{{\includegraphics[width=0.95\textwidth]{obrazky-figures/lfRepre.pdf} }} \\
  \subfloat[\centering Practical light field approximation consists of a set of images.]{{\includegraphics[width=0.95\textwidth]{obrazky-figures/lfDiscRepre.jpg} }} 
  \caption{\label{fig:lfVSGeom}
  3D scene can be represented by a definition of its geometry and materials.
  A window in the Blender software depicts such a representation.
  A field of light can be defined for any point, any ray, from any direction.
  In practice, such a representation is approximated by a set of images that capture the scene from different positions.
}
\end{figure}

Light field contains visual information for a defined position and a direction in space.
Structural representations describe the scene using information about its surface or volume attributes, such as definition of the shape of objects, textures, lights, or material settings~\cite{gregory2018game}. 
Light field is usually represented by a set of images that contain differently positioned views of the same scene.
Fig.~\ref{fig:mainTeaser} demonstrates the standard light field rendering pipeline.
The scene is first captured in multiple views from defined camera positions.
The images are aligned and passed to the light field rendering method.
The rendering method selects the necessary images and samples their pixels that are used in the synthesis of the novel view.

\begin{figure}[!htbp]\centering
  \includegraphics[width=\linewidth]{obrazky-figures/mainTeaser.pdf}
  \caption{\label{fig:mainTeaser}
A grid of cameras captures a synthetic or real scene.
A set of images, containing various views of the scene, is stored or streamed directly to the rendering method.
Novel views are generated using light field rendering methods.
These views can be used in 2D and 3D display devices to create a 3D experience in real time.
}
\end{figure}

\subsection{Practical Highlights of Light Field}
Light field representation of the scene is suitable for rendering purposes.
The advantages of light field rendering methods are mainly the constant, content-independent rendering time, and straightforward capture of a synthetic or real-life scene.
The drawback of the light field is that it does not implicitly contain all necessary information about the scene geometry for advanced interaction with the scene, such as physics simulation or collision detection.

A quick motivational experiment was conducted to show how light field rendering compares to path-tracing and rasterization; see Tab.~\ref{tab:advantages}.
NVIDIA OptiX render and denoising with adaptive sampling in Cycles engine and rasterization (Eevee engine) in a 3D modeling and rendering software Blender 3.6 was compared with a light field rendering method~\cite{chlubna2024accel}.
The measurements were performed on NVIDIA\textsuperscript{\textregistered} GeForce RTX\textsuperscript{\texttrademark} 2070.

\begin{table}[ht]\centering
\resizebox{\textwidth}{!}{
\begin{tabular}{lrrl|rrrl|lll}
\footnotesize
\input{tables/advantages.txt}
\end{tabular}
}
\caption{\label{tab:advantages}
    The table contains a comparison of path tracing, rasterization, and light field rendering.
    Light field requires constant memory footprint comparable with other techniques but offers up to 60$\times$ faster visualization to path tracing with one sample and 20$\times$ faster to rasterization.
    Four different scenes from a 4K light field dataset~\cite{chlubna2024accel} were compared.
    \textbf{PT} refers to path tracing, \textbf{PT 1s} to path-tracing with one sample and denoising, \textbf{Raster} to rasterization and \textbf{LF} to light field rendering.
    \textbf{PT} was used as a reference for the PSNR measurement.}
\end{table}

The results depend on the type of the scene.
Note that path tracing can be used to generate scenes for light field rendering.
From a theoretical point of view, path-tracing can be viewed as an algorithm that populates a discrete subset of the light field by probing the ray directions defined by the camera parameters.
The experiment is focused on the rendering of the novel view, based on existing light field dataset.
The production of the dataset itself is not relevant for this motivational measurement.
The main focus of this experiment is to evaluate the options for real-time rendering.
This experiment demonstrates how light field rendering might be a better choice than a standard rendering implemented in the popular 3D rendering software and potentially than many other implementations.

The results show that light field rendering does not require as much memory as path tracing.
Rasterization can require less memory when only simple geometry is present without a lot of textures.
Light field rendering offers a significant rendering speedup compared to the other approaches.
Although rasterization is commonly used in real-time graphics, a lot of effects need to be simulated and are not possible to physically compute.
The speed of light field rendering depends solely on the number of necessary texture read operations, which would be five per pixel for rendering with precomputed focus distances.
Note that the rasterization implementation in Blender might not be optimal and the times may be faster, but the visual quality of the result would still be lacking due to its inability to fully simulate path-tracing results.
This is also the reason why the GPU industry has focused on GPU-accelerated ray tracing in recent years~\cite{sanzharov2020survey}.
Path tracing with only one sample and denoising is still worse in both speed and visual quality compared to light field rendering.
Generally, the usability of light fields increases with higher complexity of the scene and the amount of materials and assets used. 
This is reflected in the results, where the difference in performance between light field rendering and path tracing is lower with more simple scenes.
Therefore, light field rendering has a significant potential to be used in 3D simulations.

Fig.~\ref{fig:lfvspt} shows how the results differ visually. 
The chosen scene contains only diffuse and slightly specular materials so that it can be visually compared.
Note that other more complex materials would show more significant changes and the visual comparison focused on the common issues of each method would be more difficult to visualize.

\begin{figure}[!htbp]\centering
  \includegraphics[width=0.99\textwidth]{obrazky-figures/lfVSpt.pdf}
  \caption{\label{fig:lfvspt}
A simple scene is rendered using different methods and visually compared in the figure.
Problematic parts are highlighted.  
One-sample path tracing shows blurry results with vanishing details as visible in the \textcolor{mainRed}{red} zoom.
Rasterization has problems with precise computation of the shadows and lighting effect as visible in the \textcolor{mainBlue}{blue} zoom.
Light field rendering can produce focusing artifacts as visible in the \textcolor{mainOrange}{orange} zoom.
  }
\end{figure}

Fast light field rendering, which is the main topic of this thesis, can be crucial for use cases, such as streaming of light field movies or integration in 3D scenes; see Fig.~\ref{fig:lfUseCase}.

\begin{figure}[!htbp]\centering
  \includegraphics[width=0.95\textwidth]{obrazky-figures/lfUseCase.pdf}
  \caption{\label{fig:lfUseCase}
Real-time light field rendering, is important for streaming of light field films for various 2D or 3D displays.
Any view of the scene has to be accessible by a quick synthesis so the user can change the viewing position.
In 3D simulations, light field could replace geometry that is not fully accessible by the camera and would be time-consuming to render.
  }
\end{figure}

An example of light field video usage is in cinematography for post-processing purposes.
The video, captured as a light field, would allow for a change of focus distance, camera position, and would natively support a stereo or multi-view format export~\cite{trottnow2019potential}; see Fig.~\ref{fig:standardVSLF}.

\begin{figure}[!htb]\centering
    \subfloat[\centering Standard photography has a static camera and one focus distance.]{{\includegraphics[width=0.49\textwidth]{obrazky-figures/staticScene.pdf} }} 
    \subfloat[\centering Light field with post-process options can change the camera position or focus distance.]{{\includegraphics[width=0.49\textwidth]{obrazky-figures/dynamicScene.pdf} }} 
    \\
    \subfloat[\centering The images are examples of the refocused and differently positioned views.]{{\includegraphics[width=0.99\textwidth]{obrazky-figures/dynamicScenePreviews.pdf} }} 
    \\
  \caption{\label{fig:standardVSLF} 
The figure illustrates the difference between the standard photography and light field.
Light fields can be refocused and the viewing camera can be moved in the post-processing.
The red plane at the geometry depicts the focus distance.}
\end{figure}

In 3D simulations, like computer games, light field assets would be an optimal way to render computationally demanding scenes~\cite{horn2007lightshop}, where the player's movement is limited to avoid reaching uncaptured parts of the scene~\cite{todt2008light}.
Light fields assets for 3D environment would be optimal for scenes such as views through windows, inside rooms, into boxes with various objects, at distant scenery, etc.

\subsection{Light Field in Already Existing Software}
\label{sec:lfProd} 
Google introduced a VR light field showcase application called Welcome to Light Fields~\cite{overbeck2018making}.
It serves as a virtual tour that shows a real-life captured scene with a rotating camera array.
The entire scene is scanned from a particular position.
To minimize GPU memory requirements, tiled streaming and caching is used along with modified VP9 compression. 
A disk-based reconstruction method is used to render the final image.
The whole scene is represented by a collection of disk-shaped windows with tessellated geometry based on depth values estimated by multi-view stereo algorithms. 

Leia LumePad, a 3D tablet, comes with a LeiaPix application.
It is a light field editor that allows unified editing of stereo photos, which are further interpolated into a higher density light field; see Fig.~\ref{fig:leiaEditor}.

\begin{figure}[!htbp]\centering
  \includegraphics[width=0.99\textwidth]{obrazky-figures/leiaEditor.jpg}
  \caption{\label{fig:leiaEditor}
    The figure contains two screenshots from the LeiaPix light field editing software on 3D tablet LumePad.
    The images are displayed in 3D and basic edits can be conducted.
  }
\end{figure}

An open-source light field imaging application PlenoptiCam~\cite{Hahne16} exists and allows the user to display photos from Lytro or similar plenoptic cameras.
It allows for various operations with plenoptic images such as depth map generation, refocusing or interactive camera motion; see Fig.~\ref{fig:plenopticam}.
A similar application Light Field Video Viewer~\cite{wang2017light} also supports viewing light field videos. 
The author proposed using plenoptic camera along with classic 2D camera to capture the video at the same time and then use the angular information from plenoptic camera to enhance the 2D video adding all the attributes of light field video. 
In this way, the bandwidth is reduced since all the data from the plenoptic camera are not being transferred and read during the final render.
LFDisplay software~\cite{levoy2006micro} can be used to view plenoptic images from light field microscopes; see Fig.~\ref{fig:lfdisplay}.
It supports tilting and refocusing of the captured sample.
Light Field Toolbox~\cite{dansereau2015linear} is a set of algorithms used to analyze light fields in Matlab software.
It mainly supports Lytro plenoptic images but can load other formats as well.
The toolbox can be used to calibrate, rectify, refocus and visualize light fields.
Light Field Suite~\cite{wanner2013} is a set of tools that can be used to estimate disparity from light field views.
Similar features for plenoptic camera images are contained in Plenoptic Toolbox~\cite{palmieri2018} that supports rendering of the 3D views from such photos.
However, none of these products would be able to replace the proposal of this thesis because of their inability to process streamed, high-resolution, widely spaced light field data in real time.

\begin{figure}[!htbp]\centering
  \includegraphics[width=0.92\textwidth]{obrazky-figures/plenopticam.jpg}
  \caption{\label{fig:plenopticam}
The figure shows import (top left) and result (bottom) windows of PlenoptiCam.
The result can be viewed from different viewing angles.
Point cloud and mesh can be exported (top right).
INRIA plenoptic camera dataset was used~\cite{shi2019framework}.
  }
\end{figure}

\begin{figure}[!htb]\centering
    \subfloat[\centering The object can be viewed in 3D.]{{\includegraphics[width=0.49\textwidth]{obrazky-figures/lfdisplay1.jpg} }} 
    \subfloat[\centering The lenslet light field image can be displayed.]{{\includegraphics[width=0.49\textwidth]{obrazky-figures/lfdisplay2.jpg} }} 
    \\
  \caption{\label{fig:lfdisplay}
The figure shows two viewing modes of microscopic light field images in LFDisplay software.
  }
\end{figure}
\clearpage

\section{Physical Background}
\label{sec:lfBack}
In physics, light is defined as electromagnetic radiation of a certain wavelength~\cite{ditchburn2013light}. 
Wavelengths $\lambda \in (380,750)$ in nanometers are regarded as visible light with respect to the human visual system~\cite{buser1992vision, starr2014biology}. 
Light transmission is described by geometrical (ray) or physical (wave) optics~\cite{guenther2015modern}.
Light is propagated in space by elementary particles called photons.
Light can also be viewed as a field defined by a vector function~\cite{faraday1846liv, gershun1939light}.
Specific wave or particle properties of light are not considered in this model.
The model describes a field of rays in any possible directions.
To model such a field, a 7D plenoptic function~\cite{bergen1991plenoptic} $\textrm{L}$ can be defined as in Eq.~\ref{eq:pleno}.
\begin{equation}
I=\textrm{L}(p_x,p_y,p_z,\theta, \varphi, m, \lambda).
\label{eq:pleno}
\end{equation}
This function returns the light intensity $I$ of an incoming ray to a point in 3D space with coordinates $p_x,p_y,p_z$ from a given direction represented by two spherical angles $\theta, \varphi$; see Fig.~\ref{fig:plenoptic}.
Assuming a non-static scene, the light can vary for each moment $m$ in time and for different wavelengths $\lambda$. 

\begin{figure}[!htbp]\centering
  \includegraphics[width=0.6\linewidth]{obrazky-figures/plenoptic.pdf}
  \caption{\label{fig:plenoptic}
The camera position and direction can be parametrized in 3D space as arguments of a plenoptic function. 
The function can be discretized and sampled by grid.
  }
\end{figure}

In physics, the intensity of the light along a ray would be called radiance.
The plenoptic function is defined for a point and differs from the radiance, which is defined per unit area.
It resembles the definition of radiant intensity.
Light fields are also sometimes called radiance fields.
The incoming light to the given point can originate from many different sources.
The light sources illuminate the scene and the light is absorbed, reflected, or refracted.
The intensity of the light of a given wavelength changes due to its interaction with the environment. 
The resulting intensity can be a result of a combination of several light sources.

The field contains the continuous information about the light so a full reconstruction is always possible when each query point is contained in the light field.
Light field defined as a vector field of rays is a theoretical concept.
The storage of continuous signal information is challenging in computer science, and several approximation techniques have to be used in practice.

\section{Discrete Approximation}
\label{sec:lfDiscrete}
Light information in computer science is often discretely encoded by a matrix of values of a given color model representing an image~\cite{ibraheem2012understanding}.
A commonly used RGB additive color model defines light as a vector of three channels: red, green, and blue, which corresponds to the biological processing of light in the human eye~\cite{loesdau2014hue}.
This vector defined for a specific coordinate in an image is called the pixel.
Classic cameras usually quantize the incoming light to the sensor to one 8-bit value per channel.
High Dynamic Range (HDR) devices, which aim to capture real light more accurately, use higher precision incoming light information such as 10-bit, 16-bit, or 32-bit value per channel~\cite{punchihewa2011from}.
The light field can be approximated by a set of images, where each image represents a view on the scene taken with a different camera position.
The views are also sometimes called sub-aperture images.

For static scenes and the definition of the return value $I$ as a light color, for example, RGB vector, the 7D plenoptic function can be approximated by a 5D function. The wavelength $\lambda$ and time moment $m$ arguments are omitted.
In order to efficiently represent a scene by light field, the viewing location is limited to the outside of a convex hull of the scene.
Light field would ideally capture a free space without occluders.
When viewing only the light field scene without additional 3D environment, the viewing and capturing cameras are placed exactly on the scene convex hull boundary, which can be a simple bounding box.
Fig.~\ref{fig:lfHull} shows how a scene can be enclosed and sampled by virtual cameras as light field.
The surface of the convex hull of the scene contains the visual information about the enclosed scene for each direction.

\begin{figure}[!htbp]\centering
  \includegraphics[width=0.9\linewidth]{obrazky-figures/lfHull.pdf}
  \caption{\label{fig:lfHull}
The scene is enclosed in a bounding box.
The surface of the box contains visual information about the enclosed scene in all directions.
The cameras around the box can sample the surface from all positions around it.
  }
\end{figure}

When casting a ray from the viewing camera, the light information has to stay constant along the whole ray.
In this way, the function can be further simplified to 4D, commonly referred to as the light slab~\cite{levoy1996light} or the lumigraph~\cite{gortler1996lumigraph}.
The reduced 4D function $\textrm{L'}$ in Eq.~\ref{eq:lf4d} can be geometrically represented as a ray intersection of two planes or other predefined surfaces~\cite{lin2004a}, for example, sphere-plane~\cite{camahort1998uniformly}, direction-point~\cite{camahort1999geometric}, or positional-directional sphere~\cite{ihm1997rendering}.
\begin{equation}
I=\textrm{L}'(s,t,u,v)
\label{eq:lf4d}
\end{equation}
The ray would intersect the closer plane at a point with local coordinates of $(s,t)$ and the further plane at $(u,v)$; see Fig.~\ref{fig:twoPlanes}.
The $(s,t)$ coordinates, also called angular, on the camera plane are used to identify which of the input views are relevant for the interpolation of the novel view.
The views are mapped on the plane since their capturing cameras were distributed over a planar grid.
The $(u,v)$ coordinates, also called spatial, on the focal plane are used to sample the correct pixels from the selected views.

\begin{figure}[!htbp]\centering
  \includegraphics[width=\linewidth]{obrazky-figures/twoPlanes.pdf}
  \caption{\label{fig:twoPlanes}
The scene is placed between two invisible planes.
The rays emitted from the camera intersect the planes to find out which views and which pixels will be used in the novel view synthesis.
  }
\end{figure}

The described representations are still continuous in theory.
Their dimensions are lower than those of the all-defined light field.
All possible orientations of the rays in the described representations cannot be fully defined without errors due to limited memory.
That is the reason why only some rays are defined, and the rest needs to be interpolated.
In practice, the definition of the rays is implemented by the discrete images.
Fig.~\ref{fig:lfInterpolation} describes the issue of missing information caused by the light field discretization.

\begin{figure}[!htbp]\centering
  \includegraphics[width=\linewidth]{obrazky-figures/lfInterpolation.pdf}
  \caption{\label{fig:lfInterpolation}
Two cameras were used to capture light field in a form of two images.
They cover some rays that would sample the light field scene enclosed in a convex hull.
The rays that are not directly captured by the cameras need to be interpolated.
The pixels from the images can be, for example, simply blended together.
The images were shifted so that the highlighted pixels overlap each other.
  }
\end{figure}

\section{Focusing Methods}
\label{sec:lfFocus}
The light field views can be placed on top of each other and blended together.
In such a case, only certain areas of the scene would be clearly visible because the areas that are further or closer to the camera would not be properly aligned in the views.
The input images are expected to be without blurred areas, captured with infinite depth of field.
The focusing for one pixel in the resulting novel view is defined as a change of the coordinates of the sampled pixels that are used for the final color computation.
In this thesis, the pixel in the novel synthetic view is called \textit{focused} if it is interpolated from the pixels, sampled from the input views, that belong to the same 3D spot in the scene.

The resulting view visually resembles a depth-of-field effect where certain parts of the scene are sharp and the rest are blurry.
The view can be focused at one distance, or depth information can be used to correctly focus each part of the image.
The position of the focal plane determines the focus distance in the scene, that is, which part of the scene is visually sharp; see Fig.~\ref{fig:focusScheme}.

\begin{figure}[!htbp]\centering
  \includegraphics[width=\imageSize]{obrazky-figures/focusScheme.pdf}
  \caption{
The scheme shows side view of a scene with a depth of $z$. 
Two rays from the virtual camera $c$ intersect the geometry at points $a$ and $b$.
The capturing cameras $c_i$ are evenly distributed along the $st$ plane.
Sampling rays are emitted from the closest cameras ($c_5$, $c_6$ and $c_7$) to the direction of $a$ and $b$ converging at the focus distance $f$ on the camera $c$ ray vector.
$a_{st}$ and $b_{st}$ mark the ideal intersection of the virtual camera with the $st$ plane.
The intersection points $a'_{uv}$, $a''_{uv}$ and $b'_{uv}$, $b''_{uv}$ determine which pixels, from the projected image on the $uv$ plane, are taken into the interpolation.
Points $a'_{uv}$, $a''_{uv}$ demonstrate a situation where the rays intersect the geometry in a correct place.
Points $b'_{uv}$, $b''_{uv}$ intersect the $uv$ plane, ignoring the geometry of the scene that leads to a defocused image.}
  \label{fig:focusScheme}
\end{figure}

The change of the focus in the novel view is called \textit{refocusing}.
Refocusing is simulated by the interpolation algorithm, which blends the input images together.
The nature of the light field refocusing is different from the optical refocusing used in cameras, although it looks similar.
In standard photography, light rays that pass through the lens hit the sensor.
The adjustment of the lens and the size of the aperture affect how the rays propagate further to the camera body.
If the rays converging at the given point on the sensor do not come from the same spot in the scene, such area is blurry.
In light field, such an effect is also caused by the color information not coming from the same spot in the scene.
The incoming rays on the sensor are simulated by sampling pixels at different coordinates in the input images.
The areas that are out of focus can contain sharp ghosting artifacts due to a low angular resolution of the light field data~\cite{chai2000plenoptic}; see Fig.~\ref{fig:refocus}.
The defocused areas are caused by the disparity of views. 
A smaller distance between the capturing cameras can reduce the ghosting, but it can also reduce the resulting 3D effect and viewing range.
Denser angular sampling can reduce the artifacts, leading to smooth defocus, but increases the space requirements for the light field data due to increasing redundancy between the views.

\begin{figure}[!htbp]\centering
   \def \rowLabelSize{0.02\linewidth}
   \def \columnSize{0.45\linewidth}
    \captionsetup[subfigure]{labelformat=empty}
    \subfloat[\centering ]{{\includegraphics[width=\imageSize]{obrazky-figures/refocus.pdf} }} 
    \makebox[\rowLabelSize]{ }
    \makebox[\columnSize]{\centering $8{\times}8$ grid}
    \makebox[\columnSize]{\centering $16{\times}16$ grid}
    \\
    \makebox[\rowLabelSize]{\raisebox{55pt}{\rotatebox[origin=c]{90}{\centering focused at $z_1$}}}
    \subfloat[\centering ]{{\includegraphics[width=\columnSize]{obrazky-figures/refocus8Z1.jpg} }} 
    \subfloat[\centering ]{{\includegraphics[width=\columnSize]{obrazky-figures/refocus16Z1.jpg} }}
    \\
    \makebox[\rowLabelSize]{\raisebox{55pt}{\rotatebox[origin=c]{90}{\centering focused at $z_2$}}}
    \subfloat[\centering ]{{\includegraphics[width=\columnSize]{obrazky-figures/refocus8Z2.jpg} }} 
    \subfloat[\centering ]{{\includegraphics[width=\columnSize]{obrazky-figures/refocus16Z2.jpg} }}
    \\
  \caption{\label{fig:refocus} 
The figure shows light field scene focused at two different distances. 
The amount of artifacts decreases with increasing angular resolution.}
\end{figure}

In practice, the novel $\textrm{view}$ at angular coordinates $s', t'$ can be produced as a linear combination of shifted input views, weighted by the defined aperture function $\textrm{A}(s,t)$, using the shift-sum algorithm~\cite{alain2019} according to a general Eq.~\ref{eq:shiftSum}.
The size of the input data grid in the number of images is $S{\times} T$.

\begin{equation}
\textrm{view}(s', t') = \int_{1}^{S}\int_{1}^{T} \textrm{L}'(s,t,u,v)\textrm{A}(s,t) \mathrm{d}s\mathrm{d}t
\label{eq:shiftSum}
\end{equation}

Compared to the general 4D light field representation, shift-sum works on the basis of pure view interpolation.
The synthetic novel view in 4D representation can be produced so that each part of the view can be interpolated from different input views.
The intersection of the two planes can inherently identify the angular and spatial coordinates used to sample the input views for each pixel in the novel view.
In this way, the whole light field grid can be utilized.
Shift-sum uses only the nearest input views to the virtual camera position and is thus less demanding regarding the sampling of the light field grid.
This fact can be advantageous for the GPU rendering due to the aligned memory access and SIMD (Single Instruction, Multiple Data) nature where each thread in a warp executes the same instruction on different data.

The refocusing is implemented by changing the sampling coordinates.
The sampling spatial coordinates on the focal plane are transformed according to the defined focus distance $f$. 
The vector of angular coordinates of the currently sampled view $\mathbf{st}$ and the vector of angular coordinates of the novel view $\mathbf{st}_{v}$ are used to obtain a new focused coordinates vector $\mathbf{uv}_{f}$~\cite{ng2005light} with spatial coordinates $\mathbf{uv}$; see Eq.~\ref{eq:trans}.
Note that the signs might change according to the orientation of the view grid.

\begin{equation}
\mathbf{uv}_{f} = \mathbf{uv}-(\mathbf{st}-\mathbf{st}_{v})f
\label{eq:trans}
\end{equation}

Eq.~\ref{eq:shiftSumRef} produces a refocused $\textrm{view}_{f}$ with the $\textrm{L'}_f$ function extended with the focusing parameter $f$, internally sampling the input views according to Eq.~\ref{eq:trans}.

\begin{equation}
\textrm{view}_{f}(s', t', f) = \int_{1}^{S}\int_{1}^{T} \textrm{L'}_{f}(s,t,u,v,f)\textrm{A}(s,t) \mathrm{d}s\mathrm{d}t
\label{eq:shiftSumRef}
\end{equation}

Eq.~\ref{eq:shiftSum} can be approximated by the weighted sum in Eq.~\ref{eq:shiftSumImplem}.
The weights $w_{i}$ approximate the shape of the aperture.
They depend on the distance between the sampled view coordinates and the novel view coordinates.
They can, for example, be sampled from a Gaussian function with its maximal value at the novel view.

\begin{equation}
	\textrm{view}(s', t', f) = \frac{\sum_{s=1}^{S} \sum_{t=1}^{T}  \textrm{L'}_{f}(s,t,u,v,f)\cdot w_{st}}{\sum_{s=1}^{S} \sum_{t=1}^{T} w_{st}}
	\label{eq:shiftSumImplem}
\end{equation}

Fig.~\ref{fig:focusShift} shows the principle of shifting and blending the images to focus on a certain area in the scene.
The focusing process can be performed pixel-wise. 
Each pixel of the scene can be focused by its own local focal plane; see Fig.~\ref{fig:allFocused}.

\begin{figure}[!htbp]\centering
  \includegraphics[width=\linewidth]{obrazky-figures/focusShift.pdf}
  \caption{
  Four input images from the grid are shifted to the novel view in the middle of them.
  They are blended together.
  The images are shifted so that the roof of the front house is focused.
}
  \label{fig:focusShift}
\end{figure}

\begin{figure}[!htbp]\centering
    \subfloat[\centering The scene is focused at the front house.]{{\includegraphics[width=0.49\linewidth]{obrazky-figures/focusFront.jpg} }}
    \subfloat[\centering The scene is all-focused, sharp everywhere.]{{\includegraphics[width=0.49\linewidth]{obrazky-figures/focusAll.jpg} }} \\
  \caption{\label{fig:allFocused} 
The figure contains two views on a 3D scene, demonstrating the effect of all-focused scene.
The first one is focused on the front building with a smooth depth-of-field simulation.
The second one contains all parts of the scene in focus.
Classic 3D rendering methods automatically produce all-focused results and focusing needs to be simulated with additional methods.
Simple light field rendering supports only one focus distance and all-focused results need to be produced by more advanced methods.}
\end{figure}

If each pixel in the novel view is focused, such view is called \textit{all-focused}.
The correct focus distance can be obtained by the ray intersection with the geometry of the scene.
A precalculated depth map provides the depth value for each pixel.
This value can be used directly or recalculated, according to the capturing camera attributes, as the focus distance~\cite{isaksen2000dynamically}.
A simplified geometric proxy can be used for the surface estimation instead of the depth maps.
The depth can be stored for each pixel in any light field parametrization~\cite{todt2007fast}.
This approach uses the principles of ray tracing with depth search~\cite{vogelgsang2000adaptive}. 
The mutual orientation between the intersection point on the geometric proxy and the arbitrarily positioned input camera can be used without a regular grid to determine which pixels need to be sampled~\cite{buehler2001}.

\subsection{Depth Extraction for Light Field Focusing}
Depth maps can be available during the capture process.
Their extraction is straightforward with synthetic scenes, where the scene geometry is known.
Depth is usually the position of a pixel in a 3D scene on the axis perpendicular to the camera grid plane. 
Sometimes, values in depth maps are distances of the pixels from the center of camera projection.
Fig.~\ref{fig:depthMaps} shows how the input light field data with depth maps would look like.

\begin{figure}[!htbp]\centering
  \includegraphics[width=\linewidth]{obrazky-figures/depthMaps.pdf}
  \caption{\label{fig:depthMaps}
The depth maps (on the right) can be used to approximate the scene geometry in light field rendering.
  }
\end{figure}

Depth, disparity, or optical flow estimations are dual problems in this case.
Each one of them can be used to calculate the other with additional information, such as camera positions and calibration data.
The relationship of features between the images serves as a basis for these estimations~\cite{ilg2018occlusions, beauchemin1995computation}.
Disparity maps describe how much pixels move between two images.
This is similar to optical flow that defines vectors for pixels or blocks that describe their relative motion between two similar images.

When optical flow algorithms are applied to multiple camera outputs instead of one camera video, it is possible to calculate a scene flow~\cite{menze2018object} that partially describes the geometry.
It can be said that the scene flow is a combination of disparity/depth and optical flow~\cite{schuster2018combining}.
The optical flow has to be filtered and the outliers are removed by energy minimization methods~\cite{sun2010secrets}. 
Optical flow calculation is a search problem in which a corresponding block of pixels needs to be found for each block of the other image~\cite{horn1981determining}.
Optical flow can be estimated from light field images in a manner similar to stereo views processing~\cite{chen2017fast}.
The optical flow calculation is optimized by first performing a sparse matching and then so called spatio-aware edge-aware filter is applied estimating dense flow.
The resulting disparity maps are then aggregated using median filtering followed by one-step of energy minimization calculation.
Fig.~\ref{fig:redundancy} demonstrates the visual relation between two views of a light field grid.

\begin{figure}[!htbp]\centering
  \includegraphics[width=0.99\linewidth]{obrazky-figures/redundancy.pdf}
  \caption{\label{fig:redundancy}
The right image contains highlighted areas which are also present in the left image.
This redundancy, typical for light field, can be exploited to estimate the depth of the scene.
Block-based approaches can be used to find the same spots in the views as depicted by the white drawing connecting both images.
The redundancy, on the other hand, increases the amount of data.
  }
\end{figure}

Deep learning can also be used to estimate the depth from a set of images or even a single image~\cite{eigen2014depth, peng2018unsup}, but additional processing would be needed to make the depth maps consistent throughout the whole light field according to the 3D scene.
The principle of light field refocusing can be used to segment the image according to the resulting foused areas to estimate depth~\cite{zhou20153d}.

A semi-global matching method estimates a dense disparity from rectified stereo images by searching for most similar pixel blocks between the images in predefined directions and search range~\cite{hirsch2008}.
Fast and good quality depth maps generation for light field images using semi-global matching principle was previously proposed~\cite{anisimov2019}.
It uses principles of the disparity map approximation method for stereo images.
This method searches for best disparity values along multiple corresponding lines in the input images. 

A lightweight approach~\cite{jiang2018} manages to extract depth maps even from sparse input data using intermediate disparity images and achieves the same results as more complex, previously proposed, methods in less time.
This method uses four corner images from the grid as input.
These images contain most of the geometric information about the scene because they are captured from the most extreme positions.
Rough disparity maps are estimated from them using optical flow and are aggregated by energy minimization algorithm.
After applying bilateral filtering, the maps are warped into the rest of the views.
Possible holes that appeared due to occlusions are filled by global inpainting.
However, the missing information can produce unwanted visual artifacts in possible view interpolations.
The maps can be directly used to reconstruct a 3D geometry of the scene for the given point of view; see Fig.~\ref{fig:pointCloud3D}.

\begin{figure}[!htbp]\centering
  \includegraphics[width=0.99\linewidth]{obrazky-figures/pointCloud3D.pdf}
  \caption{\label{fig:pointCloud3D}
3D mesh can be constructed from depth map but such a mesh is limited to the same or similar point of views as in the depth map image.
  }
\end{figure}

Graph Cuts method for the energy minimization for multi-camera scene reconstruction exists~\cite{kolmogorov2002multi}. 
Plenoptic camera datasets can be processed to extract the depth with symmetry property of the focal stack~\cite{linchen2015depth}.
Focal stack representation of light field data consists of images that are focused at different focal distances with depth-of-field effect.
Another technique suitable for plenoptic camera data uses spatial variance after angular integration of the epipolar image for defocusing and angular variance for correspondence depth cues estimation~\cite{taohadap2013depth}.
Small matching windows can be used for a lot of images in the light field datasets.
Flat uniform regions which are not suitable for such approach can be analyzed in a lower resolution, leading to multi-resolution matching approaches~\cite{neri2015multi}.
The extremely narrow baseline in lenslet light field camera data causes problems when estimating depth or disparity.
This problem can be solved by exploiting phase-shift theorem in the Fourier domain to estimate sub-pixel shifts~\cite{jeon2015accurate}.
The performance of depth or disparity estimation methods is, in most cases, not sufficient for real-time usage along with rendering.
Optimized methods for light field data also usually work well with the plenoptic camera data but not with the large baseline datasets.
Deep learning methods also require excessive memory.

The multi-resolution depth estimation can also be used when working with wide-baseline sparse datasets.
The capturing and rendering pipeline using such an approach with a point-cloud projection-based final image synthesis has already been proposed~\cite{sabater2017data}.
Point cloud rendering also requires additional refinements such as smoothing of edges and filling of occluded or empty areas~\cite{rosenthal2008image}.
However, using more cameras than the $4{\times}4$ proposed grid to achieve better rendering results in this pipeline could negatively affect the performance.
The conversion of depth maps to point cloud is straightforward; see Fig.~\ref{fig:pointCloud}.

\begin{figure}[!htbp]\centering
  \includegraphics[width=0.9\linewidth]{obrazky-figures/pointCloud.pdf}
  \caption{\label{fig:pointCloud}
The light field depth maps can be converted to point clouds which then can be merged to a 3D scene.
  }
\end{figure}

If the depth of the scene is known, novel views can also be rendered using image warping~\cite{schmeing2011depth}.
Light field rendering is potentially more GPU friendly and can maintain constant speed. 
Warping techniques must solve issues with regard to the ordering of pixels in occluded parts and the search for pixels in areas with missing data during the reprojection~\cite{lee2018}.

All search problems in algorithms described above can be optimized using the rules of epipolar geometry~\cite{xu2013epipolar} which can create search bounds to find the corresponding pixels.
For such an approach, the camera parameters and their exact positions are needed.
One point on one image can be found on a line on the other image.

\subsection{Light Field Focusing without Input Depth}
Additional depth maps can improve the quality of light field rendering.
However, they increase the memory and streaming bandwidth requirements and also the number of texture read operations, which is a crucial performance parameter for GPU algorithms.
Even if depth maps were available for each light field image, the depth information could be missing for certain, previously occluded, parts of the novel view.
The memory requirements can be partially solved by selective streaming of the light field data only for the currently visible part of the scene~\cite{erat2019}.
Light field rendering methods that do not rely on the input depth maps might work as alternative solutions.

A simplified geometry of the scene can be used in a classic rendering pipeline with the fine details substituted by a texture. 
To simulate view-dependent material attributes and to mask geometry inaccuracies, each polygon of the scene can be rendered with variable texture~\cite{Debevec1998Efficient}.
The method determines for each polygon in which input views it appears by projecting them on image plane from the corresponding viewing angle. 
Partially visible polygons are split.
Polygons that are not visible in any of the input images are filled with interpolated colors from neighboring polygons in object space. 
For each polygon, a view map is generated simulating viewing hemisphere assigning closest input images to the viewing angles.
This method combines both image-based and geometric rasterization rendering.
Fig.~\ref{fig:viewDependent} demonstrates the principle of variable texturing.

\begin{figure}[!htbp]\centering
  \includegraphics[width=\linewidth]{obrazky-figures/viewDependent.pdf}
  \caption{\label{fig:viewDependent}
A simple geometry can be rendered with dynamic textures that change according to the viewing angle.
It is a combination of image-based rendering and rasterization.
The images on the right correspond to the cameras and show the captured view (left image) and used textures for the same mesh area on the roof (right image).
  }
\end{figure}

Multiple unsynchronized video streams can be used as input for light field rendering~\cite{wang2007}.
In the first phase, a sparse, bidirectional optical flow is used to find correspondences between the input videos.
The result can be refined by applying an epipolar constraint that can serve to detect outliers.
Time offsets can be calculated to synchronize the video streams, and temporal interpolation of new video frames is carried out using the image morphing method. 
The quality is improved by an additional calculation of virtual edges between corresponding feature points. 
For spatial interpolation, the unstructured lumigraph rendering~\cite{buehler2001} method is used.
This method requires a geometric proxy of the scene, which requires preprocessing of the data with 3D reconstruction or depth extraction algorithms.

A light field movie can first be rendered offline using only the necessary set of cameras to cover the entire scene of interest~\cite{koniaris2017real}.
Each camera acquires a color and depth cubemap, which is compressed using a similar concept to video compression methods.
Raymarching algorithm is used to render the scene in real time using only necessary parts of cubemaps from the original cameras.
Textures from cubemaps are chosen according to the distance of the original cameras from the current virtual camera, viewing angle, and field of view. 

Tens of differently focused views for a given viewpoint, using standard light field rendering methods can be analyzed to produce an all-focused view~\cite{takahashi2003all}.
Areas in focus are then chosen~\cite{sugita2004focus} from the previously generated views and the final image is constructed from them; see Fig.~\ref{fig:allFocusComposition}.
However, this approach was demonstrated only on small-resolution images with a small distance between the cameras.
It also uses multiple synthesis filters, exploiting the density of Lytro dataset, which might not work well on the sparse datasets.
The method was further improved, but it is still unusable for real-time rendering~\cite{kubotaa2004all}.
This principle is similar to the main proposal of this thesis.
However, the proposed method works even with sparse data and on a per-pixel basis suitable for GPU acceleration.

\begin{figure}[!htbp]\centering
  \includegraphics[width=0.99\linewidth]{obrazky-figures/allFocusComposition.pdf}
  \caption{\label{fig:allFocusComposition}
    One possible method to achieve all-focused novel view is to produce several views that are focused at different distances.
    Then composing the final view from the focused parts of each intermediate view.
  }
\end{figure}

All-focus image can also be generated using high dynamic range light field~\cite{lizhang2015high}, where the position, direction, and exposure time information is integrated in the light field model.
Local focus distances can also be estimated for each view or even for each triangle of the resulting viewing plane by minimizing the least squares error~\cite{heigl2000pleno}.

In a more general way, the principle of selecting the optimal focus distance per pixel or pixel block can be defined by the 3D cost volume model~\cite{xiaodong2020}; see Fig.~\ref{fig:costVolume}.
The volume is defined by its resolution, which could be the same as the desired pixel resolution of the novel view.
In general, the resolution is the density of the result and higher resolution provides higher quality results but is more computationally demanding.
The depth of the volume is the hypothesis range.
The hypothesis planes are possible results that are being searched.
The search algorithm uses a cost function to determine which hypothesis is the optimal one for the given spatial coordinate.
Note that this model is ideal for massive parallelism computations, such as with GPU architectures.
This model is used for disparity or depth estimation in multi-view processing, as well as in the main proposal of this thesis.

\begin{figure}[!htbp]\centering
  \includegraphics[width=0.99\linewidth]{obrazky-figures/costVolume.pdf}
  \caption{\label{fig:costVolume}
   3D cost volume consists of cells that can be mapped to pixel blocks of light field focal stack.
   Each hypothesis plane in the depth dimension could be a differently focused light field view.
   The goal of a search algorithm would be to identify the optimal hypothesis for each spatial coordinate.
  }
\end{figure}

Light field images can also be analysed in spectral domain.
One of the depth-independent reconstruction methods exploits the sparsity in continuous Fourier domain to sample the light field efficiently~\cite{shi2015light}.
A new sampling quality metric that outperforms the maximized minimum distance and reduces the search space, using symmetry constraints, can be used in the novel view synthesis~\cite{schedl2018opti}. 

Densely sampled epipolar-plane image (EPI) reconstruction using shearlet transform can be achieved exploiting the light field sparsity in shearlet domain~\cite{vagh2018light}.
EPI is a visual representation of the scene where a visible point appears as a line with slope derived from the distance between the capturing camera and the point itself. 
The intensity along the line defines the light emission from the given point.
The depth of the scene is visible from the EPI~\cite{leistner2019} as shown in Fig.~\ref{fig:epi}.
This representation is the key principle of the EPI reconstruction method.
Depth-guided filters are used to analyze the EPI in the frequency domain to identify the optimal depth layer that can be used for the novel view reconstruction.

\begin{figure}[!htbp]\centering
  \includegraphics[width=0.95\linewidth]{obrazky-figures/epi.pdf}
  \caption{\label{fig:epi}
    Epipolar-plane image is constructed from a dense light field, where one row is selected, views are sorted and stacked on top of each other.
    The views are cut and displayed as a slice through the volume.
    The geometry of the scene is visible in the EPI in the slopes of the lines.
    The intersections of the lines mark occlusions.
    Two EPI views are shown from two slicing positions.
  }
\end{figure}

Deep-learning rendering based on few reference images~\cite{eslami2018neural, han2019image} can be used to produce novel views from light field data. 
An unsupervised approach working with planar light fields, using one network for disparity and one for occlusion map estimation, managed to yield results comparable to supervised approaches, overcoming the drawbacks of the full supervision methods~\cite{ni2019unsup}.
The disparity estimation uses pyramid and a cost volume approach on the input images.
Warping is used to produce a novel view.
The warping is used in a cyclic manner to acquire the information about the scene geometry and occlusions.
Forward-backward warping takes two input views, warps one to the novel view, from the novel view to the second input view, and then back to the novel and first input.
In this way, inconsistencies between the warped pixels and the input views reveal possible missing information caused by occlusions; see Fig.~\ref{fig:forwardBackwardWarp}.

\begin{figure}[!htbp]\centering
  \includegraphics[width=0.99\linewidth]{obrazky-figures/forwardBackwardWarp.pdf}
  \caption{\label{fig:forwardBackwardWarp}
  The middle image is the novel view produced by blending and warping of the left and right input views.  
  To reveal occlusions, forward and backward warping is used so, for example, point $p_1$ can be warped from the first input to the novel view but not to the second input. 
  That reveals $p_1$ being occluded in the second input.
  Point $p_2$ is visible in all views and would be propagated through the warping back and forth.
  }
\end{figure}

Specific deep learning models for light field data and novel view synthesis also exist~\cite{suhail2022,suhail2022light}.
When the camera position and parameters are known, image features can be sampled along epipolar lines across several views near the virtual one. 
The neural network can evaluate the best match along these lines to synthesize all-focused view. 
These methods can deal with occlusions even with missing data.
Their real-time usage in 3D simulations with high-resolution light fields would be problematic due to their memory and time requirements.
Fig.~\ref{fig:deepEpipolar} shows the principle of epipolar features.

\begin{figure}[!htbp]\centering
  \includegraphics[width=\linewidth]{obrazky-figures/deepEpipolar.pdf}
  \caption{\label{fig:deepEpipolar}
A point in the central novel view has related points in the input view images from the same spot in the scene if no occlusions are present.
Many light field rendering methods exploit the fact that the positions of the input views are known and an epipolar line can be defined for each image.
The main task of the rendering is the search for the best match along epipolar lines (right).
  }
\end{figure}

Tremedous amount of research was conducted in the field of video frame interpolation~\cite{dong2023}.
The frame interpolation methods can be used to enhance a video, especially by increasing its framerate.
Such videos look more smooth.
Frame interpolation is a common way to improve framerate even in the field of computer games~\cite{kapse2021}.
Less information is needed to be transmitted via the network or from the host memory to the GPU.
Also, fewer views need to be rendered in case of 3D simulations with graphical output.
Frame interpolation methods can also be used for the novel view synthesis in light fields.
Fig.~\ref{fig:twoToFourInterpolation} shows the approach to exploiting frame interpolation, which usually works as a novel frame interpolator between two consecutive video frames.
The interpolation is used three times to acquire a central view between four nearest views in the light field grid.

\begin{figure}[!htbp]\centering
  \includegraphics[width=\linewidth]{obrazky-figures/twoToFourInterpolation.pdf}
  \caption{\label{fig:twoToFourInterpolation}
The interpolation is first used to synthesize the middle images between two pairs.
The results are then interpolated again into the final view in the middle of the four input views.
  }
\end{figure}

The standard frame interpolation can be implemented using optical flow, which identifies the corresponding parts of two frames and serves as the interpolation guide.
Fast GPU accelerated approaches exist, such as NVIDIA FRUC library\footnote{\href{https://developer.nvidia.com/opticalflow-sdk}{developer.nvidia.com/opticalflow-sdk}} which can be directly used to interpolate intermediate video frames.
A GPU-accelerated high-accuracy optical flow~\cite{brox2004high} is implemented in OpenCV framework and can be used to interpolate, for example, new video frames between two original ones.
This approach can be used for light field data to make the input image grid more dense or to generate intermediate views.
The limitation is the processing time and the inability to use additional information from surrounding views, except for the two reference ones.
Utilizing optical flow-based methods can be problematic when the scene contains a lot of occluded parts, reflections, or areas without detailed textures.

Deep learning provides better results in such cases.
The frame interpolation seems to work better on light field data than general deep-view interpolation methods \cite{chlubna2023}.
Light field data usually do not capture the scene from significantly different angles and positions, which makes them unsuitable for methods designed to work with free-look data.

A combination of meta-learning and test-time adaptation leads to a robust network for frame interpolation~\cite{choi2020}.
Meta-learning ensures better adaptability for various input tasks as it modifies the learning algorithm itself according to the given task.
Frame triplets were used as the training samples.
The triplet consists of three consecutive video frames, where the middle one is considered as the expected ground-truth result.
Optical flow and pyramidal feature extractors can be combined with deep learning and softmax splatting, which guarantees better interpolation results than optical flow or deep learning alone~\cite{niklaus2020}.
Bidirectional optical flow between the two input images is first estimated.
The network is used to synthesize the novel view between them, guided by the warped images according to the flow.
Softmax splatting solves the problems with sampling and splatting; see Fig.~\ref{fig:splat}.
It can take into account the depth relation between the pixels and separate background and foreground correctly.

\begin{figure}[!htbp]\centering
  \includegraphics[width=\linewidth]{obrazky-figures/splat.pdf}
  \caption{\label{fig:splat}
When image is transformed, for example, warped by disparity map to the novel camera position, the result can be produced in two simple ways.
By splatting, sending the input pixels directly to the resulting pixel grid which can produce holes and overlaps.
By sampling, where each pixel in the destination grid looks for the nearest source pixels and computes the new value, for example, by averaging them.
The figure shows how rotated pixels are propagated.
  }
\end{figure}

FLAVR framework uses 3D spatio-temporal kernels to directly learn motion properties from
unlabeled videos~\cite{kalluri2021flavr}.
This method can provide fast results in a good quality.
The improved texture synthesis in ST-MFNet outperforms the previous approaches in terms of visual quality and uses four closest frames in the video instead of two~\cite{danier2022}.

General multi-view synthesis neural approaches can also provide acceptable results~\cite{wang2021nerfmm, mueller2022instant} but are often not designed for light field data where views are aligned in a grid.
To reduce the memory requirements, it is optimal to use only the nearest views, which seems to be problematic for this kind of network.
The problem is similar to the 3D reconstruction methods, which need more 3D information about the scene derived from the views.
These issues can be partially solved by identifying the camera rays projected in all views. 
The color information along the rays can be used as the input of the network which significantly shrinks the search domain for the correct synthesis of the novel view~\cite{wang2021ibrnet}.
IFRNet is a real-time solution that reaches the same quality levels as the other works~\cite{kong2022}.
Similarly to other approaches, feature pyramid is extracted from the input views.
The output of the feature extraction encoder is passed to the coarse-to-fine decoders.
Intermediate flow fields are gradually refined in a sequence of decoders.
The performance of the model is improved by novel objective functions: task-oriented flow distillation loss and feature space geometry consistency loss.
Pyramidal approach is used in computer vision often to detect features and objects at different scales; see Fig.~\ref{fig:pyramids}.

\begin{figure}[!htbp]\centering
  \includegraphics[width=\imageSize]{obrazky-figures/pyramidGauss.jpg} \\
  \includegraphics[width=\imageSize]{obrazky-figures/pyramidLaplace.jpg}
  \caption{\label{fig:pyramids}
The input image is scaled down multiple times to produce several levels of scale in the Gaussian pyramid (top image).
This pyramid can be used in scale-invariant feature or object detection.
Laplacian pyramid (bottom image) is formed as a subtraction of two successive levels in the Gaussian pyramid and is used for image compression.
The pyramids are often produced in grayscale, this figure contains RGB versions and its brightness is increased for better readability.
  }
\end{figure}

The deep interpolation is ready to be used even for modern standards like 4K videos~\cite{park2023BiFormer}.
Large spaces between the frames and high resolution of the video introduce drawbacks to the existing methods.
The main issue is larger pixel distance between the features in the frames.
Search windows would have to be excessively big to quickly match the detected inter-frame features.
Global motion fields at a coarse scale are computed.
The issue of vanishing details in coarser scale is solved by refinement of the fields with an upscaling module.
Fig.~\ref{fig:scaleDisparity} shows the problem of uneven distances between original and downscaled images.

\begin{figure}[!htbp]\centering
  \includegraphics[width=0.99\linewidth]{obrazky-figures/scaleDisparity.pdf}
  \caption{\label{fig:scaleDisparity}
Search windows used to find the corresponding features in two views might not be optimal for high-resolution inputs.
The image can be downscaled to reduce the distance in pixels between the features.
Important details might be lost in such process due to the general loss of information introduced by downscaling.
The figure shows the original and $10\times$ downsampled views.
Right bottom corner of the original blended image (top-right image) shows the size of their difference.
  }
\end{figure}

Networks such as FILM~\cite{reda2022}, that support large motion between the frames are the most relevant for light field synthesis, as they are not limited to close views only.
Input images are processed in a pyramid manner with a feature extractor.
Scale agnostic bi-directional flows are computed from the features, and the final view is synthesized by UNet decoder.

\subsection{3D Reconstruction}
\label{sec:lf3D} 
The light field images can be used in geometry reconstruction algorithms~\cite{moons20103d}.
The quality of the results depends on the density of input views, existence of additional information, for example, depth and camera parameters, lighting conditions, or image quality.
Standard photogrammetry and structure-from-motion methods, where the 3D scene or object is reconstructed based on the corresponding features and similarities detected in the input pictures~\cite{linder2009digital}, can be used with light field data.
Software such as COLMAP, Meshroom, VisualSFM, Regard3D, Metashape, Autodesk ReCap, 3DF Zephyr exists and can generate 3D representations of the light field scene with quality dependent on the dataset and methods implemented in each program. 
However, according to the conducted experiments, these methods might fail with light field datasets because of the lack of free-look views around the scene.
Fig.~\ref{fig:reconstruct} shows the results of 3D reconstruction experiments on standard light field datasets. 

\begin{figure}[!htbp]\centering
    \subfloat[\centering Central view is presented.]{{\includegraphics[width=0.49\linewidth]{obrazky-figures/pavilion.jpg} }}
    \subfloat[\centering Scene is reconstructed with Metashape.]{{\includegraphics[width=0.49\linewidth]{obrazky-figures/metashape.jpg} }} \\
    \subfloat[\centering Scene is reconstructed with 3DF Zephyr.]{{\includegraphics[width=0.49\linewidth]{obrazky-figures/3df.jpg} }}
    \subfloat[\centering Scene is reconstructed with Visual SFM.]{{\includegraphics[width=0.49\linewidth]{obrazky-figures/vsfm.jpg} }}
  \caption{\label{fig:reconstruct} 
The existing reconstruction software does not work well with light field data.
Metashape estimated the camera positions correctly and produced an rough and incomplete 3D mesh.
3DF Zephyr reconstructed the scene and produced an accurate mesh estimation in the shortest time, but the model contains a lot of holes.
Visual SFM estimated the camera positions, reconstructed the sparse point cloud but could not produce a viable dense cloud.
The reconstruction took several minutes, measured with Intel\textsuperscript{\textregistered} Core\textsuperscript{\texttrademark} i9-9900K and NVIDIA\textsuperscript{\textregistered} GeForce RTX\textsuperscript{\texttrademark} 4090.}
\end{figure}

COLMAP, Meshroom and Regard3D were not able to produce an acceptable result.
Most of the scenes that work well in light field rendering methods are not usable at all in the reconstruction software, even with high-quality settings.
The reconstruction sometimes takes tens of minutes to complete, even on a high-end machine.
These approaches need to preprocess the input data, calibrate the camera, find the corresponding features between the images, create the mesh, and apply textures~\cite{kang2020review}.
Generally, light field datasets with wider space between the capture cameras are more suitable for 3D reconstruction algorithms.
Light field rendering methods, on the other hand, generate more artifacts with such datasets.

It is possible to reconstruct a scene in real time, for example, with a stereo camera~\cite{geiger2011stereoscan}, but the possible combinations of light field images increase the necessary processing time.
Real-time methods such as ORB-SLAM3~\cite{campos2021orb} for camera pose estimation can usually also produce a dense point cloud of the environment that can be used for 3D reconstruction. 
These methods usually depend on the already calibrated camera parameters, and the process should be initialized with enough input views of various orientations~\cite{taketomi2017visual}.
A similar approach to 3D reconstruction is point-cloud reprojection that also requires depth information~\cite{perra2016analysis}.
Having an image and depth map, the 3D geometry can be reconstructed, but information from different viewing angles might be missing~\cite{battiato20043d}.
Therefore, the viewing cone would be limited.
The scene can be represented as a set of textured planes~\cite{waschbusch20073d} or other simple objects such as spheres~\cite{broxton2020immersive}.
The depth associated with the pixels can be quantized, segmented, and processed to produce a very simple geometry. 
The set of billboards would result in a fast rendering; see Fig.~\ref{fig:billboards}.
However, some parts of the scene might create the cardboard effect~\cite{yamanoue2000} due to the lack of depth-related details.

\begin{figure}[!htbp]\centering
    \subfloat[\centering Front view on the billboards looks natural.]{{\includegraphics[width=0.32\linewidth]{obrazky-figures/billboardFront.jpg} }} 
    \subfloat[\centering Shifted view looks natural but some 3D details are missing.]{{\includegraphics[width=0.32\linewidth]{obrazky-figures/billboardSide.jpg} }} 
    \subfloat[\centering Side view reveals the incomplete geometry.]{{\includegraphics[width=0.32\linewidth]{obrazky-figures/billboardTrue.jpg} }} 
  \caption{\label{fig:billboards} 
  The figure shows how billboards can replace a 3D scene.
  They are constructed according to one viewing direction.
  They can produce acceptable results when viewed from this direction and slightly shifted camera positions.
  Similarly to light fields, they cannot be used for views from completely different camera positions.  
}
\end{figure}

Real-time global illumination methods often use probes as an acceleration of rendering instead of raw ray tracing of the scene~\cite{guo2022eff}.
The probes can store the 3D incoming light information in the form of a light field~\cite{mcguire2017real}.
This approach can be further extended to handle dynamic lighting~\cite{chen2021}.
The geometric and radiance information can be separated and the radiance reconstructed using the principles of deferred shading.

Real-time image-based light field rendering methods are potentially faster and less memory demanding than scene reconstruction and rendering of the result because of a reduced number of necessary operations. 
3D reconstructions might also fail completely when the necessary correspondences between the views are not found or are not distributed properly in the 3D space.
Light field rendering always produces at least the best-effort result, where the scene is reconstructed with visible artifacts in the worst case.

\section{Capturing of Discrete Light Field}
\label{sec:lfAcq} 
The discrete approximation of light field, in the form of multiple images, can be captured by a grid of cameras as shown in Fig.~\ref{fig:capture}.
The grid can be replaced with one moving camera in the case of static scenes.
The shape of the grid depends on the rendering method used.
The grid is generally rectangular and planar with constant spaces between the cameras~\cite{shi2019framework,vaish,honauer2017dataset,shekhar2018light}.

\begin{figure}[!htbp]\centering
    \subfloat[\centering Camera array captures a scene.]{{\includegraphics[width=0.4\linewidth]{obrazky-figures/cameraArray.pdf} }} 
    \subfloat[\centering Resulting views are stored as images.]{{\includegraphics[width=0.59\linewidth]{obrazky-figures/viewArray.jpg} }}
  \caption{\label{fig:capture} 
The figure shows a $4{\times}4$ array of cameras in a synthetic scene and the corresponding grid of captured views.}
\end{figure}

Another way to capture light field data is by using a plenoptic camera~\cite{rerabek2016new}.
The set of images is acquired in the same way as with a camera grid except that only one sensor is used and the incoming rays pass through a microlens (lenslet) array that is between the main lens and the sensor~\cite{ng2005light}.
Lytro company~\cite{georgiev2013lytro} had released three models of plenoptic cameras, where Lytro Illum became the most popular and capable of acquiring 4 megapixel views.
However, the extracted views from such cameras are very close to each other, so the main use-case for such photos is only the refocusing and related effects.
Fig.~\ref{fig:plenoCams} contains three handheld devices that can be used to capture light field views.
The resulting photos can be viewed in Fig.~\ref{fig:plenoCamsResults}.

\begin{figure}[!htbp]\centering
  \includegraphics[width=0.6\linewidth]{obrazky-figures/plenoCams.jpg}
  \caption{\label{fig:plenoCams}
Three devices able to capture light field data are displayed in the photo.
It is a standard Samsung Galaxy NX camera with 3D stereo lens, Lytro plenoptic camera, and 3D stereo camera on Leia LumePad 3D tablet.
  }
\end{figure}

\begin{figure}[!htbp]\centering
  \subfloat[\centering 3D lens provides a small displacement.]{{\includegraphics[width=0.49\textwidth]{obrazky-figures/samsung.jpg} }} 
  \subfloat[\centering 3D tablet also provides close stereo images.]{\includegraphics[width=0.49\textwidth]{obrazky-figures/leia.jpg}} \\
  \subfloat[\centering Plenoptic camera obtains lots of close views used for refocusing.]{\includegraphics[width=0.9\textwidth]{obrazky-figures/lytro.jpg}}
  \caption{\label{fig:plenoCamsResults}
The results of the 3D lens and the 3D tablet are simple stereo images displayed in the figure as side-by-side views.
The plenoptic camera stores a plenoptic image that can be refocused or tilted, as shown in two different views in the official Lytro Desktop software.
  }
\end{figure}

Camera grids have the advantage of being scalable, having a larger distance between the cameras, and potentially having higher resolution because each view is captured by the whole sensor.
A similar solution targeting industry and science is Raytrix 3D Camera~\cite{perwass2013digital}. 
Plenoptic acquisition principles are also used in Fourier integral microscopes~\cite{scrofani2018}.
Multiple sensor cameras can be used to increase resolution and reduce noise in standard photography~\cite{sahin2017}.
This approach is often used in modern smartphones to widen the effective image sensor area for better zooming and telephoto options~\cite{blahnik2021}.
A two-sensor camera is used in the LumePad 3D tablet to acquire a stereophoto that can be viewed directly on the device~\cite{fattal2021}.
3D information can also be captured with the standalone depth sensor.
An example of this approach is Microsoft Azure Kinect~\cite{zhang2012microsoft}.

The distance between views is an important parameter that affects the quality of the resulting image.
The closer the images are to each other, the fewer artifacts occur in the final render because the differences between images are very small.
The 3D effect is less visible if the angular resolution remains the same while the distance between the cameras is decreased.
The sampling rate of light fields during acquisition depends on the content of the scene and the final use case of the rendering.
Optimal sampling can be estimated by spectral or optical analysis of the scene~\cite{chai2000plenoptic}.

A study, conducted by the author of this thesis, revealed that input light field images captured in logarithmic color profiles can improve the visual quality of the rendering methods~\cite{chlubna2024profile}.
Cameras offer various capturing color profiles that map the input light intensity into digital color values by different functions.
Synthetic scenes can also be rendered in such profiles.
The logarithmic profile CanonLog3 is the most efficient.
Logarithmic profiles reveal details in areas that are too dark and too bright~\cite{gupta2019}.
They are often used for artistic purposes as they enhance the post-processing options.

\section{Compression of Discrete Light Field}
\label{sec:lfComp} 
One of the main drawbacks of light field rendering are the high memory space requirements.
Light field data are often larger than standard 3D geometrical representation.
The capturing cameras usually record the same geometry with a lot of redundancy between the views~\cite{chai2000plenoptic}.
The memory requirements on the GPU can be reduced by efficient data reuse approaches and progressive rendering~\cite{lee2022}.
Efficient compression methods are still necessary and can be used even in combination with smart streaming of the necessary parts of the light field grid.
Compression methods can also improve the visual quality of the novel view rendering.
If a highly efficient compression is used, the light field might be captured extremely densely without large spaces between the views.
Closer views are easier to interpolate
If the density of the views is larger than the necessary sampling density, the novel view rendering might not even be necessary.

\subsection{Standard Methods for Light Field Compression}
The performance of the main image coding standards, JPEG, JPEG 2000, H.264/AVC intra profile, and H.265 intra profile was evaluated~\cite{Alves2016}.
The \textit{intra} suffix refers to the fact that the individual views were compressed independently (intra profile).
The H.265 intra profile proved to be the most efficient compression method.

Three strategies using the H.265 were also compared~\cite{Viola2017}.
The first strategy performs compression directly on the lenslet image.
Another strategy arranges 4D light field views in spiral order and subsequently compresses them.
The last strategy compresses a subset of lenslet images through the transformation to 4D light field.
The results show that coding 4D light field leads to better performance compared to coding the lenslet images directly.

The performance of JPEG, JPEG 2000, and SPIHT (Set Partitioning in Hierarchical Trees) directly on lenslet images was also compared~\cite{Higa2013}.
The JPEG 2000 exhibits the best compression performance.

4D light field views can be rearranged as tiles of a large rectangular image~\cite{Perra2017}.
This image is then compressed using the JPEG 2000 coder.
The proposed scheme was compared against standard image coding algorithms, namely JPEG 2000 and JPEG XR.
However, it is unclear how these standard coding algorithms were exactly applied to the 4D light field data.

The similarity between light field images is high.
Fig.~\ref{fig:flowCompression} shows a Farneback dense optical flow calculated between two views from the light field grid.
The motion vectors correspond to the structure of the scene and identify a lot of redundant data.

\begin{figure}[!htbp]\centering
  \subfloat[\centering First image is taken.]{{\includegraphics[width=0.32\textwidth]{obrazky-figures/flowCompressionFirst.jpg} }}
  \subfloat[\centering Second image is taken.]{{\includegraphics[width=0.32\textwidth]{obrazky-figures/flowCompressionSecond.jpg} }}
  \subfloat[\centering Optical flow shows a motion.]{{\includegraphics[width=0.32\textwidth]{obrazky-figures/flowCompressionFlow.jpg} }}
  \caption{\label{fig:flowCompression}
  The dense flow shows that similar pixel blocks are correctly detected between the neighboring images in light field grid.
  That suggests that compression methods exploiting this correspondence might be efficient.
}
\end{figure}

4D light field can also be rearranged into a three-dimensional body~\cite{Aggoun2006}.
The three-dimensional volume is then encoded using the 3D DCT scheme on $8{\times}8{\times}8$ blocks, similarly as in the JPEG coding system.
Fig.~\ref{fig:sequenceVolume} shows how light field views can be rearranged for different compression methods.

\begin{figure}[!htbp]\centering
  \includegraphics[width=\linewidth]{obrazky-figures/sequenceVolume.pdf}
  \caption{\label{fig:sequenceVolume}
Two main approaches are used in light field compression.
The views are rearranged from the grid to a pseudo-sequence which is encoded, for example, using video encoders, as depicted on the left image.
The views can be also aligned to form a volume which can be compressed by exploiting the redundant parts which overlay each other, as depicted on the right image.
  }
\end{figure}

H.264 standard describes the Multiview Video Coding (MVC) extension, which is also described in the H.265 (HEVC) standard (MV-HEVC) for optimal coding of animated multiview sequences~\cite{mesloul2018}.
This extension is capable of encoding multiple views and exploiting their redundancy.
It was designed for stereoscopic videos.
The main downside of these extensions is the lack of implementations, especially in open-source software and libraries.
Hardware acceleration is almost non-existent.
It was used as part of the light field compression scheme with hierarchical organization of the views~\cite{ahmad2019computationally}.
An efficient 3D prediction structure for MV-HEVC was proposed~\cite{khoury2019new} and proved to be better than similar previous approaches~\cite{wang2016}.
MV-HEVC can be easily adjusted to support the light field grid and several prediction schemes were compared~\cite{avramelos2020random}.
The results show that MV-HEVC can be used for a random-access encoding which is suitable for light field data.

\subsection{Video Methods for Light Field Compression}
Light field compression with video codecs outperforms other 2D or 3D extended image compression methods according to previous research~\cite{barina2022, barina2019, ahmad2019computationally, SANTOS201821}.
Another reason for the promotion of video codecs is their widespread usage, the variety of available implementations, and the increasing support on modern GPUs~\cite{palau2021} with the new AV1 support on the NVIDIA\textsuperscript{\textregistered} RTX\textsuperscript{\texttrademark} 3xx series and possible future support of current state-of-the-art XVC~\cite{samuelsson2019} and VVC~\cite{bross2021} (Versatile Video Coding) formats.
H.265 standard~\cite{sullivan2012}, similarly to alternative formats, offers intra-frame coding, which compresses the frame independently.
This approach uses techniques similar to other image compression algorithms such as JPEG~\cite{wallace1992}, for example, block-based processing that exploits spatial redundancy in the image, quantization of data in the frequency domain or chroma subsampling.

Inter-frame coding, available in all standard video compression methods, exploits temporal redundancy.
Similar blocks in multiple frames can be identified using block-matching algorithms~\cite{srinivas2022} and encoded as blocks with motion vectors instead of multiple blocks. 
The input video frames, shown in Fig.~\ref{fig:frames}, are compressed in three basic ways~\cite{bhaskaran1997}:
\begin{itemize}
    \item \textbf{I-frame} is a keyframe that is encoded only by intra-frame methods. It can be decoded without information from any other frames.
    \item \textbf{P-frame} is a difference frame which encodes only changes (new parts of the image and motion vectors) from the previous frames. It is necessary to have the relevant reference frames already decoded.
    \item \textbf{B-frame} is similar to the P frame but can also use later frames as reference for decoding.
\end{itemize}

\begin{figure}[!htbp]\centering
  \includegraphics[width=0.7\linewidth]{obrazky-figures/frames.pdf}
  \caption{\label{fig:frames}
  Three standard types of frames used in video compression are depicted in the figure.
  I-frame can be encoded separately, P-frame depends on the last I-frame, and B-frame depends on I-frames on both sides.}
\end{figure}

The encoded video stream is a numbered sequence of packets, where usually one packet corresponds to one frame.
Modern encoding standards can also split frames into macroblocks or slices, where each slice can be independently encoded as one of the three mentioned types~\cite{schwarz2014}.
Group of pictures (GOP) is a sequence of encoded frames beginning with an I-frame.
The rest of the frames are P or B-frames that use one or more frames from the same GOP as references.
When random access in a video stream is needed, the closest GOP beginning is found.
The subsequent frames are decoded until the desired one is reached.
Light field data can be encoded as a pseudo-sequence that is designed for optimal compression of the whole grid~\cite{mehajabin2019efficient}.
Another pseudo-sequence approach exploits the redundancies in sub-aperture image intra spatial, sub-aperture image inter-view, intra-micro-image, and inter-micro-image space~\cite{monteiro2020}.
Neural video compression methods can also be used instead~\cite{li2023neural, yang2020Learning}.

\subsection{Specialized Methods for Light Field Compression}
One of the first proposals~\cite{levoy1996light} was to use vector quantization to reduce redundancy between views.
2D or 4D tiles of the light field are encoded as vectors and saved as references to a codebook constructed during the training phase.
This compression is lossy because the closest value in the codebook is assigned to the input vectors.
A training-based dictionary technique~\cite{miandji2019unified}  provides a way to compress light field images and videos.
This approach uses a multidimensional dictionary ensemble instead of single 1D dictionaries used in different proposals.
Simple approaches such as selective subsampling of the views in the grid can improve the compression ratio for many light field encoders.
This approach is mainly efficient with dense light field grids~\cite{chen2022view}.

Another approach divides the 4D light field into blocks and then converts to the YUV color space with downsampled chrominance~\cite{magnor2000data}. 
Evenly distributed I-frames are chosen from the input images and compressed using block-based discrete cosine transform and coefficient quantization.
P-frames that are used as difference frames between the closest I-frames are then calculated using the disparity between the compressed image and the adjacent I-frame.
Because the shift between images is known, the disparity direction can be exactly defined.

Hierarchical motion-compensated compression~\cite{pratapa2019hmlfc} was proposed as an efficient solution for interactive rendering.
This method analyzes the relative motion between input images and applies image transformations to minimize redundancy in a hierarchical manner.
Both motion compensation and hierarchical approaches are combined into one hybrid method.
The main advantage of this approach is a fast GPU-friendly decoding.
Similarly to other methods, this method loses efficiency as the distance between the input images increases. 
Fig.~\ref{fig:dictionary} shows an example of the identification of common elements in light field.

\begin{figure}[!htbp]\centering
  \includegraphics[width=\linewidth]{obrazky-figures/dictionary.pdf}
  \caption{\label{fig:dictionary}
  Blocks which contain the same visual information in all light field views can be detected.
  These blocks can be used in dictionary-based compression or for inter-view prediction.
  Note that simply identifying blocks of pixels in multiple views that come from the same spot in the scene is not enough.
  Reflections and other lighting effects produce view-dependent changes in some materials.}
\end{figure}

Other standard image or video compression methods are used directly or in hybrid approaches, such as displacement intra prediction~\cite{Li2016}.
The image displacement in the 4D light field structure is estimated using the intra prediction of the H.265 standard in both the time and spatial domains.

Sparse coding scheme for the entire 4D LF based on several optimized key views can be used for an efficient compression~\cite{Chen2018}.
Another method decomposes the 4D light field into homography parameters and residual matrix~\cite{Jiang2017}.
The matrix is then factored as the product of a matrix containing $k$ basis vectors and a smaller matrix of coefficients.
The basis vectors are then encoded using the H.265 intra profile.
Similarly, H.265 can be used to remove redundancies in dynamic mode decomposition compression~\cite{ravishankar2023data}.
A hierarchical coding structure for 4D light fields can also be used~\cite{Li2017}.
The 4D LF is decomposed into multiple views and then organized into a coding structure according to the spatial coordinates.
The scheme is implemented in the reference H.265 software.
Another approach is a coding scheme that splits the 4D light field into several central views and remaining adjacent views~\cite{Han2018}.
The adjacent views are subtracted from the central views and then both groups are encoded using the H.265 coder.
The 4D light field can be fed into the H.265 exploiting the inter-prediction mode for individual LF views~\cite{Li2014}.
Disparity-based sparse prediction~\cite{astola2019} warps the reference views to the compressed ones with normalized disparity maps.
Then it reconstructs the view with inpainting and filtering based on the result.
JPEG2000 is used to encode the disparity views and H.265 to encode the residual and color views.
This method can be used for both dense and sparse datasets and outperforms alternative plenoptic and video encoders.

A lot of research focuses on compression based on convolutional neural networks~\cite{Hou2019,Jia2019, zhao2019light}.
The learning process might often be time-consuming.
Deep image prior~\cite{lempitsky2018}, with random neural network initialization, can be used to compress light field data without the training process~\cite{jiang2022}.
It is compact and lightweight and can reach better compression ratios than alternative networks.
All training-based methods depend on the quality of the training phase and are usually not suitable for memory-efficient and fast GPU decoding.

Various compression schemes have previously been proposed for different light field formats or live-streaming scenarios~\cite{kovacs2017}.
Data streaming can be optimized by analyzing the current light field coverage of the scene so that only the data necessary to improve the result are uploaded to the memory~\cite{erat2019}.

Light field rendering and compression are not just connected as different steps of the light field workflow.
Novel view rendering and synthesis can also be used as a compression method.
The desired views can be synthesized from other existing views similarly to reference and prediction frames in video compression methods but with fewer information about the novel view.
Deep learning can be used to estimate the 2D view from the sparse sets of 4D views~\cite{Bakir2018}.
A sparse set of images is encoded using a hybrid video encoder, and the missing views are approximated from the decoded ones.
Similarly, multi-disparity geometry structure estimated from a sparse set of images can be used in a multi-stream dense view reconstruction network to synthesize a novel image for decompression purposes~\cite{liu2022}.
This method handles well the occluded geometry and reveals texture details in high quality.
Residual CNN-Assisted JPEG compression method~\cite{hedayati2021} outperforms the GPU-accelerated H.265 decoding speed $10{\times}$ for light field compression.
The method uses depth-based warping of the central image from Lytro camera scenes.
The central image is encoded in JPEG format.
A combination of novel video compression format VVC and deep learning for missing view synthesis uses advantages of both approaches~\cite{bakir2020}.

Disparity-assisted models can be more useful in light field compression due to the spatial distribution of information across the grid~\cite{singh2021}.
However, novel standard image/video compression methods can still reach results close to the schemes tailored for light field.
Especially when the views are far from each other and not captured by a plenoptic lenslet camera, the disparity-based warping can introduce a significant amount of artifacts.

\input{chapters/LightFieldCompression}