\section{Real-time Per-Pixel Focusing}
\label{sect:realtime}
This section is based on an original method of the author~\cite{chlubna2021}.
The basic method described here addresses the claims of \hyperref[hypothesis]{Hypothesis} that the proposed method is computationally more efficient than existing methods and does not require additional depth information about the input scene.

This section describes a real-time light field rendering method that can be used with contemporary consumer hardware.
The inputs of this method are a discrete light field approximation, as a grid of images, and a position of the virtual camera.
The output is a novel view that is focused in all parts of the scene. 
Various open-source applications were implemented during this research and an interactive web demo\footnote{The demo is available as a part of presentation website for one of the original papers~\cite{chlubna2024accel} here:\\\href{https://www.fit.vutbr.cz/~ichlubna/lf/}{fit.vutbr.cz/$\sim$ichlubna/lf}} is available.
The main principle, allowing for the real-time usage, is the design of a highly parallel optimal algorithm for GPU.

First, a focus map with the optimal focus distance is computed for each pixel.
Then, each pixel of the novel view is computed as a combination of pixels from the input images, so that the contributing pixel colors are sampled from the same spot in the scene.
The focus map contains the mutual displacement of the images to achieve the all-focused sampling and simulates the scene surface.
The focus map can resemble a depth or disparity map, but it does not contain the real depth of the scene.
The scene structure does not have to be visible on the focus map; see Fig.~\ref{fig:focusDepth}.

\begin{figure}[!htbp]\centering
  \includegraphics[width=0.9\linewidth]{obrazky-figures/scan/focusDepth.jpg}
  \caption{
The first image is a depth map.
The second one is a focus from the proposal.
It differs from the depth, but leads to artifacts-free novel view in the third image.
  }
  \label{fig:focusDepth}
\end{figure}

Compared to the disparity map, the focus map contains the offsets for all light field images and is not computed only for stereo pairs~\cite{hamzah2016literature}.
The values in the focus map are calculated for the new synthetic view and are not explicitly aligned with any of the input images.
Color similarity is the main metric used during the focus map estimation as the scene geometry correspondence in the views is not available.
The main goal is to minimize the final rendering error.
Therefore, focus maps are tailored to be used in the light field image-based rendering but not for 3D surface or point-cloud reconstruction.
The scene can be viewed from an arbitrary point of view within the bounds defined by the capturing grid.
Fig.~\ref{fig:perPixelTeaser} shows the principle of focus map usage in light field rendering.

\begin{figure}[!htbp]\centering
\includegraphics[width=\textwidth]{obrazky-figures/perPixel/teaser.pdf}
  \centering
   \caption{Left: Comparison of fully focused light field image as a result of the proposed method with light field focused at a single distance.
   Right: From the input set of images taken by camera grid, a new synthetic view of the scene is generated having every location in the scene focused as if captured by a pinhole camera.
   focus distance values for each pixel are estimated and stored in a focus map which resembles disparity or depth map of the scene. 
   This map is used to achieve a correct focusing of each pixel in the novel view synthesis.}
 \label{fig:perPixelTeaser}
\end{figure}

Compared to previous methods~\cite{anisimov2019, brox2004high, shi2015light, vagh2018light, sugita2004focus, lizhang2015high}, the proposed algorithm is designed to optimally utilize GPU for real-time rendering scenarios and also outperforms the methods in the resulting visual quality.
It works well with sparse, dense, and high-resolution light field datasets with no excessive hardware requirements.
As proved in the experiments, the method also reaches better quality than methods in the same category and even reaches similar quality levels as deep learning approaches.
Deep learning usually depends on the long and expensive training process, and the rendering, using the pre-trained model, is not reaching the times and memory requirements suitable for real-time usage with high-resolution data.

The proposed method is lightweight in all aspects.
No additional depth information is needed as the input, which also saves memory and allows for the usage of the input images without preprocessing.
One of the notable advantages of the proposed approach is its independence from the scene content.
The computation time is constant and depends only on the desired resolution of the resulting view and quality settings.
This method can be used, for example, to render scenes behind windows or in a far distance in computer games where the player's movement is limited by the environment.
It can also be used to provide a real-time preview of light field video, for example, on a film set, when light field capturing technology is used.

\subsection{Per-Pixel Scanning}
\label{sec:perpi}
The key part of the method is the generation of the focus map.
It uses principles similar to stereo disparity estimation methods~\cite{hirsch2008} but compares colors from multiple images of the input light field grid.
For each pixel, the method iterates over a focus distance range, interpolates the sampled colors from the input images at the given focus distance, computes the variance of the colors used, and stores the focus distance with the lowest color variance.
The weighted shift-sum algorithm~\cite{alain2019} is used for color interpolation, where the weights are computed by a linear function of the distance of the sampled image from the position of the virtual novel view in the grid.
The focus map then contains for each pixel its lowest-variance focus distance. 
The variance is computed based on the Chebyshev distance between the pixel values.
Generally, the choice of the color distance metric for a given task is problematic~\cite{mahama2016choice}.
Chebyshev was experimentally proven to be the most suitable metric in this case.
The whole process is described in Alg.~\ref{alg:focusMapEstim}.

\begin{algorithm}[!htbp]
 \KwData{Position of virtual $camera$, $grid$ of images, focus $range$}
 \KwResult{Focus $map$, novel $image$}
 \ForEach {$coordinate \oper{\in} image$}{
        $variances \oper{=} \fce{emptyArray}()$\;
        $colors \oper{=} \fce{emptyArray}()$\;
      \ForEach {$focusDistance \oper{\in} range$}{
            $newPixel \oper{=} \fce{shiftSum}(camera, focusDistance, coordinate, grid)$\;
            $variances \operN{[}focusDistance\operN{]} \oper{=} newPixel \xrightarrow{} variance$\;
            $colors\operN{[}focusDistance\operN{]} \oper{=} newPixel \xrightarrow{} color$\;
      }
      $optimalDistance \oper{=} \fce{indexOfMinimum}(variances)$\;
      $map\operN{[}coordinate\operN{]} = optimalDistance$\; 
      $image\operN{[}coordinate\operN{]} \oper{=} colors\operN{[}optimalDistance\operN{]}$\;
  }\
 \caption{
Function $\fce{shiftSum}()$ uses the shift-sum algorithm and returns the final color and variance of the colors contributing in the summation.
This algorithm is generalized, returning also the focused pixel color.
However, image synthesis is separated in the reference implementation.}
 \label{alg:focusMapEstim}
\end{algorithm}

In terms of 3D cost volume~\cite{xiaodong2020}, the focus distances are the hypothesis planes, the spatial resolution of the focus map is the spatial resolution of the cost volume, and the cost function is the variance of the colors in the given cell.
When the focus map is ready, the shift-sum algorithm is used again to synthesize the final image, where each pixel is interpolated at the previously computed optimal focus distance.
The focus map computation and rendering of the novel view are separated in two stages so that optimizations such as focus map subsampling can be used.
The example of both the final image and the focus map is shown in Fig.~\ref{fig:mapAndResult}.

\begin{figure}[!htbp]\centering
  \subfloat[\centering A view is rendered from light field.]{{\includegraphics[width=0.49\linewidth]{obrazky-figures/perPixel/result.jpg} }}
    \subfloat[\centering A focus map is used in the process.]{{\includegraphics[width=0.49\linewidth]{obrazky-figures/perPixel/map.jpg} }} \\
  \caption{
Focused result using the \textit{Pavilion} dataset with a correspondent focus map are shown side by side.
The focus map contains estimated focus distance for each pixel of the final image.
The map resembles depth or disparity map for the given synthetic view because the focus distances depend on the distance of the geometry one each pixel from the camera.}
  \label{fig:mapAndResult}
\end{figure}

The 3D effect is simulated by the weights assigned to the input images and the novel view camera coordinates that are used in the shift-sum algorithm to define the relative displacement of the input images.
The color vector of the output pixel $\mathbf{p}_{o}$ can be obtained by iterating over $n$ input images.
The function $\textrm{p}_{i}$ in Eq.~\ref{eq:shiftSumPerPixel} returns the pixel of the $i$th image at the given coordinates.
Each input color is weighted by $w_{i}$.
The output view coordinates $\mathbf{c}$ are used to shift the sampled images by the offset $\mathbf{o}_{i}$ between the images in the grid.
The amount of shifting is adjusted by the desired focus distance $f$.

\begin{equation}
	\mathbf{p}_{o} = \frac{\sum_{i=1}^{n}p_{i}(\mathbf{c}+\mathbf{o}_{i}\cdot f)\cdot w_{i}}{\sum_{i=1}^{n}w_{i}}
	\label{eq:shiftSumPerPixel}
\end{equation}

An iteration of the used shift-sum algorithm is depicted in Fig.~\ref{fig:shiftSumIteration}.

\begin{figure}[!htbp]\centering
  \includegraphics[width=\imageSize]{obrazky-figures/perPixel/shiftSumIteration.pdf}
  \caption{
The figure shows one iteration of the shift-sum based image synthesis where a pixel from $ith$ image (\textcolor{mainBlue}{blue} rectangle) is taken into the summation of the output pixel $\mathbf{p}_o$.
The \textcolor{mainOrange}{orange} rectangle depicts the new synthetic image.
The \textcolor{mainRed}{red} lines show the offsets relative to the currently sampled image and the distance between the two images.
The currently sampled pixel's weight $w_i$ depends on the distance between the two images.
The $x$ and $y$ superscript denotes the first and the second element of the vector variable.}
  \label{fig:shiftSumIteration}
\end{figure}

\section{Improvements of the Focusing Algorithm}
\label{sec:improvFocus}
Additional improvements in quality and performance, optimal scheme for light field video playback, and automatic detection of important parameters of the original method were further proposed and published~\cite{chlubna2023}.
\hyperref[hypothesis]{Hypothesis} claims that the proposed method achieves better visual quality than the state of the art.
The improvements described here mitigate the artifacts that can occur when the focus map contains inaccurately estimated focus values for some pixels.

Objects of one-pixel size are usually not present in ordinary scenes, and such pixels can be assumed to be focused at the same distance as their neighbors.
The first straightforward enhancement of the focus map is filtering.
Median filtering is used to improve the quality of the focus map, as shown in Fig.~\ref{fig:median}.
A GPU-optimized min-max median algorithm~\cite{salvador2018} is an efficient solution.

\begin{figure}[!htbp]\centering
\includegraphics[width=0.85\linewidth]{obrazky-figures/perPixel/median.pdf}
  \centering
   \caption{The noise in focus map resembles salt-and-pepper noise. 
   Median filtering can be used to get rid of incorrectly focused pixels.}
 \label{fig:median}
\end{figure}

\subsection{Variance Course Analysis}
False focus distance detection in the shift-sum algorithm is the source of visual quality degradation. 
Large segments filled with a single color, repetitive patterns, or coincidentally similar areas around one pixel can be problematic.
An analysis of the variance course for one pixel in the problematic area of the image (thin edge) can be seen in Fig.~\ref{fig:varianceSaddle}.
The global minimum does not correspond to the correct focus value. 
However, the desired detail causes a peak in the variance chart.
This steep change in the variance value, which is close to the minimal value, can indicate such details. 
After the variances are computed, a fast shuffle operation for interthread communication on GPU can be used to test each triplet of neighboring variances, detecting the difference that is above a certain threshold.
This scheme expects each GPU thread to evaluate one focus distance.
Prioritizing the focus distance, which is low enough and at the bottom of a saddle, reveals some of the previously distorted details in the image.
The method is described in Alg.~\ref{alg:saddle}.

\begin{figure}[!htbp]
\pgfplotstableread{charts/perPixel/variance1.txt}\varianceSaddle
\pgfplotstableread{charts/perPixel/variance1H.txt}\varianceSaddleCorrect
\begin{tikzpicture}
\begin{axis}[
  width=0.6\linewidth,
  ylabel={$Variance$},
  xlabel={$Focus\ distance$},
  legend style={at={(0.95,0.9)},anchor=east},
  legend entries={correct value}
]
\addplot[color=mainBlue, mark=o, ultra thick,forget plot] table[x=focus, y=variance]{\varianceSaddle};
\addplot[opacity=0.6, color=mainOrange, mark=*, mark size=5,only marks] table[x=focus, y=variance]{\varianceSaddleCorrect};
\end{axis}
\end{tikzpicture}
\centering
\includegraphics[width=0.4\linewidth]{obrazky-figures/perPixel/saddle.pdf}
\caption{\label{fig:varianceSaddle}
    The chart shows the course of the variance coming from the shift-sum for one pixel. 
    The global minimum value does not correspond to the best focus distance in this case. 
    \textcolor{mainOrange}{Orange} node indicates the correct distance. 
    The steep saddle is a thin detail, the frame of the window highlighted in the picture.}
\end{figure}

\begin{algorithm}[!htbp]
 \KwData{Array of $variances$ associated with focus distances, $sensitivity$}
 \KwResult{Edited array of variances $editedVariances$}
  $editedVariances \oper{=} \fce{emptyArray}()$\;
  $editedVariances \xleftarrow{\fce{insert}} \fce{firstElement}(variances)$\;
  $max \oper{=} \fce{maximalElement}(variances)$\;
  \ForEach {three consecutive variances $var_{previous, current, next} \oper{\in} variances$}{
    $var_{new} \oper{=} 0$\;
    \uIf{$var_{current} \oper{<} var_{prev} \:\mathbf{and}\: var_{current} \oper{<} var_{next}$}
     {
        $norm \oper{=} \frac{var_{current}}{max}$\;
        $prev \oper{=} \left |norm \oper{-} \frac{var_{prev}}{max} \right |$\;
        $next \oper{=} \left |norm \oper{-} \frac{var_{next}}{max}\right |$\;
        $saddle \oper{=} prev \oper{+} next$\;
        $var_{new} \oper{=} var_{current} \oper{-} saddle \oper{\cdot} sensitivity$\;
    }    
    \Else
    {
        $var_{new} \oper{=} var_{current}$\;
    }
    $editedVariances \xleftarrow{\fce{insert}} var_{new}$\;
   }
   $editedVariances \xleftarrow{\fce{insert}} \fce{lastElement}(variances)$\;
 \caption{Saddle detection algorithm that prioritizes local minimum over the global one in the array of color variances during the focus map generation phase. 
 The bigger the difference between the minimum and its neighbors, the more the minimum shifts to outperform the global one.}
 \label{alg:saddle}
\end{algorithm}

The saddle detection reveals problematic details, but it can lead to a false positive and cause artifacts in the image, especially near low-frequency areas. 
Many scenes contain various kinds of background, such as sky, walls, etc. 
Such backgrounds are bound to be in focus at the beginning or end of the focus range.
Sampled distances at the boundaries of the range can converge to very low variance values. 
If the values are small enough and their difference is below a certain threshold, it can be assumed that the tested pixel is in focus at the border of the range. 
Alg.~\ref{alg:border} describes this phase, and an example is shown in Fig.~\ref{fig:varianceBorder}.

\begin{algorithm}[!htbp]
 \KwData{Edited array of variances from saddle detection phase $editedVariances$, number of border elements to check $checkNum$, detection $threshold$, $tolerance$ marking the same variance level}
 \KwResult{Edited array of variances $editedVariances$ with new minimums}
 $index \oper{=} 0$\;
  \uIf{$\fce{firstElement}(editedVariances) \oper{>} \fce{lastElement}(editedVariances)$}{
        $index \oper{=} \fce{sizeOf}(editedVariances) \oper{-} checkNum$\;
    }
  $same \oper{=} false$\;    
   \uIf{$editedVariances\operN{[}index\operN{]} \oper{<} threshold$}{
      \ForEach {$\left \{  i \oper{\in} \mathbb{Z} \mid 0 \oper{\leq} i \oper{<} checkNum \oper{-} 1 \right \}$}{
        $j \oper{=} index \oper{+} i$\;
        $diff \oper{=} \left | editedVariances\operN{[}j\operN{]} \oper{-} editedVariances\operN{[}j\oper{+}1\operN{]} \right |$\;
        \uIf{$diff \oper{>} tolerance$}{
            $same \oper{=} false$\;
            \textbf{break}\;
            }
        }
        \uIf{same}{
            $editedVariances\operN{[}index \operN{]} \oper{=} \operN{-}1$\;
        }
    }
 \caption{The variances are adjusted by prioritizing the values at the borders of the course which are low enough and surrounded by same or similar values.
 Such cases occur, for example, when a distant background is present that stays in focus for a number of distances at the end of the range.}
 \label{alg:border}
\end{algorithm}

\begin{figure}[!htbp]
\pgfplotstableread{charts/perPixel/variance2.txt}\varianceBorder
\pgfplotstableread{charts/perPixel/variance2H.txt}\varianceBorderCorrect
\begin{tikzpicture}
\begin{axis}[
  width=0.6\linewidth,
  ylabel={$Variance$},
  xlabel={$Focus\ distance$},
  legend style={at={(0.98,0.9)},anchor=east},
  legend entries={correct value}
]
\addplot[color=mainOrange, mark=o, ultra thick,forget plot] table[x=focus, y=variance]{\varianceBorder};
\addplot[opacity=0.45, color=mainBlue, mark=*, mark size=5, only marks] table[x=focus, y=variance]{\varianceBorderCorrect};
\end{axis}
\end{tikzpicture}
\centering
\includegraphics[width=0.4\linewidth]{obrazky-figures/perPixel/border.pdf}
\caption{\label{fig:varianceBorder}
    The chart is showing the course of the variance coming from the shift-sum for one pixel. 
    The saddle point in this case causes incorrect focus as shown on the picture. 
    The \textcolor{mainBlue}{blue} nodes are correct.}
\end{figure}

\subsection{Pixel Sampling}
Instead of sampling one pixel per scan from the transformed coordinates, sampling multiple pixels at a certain distance can lead to better results.
It is based on the assumption that pixels that are adjacent or very close neighbors belong to the same object in the scene.
This block matching resembles matching cost computation and aggregation used in stereo disparity estimation methods~\cite{scharstein2002taxonomy}.
However, increasing the number of texture-reading operations slows down the computation.
The pixels can be sampled using various patterns.
Fig.~\ref{fig:sampling} shows the best patterns chosen according to the performance results. 

\begin{figure}[!htbp]
\includegraphics[width=\imageSize]{obrazky-figures/perPixel/sampling.pdf}
  \centering
   \caption{The figure depicts three tested texture sampling patterns: a) is a full 3x3 window, b) exploits the texture interpolation units in GPU, getting colors from coordinates between four pixels (similar to Kawase blur~\cite{kawase}), c) alternates over modulo, using four three-read patterns.
   Note that these patterns were also used in 2 px radius version.}
 \label{fig:sampling}
\end{figure}

\subsection{Focus Range Detection}
Predefined focus range can be inconvenient for dynamic scenes with a variable depth range. 
The focus range might also not always be known for the given light field.
However, the range is an important parameter for the scan during focus map generation.
Automatic fast detection of the start and the end of the range is necessary.
An extra wide range is defined to ensure coverage of all standard datasets.
Then this range shrinks into a tighter one.
The image is divided into $16{\times}16$ pixel blocks.
Variance is computed with the shift-sum algorithm in the same way as in the focus map generation. 
The variances are averaged through the entire block at each distance. 
The minimal average variance is then chosen as optimal.
The minimum and maximum of these minimal averages from all blocks are selected as the final focus range.

Only four nearest central images are used in this computation.
Using images that are far from each other, such as corner images of the whole grid, yields inaccurate results because of the reduced similarity between them. 
The local minimum variance of each $16{\times}16$ block is stored in an array, and then the global minimum and maximum are found; see Alg.~\ref{alg:range}.
The array of local minimum variances is statistically filtered using the standard score method to remove outliers. 

\begin{algorithm}[!htbp]
 \small
 \KwData{Four neighbouring input $images$, scanning $distance$, size of one scanning $step$}
 \KwResult{narrowed focus $range$}
 $coef \oper{=} \frac{1}{distance^2}$\;
 $range \oper{=} \fce{2DVector}(distance, \operN{-}distance)$\;
 \ForEach {$16{\times}16$ $block \oper{\in} result$}{
    $bestFocus \oper{=} 0$\;
    $minVariance \oper{=} \fce{maximalValue}()$\;
     \ForEach {$fi$ \textbf{from} $\operN{-}distance$ \textbf{to} $distance$ \textbf{with} $step$}{
        $f \oper{=} coef \oper{\cdot} fi^3$\;
        $variance \oper{=} 0$\;
        \ForEach {pixel$\oper{\in} block$}{
            $colors \oper{=} \fce{emptyArrayOfColors}()$\;
            $avgCol \oper{=} \fce{emptyColor}()$\;
            \ForEach {$\left \{  i \oper{\in} \mathbb{Z} \mid 0 \oper{\leq} i \oper{<} 4 \right \}$}{
                $colors\operN{[}i\operN{]} \oper{=} \fce{sample}(images\operN{[}i\operN{]}, pixel_{xy}, f)$\;
                $avgCol$ \oper{=} $avgCol\oper{+}colors\operN{[}i\operN{]}$;
            }
            $avgCol \oper{=} \frac{avgCol}{4}$\;
            \ForEach {$\left \{  i \oper{\in} \mathbb{Z} \mid 0 \oper{\leq} i \oper{<} 4 \right \}$}{
                $variance$ \oper{=} $variance\oper{+}\fce{distance}(avgCol, colors\operN{[}i\operN{]})$\;
            }
        }
         \uIf{$variance \oper{<} minVariance$}{
            $minVariance \oper{=} variance$\;
            $bestFocus \oper{=} f$\;
        }
     }
     $range \oper{=} \fce{minMax}(bestFocus, range_{max}$)\;
     $range_{min} \oper{=} \fce{min}(bestFocus, range_{min}$)\;    
 }
 \caption{Automatic range detection iterates over a wide focus range. 
 The new narrow range is constructed using focus distances with minimal variance of each $16{\times}16$ block of pixels.}
 \label{alg:range}
\end{algorithm}

Experiments showed that non-linear focusing scan with a constant step is more robust for this method. 
The desired focus range is always defined around the zero focus distance.
Therefore, this area is searched more densely according to Eq.~\ref{eq:exponentialA}, \ref{eq:exponentialB}, and \ref{eq:exponentialC}.
This is achieved by transformation of the linear range, using a third degree polynomial, with only the highest degree coefficient set as non-zero as follows:

\begin{equation}
f(x)=ax^3.
	\label{eq:exponentialA}
\end{equation}

The transformation of the linear value into exponential space is calculated as:

\begin{equation}
f_{e} = Af_{l}^3.
	\label{eq:exponentialB}
\end{equation}

The coefficient $A$ is calculated using the upper limit of the range as:

\begin{equation}
A=\frac{1}{f_{max}^2},
	\label{eq:exponentialC}
\end{equation}

where $f_{l}$ is the focus distance in the linear range, $f_{e}$ is the focus distance in the exponential range, and $f_{max}$ is the upper limit of the range.

\subsection{Depth of Field}
The focus map can be used to generate post-processing effects, such as the depth of field.
The defocused area is distorted by sharp ghosting artifacts in many light field applications~\cite{levoy1996light}.
The artifacts can be mitigated with a blur filter that smoothens the defocused parts. 
In the reference implementation, a two-pass separable linear blur was used, where the amount of blur linearly decreases as the focus distance is closer to the chosen target distance; see Alg.~\ref{alg:dof}.

\begin{algorithm}[!htbp]
 \small
 \KwData{All-focused light field rendered $image$; focus $map$, DOF $range$, DOF $distance$, size of the blur kernel $kernelSize$, coordinate-space size of the half of the pixel $halfPixel$}
 \KwResult{Rendered $image$ with depth of field effect}
 $focus \oper{=} \fce{round}(distance \oper{\cdot} totalLevels)$\;
 \For {$pass \oper{\in} \left\{x, y\right\}$}{
    \ForEach {$pixel \oper{\in} image$}{
        $coord \oper{=} pixel\xrightarrow{}coord$\;
        $dist \oper{=} \left |focus\oper{-}\fce{sample}(map, coord) \right |$\;
        $kernelSize \oper{=} \fce{round}(dist \oper{\cdot} range)$\;
        $w \oper{=} kernelSize\oper{+}1$\tcp*{weight}
        $color \oper{=} \fce{sample}(image, coord)\oper{\cdot} w$\;
        \ForEach {$\left \{  i \oper{\in} \mathbb{Z} \mid 0 \oper{\leq} i \oper{<} kernelSize \right \}$}{
            $o \oper{=} \fce{2DVector}(0,0)$\tcp*{offset}
            $o\operN{[}pass\operN{]} \oper{=} (3 \oper{+} 4 \oper{\cdot} i) \oper{\cdot} halfPixel$\;
            $w \oper{=} kernelSize\oper{-}i$\;
            $color \oper{=} color\oper{+}\fce{sample}(image, coord \oper{+} o) \oper{\cdot} w$\;
            $color \oper{=} color\oper{+}\fce{sample}(image, coord \oper{-} o) \oper{\cdot} w$\;
        }
        $color$ \oper{=} $\frac{color}{(kernelSize\oper{+}1)^2}$\;
        $outImage\operN{[}coord\operN{]} \oper{=} color$\;
    }
    $\fce{switchImages}(image, outImage)$\;
 }
 \caption{Simulation of the depth-of-field effect on the rendered light field image uses the computed focus map. 
 Half-pixel offsets are used to sample pixels for two-pass separable linear blur, exploiting the interpolation of texturing units in GPU.}
 \label{alg:dof}
\end{algorithm}

The results in Fig.~\ref{fig:dof} show that this method can be extended to support multiple focus distances. 
The proposed improvements to the focus map reduce the number of artifacts in this effect; see Fig.~\ref{fig:dofArtifactacts}.

\begin{figure}[!htbp]\centering
\includegraphics[width=\textwidth]{obrazky-figures/perPixel/dof.pdf}
  \centering
   \caption{Focus map was used to simulate the depth of field effect, applying box blur on the defocused parts of the scene.
   A comparison with the original shift-sum algorithm is shown in the figure.
   Box-like artifacts are visible without the proposed improvements.}
 \label{fig:dof}
\end{figure}

\begin{figure}[!htbp]\centering
\includegraphics[width=\textwidth]{obrazky-figures/perPixel/dofArtifacts.pdf}
  \centering
   \caption{The left image shows a part of the depth-of-field effect in \textit{Pavilion} scene using the focus map with the proposed improvements.
   The effect in the right image uses the focus map generated without the proposed improvements and contains more visible artifacts. 
   The rightmost image is a difference between the top images.}
 \label{fig:dofArtifactacts}
\end{figure}

\subsection{GPU Utilization}
The method is designed to exploit massive parallelism available on GPU architectures, which is important for the desired real-time usage.
The focus map generation is performed in a GPGPU compute shader or kernel.
Each warp (32 threads on NVIDIA cards) is assigned to one pixel.
Each workgroup consists of 8 neighboring pixels.
This scheme offers a good GPU occupancy and memory access coherency, allowing an in-warp data transfer between the threads, which is much faster than using the global or local memory.
Each thread computes one focus distance (or more when denser search is required), using the weighted shift-sum and the Welford's variance algorithm~\cite{welford1962note}; see Alg.~\ref{alg:onlineVariance}.

\begin{algorithm}[!htbp]
 \KwData{Stream of $input$ pixel values}
 \KwResult{Estimated variance $m_2$}
 $n \oper{=} 0$\;
 $mean \oper{=} 0$\;
 $m_2 \oper{=} 0$\;
 \ForEach {$pixel \oper{\in} input$}{
  $n \oper{=} n \oper{+} 1$\;
  $delta \oper{=} pixel\oper{-}mean$\;
  $distance \oper{=} \fce{pixelDistance}(pixel, mean)$\;
  $mean \oper{=} mean\oper{+}delta\oper{/}n$\;
  $m_2 \oper{=} m_2\oper{+}distance \oper{\cdot} \fce{pixelDistance}(pixel, mean)$\;
  }
 $m_2 \oper{=} m_2\oper{/}(n\oper{-}1)$\;\
 \caption{
Welford's method for computing online variance in one pass, adjusted to pixel values (RGB colors in reference implementation) comparison purpose.
This algorithm is used in the shift-sum, analyzing new color values coming into the summation.}
  \label{alg:onlineVariance}
\end{algorithm}

This way of variance computation improves the GPU occupancy by reducing the necessary number of registers.
In the end, the minimal variance value within a warp is found using parallel reduction with ballot operation.
In the fragment shader, a surface representing the light field is rendered.
The focus map and the input images are stored as textures.
Therefore, the missing pixels can be interpolated in texturing units if the resolutions of the result and the focus map differ.
Fig.~\ref{fig:gpuScheme} describes the work distribution on GPU. 

\begin{figure}[!htbp]\centering
  \includegraphics[width=\imageSize]{obrazky-figures/perPixel/gpuScheme.pdf}
  \caption{
Work distribution on the GPU for focus map generation.
The compute shader analyses the input images, going through the focus range and saving the focus distance with a minimal variance in the focus map.
Because the workload is divided into warp-sized elements, no global or local synchronization, that would slow down the computation, is needed.}
  \label{fig:gpuScheme}
\end{figure}

\subsubsection{Video Streaming}
Classic datasets~\cite{vaish} contain frames in the $8{\times}8$ grids.
Current video standards expect a resolution of at least $1920{\times}1080$ (FulllHD), slowly rising to $3840{\times}2160$ (4K).
This leads to almost 10 GB/s when streaming 25 fps FullHD 8-bit RGB video.
Such a transfer from RAM memory via PCI-Express into GPU memory is hardly achievable in real time. 

Video compression was shown to be an optimal solution for light field data~\cite{barina2022}.
It was discovered that compression artifacts are mitigated by light field rendering.
The same fact was proven for downscaling with H.265 compression; see Fig.~\ref{fig:downscale}.
PSNR difference is higher when comparing the raw data frame-by-frame than when comparing synthetic views produced by the light field rendering method.
SSIM seems to be a bit worse in the light field case, but the difference is very small.

\begin{figure}[!htbp]\centering
\centering
\pgfplotsset{try min ticks=5}
\pgfplotsset{width=0.95\linewidth, height=3.5cm, every axis plot post/.append style={ultra thick, draw=none, mark=none}}
\pgfplotstableread{charts/perPixel/downscalePSNR.txt}\downscalePSNR
\pgfplotstableread{charts/perPixel/downscaleSSIM.txt}\downscaleSSIM
\begin{tikzpicture}
\begin{axis}[xbar,yticklabels from table={\downscalePSNR}{method},
  ytick=data,
  reverse legend,
  xlabel={$Error\ [\%]$},
  ylabel={$Format$},
  legend style={at={(0.9,0.8)},anchor=east},
  legend entries={PSNR, SSIM},
  bar width=1cm,
  enlarge y limits={abs=0.5cm},
  area legend
]
\addplot[color=white, fill=mainBlue] table[x expr={\thisrow{avgErr}*100},y expr=\coordindex]{\downscalePSNR};
\addplot[color=white, fill=mainOrange] table[x expr={\thisrow{avgErr}*100},y expr=\coordindex]{\downscaleSSIM};
\end{axis}
\end{tikzpicture}
\caption{\label{fig:downscale}
An average error, over all compression levels defined by H.265 CRF parameter, of PSNR and SSIM comparison of downscaled (HD) and original (FullHD) light field dataset.
The downscaled dataset was compared to the original frame-by-frame as a video and as a light field, comparing the new synthetic views, produced using the original and downscaled input.}
\end{figure}

Video packets can be uploaded asynchronously, decompressed on GPU, and converted\footnote{GL\_NV\_vdpau\_interop extension for OpenGL can be used.} to textures.
The scheme in Fig.~\ref{fig:gpuDecoding} describes the approach used in the experimental implementation.

\begin{figure}[!htbp]\centering
\includegraphics[width=\imageSize]{obrazky-figures/perPixel/gpuDecoding.pdf}
  \centering
   \caption{Light field images are compressed in a video stream which is transferred asynchronously to the GPU.
   Video packets are decoded on GPU, using vdpau, and converted into OpenGL textures.
   Textures can be then used for rendering, using classic rendering pipeline and compute shaders.}
 \label{fig:gpuDecoding}
\end{figure}

\section{Experimental Evaluation with Standard Datasets}
\label{sec:evalStand}
The proposed method was evaluated using an experimental implementation.
Datasets used in the experiments captured with camera array come from Stanford light field archives~\cite{vaish}, light field captured by the plenoptic camera Lytro Illum belongs to EPFL light field dataset~\cite{rerabek2016new}, and synthetic dataset was rendered on \textit{Barcelona Pavilion} scene, which is available at the Blender demo files page; see Fig.~\ref{fig:perPixelDataset}.
Only one Lytro dataset was used because the distance between Lytro views is very small and cannot be used to create an appreciable 3D effect.
In all experiments, a ground truth center view from the original dataset was chosen as a reference and it was compared using SSIM and PSNR metrics to a new synthetic view rendered by the proposed method.
All experiments were executed on a machine equipped with NVIDIA\textsuperscript{\textregistered} GeForce RTX\textsuperscript{\texttrademark} 2070 GPU and Intel\textsuperscript{\textregistered} Core\textsuperscript{\texttrademark} i5-8500 CPU 3.00 GHz CPU, running Arch Linux.

\begin{figure}[!htbp]\centering
    \def \datasetFrameSize {0.3\linewidth}
    \subfloat[\centering \textit{Bunny} contains a simple object.]{{\includegraphics[width=\datasetFrameSize]{obrazky-figures/perPixel/datasetBunny.jpg} }}
    \subfloat[\centering \textit{Lego} contains small details.]{{\includegraphics[width=\datasetFrameSize]{obrazky-figures/perPixel/datasetLego.jpg} }}
    \subfloat[\centering \textit{Buldozer} contains one complex object.]{{\includegraphics[width=\datasetFrameSize]{obrazky-figures/perPixel/datasetBuldo.jpg} }}
    \\
    \subfloat[\centering \textit{Chess} contains reflective materials.]{{\includegraphics[width=\datasetFrameSize]{obrazky-figures/perPixel/datasetChess.jpg} }}
    \subfloat[\centering \textit{Pavilion} scene is a complex outdoor scene.]{{\includegraphics[width=\datasetFrameSize]{obrazky-figures/perPixel/datasetPavilion.jpg} }}
    \subfloat[\centering \textit{Lytro} is an outdoor scene taken by a plenoptic camera.]{{\includegraphics[width=\datasetFrameSize]{obrazky-figures/perPixel/datasetLytro.jpg} }}
  \caption{The images are reference views from each dataset used in the experiments.}
  \label{fig:perPixelDataset}
\end{figure}

\subsection{Color Distance Metric}
The variance computation phase in the proposed algorithm requires a pixel color value distance metric to decide how much two pixels differ in terms of color similarity.
The right choice of the metric depends on various aspects, such as the expected color range, the type of images, or a final use case. 
Various color distance metrics were compared to find out which one would yield the best visual quality results for light field datasets; see Fig.~\ref{fig:colorMetric}.
The quality differences were not significant, but the computational complexity of the metrics differed and could negatively affect performance. 
Chebyshew metric was chosen for further experiments because of high-quality results and computational simplicity. 

\begin{figure}[!htbp]
\centering
\pgfplotsset{width=0.42\linewidth, every axis plot post/.append style={ultra thick, draw=none, mark=none}}
\pgfplotstablesort[sort key={average}]{\colorMetricData}{charts/perPixel/colorPSNR.txt}
\begin{tikzpicture}
\begin{axis}[xbar,yticklabels from table={\colorMetricData}{metric},
  ytick=data,
  xlabel={$PSNR\ [dB]$},
  ylabel={$Metric$}
]
\addplot[color=gray, fill=mainBlue] table[x=average,y expr=\coordindex]{\colorMetricData};
\end{axis}
\end{tikzpicture}
\pgfplotstablesort[sort key={average}]{\colorMetricDataS}{charts/perPixel/colorSSIM.txt}
\begin{tikzpicture}
\begin{axis}[xbar,yticklabels from table={\colorMetricDataS}{metric},
  ytick=data,
  xlabel={$SSIM$},
  %ylabel={$Metric$},
  xticklabel style={/pgf/number format/.cd,fixed,fixed zerofill,precision=3}
]
\addplot[color=gray, fill=mainOrange] table[x=average,y expr=\coordindex]{\colorMetricDataS};
\end{axis}
\end{tikzpicture}
\caption{\label{fig:colorMetric}
A comparison of the RGB color distance metrics for the pixel similarity test during the variance computation phase.
The $W$ suffix in the metric name stands for weighted metrics.
The average results of all the datasets tested are presented.}
\end{figure}

\subsection{Base Quality}
For each dataset, the best focus range and search step were manually found and the resulting images were compared to the reference.
The final visual quality is evaluated in Fig.~\ref{fig:best}. 
The images are focused in all parts, but interpolation artifacts are visible around thin edges or near similarly colored areas.

\begin{figure}[!htbp]
\pgfplotsset{height=0.28\textwidth}
\centering
\pgfplotsset{width=0.45\linewidth, every axis plot post/.append style={ultra thick, draw=none, mark=none}}
\pgfplotstablesort[sort key={psnr}]{\bestData}{charts/perPixel/best.txt}
\begin{tikzpicture}
\begin{axis}[xbar,yticklabels from table={\bestData}{dataset},
  ytick=data,
   y tick label style={font=\itshape},
  xlabel={$PSNR\ [dB]$},
  ylabel={$Dataset$}
]
\addplot[color=gray, fill=mainBlue] table[x=psnr,y expr=\coordindex]{\bestData};
\end{axis}
\end{tikzpicture}
\pgfplotstablesort[sort key={ssim}]{\bestDataS}{charts/perPixel/best.txt}
\begin{tikzpicture}
\begin{axis}[xbar,yticklabels from table={\bestDataS}{dataset},
  ytick=data,
   y tick label style={font=\itshape},
  xlabel={$SSIM$},
  %ylabel={$Dataset$},
  xticklabel style={/pgf/number format/.cd,fixed,fixed zerofill,precision=3}
]
\addplot[color=gray, fill=mainOrange] table[x=ssim,y expr=\coordindex]{\bestDataS};
\end{axis}
\end{tikzpicture}
\caption{\label{fig:best}
Best results are presented of rendering a new view from each dataset compared to the ground truth.}
\end{figure}

A detailed look at the interpolation artifacts is captured in Fig.~\ref{fig:artifacts}. 
\textit{Bunny} dataset contains only diffuse material and is clearly separated from the black background.
Therefore, the reconstruction had minimum artifacts.
Although the \textit{Chess} dataset contains a lot of reflections, the chessboard pattern and a relatively small distance between the views improved the quality of the result.
\textit{Buldozer} contains a lot of small details that are clearly separated from the yellow construction of the model, which causes higher variance values when mixing nearby pixels.
\textit{Lego} dataset is filled with a single-color area where, for example, the thin edges or details are hard to detect, and the pixels interpolated from the surrounding area might yield lower variance.
The distance between \textit{Lytro} cameras is small, so the result was expected to be better.
However, due to the technical drawbacks of the camera, the input images contain subtle noise that negatively affects the evaluation.
\textit{Pavilion} contains both similar colored areas and complex objects with many details, but the distance between cameras is large.
Fig.~\ref{fig:time} shows the elapsed time of focus map generation and the final composition of pixels from each dataset.

\begin{figure}[!htbp]
    \def \frameSize {0.25\linewidth}
  \centering
    \subfloat[\centering Reference image is taken from the input light field.]{{\includegraphics[width=\frameSize]{obrazky-figures/perPixel/legoRef.jpg} }}
    \subfloat[\centering Rendered view shows missing details.]{{\includegraphics[width=\frameSize]{obrazky-figures/perPixel/legoRend.jpg} }}
    \subfloat[\centering Zoomed reference contains the edges of the bricks.]{{\includegraphics[width=\frameSize]{obrazky-figures/perPixel/legoZoomRef.jpg} }}
    \subfloat[\centering Zoomed render shows vanishing details.]{{\includegraphics[width=\frameSize]{obrazky-figures/perPixel/legoZoomRend.jpg} }}
  \caption{\label{fig:artifacts}
Reference images are placed in the left column.
Right column contains rendered reconstructed images with zoomed detail of interpolation artifacts caused by incorrect focus distance estimation in the affected pixels.}
\end{figure}

\begin{figure}[!htbp]
\pgfplotsset{every axis plot post/.append style={ultra thick, draw=none, mark=none}}
\pgfplotsset{width=.9\textwidth, height=.6\textwidth}
\pgfplotstableread{charts/perPixel/timeCs.txt}\timeDataCs
\pgfplotstableread{charts/perPixel/timeDr.txt}\timeDataDr
\begin{tikzpicture}
\pgfplotsset{xbar stacked,yticklabels from table={\timeDataCs}{dataset},
  yticklabel style={align=right},
  ytick=data,
   y tick label style={font=\itshape},
  xlabel={$Elapsed\ time\ [ms]$},
  ylabel={$Dataset$}, 
  xmax=20,
  xmin=2}
\begin{axis}[bar shift=10pt]
\addplot[color=gray, fill=mainBlue] table[x=0,y expr=\coordindex]{\timeDataCs};
\addplot[color=gray, fill=mainLightBlue] table[x=0,y expr=\coordindex]{\timeDataDr};
\end{axis}
\pgfplotsset{ticks=none, xlabel={}, ylabel={}}
\begin{axis}[hide axis]
\addplot[color=gray, fill=mainRed] table[x=4,y expr=\coordindex]{\timeDataCs};
\addplot[color=gray, fill=mainLightRed] table[x=4,y expr=\coordindex]{\timeDataDr};
\end{axis}
\begin{axis}[bar shift=0-10pt, hide axis, legend entries={\textbf{Focus map},Full,1/4,1/8,\textbf{Drawing},Full,1/4,1/8}, legend style={at={(1,0.718)},anchor=east, nodes={scale=0.9}}]
\addlegendimage{opacity=0.0}
\addlegendimage{fill=mainBlue, draw=white}
\addlegendimage{fill=mainRed, draw=white}
\addlegendimage{fill=mainOrange, draw=white}
\addlegendimage{opacity=0.0}
\addlegendimage{fill=mainLightBlue, draw=white}
\addlegendimage{fill=mainLightRed, draw=white}
\addlegendimage{fill=mainLightOrange, draw=white}
\addplot[color=gray, fill=mainOrange] table[x=8,y expr=\coordindex]{\timeDataCs};
\addplot[color=gray, fill=mainLightOrange] table[x=8,y expr=\coordindex]{\timeDataDr};
\end{axis}
\end{tikzpicture}
\caption{\label{fig:time}
Elapsed time of the focus map generation and drawing which depends on the focus map and resulting image resolution respectively.
Full, 1/4 and 1/8 sized focus map is used in these measurements.
The drawing time slightly increases when using a smaller focus map, most likely due to coordinates interpolation in texturing units due to resolution mismatch.}
\end{figure}

\subsection{Focus Map Resolution}
One of the key features of the proposed method is the option to separate the focus map generation from the interpolation of the final result.
Fig.~\ref{fig:subdiv} shows how the reduction of the focus map size affects the computation time and visual quality of the final image.

\begin{figure}[!htbp]
\pgfplotsset{height=6cm}
\pgfplotstableread{charts/perPixel/subdiv.txt}\subdivData
\begin{tikzpicture}
\begin{axis}[
  ylabel={$PSNR\ [dB]$},
  xtick={5,10,15,20},
  legend style={at={(0.98,0.09)},anchor=east}
]
\addlegendimage{legend image with textB=$x$}
        \addlegendentry{Focus map divisions}
\addplot[color=mainBlue, mark=o, ultra thick] table[x=time, y=psnr]{\subdivData}
node [pos=0.0, above,color=mainBlue]{$Avg$}
node [pos=0.0, below,color=mainLightBlue]{$1$}
node [pos=0.64, below,color=mainLightBlue]{$4$}
node [pos=0.83, below,color=mainLightBlue, yshift=0-1mm,  xshift=1mm]{$10$}
node [pos=1.0, below,color=mainLightBlue, xshift=0-3mm]{$50$};
\addplot[color=mainBlue, opacity=0.3, mark=o, ultra thick] table[x=time, y=minPsnr]{\subdivData} 
node [pos=0.0, above,color=mainBlue]{$Min$};
\addplot[color=mainBlue, opacity=0.3, mark=o, ultra thick] table[x=time, y=maxPsnr]{\subdivData}
node [pos=0.0, above,color=mainBlue]{$Max$};
\end{axis}
\end{tikzpicture}
\begin{tikzpicture}
\begin{axis}[
  ylabel={$SSIM$},
  legend style={at={(0.98,0.09)},anchor=east}
]
]
\addlegendimage{legend image with textR=$x$}
        \addlegendentry{Focus map divisions}
\addplot[color=mainOrange, mark=o, ultra thick] table[x=time, y=ssim]{\subdivData}
node [pos=0.0, above,color=mainOrange]{$Avg$}
node [pos=0.0, below,color=mainLightOrange, xshift=2mm, yshift=2mm]{$1$}
node [pos=0.73, below,color=mainLightOrange]{$4$}
node [pos=0.91, below,color=mainLightOrange, yshift=0-1mm]{$10$}
node [pos=1.0, below,color=mainLightOrange, xshift=0-1mm]{$50$};
\addplot[color=mainOrange, opacity=0.5, mark=o, ultra thick] table[x=time, y=minSsim]{\subdivData}
node [pos=0.0, above,color=mainOrange]{$Min$};
\addplot[color=mainOrange, opacity=0.5, mark=o, ultra thick] table[x=time, y=maxSsim]{\subdivData}
node [pos=0.0, above,color=mainOrange]{$Max$};
\end{axis}
\end{tikzpicture}
\\
\makebox[\linewidth]{\centering $Time\ [ms]$}
\caption{\label{fig:subdiv}
The chart shows the relation of visual quality, computation time, and amount of focus map dimensions division.
The results are averaged from all tested datasets.}
\end{figure}

Surprisingly, the quality does not decrease rapidly even with a significant focus map downscaling. 
In certain cases, the quality even improves because some areas with incorrect focus distances are smoothed due to filtering caused by resizing.
Too small map size can lead to the same focus distance on nearby objects that might not lie in the same distance, causing out-of-focus artifacts as shown in Fig.~\ref{fig:subdivImg}.

\begin{figure}[!htbp]
  \centering
  \begin{overpic}[width=.99\linewidth]{obrazky-figures/perPixel/subdiv.jpg}
  \color{white}
  \put(1,27){\textbf{Ground truth}}
  \put(1,12){\textbf{$2{\times}$ smaller}}
  \put(51,27){\textbf{$20{\times}$ smaller}}
  \put(51,12){\textbf{$50{\times}$ smaller}}
  \end{overpic}
  \caption{\label{fig:subdivImg}
Focusing artifacts caused by low resolution focus map.
The first image shows reference image with generated depth map.
The other ones are focus maps with their dimensions divided by 2, 20, and 50 with the final rendered results.}
\end{figure}

\subsection{Focus Range Density}
Increasing the number of search samples when iterating over the focus distances in a given range does not significantly help the visual quality and unnecessarily slows down the computation unnecessarily; see Fig.~\ref{fig:searchDivQuality}.
32 samples proved to be an optimal choice for most of the datasets.
The most significant difference in quality was measured on \textit{Pavilion} dataset which has the largest depth range.
That is the only case where a denser search is necessary.
A denser search is needed, especially when the objects in the scene are linearly distributed over the whole depth range.

\begin{figure}[!htbp]
\pgfplotsset{height=6cm}
\pgfplotstableread{charts/perPixel/searchDivision.txt}\seachQData
\begin{tikzpicture}
\begin{axis}[
  ylabel={$PSNR\ [dB]$},
  legend style={at={(0.98,0.2)},anchor=east}
]
\addlegendimage{legend image with textB=$x$}
        \addlegendentry{Number of search samples}
\addplot[color=mainBlue, mark=o, ultra thick] table[x=time, y=psnr]{\seachQData}
node [pos=0.0, above,color=mainBlue]{$Avg$}
node [pos=0.0, below,color=mainLightBlue]{$32$}
node [pos=0.45, below,color=mainLightBlue]{$128$}
node [pos=1.0, below,color=mainLightBlue]{$256$};
\addplot[color=mainBlue, opacity=0.3, mark=o, ultra thick] table[x=time, y=minPsnr]{\seachQData} 
node [pos=0.0, above,color=mainBlue]{$Min$};
\addplot[color=mainBlue, opacity=0.3, mark=o, ultra thick] table[x=time, y=maxPsnr]{\seachQData}
node [pos=0.0, above,color=mainBlue]{$Max$};
\end{axis}
\end{tikzpicture}
\begin{tikzpicture}
\begin{axis}[
  ylabel={$SSIM$},
  legend style={at={(0.98,0.2)},anchor=east}
]
\addlegendimage{legend image with textR=$x$}
        \addlegendentry{Number of search samples}
\addplot[color=mainOrange, mark=o, ultra thick] table[x=time, y=ssim]{\seachQData}
node [pos=0.0, above,color=mainOrange]{$Avg$}
node [pos=0.0, below,color=mainLightOrange]{$32$}
node [pos=0.45, below,color=mainLightOrange]{$128$}
node [pos=1.0, below,color=mainLightOrange]{$256$};
\addplot[color=mainOrange, opacity=0.5, mark=o, ultra thick] table[x=time, y=minSsim]{\seachQData}
node [pos=0.0, above,color=mainOrange]{$Min$};
\addplot[color=mainOrange, opacity=0.5, mark=o, ultra thick] table[x=time, y=maxSsim]{\seachQData}
node [pos=0.0, above,color=mainOrange]{$Max$};
\end{axis}
\end{tikzpicture}
\\
\makebox[\linewidth]{\centering $Time\ [ms]$}
\caption{\label{fig:searchDivQuality}
Effect of focus range scan density on visual quality is shown.}
\end{figure}

\subsection{Camera Grid Sampling Distance}
The experimental results in Fig.~\ref{fig:sampleDistance} show how many images need to be sampled for the resulting pixel sum.
The sampling window in the input grid gives optimal results when having a radius about 2 grid views wide.
The wider radius leads to more texture reads and excessive memory access, which slows down the computation most.
Surrounding images from the grid in distance from zero to the sampling distance radius are taken into account during the interpolation.
When the radius is too wide, images from distant places in the grid might add unwanted ghosting artifacts in the final result.
The images capture the scene from a different angle than expected.

\begin{figure}[!htbp]
\pgfplotsset{height=5.45cm}
\pgfplotstableread{charts/perPixel/sampleDistance.txt}\seachDistance
\begin{tikzpicture}
\begin{axis}[
  ylabel={$PSNR\ [dB]$},
    legend style={at={(0.98,0.25)},anchor=east}
]
\addlegendimage{legend image with textB=$x$}
        \addlegendentry{Max sample distance}
]
\addplot[color=mainBlue, mark=o, ultra thick] table[x=time, y=psnr]{\seachDistance}
node [pos=0.0, above,color=mainBlue]{$Avg$}
node [pos=0.0, below,color=mainLightBlue]{$1.5$}
node [pos=0.4, below,color=mainLightBlue]{$3$}
node [pos=1.0, below,color=mainLightBlue]{$4.5$};
\addplot[color=mainBlue, opacity=0.3, mark=o, ultra thick] table[x=time, y=minPsnr]{\seachDistance} 
node [pos=0.0, above,color=mainBlue]{$Min$};
\addplot[color=mainBlue, opacity=0.3, mark=o, ultra thick] table[x=time, y=maxPsnr]{\seachDistance}
node [pos=0.0, above,color=mainBlue]{$Max$};
\end{axis}
\end{tikzpicture}
\begin{tikzpicture}
\begin{axis}[
  ylabel={$SSIM$},
    legend style={at={(0.98,0.15)},anchor=east}
]
\addlegendimage{legend image with textR=$x$}
        \addlegendentry{Max sample distance}
]
\addplot[color=mainOrange, mark=o, ultra thick] table[x=time, y=ssim]{\seachDistance}
node [pos=0.0, above,color=mainOrange, xshift=0-2mm]{$Avg$}
node [pos=0.0, below,color=mainLightOrange]{$1.5$}
node [pos=0.4, below,color=mainLightOrange]{$3$}
node [pos=1.0, below,color=mainLightOrange]{$4.5$};
\addplot[color=mainOrange, opacity=0.5, mark=o, ultra thick] table[x=time, y=minSsim]{\seachDistance}
node [pos=0.0, above,color=mainOrange, xshift=0-1.5mm, yshift=7mm]{$Min$};
\addplot[color=mainOrange, opacity=0.5, mark=o, ultra thick] table[x=time, y=maxSsim]{\seachDistance}
node [pos=0.0, above,color=mainOrange, xshift=0mm, yshift=2mm]{$Max$};
\end{axis}
\end{tikzpicture}
\\
\makebox[\linewidth]{\centering $Time\ [ms]$}
\caption{\label{fig:sampleDistance}
Effect of maximal sample distance to quality and computation time is shown.}
\end{figure}

\subsection{Camera Grid Parameters}
The \textit{Pavilion} dataset was used to measure the relation between the visual quality of the reconstructed view and the distance between cameras with various focal lengths.
The distance between cameras, field of view, total depth range in the scene, and position of the camera grid in the scene affect the quality of the reconstruction; see Fig.~\ref{fig:fov}.

\begin{figure}[!htbp]
\pgfplotsset{height=6.5cm}
\pgfplotstableread{charts/perPixel/fov.txt}\fovData
\begin{tikzpicture}
\begin{axis}[
  ylabel={$PSNR\ [dB]$},
  legend entries={\textbf{Focal length},25 mm, 35 mm, 45 mm, 55 mm}
]
\addlegendimage{opacity=0.0}
\addlegendimage{color=mainRed, mark=o, ultra thick}
\addlegendimage{color=mainLightRed, mark=o, ultra thick}
\addlegendimage{color=mainBlue, mark=o, ultra thick}
\addlegendimage{color=mainLightBlue, mark=o, ultra thick}

\addplot[color=mainRed, mark=o, ultra thick] table[x=scale, y=25mm_psnr]{\fovData};
\addplot[color=mainLightRed, mark=o, ultra thick] table[x=scale, y=35mm_psnr]{\fovData};
\addplot[color=mainBlue, mark=o, ultra thick] table[x=scale, y=45mm_psnr]{\fovData};
\addplot[color=mainLightBlue, mark=o, ultra thick] table[x=scale, y=55mm_psnr]{\fovData};
\end{axis}
\end{tikzpicture}
\begin{tikzpicture}
\begin{axis}[
  ylabel={$SSIM$},
  legend entries={\textbf{Focal length},25 mm, 35 mm, 45 mm, 55 mm},
  legend style={}
]
\addlegendimage{opacity=0.0}
\addlegendimage{color=mainRed, mark=o, ultra thick}
\addlegendimage{color=mainLightRed, mark=o, ultra thick}
\addlegendimage{color=mainOrange, mark=o, ultra thick}
\addlegendimage{color=mainLightOrange, mark=o, ultra thick}

\addplot[color=mainRed, mark=o, ultra thick] table[x=scale, y=25mm_ssim]{\fovData};
\addplot[color=mainLightRed, mark=o, ultra thick] table[x=scale, y=35mm_ssim]{\fovData};
\addplot[color=mainOrange, mark=o, ultra thick] table[x=scale, y=45mm_ssim]{\fovData};
\addplot[color=mainLightOrange, mark=o, ultra thick] table[x=scale, y=55mm_ssim]{\fovData};
\end{axis}
\end{tikzpicture}
\\
\makebox[\linewidth]{\centering $Camera\ grid\ scale$}
\caption{\label{fig:fov}
The camera grid contains $8{\times}8$ cameras and is initially 2 m wide in the scene-space.
The first visible surface is about 1 m away from the grid, and the farthest visible spot excluding the sky is about 90 m away.
The camera grid is uniformly scaled up and down to change the distance between cameras.
The meters units are used according to the world space of the scene.}
\end{figure}

The camera setup used in the scene can be viewed in Fig.~\ref{fig:fovSetup}.
With an increasing space between cameras or a decreasing field of view (increasing focal length), the differences between views increase, and the interpolation is more prone to visual artifacts.
On the other hand, the farther the cameras are, the more freedom is gained for the virtual camera.
This issue can be solved with denser sampling~\cite{chai2000plenoptic}, providing more views in the grid, increasing its dimensions.
However, this leads to higher memory and bandwidth requirements.

\begin{figure}[!htbp]
  \centering
  \includegraphics[width=.45\linewidth]{obrazky-figures/perPixel/fovSetup.jpg}
  \includegraphics[width=.47\linewidth]{obrazky-figures/perPixel/views.jpg}
  \caption{\label{fig:fovSetup}
The size of the grid (red circle) in \textit{Pavilion} scene and the value of field of view was animated and resulting reconstruction quality was measured.
Two views from the corners of the grid using 25 mm focal length are placed in the middle and 55 mm ones at the bottom.
The difference between views is bigger in the 55 mm version.}
\end{figure}

\subsection{Comparison to Other Methods}
An accurate comparison of performance with state-of-the-art methods is complicated due to different methodology and outputs.
The proposed method generates a focus map for the new synthetic view used in the rendering stage.
The process can be roughly compared to depth or disparity map estimation algorithms.
Tab.~\ref{tab:performance} is an indicative overview of the computation times of this stage.

\begin{table}[ht]
\centering
\begin{tabular}{llrr}
%\hline
\textbf{Method}                             & \textbf{Architecture} & \textbf{Resolution}                                   & \textbf{Time} \\ \hline
Proposed                                         & RTX 2070       & $1920{\times}1080{\times}64$                                          & 18 ms         \\ %\hline
\cite{banz2011real}           & Tesla C2050    & $640{\times}480{\times}2$                                             & 16 ms         \\ %\hline
\cite{anisimov2019}   & E3-1245 V2 & $541{\times}376{\times}9$                                             & 1.5 s         \\ %\hline
\cite{jiang2018}         & i7 2.8GHz  & \begin{tabular}[c]{@{}l@{}}$512{\times}512{\times}49$\end{tabular} & 13 min        \\ %\hline
\cite{chen2017fast}           & i7-6700k   & $512{\times}512{\times}49$                                            & 0.8 s         \\ %\hline
\cite{chuchvara2020} & Quadro M1000M  & $1920{\times}1080{\times}45$                                          & 1.58 s        \\ %\hline
\end{tabular}
\caption{\label{tab:performance}
The table contains an overview of computation times of state-of-the-art depth or disparity estimation methods from light fields.}
\end{table}

The core of the method, consisting of the focus range scan, was first evaluated.
The improvements, such as saddle and border detection, block sampling, or map filtering, were then additionally measured to find out how efficient they were.
A side-by-side visual quality comparison with state-of-the-art methods is shown in Fig.~\ref{fig:comparison}.

\begin{figure}[!htbp]\centering
  \includegraphics[width=0.98\linewidth]{obrazky-figures/perPixel/comparison.pdf}
  \caption{
Side-by-side visual comparison with other state of the art methods, rendering a new synthetic view.
The proposed method without improvements outperforms other general methods but does not reach the same quality as learning-based methods.
The results of direct methods do not show any significant differences and are almost identical with difference below 1 dB from the proposed method.
The proposed method produces the sharpest result.
The proposed method was compared to Vagharshakyan~\cite{vagh2018light}, Shi~\cite{shi2015light}, Brox~\cite{brox2004high}, and Ni~\cite{ni2019unsup}.}
  \label{fig:comparison}
\end{figure}

Methods capable of producing the synthetic view directly from the images were chosen for the evaluation.
The proposed method outperforms other similar approaches.
View reconstruction on \textit{Bunny} dataset, using the biggest competitor, the shearlet approach~\cite{vagh2018light}, measured on GeForce GTX\textsuperscript{\texttrademark} Titan X takes 5 s, which is not suitable for real-time rendering.

The proposed method without further improvements does not reach the same visual quality as the newer learning-based methods~\cite{ni2019unsup} when measured on the same dataset used in the original paper, but slightly outperforms older methods~\cite{kalan2016learning} (indirect comparison on \textit{Kitchen} and \textit{Museum} datasets, difference about 1 dB~\cite{ni2019unsup}).
However, the proposed method does not depend on the training process, is faster, and is less memory demanding.

Measurements show that the new proposed method is comparable to the other published algorithms in terms of visual quality, reaching a performance suitable for real-time rendering. Fig.~\ref{fig:focusimprov} shows how saddle detection improved the results, compared to the original scan.

\begin{figure*}[!htbp]\centering
\includegraphics[width=\textwidth]{obrazky-figures/perPixel/focusimprov.pdf}
  \centering
   \caption{The figure contains visual comparison of the proposed improvements, captured on \textit{Lego} and \textit{Pavilion} datasets. Saddle and borders detection leads to huge improvements in visual quality in \textit{Lego} dataset but creates additional noise and artifacts in other ones. 
   Block sampling, using texture interpolation, was used (method \textit{b)} in Fig.~\ref{fig:sampling}. 
   Two combinations of the improvements were also measured.}
 \label{fig:focusimprov}
\end{figure*}

The saddle detection and median filtering bring about quality improvement without significant performance loss; see Tab.~\ref{tab:performanceImpr}.
How this improvement made the method comparable even with deep learning approaches is visible in Tab.~\ref{tab:comparison}.
The block sampling patterns are further referenced as \textit{bA, bB, bC} according to Fig.~\ref{fig:sampling}.
Fig.~\ref{fig:bestCase} shows the best-settings quality improvements for each scene.
Fig.~\ref{fig:block} shows the comparison between the proposed pixel sampling patterns.
The automatic focus range detection shows very good results, measured on the data annotated by human; see Fig.~\ref{fig:detection}.

\begin{table}[!htbp]
\centering
\begin{tabular}{lccc}
\textbf{Method} & \multicolumn{3}{c}{\textbf{Time [ms]}} \\
 {} & $\mathbf{1920{\times}1080}$ & $\mathbf{1024{\times}1024}$ & $\mathbf{624{\times}432}$ \\
 \midrule
 raw & 18.9 & 9.6 & 3.2 \\
 saddle & 19.4 & 9.9 & 3.3 \\
 borders & 19.4 & 9.9 & 3.4 \\
 bA & 30.9 & 15.7 & 5.4 \\
 bB & 22.4 & 11.5 & 3.9 \\
 bC & 43.4 & 21.1 & 5.9 \\
 median & 19 & 9.6 & 3.2 \\
 bB+m & 24.5 & 11.8 & 4.1 \\
 bB+m+s & 24.6 & 11.9 & 4.1 \\
\\
\end{tabular}
\caption{\label{tab:performanceImpr}
    The table contains absolute computational times of focus map generation for all improvements and three common light field resolutions in the tested datasets. 
    The raw label marks the per-pixel scan without the proposed improvements. 
    The time depends only on the resolution of the dataset and does not depend on the content of the scene.}
\end{table}

\begin{table}[!htbp]\centering
\begin{tabular}{lc}
\textbf{Method} & \textbf{PSNR [db]} \\
 \midrule
 film ($4{\times}4$ images)~\cite{film-tf} & 42.73 \\
 \textcolor{red}{proposed (median+block)} & \textcolor{red}{42.62}  \\
 film ($2{\times}2$ images)~\cite{film-tf} & 42.55 \\
 Jiang~\cite{choi2020} & 42.35 \\
 Chlubna~\cite{chlubna2021} & 41.91 \\
 Brox~\cite{brox2004high} & 41.54 \\
 Vagharshakyan~\cite{vagh2018light} & 41.29 \\
 Shi~\cite{shi2015light} & 41.16 \\
\\
\end{tabular}
\caption{\label{tab:comparison}
    A qualitative comparison with other methods is presented here. 
    PSNR was measured on the \textit{Bunny} scene and compared to other results. 
    The best improvements were used in the proposed method.}
\end{table}

\begin{figure}[!htbp]
\begin{minipage}[c]{0.48\linewidth}
\pgfplotsset{every axis plot post/.append style={ultra thick, draw=none, mark=none}}
\pgfplotstableread{charts/perPixel/bestCasePSNR.txt}\bestCasePSNR
\pgfplotstableread{charts/perPixel/bestCaseSSIM.txt}\bestCaseSSIM
\begin{tikzpicture}
\begin{axis}[ybar,xticklabels from table={\bestCasePSNR}{dataset},
  width = \linewidth,
  height = 0.9\linewidth,
  xtick=data,
  xticklabel style={font=\tiny},
  ylabel={$Improvement\ [\%]$},
  xlabel={$Dataset$},
  legend style={at={(0.8,0.8)},anchor=east},
  legend entries={PSNR, SSIM},
  bar width=0.3cm,
  enlarge x limits={abs=2*\pgfplotbarwidth}
]
\addplot[color=white, fill=mainBlue] table[y expr={\thisrow{improvement}},x expr=\coordindex]{\bestCasePSNR};
\addplot[color=white, fill=mainOrange] table[y expr={\thisrow{improvement}},x expr=\coordindex]{\bestCaseSSIM};
\end{axis}
\end{tikzpicture}
\caption{\label{fig:bestCase} The figure shows the best visual quality improvements for each scene in the dataset. 
The improvement value is relative to the per-pixel scan without additional operations.
The complex scenes show higher quality gains than the simple ones, but all scenes are improved.}
\end{minipage}
\hfill
\begin{minipage}[c]{0.48\linewidth}
\pgfplotsset{every axis plot post/.append style={ultra thick, draw=none, mark=none}}
\pgfplotstableread{charts/perPixel/blockPSNR.txt}\blockPSNR
\pgfplotstableread{charts/perPixel/blockSSIM.txt}\blockSSIM
\begin{tikzpicture}
\begin{axis}[ybar,xticklabels from table={\blockPSNR}{method},
  width = \linewidth,
  height = 0.9\linewidth,
  xtick=data,
  xticklabel style={font=\tiny},
  ylabel={$Improvement\ [\%]$},
  xlabel={$Sampling\ method$},
  legend style={at={(0.53,0.8)},anchor=east},
  legend entries={PSNR, SSIM},
  bar width=0.3cm,
  enlarge x limits={abs=2*\pgfplotbarwidth}
]
\addplot[color=white, fill=mainBlue] table[y expr={100-(( 31.8013761666667/\thisrow{avg})*100)},x expr=\coordindex]{\blockPSNR};
\addplot[color=white, fill=mainOrange] table[y expr={100-((0.9219145/\thisrow{avg})*100)},x expr=\coordindex]{\blockSSIM};
\end{axis}
\end{tikzpicture}
\caption{\label{fig:block} The chart shows the visual quality comparison of the three block-sampling patterns.
Two distance variations were compared, the first samples the closest pixels, and the second the pixels that are one pixel further away than the first according to the sampling pattern.}
\end{minipage}
\end{figure}

\begin{figure}[!htbp]
\centering
\pgfplotsset{every axis plot post/.append style={ultra thick, draw=none, mark=none}}
\pgfplotstableread{charts/perPixel/detectionPSNR.txt}\detectionPSNR
\pgfplotstableread{charts/perPixel/detectionSSIM.txt}\detectionSSIM
\begin{tikzpicture}
\begin{axis}[ybar,xticklabels from table={\detectionPSNR}{dataset},
width=0.78\linewidth,
height=0.6\linewidth,
  xtick=data,
  ylabel={$Error\ [\%]$},
  xlabel={$Dataset$},
  legend style={at={(0.3,0.14)},anchor=east},
  legend entries={PSNR, SSIM},
  bar width=1.5cm,
  enlarge x limits={abs=0.7*\pgfplotbarwidth}
]
\addplot[color=white, fill=mainBlue] table[y expr={\thisrow{error}*100},x expr=\coordindex]{\detectionPSNR};
\addplot[color=white, fill=mainOrange] table[y expr={\thisrow{error}*100},x expr=\coordindex]{\detectionSSIM};
\end{axis}
\end{tikzpicture}
\caption{\label{fig:detection} Automatic focus range detection is compared to manual focus set according to subjective visual evaluation of the result. 
The average error of automatic detection using the PSNR metric is 1.46\% and 1.19\% using SSIM. 
The detection was able to outperform the manual setting in the case of the \textit{Chess} dataset, measuring PSNR.
This was probably caused by the inaccuracy of the PSNR metric according to human visual perception.}
\end{figure}

\section{Focusing Performance Optimizations}
\label{sec:scanOptim}
The generation of the focus map can be accelerated to produce a better quality/speed ratio~\cite{chlubna2024accel}. 
The optimal computational efficiency mentioned in \hyperref[hypothesis]{Hypothesis} is the main goal of this research.
This section describes the proposed optimization of the above-described light field rendering method; see Fig.~\ref{fig:optimTeaser}.

\begin{figure*}[!htbp]\centering
  \includegraphics[width=0.99\linewidth]{obrazky-figures/scan/teaser.pdf}
  \caption{
Real-life or synthetic scene is captured by multiple cameras which are optimally positioned according to the proposal.
The captured images, closest to the virtual camera, are mutually shifted to simulate light field focusing.
Multiple focus distances are scanned using a novel optimal strategy with an improved color dispersion metric.
Optimal focusing for each pixel is stored in a focus map which is filtered and used to synthesize a novel view.
The result is produced by the proposed method with \textit{Cornell box} scene.
  }
  \label{fig:optimTeaser}
\end{figure*} 

\subsection{Focus Range Scanning}
Scanning of the focus range and looking for the minimal color dispersion was previously~\cite{chlubna2021, chlubna2023} carried out using a brute-force approach.
A fixed number of steps with constant spaces was used.
This brute-force scan has two disadvantages.
First, it can skip the optimal value due to the constant spacing of the samples.
Second, it is computationally demanding because each sample involves GPU memory access operations that slow down the process.
The goal of this research is to find a better strategy that adaptively scans the relevant parts of the range and skips the parts that do not contain any potentially desired values.
Several alternative scanning methods are proposed and tested:
\begin{itemize}
  \item \textbf{BF} (brute force) is a simple scan with fixed number of steps and constant spaces. 
  \item \textbf{BFET} (brute force with early termination) is the BF scan that stops if the last optimal value does not change in a defined number of steps.
  \item \textbf{VS} (variable step) shrinks the step size when a new optimal value is found and grows otherwise.
  \item \textbf{VSET} (variable step with early termination) is the VS scan that stops if the last optimal value does not change in a defined number of steps.
  \item \textbf{RAND} (random) scans the range at random positions with an early termination mechanism.
  \item \textbf{TD} (top down) starts with three samples, and in each iteration scans the center positions of the remaining spaces between the previously sampled positions.
  \item \textbf{HIER} (hierarchical) uses bisection method to scan the range sparsely and shrink the scanning interval in each iteration.
  \item \textbf{DESC} (gradient descent~\cite{haji2021comparison}) starts at multiple uniformly selected positions in the range. 
  Each scan branch uses a variable scan step that changes according to the difference between the last focus value and the new optimum. 
  The best value of all branches is selected in the end.
  \item \textbf{PYR} (pyramid) performs the first scan on downsampled and blurred data. 
  A tight area around the found optimum is scanned again on the full-resolution data.
  The downsampling might help to identify the area that most likely contains the desired values, and large disparities or single colored areas might be easier to detect~\cite{trinidad2019multi}. This method can exploit texture mipmapping.
\end{itemize}
Fig.~\ref{fig:methods} describes the methods.
Stochastic variations of the proposed methods were tested too, but they showed too much thread divergence on GPU and were too slow.

\begin{figure}[!htbp]\centering
  \includegraphics[width=\linewidth]{obrazky-figures/scan/methods.pdf}
  \caption{
Scanning methods compared in this study are depicted in the figure.
The blue line marks the whole focus range.
The orange arrows are samples in the scan.
Each method scans the range with different number of steps and at different positions.
  }
  \label{fig:methods}
\end{figure}

\subsection{Color Dispersion Metric}
A statistical dispersion metric is used when the colors of the pixels sampled are compared.
This metric is used to estimate the probability that the sampled pixels belong to the same spot in the scene.
The assumption is that the colors of such pixels would be similar.
The color function $\textrm{C}$ in Eq.~\ref{eq:dispA} for a specific pixel and synthetic view position is a mapping between the focus level $f$ and the light field image grid coordinates $\textbf{st}$ into the $m$-dimensional color space (typically $m=3$ for RGB or YUV spaces):
\begin{equation}
	\textrm{C}(\textbf{st},f) : \mathbb{R}^2\times\mathbb{R} \rightarrow \mathbb{R}^m.
	\label{eq:dispA}
\end{equation}
An array of colors for all images $\textbf{X}(f)$ from $S{\times} T$ sized grid is processed in Eq.~\ref{eq:dispB} as:
\begin{equation}
	\textbf{X}(f) = \left[
    \textrm{C}(\textbf{u},f)
    \right]\ \ \ \ \textbf{st} \in [1 \dots S] \times [1 \dots T].
	\label{eq:dispB}
\end{equation}
The optimal focus distance $f_o$ in Eq.~\ref{eq:dispC} is found from the focus range $[f_s,f_e]$, based on the minimal color dispersion value $\textrm{disp}(\textbf{X}(f))$:
\begin{equation}
	f_o = \textrm{arg\,min}_{f \in [f_s,f_e]}\left \{\textrm{disp}(\textbf{X}(f))\right \}.
	\label{eq:dispC}
\end{equation}

This study contains a comparison of several dispersion metrics that differ in quality and computational complexity:
\begin{itemize}
  \item \textbf{VAR} (variance) is standard statistical variance. 
  \item \textbf{RANGE} (total range) measures the maximal and minimal distance from the origin, and their difference is marked the dispersion.
  \item \textbf{ERANGE} (elementwise range) is a metric where maximal and minimal values are stored for each color vector element. 
  The distance between the maximal and minimal vectors marks the dispersion.
  \item \textbf{MAD} (mean absolute difference): would be too complex to compute, so a simple approximation is used.
  The difference of the new color and the last one and the difference of the new color and one sample from the history are added to the final dispersion.
\end{itemize}
Interquartile range metric~\cite{dekking2005} was also considered but not measured because the preliminary test showed that this metric is computationally too demanding and the results in terms of visual quality were not promising.

\subsection{Pixel Sampling}
Memory requirements of light fields are one of the main issues in the rendering pipelines.
Encoding in video formats was previously proposed~\cite{chlubna2023} to stream the necessary packets in the GPU memory, where the frames are decoded in the HW decoders.
Most decoders, such as NVDEC, return frames in YUV format.
Conversion between color spaces might bring additional overhead.
The attributes of YUV might be exploited to achieve better results than in standard RGB.
Given the assumption that most of the materials in the real world are partially diffuse and specular, the color of the pixel at the same spot on the scene geometry should not change with the changing camera position. 
The luminance of the pixel can change.
The Chebyshev distance~\cite{mahama2016choice} proved to be the best metric to measure color similarity in focus map generation~\cite{chlubna2021}.
This distance function can be adjusted to attenuate the luminance component in Eq~\ref{eq:chebyshev} as:
\begin{equation}
    d = \textrm{max}\left (\left | c_{Y}^{A} - c_{Y}^{B} \right |\cdot w_Y, \left | c_{U}^{A} - c_{U}^{B} \right |, \left | c_{V}^{A} - c_{V}^{B} \right |  \right ),
    \label{eq:chebyshev}
\end{equation}
where $d$ is the final distance, $c^A$ and $c^B$ are the input colors with subscripts $c_{Y,U,V}$ representing the color components and $w_Y$ is the attenuation weight for the $Y$ component distance.

Additional strategies for sampling the texture outside its borders were proposed, as the shifted textures always end up being sampled outside.
Blending of the pixels at the border, where the radius grows with the distance from the border and alternating of the last pixels instead of copying of the last one might create more distinctive details.

\section{Experimental Evaluation with Novel 4K Dataset}
\label{sec:eval4K}
The implementation uses CUDA kernels for the focus map computation.
Visual quality was measured using PSNR and SSIM metrics.
Quality was measured relative to the basic BF method and averaged.
The number of scan steps was set to $32$ according to previous measurements~\cite{chlubna2021}, as it ensures the optimal speed/quality ratio.
Decreasing the steps leads to quality loss with only linear speed-up.
Additional no-reference metrics LIQE~\cite{zhang2023liqe} for standard artifact detection and NIQSV+~\cite{tian2018} for multi-view image synthesis evaluation were used for further analysis of the results.
MNSS~\cite{gu2020} metric designed for DIBR (depth-image-based rendering) was also tested, but the results did not correspond to the actual quality of the novel view.
The DIBR artifacts are apparently of a different nature than artifacts appearing in light field rendering.
Measurements were carried out on all scenes of the dataset, unless explicitly stated otherwise.
Five reference views for each measurement were linearly selected, with constant spaces, from the reference dataset grid and evaluated.
The dataset was generated by the method described in Section~\ref{sec:novelCapture}.

\subsection{Important Parameters Identification}
Initially over six million test combinations were considered, but due to the size being unreasonable, only the most promising parameters were selected for full testing.
A quick test was conducted on \textit{street animation} scene to eliminate parameters that did not show significant improvements.
The scene was selected because it resembles an ordinary real-life photo and contains various materials.

\subsubsection{Reduction of Scan Methods}
RAND, VSET, VS and BFET methods are considered to be only minor optimizations.
Only VSET was selected for the further measurement, according to Fig.~\ref{fig:firstTestScanMethods}.

\begin{figure*}[!htbp]
\pgfplotsset{every axis plot post/.append style={ultra thick, draw=none, mark=none}}
\pgfplotsset{
  xbar,
  enlarge x limits={abs=0pt},  
  height=.2\linewidth,
  width=.51\linewidth,
  ytick=data,
  xmin=0,}

\pgfplotstableread{charts/scan/firstTestScanMethodsTime.txt}\dataA
\pgfplotstableread{charts/scan/firstTestScanMethodsQuality.txt}\dataB
\begin{tikzpicture}
\begin{axis}[yticklabels from table={\dataA}{method},
  xlabel={$Times\ faster$},
  ylabel={$Method$},
  legend style={at={(0.8,0.8)},anchor=east},
  area legend,
]
\addplot[color=white, fill=mainBlue] table[x=value,y expr=\coordindex]{\dataA};
\end{axis}
\end{tikzpicture}
\begin{tikzpicture}
\begin{axis}[yticklabels from table={\dataB}{method},
  xlabel={$Quality\ change\ [\%]$},
  yticklabels={,,},
]
\addplot[color=white, fill=mainOrange] table[x=value,y expr=\coordindex]{\dataB};
\end{axis}
\end{tikzpicture}
\caption{\label{fig:firstTestScanMethods}
These four similar scanning methods were compared to select only one for further measurements.
The quality does not change significantly, but the speedup is the best at VSET. 
The measurements are relative to the base BF method.}
\end{figure*}

Although the quality metrics were not in favor of RAND, this scan method might be useful for early preview purposes similarly to progressive path tracing, where the amount of noise decreases with every new sample~\cite{xing2021progressive}.
The advantage of RAND is that the error distribution is uniform throughout the image~\cite{luengo2020survey}. 
This method can transform the typical light field ghosting artifacts to classic, per-pixel, color noise.
Other methods usually leave whole areas distorted, so the area-wise error is more visible to human observer than the pixel-wise error; see Fig.~\ref{fig:firstTestScan}.
Perceptual LIQE metric score for RAND was $1.97$ compared to $1.73$ for VSET and $3.382812$ for the reference image.
NIQSV+ score was $7.0$ for RAND and $6.9$ for VSET.

\begin{figure}[!htbp]\centering
   \def \imageSize{0.32\linewidth}
    \subfloat[\centering Reference image is taken from the input light field. ]{{\includegraphics[width=\imageSize]{obrazky-figures/scan/firstTestScanREF.jpg} }} 
    \subfloat[\centering RAND focuses the scene but adds noise everywhere.]{{\includegraphics[width=\imageSize]{obrazky-figures/scan/firstTestScanRAND.jpg} }}
    \subfloat[\centering VSET leaves some parts of the scene out of focus.]{{\includegraphics[width=\imageSize]{obrazky-figures/scan/firstTestScanVSET.jpg} }}
  \caption{\label{fig:firstTestScan} 
The figure shows how VSET scan can reconstruct many small details but can contain large areas with artifacts. 
RAND contains more pixels with error, but the error might not be visible because it is distributed uniformly.
The images are zoomed parts of the \textit{street animation} scene.
}
\end{figure}

\subsubsection{Map Filtering}
The final focus map usually contains noise. 
The map was filtered by three methods: median filter, symmetric nearest neighbor~\cite{hall2007smooth}, and Kuwahara filter~\cite{Kuwahara1976}.
Median and Kuwahara produced better quality results.
Kuwahara filter was identified as the best option because the same quality of the result could be achieved in half the time of the previously proposed median filter.

\subsubsection{Color Sampling}
The pixels are sampled by blocks to obtain information about the surrounding details.
Previous measurements showed that pixels can be optimally sampled in certain patterns with a kernel with a width of 2 pixels~\cite{chlubna2021}. 
Only 5 pixels are sampled in the distance defined by the radius.
One pixel in the middle and four interpolated, each between the four pixel coordinates in the corners.
The measurements on 4K data showed that the visual quality improved with increasing block radius up to 10 pixels.
The quality reached an improvement of 13\% compared to the single-pixel sampling.
Experiments on various sizes of the data showed that the radius can be linearly adjusted according to the total number of pixels, for example, 2.5 for FullHD resolution.

The change of color spaces does not significantly affect the result. 
It can be safely assumed that the proposed method works with the same efficiency in both RGB and YUV.
YUV formats, such as widely used NV12, store the Y values in a separate array from UV. 
Reading the whole YUV triplet would cause inconsistent read operations on GPU which might negatively affect the performance.
The measurements showed that the color distance can be replaced by the simple difference of Y components of the colors with the loss of quality by $3\%$.
The proposed weighted YUV Chebyshev distance, see Eq.~\ref{eq:chebyshev}, slightly outperforms the other mentioned approaches with the attenuation weight set to $0.25$ which was empirically discovered.
The quality improvement is $0.2\%$ compared to standard RGB and $0.8\%$ to YUV Chebyshew.

\subsubsection{Other Parameters}
\label{sec:parameters}
Measurement of the behavior of the out-of-texture sampling showed that the computational time does not change significantly.
The visual quality of standard wrapping, mirroring, and solid border modes is the lowest.
Clamping, blending, and alternating produced similar results, with alternating slightly outperforming the rest by $0.34\%$.

The effect of the order of the color distance ($distance^{order}$) function was measured, but the effect on the result is not statistically significant.
The focus range was originally scanned linearly. 
The order of non-linear transformation ($range^{order}$) was also measured since the details that are far from the camera might not be as dense and might need a sparser scan than the objects nearby.
This assumption was not confirmed, and the linear range showed the best results.
Both orders were kept at the value of $1$ in all other experiments.

The average quality measured in all tests showed that the input views can be reduced to only $4$ nearest neighbors to the synthetic view, compared to the previously proposed $16$ views.
The quality loss is only $0.3\%$, but the reduced number of texture read operations drastically speeds up the process $10.6{\times}$ when measured over all the approaches mentioned.

\subsection{Color Dispersion}
The dispersion metrics were compared by averaging all measurements with all scanning methods; see Fig.~\ref{fig:dispersionResult}.
The previously proposed VAR metric for the evaluation of color dispersion can be replaced by ERANGE which is $4.14{\times}$ faster and reaches better quality than VAR by $3\%$.

\begin{figure*}[!htbp]
\pgfplotsset{every axis plot post/.append style={ultra thick, draw=none, mark=none}}
\pgfplotsset{
  xbar,
  enlarge x limits={abs=0pt},  
  height=.19\linewidth,
  width=.50\linewidth,
  ytick=data,
  xmin=0,}

\pgfplotstableread{charts/scan/dispersionTime.txt}\dataA
\pgfplotstableread{charts/scan/dispersionQuality.txt}\dataB
\begin{tikzpicture}
\begin{axis}[yticklabels from table={\dataA}{method},
  xlabel={$Times\ faster$},
  ylabel={$Method$},
  legend style={at={(0.8,0.8)},anchor=east},
  area legend,
]
\addplot[color=white, fill=mainBlue] table[x=value,y expr=\coordindex]{\dataA};
\end{axis}
\end{tikzpicture}
\begin{tikzpicture}
\begin{axis}[yticklabels from table={\dataB}{method},
  xlabel={$Quality\ change\ [\%]$},
  yticklabels={,,},
]
\addplot[color=white, fill=mainOrange] table[x=value,y expr=\coordindex]{\dataB};
\end{axis}
\end{tikzpicture}
\caption{\label{fig:dispersionResult}
The color dispersion metrics were compared over all measurements.
The ERANGE metric outperforms all other approaches in time and quality.
MAD metric brings no improvement.
The measurements are relative to the base VAR method.}
\end{figure*}

\subsection{Focus Scanning}
The comparison of the scanning methods is performed with the ERANGE metric, but the results are similar even with the other ones; see Fig.~\ref{fig:scanResult}.
The PYR method reaches almost the same quality as the base BF method, the quality loss is $2.86\%$, but it is $1.74{\times}$ faster.
The TD method is the fastest, $2.96{\times}$ faster, but the quality loss is $12.83\%$. 
HIER reaches $1.06{\times}$ speedup with $3\%$ quality loss.

\begin{figure*}[!htbp]
\pgfplotsset{every axis plot post/.append style={ultra thick, draw=none, mark=none}}
\pgfplotsset{
  xbar,
  enlarge x limits={abs=0pt},  
  height=.25\linewidth,
  width=.52\linewidth,
  ytick=data,
  xmin=0,}

\pgfplotstableread{charts/scan/scanTime.txt}\dataA
\pgfplotstableread{charts/scan/scanQuality.txt}\dataB
\begin{tikzpicture}
\begin{axis}[yticklabels from table={\dataA}{method},
  xlabel={$Times\ faster$},
  ylabel={$Method$},
  legend style={at={(0.8,0.8)},anchor=east},
  area legend,
]
\addplot[color=white, fill=mainBlue] table[x=value,y expr=\coordindex]{\dataA};
\end{axis}
\end{tikzpicture}
\begin{tikzpicture}
\begin{axis}[yticklabels from table={\dataB}{method},
  xlabel={$Quality\ change\ [\%]$},
  yticklabels={,,},
  xmax=100
]
\addplot[color=white, fill=mainOrange] table[x=value,y expr=\coordindex]{\dataB};
\end{axis}
\end{tikzpicture}
\caption{\label{fig:scanResult}
The scanning methods were compared, showing that all methods are significantly faster than the base BF.
The PYR method outperforms all other approaches in time and quality.
The measurements are relative to the base VAR method.}
\end{figure*}

No-reference metrics were also used to evaluate how the artifacts manifest in the best scan.
NIQSV+ and LIQE scores for BF and PYR were $5.44$, $5.41$ and $1.86$, $1.84$ respectively. 
This shows that the perceptual distortion of the PYR scan is minimal compared to the BF.

Fig.~\ref{fig:error} shows the common errors in the results. 
Although the number of scanning levels can be reduced to speed up the original BF method, the constant scan leads to a crude focusing approximation where certain ranges are never scanned at all. 
The proposed scan methods are less prone to vanishing details and ghosting due to the possibility of denser scan when necessary.

\begin{figure*}[!htbp]\centering
   \def \imageSize{0.32\linewidth}
    \subfloat[\centering Reference image is taken from the input light field. ]{{\includegraphics[width=\imageSize]{obrazky-figures/scan/errorRef.pdf} }} 
    \subfloat[\centering BF method creates incomplete surfaces.]{{\includegraphics[width=\imageSize]{obrazky-figures/scan/errorBF.pdf} }}
    \subfloat[\centering PYR significantly mitigates the artifacts.]{{\includegraphics[width=\imageSize]{obrazky-figures/scan/errorPYR.pdf} }}
  \caption{
Parts of the most challenging \textit{high frequency} scene are shown on the pictures. 
The number of steps of the BF method was lowered to reach the same time as PYR.
}
  \label{fig:error}
\end{figure*} 

\subsection{Optimal Setup}
According to the above measurements, the optimal parameters are identified and the best method for focus map generation is proposed with parameters listed in Tab.~\ref{tab:best}.
The comparison with the previously proposed and other state-of-the-art methods is shown in Tab.~\ref{tab:compare}.

\begin{table*}[!htbp]
\centering
\begin{tabular}{ccccccc}
\input{tables/bestScan.tex}
\end{tabular}
\caption{
The best-quality and fastest computation parameters were identified.
}
\label{tab:best}
\end{table*}

\begin{table}[!htbp]
\centering
\begin{tabular}{ccccccc}
\input{tables/compareScan.tex}
\end{tabular}
\caption{
Previous focus map generation method is compared to the improved proposal.
}
\label{tab:compare}
\end{table}

Fig.~\ref{fig:comparisonScan} shows a visual and measured comparison of the proposed method and other recently published approaches.
Furthermore, the FLAVR~\cite{kalluri2021flavr}, ST-MFNet~\cite{danier2022} and SAVFI~\cite{choi2020} methods were tested but required more views than $4$ closest to produce a reasonable result.
Instant NGP~\cite{mueller2022instant} was able to use $4$ views, but the result was still blurry even after minutes of processing.

\begin{figure}[!htbp]\centering
  \includegraphics[width=0.96\linewidth]{obrazky-figures/scan/comparison.pdf}
  \caption{
View interpolation methods were compared on \textit{Cornell box} scene.
The other methods are: previous scan~\cite{chlubna2023} for focus map generation, deep interpolations such as softmax~\cite{niklaus2020}, ifrnet~\cite{kong2022}, biformer~\cite{park2023BiFormer}, film~\cite{film-tf}, and optical-flow based fruc by NVIDIA.
All methods were measured on GPU.
Inputs of some methods had to be downscaled.
}
  \label{fig:comparisonScan}
\end{figure} 

The $4$ closest images are problematic for the 3D and neural~\cite{wang2021ibrnet} reconstructions, as the 3D information is difficult to obtain from only four views positioned on a plane. 
For example, Colmap library, Meshroom, or 3DF Zephir were unable to correctly position the cameras for dense point-cloud reconstruction.
All these results show that the proposed method is valuable for real-time processing as it saves memory, produces a high-quality synthetic view, and is significantly faster than other methods.
The previous focus map scan used $16$ nearest views.
The implementation of the previous method used in the experiments was adjusted to use $4$ views, otherwise the speed-up would be even higher.

The memory and time requirements of the proposed method are lower than state of the art.
To prove that the method reaches the same level of visual quality as existing methods, a qualitative comparison was conducted on different existing datasets.
A zoomed detail of the \textit{chest} scene is compared in Fig.~\ref{fig:qualitativeZoom} where many methods ignore the details on the crystal and blur them.
Optical-flow-based fruc, on the other hand, unnecessarily highlights the details.

\begin{figure*}[!htbp]\centering
  \includegraphics[width=0.99\linewidth]{obrazky-figures/scan/qualitativeZoom.pdf}
  \caption{
A problematic zoomed region of \textit{chess} \cite{vaish} scene is compared across the interpolation methods.
The reflective material of the crystal with the small crack-like shiny details is hard to reconstruct.
}
  \label{fig:qualitativeZoom}
\end{figure*} 

Fig.~\ref{fig:qualitative} shows full novel views views from different scenes.
The results look visually the same with minor differences.

\begin{figure*}[!htbp]\centering
  \includegraphics[width=\linewidth]{obrazky-figures/scan/qualitative.pdf}
  \caption{
Existing view interpolation methods are qualitatively compared using scenes from different datasets: \textit{board games} \cite{honauer2017dataset}, \textit{chess} \cite{vaish}, \textit{chest} \cite{vaish}, \textit{flower} \cite{vaish}, and \textit{simple setting} (original dataset). 
The \textcolor{mainOrange}{orange numbers} in the bottom-left corners are SSIM results.
}
  \label{fig:qualitative}
\end{figure*} 

The proposed method can produce artifacts similar as the existing methods but is stable.
Deep learning methods can fail when the pretrained model does not contain the necessary information for the specific kind of scene.
The example of such an issue is especially visible in the \textit{simple setting} scene with softmax and ifrnet.
Although deep learning methods can sometimes produce higher-quality results, the proposed method is more stable and lightweight.

\subsection{Scene Types}
The original dataset proved to be a good benchmark of various types of scenes.
The best quality, PSNR above 40 dB and SSIM over $0.95$, was measured at \textit{Cornell box}, \textit{low frequency} and \textit{simple setting} scenes.
The worst quality, PSNR below 25 dB and SSIM below $0.75$, was measured at \textit{high frequency} and \textit{large depth} scenes.
It can be assumed that two crucial parameters regarding the resulting quality of the proposed method exist.
The first is the frequency of the features in the scene, as the sampling can miss the high frequency patterns according to the Nyquist–Shannon theorem~\cite{chai2000plenoptic}.
The second is the depth range, where the wide range is more difficult to properly scan without skipping important details in the scene.
Both parameters need to be high to decrease the quality.
For example, \textit{low frequency} contains a large depth range but belongs to the best results.
The method is efficient with reflective materials, volumetric effects, and single-colored areas, which are often considered problematic for image-based rendering methods.
Surprisingly, the \textit{reflective} scene reached slightly better quality of the novel view than its counterpart \textit{diffuse}.
Both scenes contain the same geometry and differ only in material.

The views need to be synchronized in time~\cite{wang2007} and captured by the same type of camera.
The proposed method can produce a novel view even from temporarily unsynchronized views with different brightness levels.
The color similarity metric ensures that the least error value is produced, so the unsynchronized results are simply motion blurred.

Several filters including CLAHE contrast~\cite{sonali2019}, edge detection, sharpening, histogram equalization, bilateral filtering, sinusoidal color transformation, etc. were tested for both denoising and detail highlighting.
The quality remained the same, and the filtering did not aid the algorithm.
However, this proved that the method is robust enough and can tolerate such filters and applied effects on the views.

\section{Novel Dataset Capturing Method}
\label{sec:novelCapture}
This section proposes an optimal method for capturing light field data~\cite{chlubna2024accel}.
It supports the \hyperref[hypothesis]{Hypothesis} by creating optimal dataset for the proposed method.
This data set ensures the best visual quality of the results produced by the proposed rendering.

Narrowly spaced data, for example, taken by plenoptic cameras, are useful for refocusing of the final photo at different distances~\cite{zhaolin2022}.
They can be represented as a grid of images with small spaces or by a focal stack, where the images are taken from the same position but differently focused~\cite{pendu2019}.
Widely spaced data allow for 3D movement in the scene~\cite{wilburn2005}.
Widely spaced discrete light field data are the main topic of this research.

The shape of the camera grid used is usually defined by the technical limitations of the rig or by subjective visual evaluation of the results.
Existing datasets usually use constant spacing between cameras~\cite{levoy1996light}, or different fixed spaces in both axes depending on the capturing mechanism~\cite{herfet2018enabling}.
Reconfigurable camera arrays also exist, where camera positions are adjusted based on the rendering quality of the result~\cite{zhang2004self}.
The cameras are usually positioned to achieve a defined overlap, for example, complete view overlap at 10 feet from the array~\cite{rerabek2016new}, instead of depth-aware adjustment.
This section proposes a capturing method designed to produce the best results with the proposed rendering method by having the same amount of view overlap in both axes.

Two main parameters can be defined by the user to affect the positioning of the cameras in the grid: the distance from the cameras plane to the geometry of the scene and the amount of overlap between the cameras.
These parameters are more intuitive for the user than, for example, size of the camera grid, because they reflect how much of the scene content the resulting views share between each other.

The geometry of the scene is usually not planar. 
The distance to the geometry needs to be defined as either a distance to the object of interest or to the average position of the whole visible geometry.
Occluded vertices or varying density of the geometry in a synthetic scene disallow the method to analyze the geometry itself.
A screen-space approach is proposed to solve this issue.
In case of a synthetic scene, the scene is quickly rendered from the reference camera with all object materials overridden by a special material, which converts world-space coordinates of the fragments into colors.
The average of such a rendered image is a decent approximation of the average position of the visible geometry.
One, even a low-quality, depth sensor would suffice for real-life camera grids.
The camera grid is then created, taking a reference camera position and orientation as the center of the grid.

Having square-shaped cameras with the same width and height and the same horizontal and vertical spaces between the cameras always results in the same-sized overlapping areas in the views.
The aspect ratio between the width and height of cameras in modern cinematography or photography is usually used with a wider width, for example, $16:9$~\cite{cardwell2015sense}.
Regular spaces between such cameras would lead to unequal overlap.
Spacing simply scaled by the aspect ratio would ensure that the cameras exactly touch their borders when the overlap is zero but would be unequal when it is greater.
The spacing needs to be recalculated based on the input parameters. 
Standard 50 mm lens was used in the implementation, as it creates the most natural-looking image for a human observer~\cite{banks2014camera}.
Fig.~\ref{fig:cameraArray} demonstrates the described issue.

\begin{figure*}[!htbp]\centering
  \includegraphics[width=\linewidth]{obrazky-figures/scan/cameraArray.pdf}
  \caption{
  The figure demonstrates the problem of overlaping view cones in light field camera grids.
  A non-uniform aspect ratio, e.g., standard $16:9$, creates different overlapping areas of the image planes between the neighboring cameras in the vertical and horizontal directions.
  Recalculation of the camera spacing by the proposed method is necessary to ensure the same amount of overlap in both axes.
  }
  \label{fig:cameraArray}
\end{figure*}

Fig.~\ref{fig:cameraOverlap} depicts the positions of the cameras in relation to the scene and along with the 2D-aligned view in Fig.~\ref{fig:cameraOverlap2D} contains definitions of the variables used below.

\begin{figure}[!htbp]
\begin{minipage}[c]{0.48\linewidth}
  \includegraphics[width=\linewidth]{obrazky-figures/scan/cameraOverlap.pdf}
  \caption{
  Two capturing cameras placed at two neighboring rows and columns are depicted.
  The $s_h$, $s_v$, and $d$ lines are going through the centers of projection.
  The camera cones overlap in both axes with amount $o_{h}$ and $o_{v}$.
  The goal of the method is to achieve the state where $o_{h} = o_{v}$.
  }
  \label{fig:cameraOverlap}
\end{minipage}\hspace{0.5cm}
\begin{minipage}[c]{0.48\linewidth}
  \includegraphics[width=\linewidth]{obrazky-figures/scan/cameraOverlap2D.pdf}
  \caption{
  2D view of two neighboring cameras is shown.
  The spacing $s_h$ between the cameras is the desired result.
  The ends of the view cones represent the image planes.
  }
  \label{fig:cameraOverlap2D}
\end{minipage}
\end{figure}

Given a horizontal field of view $f_h$, horizontal $l_h$, and vertical $l_v$ camera length or resolution, the tangent of half of the vertical field of view $f_v$ can be obtained from
\begin{equation}
    \frac{\tan \left( \frac{f_{h}}{2} \right)} {\tan \left( \frac{f_{v}}{2} \right)} = \frac{l_{h}}{l_v}.
    \label{eq:overlapStart}
\end{equation}
The horizontal distance between cameras $s_h$ can be expressed from
\begin{equation}
    \tan \left( \frac{f_{h}}{2} \right) = \frac{s_{h}}{2d_{1}}
\end{equation}
as
\begin{equation}
    s_{h}=2\tan \left( \frac{f_{h}}{2} \right) d_{1},
    \label{eq:overlapD}
\end{equation}
using the distance $d_1$ from the cameras plane to the closest point of camera frustum overlap.
This distance can be obtained from
\begin{equation}
    d_1 = d - d_2.
    \label{eq:overlapA}
\end{equation}
The distance $d$ is the total distance from the cameras plane to the scene geometry and $d_2$ is the distance from the aforementioned point of overlap to the scene geometry. 
Both $d_1$ and $d_2$ are unknown but can be obtained by expressing $d_2$ from
\begin{equation}
    \tan \left( \frac{f_{h}}{2} \right) = \frac{o_{h}}{2d_{2}},
    \label{eq:overlapB}
\end{equation}
where $o_{h}$ denotes the amount of overlap between the cameras.
Combining the equations~\ref{eq:overlapA} and~\ref{eq:overlapB} expresses $d_1$ as
\begin{equation}
    d_1=d-\frac{o_{h}}{2\tan \left( \frac{f_{h}}{2} \right)}.
    \label{eq:overlapC}
\end{equation}
Combining equations~\ref{eq:overlapD} and~\ref{eq:overlapC} results in
\begin{equation}
     s_{h}=2\tan \left( \frac{f_{h}}{2} \right) \left(d - \frac{o_h}{2\tan \left( \frac{f_{h}}{2} \right)}\right),
\end{equation}
which can be simplified to the final formula:
\begin{equation}
     s_{h}=2\tan \left( \frac{f_{h}}{2} \right)d - o_h.
\end{equation}
The same approach can be used to analogously express the vertical distance between the cameras $s_{v}$:
\begin{equation}
     s_{v}=2\tan \left( \frac{f_{v}}{2} \right)d - o_v.
\end{equation}
The horizontal $o_h$ and vertical $o_v$ amount of overlap should be the same according to the desired constraint:
\begin{equation}
     o_{h}=o_{v}.
     \label{eq:overlapEnd}
\end{equation}

This method, described by Eq.~\ref{eq:overlapStart}--\ref{eq:overlapEnd}, was used to produce a novel 4K synthetic dataset containing 22 static and 4 animated light field scenes with the size of the grid $15{\times}15$ views.
The scenes were selected to cover most of the commonly measured scenarios such as diffuse/reflective materials, wide/narrow depth range, high/low frequency features, single/multiple objects, near/far objects, all-focused/depth-of-field, solid/volumetric objects, and their combinations.
The currently existing datasets are usually available in lower quality and do not cover all the mentioned scenarios suitable for benchmarking.

\subsection{Evaluation}
The amount of overlap in the input views can affect the quality of the novel synthetic view.
The distance between the cameras depends on the defined minimal overlap value.
When the view is not in $1{:}1$ ratio, the minimal overlap is defined on the vertical axis.
With constant spaces between the cameras, the views would overlap more on the horizontal axis.
This might be redundant.
Using the proposed camera distance calculation method ensures that the minimal overlap will be used in both axes.

An experiment was conducted to measure how the quality changes when the aforementioned approaches are used.
The 4K \textit{street animation} scene from the dataset was used with the default camera settings documented in the dataset files.
Tab.~\ref{tab:gridTest} shows that the quality does not change significantly.
However, the proposed method allows for larger spaces between the cameras, which means that a larger portion of the scene is captured without quality degradation.
For comparison, an even larger spacing was measured in the table to show how the quality decreases when the distance is too large.
The proposed method ensures the best quality of the result with the largest possible spacing.
Note that the spacing difference in favor of the proposed method might be even larger with other camera configurations.

\begin{table}[ht]
\centering
\small
\begin{tabular}{lccccc}
\input{tables/gridTest.tex}
\end{tabular}
\caption{
Various light field camera grids are compared.
The proposed method ensures the largest grid width without visible quality degradation.
The grid height remains the same in all measurements.
When the width of the grid is too large, the quality loss is visible.
}
\label{tab:gridTest}
\end{table}