\subsection{Camera Trajectory of Input Views}
\label{sec:cameraMotion}
Input views for 3D displays need to follow defined rules regarding their camera positions.
Methods mentioned in this section are tools used to identify camera motion in a sequence of such views.
These methods are used in the following sections.

Camera motion following a straight horizontal line trajectory is called \textit{truck} in cinematography. 
It is visually similar to \textit{pan} which consists only of rotation around vertical axis.
Dense~\cite{farneback} or sparse~\cite{lucasKanade} optical flow can be used to detect camera motion for consecutive video frames.
Another option is to use feature matching algorithms with descriptors such as SIFT~\cite{arya1998}, SURF~\cite{bay2006}, KAZE~\cite{alcantarilla2012}, or ORB~\cite{rublee2011} and find the displacement of the found features. 

For a sparse optical flow, the tracking features are selected using the Shi-Tomasi corner detector~\cite{jianbo1994}. 
The dominant camera motion in a video sequence is usually estimated by averaging optical flow vectors in predefined regions of the image/flow~\cite{younis2018}.
A set of rules is then used to estimate camera motion based on the dominant direction in each region~\cite{xiong1998efficient, lee2002}.

Knowing the intrinsics of the camera, a combination of multiple optical flow models can provide more accurate results~\cite{park2004}.
A spatio-temporal derivative of the intensity in two successive frames can be used~\cite{bouthemy1999unified}.
Motion vectors can be computed using temporal gradient-based block matching.
The resulting motion is estimated using the motion vectors of interest, which are chosen from the field according to their significance and consistency~\cite{hasan}. 
If encoded video files are used as input data, the necessary motion vectors can be extracted from the compressed stream~\cite{almeida, weng2011, tiburzi2007}. 
Hidden Markov Models~\cite{naito} or Transferable Belief Model~\cite{guironnet2006} might be more robust alternatives to solid thresholds in the camera motion estimation rules.
Multiresolution least-squares methods might be used to fit the motion model even with noisy data~\cite{odobez1995}.
Model-based estimations might not cover the whole range of possible videos and camera motions. 
Machine learning approaches to overcome this problem were also proposed~\cite{lingyu2006, liu2010, chen2004, chang2004, geng2006}, but their quality depends on the training process. 

Structure-from-motion and visual simultaneous localization and mapping techniques are frequent approaches that are used to reconstruct a point cloud of a 3D scene or to retrieve the camera trajectory from a sequence of images. 
ORB\_SLAM3~\cite{campos2021orb} is based on a previous SLAM research~\cite{murartal2015,murartal2017}.
ORB\_SLAM3 approach consists of tracking, mapping, relocalization, and loop closing based on ORB features extracted from the image frames. 
Camera pose information can be retrieved in real time for each frame after the initialization phase when the scene map is created. 
The quality of the result, aside from the quality of the input sequence, depends on the camera intrinsics, especially the focal length value.

\section{3D-Display-Friendly Frames Extraction}
\label{sec:extraction}
Suitable views of scenes for the 3D display can be captured intentionally, but many shots can be found, for example, in movies or gameplay videos.  
The method proposed in this section \cite{chlubna2024auto} can automatically extract suitable views of a scene from any video footage or detect that no such scene exists.
This proposal describes light field capturing from existing data.
It is important to ensure that such light field is suitable for the 3D display so that the main proposal of this thesis can also be applied in this field.

With minor adjustments, the method could be used to assist users during a capture process on a set.
Without the aid of the method, the user would need to use special equipment to ensure the optimal motion of the camera along the desired trajectory.
In case of already recorded videos, manual, time-consuming selection of the video frames would be necessary with additional editing and repeated evaluation on the display. 
Automatically extracted sequential frames, the so-called \textit{quilt}, are ready to be displayed on a 3D display.
LKG was used as a testing device.
This method proposes the entire pipeline to produce a visually appealing result. 
The input video is analyzed, suitable frames are extracted, resampled, and the optimal focus distance is identified to reduce out-of-focus artifacts; see Fig.~\ref{fig:lkgModel}.

\begin{figure*}[!htbp]\centering
  \begin{overpic}[width=.95\linewidth]{obrazky-figures/detection/lkgModel.pdf}
  \color{black}
  \put(-3,5){a)}
  \put(-3,1){b)}
  \put(75,-1.3){c)}
  \end{overpic}
  \caption{The figure is an overview of the proposed method: a) The identification of video frames captured by horizontally moving camera. b) Selection of desired images for the sequence. c) The conversion of desired images into proper format which can be displayed on the LKG. 
  Different virtual views are visible from different real-life viewing angles.}
  \label{fig:lkgModel}
\end{figure*}

\subsection{View Synthesis Approach}
One possibility of obtaining the views for LKG is to pick a few frames from a video, reconstruct the scene, and render the missing views.
The 3D reconstruction fails in most cases because the scene is captured only from one direction in the input views. 
3D reconstruction algorithms usually expect the scene to be captured from many different positions.
The suitable sequences for LKG do not contain views positioned freely in the space. 
For example, the expected LKG-friendly \textit{Barber} scene from the original dataset was used in INGP~\cite{mueller2022instant}, FILM~\cite{film-tf}, Meshroom, and 3DF Zephyr. 
The scene was reconstructed in the two programs, but the result is not optimal, as shown in Fig.~\ref{fig:3DRec}. 
INGP pre-processing of the data by Colmap failed. 
The FILM deep learning frame interpolation produced a frame between the boundary frames from the video. 
However, the frames had to be downscaled from FullHD to HD due to insufficient resources on NVIDIA\textsuperscript{\textregistered} RTX\textsuperscript{\texttrademark} 2070.
The interpolated frames contained visible artifacts even when the distance between the views was halved. 
RGBD photos can also be processed by LKG software and displayed, but the lack of parallax and missing reflections create the unwanted cardboard effect~\cite{yamanoue2000}. 

\begin{figure}[!htbp]
    \def \frameSize {0.49\linewidth}
  \centering
    \subfloat[\centering Meshroom reconstructed only standalone polygons.]{{\includegraphics[width=\frameSize]{obrazky-figures/detection/3DMeshroom.jpg} }}
    \subfloat[\centering 3DF Zephyr produced a distorted but full geometry.]{{\includegraphics[width=\frameSize]{obrazky-figures/detection/3DZephyr.jpg} }}
    \\
    \subfloat[\centering Original middle view was taken from the input light field.]{{\includegraphics[width=\frameSize]{obrazky-figures/detection/3DRealFull.jpg} }}
    \subfloat[\centering FILM produced a distorted middle view.]{{\includegraphics[width=\frameSize]{obrazky-figures/detection/3DInterFull.jpg} }}
    \\
    \subfloat[\centering Original quarter view was taken from the input light field.]{{\includegraphics[width=\frameSize]{obrazky-figures/detection/3DRealHalf.jpg} }}
    \subfloat[\centering FILM produced a distorted quarter view with less artifacts.]{{\includegraphics[width=\frameSize]{obrazky-figures/detection/3DInterHalf.jpg} }}
  \caption{\label{fig:3DRec}
Zephyr produced the best result when reconstructing the scene, but the result still contains a lot of artifacts.
The deep FILM interpolation produced distorted results for both full and halved distance between the views from \textit{Barber} scene in the used dataset.}
\end{figure}

The only reliable way to produce a good-quality quilt from videos is the proposed method with possible interpolation on a small scale if the views are not distributed linearly along the trajectory.
The proposed focusing metrics are still necessary for the extracted or interpolated views.
The user can capture the scene in a similar way as when taking a panorama photo with a mobile phone. 
The only difference is that instead of rotating the phone, the user would move to the side. 
Such captured footage would be automatically prepared by the proposed method for direct rendering on the 3D display.

\subsection{Extraction Overview}
The proposed method consists of the following steps:
\begin{enumerate}
  \item \textbf{Quilt extraction} where 
  the input video stream is analyzed and suitable frames are chosen for further processing. 
  The suitable frames are those that capture the scene having their camera positions lying on a horizontal trajectory.
  \item \textbf{Frame resampling} where
  The camera in the selected sequence can be subject of acceleration or uneven capturing frequency producing uneven spacing between the views.
  Equally-distanced frames are selected from the sequence to ensure the best visual quality of the rendered result.
  \item \textbf{Automatic focusing} solving the problem of 3D displays support for only one distance with a fairly limited depth of field, where the scene is visually sharp and focused. 
  The optimal focus distance is estimated to ensure that most of the scene is in focus.
\end{enumerate}

Sparse optical flow, dense optical flow, feature matching, and SLAM approaches were used to estimate the camera trajectory and compared in the reference implementation.
In the first phase, a rough camera motion estimation is performed, detecting truck and pan motion sequences. 
Pure truck sequences are ideal for LKG.
The camera can also do a combination of truck and pan, which might still be acceptable. 
Excessive panning motion reduces the parallax effect.
The second phase determines the amount of pan motion in the sequence.
Additional checks are performed, such as motion blur and shakiness detection. 
A score that evaluates the suitability of the sequence is the result of the second phase.
An approach using ORB\_SLAM3 does both tasks in one phase.

\subsection{Horizontal Sequence Detection}
Pairs of pixel blocks are identified as belonging to the same spot in the scene in two consecutive video frames.
The field of motion vectors (optical flow) is then known for each frame pair. 
The mean of these vectors is a good guide to determine the overall motion of the camera. 
Monocular visual SLAM methods also exploit a similar concept of feature matching in two frames, and based on them, the camera pose or scene map is estimated. 
Fig.~\ref{fig:lkgTrajectory} shows a simple scenario in which the ideal horizontal quilt sequence is mapped to a subset of suitable video frames.

\begin{figure}[!htbp]\centering
  \includegraphics[width=0.6\linewidth]{obrazky-figures/detection/lkgTrajectory.pdf}
  \caption{The \textcolor{blue}{camera~positions} on the trajectory belong to frames in the input video. 
  The frames sample the camera motion with a constant framerate.
  The required ideal \textcolor{mainOrange}{horizontal~sequence}, capturing the scene from a desired spot and with a given spacing, is then mapped on the closest input frames.}
  \label{fig:lkgTrajectory}
\end{figure}

A possible scene change can be detected by a simple histogram comparison or by setting a threshold for the estimated camera pose change.
The result of the analysis is an array of estimated camera pose differences between frames. 
The analysis phase is the most computationally expensive part of the processing.

In many cases, the mean value of these vectors can be used directly to decide whether the truck/pan motion is dominant in the shot. 
When the truck motion is combined with pan, the parallax can be inverted, depending on the distance of the objects in the scene from the camera.
To solve this issue, the direction of the sequence can be ignored, working only with the magnitude of horizontal motion. 
The method accepts the $i$th frame $f_{i}$ in the sequence $seq_i$ until the vertical velocity $\textrm{vel}_{vert}$ is above a defined threshold $t_{vert}$ or the horizontal velocity $\textrm{vel}_{hor}$ is close to zero; see Eq.~\ref{eq:acceptance} and Alg.~\ref{alg:sequence}.

\begin{equation}
seq_i =
  \begin{cases}
  accept & \text{if} \ \textrm{vel}_{hor}(f_{i}, f_{i-1}) >  0 \land \textrm{vel}_{vert}(f_{i}, f_{i-1}) \leq t_{vert}  \\
  reject & \text{otherwise}
  \end{cases}
	\label{eq:acceptance}
\end{equation}

\begin{algorithm}[ht]
 \KwData{Video frames sequence $fr$}
 \KwResult{Clips suitable for LKG quilt $clips$}
 $motions \oper{=} \fce{emptyArray}()$\;
 \ForEach {two consecutive frames prev and next$\oper{\in} fr$}{
    $motions \xleftarrow{\fce{insert}} \fce{motion}(prev,next)$\;
 }
 $clips \oper{=} \fce{emptyArray}()$\;
 $clips \xleftarrow{\fce{insert}}  \fce{emptyClip}()$\;
 \ForEach {id and motion$\oper{\in}motions$}{
    $\{x,y\} \oper{=} |motion_{xy}|$\;
    \uIf{$x \oper{>} x_{min}$ \textbf{and} $y \oper{<} y_{max}$}
    {
        $\fce{last}(clips) \xleftarrow{\fce{insert}} id$\;
    }
    \Else
    {
        \uIf{$\fce{hasSufficientLength}(\fce{last}(clips))$}
        {$clips \xleftarrow{\fce{insert}} \fce{emptyClip}()$\;}
        \Else
        {$\fce{last}(clips) \oper{=} \fce{emptyClip}()$\;}
    }
 }
 \caption{The algorithm iterates over the calculated camera offsets between all consecutive frame pairs and is adding frames into the sequence as long as the motion meets the requirements. 
 Note that in SLAM-based approaches, where the camera rotation is explicitly known, the maximal rotational bounds and checks can be implemented in addition to the vertical velocity test.}
 \label{alg:sequence}
\end{algorithm}

\subsection{Pan Elimination}
Fig.~\ref{fig:apparent} shows images composed of multiple video frames. 
The pan shots are very similar to the truck ones, except for the distortion in the corners of the field.
The angles between the motion vectors and the horizontal axis are higher than zero. 
Dense optical flow can be computed for each corner of the image pair.
The mean motion vector should point down on one side and up on the other side of the image or vice versa, according to the camera motion direction.
Each fifth pair is tested in the reference implementation for performance reasons.

\begin{figure}[!htbp]\centering
  \begin{overpic}[width=.24\linewidth]{obrazky-figures/detection/classt.jpg}
  \color{white}
  \put(0,0){\includegraphics[width=.24\linewidth]{obrazky-figures/detection/classt.pdf}}
  \put(2,2){\textbf{Truck}}
  \end{overpic}  
  \begin{overpic}[width=.24\linewidth]{obrazky-figures/detection/classp.jpg}
  \color{white}
  \put(0,0){\includegraphics[width=.24\linewidth]{obrazky-figures/detection/classp.pdf}}
  \put(2,2){\textbf{Pan}}
  \end{overpic}
  \begin{overpic}[width=.24\linewidth]{obrazky-figures/detection/classpat.jpg}
  \color{white}
  \put(2,2){\textbf{Pan and truck}}
  \put(0,0){\includegraphics[width=.24\linewidth]{obrazky-figures/detection/classpat.pdf}}
  \end{overpic}
  \includegraphics[width=0.24\linewidth]{obrazky-figures/detection/flowSegmentation.pdf}
 
  \definecolor{darkGreen}{rgb}{0.0,0.7,0.2}
  \caption{The apparent motion of the pixels is demonstrated in the images for truck, pan, and combined camera motion. 
  Frames from the testing sequences were mixed together and the image gradient of the result was \textcolor{darkGreen}{highlighted}.
  The rightmost image shows how pan could be identified from the motion field based on the previous images.}
  \label{fig:apparent}
\end{figure}

A score is calculated for each processed sequence; see Eq.~\ref{eq:score}.
The calculation of the score $s$ takes into account the overall amount of motion $blur$ in the sequence, $shake$ (amount of vertical motion), the average presence of the $pan$ pattern, and the difference $dist$ between the average amount of vertical motion at the center of the image $y_{cent}$ and at the corners $y_{cor}$ normalized by the maximal allowed limit $maxDiff$; see Eq.~\ref{eq:diff}.
The amount of blur is measured using the variance of Laplacian method~\cite{pech2000, pertuz2012}.
Lower-scored sequences are assumed to be more suitable for LKG. 
The weights $w_i$ can be adjusted according to the specific requirements of the quilt. 
In the measurements, the weights were set to $0.2, 0.5, 0.1, 0.2$ in this order.
The values were determined according to a previous study on user experience related to camera trajectory distortions on LKG~\cite{chlubna2024how}.
It shows that the pan pattern can negatively affect the result quickly, while shaking can be tolerated to a certain extent.

\begin{equation}
s=w_1 \cdot dist + w_2 \cdot pan + w_3 \cdot shake + w_4 \cdot blur
	\label{eq:score}
\end{equation}

\begin{equation}
dist=\textrm{min}\left (1, \frac{y_{cor}-y_{cent}}{maxDiff} \right )
	\label{eq:diff}
\end{equation}

In the case of ORB\_SLAM3 analysis, the camera pose matrix is decomposed into translation and rotation. 
The translation difference can be obtained simply by subtracting the translation vectors.
The rotation difference $diff_{rot}$ is estimated by comparing the rotational quaternions $\mathbf{q}_1$ and $\mathbf{q}_2$ according to Eq.~\ref{eq:quaternion}.
Pure pan motion is not suitable for SLAM reconstruction and is rejected automatically.

\begin{equation}
diff_{rot} = 1-\left|\frac{2\textrm{acos}(\mathbf{q}_1 \cdot \mathbf{q}_2)}{\pi}-1\right|
	\label{eq:quaternion}
\end{equation}

\subsection{Non-linear Camera Velocity and Noise}
The motion vectors are available from the analysis phase and can be used to improve the quality of the accepted sequence. 
The vertical component of the average motion vector can be used to reduce the shaking of the camera on the vertical axis. 
The camera motion perpendicular to the viewing plane (\textit{dolly} motion) can be used to change the scale of the images to reduce the noise on the depth axis. 

For the best viewing experience on LKG, equally distanced views of the scene are optimal~\cite{chlubna2024how}.
This requires a constant camera velocity and frame rate, which is not guaranteed in the input. 
If the frames are not equally spatially distanced, the sequence has to be sampled non-uniformly. 
The frame distance can be represented by a horizontal component of the average motion vector between frames. 
Horizontal position changes are accumulated in $acc$ frame-by-frame, with an increasing index $i$.
The nearest neighbor can be accepted as the $n$th frame in the resampled sequence $res_n$ when the accumulated motion exceeds the maximal allowed value $m_{max}$.
The difference is labeled as $d$ in Eq.~\ref{eq:delta}.
The conditions that decide the index of the accepted frame that is closer to the desired position are described in Eq.~\ref{eq:resample} and Alg.~\ref{alg:linear}.

\begin{equation}
d = acc-m_{max}
	\label{eq:delta}
\end{equation}

\begin{equation}
res_n =
  \begin{cases}
  i & \text{if} \ d < \left | d-\left |m_{i-1} \right | \right | \\ 
  i-1 & \text{otherwise}
  \end{cases}
	\label{eq:resample}
\end{equation}

\begin{algorithm}[ht]
 \KwData{X axis motion values for a sequence of selected frames $clip_{x}$}
 \KwResult{Sequence of frame indices with linearized motion $clip_{lin}$}
    $motion_{max} \oper{=} \fce{max}(clip_{x})$\;
    $acc \oper{=} |\fce{first}(clip_{x})|$\;
    $clip_{lin} \oper{=} \fce{emptyArray}()$\;
    \ForEach {two consecutive id and motion$\oper{\in} clip_{x}$}{
        $acc$ \oper{=} $ acc \oper{+} |motion_{next}|$\;
        \uIf{$acc \oper{\geq} motion_{max}$}
        {
            $delta \oper{=} acc\oper{-}motion_{max}$\;
            \uIf{delta $\oper{<} |delta\oper{-}|motion_{prev}||$}
            {$clip_{lin} \xleftarrow{\fce{insert}} id_{next}$\;}
            \Else
            {$clip_{lin} \xleftarrow{\fce{insert}} id_{prev}$\;}
            $acc$ \oper{=} $acc \oper{-} motion_{max}$\;
        }
    }
 \caption{The sequence is resampled in the algorithm to ensure a constant camera position difference between the frames. 
 Note that instead of choosing the closer frame according to the velocity accumulator, a frame interpolation can be used to create a new synthetic one.}
 \label{alg:linear}
\end{algorithm}

The distances between frames have to be large enough to create sufficient 3D perception. 
If the frames are too close, the LKG result looks flat without significant parallax. 
On the other hand, too large distance reduces the amount of focused area in the scene.
The sequence can be resampled according to the optimal frame distance; see Fig.~\ref{fig:lkgWindow}. 
The computation of the best alignment offset $o_{best}$ from the range ($o_{s}, o_{e}$) of the detected sequence is described in Eq.~\ref{eq:fitError}.
The $\mathbf{s_{optimal}}$ is a vector that contains the positions of the views in the optimal sequence with constant spacing.
The $\mathbf{s_{detected}}$ is a vector that contains the positions of the views in the sequence that was detected in the previous phase.
The optimal sequence is shifted by the offset $o$ and the error between the sequences is computed, looking for its minimum.
The vector and scalar addition is defined as the addition of the scalar value to all values in the vector.

\begin{equation}
	o_{best} = \mathop{\textrm{arg\,min}}_{o \in [o_s,o_e]}\{\left | (\mathbf{s_{optimal}}+o) - \mathbf{s_{detected}} \right |\}
	\label{eq:fitError}
\end{equation}

Eq.~\ref{eq:fitError} is implemented in practice by Alg.~\ref{alg:bestFit} that describes a quilt window with the desired distance.
This window is sliding over the extracted frames, and the error is computed from the distance between the actual frames and the positions in the window. 
The window position with the lowest error marks the best sequence according to the given requirement.
The optimal distance depends on the content of the scene and the user preferences.

\begin{figure}[!htbp]\centering
  \includegraphics[width=0.8\linewidth]{obrazky-figures/detection/lkgWindow.pdf}
  \caption{The \textcolor{gray}{middle row} depicts the video frames spaced according to their camera displacement.
  A quilt window with fixed spacing is placed at the beginning in the \textcolor{blue}{bottom~row} and at a better position in the \textcolor{orange}{top~row}. 
  The distances between the positions in the sampling windows and the positions of the nearest input frames are defining the overall \textcolor{red}{error}.}
  \label{fig:lkgWindow}
\end{figure}

\begin{algorithm}[!htbp]
 \KwData{Sequence of frame positions $pos$, desired spacing between the views $space$, location of interest in the input sequence $loc$, searching distance around the defined position $dist$, $resolution$ of the search, number of the quilt views $quiltSize$}
 \KwResult{An offset of the window with desired parameters with the lowest error $clip_{best}$}
    $clips \oper{=} \fce{emptyArray}()$\;
    $step\oper{=} \frac{2 \oper{\cdot} dist}{resolution}$\;
    $range \oper{=} (loc\oper{-}dist, loc\oper{+}dist)$\;
    \ForEach {$offset \oper{\in} range\ \textbf{with}\ step$}{
        $error\oper{=}0$\;
        $window \oper{=} \fce{emptyArray}()$\;
        \ForEach {$\left \{  i \oper{\in} \mathbb{Z} \mid 0 \oper{\leq} i \oper{<} quiltSize \right \}$}{
            $sample\oper{=}offset\oper{+}i \oper{\cdot} space$\;
            $nearest \oper{=} \fce{findNearest}(pos, sample)$\;
            $diff \oper{=} |nearest_{pos}\oper{-}sample|$\;
            $error \oper{=} error \oper{+} diff^2$\;
            $window \xleftarrow{\fce{insert}} nearest_{id}$\;
        }
        \tcp{if duplicates are not allowed}
        \uIf{$\fce{hasNoDuplicates}(window)$}
        {
            $clip \oper{=} \fce{newClip}(error, offset)$\;
            $clips \xleftarrow{\fce{insert}} clip$\;
        }
    }
    $clip_{best}\oper{=}\fce{lowestErrorClip}(clips)$\;
 \caption{A quilt window is sliding along the sequence, computing the squared distance between the closest samples and predefined positions in the window.
 The window position with the lowest error is then marked as the optimal one.}
 \label{alg:bestFit}
\end{algorithm}

\subsection{Focusing}
The display blends multiple frames to avoid discrete frame changes when users change viewing position.
Scene content positioned at \textit{zero-parallax plane} is always sharp and maintains its screen-space coordinates; see Fig.~\ref{fig:focusPlanes}.
The further the content is from the plane, the more parallax is present along with the out-of-focus artifacts. 
The zero-parallax plane position can be changed by shifting the views so that the same area of the scene lies on the same screen-space coordinates. 

\begin{figure}[!htbp]\centering
  \begin{overpic}[width=\linewidth]{obrazky-figures/detection/focusPlanes.pdf}
  \end{overpic}
  \caption{The first picture shows the 3D scene with two zero parallax plane positions. 
  The two other pictures are the actual results displayed on the LKG showing how the plane position affects the focusing in the rendered image.}
  \label{fig:focusPlanes}
\end{figure}

The positions of the start and end points of the acceptable focus range, where at least parts of the scene are focused, depend on the depth range of the captured scene; see Fig.~\ref{fig:focusExample}. 
The proposed method finds the optimal zero-parallax plane position that makes most of the scene focused.

\begin{figure}[!htbp]\centering
  \begin{overpic}[width=\linewidth]{obrazky-figures/detection/focusExample.pdf}
  \color{white}
  \put(2,70){\textbf{Start}}
  \put(2,37){\textbf{Optimum}}
  \put(2,3){\textbf{End}}
  \end{overpic}
  \caption{Scene is focused at the borders of the focus range and in the optimal distance where most of the scene appears to be focused.}
  \label{fig:focusExample}
\end{figure}

The unique internal image format, displayed directly on the LKG screen, where all quilt views are combined into one image, is analyzed. The areas of the image outside the zero-parallax plane are distorted by a diagonal blur-like pattern. 
The more the pattern is visible, the less focused the given part of the image is on LKG.
Two focus estimation metrics are proposed. 

The first is a difference of Gaussians (DoG) edge detection on the internal LKG format image.
The second performs a subtraction of two internal LKG images with different display calibration settings.
The diagonal blur pattern is different for each calibration. 
In both cases, the out-of-focus pixels in the result contain high values. 
A similar principle was used in all-focused light field research~\cite{kubotaa2004all}. 
Fig.~\ref{fig:focusMetric} shows both the metrics and their results. 

\begin{figure}[!htbp]\centering
  Out of focus
  \vspace{5pt}
  \\
  \begin{overpic}[width=.49\linewidth]{obrazky-figures/detection/focus3_354.jpg}
  \color{white}
  \put(2,5){\textbf{Pitch: 354}}
  \put(0,0){\includegraphics[width=.49\linewidth]{obrazky-figures/detection/focus03_354_over.pdf}}  
  \end{overpic}
  \begin{overpic}[width=.49\linewidth]{obrazky-figures/detection/focus3_354DOG.jpg}
  \color{white}
  \put(2,51){\textbf{DoG}}
  \put(2,5){\textbf{Avg: 47.6676}}
  \end{overpic}
  \\
  \begin{overpic}[width=.49\linewidth]{obrazky-figures/detection/focus3_100.jpg}
  \color{white}
  \put(2,5){\textbf{Pitch: 100}}
  \put(0,0){\includegraphics[width=.49\linewidth]{obrazky-figures/detection/focus03_100_over.pdf}}  
  \end{overpic}
  \begin{overpic}[width=.49\linewidth]{obrazky-figures/detection/focus3SUB.jpg}
  \color{white}
  \put(2,51){\textbf{Subtraction}}
  \put(2,5){\textbf{Avg: 20.4949}}
  \end{overpic}
  \\
  Most of the scene in focus
  \vspace{5pt}
  \\
  \begin{overpic}[width=.49\linewidth]{obrazky-figures/detection/focus093_354.jpg}
  \color{white}
  \put(2,5){\textbf{Pitch: 354}}
  \put(0,0){\includegraphics[width=.49\linewidth]{obrazky-figures/detection/focus093_354_over.pdf}}  
  \end{overpic}
  \begin{overpic}[width=.49\linewidth]{obrazky-figures/detection/focus093_354DOG.jpg}
  \color{white}
  \put(2,51){\textbf{DoG}}
  \put(2,5){\textbf{Avg: 25.2654}}
  \end{overpic}
  \\
  \begin{overpic}[width=.49\linewidth]{obrazky-figures/detection/focus093_100.jpg}
  \color{white}
  \put(2,5){\textbf{Pitch: 100}}
  \put(0,0){\includegraphics[width=.49\linewidth]{obrazky-figures/detection/focus093_100_over.pdf}}  
  \end{overpic}
  \begin{overpic}[width=.49\linewidth]{obrazky-figures/detection/focus093SUB.jpg}
  \color{white}
  \put(2,51){\textbf{Subtraction}}
  \put(2,5){\textbf{Avg: 10.4709}}
  \end{overpic}
  \caption{Focused and out-of-focus images are shown. 
  Left ones are in the LKG internal format with two different display calibration settings.
  The first proposed metric is the difference of Gaussians from a single LKG image, and the second one is a subtraction of the two images with different calibrations. 
  The more focused the image, the less energy (right images) the metrics yield.}
  \label{fig:focusMetric}
\end{figure}

The algorithm iterates over a wide focus range and searches for the minimum of the given metric, as shown in Fig.~\ref{fig:focusingChart}.
The principle is the opposite of the standard blur measurement methods~\cite{elder1998local}.
The Gaussian metric is described by Eq.~\ref{eq:gaussMetric} and subtraction by Eq.~\ref{eq:subMetric}.
The optimal focus distance $f_{o}^{DoG}$ and $f_{o}^{sub}$ searched as $f$ in the interval $(f_s,f_e)$ is computed, for the first metric, as a difference of internal LKG format images $I_{lkg}$ with calibration parameter $c_1$, processed by Gaussian filter $G$ with filter parameters $p_1$ and $p_2$.
The second metric subtracts two different internal images with calibration parameters $c_1$ and $c_2$. 
The DoG parameters used in the measurement were $radius=10, \sigma_1=0, \sigma_2=10$, but different reasonable parameters showed the same results.
The calibration parameters used in Eq.~\ref{eq:internal}, $c_1$ were set to $t=-0.1153, p=354.42108, c=0.04239, v=0.99976$ according to a physical device used in the tests, and the parameters $c_2$ were changed to $t=0.05, p=150$ which showed the most visible changes after subtraction.

\begin{equation}
	f_{o}^{DoG} = \mathop{\textrm{arg\,min}}_{f \in [f_s,f_e]}\left \{G(p_1, \textrm{I}_{lkg}(c_1, f))-\textrm{G}(p_2, \textrm{I}_{lkg}(c_1, f))\right \}
	\label{eq:gaussMetric}
\end{equation}

\begin{equation}
	f_{o}^{sub} = \mathop{\textrm{arg\,min}}_{f \in [f_s,f_e]}\left \{\textrm{I}_{lkg}(c_1, f)-\textrm{I}_{lkg}(c_2, f)\right \}
	\label{eq:subMetric}
\end{equation}

\begin{figure}[!htbp]
\centering
    \pgfplotstableread{charts/detection/focusTeddy.txt}\focus
    \begin{tikzpicture}
    \begin{axis}[
     width=.6\linewidth,
     height=.5\linewidth,
      ylabel={$Average\ image\ energy$},
      xlabel={$Focus\ distance$},
      legend style={at={(0.45,0.53)},anchor=east},
      legend entries={optimum, \textbf{Energy of}, DoG, subtraction},
      ymin=9
    ]
    \addplot +[dashed, color=red, mark=none] coordinates {(0.093, 0) (0.093, 52)};
    \addlegendimage{opacity=0.0}
    \addplot[color=mainBlue, mark=none, ultra thick] table[x=focus, y=dog]{\focus};
    \addplot[color=mainOrange, mark=none, ultra thick] table[x=focus, y=sub]{\focus};
    \end{axis}
    \end{tikzpicture}
    \caption{Two focusing metrics are plotted with manually marked optimum distance where most of the image is focused. 
    The location of the global minimum of the energy marks the best focus.}
    \label{fig:focusingChart}
\end{figure}

\subsection{Results \& Discussion}\label{sec:eval}
The overall quality of the detector was evaluated in terms of efficiency and performance, see Tab.~\ref{tab:resultsDetector}, in the following experiments:
\begin{enumerate}
  \item \textbf{Quilt extraction} which aim was to measure the accuracy of automatic quilt extraction from a video.
  The classification is evaluated using an annotated dataset. 
  \item \textbf{Frame resampling} experiment which proves that frames can be sampled in an optimal way, resulting in a lower error compared to a simple first frame alignment.
  \item \textbf{Automatic focusing} of the resulting quilt proves that the optimal focus distance where most of the scene lies on the zero-parallax plane can be found automatically, which makes the resulting quilt ready to be displayed.
\end{enumerate}

\begin{table}[ht]\centering
\begin{tabular}{lcc}
\multicolumn{3}{c}{\textbf{Horizontal sequence extraction}}\\
 \midrule
 \textbf{analysis method} & \textbf{true [\%]} & \textbf{false [\%]} \\
 \midrule
 \textcolor{red}{ORB\_SLAM3} & \textcolor{red}{79.7} & \textcolor{red}{20.3} \\
 ORB & 75.6 & 24.4 \\
\\

\multicolumn{3}{c}{\textbf{Optimal window positioning}}\\
 \midrule
  \textbf{scene} & \multicolumn{2}{c}{\textbf{squared error}} \\
 & \textbf{first frame aligned} & \textcolor{red}{\textbf{optimized}} \\
 \midrule
 \textit{teddy} & 382.988 & \textcolor{red}{251.594} \\
 \textit{hut} & 502.925 & \textcolor{red}{300.505} \\
 \textit{class} & 115.437 & \textcolor{red}{82.4649} \\
 \textit{pavilion} & 88.3094 & \textcolor{red}{72.9947} \\
\\

\multicolumn{3}{c}{\textbf{focus distance detection}}\\
 \midrule
  \textbf{method} & \multicolumn{2}{c}{\textbf{absolute error}} \\
 & \textbf{DoG} & \textcolor{red}{\textbf{subtraction}} \\
 \midrule
 average & 0.0475 & \textcolor{red}{0.0441} \\
 min & 0 & \textcolor{red}{0} \\
 max & 0.549 & \textcolor{red}{0.549} \\
\end{tabular}

\caption{\label{tab:resultsDetector}The table contains important results from the experiments with the best results \textcolor{red}{highlighted}. 
ORB\_SLAM3 and ORB showed the best results for quilt extraction. True values are considered $TN+TP$ and false $FN+FP$ ($T/F$ is true/false, $P/N$ is positive/negative) according to the annotated dataset.
The alignment errors of the first frame of the quilt window and optimal placement are compared, proving that the proposed algorithm can find a better position in the sequence.}
\end{table}

OpenCV implementations of the Lucas-Kanade, Farneback, and ORB methods were used along with the original ORB\_SLAM3 implementation for the quilt extraction. 
All experiments were executed on a machine equipped with NVIDIA\textsuperscript{\textregistered} GeForce RTX\textsuperscript{\texttrademark} 2070 GPU and Intel\textsuperscript{\textregistered} Core\textsuperscript{\texttrademark} i5-8500 CPU 3.00 GHz CPU, running Arch Linux. 

The dataset consists of 182 FullHD, 25 fps, synthetic and handheld camera scenes encoded in H.265 format.
The scenes are divided into 7 categories according to the dominant camera motion: chaotic, dolly, pan, pan and truck combined, pedestal, tilt, and truck.
The duration of the videos is from $2$ to $13$ seconds.
The videos were captured to cover all types of scenes with various depths, close and far objects of interest, etc.
The synthetic shots were rendered in Blender with pre-made scenes from Blender demo files.
They were captured in two versions: with ideal smooth camera motion and with additional shaking and motion blur. 
The real-life videos were captured by Panasonic HC-VX980 Camcorder without any special equipment.

\subsection{Quilt Extraction}\label{sec:evalHor}
The desired point in Fig.~\ref{fig:roc} marks the maximal vertical motion between two frames that is accepted as valid.
The position was empirically determined by watching the amount of artifacts on LKG.
An experiment was conducted in which identical frames with a simple pattern were vertically shifted with an increasing offset. 
The first kind of artifact is caused by the mixing of the frames, which occurs throughout the displaying range to simulate the binocular parallax.
Fig.~\ref{fig:checkerDisp} shows the difference between the vertically displaced quilt and the aligned one. 
These artifacts start to occur with displacement around 1.5 px.
The second type is visible in between two frame positions in the monocular motion parallax when watching the display from a close distance. 
This kind of artifact is more sensitive to displacement and was detected starting at 0.5 px displacement. 
A value between these two, 1 px, was selected as desired; see Fig.~\ref{fig:lkgTolerance}.

%----------------------------------------
%ROC OUR DATASET
\begin{figure}[!htbp]
\centering
\pgfplotstableread{charts/detection/m0g1.txt}\methodA
\pgfplotstableread{charts/detection/m1g1.txt}\methodB
\pgfplotstableread{charts/detection/m2g1.txt}\methodC
\pgfplotstableread{charts/detection/slam.txt}\methodD

\begin{tikzpicture}
\begin{axis}[
  xlabel={$False\ positive\ rate$},
  width=.8\linewidth,
  height=.7\linewidth,
  ylabel={$True\ positive\ rate$},
  xmin=0,
  ymin=0,
  legend style={at={(0.98,0.3)},anchor=east},
  legend entries={Lucas-Kanade, Farneback, ORB, ORB\_SLAM3, Best ratio point, Desired point}
]
\addplot[color=mainOrange, mark=none, ultra thick]table[x=FPR, y=TPR]{\methodA};
\addplot[color=mainBlue, mark=none, ultra thick] table[x=FPR, y=TPR]{\methodB};
\addplot[color=green, mark=none, ultra thick] table[x=FPR, y=TPR]{\methodC};
\addplot[color=yellow, mark=none, ultra thick] table[x=FPR, y=TPR]{\methodD};
\addplot[only marks, mark=*, color=red, mark size=6pt] coordinates {(0.206611570247934,0.836065573770492)(0.302521008403361, 0.885245901639344)};
\pgfplotsset{
every axis plot post/.append style={mark=+},
} 
\addplot[only marks, mark=+, color=black, mark size=6pt] coordinates {(0.223140495867769,0.836065573770492)(0.327731092436975,0.9180327868852462)};
\addplot[domain=0:1, color=gray, dashed]{x};
\end{axis}
\end{tikzpicture}

\caption{\label{fig:roc}
  The ORB\_SLAM3 and ORB are the best candidates for quilt detection. The threshold parameter is the limit of vertical motion. 
  ORB\_SLAM3 does not cover the whole ROC range due to its inability to analyze all the sequences. 
  The desired point (1 px vertical motion tolerance) was chosen according to empirically found maximal acceptable threshold and the best ratio (0.9 px tolerance) according to the commonly used metrics.}
\end{figure}

\begin{figure}[!htbp]\centering
  \begin{overpic}[width=.9\linewidth]{obrazky-figures/detection/checkerNoDisp.jpg}
  \end{overpic}
  \vspace{10pt}\\
  \begin{overpic}[width=.9\linewidth]{obrazky-figures/detection/checkerDisp.jpg}
  \end{overpic}
  \caption{The first picture shows a checkerboard quilt with views that are aligned in one horizontal line. 
  The second picture shows the same quilt with vertical displacement between views where artifacts are visible.}
  \label{fig:checkerDisp}
\end{figure}

\begin{figure}[!htbp]\centering
  \begin{overpic}[width=.98\linewidth]{obrazky-figures/detection/lkgTolerance.pdf}
  \color{black}
  \put(18, 98){Testing pattern}
  \put(63,98){Pattern on LKG}
  \put(19,51){Center of frame}
  \put(67,51){Between frames}
  \put(-1,44){0.2}
  \put(-1,38){0.5}
  \put(-1,31){0.7}
  \put(-1,25){0.9}
  \put(-1,19){1.5}
  \put(-1,13){2.0}
  \end{overpic}
  \caption{A simple pattern was chosen to measure how vertical motion limit affects the quality of the result. 
  The border of the pattern is getting distorted with the increasing vertical displacement (values on the left). 
  The breaking points were detected at 1.5 px in the center case and 0.5 px in the in-between case.}
  \label{fig:lkgTolerance}
\end{figure}

Three metrics, the Youden index, closest to point $(0,1)$, and maximum area, were used to identify the best ratio threshold in the ROC curves~\cite{dong2015}, which corresponds to a vertical motion tolerance of 0.9 px.
Fig.~\ref{fig:datasetPositive} shows the distribution of negative and positive results among the categories in the dataset.
Fig.~\ref{fig:precision} shows the accuracy and precision results.

ORB-based feature detection method outperforms the optical flow approach. 
ORB\_SLAM3 shows better results than all other approaches, but it is limited by its inability to analyze all possible sequences. Most of the problematic sequences are true negatives, but some positive sequences were also rejected. 
That is the reason why its ROC curve does not reach the right top corner as the rest. 

An unknown focal length is assumed in the measurements, according to the use case, where the user can provide a random video without further information.
The focal length was estimated by Meshroom photogrammetry software, as it is a necessary parameter for ORB\_SLAM3 method. 

The results of the computational time measurements are presented in Fig.~\ref{fig:timesDetect}.
ORB might be more versatile and robust, while ORB\_SLAM3 is more accurate in classification.
However, the efficiency of ORB\_SLAM3, depends on the focal length estimation, which might be problematic in scenes shot by multiple different cameras or shots containing zooming.

\begin{figure*}[!htbp]
\pgfplotsset{every axis plot post/.append style={ultra thick, draw=none, mark=none}}
\pgfplotstableread{charts/detection/datasetFalse.txt}\datasetFalse
\pgfplotstableread{charts/detection/datasetTrue.txt}\datasetTrue
\pgfplotstableread{charts/detection/datasetFalseSLAM.txt}\datasetFalseSLAM
\pgfplotstableread{charts/detection/datasetTrueSLAM.txt}\datasetTrueSLAM
\begin{tikzpicture}
\begin{axis}[xbar,yticklabels from table={\datasetFalse}{category},
  bar width=0.3cm,
    enlarge x limits={abs=0pt},
  height=.3\linewidth,
  width=.5\linewidth,
  ytick=data,
  ylabel={$Camera\ motion$},
  xlabel={$ORB$},
  legend style={at={(0.8,0.8)},anchor=east},
  area legend,
  xmin=0,
]
\addplot[color=white, fill=mainBlue, forget plot] table[x=positive,y expr=\coordindex]{\datasetFalse};
\addlegendimage{area legend,fill=mainBlue,draw=white, mark=bar}
\addlegendentry{False};
\addplot[color=white, fill=mainOrange] table[x=positive,y expr=\coordindex]{\datasetTrue};
\addlegendimage{area legend,fill=mainOrange,draw=white, mark=bar}
\addlegendentry{True};
\end{axis}
\end{tikzpicture}
\begin{tikzpicture}
\begin{axis}[xbar,yticklabels from table={\datasetFalseSLAM}{category},
  bar width=0.3cm,
    enlarge x limits={abs=0pt},
  height=.3\linewidth,
  width=.5\linewidth,
  ytick=data,
  xlabel={$ORB\_SLAM3$},
  yticklabels={,,},
  xmin=0,
]
\addplot[color=white, fill=mainBlue, forget plot] table[x=positive,y expr=\coordindex]{\datasetFalseSLAM};
\addplot[color=white, fill=mainOrange] table[x=positive,y expr=\coordindex]{\datasetTrueSLAM};
\end{axis}
\end{tikzpicture}
\put(-220,-10){$Positive\ [\%]$}
\caption{\label{fig:datasetPositive} The amount of positive ORB and ORB\_SLAM3 results in each category of the dataset is depicted.
Only pure truck or truck-and-pan combined camera motions are true positives.
Truck has the most positive results, followed by pan and truck, which can still be displayed but are prone to artifacts and a flat look.}
\end{figure*}

\begin{figure*}[!htbp]
\pgfplotsset{every axis plot post/.append style={ultra thick, draw=none, mark=none}}
\pgfplotstableread{charts/detection/precision.txt}\precision
\pgfplotstableread{charts/detection/accuracy.txt}\accuracy
\begin{tikzpicture}
\begin{axis}[xbar,yticklabels from table={\accuracy}{method},
  bar width=0.3cm,
  height=.32\linewidth,
  width=.47\linewidth,
  ytick=data,
  ylabel={$Method$},
  xlabel={$Accuracy$},
  xmin=0,
    enlarge x limits={abs=0pt},
  enlarge y limits={abs=2*\pgfplotbarwidth}
]
\addplot[color=white, fill=mainOrange] table[x=first,y expr=\coordindex]{\accuracy};
\addplot[color=white, fill=mainBlue] table[x=both,y expr=\coordindex]{\accuracy};
\node[color=mainBlue] at (axis cs:0.1,3.2) {$N/A$};
\end{axis}
\end{tikzpicture}
\begin{tikzpicture}
\begin{axis}[xbar,yticklabels from table={\precision}{method},
  ytick=data,
  bar width=0.3cm,
  height=.32\linewidth,
  width=.47\linewidth,
  yticklabels={,,},
  xlabel={$Precision$},
  xmin=0,
    enlarge x limits={abs=0pt},
  legend style={at={(0.8,0.98)},anchor=east},
  legend entries={First phase, Both phases},
  area legend,
  enlarge y limits={abs=2*\pgfplotbarwidth}
]
\addplot[color=white, fill=mainOrange] table[x=first,y expr=\coordindex]{\precision};
\addplot[color=white, fill=mainBlue] table[x=both,y expr=\coordindex]{\precision};
\node[color=mainBlue] at (axis cs:0.1,3.2) {$N/A$};
\end{axis}
\end{tikzpicture}
\caption{\label{fig:precision} The figure shows precision and accuracy values for all methods.
The vertical motion limit is set to 1 px. 
ORB\_SLAM3 method is the best choice for the analysis.}
\end{figure*}


\begin{figure}[!htbp]
\pgfplotsset{every axis plot post/.append style={ultra thick, draw=none, mark=none}}
\pgfplotstableread{charts/detection/timesCPU.txt}\timesCPU
\pgfplotstableread{charts/detection/timesGPU.txt}\timesGPU
\begin{tikzpicture}
\begin{axis}[xbar,yticklabels from table={\timesCPU}{method},
  bar width=0.3cm,
  height=.4\linewidth,
  width=.8\linewidth,
  ylabel={$Method$},
  ytick=data,
  xlabel={$Time\ per\ frame\ [s]$},
   x tick label style={
        /pgf/number format/.cd,
            fixed,
            fixed zerofill,
            precision=2,
        /tikz/.cd
    },
  legend style={at={(0.75,0.6)},anchor=east},
  legend entries={CPU, GPU},
  area legend,
    enlarge x limits={abs=0pt},
  xmin=0,
  reverse legend,
  enlarge y limits={abs=0.5cm},
  bar width=1cm,
]
\addplot[color=white, fill=mainBlue] table[x=time,y expr=\coordindex]{\timesCPU};
\addplot[color=white, fill=mainOrange] table[x=time,y expr=\coordindex]{\timesGPU};
\node[color=mainOrange] at (axis cs:0.05,3.25) {$N/A$};
\end{axis}
\end{tikzpicture}
\caption{\label{fig:timesDetect} The chart compares the computation times of both CPU and GPU implementations of the analysis methods. 
Times were measured and averaged over the whole dataset. 
The resolution of the processed sequences is $1920{\times}1080$.}
\end{figure}

\subsection{Frame Resampling}\label{sec:evalWind}
\textit{teddy} sequence from the \textit{truck} category in the dataset was chosen because it is recorded with a handheld camera and is bound to have different spaces between the frames. 
Fig.~\ref{fig:windowErr} shows how the resampling algorithm selects the optimal offset within the defined range to minimize error. 

\begin{figure}[!htbp]
\pgfplotsset{every axis plot post/.append style={ultra thick, draw=none, mark=none}}
\pgfplotsset{width=.4\textwidth, height=0.33*\textwidth,xmin=0,ymin=0.1}
\pgfplotstableread{charts/detection/window.txt}\window
\begin{tikzpicture}
\begin{axis}[xbar,
 bar width=0.3cm,
 ylabel={$Scene$},
 y tick label style={font=\itshape},
  height=.4\linewidth,
  width=.92\linewidth,
    enlarge x limits={abs=0pt},
 yticklabels from table={\window}{dataset},
 legend entries={First frame aligned, Lowest error},
  area legend,
  ytick=data,
  xlabel={$Absolute\ error$},
  enlarge y limits=0.2,
  reverse legend,
]
\addplot[color=white, fill=mainBlue] table[x=aligned,y expr=\coordindex]{\window};
\addplot[color=white, fill=mainOrange] table[x=best,y expr=\coordindex]{\window};

\end{axis}
\end{tikzpicture}
\caption{\label{fig:windowErr} The chart shows the results of the experiment with the adjustment of the position of the quilt window in scenes from the \textit{truck} category.
The distance between the frames was twice the average distance in the sequence to avoid duplicates.
The search range was set to one quarter of the distance.}
\end{figure}

\subsection{Automatic Focusing}\label{sec:evalFocus}
The focusing metrics were evaluated on the quilts created from the truck sequences. 
The range and optimal value were manually annotated, evaluating the result on LKG. 
The absolute error of the estimated optimal value and the annotated one was calculated and normalized by the whole scanning range. 
SMAPE was used as a second evaluation metric. 
The results of the two proposed focusing metrics do not differ significantly, as shown in Fig.~\ref{fig:focusErr}. 
The optimal focus distance is in all quilts very close to the metric minimum. 

\begin{figure}[!htbp]
\makebox{\rotatebox[origin=c]{90}{\centering $Focus\ metric$}}
\pgfplotsset{every axis plot post/.append style={ultra thick, draw=none, mark=none}}
\pgfplotsset{width=\textwidth, height=0.19*\textwidth,xmin=0,ymin=0.1}
\pgfplotstableread{charts/detection/focusErr.txt}\focusErr
\begin{minipage}[c]{0.88\linewidth}
\begin{tikzpicture}
\begin{axis}[xbar,yticklabels={DoG, Subtraction},
  ytick=data,
enlarge x limits={abs=0pt},
 bar width=0.3cm,
  xmin=0.044,
  xmax=0.048,
  legend style={at={(0.95,1.3)},anchor=east},
  x tick label style={
    /pgf/number format/fixed,
     /pgf/number format/precision=5
    },
    scaled x ticks=false,
      enlarge y limits={abs=0.3cm},
  bar width=1cm,
]
\addplot[color=white, fill=mainBlue, forget plot] table[x=abs,y expr=\coordindex]{\focusErr};
\addlegendimage{area legend,fill=mainBlue,draw=white, mark=bar}
\addlegendentry{Normalized absolute error};
\addlegendimage{area legend,fill=mainOrange,draw=white, mark=bar}
\addlegendentry{SMAPE};
\end{axis}
\end{tikzpicture}
\\
\begin{tikzpicture}
\begin{axis}[xbar,yticklabels={DoG, Subtraction},
  bar width=0.3cm,
  enlarge x limits={abs=0pt},
  xtick={0.2259, 0.226, 0.2261, 0.2262},
  ytick=data,
  xmin = 0.2259,
  xmax = 0.2262,
  yshift = 100pt,
  xlabel={$Error\ [\%]$},
    x tick label style={
    /pgf/number format/fixed,
     /pgf/number format/precision=5
    },
    scaled x ticks=false,
      enlarge y limits={abs=0.3cm},
  bar width=1cm,
]
\addplot[color=white, fill=mainOrange] table[x=smape,y expr=\coordindex]{\focusErr};
\end{axis}
\end{tikzpicture}
\end{minipage}
\caption{\label{fig:focusErr} Bars are showing the errors of both focus estimation metrics. 
The differences are not significant.}
\end{figure}

\subsection{Comparison to Existing Methods}
To prove the novelty of the proposal, the method was compared experimentally with similar methods. 
No alternative full-quilt detection methods for 3D displays were proposed before.
Only the subtasks of the proposal were compared in standalone tests.
The results show that the proposal cannot be replaced by existing standard approaches.

\subsubsection{Quilt Extraction}
3D reconstruction software, such as Meshroom, can be used to estimate camera poses, but it takes minutes to process one frame compared to near real-time processing proposed in this paper. 
It would also be problematic to process a video file without knowing how many frames are necessary to include in the computation.
The proposal is focused on processing of the two subsequent frames, retaining the context of the previous results.
In most cases, two frames are not enough for a successful 3D reconstruction.

Panorama stitching software is another option as it detects the pan motion with possible shaking when the capturing device is handheld. 
Experiments with the Autostitch program~\cite{Brown2007} showed that pure truck motion shots can be properly detected as a suitable sequence and processed by the software.
However, the combination of pan and truck does not yield acceptable results, and this category of potentially suitable quilts would be rejected.
Simple averaging of motion vectors from optical flow would also make the distinction between small amount of camera vertical shake, pan, and truck impossible.

Research in video analysis~\cite{rao2020unified} usually focuses only on the general classification of the shot type such as \textit{moving} or \textit{static} and these approaches cannot be used for the quilt detection.
CAMHID~\cite{hasan} method can be used to classify camera motion, but the results presented by the authors mention only static, pan, tilt, and zoom categories without the necessary distinction between pan and truck, which is necessary for the proposed method.
Similar categories translation (pan and/or tilt), zoom, and static are defined in other works~\cite{guironnet2006, delibasoglu2022, wang2008}.

Truck motion is distinguished from pan in a novel deep learning approach of camera motion detection for story and multimedia information convergence~\cite{bak2023camera} where $8$ frame intervals are processed.
An advanced motion vector extractor preprocesses the shots, and ResNet is used to classify the resulting sequences.

The available implementation of the network was trained with the same data that the authors provided.
The network was then used to measure its accuracy on the data tested in this paper to determine if the network can be used instead of the proposal.
The network can classify a shot into pan, pedestal, tilt, truck, zoom-in, and zoom-out categories.
This classification is not detailed enough for the quilt detection because truck with a small amount of pan is also acceptable.
This motion is not distinguishable by the network.
The measurement was slightly adjusted in favor of the network approach, and any shot from the combined truck and pan category classified by the network as either standalone pan or truck is considered to be successful true positive, and false classification between other categories is not penalized.
Tab.~\ref{tab:comparisonResults} shows that the proposal in this paper is more suitable for the task than the alternative deep learning network.

\begin{table}[h!]\centering
\begin{tabular}{lcc}
 \textbf{analysis method} & \textbf{true [\%]} & \textbf{false [\%]} \\
 \midrule
 \textcolor{red}{proposed} & \textcolor{red}{79.7} & \textcolor{red}{20.3} \\
 deep~\cite{bak2023camera} & 68.1 & 46.9 \\
\\
\end{tabular}
\caption{\label{tab:comparisonResults}The table compares alternative existing detection method with the proposal. True values are considered $TN+TP$ and false $FN+FP$ according to the custom annotated dataset.}
\end{table}

The frames containing the motion vectors need to be downsampled to resolution of $600{\times}300$ compared to $1920{\times}1080$ in this paper's proposal.
The time of the classification for one frame in the network is $0.006$ seconds and the time of optical flow preprocessing is $0.1$ seconds compared to $0.025$ seconds for ORB and $0.14$ seconds for ORB\_SLAM3 in the proposal of this paper.
The proposal is not a real-time solution and both compared methods can be labeled as the same speed category since the difference is not significant.
If the same resolution frames were used as in the proposal, the deep classification time would be at least $0.07$ seconds and the preprocessing would be $1.15$ seconds due to the processing of $11.52{\times}$ more data.

\subsubsection{Focusing}
To prove that a novel focusing metric is necessary for the quilt detection task, an attempt to identify focus distance with three other methods was conducted.
The first is a standard contrast-based method~\cite{shih2007} with root mean square (RMS) contrast metric~\cite{kukkonen1993}.
The second is a no-reference image quality metric LIQE~\cite{zhang2023liqe} which should be able to detect various artifacts and defocused areas.
Both metrics were used on the internal LKG format like in the proposal and on the blended views according to Eq.~\ref{eq:focus} with all weights set to the same constant, resulting in averaging of the refocused images.
The third is a PSNR comparison of the refocused views according to the central one.
Each view is shifted according to the focus distance, compared with the central one with the PSNR metric, and the average of the PSNR values is taken as the metric.
The results are shown in Fig.~\ref{fig:focusComparison}.

The maximal values of both metrics on the internal LKG format are close to the optimum but not as close as proposed metrics in Fig.~\ref{fig:focusingChart}.
The error of the proposed metrics on \textit{teddy} scene is only 15 px in the focus displacement while the contrast metric error is 36 px which is more than $2{\times}$ higher than in the proposal.
The charts of the proposed metrics also show steeper curve towards the minimum which identifies the area of potential other focus distances that the user might want to choose. 
The slope of the alternative methods is either noisy or not that steep so such identification would be more problematic.
The idea of evaluation of the internal format is proposed in this paper, so the internal versions of the metrics are only semi-alternative.
The versions with blending, which can be considered as an alternative approach, are not suitable at all.
The conventional methods cannot be used in this case.

\begin{figure}[!htbp]
    \pgfplotsset{every axis plot post/.append style={very thick, mark=none}}
    \centering
    \pgfplotstableread{charts/detection/focusComp.txt}\focus
    \begin{tikzpicture}
    \begin{axis}[
    width=.6\linewidth,
     height=.5\linewidth,
      ylabel={$Metric\ normalized\ value$},
      xlabel={$Focus\ distance$},
      legend style={at={(1.6,0.5)},anchor=east, fill opacity=0.7, text opacity=1, draw opacity=1, legend columns=2},
      legend entries={optimum, psnr, \textbf{contrast}, \ , blend, internal, \textbf{LIQE}, \ , blend, internal}
    ]
    \addplot +[dashed, color=red] coordinates {(0.093, 0) (0.093, 1)};
    \addplot[color=red] table[x=focus, y=psnr]{\focus};
    \addlegendimage{opacity=0.0}
    \addlegendimage{opacity=0.0}
    \addplot[color=mainOrange] table[x=focus, y=contBlend]{\focus};
    \addplot[color=mainOrange, dashed] table[x=focus, y=contInt]{\focus};
    \addlegendimage{opacity=0.0}
    \addlegendimage{opacity=0.0}
    \addplot[color=mainBlue] table[x=focus, y=liqeBlend]{\focus};
    \addplot[color=mainBlue, dashed] table[x=focus, y=liqeInt]{\focus};
    \end{axis}
    \end{tikzpicture}
    \caption{Alternative existing metrics that can be used to measure focus in image were compared on \textit{teddy} scene. The values were transformed to the range of (0,1) because only possible minimum or maximum are the desired information. The results show that these metrics are not suitable for 3D displays. The metrics were tested on internal LKG format and blended views.
    \label{fig:focusComparison}}
\end{figure}

\newpage
\section{Focusing Artifacts Mitigation}
\label{sec:artifactsMit}
Another study was conducted to improve the quality of the quilts displayed on LKG \cite{chlubna2024out}.
The focusing on 3D displays and possible transformations of the input scenes are necessary to ensure the best viewing comfort for the user.
The light field rendering method proposed in this thesis is capable of producing the input views for 3D displays but does not address the necessary adjustments of such views for optimal viewing on the devices.
This section addresses this issue.

The novel saliency-based autofocusing method was compared to the previously proposed one and proved to be more efficient. 
Additionaly, a data prefiltering step is proposed to improve the existing method.
Low-pass filtering to reduce interperspective aliasing was implemented as a depth-of-field blur to mitigate out-of-focus artifacts. 
The measurements showed that such filtering, proposed in previous works, can improve the result but might not be optimal in all cases. 
The theoretical basis and the results of the experiment concerning the relation of the distance between objects in the scene and the focus settings are documented.
A novel scene-warping method was implemented to increase the focus areas in the result and was proved to be efficient and desirable by users. 

\subsection{Edge Detection Focusing}
The previously described focusing method, based on DoG is extended.
The output values of the metric for each focusing distance need to be prefiletered prior to the minimal value identification. 
When the scene is extremely defocused, the total energy of the DoG image tends to get very low; see Fig.~\ref{fig:dogFocus}.
This happens at the borders of the focus range; see Fig.~\ref{fig:dogFiltering}.
The global minimum would not mark the correct focus.
Therefore, the average energy value is calculated and all energy values from the borders of the array are removed until they are higher than the average. 
The whole process is described in Alg.~\ref{alg:edgeDetection}. 

\begin{figure}[!htbp]\centering
   \captionsetup[subfigure]{labelformat=empty}
   \def \rowLabelSize{0.05\linewidth}
   \def \imageSize{0.43\linewidth}
    \makebox[\rowLabelSize]{ }
    \makebox[\imageSize]{\centering completely defocused}
    \makebox[\imageSize]{\centering focused on the body}
    \\
    \makebox[\rowLabelSize]{\raisebox{50pt}{\rotatebox[origin=c]{90}{\centering internal format}}}
    \subfloat[\centering ]{{\includegraphics[width=\imageSize]{obrazky-figures/mitigation/internalDefocus.png} }} 
    \subfloat[\centering ]{{\includegraphics[width=\imageSize]{obrazky-figures/mitigation/internalFocus.png} }}
    \\
    \makebox[\rowLabelSize]{\raisebox{50pt}{\rotatebox[origin=c]{90}{\centering DoG}}}
    \subfloat[\centering Total energy: 17.1049]{{\includegraphics[width=\imageSize]{obrazky-figures/mitigation/dogDefocus.png} }} 
    \subfloat[\centering Total energy: 6.00614]{{\includegraphics[width=\imageSize]{obrazky-figures/mitigation/dogFocus.png} }}
    \\
  \caption{\label{fig:dogFocus} The first row contains the internal LKG format images with two different zero-parallax plane positions. 
  The second row shows the same images after DoG edge detection. The resulting energy of the image is lower in the focused case.}
\end{figure}

\begin{figure}[!htbp]
\centering
\pgfplotsset{height=7.5cm, width=10cm}
\pgfplotstableread{charts/mitigation/dogFiltering.txt}\dogFiltering
\pgfplotstableread{charts/mitigation/dogFilteringExclude.txt}\dogFilteringExclude
\begin{tikzpicture}
\begin{axis}[
  ylabel={$Objects\ distance$},
  xlabel={$Focus\ distance$},
  legend style={at={(0.4,0.7)},anchor=east},
  legend entries={Average, Focusing, Excluded, Optimal}
]

\addplot[dashed, no marks, color=red, mark=none] coordinates {(-2.0, 6.4631285) (2.0, 6.4631285)};
\addplot[color=mainBlue, mark=o, ultra thick] table[x=focus, y=energy]{\dogFiltering};
\addplot[color=mainOrange, mark=o, only marks, ultra thick] table[x=focus, y=energy]{\dogFilteringExclude};
\addplot[color=red, mark=x, only marks, ultra thick] coordinates {(0.43999999999999995, 5.33707)};
\end{axis}
\end{tikzpicture}
\caption{\label{fig:dogFiltering} The minimal energy value of the entire scan range would not be correct due to the very low values at the boundaries of the range. 
Therefore, the border values are excluded until they reach the average value. The values are taken from the evaluation of the \textit{Buddha} dataset.}
\end{figure}

\begin{algorithm}[!htbp]
 \KwData{The input set of $images$, discrete $focusRange$ with defined step}
 \KwResult{Optimal focus $distance$}
    $values \oper{=} \fce{emptyArray}()$\;
    \ForEach {focus$\oper{\in} focusRange$}{
        $interal \oper{=} \fce{generateInternal}(images, focus)$\;
        $dog \oper{=} \fce{differenceOfGaussians}(internal)$\;
        $e \oper{=} \fce{energy}(dog)$\;
        $values \xleftarrow{\fce{insert}} \{focus,e\}\;$
    }
    $avg \oper{=} \fce{getAvg}(values)$\;
    \ForEach {v$\oper{\in} values$}{
        \If{$v_{energy} \oper{>} avg$}
        {   \textbf{break}\;}
        $\fce{setAsInvalid}(v)$\;
    }
    \ForEach {v$\oper{\in} \fce{reverse}(values)$}{
        \If{$v_{energy} \oper{>} avg$}
        {    \textbf{break}\;}
        $\fce{setAsInvalid}(v)$\;
    }
    $optimal \oper{=} \fce{minEnergy}(values)$\;
    $distance \oper{=} optimal_{focus}$\;
 \caption{\label{alg:edgeDetection} The method iterates over a focus range and returns a distance, where the DoG of the internal LKG format has the lowest energy.}
\end{algorithm}

\subsection{Saliency Based Focusing}
This method is content-aware. 
First, the potential object of interest is detected, assuming that the user would like to see this object focused~\cite{borji2013stands}. 
The goal of the method is to position the zero-parallax plane near the detected object to minimize visual discomfort~\cite{jung2015visual}.
To identify the object of interest, the deep learning method GICD~\cite{zhang2020} is used. 
For each image of the input set, a saliency map is computed and saved. 
The map contains white pixels in the area of interest and black elsewhere. 
The method iterates over a focus range and shifts the maps according to the zero-parallax plane position. 
The intersection of the shifted maps is then computed; see Fig.~\ref{fig:saliencyFocus}.
The intersection is implemented by median filtering across the images to avoid completely black saliency maps clearing the result.
The focus distance with the highest energy saliency map intersection is then the one which has the object of interest in focus; see Alg.~\ref{alg:saliency}.

\begin{figure}[!htbp]\centering
   \centering
    \subfloat[\centering Saliency maps are computed from three different views.]{{\includegraphics[width=0.9\linewidth]{obrazky-figures/mitigation/saliencyMaps.png} }} 
    \\
    \subfloat[\centering Completely defocused maps are mixed.]{{\includegraphics[width=0.45\linewidth]{obrazky-figures/mitigation/saliencyMix.png} }} 
    \subfloat[\centering Maps are focused on the body and mixed.]{{\includegraphics[width=0.45\linewidth]{obrazky-figures/mitigation/saliencyMixFocus.png} }}
    \\
  \caption{\label{fig:saliencyFocus} The first row contains three generated saliency maps. 
  The second row shows two images which are results of a simple mix blending of the maps to demonstrate, that the defocused case would result in completely black multiplication of the images while the focused one would contain white areas.}
\end{figure}

\begin{algorithm}[!htbp]
 \KwData{The input set of $images$, discrete $focusRange$ with defined step}
 \KwResult{Optimal focus $distance$}
    $maps \oper{=} \fce{saliencyMaps}(images)$\;
    $optimal \oper{=} \fce{minimalValue}()$\;
    \ForEach {focus$\oper{\in} focusRange$}{
        $maps \oper{=} \fce{shiftImages}(maps, focus)$\;
        $mix \oper{=} \fce{multiply}(maps)$\;
        $e \oper{=} \fce{energy}(mix)$\;
        \If{$e \oper{>} optimal \xrightarrow{} e$}
        {
            $optimal \oper{=} \{e, focus\})$\;
        }
    }
    $distance \oper{=} optimal \xrightarrow{} focus$\;
 \caption{\label{alg:saliency} The method iterates over a focus range and returns a distance, where the intersection of saliency maps has the highest energy.}
\end{algorithm}

\subsection{Depth of Field Enhancement}
Due to the discrete light field representation and optical mixing of the views, a ghosting artifact appears in the parallax area. This artifact is similar to defocusing in a human visual system when the eyes are fixed at a certain distance~\cite{faubert2001}. 
The images can be pre-filtered to mitigate this interperspective aliasing~\cite{moller2005, zwicker2006}. 
Prefiltering removes high frequencies in the scene that are breaking the bounds defined by the Nyquist-Shannon Theorem. Depth-of-field (DoF) simulation blurs the expected defocused areas~\cite{lambooij2007}; see Fig.~\ref{fig:dofExample}.
The depth map or information on the geometry of the scene is usually necessary to increase the amount of blur with increasing distance from the zero-parallax plane~\cite{barsky2008}. 
Depth estimation along with the blur effect can be achieved using deep learning approaches, for example, DeepLens~\cite{deeplens2018} which is used in the experiments.

\begin{figure}[!htbp]\centering
   \def \imageSize{0.42\linewidth}
    \subfloat[\centering without DoF ]{{\includegraphics[width=\imageSize]{obrazky-figures/mitigation/dofOff.jpg} }} 
    \subfloat[\centering with DoF ]{{\includegraphics[width=\imageSize]{obrazky-figures/mitigation/dofOn.jpg} }}
  \caption{\label{fig:dofExample} The ghosting artifacts are visible at the defocused part of the scene (back object).
  The preprocessed images with DoF effect mitigate the out-of-focus sharp artifacts.}
\end{figure}

The proposed DoF algorithm needs a screen-space position of a zero-parallax plane point. 
When the depth map is estimated, a coordinate in the image is sufficient to identify the focus distance.
In the case of DoG focusing, the coordinate can be found by generating the focus map as in light field rendering methods~\cite{chlubna2021} based on pixel color deviations. 
The average coordinates of the focused pixels can lie outside a concave area, so the closest pixel is found from the focused ones, and its coordinates are selected as the DoF focus point. 
Multiplication of saliency maps can be used instead of the focus map to identify the focus point. 
Alg.~\ref{alg:dofMitig} describes both cases.

\begin{algorithm}[!htbp]
 \KwData{The input set of $images$, used focusing $method$}
 \KwResult{$DoF$ images}
    \If{$method\ \mathbf{is}\ DoG$}
    {$map \oper{=} \fce{dogFocusMap}(images)$\;}
    \ElseIf{$method\ \mathbf{is}\ Saliency$}
    {$map \oper{=} \fce{saliencyIntersection}(images)$\;}
    $map \oper{=} \fce{threshold}(map)$\;
    $map \oper{=} \fce{medianFilter}(map)$\;
    $avgCoord \oper{=} \fce{averageWhitePx}(map)$\;
    $coord \oper{=} \fce{closestWhitePx}(map, avgPoint)$\;
    $DoF \oper{=} \fce{simulateDoF}(images, focusPoint)$\;

 \caption{\label{alg:dofMitig} One of the focused pixels is chosen to identify the focus point for the depth of field simulation.}
\end{algorithm}

\subsection{Experiments \& Results}
15 people participated in the experiments in a laboratory equipped with a 15.6$''$ Looking Glass 4K model.
Users were expected to be LKG users interested in IT in the age range between 20-65.
None of the users was diagnosed with serious sight-related problems. 
All users passed a quick entrance test to exclude people with limited depth perception.
They were instructed to read the text on the display and distinguish 2D and 3D parts of the scene.
The users were sitting in front of the LKG at an initial distance of 55 cm, as recommended~\cite{rempel2007}.
The average time of one session was approximately 14 minutes with 11 seconds per test.
The tests were randomly shuffled to avoid bias when evaluating the same scene or type.

The rendering camera setup is based on Eq.~\ref{eq:travel} from the official LKG documentation\footnote{\href{https://docs.lookingglassfactory.com}{docs.lookingglassfactory.com}}, where the $travel$ value is the length of the capturing camera trajectory, the $distance$ is measured from the camera to the geometry of the expected area of interest, and $fov$ is the field of view of the camera.

\begin{equation}
  travel = distance \cdot \textrm{tan}(fov/2) \cdot 2
  \label{eq:travel}  
\end{equation}

\subsubsection{Focus Range Identification}
The depth range (the distance between the closest and farthest points from the camera) affects the way the user might select the optimal focus distance. 
If the scene does not contain large depth differences, the zero-parallax plane might be set by the user between multiple objects to keep them all more or less focused. 
Otherwise, the user would focus on a specific area of interest, leaving the rest of the scene unfocused.

For a given pixel representing a point in a perpendicular distance from the camera plane $d$, distance from the camera to the zero-parallax plane $f$, horizontal field of view $fov_x$, horizontal distance between the two neighboring capturing cameras $\Delta c_x$, and horizontal resolution of the image $res_x$, the displacement of the pixel representing the width of ghosting artifact $ghost$, can be theoretically computed by Eq.~\ref{eq:ghost}.

\begin{equation}
    ghost = \frac{\Delta c_x \cdot ( 1/f - 1/d )}{2 \cdot \textrm{tan}(fov_{x}/2)} \cdot res_x
    \label{eq:ghost}
\end{equation}

In a simple case, the optimal 2D vector representing resolution of one $\mathbf{view}$ would be calculated from the resolution of the device $\mathbf{screen}$ and the number of $views$ according to Eq.~\ref{eq:viewResolution}.

\begin{equation}
    \mathbf{view} = \frac{\mathbf{screen}}{\sqrt{views}}
    \label{eq:viewResolution}
\end{equation}

For a 4K ($3840 {\times} 2140$) display with 45 views in a quilt, this would lead to a resolution of $572 {\times} 322$.
According to the observations conducted, this resolution is still not high enough, and the improvement in visual quality is visible when the resolution is further increased.
The low-resolution artifacts cease to be noticeable at about $800 {\times} 583$. 
This leads to an approximation of Eq.~\ref{eq:ratio}, for the optimal $\mathbf{view}$ resolution calculation from the $\mathbf{screen}$ one.
The vector elements are accessed by the $x,y$ subscripts.

\begin{equation}
    \mathbf{view} = (0.21 \cdot \mathbf{screen}_x, 0.27 \cdot \mathbf{screen}_y)
    \label{eq:ratio}
\end{equation}

Note that the ratio between horizontal and vertical resolutions is less than the screen aspect ratio $16{:}9$ and the quilt rows and columns ratio $9{:}5$.
This might be caused by the rotation of the lenticular strips. Fig.~\ref{fig:ghostTheory} shows the theoretical amount of ghosting for the \textit{Crash} scene where the field of view of the camera is 39.6\degree, distance between the cameras is 0.050702 meters, and the horizontal resolution is 800 px.
The amount of artifacts is increasing not only with a higher distance of the pixel from the zero-parallax plane but also with the zero-parallax plane position being closer to the camera. 
Scenes focused on very close objects to the camera can be more difficult to setup.

\begin{figure}[!htbp]
\pgfplotsset{every axis plot post/.append style={very thin, draw=none, mark=none}}
\centering
\begin{tikzpicture}
\begin{axis}[
width=0.5\textwidth,
height=0.5\textwidth,
xlabel={$f$},
ylabel={$d$},
zlabel={$ghost\ [px]$},
colormap={example}{
color(0)=(mainBlue)
color(100)=(mainOrange)
},
]
\addplot3[surf,faceted color=black,
			 samples=32,
			 domain=0.05:0.5]
			{800*(0.050702 * (1/x-1/y))/(2*tan(19.8))};

\end{axis}
\end{tikzpicture}
\caption{\label{fig:ghostTheory} The chart shows the amount of ghosting artifacts produced according to the position of the zero-parallax plane $f$ and the depth of the pixel $d$.}
\end{figure}

The goal of this experiment is to find the distance between two objects, where users switch from focusing on one object to moving the zero-parallax plane in between them.
The testing scene contains two same objects next to each other. 
The one in the front is static, and the second one starts at the same $Z$ position (depth) as the first one. 
Then it is moved further away in each measurement. The moving object is scaled in each step so that it covers the same area of the image as the static one (approximately 7.5\% of the image). 
Rescaling is conducted to avoid a perceptional bias that might make users focus more on the larger object. 

The measured focus distances were clustered in two groups according to their distance to the optimal focus distance of the closer and further object. 
The variances of these two groups were averaged and compared to the global variance without clustering in Fig.~\ref{fig:focusRangeTest}. 

According to the theoretical calculation above, given the zero-parallax plane positioned at the first object and the views rendered in FullHD, with the scene dimensions shown in Fig.~\ref{fig:crashScene}, the artifacts should become visible ($ghost \geq 1$ px according to the plenoptic sampling theory~\cite{chai2000plenoptic}) at the distance from the camera plane of $2.823402$ meters. 
The distance between the object should be $3.01-2.823402=0.186598$ meters. 
The graphs start to noticeably diverge around $0.8$ meters. 
The equation is more strict than the real tolerance of the users, but it can be used as a good estimation.

\begin{figure}[!htbp]
\begin{minipage}[c]{0.49\linewidth}
\centering
\centering
\pgfplotsset{height=6.9cm}
\pgfplotstablesort{\focusRangeTest}{charts/mitigation/focusRangeTest.txt}
\pgfplotsset{every axis plot post/.append style={mark=none}}
\begin{tikzpicture}
\begin{axis}[
  width=\linewidth,
  xlabel={$Objects\ distance$},
  ylabel={$Variance$},
  legend style={at={(0.05,0.8)},anchor=west},
  legend entries={Global, Clustered, Theoretical},
]
\addplot[color=mainOrange, mark=o, ultra thick] table[x=distance, y=globalvariance]{\focusRangeTest};
\addplot[color=mainBlue, mark=o, ultra thick] table[x=distance, y=clusteredvariance]{\focusRangeTest};
\addplot[dashed, mark=none, color=red] coordinates {(0.186598,-0.0001)(0.186598,0.01)};

\end{axis}
\end{tikzpicture}
\caption{\label{fig:focusRangeTest} The figure shows how the variance of the focusing measurements increases with larger distance between objects. 
The average variance of the clustered measurements is close to the global variance until a slightly larger distance but close to the theoretical estimation.}
\end{minipage}
\hfill
\begin{minipage}[c]{0.49\linewidth}
\centering
  \includegraphics[width=\linewidth]{obrazky-figures/mitigation/crashScene.pdf}
  \caption{\label{fig:crashScene} Dimensions of the testing \textit{Crash} scene in meters and degrees.}
\end{minipage}
\end{figure}

Some users noticed that the object that is closer to the camera is more artifact-prone but also has more 3D-looking when focused.
This confirms the assumption shown in Fig.~\ref{fig:ghostTheory}. Fig.~\ref{fig:clustering} reveals that users usually choose the object that looks \textit{more 3D} despite the higher amount of artifacts.

\begin{figure}[!htbp]\centering
\pgfplotsset{every axis plot post/.append style={ultra thick, draw=none, mark=none}}
\pgfplotsset{width=0.99\linewidth, height=0.18\linewidth}
\pgfplotstableread{charts/mitigation/clustering.txt}\clustering
\begin{tikzpicture}
\begin{axis}[xbar,
  ymajorticks=false,
  xlabel={$Users\ [\%]$},
  legend style={at={(0.9,0.87)},anchor=east},
  legend entries={Front, Back},
  reverse legend,
  ylabel={$Object$},
  enlarge y limits={abs=0.5cm},
  bar width=1cm,
  ymax = 1,
  area legend
]
\addplot[color=white, fill=mainBlue] table[x expr={\thisrow{start}},y expr=\coordindex]{\clustering};
\addplot[color=white, fill=mainOrange] table[x expr={\thisrow{end}},y expr=\coordindex]{\clustering};
\end{axis}
\end{tikzpicture}
\caption{\label{fig:clustering} The figure shows that most users chose to focus on the front object in the \textit{Crash} scene.}
\end{figure}

\subsubsection{Depth Warping Evaluation}
The goal of this experiment is to determine whether the scene can be warped to reduce the depth range. 
The zero-parallax plane would be closer to more geometry, which leads to more focused scenes. 
The disadvantage is that the 3D effect and the amount of parallax would be less significant, similarly to cardboard effect caused by the low separation of the capturing camera~\cite{yamanoue2000}.
The users were choosing a more visually appealing result from pairs of scenes, where they compared differently warped scenes.
Fig.~\ref{fig:depthWarp} shows an example of the difference between the warped and original scene on LKG. 
Given the camera position $\mathbf{cam}$, expected $focus$ distance, and the $factor$ of the depth ($z$ axis) shrinking, the warping can be performed for each vertex $\mathbf{vert}$ of the scene using Eq.~\ref{eq:warpingA},~\ref{eq:warpingB},~\ref{eq:warpingC}, producing vertices with new positions $\mathbf{vert}^w$. 
The $x,y,z$ subscripts access the corresponding vector elements.
The objects keep the same relative size in the camera view but are shrinked towards the focus plane position and the camera.

\begin{equation}
    \mathbf{vert}^w_z = factor\cdot(\mathbf{vert}_z-focus)+focus
  \label{eq:warpingA}  
\end{equation}

\begin{equation}
    \mathbf{vert}^w_x = \frac{\mathbf{vert}_x-\mathbf{cam}_x}{\mathbf{vert}_z-\mathbf{cam}_z}\cdot(\mathbf{vert}^w_z-\mathbf{cam}_z)-\mathbf{cam}_x
  \label{eq:warpingB}  
\end{equation}

\begin{equation}
    \mathbf{vert}^w_y = \frac{\mathbf{vert}_y-\mathbf{cam}_y}{\mathbf{vert}_z-\mathbf{cam}_z}\cdot(\mathbf{vert}^w_z-\mathbf{cam}_z)-\mathbf{cam}_y
  \label{eq:warpingC}  
\end{equation}

\begin{figure}[!htbp]\centering
   \def \rowLabelSize{0.05\linewidth}
   \def \imageSize{0.43\linewidth}
    \makebox[\rowLabelSize]{ }
    \makebox[\imageSize]{\centering warped}
    \makebox[\imageSize]{\centering original}
    \\
    \makebox[\rowLabelSize]{\raisebox{50pt}{\rotatebox[origin=c]{90}{\centering 3D view}}}
    \subfloat[\centering ]{{\includegraphics[width=\imageSize]{obrazky-figures/mitigation/depthWarpScene.pdf} }} 
    \subfloat[\centering ]{{\includegraphics[width=\imageSize]{obrazky-figures/mitigation/depthNoWarpScene.pdf} }}
    \\
    \makebox[\rowLabelSize]{\raisebox{50pt}{\rotatebox[origin=c]{90}{\centering center view}}}
    \subfloat[\centering ]{{\includegraphics[width=\imageSize]{obrazky-figures/mitigation/depthWarp.jpg} }} 
    \subfloat[\centering ]{{\includegraphics[width=\imageSize]{obrazky-figures/mitigation/depthNoWarp.jpg} }}
    \\
    \makebox[\rowLabelSize]{\raisebox{50pt}{\rotatebox[origin=c]{90}{\centering left view}}}
    \subfloat[\centering ]{{\includegraphics[width=\imageSize]{obrazky-figures/mitigation/depthWarpPar.jpg} }} 
    \subfloat[\centering ]{{\includegraphics[width=\imageSize]{obrazky-figures/mitigation/depthNoWarpPar.jpg} }}
    \\
  \caption{\label{fig:depthWarp} The zero-parallax plane is positioned at the tree in the front in all cases. 
  The warped scene has $20{\times}$ smaller depth range and has more objects behind the tree in focus than the original.
  However, the parallax is less visible in the warped case.}
\end{figure}

Fig.~\ref{fig:warping} shows the users' choices when presented in each test with two scenes with different amounts of warping.
The users chose the more warped scene almost every time.
The only exception was in the case where the original scene was compared to the first warping iteration.
The reason is a high amount of artifacts that made both scenes relatively noisy and hard to distinguish which is better.
The testing scene was intentionally designed with a large depth range as an expected candidate for the warping method. 
The measurements prove that the warping of the scene is a suitable approach to improve the quality of the displayed result.

\begin{figure}[!htbp]
\centering
\pgfplotsset{height=6.9cm, width=0.7\linewidth}
\pgfplotsset{every axis plot post/.append style={mark=none}}
\pgfplotsset{tick label style={font=\footnotesize}, y dir=reverse, ytick=data}
\pgfplotstableread{charts/mitigation/warping.txt}\warping
\begin{tikzpicture}[] 
\begin{axis} [
    yticklabels from table={\warping}{test},
    xlabel={$Users\ [\%]$},
    enlarge x limits={abs=0pt},
    ylabel={$Warping\ [times\ shrinked]$},
    legend style={at={(1.0,0.5)},anchor=west},
    legend entries={Unwarped, Warped},
]
\addlegendimage{draw=mainBlue,very thick}
\addlegendimage{draw=mainOrange,very thick}
\def \barWidth {7pt}
\addplot+[mainBlue, forget plot,only marks, mark=square] 
  plot[ultra thick, error bars/.cd, x dir=minus, x explicit, error bar style={line width=\barWidth}]
  table[y expr=\coordindex,x=upperVal,x error expr={\thisrow{upperVal}}] {\warping};
\addplot+[mainOrange, only marks, mark=square, xticklabels=\empty] 
  plot[very thick, error bars/.cd, x dir=plus, x explicit, error bar style={line width=\barWidth}]
  table[y expr=\coordindex,x=upperVal,x error expr={\thisrow{lowerVal}}] {\warping};
\addplot[dashed, mark=none, color=red] coordinates {(0,6.5)(100,6.5)};
  
\end{axis} 
\end{tikzpicture}
\caption{\label{fig:warping} The charts show how users prefer higher amount of warping of the scene geometry. 
The warping reduces parallax but mitigates the out-of-focus artifacts.}
\end{figure}

\subsubsection{Depth of Field Evaluation}
The scenes are focused with the saliency method, and then the images are post-processed with a shallow depth-of-field framework~\cite{deeplens2018}.
The DoF-based ghosting mitigation assumption is based on the sampling theory, effectively removing frequencies above the Niquist limit~\cite{zwicker2007}.
The filtering was applied on the full-resolution input views to avoid aliasing~\cite{chai2000plenoptic}.
Users were presented with prefocused scenes in two variants for each scene, with and without depth of field.
Their task was to choose which of each pair is more appealing in terms of visual quality and 3D experience on the LKG.
The prefiltered scenes were manually validated.

Surprisingly, Fig.~\ref{fig:dofResult} shows that most users chose the scene without DoF as a more appealing result.
Therefore, DoF cannot be used without a prior analysis of the content of the scene. 
Although the interperspective aliasing artifacts are smoothed, the added blur can decrease the visual comfort of the imagery on LKG. 
Fig.~\ref{fig:dofTest} shows the results for each scene. 
The \textit{Cars} scene was mostly preferred with DoF, the \textit{Knight} scene without, and the expected optimal candidate for DoF, \textit{Angel}, was slightly more preferred without DoF; see Fig.~\ref{fig:dofScenes}. 
Few observations can be derived from the measurements: 
\begin{itemize}
    \item DoF reduces the quality of the result in scenes with small depth range, for example, \textit{Knight, Table}.
    \item The best choice for DoF is a scene with sharp edges that are only slightly out-of-focus, for example, \textit{Cars, Room}.
    \item Applying DoF on a distant background does not always improve the result, for example, \textit{Angel}.
\end{itemize}

\begin{figure}[!htbp]
\begin{minipage}[c]{0.49\linewidth}
\centering
\pgfplotsset{every axis plot post/.append style={ultra thick, draw=none, mark=none}}
\pgfplotstableread{charts/mitigation/dofResult.txt}\dofResult
\begin{tikzpicture}
\begin{axis}[xbar,
  width=0.9\linewidth,
  height=0.9\linewidth,
  ymajorticks=false,
  xlabel={$Users\ [\%]$},
  ylabel={$DoF\ effect$},
  legend style={at={(0.9,0.25)},anchor=east},
  legend entries={With DoF, Without DoF},
  area legend,
  bar width=2cm,
  reverse legend,
  ymax = 1,
  enlarge y limits={abs=1.0*\pgfplotbarwidth},
]
\addplot[color=white, fill=mainBlue] table[x expr={\thisrow{dof}},y expr=\coordindex]{\dofResult};
\addplot[color=white, fill=mainOrange] table[x expr={\thisrow{nodof}},y expr=\coordindex]{\dofResult};
\end{axis}
\end{tikzpicture}
\caption{\label{fig:dofResult} The overall amount of users that preferred original or DoF scene across all tests and scenes.}
\end{minipage}
\hfill
\begin{minipage}[c]{0.49\linewidth}
\centering
\pgfplotsset{every axis plot post/.append style={ultra thick, draw=none, mark=none}}
\pgfplotsset{height=6.9cm}
\pgfplotsset{tick label style={font=\footnotesize}, y dir=reverse, ytick=data}
\pgfplotstableread{charts/mitigation/dofTest.txt}\dofTest
\begin{tikzpicture}[] 
\begin{axis} [
    width=0.9\linewidth,
    y tick label style={font=\itshape},
    yticklabels from table={\dofTest}{scene},
    xlabel={$Users\ [\%]$},
    enlarge x limits={abs=0pt},
    ylabel={$Scene$},
    legend style={at={(0.45,0.4)},anchor=west},
    legend entries={Measurement, With DoF, Without DoF}
]
\addlegendimage{draw=black,line width=4pt, xscale=0.1, style={xshift=80}}
\addlegendimage{draw=mainBlue,very thick}
\addlegendimage{draw=mainOrange,very thick}
\def \barWidth {7pt}
\def \minMaxWidth {2pt}
\def \markScale {2.2}
\addplot+[mainBlue, forget plot,only marks, mark=square, mark options={color=black, xscale=0.07, yscale=\markScale}] 
  plot[ultra thick, error bars/.cd, x dir=minus, x explicit, error bar style={line width=\barWidth}]
  table[y expr=\coordindex,x=dof,x error expr={\thisrow{dof}}] {\dofTest};
\addplot+[mainOrange, only marks, mark=square, mark options={color=black, xscale=0.07, yscale=\markScale},xticklabels=\empty] 
  plot[very thick, error bars/.cd, x dir=plus, x explicit, error bar style={line width=\barWidth}]
  table[y expr=\coordindex,x=dof,x error expr={\thisrow{nodof}}] {\dofTest};
  
\end{axis} 
\end{tikzpicture}
\caption{\label{fig:dofTest} The chart shows the user preferences regarding the DoF effect for all tested scenes.}
\end{minipage}
\end{figure}

\begin{figure}[!htbp]\centering
   \def \rowLabelSize{0.05\linewidth}
   \def \imageSize{0.46\linewidth}
    \makebox[\rowLabelSize]{ }
    \makebox[\imageSize]{\centering without DoF}
    \makebox[\imageSize]{\centering with DoF}
    \\
    \makebox[\rowLabelSize]{\raisebox{50pt}{\rotatebox[origin=c]{90}{\centering \textit{Cars}}}}
    \subfloat[\centering ]{{\includegraphics[width=\imageSize]{obrazky-figures/mitigation/dofOffCars.pdf} }} 
    \subfloat[\centering ]{{\includegraphics[width=\imageSize]{obrazky-figures/mitigation/dofOnCars.pdf} }}
    \\
    \makebox[\rowLabelSize]{\raisebox{50pt}{\rotatebox[origin=c]{90}{\centering \textit{Knight}}}}
    \subfloat[\centering ]{{\includegraphics[width=\imageSize]{obrazky-figures/mitigation/dofOffKnight.pdf} }} 
    \subfloat[\centering ]{{\includegraphics[width=\imageSize]{obrazky-figures/mitigation/dofOnKnight.pdf} }}
    \\
    \makebox[\rowLabelSize]{\raisebox{50pt}{\rotatebox[origin=c]{90}{\centering \textit{Angel}}}}
    \subfloat[\centering ]{{\includegraphics[width=\imageSize]{obrazky-figures/mitigation/dofOffAngel.pdf} }} 
    \subfloat[\centering ]{{\includegraphics[width=\imageSize]{obrazky-figures/mitigation/dofOnAngel.pdf} }}
    \\
  \caption{\label{fig:dofScenes} Examples of the most interesting scenes details on LKG with and without DoF.}
\end{figure}

\subsubsection{Focusing Method Comparison}
The edge detection (DoG) approach selects as the optimal zero-parallax plane position a distance where most of the scene is focused. 
The method does not take into account human perception.
The saliency method focuses on a specific area in the scene that is most likely to be looked at by the user. 
The difference is shown in Fig.~\ref{fig:comparisonPlanes}.

\begin{figure}[!htbp]\centering
  \includegraphics[width=\linewidth]{obrazky-figures/mitigation/comparisonPlanes.pdf}
  \caption{\label{fig:comparisonPlanes} Two zero-parallax planes are depicted in the 3D scene, one is aligned with the front object which is regarded the object of interest as the result of the \textcolor{mainBlue}{saliency~method} and the second one is roughly between the objects (a bit closer to the one with larger area in the final image) trying to find the best average focus distance as the result of \textcolor{mainOrange}{DoG~method}.}
\end{figure}

The autofocusing methods are time-consuming due to the scanning process, where all input images must be analyzed, filtered, and evaluated for all steps of the focus range. 
A measurement was carried out to analyze the performance of the methods in relation to the resolution of the input images.
Fig.~\ref{fig:methodsPerformance} shows the execution times of both methods and the resulting focus distances.
The measurement was carried out on an Arch Linux machine equipped with Intel\textsuperscript{\textregistered} Core\textsuperscript{\texttrademark} i5-8500 CPU 3.00 GHz and 16 GB DDR4 RAM.

\begin{figure}[!htbp]
\centering
\pgfplotsset{height=6.9cm}
\pgfplotsset{try min ticks=9}
\pgfplotstableread{charts/mitigation/methodsPerformance.txt}\methodsPerformance
\begin{tikzpicture}
\begin{axis}[
  ylabel={$Execution\ time\ [minutes]$},
  xlabel={$Resolution\ shrinking\ step$},
  legend style={at={(0.9,0.7)},anchor=east},
  xmax=9,
  legend entries={DoG, Saliency, Value change}
]
\addplot[color=mainOrange, mark=o, ultra thick] table[x=resolution, y=dogTime]{\methodsPerformance}
node [pos=0.15, above,color=red]{$0.54$}
node [pos=0.5, above,color=red]{$0.48$}
node [pos=0.8, above,color=red]{$0.46$}
node [pos=0.975, above,color=red]{$1.58$};
\addplot[color=mainBlue, mark=o, ultra thick] table[x=resolution, y=deepTime]{\methodsPerformance}
node [pos=0.1, right,color=red]{$0.46$};
\addplot[color=red, mark=o, only marks, ultra thick] coordinates {
(0, 49.5420562465985)
(0, 235.476424292723)
(1, 33.504799091816)
(4, 6.53886620203653)
(7, 1.15320069789887)
};
\end{axis}
\end{tikzpicture}
\caption{\label{fig:methodsPerformance} The figure shows the performance of both autofocusing methods. 
The resolution of the input images shrinks by $3/4$ of the previous one along the $x$ axis, starting at 4K.
The range was scanned with 200 steps on the \textit{Crash} scene. 
The red nodes indicate how the value of the focus changed.}
\end{figure}

The density of the sampling range affects the performance/quality ratio linearly because it only adds more iterations.
The density is set to $400$ steps in the $[-1;1]$ range, which was empirically identified as sufficient to cover the entire dataset. 

The DoG method is approximately $6{\times}$ faster than the saliency one, but the detection of a focus point for the DoF simulation is approximately $30{\times}$ faster in the saliency method. 
However, the DoG method is faster in the total time comparison, because the detection of the point takes only $15\%$ of the total DoG time and $0.09\%$ of the saliency time. 
The saliency method is stable in all resolutions, whereas the DoG method slightly changes the result in different resolutions and returns a completely wrong focus distance for resolutions below $512{\times}288$.

The saliency method had problems in four scenes (\textit{Brick, Cubes, Rock, Cars}).
The reason is that the generated saliency maps did not mark the same objects in the scene as salient. 
The DoG method always returns the best overall focus distance.

The experimental task for the users was to move the slider to optimally focus the scenes.
The user might move the zero-parallax plane to an object of interest or to the position that makes a bigger part of the scene focused, sacrificing the sharpness of other parts. 

Fig.~\ref{fig:comparisonResult} shows that the saliency method outperforms the DoG one and simulates the human choice more precisely with respect to the overall error. 
The total number of measurements that were closer to the DoG method is higher; see Fig.~\ref{fig:comparisonResultCloser}.

\begin{figure}[!htbp]
\pgfplotsset{every axis plot post/.append style={ultra thick, draw=none, mark=none}}
\pgfplotsset{width=0.99\linewidth, height=0.4\linewidth}
\begin{minipage}[c]{0.49\linewidth}
\centering
\pgfplotstableread{charts/mitigation/comparisonResult.txt}\comparisonResult
\begin{tikzpicture}
\begin{axis}[xbar,
  width=\linewidth,
  ylabel={$Method$},
  ymajorticks=false,
  xlabel={$Error\ [focus\ distance]$},
  legend style={at={(0.9,0.8)},anchor=east},
  legend entries={DoG, Saliency},
  ymax = 1,
  scaled ticks=false,
    x tick label style={
    /pgf/number format/fixed,
     /pgf/number format/precision=3},
  area legend,
  reverse legend,
  enlarge y limits={abs=0.5cm},
  bar width=1cm
]
\addplot[color=white, fill=mainBlue] table[x expr={\thisrow{dogErr}},y expr=\coordindex]{\comparisonResult};
\addplot[color=white, fill=mainOrange] table[x expr={\thisrow{saliencyErr}},y expr=\coordindex]{\comparisonResult};
\end{axis}
\end{tikzpicture}
\caption{\label{fig:comparisonResult} The chart shows the average difference of the automatically detected focus distances from the users' measurements.}
\end{minipage}
\hfill
\begin{minipage}[c]{0.49\linewidth}
\centering
\pgfplotstableread{charts/mitigation/comparisonResultCloser.txt}\comparisonResultCloser
\begin{tikzpicture}
\begin{axis}[xbar,
  width=\linewidth,
  ymajorticks=false,
ylabel={$Method$},
  xlabel={$Users\ closer\ [\%]$},
  legend style={at={(0.9,0.8)},anchor=east},
  legend entries={DoG, Saliency},
  area legend,
  reverse legend,
  ymax = 1,
  enlarge y limits={abs=0.5cm},
  bar width=1cm
]
\addplot[color=white, fill=mainBlue] table[x expr={\thisrow{dog}},y expr=\coordindex]{\comparisonResultCloser};
\addplot[color=white, fill=mainOrange] table[x expr={\thisrow{saliency}},y expr=\coordindex]{\comparisonResultCloser};
\end{axis}
\end{tikzpicture}
\caption{\label{fig:comparisonResultCloser} The bars show how many of the users measured the focusing closer to the given method in 180 measurements.}
\end{minipage}
\end{figure}

Note that in some cases, such as in many measurements of \textit{Knight} scene, the measurements of both methods were very close.
Therefore, the error is taken as the most representative metric. 
Fig.~\ref{fig:focusingComparison} shows the average, minimal, and maximal measured values and values which were automatically detected by the two proposed methods for each scene.
The DoG method is not optimal for scenes with a large depth range and scenes with a lot of high-frequency features. 
The saliency method performs slightly worse in scenes with low frequencies and large solid-colored areas. 

\begin{figure}[!htbp]
\centering
\pgfplotsset{height=0.42\linewidth, width=0.9\linewidth}
\pgfplotsset{tick label style={font=\footnotesize}, y dir=reverse, ytick=data}
\pgfplotstableread{charts/mitigation/focusingComparison.txt}\focusingComparison
\begin{tikzpicture}[] 
\begin{axis} [
    yticklabels from table={\focusingComparison}{scene},
    xlabel={$Focus\ distance$},
    y tick label style={font=\itshape},
    ylabel={$Scene$},
    legend style={at={(0.027,0.65)},anchor=west},
    legend entries={User, DoG, Saliency}
]
\addlegendimage{draw=black,line width=4pt, xscale=0.1, style={xshift=80}}
\addlegendimage{draw=mainBlue,very thick}
\addlegendimage{draw=mainOrange,very thick}
\def \barWidth {7pt}
\def \minMaxWidth {2pt}
\def \markScale {2.2}
\addplot+[mainBlue, opacity=0.5, forget plot,only marks, mark=square, mark options={color=black, xscale=0.07, yscale=\markScale}] 
  plot[ultra thick, error bars/.cd, x dir=plus, x explicit, error bar style={line width=\barWidth}]
  table[y expr=\coordindex,x=avg,x error expr={\thisrow{dog}-\thisrow{avg}}] {\focusingComparison};
\addplot+[mainOrange, opacity=0.5, only marks, mark=square, mark options={color=black, xscale=0.07, yscale=\markScale},xticklabels=\empty] 
  plot[very thick, error bars/.cd, x dir=minus, x explicit, error bar style={line width=\barWidth}]
  table[y expr=\coordindex,x=avg,x error expr={\thisrow{avg}-\thisrow{deep}}] {\focusingComparison};
\addplot+[black, only marks, mark=square, mark options={color=black, xscale=0, yscale=0},xticklabels=\empty] 
  plot[thick, error bars/.cd, x dir=minus, x explicit, error bar style={line width=\minMaxWidth}]
  table[y expr=\coordindex,x=avg,x error expr={\thisrow{avg}-\thisrow{min}}] {\focusingComparison};
\addplot+[black, only marks, mark=square, mark options={color=black, xscale=0, yscale=0},xticklabels=\empty] 
  plot[thick, error bars/.cd, x dir=plus, x explicit, error bar style={line width=\minMaxWidth}]
  table[y expr=\coordindex,x=avg,x error expr={\thisrow{max}-\thisrow{avg}}] {\focusingComparison};
  
\end{axis} 
\end{tikzpicture}
\caption{\label{fig:focusingComparison} The automatically detected focus distances of both methods and the average measured focus distances with their minimal and maximal values are visible in the chart for each scene.}
\end{figure}

The DoG method serves as a good first guess for the focusing, expecting the user to adjust the result for desired result.
The saliency method is a better approach for reaching an artistic result without further adjustments.
The experiments show, that a general focusing approach would consist of both methods, first performing the saliency one. 
If the accuracy of the saliency method is too low, for example, low amount of overlapping saliency masks, the DoG method can be used as a fallback. 
This approach would improve the accuracy of the focusing, compared to the saliency method, by more than 50\% according to the measurements.

\clearpage
\section{Capturing Camera Trajectory Distortion}
\label{sec:trajectoryDist}
The main goal of this section is to discover the limitations and most crucial parameters related to the shape of the capturing camera trajectory when creating input views for 3D display.
An evaluation method was proposed and experimentally measured in a user study~\cite{chlubna2024how}.
This study reveals specific attributes of 3D displays that define the limitations for the input light field data.
For example, the proposed light field rendering method can be used accordingly.

The linear trajectory of the cameras was distorted in various ways. 
The users were asked to adjust the amount of the distortion to the maximum acceptable values while evaluating the visual quality of the result on the LKG; see Fig.~\ref{fig:trajectoryTeaser}.
Some distortions of the trajectory might affect the resulting visual quality in a different way or with a different impact on the human visual system than others. 

\begin{figure*}[!htbp]\centering
  \includegraphics[width=0.98\linewidth]{obrazky-figures/trajectory/teaser.pdf}
  \caption{The testing 3D scene was captured by cameras placed along the optimal trajectory.
  The views obtained were displayed on a Looking Glass Factory 3D display.
  The users adjusted the amount of simulated distortions of the trajectory.
  Each distortion type was examined in a separate measurement.
  The users evaluated the resulting 3D experience on the display and were instructed to set the amount of the distortion to the maximal value where the possible visual discomfort was not noticeable yet.}
  \label{fig:trajectoryTeaser}
\end{figure*}

Users might capture the scene with a handheld camera or extract the scene from arbitrary video.
The discovered limits might be used to produce a score of scene suitability and estimate the level of visual comfort for such a scene.
The perfect camera trajectory is difficult to achieve in real life without special equipment.
Proprietary software, such as games, can use tools like NVIDIA Ansel to enable restricted free-look camera view.
The proposed guidelines might help to correctly setup the capturing process when intentionally creating suitable views and also when processing already existing videos or photos which are not suitable to display on LKG without additional edits.
A 2D-reprojection-based correction method of the images is also proposed and evaluated.
The method is capable of improving the visual quality of the unsuitably captured scenes, thus loosening the required capturing rules. 
The limits discovered might lead to a more comfortable experience in the field of 3D displays, which had been gaining popularity and various applications in recent years~\cite{yang2016see}.

Compared to previous studies, more camera trajectory distortions were tested with additional adjustments of user behavior and scene content.
More distortion types (translation and rotation in 3D and their combination) were implemented in this study, and they were also measured in more variations (with or without background, with user's head moving and being static). 
The study does not focus on possible misperceptions and depth accuracy issues~\cite{held2008} or visual discomfort caused by specific disparity-related problems~\cite{terzic2016}.
The measurements discovered limitations valid for the current state-of-the-art 3D display that might differ from previous tests with older devices.
The device in this study is not experimental as devices in previously published papers, but it is a widely used consumer 3D display device.
Other technologies, such as Lumepad 3D tablet, provide only a limited functionality of LKG and the results would be similar, so only LKG was considered during the testing.

\subsection{Evaluation Method}\label{section:experiments}
All experiments require the users to adjust the normalized slider in the $[0,1]$ range, which affects the distortion of the capture camera.
At the same time, they visually evaluate the result on the LKG.
In this way, users find the optimal or maximum value of the parameter, keeping the result visually acceptable. 
The four main objectives of the experiments are as follows:
\begin{enumerate}
\item Find the maximum acceptable amount of the distortions.
\item Identify the most crucial and the least crucial type of distortion.
\item Reveal the effect of the content of the scene or user behavior on the visual quality of the distorted scene.
\item Measure the efficiency of 2D reprojection of the images that compensates for the distortions.
\end{enumerate}

An OpenGL application with a simple 3D scene with two boxes and a background plane was implemented to conduct the experiments; see Fig.~\ref{fig:app}.
The zero-parallax plane distance was set to keep most of the area containing the boxes in the middle of the scene focused. Fig.~\ref{fig:scene} is the blueprint of the scene. The experiments were conducted with 10 people in a laboratory equipped with a 15.6$''$ Looking Glass 4K model, rendering 45 views of the scene. Users were expected LKG users, people interested in IT, in the 20-40 age range with no serious sight indisposition. During the experiments, the participants were sitting in front of the LKG at an initial distance of 55 cm, which is within the recommended viewing distance range~\cite{rempel2007}. A supervisor noted the relevant comments of the users. The results were statistically processed and averaged.

\begin{figure}[!htbp]
\begin{minipage}[c]{0.48\linewidth}
    \includegraphics[width=\textwidth]{obrazky-figures/trajectory/app.jpg}
  \caption{2D view of the testing application where the user's task was to move the slider to the maximal value while evaluating the result on the LKG.}
  \label{fig:app}
\end{minipage}
\hfill
\begin{minipage}[c]{0.48\linewidth}
  \includegraphics[width=\textwidth]{obrazky-figures/trajectory/scene.pdf}
  \caption{The figure depicts the 3D scene used in the experiments. The measures are in the world space units. The cameras are set according to the official LKG recommendations.}
  \label{fig:scene}
\end{minipage}
\end{figure}

During the first set of measurements, users were instructed not to move their heads to evaluate only the artifacts caused by the interpolation of the views.
The use case for this scenario is, for example, watching a 3D movie where users are expected to sit still. 
During the second set, they moved their heads from side to side to see the parallax and to evaluate the quality of the seeming motion of the objects in the scene. 
The user might do so, for example, when playing games or exploring 3D models. 

The user might focus solely on the main object in the scene (boxes) or may also evaluate the artifacts in the background. 
Therefore, another categorization was used, using the same scene with and without the background. The categories are depicted in Fig.~\ref{fig:experiments}.
A total of 128 measurements were conducted with two output values for each measurement, the slider value, and the time duration of the particular measurement.
At the end of the session, users were asked additional questions about the experiments:
\begin{enumerate}
\item What distortions were the most problematic and why?
\item What kinds of artifacts did the participant notice?
\end{enumerate}

\begin{figure}[!htbp]\centering
  \includegraphics[width=0.95\linewidth]{obrazky-figures/trajectory/experiments.pdf}
  \caption{The scheme describes the categories of the measured tasks in the experiments.}
  \label{fig:experiments}
\end{figure}

\subsubsection{Trajectory Distortion}
The shape of the capturing camera trajectory can be described by a function. 
Different shapes might result in different kinds of artifacts and user experience, even if the underlying camera transformation is the same. 
Theoretically, $n$ basis functions $\textrm{f}_i$ can be defined and combined with weights $w_i$ to simulate a large set of camera trajectory shapes according to Eq.~\ref{eq:basis}. 
The resulting $x$th camera displacement $\textrm{d}(x)$ can be calculated by multiplying the amount of transformation $\textrm{t}(x)$ and the weighted sum of the basis shape functions.

\begin{equation}
\textrm{d}(x)=\textrm{t}(x) \cdot \sum_{i=0}^{n} (w_{i} \cdot \textrm{f}_{i}(x))
	\label{eq:basis}
\end{equation}

Two basis functions were defined and used separately (only one weight is 1 and the second is zero): a sine function to simulate camera shaking and a linear function to simulate gradual camera change.
Therefore, the distortions were implemented in two variants: 
\begin{itemize}
    \item \textbf{Linear} where each camera is affected by the distortion relative to the previous one (incrementally).
    \item \textbf{Jagged} where the cameras are distorted in an alternating pattern.
\end{itemize}
Fig.~\ref{fig:camerasVertical} shows how the cameras are placed. 
Fig.~\ref{fig:distortions} is an example of the resulting quilts. 
Six basic camera transformations were implemented: rotation and translation on the three basic axes; see Fig.~\ref{fig:transformations}. 

\begin{figure}[!htbp]
    \centering
    \def \subImageSize {0.27\textwidth}
    \subfloat[\centering no distortion]{{\includegraphics[width=\subImageSize]{obrazky-figures/trajectory/camerasNormal.pdf} }}\   
    \subfloat[\centering linear]{{\includegraphics[width=\subImageSize]{obrazky-figures/trajectory/camerasLinear.pdf} }}\ 
    \subfloat[\centering jagged]{{\includegraphics[width=\subImageSize]{obrazky-figures/trajectory/camerasJagged.pdf} }}
    \caption{The distortions can be applied in two modes, linearly with an offset relative to the distance from the first camera and jagged where the camera moves up and down repeatedly. 
    Vertical translation is used here for the demonstration.}
    \label{fig:camerasVertical}
\end{figure}

\begin{figure*}[!htbp]
    \centering
    \subfloat[no distortion]{{\includegraphics[width=1.0\linewidth]{obrazky-figures/trajectory/noDistortion.pdf} }}\\
    \subfloat[linear]{{\includegraphics[width=1.0\linewidth]{obrazky-figures/trajectory/linearDistortion.pdf} }}\\
    \subfloat[jagged]{{\includegraphics[width=1.0\linewidth]{obrazky-figures/trajectory/jaggedDistortion.pdf} }}
    \\\setcounter{subfigure}{0}

    \caption{The figure shows different kinds of distortions. Each line shows a subset of views from the whole quilt.
    The \textcolor{mainBlue}{blue~line} tracks the position of the same 3D point.}
    \label{fig:distortions}
\end{figure*}

\begin{figure}[!htbp]\centering
  \includegraphics[width=0.5\textwidth]{obrazky-figures/trajectory/transformations.pdf}
  \caption{The basic transformations used to distort the trajectory: translation and rotation (roll, pitch, yaw) in $X,Y,Z$ axes.}
  \label{fig:transformations}
\end{figure}

Linear distortion causes more ghosting in the static view; see Fig.~\ref{fig:ghosting}.
The motion artifacts, when moving the head around the display, are continuous.
The objects move in the direction of the distortion smoothly; see Fig.~\ref{fig:linearArtifacts}. 
The jagged distortion creates less ghosting.
Two images of three might be placed on the horizontal line in the best case.
However, the motion of the objects suffers from wobbling artifacts; see Fig.~\ref{fig:wooble}. 
All proposed tests simulate the distortions that the user might create, for example, when capturing the scene with a handheld camera.

\begin{figure}[!htbp]
\begin{minipage}[c]{0.54\linewidth}
  \begin{overpic}[width=\textwidth]{obrazky-figures/trajectory/ghosting.jpg}
  \put(0,0){\includegraphics[width=1.0\textwidth]{obrazky-figures/trajectory/ghostingOver.pdf}}
  \end{overpic}
  \caption{The figure shows the LKG imagery captured with the ideal camera trajectory in the top row and increasing jagged vertical distortion below. 
  The ghosting effect is visible at the edges of the objects.}
  \label{fig:ghosting}
\end{minipage}
\hfill
\begin{minipage}[c]{0.44\linewidth}
  \begin{overpic}[width=\textwidth]{obrazky-figures/trajectory/linearArtifacts.jpg}
  \put(0,0){\includegraphics[width=1.0\textwidth]{obrazky-figures/trajectory/linearArtifactsOver.pdf}}
  \end{overpic}
  \caption{The figure shows the LKG view from rightmost, center, and leftmost viewing angle.
  The objects move down according to the linear distortion with a motion blur.}
  \label{fig:linearArtifacts}
\end{minipage}
\end{figure}

\begin{figure*}[!htbp]\centering
  \includegraphics[width=0.9\linewidth]{obrazky-figures/trajectory/wooble.pdf}
  \caption{The figure shows two zoomed areas of the LKG imagery captured from different but close angles. 
  The jagged vertical distortion leads to wobbling artifacts of the edges. 
  The lines represent the expected edge. 
  The ghosting artifact oscillates around the expected edge.}
  \label{fig:wooble}
\end{figure*}

An extra distortion is a circular shape of the trajectory. 
Two versions are used, a circle around the objects where each camera points to the center of the circle and an inverted one, where the cameras point outside; see Fig.~\ref{fig:camerasCircle}.
This simulates the case where the user might circle around the object with a camera to capture it from all angles or when capturing the scene standing at one place and rotating the camera around, such as when taking a panorama photo.

\begin{figure}[!htbp]
    \centering
%    \subfloat[\centering no distortion]{{\includegraphics[width=0.8\linewidth]{obrazky-figures/trajectory/cameraNormalCircle.pdf} }} 
%   \\
    \subfloat[\centering circular]{{\includegraphics[width=0.4\textwidth]{obrazky-figures/trajectory/cameraCenterCircle.pdf} }}
    \subfloat[\centering inverted circular]{{\includegraphics[width=0.4\textwidth]{obrazky-figures/trajectory/cameraInvertedCircle.pdf} }}
    \caption{The circular distortion interpolates the camera positions between the original and the circular trajectories. 
    Two variants are implemented, half-circle around the object where the cameras point at the center and inverted half-circle as if taking a panorama photo.}
    \label{fig:camerasCircle}
\end{figure}

\subsubsection{Extra Measurements}
Additional measurements were included. 
The first was to discover the limits and optimal value of the view cone (distance between the cameras). 
According to the documentation, $35\degree$ is the optimal value of the total bounding view cone for all cameras combined according to the physical parameters of the display. 
This experiment was conducted to verify this claim and to find the possible range of acceptable view-cone values. 

Two additional tests, with the scene using a $30\degree$ and $60\degree$ view cone, measuring the depth-of-field effect were also included. 
These angles were chosen to represent narrow and wide view cones. Users were asked to set the amount of background blur (Gaussian blur radius) that can mitigate out-of-focus artifacts~\cite{jung2013}.

The default view angle of the camera from Fig.~\ref{fig:scene} is used in all measurements except for the depth-of-field test, where the angle is changed to test the effect of the wide and narrow view, and in the view cone test, where the users change it directly.

\subsubsection{Reprojection}
All distortion measurements had corresponding versions with a 2D correction of the images back to the optimal position. 
The pixels of the rendered 2D images are completely reprojected by the inverse transformation of the distortion with a specified constant depth. 
The goal of these tests is to evaluate the efficiency of this correction process in fixing inappropriately captured scenes. 
This is the evaluation of the ideal case, where the transformations are known: the view matrix $\mathbf{view}_{opt}$ of the camera in an optimal position, the distorted view matrix $\mathbf{view}_{dist}$, and the projection matrix $\mathbf{proj}$. 
The reprojected coordinates $\mathbf{cr}$ are obtained in Eq.~\ref{eq:reproj} where $\mathbf{cv}$ are the view-space coordinates of the pixels in the distorted image.
The inverse matrix of the distortion is used to correct the image and transform it according to the optimal position of the camera.
\begin{equation}
    \mathbf{cr} = \mathbf{proj} \cdot \mathbf{m}_{dist}^{-1} \cdot \mathbf{cv}
	\label{eq:reproj}
\end{equation}
The distortion is represented by a transformation matrix $\mathbf{m}_{dist}$, obtained as described in Eq.~\ref{eq:camera}, as the difference matrix of the optimal view matrix and the distorted one.
\begin{equation}
	\mathbf{m}_{dist} = \mathbf{view}_{dist} \cdot \mathbf{view}_{opt}^{-1} 
	\label{eq:camera}
\end{equation}
The $\mathbf{cv}$ denotes the view space coordinates acquired by Eq.~\ref{eq:view} from the input normalized texture coordinates (NTC) $\mathbf{co}$ of the image. 
$\mathbf{1}$ denotes vector filled with the value one.
\begin{equation}
    \mathbf{cv} = \mathbf{proj}^{-1} \cdot \left(2 \cdot \mathbf{co} - \mathbf{1}, d, 1\right)
	\label{eq:view}
\end{equation}
The coordinates must first be converted to the normalized device coordinates (NDC).
A predefined depth $d$ that corresponds to the view-space depth of the focused objects (boxes) is used as a constant global depth for all pixels in the image.
The projection is then reversed using the inverted projection matrix.
The coordinates need to be divided by their fourth element to compensate for the perspective-divide operation; see Eq.~\ref{eq:division}.
The subscript ($x,y,z,w$) denotes the selection of the given coordinates.
\begin{equation}
    \mathbf{cv} = \frac{\mathbf{cv}_{xyzw}}{\mathbf{cv}_w} 
	\label{eq:division}
\end{equation}
Eq.~\ref{eq:reprojFinal} is used to perform the perspective division and to get the coordinates back into NTC, resulting in the final $\mathbf{cr}$.

\begin{equation}
\mathbf{cr} = \left (\frac{\mathbf{cr}_{xy}}{\mathbf{cr}_{w}}+1  \right ) \cdot \frac{1}{2}
	\label{eq:reprojFinal}
\end{equation}

In practice, 3D reconstruction methods such as ORB-SLAM3~\cite{campos2021orb} would be necessary to estimate the camera positions and their distortions from the optimal path.
Views in the quilt should also be $2{\times}$ zoomed in to hide the areas at the edges of the reprojected image, where visual information is not available. 
Fig.~\ref{fig:compensationExample} shows an example of the reprojection.

\begin{figure*}[!htbp]
    \centering
    \subfloat[The images are reprojected without zoom and crop.]{{\includegraphics[width=1.0\linewidth]{obrazky-figures/trajectory/compensationNoZoom.pdf} }}\\
    \subfloat[The images are reprojected with zoom and crop.]{{\includegraphics[width=1.0\linewidth]{obrazky-figures/trajectory/compensationZoom.pdf} }}
    \caption{The figure demonstrates the effect of the reprojection of the 2D images into the correct positions. 
    The \textcolor{mainOrange}{orange~line} marks the area with missing image data. 
    The images are zoomed to mitigate border artifacts where the image was not rendered.}
    \label{fig:compensationExample}
\end{figure*}

\subsection{Experiments \& Results}
\def \smallChartHeight {0.5\textwidth}
\def \smallChartWidth {0.5\textwidth}

The user experience measurement criteria~\cite{tullis2013} do not focus on the general usability or efficiency of the product in these experiments.
The time of each task or the overall opinion of the user about the LKG is not important.
The main metric is the visual appeal of the result and the visual comfort related to human perception, based on the amount of distortion.
The effort of the user to complete the task was also checked and noted to identify possibly problematic types of distortion.

The duration of the session for one participant was approximately 29 minutes, where one measurement took 14 seconds on average. 
Half of the participants felt slight fatigue after the experiments, but none reported sickness, such as when watching uncomfortable virtual reality imagery. 
The most significant artifacts according to the users were vanishing edges, motion-blur-like noise, and ghosting.
When evaluating the results while moving their heads, unrealistic motions of the objects and woobling artifacts were reported. 
Tab.~\ref{tab:notation} explains the names of the measurements used in this section.

\begin{table}[!htbp]\centering
\begin{tabular}{ p{0.1\linewidth} p{0.85\linewidth} }
 name & description \\
 \toprule
 linear & The cameras are distorted incrementally. $n$th camera is distorted $n$ times.\\  
 jagged & The cameras are distorted alternately. Positive direction, no distortion, negative direction, no distortion, etc.).\\  
 static & The participants were instructed not to move their heads to evaluate only the stereo image.\\
 moving & The participants were instructed to evaluate the result from different viewing angles.\\ \\
\end{tabular}
\caption{\label{tab:notation} The table explains the terms used in the measurements.}
\end{table}

The units used in the measurements are listed in Tab.~\ref{tab:units} that shows how much the view image changes when its camera is transformed. 
Two metrics are used: SSIM over the whole image compared to the undistorted one and displacement of one pixel placed in the area of interest, on the zero parallax plane. 
The pixel is placed at the top left corner of the smaller box.
This pixel was chosen because of its position at the edge where most of the artifacts are noticeable.
Note that other pixels in the scene might move with a different velocity due to the perspective projection, their depth, and the nature of the applied transformation.
SSIM metric was chosen as a commonly used standard image similarity metric which better reflects the image differences for the human observer than other metric such as PSNR~\cite{choudh2019}.

\begin{table}[!htbp]\centering
\begin{tabular}{ c c c c c }
 transformation & amount & axis & SSIM & displacement [px] \\
 \toprule
 \multirow{3}{*}{rotation}  & \multirow{3}{*}{1\degree{}}      & x  & 0.636040 & 19 \\ 
                            &                                & y & 0.596173 & 19 \\ 
                            &                                & z & 0.651903 & 3.2 \\
\midrule 
\multirow{3}{*}{translation} & \multirow{3}{*}{0.01 ws unit} & x  & 0.850915 & 5 \\ 
                             &                               & y & 0.891480 & 5 \\ 
                             &                               & z & 0.938095 & 2 \\
                             \\
\end{tabular}
\caption{\label{tab:units} The table shows the relationship between the given transformation and the difference in pixels between two neighboring $1920{\times}1080$ views.
The pixel displacement is the length of the trajectory that the chosen pixel (a corner pixel in the area of interest) traveled due to the applied transformation.
Identical images would be evaluated by SSIM value of 1.
The world-space units in the 3D scene are labeled as ws units.}
\end{table}

The main questions defined in Section~\ref{section:experiments} are answered as follows:
\begin{enumerate}
\item The maximum distortion should generally not be greater than $0.3\degree$ in rotations and 0.02 world units (4 px in the area of interest) in translations.
\item Users mostly tolerated rotational distortions followed by translational and circular. 
The jagged variant is also more artifact-prone than the linear one.
\item A featureless or solid background can, in some cases, increase the tolerance to distortions. 
The more users move their heads around the LKG, the more unpleasant the distortion artifacts appear.
\item The 2D reprojection can be used to significantly improve the quality of the distorted quilts, up to 60 times in the best cases.
\end{enumerate}

The pixel difference between the distorted and optimal quilt is not enough to predict the effect of the distortion on the resulting visual quality. 
Fig.~\ref{fig:ssimGeneral} shows the SSIM metric values between the optimal and distorted quilts according to the measured limits.

\begin{figure}[!htbp]
\pgfplotsset{every axis plot post/.append style={ultra thick, draw=none, mark=none}}
\centering
\pgfplotstableread{
type ssim
rotations	0.637381916666667
}\rotations
\pgfplotstableread{
type ssim
translations	0.735853083333333
}\translations
\pgfplotstableread{
type ssim
linear	0.465248958333333
}\linear
\pgfplotstableread{
type ssim
jagged	0.907986041666667
}\jagged
\begin{tikzpicture}
\begin{axis}[xbar,yticklabels from table={\rotations}{type},
  width=\linewidth,
  height=0.22\textwidth,
  ymode=normal,
  xmin=0.4,
  xmax=0.91,
  ymajorticks=false,
  xlabel={$SSIM$},
  ylabel={$Transform$},
  legend style={at={(0.9,0.5)},anchor=west, fill opacity=0.9, text opacity=1, draw opacity=1, nodes={scale=1, transform shape}},
  legend entries={rotations, translations, linear, jagged, bckg, nobckg},
  area legend,
  reverse legend,
  enlarge y limits={abs=0.7cm},
  ymax = 1,
  bar width=1cm
]
\def \shift {0.0\linewidth}
\addplot[xshift=\shift, color=white, fill=mainLightBlue] table[x=ssim,y expr=\coordindex]{\rotations};
\addplot[xshift=\shift, color=white, fill=mainBlue] table[x=ssim,y expr=\coordindex]{\translations};
\addplot[xshift=\shift, color=white, fill=mainLightRed] table[x=ssim,y expr=\coordindex]{\linear};
\addplot[xshift=\shift, color=white, fill=mainRed] table[x=ssim,y expr=\coordindex]{\jagged};
\end{axis}
\end{tikzpicture}
\caption{\label{fig:ssimGeneral} The chart shows SSIM comparison between the optimal and average distorted quilt. 
A lower value means higher tolerance.}
\end{figure}

In this way, the measured results are normalized by the image difference and different transformations can be compared. 
Users tolerate a higher amount of rotational than translational, and linear over jagged variants of the distortions.
The presence of background does not show a significant difference in this overview.
The $z$ axis rotation is the most tolerated, while the rotation of the $x$ and $y$ axes shows very similar limits. 
The same facts are not valid for translations, where the $x$ axis has the highest distortion limit, followed by $z$, and leaving $y$ as the most artifact-prone transformation; see Fig.~\ref{fig:translationsRotationsGeneral}.
Visually, a small amount of $y$ rotation is similar to $x$ translation and $x$ rotation to $y$ translation.
Despite that, the results do not correlate due to the parallax effect present in the translation.

\begin{figure*}[!htbp]\centering
\pgfplotsset{every axis plot post/.append style={ultra thick, draw=none, mark=none}}
\centering
\pgfplotstableread{
type angle
x	0.0755664228056484
}\xr
\pgfplotstableread{
type angle
y	0.0674607541063248
}\yr
\pgfplotstableread{
type angle
z	0.336333423533708
}\zr
\subfloat[Rotation]{
\begin{tikzpicture}
\begin{axis}[xbar,yticklabels from table={\xr}{type},
  width=0.979\textwidth,
  height=0.2\textwidth,
  xmode=normal,
  xmin=0,
  enlarge x limits={abs=0pt},
  xmax=0.34,
    xticklabel style={
        /pgf/number format/fixed,
        /pgf/number format/precision=2
},
  ymajorticks=false,
  xlabel={$Angle\ [\degree]$},
  ylabel={$Axis$},
  enlarge y limits={abs=0.5cm},
  ymax = 1,
  bar width=1cm
]
\def \shift {0.0\linewidth}
\addplot[xshift=\shift, color=white, fill=mainRed] table[x=angle,y expr=\coordindex]{\xr};
\addplot[xshift=\shift, color=white, fill=mainBlue] table[x=angle,y expr=\coordindex]{\yr};
\addplot[xshift=\shift, color=white, fill=mainOrange] table[x=angle,y expr=\coordindex]{\zr};
\end{axis}
\end{tikzpicture}
}
\\
\pgfplotstableread{
type disp
x	0.0269093826015625
}\xt
\pgfplotstableread{
type disp
y	0.003004800296875
}\yt
\pgfplotstableread{
type disp
z	0.015575850875
}\zt
\subfloat[Translation]{
\begin{tikzpicture}
\begin{axis}[xbar,yticklabels from table={\xt}{type},
  width=\textwidth,
  height=0.2\textwidth,
  xmode=normal,
  enlarge x limits={abs=0pt},
  xmin=0.0,
  xmax=0.027,
  xticklabel style={
        /pgf/number format/fixed,
        /pgf/number format/precision=3
},
  scaled x ticks=false,
  ymajorticks=false,
  xlabel={$Shift\ [ws\ units]$},
  ylabel={$Axis$},
  x tick label style={font=\footnotesize},
  legend style={at={(0.8,0.6)},anchor=west, fill opacity=0.9, text opacity=1, draw opacity=1, nodes={scale=1, transform shape}},
  legend entries={x,y,z},
  area legend,
   reverse legend,
  enlarge y limits={abs=0.5cm},
  ymax = 1,
  bar width=1cm
]
\def \shift {0.0\linewidth}
\addplot[xshift=\shift, color=white, fill=mainRed] table[x=disp,y expr=\coordindex]{\xt};
\addplot[xshift=\shift, color=white, fill=mainBlue] table[x=disp,y expr=\coordindex]{\yt};
\addplot[xshift=\shift, color=white, fill=mainOrange] table[x=disp,y expr=\coordindex]{\zt};
\end{axis}
\end{tikzpicture}
}
\caption{\label{fig:translationsRotationsGeneral} The charts show the average measured limits of the translations and rotations along the three basic Cartesian axes. 
Translation along the $x$ axis and rotation along the $z$ axis are the ones that users tolerate most.}
\end{figure*}

Reprojection can significantly loosen the limits according to Fig.~\ref{fig:improvementGeneral}.
This correction of the distorted images is most efficient at rotational distortions because of the absence of parallax-related artifacts compared to the translational ones. 
Generally, the reprojection can be used in almost all cases to correct the distorted quilt by hundreds of percent.

\begin{figure}[!htbp]
\pgfplotsset{every axis plot post/.append style={ultra thick, draw=none, mark=none}}
\centering
\pgfplotstableread{
type impr
translation	303.053646877781
rotation	2102.5154152022
}\x
\pgfplotstableread{
type impr
translation	530.762196104076
rotation	1933.27110750022
}\y
\pgfplotstableread{
type impr
translation	362.04790725395
rotation	762.965228843212
}\z
\begin{tikzpicture}
\begin{axis}[xbar,yticklabels from table={\x}{type},
  width=0.9\textwidth,
  height=0.25\textwidth,
  xmode=normal,
  enlarge x limits={abs=0pt},
  xmin=100,
  xmax=2150,
  ymin=0,
  xticklabel style={
        /pgf/number format/fixed,
        /pgf/number format/precision=0
},
  scaled x ticks=false,
  xlabel={$Improvement\ [\%]$},
  ylabel={$Transform$},
  ytick=data,
  legend style={at={(0.85,0.3)},anchor=west, fill opacity=0.9, text opacity=1, draw opacity=1, nodes={scale=1, transform shape}},
  legend entries={x,y,z},
  bar width=0.5cm,
  area legend,
  reverse legend,
  enlarge y limits={abs=0.5cm},
  bar width=1cm
]
\def \shift {0.0\linewidth}
\addplot[xshift=\shift, color=white, fill=mainRed] table[x=impr,y expr=\coordindex]{\x};
\addplot[xshift=\shift, color=white, fill=mainBlue] table[x=impr,y expr=\coordindex]{\y};
\addplot[xshift=\shift, color=white, fill=mainOrange] table[x=impr,y expr=\coordindex]{\z};
\end{axis}
\end{tikzpicture}
\caption{\label{fig:improvementGeneral} The effect of the reprojection is demonstrated in this chart. 
The percentage improvement of the measured limit is most significant in rotations, especially around the $x$ axis.}
\end{figure}

\input{misc/chartCode.tex}
\subsubsection{SSIM Comparison}
The optimal quilt was compared, using the SSIM metric, with the quilt acquired with the amount of the distortion corresponding to the average measured value.
This ensures an objective measurement of the difference in the views after the trajectory distortion; see Fig.~\ref{fig:ssim}. 
The chart indicates two main facts. Users tolerate more pixel difference in:
\begin{itemize}
\item rotations than in translations,
\item linear distortions than in jagged ones.
\end{itemize}

\chart{$SSIM$}{0.0,0.5,1.0}{1}{ssim}{The chart contains SSIM values of the distorted quilt compared to the optimal. 
The amount of the distortions corresponds to the average values from the measurements. 
The lower the value, the greater the tolerance of the participants for the given camera distortion.}{0.42}{1.05}{0.5}

This result proves that the perception of the image on the LKG does not depend only on the pixel difference, but also on the type of transformation which might affect how the human brain interprets the artifacts. 
Note that the values in the chart would not differ if the pixel difference of the quilt was the only parameter that impacted quality.
The high SSIM values of jagged variants of the distortions are also caused by the fact that some images are the same as in the optimal quilt.
It might be surprising that the participants did not tolerate even higher distortions in the rest of the images in the jagged variants. 
Apparently, the resulting quality depends on each of the views and cannot be evaluated globally over the whole quilt because the imagery the user perceives is always a mix of three or more views.
In most of the cases, the motion of the users' head also lowers their tolerance to the artifacts.

\subsubsection{Rotations}
Fig.~\ref{fig:rotations} summarizes the measurements of the maximum angles. 
The participants marked the $z$ rotation as an unexpected behavior of the objects according to the position of the head. 
The difference between the static and moving variants is the most significant. 
Nevertheless, the measurements show that this rotation allows for the most freedom. 
The values obtained are generally slightly lower than the results of other studies on stereoscopic images~\cite{speranza2006effect}.
However, certain cases show greater tolerance, for example, the static jagged variant of the $z$ rotation without background image.
The linear variant of the tests shows higher user tolerance in the evaluation without head motion.
The situation is the same in the jagged tests with a black background. 

\chart{$Angle\ [\degree]$}{0.0,0.08,0.2,0.5,1,2.02}{0}{rotations}{The chart shows the maximum rotational angles in all three main Cartesian axes that the users marked as visually acceptable.}{0.44}{0.3}{30}

\subsubsection{Translations}
The results in Fig.~\ref{fig:translations} reveal that the translational distortion in the $x$ axis is quite robust to the artifacts, followed by the $z$ axis, which visually creates a zoom effect in the view. 
The $x$ axis translation moves the view along the direction of the optimal line trajectory, which corresponds to the orientation of the human eyes. 
The $z$ axis distortion might look more natural to the user than the $y$ axis, because the vertical motion of the objects on the $y$ axis resembles shaking of the 2D imagery, while the zooming of the objects might resemble the deformation caused by a perspective projection. 
The sitting position of the participants also reduces the expectation of the vertical motion of the rendered objects and does not exclude the possible motion of the user's head closer and farther away from the screen, which resembles the $z$ axis distortion.

\chart{$Translation$}{0.0,0.01,0.04,0.1,0.17}{0}{translations}{The chart shows the maximum amount of translation in the world space units in all three main Cartesian axes that the users marked as visually acceptable.}{0.44}{0.3}{50}

\subsubsection{Reprojection}
The 2D reprojection of the view images can significantly increase the maximum limit of the transformations; see Fig.~\ref{fig:compensation}.
The reprojection is most efficient at rotational distortions because of the lack of parallax effect.
As expected, the removal of the background significantly increases the efficiency of the reprojection in almost all cases.
The expected depth used in the reprojection process (Eq.~\ref{eq:view}) corresponds to the zero-parallax plane that is placed at the boxes.
The best cases for reprojection ($45-60{\times}$ improvement) are the jagged variants of $x$ and $y$ axis rotations. 
The most significant improvement was detected at the inverted circular trajectory ($30{\times}$).

\chart{$Improvement\ [\%]$}{0.0,100,1000,6000.5}{2}{compensation}{The chart shows the average improvement of the measured limits at each transformation after applying the 2D reprojection to correct the distortions.
The original limit is considered to be $100\%$. 
The values are calculated from the average measured limits in each category.}{0.5}{0.8}{60}


\subsubsection{Extra Measurements}
The linear trajectory can be interpolated into half circle only by the factor of 0.08 ($1=$ half circle), which proves that circular capture is unusable.
Users marked the inverted circular distortion as the most artifact-prone transformation of all measurements. 

For a simple stereoscopic image, where the user is not expected to move around, the view cone can be lowered almost to half of the recommended value for the background scene. 
According to the measurements, the view cone value can generally be set in $[13\degree;50\degree]$ range.

The participants mostly set the value of the depth of field low enough to mitigate the ghosting artifacts. 
The optimal amount of blur seems to be directly proportional to the view-cone value. Doubled view-cone requires doubled radius of the blur.
Users tended to increase the blur when head motion was allowed; see Tab.~\ref{tab:dof}.

\begin{table}[ht!]\centering
\begin{tabular}{ c c c }
 view cone $[\degree]$ & head & avg radius \\
 \toprule
 \multirow{2}{*}{30} & static & 0.562100 \\ 
                     & moving & 1.865934 \\
 \multirow{2}{*}{60} & static & 2.287247 \\
                     & moving & 2.383613 \\
\end{tabular}
\caption{\label{tab:dof} Measured average radius of the Gaussian blur in the depth-of-field effect simulation is shown in the table.}
\end{table}