\section{3D Displays}
\label{chap:displays}
This section describes the technology of 3D displays that simulate 3D perception for the user. 
These devices are tightly related to light fields, as they physically implement their multi-view and optical features.
Fig.~\ref{fig:displayModels} shows various types of 3D display devices.

\begin{figure}[!htbp]\centering
  \begin{overpic}[width=\linewidth]{obrazky-figures/displayModels.jpg}
  \color{white}
  \put(2,45){\textbf{2D display}}
  \put(45,39){\textbf{Anaglyph}}
  \put(46,36){\textbf{glasses}}
  \put(64,40){\textbf{Stereoscopic display}}
  \put(57,25){\textbf{3D tablet}}
  \put(27,20){\textbf{Stereoscopic}}
  \put(27,15){\textbf{volumetric display}}
  \put(77,15){\textbf{Virtual reality}}
  \end{overpic}
   \caption{
Multiple 3D display devices that represent categories from large displays for movie watching, smaller sizes for interactive usage or previews, almost pocket-size devices, and equipment wearable on user's head.}
  \label{fig:displayModels}
\end{figure}

On classic displays, users can view a 2D projection of a 3D scene. 
The depth of the scene can be perceived from the motion of the camera~\cite{ferris1972motion, gibson1959motion}.
Objects at different depths would have different apparent velocity.
The perceived velocity difference is called the motion parallax.
Moving objects in the scene can also serve as a depth cue, even without parallax~\cite{swanston1986perceived, sperling1994depth}.
When objects move closer and farther from the camera, the projected size change indicates the distance of the object.
Perspective projection can similarly indicate depth relations in the scene.
Textures, transformed by geometry from the original 2D view, are another important depth cue~\cite{o2000texture}.
Other depth cues exist, such as eye accommodation, learned expectations, contextual analysis of the scene, shadows, etc.~\cite{okoshi2012three}.

Binocular depth cues~\cite{mehrabi2013, el2019survey} play an important role in human depth perception.
A straightforward way to simulate this perception is to capture two images of the same scene from a slightly different position.
These images can be projected separately into each eye, creating the 3D illusion~\cite{boas2013overview}.
Binocular parallax creates the 3D feel by a projection of two 2D images that capture the scene from different positions, one for each eye.
Monocular motion parallax is based on the occlusion and motion of the objects in the scene according to the position of the users' heads. 
Several models of devices, called head-mounted displays (HMDs), are commonly used among consumers, usually referred to as virtual reality devices.
Another way to project different images into each eye is to use light-blocking glasses~\cite{cho2013effects}.
The two images can be simultaneously displayed on a screen, each image with different polarization.
Polarized glasses on users' heads then pass through only one of the images to each eye.
An alternate approach is to use a shutter display.
The display cyclically switches between the two images at high frequency.
The shutter glasses are synchronized with this switching and pass through the currently projected image only to the associated eye.

Volumetric displays use a rotating mirror and a projector with a high frame rate that is synchronized with the mirror rotation frequency~\cite{jones2007rendering}. 
3D scene can then be rendered from multiple angles, and each of the views is then reflected by the mirror in the right direction.
Users see different parts of the scene depending on their position and viewing angle at the mirror.
A more robust solution using a similar principle with a rapidly vertically moving flat screen, on which a different part of the 3D image is projected depending on its current position, are Voxon Photonics products~\cite{keane2019volumetric}.

A different approach that uses the viewing angle-based ray distribution is the Tensor Display~\cite{hirsch2012tensor}.
It uses multiple light-attenuating layers that modify the light coming from the source display in the form of directional backlight, which can be simulated by the lenslet array on the display. 
A compact solution in the form of a 3D tablet by Leia Inc. uses diffractive light field backlighting~\cite{kimmel2012diffractive, fattal2021}.
Four views of the same scene are projected in 16 directions with a 2D backlight, proprietary optical layer, and an LCD panel.
Rapid temporal modulation showing different frames in a short period of time helps to widen the angle of view showing different parts of the scene through the synchronized layers.
Similar technology was used in the RED Hydrogen One holographic smartphone~\cite{williams2019}.
A holographic screen, where each point modifies light rays emitted from the optical module layer in various directions, is used in the HoloVizio display by Holografika~\cite{balogh2007holovizio}. 

\subsection{Visual Discomfort}
\label{sec:discomfort}
An unnatural stereoscopic input into the human visual system could cause discomfort and eye strain, resulting in excessive brain activity or unpleasant eye conditions such as heterophoria~\cite{karpicka2013}.
This discomfort can lead to cybersickness~\cite{Alhassan2021}.
The effect of varying camera separation, display duration, focal length, and convergence distance on visual perception using stereoscopic displays has already been documented~\cite{wijnand2000, IJsselsteijn2000}.
The eye strain is more severe with increasing disparity between images.
The vertical parallax in virtual reality devices is not well tolerated by the human visual system and should not exceed 20$''$~\cite{moreau2014} or 7 mm for two homologous points~\cite{woods2002}.
An experiment carried out revealed that tilt angle differences of the 3D screen greater than 30\degree{} cause noticeable visual discomfort~\cite{sugita2019}.
Similar rules are valid even in the field of augmented reality~\cite{kruijff2010}.
Correction of inappropriately shifted or rotated stereoscopic views was proposed and tested by image transformation, revealing the difference between translational and rotational disparities, where rotation was shown to be easier to correct~\cite{livingston2006}.
An objective metric exists to predict visual fatigue from stereoscopic images that measure vertical and horizontal disparities, but considers only two views~\cite{donghyun2011}.
The effect of compression was also evaluated, revealing that depth perception does not change significantly with increasing compression ratio, but eye strain is much more significant due to the reduction in image sharpness~\cite{seuntiens2006}.
However, compression artifacts are less visible in scenes with large disparities~\cite{mohona2021subjective}.

Previously published papers~\cite{gao2018, li2014gen, woods2002} addressed stereoscopic screens and input image distortions with respect to user depth perception. 
However, novel 3D displays combine stereoscopy with the view-dependent 3D effect in a relatively large viewing cone in front of the screen.
The previously discovered rules might not be exactly the same for current state-of-the-art 3D display technology.
The issue of distorted capturing camera trajectory is usually not present in already well-explored stereoscopic images because the stereo cameras are usually physically bound together and cannot move independently. 
Camera distortions can be partially fixed with stereo stabilization algorithms~\cite{guo2018, liu2013}.
Parallax of 2\degree{} was shown to cause visual discomfort with stereoscopic displays~\cite{liu2021}. 
Another work stated 40$'$ as the upper limit of the comfort zone parallax~\cite{wang2023assessment}. 
The same paper contained information that young people are more prone to visual sickness than old ones.
Research in the field of visual discomfort shows that input views presented on 3D displays need to follow certain rules.
Otherwise, the quality of user experience might deteriorate.

\subsection{Looking Glass}
\label{sec:lkgTheory}
The state-of-the-art multiview display by Looking Glass Factory (LKG)~\cite{frayne2018advanced, frayne2019superstereoscopic} is designed to directly display a 3D scene without any additional equipment, such as glasses, similarly to lenticular displays~\cite{takaki2010multi}.
This device consists of a classic 2D display, a microlens layer, and a glass block on top.
USB output is also included, since the device calibration with the display and microlense parameters is sent to the rendering computer to ensure the correct pixel alignment and optimal 3D projection.

Multiple views are displayed at once on the underlying display in a defined pattern; see Fig.~\ref{fig:lkgProcess}.
The microlenses redirect the rays per pixel in the right directions, so that it is possible to see different views according to the user's position.
The glass layer refracts the rays coming out of the background display so that they reach wider viewing angles.
One of the limitations of this display is only horizontal parallax, where users are expected to be sitting in front of the screen and tilting their heads from side to side.
This device can be considered as a hardware implementation of the one-directional discrete light field approximation.
Similarly to 1D light field rendering, an array of images, taken along a horizontal line with constant spaces between the camera positions, is used as an input; see Fig.~\ref{fig:lkgCapture}.
The display renders the appropriate view based on the user's viewing angle and also blends neighboring views to mask out abrupt changes of the views.

\begin{figure}[!htbp]\centering
  \includegraphics[width=\linewidth]{obrazky-figures/lkgProcess.pdf}
  \caption{
Evenly distributed cameras on a horizontal line capture the scene as a set of images that are merged into one matrix (quilt) representing a 1D light field.
The top left image is the first one from the linear sequence and the bottom right the last one. 
The quilt is then transformed into the internal LKG format which can be directly displayed on LKG. 
The optical block on the device distributes the pixels in the correct directions.}
  \label{fig:lkgProcess}
\end{figure}

\begin{figure}[!htbp]\centering
  \includegraphics[width=0.7\linewidth]{obrazky-figures/detection/lkgCapture.pdf}
  \caption{The figure shows the optimal orientation of the \textcolor{orange}{capturing cameras} which are placed in the scene according to the horizontal 3D display orientation. \textcolor{blue}{Users} then see a combination of the captured views according to their viewing position.}
  \label{fig:lkgCapture}
\end{figure}

\subsubsection{Focusing}
Focusing of the scene is closely related to a similar issue in light field rendering~\cite{levoy1996light}.
A distance from the camera's near plane where all projected pixels are in focus is called the zero-parallax plane. 
Such pixels are spatially static and do not show any parallax motion when users change their viewing angle.
Their color can change due to reflections or other lighting effects.
The parallax is visible in the defocused parts of the scene and is one of the important depth cues that simulate the resulting 3D effect~\cite{reichelt2010}.
An example of different focus distances and the parallax effect on the physical device is visible in Fig.~\ref{fig:focusingExample}.
The geometric representation of the focus problem is shown in Fig.~\ref{fig:lkgDisparity}.

\begin{figure*}[!htbp]\centering
    \def \imageSize{0.25\linewidth}
    \subfloat[\centering  ]{{\includegraphics[width=\imageSize]{obrazky-figures/mitigation/focusExampleLeftFront.jpg} }} 
    \subfloat[\centering ]{{\includegraphics[width=\imageSize]{obrazky-figures/mitigation/focusExampleRightFront.jpg} }}
    \subfloat[\centering
    ]{{\includegraphics[width=\imageSize]{obrazky-figures/mitigation/focusExampleLeftBack.jpg} }} 
    \subfloat[\centering ]{{\includegraphics[width=\imageSize]{obrazky-figures/mitigation/focusExampleRightBack.jpg} }}
    \\
    \subfloat[\centering Parts of defocused areas from each view are zoomed.]{{\includegraphics[width=0.8\linewidth]{obrazky-figures/mitigation/focusExampleZoom.jpg} }}
    \\
  \caption{\label{fig:focusingExample} The figure contains a 15.6$''$ LKG device with the same scene, captured from the left (a,c) and the right (b,d) side and with zero-parallax plane positioned at the front (a,b) and back (c,d) object. 
  The parallax is visible when the view angle changes.
  The out-of-focus blur resembles ghosting artifacts due to the mixing of the rendered views.}
 \end{figure*}

\begin{figure}[!htbp]\centering
  \includegraphics[width=0.6\linewidth]{obrazky-figures/mitigation/lkgDisparity.pdf}
  \caption{\label{fig:lkgDisparity} Point \textcolor{mainOrange}{$p_1$} lies on the zero-parallax plane and is focused from all cameras \textcolor{mainBlue}{$A,B,C$}.
  Point \textcolor{red}{$p_2$} lies further away and is projected on the screen with disparities \textcolor{red}{$d_{AB}, d_{BC}$} between the views.}
\end{figure}

Focusing is simulated by displacement of the input images according to their position on the capturing trajectory. 
The method is similar to the shift-sum algorithm~\cite{alain2019} used in light field rendering~\cite{isaksen2000}; see Fig.~\ref{fig:focusingDisplacement}.
Light field function $L$ describes light from a given angular $\mathbf{u}$ and spatial $\mathbf{x}$ coordinates with the refocusing parameter $focus$; see Eq.~\ref{eq:lf}. 

\begin{equation}
  focused = \int_{\mathbb{R}} \textrm{L}(\mathbf{x}-\mathbf{u} \cdot focus,\mathbf{u})\mathrm{d}\mathbf{u}
  \label{eq:lf}
\end{equation}

After shifting, the last pixel is repeated at the edge of the image to avoid visual discontinuities.
The $focused$ result, where the $i$th image of the total $N$ images of the scene is shifted ($\textrm{shiftImage}_{i}(amount)$) by the $focus$ value, which can be relative or in pixels, can be calculated by Eq.~\ref{eq:focus}. 
The summation can be a specific pixel distribution operation, for example, conversion into the internal format of the LKG. 
The images can be moved to the left or to the right, while the center image remains static.

\begin{equation}
  focused = \sum_{i=0}^{N} \textrm{shiftImage}_{i}(focus \cdot (1 - \frac{2i}{N}))
  \label{eq:focus}
\end{equation}

\begin{figure*}[!htbp]\centering
  \includegraphics[width=0.98\linewidth]{obrazky-figures/mitigation/focusingDisplacement.pdf}
  \caption{\label{fig:focusingDisplacement} The scene is captured from multiple positions. 
  The images can be mixed together as they are, leading to ghosting artifacts.
  The zero-parallax plane can be changed by shifting the images so that the desired part of the scene is in focus (constant screen position) but does not show parallax.}
\end{figure*}

The combination of the images from the quilt into the internal format is performed according to Eq.~\ref{eq:internal}, where $h$ is the index of the RGB channel ($R=0, G=1, B=2$), $\textrm{I}$ function samples an input input image at the given coordinates, and the internal format image access function $\textrm{I}_{lkg}$ is extended by the parameter to index the specific channel.
The calibration parameters, defined by the LKG manufacturer, are as follows: $s$ is subpixel shift, $t$ is tilt, $p$ is pitch, $v$ is view portion, and $c$ is center shift.
The total number of views is defined as $N$, $\lfloor \rfloor$ means floor operation, and $\{ \}$ is the extraction of fractional part.

\begin{equation}
\textrm{I}_{lkg}(x,y,h) = \textrm{I}(\lfloor \left \{(x + sh + yt)p-c\right \}N \rfloor, vx, vy)	
	\label{eq:internal}
\end{equation}

Each physical LKG device comes with a different set of these parameters.
The equation allows to directly render the quilt on the display without any proprietary software as it distributes the pixels according to the structure of the microlens array on LKG.
The focusing can be achieved by the transformation of $x,y$ coordinates similarly to Eq.~\ref{eq:trans}.

The user always sees multiple mixed views at the same time.
Their disparity leads to ghosting artifacts in parts far from the zero-parallax plane. 
Scenes with a large depth range would show more parallax and 3D feel, but they would contain more ghosting artifacts in the defocused parts.
A naive approach to create an optimal all-focused result would be to split the image into parts, where each part would be focused with a different zero-parallax plane position. 
This principle is similar to the per-pixel focusing method from Section~\ref{sect:realtime}.
However, this would cause discontinuities when changing viewing angles and an unnatural motion of the objects that would rotate instead of moving to the sides according to the parallax; see Fig.~\ref{fig:splitFocus}.

\begin{figure}[!htbp]\centering
   \def \imageSize{0.48\linewidth}
    \subfloat[\centering Left view shows abrupt splitting of the scene and duplicated objects. ]{{\includegraphics[width=\imageSize]{obrazky-figures/mitigation/splitFocusB.pdf} }} 
    \subfloat[\centering Right view shows a discontinuity on the ground. ]{{\includegraphics[width=\imageSize]{obrazky-figures/mitigation/splitFocusA.pdf} }}
  \caption{\label{fig:splitFocus} Two zero-parallax planes (different focus distance for each half of the image) were used to keep both objects focused.
  The parallax is visible and is sharply cut at the middle of the image. 
  The motion of the objects is also unnatural.
  Both are rotating instead of one seemingly moving to the sides.}
\end{figure}