\section{GPU-Accelerated Decompression of Discrete Light Field}
\label{sec:compression}
The following sections contain a deeper description of a HW accelerated decoding method~\cite{chlubna2024eff} suitable for the main proposals of this thesis.
These methods are the author's original extension of the state-of-the-art field.
Existing compression standards were investigated, and, based on the results, a novel compression scheme was proposed.
These experiments and methods are not claimed to be the main scientific contribution of this thesis but are necessary to support the main proposal.
Acceleration of decompression is important in light field rendering because the decompression process might be utilized several times during each rendering time frame.

\subsection{Evaluation of Light Field Compression Methods}
\label{sect:compressionEvaluate}
This section is based on two published papers~\cite{barina2019, barina2022}.
The impact of state-of-the-art image and video compression methods on the quality of images rendered from light field data was evaluated.
The results presented here serve as motivation for the usage of video encoding in the following light field compression scheme.
The main question was: \textit{Which of the widely used state-of-the-art image/video compression methods is the optimal choice for light field data?}
Four-dimensional light field data can be compressed much more than independent still images while maintaining the same visual quality.
The video compression methods seem to be an efficient solution for light field compression due to the best compression ratio and wide support in the existing software and hardware.

Both main light field formats are included in the measurements performed.
Light fields acquired by a single compact sensor have limited support for the viewing angle.
Light fields based on the array of cameras offer larger viewing angles at the cost of missing information in between the cameras.
Considering increasing resolution sensors, it is no surprise that the light field data reach huge sizes.
For example, a scene from the standard dataset~\cite{vaish} is captured using a $17{\times}17$ grid of cameras with image resolution $1536{\times}1152$ (rectified and cropped).
The uncompressed size easily exceeds one gigabyte.

\subsubsection{Dataset}
The first group of measured methods covers the image coding methods JPEG and JPEG 2000. 
They are sometimes referred to as self-similarity-based methods~\cite{Li2018}.
The second group comprises the video coding methods H.265, AV1, VP9, XVC, and VVC.
These methods are referred to as pseudo-sequence-based methods.
The third group consists of JPEG 3D custom extension and JPEG 2000 3D (Part 10, JP3D).
The fourth group extends the image-coding methods into four dimensions with a custom JPEG 4D extension.
To evaluate the above methods, the following list of encoders was used: 
OpenJPEG\footnote{\href{https://www.openjpeg.org}{openjpeg.org}},
x265\footnote{\href{https://github.com/videolan/x265}{github.com/videolan/x265}},
libaom (AV1 Codec Library)\footnote{\href{https://aomedia.googlesource.com/aom/}{aomedia.googlesource.com/aom}},
libvpx (VP8/VP9 Codec SDK)\footnote{\href{https://github.com/webmproject/libvpx}{github.com/webmproject/libvpx}},
Divideon xvc codec\footnote{\href{https://github.com/divideon/xvc}{github.com/divideon/xvc}},
Fraunhofer vvc codec\footnote{\href{https://github.com/fraunhoferhhi/vvenc}{github.com/fraunhoferhhi/vvenc}}
and custom implementation of the JPEG method.
Tab.~\ref{tab:datasetCompression} describes the dataset used in the measurements.

\begin{table}[ht]
\centering
\begin{tabular}{llrr}
\input{tables/compressionDataset.tex}
\end{tabular}
\caption{
Dataset used in this study.
The first and last light field are taken using a plenoptic camera; the Chessboard is captured using a camera array; the Lego Bulldozer is captured using a motorized gantry holding a camera.
The adjacent image disparity range (last column) is given in pixels.
}
\label{tab:datasetCompression}
\end{table}

\subsection{Evaluation}
A simple comparison of compressed and uncompressed raw data is not enough to evaluate the encoding of light fields.
The uncompressed data were used in a light field refocusing rendering, and the data before and after compression were compared.
Fig.~\ref{fig:dfd} describes the evaluation scheme used.
A quick experiment shows a significant disparity in the methods used; see Fig.~\ref{fig:compressionExperiment0}.
This difference is about 10 decibels in the PSNR, depending on the bitrate and compression method.
This can be explained by the fact that any pixel in the rendered view is a sum of pixels from the 4D light field.
This sum suppresses compression artifacts.
The 4D light fields can be compressed much more than independent images, while maintaining the same visual quality.

\begin{figure}[!htbp]\centering
\includegraphics[width=\textwidth]{obrazky-figures/compression/dfd.pdf}
  \centering
   \caption{The data flow diagram describes the compression performance assessment methodology used in this study.}
 \label{fig:dfd}
\end{figure}

\begin{figure}[!htbp]
\centering
\begin{tikzpicture}
\pgfplotstableread{charts/compression/experiment0/AV1.txt}\experimentAV
\pgfplotstableread{charts/compression/experiment0/AV1R.txt}\experimentAVR
\pgfplotstableread{charts/compression/experiment0/JPEG.txt}\experimentJPEG
\pgfplotstableread{charts/compression/experiment0/JPEGR.txt}\experimentJPEGR
\pgfplotstableread{charts/compression/experiment0/JPEG2000.txt}\experimentJPEGG
\pgfplotstableread{charts/compression/experiment0/JPEG2000R.txt}\experimentJPEGGR
\begin{axis}[
  xlabel={$Bitrate\ [bpp]$},
  ylabel={$PSNR\ [dB]$},
  xmode=log,
  legend entries={AV1, AV1 (refocus), JPEG, JPEG (refocus), JPEG 2000, JPEG 2000 (refocus)},
  legend style={at={(1.8,0.5)},anchor=east}
]

\addplot[color=mainLightRed, mark=o, ultra thick] table[x=br, y=psnr]{\experimentAV};
\addplot[color=mainRed, mark=o, ultra thick] table[x=br, y=psnr]{\experimentAVR};
\addplot[color=mainLightBlue, mark=o, ultra thick] table[x=br, y=psnr]{\experimentJPEG};
\addplot[color=mainBlue, mark=o, ultra thick] table[x=br, y=psnr]{\experimentJPEGR};
\addplot[color=mainLightOrange, mark=o, ultra thick] table[x=br, y=psnr]{\experimentJPEGG};
\addplot[color=mainOrange, mark=o, ultra thick] table[x=br, y=psnr]{\experimentJPEGGR};
\end{axis}
\end{tikzpicture}
\caption{
The chart shows the difference in the quality assessment using the 4D light field directly compared to using images rendered at virtual focal planes.
The results are measured with the \textit{Black Fence} scene.}
\label{fig:compressionExperiment0}
\end{figure}

\subsubsection{2D, 3D and 4D Compression}
To fairly assess the difference between multidimensional compressions, an identical method must be used.
A custom implementation of the JPEG compression method was created with the ability to process 2D, 3D, or 4D data.
Although JPEG 2000 supports 2D and 3D data compression, it cannot process 4D images.
Since the similarity of adjacent pixels in the third and four dimensions strongly depends on the camera baseline, different results can be expected depending on the baseline distance.
The result of this experiment is shown in Fig.~\ref{fig:compressionExperiment1}.

\begin{figure}[!htbp]
\centering
\newcommand{\experimentChart}[3]
{
\def \labels{JPEG 2D, JPEG 3D, JPEG 4D, JPEG 2000 2D, JPEG 2000 3D}
\def \labelX{\hspace{150pt}$PSNR\ [dB]$}
\ifnum0=#2\relax
    \def \labels{}
\fi

\ifnum0=#3\relax
    \def \labelX{}
\fi

\begin{tikzpicture}
\pgfplotstableread{charts/compression/experiment1/#1JPEG2D.txt}\experimentA
\pgfplotstableread{charts/compression/experiment1/#1JPEG3D.txt}\experimentB
\pgfplotstableread{charts/compression/experiment1/#1JPEG4D.txt}\experimentC
\pgfplotstableread{charts/compression/experiment1/#1JPEG20002D.txt}\experimentD
\pgfplotstableread{charts/compression/experiment1/#1JPEG20003D.txt}\experimentE
\begin{axis}[
  %xlabel={$Bitrate\ [bpp]$},
  width=0.45\linewidth,
  xmode=log,  
  legend entries/.expand once={\labels},
  legend style={at={(0.3,1.0)},anchor=north}
]
\addplot[color=mainBlue, mark=o, ultra thick] table[x=br, y=psnr]{\experimentA};
\addplot[color=mainLightBlue, mark=o, ultra thick] table[x=br, y=psnr]{\experimentB};
\addplot[color=mainRed, mark=o, ultra thick] table[x=br, y=psnr]{\experimentC};
\addplot[color=mainOrange, mark=o, ultra thick] table[x=br, y=psnr]{\experimentD};
\addplot[color=mainLightOrange, mark=o, ultra thick] table[x=br, y=psnr]{\experimentE};
\end{axis}
\end{tikzpicture}
}
\makebox{\rotatebox[origin=c]{90}{\centering $PSNR\ [dB]$}}
\subfloat[\centering \textit{Black Fence} is measured.]{\experimentChart{BlackFence}{0}{1}}
\subfloat[\centering \textit{Chess} is measured.]{\experimentChart{Chess}{0}{0}}
\\
\subfloat[\centering \textit{Lego} is measured.]{\experimentChart{Lego}{0}{1}}
\subfloat[\centering \textit{Palais} is measured.]{\experimentChart{Palais}{1}{0}}
\\
\subfloat{$\hspace{50pt} Bitrate\ [bpp]$}
\addtocounter{subfigure}{-1}
\caption{
Image compression methods are compared against their extensions into three and four dimensions.}
\label{fig:compressionExperiment1}
\end{figure}

3D compression methods clearly outperform their 2D counterparts on light fields with a small baseline (\textit{Black Fence} and \textit{Palais du Luxembourg}).
4D JPEG also outperforms its 3D counterpart.
Pixels at the same spatial position in adjacent views are strongly correlated. 
However, the situation changes with increasing baseline (\textit{Lego Bulldozer} and \textit{Chessboard}).
Adjacent views are less and less similar, resulting in higher amplitudes of the underlying transform coefficients.
Consequently, the tide is turning in favor of the less-dimensional compression methods.
Taking into account the JPEG method, the \textit{Lego Bulldozer} is a special case because it contains large areas of black pixels.
It turns out that it is more efficient to compress these solid areas at once using a single 4D block than using multiple 3D blocks.
Similarly, it is more efficient to use a single 3D block than multiple 2D blocks.

\subsubsection{Video Compression}
4D light fields can be compressed as a sequence of 2D frames, as multi-dimensional volume, or as a video.
The overall comparison is shown in Fig.~\ref{fig:compressionExperiment2}.
Video compression methods perform better than all image compression methods, even better than their 3D and 4D extensions.

\begin{figure}[!htbp]
\centering
\newcommand{\experimentChart}[3]
{
\def \labels{XVC, AV1, JPEG 4D, JPEG 2000 2D}
\def \labelX{$PSNR\ [dB]$}
\ifnum0=#2\relax
    \def \labels{}
\fi

\ifnum0=#3\relax
    \def \labelX{}
\fi

\begin{tikzpicture}
\pgfplotstableread{charts/compression/experiment2/#1-plot-refocus-av1.txt}\experimentA
\pgfplotstableread{charts/compression/experiment2/#1-plot-refocus-jpeg-4d.txt}\experimentB
\pgfplotstableread{charts/compression/experiment2/#1-plot-refocus-lf2.txt}\experimentC
\pgfplotstableread{charts/compression/experiment2/#1-plot-refocus-xvc.txt}\experimentD
\begin{axis}[
  width=0.45\linewidth,
  %xlabel={$Bitrate\ [bpp]$},
  %ylabel={\labelX},
  xmode=log,  
  legend entries/.expand once={\labels},
  legend style={at={(0.85,0.01)},anchor=south}
]
\addplot[color=mainBlue, mark=o, ultra thick] table[x=br, y=psnr]{\experimentD};
\addplot[color=mainLightBlue, mark=o, ultra thick] table[x=br, y=psnr]{\experimentA};
\addplot[color=mainLightOrange, mark=o, ultra thick] table[x=br, y=psnr]{\experimentB};
\addplot[color=mainOrange, mark=o, ultra thick] table[x=br, y=psnr]{\experimentC};

\end{axis}
\end{tikzpicture}
}
\raisebox{70pt}{\rotatebox[origin=c]{90}{\centering $PSNR\ [dB]$}}
\subfloat[\centering \textit{Black Fence} is measured.]{\experimentChart{Black_Fence}{0}{1}}
\subfloat[\centering \textit{Chess} is measured.]{\experimentChart{Chessboard}{1}{1}}
\\
\subfloat{$Bitrate\ [bpp]$}
\addtocounter{subfigure}{-1}
\caption{
Video compression methods perform better than all image compression methods according to the conducted experiment.}
\label{fig:compressionExperiment2}
\end{figure}

The video compression formats were compared in the refocusing scenario; see Fig.~\ref{fig:compressionExperiment3}.
Another measurement was performed to additionally evaluate the effect of compression on 3D point cloud reconstruction from light field; see Fig.~\ref{fig:compressionExperiment4}.
A distortion is apparent at lower bitrate, which is caused by noise and a reduced number of vertices present in the point clouds.
Video compression methods show similar results with the newest standards slightly outperforming the old ones. 
These methods might be an optimal way to compress light field data with a wide support in current software and hardware.

\begin{figure}[!htbp]
\centering
\newcommand{\experimentChart}[3]
{
\def \labels{XVC, AV1, VVC, H.265, JPEG 4D}
\def \labelX{$PSNR\ [dB]$}
\ifnum0=#2\relax
    \def \labels{}
\fi

\ifnum0=#3\relax
    \def \labelX{}
\fi

\begin{tikzpicture}
\pgfplotstableread{charts/compression/experiment3/#1/refocus-xvc-8bpp-yuv444p.sort.txt}\experimentA
\pgfplotstableread{charts/compression/experiment3/#1/refocus-av1-8bpp-yuv444p.sort.txt}\experimentB
\pgfplotstableread{charts/compression/experiment3/#1/refocus-vvc-8bpp-yuv444p.sort.txt}\experimentC
\pgfplotstableread{charts/compression/experiment3/#1/refocus-x265-8bpp-yuv444p.sort.txt}\experimentD
\pgfplotstableread{charts/compression/experiment3/#1/refocus-lf4-8bpp-yuv444p.sort.txt}\experimentE
\begin{axis}[
  width=0.45\linewidth,
  %xlabel={$Bitrate\ [bpp]$},
  ylabel={\labelX},
  xmode=log,  
  xmax=1,
  legend entries/.expand once={\labels},
  legend style={at={(0.85,0.01)},anchor=south}
]
\addplot[color=mainBlue, mark=o, ultra thick] table[x=br, y=psnr]{\experimentD};
\addplot[color=mainLightBlue, mark=o, ultra thick] table[x=br, y=psnr]{\experimentA};
\addplot[color=mainLightOrange, mark=o, ultra thick] table[x=br, y=psnr]{\experimentB};
\addplot[color=mainOrange, mark=o, ultra thick] table[x=br, y=psnr]{\experimentC};
\addplot[color=mainRed, mark=o, ultra thick] table[x=br, y=psnr]{\experimentE};

\end{axis}
\end{tikzpicture}
}

\subfloat[\centering \textit{Ha Toy} is measured.]{\experimentChart{hatoy}{0}{1}}
\subfloat[\centering \textit{Lego} is measured.]{\experimentChart{lego}{1}{1}}
\\
\subfloat{$Bitrate\ [bpp]$}
\addtocounter{subfigure}{-1}
\caption{
Different compression formats are compared on views rendered for multiple focal planes.}
\label{fig:compressionExperiment3}
\end{figure}

\begin{figure}[!htbp]
\centering
\newcommand{\experimentChart}[3]
{
\def \labels{XVC, AV1, VVC, H.265, JPEG 4D}
\def \labelX{$RMS\ error$}
\ifnum0=#2\relax
    \def \labels{}
\fi

\ifnum0=#3\relax
    \def \labelX{}
\fi

\begin{tikzpicture}
\pgfplotstableread{charts/compression/experiment4/#1/pointcloud-xvc-8bpp-yuv444p.sort.txt}\experimentA
\pgfplotstableread{charts/compression/experiment4/#1/pointcloud-av1-8bpp-yuv444p.sort.txt}\experimentB
\pgfplotstableread{charts/compression/experiment4/#1/pointcloud-vvc-8bpp-yuv444p.sort.txt}\experimentC
\pgfplotstableread{charts/compression/experiment4/#1/pointcloud-x265-8bpp-yuv444p.sort.txt}\experimentD
\pgfplotstableread{charts/compression/experiment4/#1/pointcloud-lf4-8bpp-yuv444p.sort.txt}\experimentE
\begin{axis}[
  width=0.5\linewidth,
  %xlabel={$Bitrate\ [bpp]$},
  ylabel={\labelX},
  xmode=log,  
  xmin=0.001,
  legend entries/.expand once={\labels},
  legend style={at={(0.85,0.5)},anchor=south}
]
\addplot[color=mainBlue, mark=o, ultra thick] table[x=br, y=rms]{\experimentD};
\addplot[color=mainLightBlue, mark=o, ultra thick] table[x=br, y=rms]{\experimentA};
\addplot[color=mainLightOrange, mark=o, ultra thick] table[x=br, y=rms]{\experimentB};
\addplot[color=mainOrange, mark=o, ultra thick] table[x=br, y=rms]{\experimentC};
\addplot[color=mainRed, mark=o, ultra thick] table[x=br, y=rms]{\experimentE};

\end{axis}
\end{tikzpicture}
}

\subfloat[\centering \textit{Ha Toy} is measured.]{\experimentChart{hatoy}{0}{1}}
\subfloat[\centering \textit{Lego} is measured.]{\experimentChart{lego}{1}{1}}
\\
\subfloat{$Bitrate\ [bpp]$}
\addtocounter{subfigure}{-1}
\caption{
Different compression formats are compared on 3D point clouds reconstructed from 4D light fields.}
\label{fig:compressionExperiment4}
\end{figure}

\subsection{Efficient Video Decoding Method}
\label{sect:videoDecoding}
The proposed method belongs to pseudo-sequence-based video compression where light field views are used as video frames; see Fig.~\ref{fig:encodingScheme}.
One selected frame serves as a fully encoded standalone reference, and other frames are encoded as motion-compensated differences from this reference.
The proposed compression is described with a 1D light field grid, where the views are captured along a horizontal line.
This method can be easily used for other shapes of the light field grids as well.

\begin{figure*}[!htbp]\centering
\includegraphics[width=\linewidth]{obrazky-figures/decoding/encodingScheme.pdf}
  \centering
   \caption{The scheme describes the video encoding principle used for efficient data streaming to GPU in light field rendering.}
 \label{fig:encodingScheme}
\end{figure*}

\subsubsection{Frame Type Sequences}
Ideally, only the necessary compressed video packets are uploaded to the GPU memory and decoded there; see Fig.~\ref{fig:lfView}.
The host and GPU memory data transfer is the bottleneck of light field streaming.
Using the classic encoding scheme, all previous packets from the GOP need to be decoded before the desired frame is acquired.
If the light field views are encoded row-by-row, then multiple GOP jumps would be necessary to decode four nearest views to the novel one.
The interval between I-frames (size of GOP) can be shortened.
The data size would be larger because of more encoded space-demanding I-frames, but the desired frame would potentially need fewer previous frames decoded.
The shortening of GOP can reach the limit, so that all frames are encoded as I-frames and can be decoded without other frames.

\begin{figure}[!htbp]\centering
\includegraphics[width=0.8\linewidth]{obrazky-figures/decoding/lfViewDepend.pdf}
  \centering
   \caption{Only a subset of the light field views is necessary for interpolation of a novel view.
   The needed frames are determined by the viewing position of the user.}
 \label{fig:lfView}
\end{figure}

\subsubsection{Compression Scheme}
The proposed optimal scheme for light field data encodes an I-frame in the middle of the sequence.
All other P-frames are encoded using only this initial I-frame as a reference.
This approach would not be efficient for arbitrary video, because of huge differences between its different parts, especially when the scene changes.
However, light field views are very similar to each other because they capture the same static scene, and the distance between the views is usually not significant enough to create large differences between the images.
Light field rendering methods often exploit this similarity. 
After decoding the initial I-frame, all other frames can be decoded simply by uploading their packets to the decoder. 
The minimum number of packets to decode a frame is one I-frame packet and one P-frame packet, but only one P-frame packet is needed to decode further frames.
The decoding speed might also be faster because the decoder only needs to read the data from one packet and also internally from one I-frame to decode the new frame.
The encoding schemes that are compared are:
\begin{itemize}
    \item \textbf{Proposed} with one I-frame in the middle of the sequence and the rest P-frames dependent on the one reference.
    \item \textbf{One-GOP} and \textbf{GOP30} are classic GOP schemes, one I-frame at the beginning, and the rest are P-frames sequentially encoded in a classic way with possible multiple dependencies and a version with I-frames enclosing every 30 frames.
    \item \textbf{All-I} where all frames are individually encoded as I-frames without any dependencies.
\end{itemize}

See Fig.~\ref{fig:packetSchemes} that compares the described schemes. 

\begin{figure}[!htbp]
    \centering
    \subfloat[All preceding frames need to be decoded sequentially in One-GOP or GOP30.]{{\includegraphics[width=0.32\linewidth]{obrazky-figures/decoding/packets.pdf} }}
    \hfill
    \subfloat[The desired frame can be decoded directly in all-I.]{{\includegraphics[width=0.32\linewidth]{obrazky-figures/decoding/packetsKey.pdf} }}
    \hfill
    \subfloat[One reference frame needs to be decoded before the desired frame in the proposal.]{{\includegraphics[width=0.32\linewidth]{obrazky-figures/decoding/packetsRandom.pdf} }}
    \caption{\textcolor{mainOrange}{P-frames} can be decoded only after all their reference frames. \textcolor{mainBlue}{I-frames} can be decoded without any other frames. \textbf{Fourth} frame is the desired one in this example.}
    \label{fig:packetSchemes}
\end{figure}

The main reason why light field assets are not commonly used instead of 3D mesh geometry is their space requirements.
One option is to keep the entire asset in the GPU memory.
The decoding and rendering would be relatively fast, but usually 3D simulations or games require a lot of GPU memory.
Storing a large number of assets like this might be inconvenient.
The second option is to stream the needed views of the light field asset in the GPU memory on demand. 
The views need to be decompressed on the CPU and transferred via PCI-Express in GPU memory.
The importance of compressed data transfer and GPU decoding has been increasing recently, as reflected in relevant proposed solutions, such as DirectStorage\footnote{\href{https://devblogs.microsoft.com/directx/directstorage-is-coming-to-pc/}{devblogs.microsoft.com/directx/directstorage-is-coming-to-pc}} by Microsoft, which allows streaming of compressed multimedia files while decoding them with GPU shaders. The workflow for the proposal is illustrated in Fig.~\ref{fig:workflow}.

\begin{figure}[!htbp]\centering
\includegraphics[width=\linewidth]{obrazky-figures/decoding/packetsWorkflow.pdf}
  \centering
   \caption{The input light field images are encoded with the proposed compression scheme with one I-frame.
   The rendering method identifies the necessary views according to the virtual camera position. 
   Usually, only the closest views are necessary. 
   These views are decoded and only their packets with the I-frame are transferred to GPU.
   The packets are decoded on GPU and used to interpolate a novel view.}
 \label{fig:workflow}
\end{figure}

\section{Decompression Evaluation}
\label{sec:videoDecodingExper}
Measurements were performed on a machine equipped with GeForce RTX\textsuperscript{\texttrademark} 2070 GPU and Intel\textsuperscript{\textregistered} Core\textsuperscript{\texttrademark} i5-8500 CPU 3.00 GHz CPU, running Arch Linux.
The relative comparison should be the same on other hardware (HW) and should not depend on the used configuration.

\subsection{Dataset}
A custom dense (500 4K views) 1D light field dataset was rendered and used in the experiments; see Fig.~\ref{fig:dataset}.
The new dataset was created to reflect the current 4K video standard and also to cover extremely dense data.
Previously used high-resolution light field datasets are usually sparse~\cite{vaish}.
The 1D dataset simulates a horizontal 3D motion.
It can be suitable for views through virtual windows, expecting the camera to move at almost constant height from the floor.
Another use case is on holographic 3D displays, such as by Looking Glass Factory, that support only horizontal view change, expecting the user to be in a sitting position.

\begin{figure}[!htbp]
    \def \subSize {0.2\textwidth}
    \centering
    %\captionsetup[subfigure]{width=\subSize}
    \subfloat[\textit{Bunny} contains one simple object with static background.]{\includegraphics[width=\subSize]{obrazky-figures/decoding/datasetBunny.jpg} }
    \subfloat[\textit{Cars} contain reflective objects with simple background.]{\includegraphics[width=\subSize]{obrazky-figures/decoding/datasetCars.jpg} }
    \subfloat[\textit{Sand} contains a landscape with small number of features.]{\includegraphics[width=\subSize]{obrazky-figures/decoding/datasetSand.jpg} }
    %\captionsetup[subfigure]{width=0.5\linewidth}
    \subfloat[\textit{Class} is a complex indoors scene.]{\includegraphics[width=\subSize]{obrazky-figures/decoding/datasetClass.jpg} }
    \subfloat[\textit{Pavilion} is a complex outdoors scene.]{\includegraphics[width=\subSize]{obrazky-figures/decoding/datasetPavilion.jpg} }  
    \caption{Center views of the five scenes in the used dataset are shown in this figure.
    The scenes cover the usual settings of light field data.}
    \label{fig:dataset}
\end{figure}

The dataset is used in three versions: full, every fifth, and every tenth view.
The greater distance between the views and the smaller view count might affect the compression efficiency and also the light field rendering methods and their sensitivity to compression artifacts.
The scenes were taken from Blender demo projects\footnote{\href{https://www.blender.org/download/demo-files/}{blender.org/download/demo-files}} where the \textit{Pavilion} scene is modelled by an external artist\footnote{\href{https://www.emirage.org}{Hamza Cheggour - emirage.org}} and the \textit{Bunny} 3D model is a classic Stanford Bunny\footnote{\href{https://graphics.stanford.edu/~mdfisher/Data/Meshes/bunny.obj}{graphics.stanford.edu/\(\sim \)mdfisher/Data/Meshes/bunny.obj}}. 

\subsection{Streaming Capabilities}
A simple experiment was conducted to compare the image decoding capabilities of modern GPUs. 
Fig.~\ref{fig:streaming} shows how H.265 (implemented using NVIDIA NVDEC via FFmpeg\footnote{\href{https://trac.ffmpeg.org/wiki/HWAccelIntro}{trac.ffmpeg.org/wiki/HWAccelIntro}}) outperforms JPEG2000 (implemented using nvJPEG2000 library\footnote{\href{https://developer.nvidia.com/nvjpeg}{developer.nvidia.com/nvjpeg}}) when decoding a sequence of 4K light field views on GPU.
H.265 has a physical HW acceleration unit on modern GPUs and also reaches a better compression ratio for light field data due to intra-frame redundancy, as shown in Fig.~\ref{fig:streamingSize}.
GPU-accelerated texture compression formats are inferior to the more advanced compression methods and are not suitable for the task.
Video formats seem to be the best choice for streaming light field data on a GPU.

\begin{figure}[!htbp]
\centering
\pgfplotstableread{charts/decoding/streaming.tex}\streaming
\begin{tikzpicture}
\begin{axis}[
  width=\imageSize,
  ylabel={$Decoding\ speed\ [fps]$},
  xlabel={$PSNR$},
  legend style={at={(0.45,0.45)},anchor=east},
  legend entries={H.265, JPEG2000}
]
\addplot[color=mainBlue] table[x=psnrVideo, y=fpsVideo]{\streaming};
\addplot[color=mainRed] table[x=psnrJP, y=fpsJP]{\streaming};
\end{axis}
\end{tikzpicture}
\caption{\label{fig:streaming}
GPU-accelerated H.265 and JPEG2000 format decoding is compared in this chart.
50 frames of \textit{Pavilion} dataset were encoded with different quality settings and decoded on GPU $50{\times}$ in a loop.
}
\end{figure}

\begin{figure}[!htbp]
\centering
\pgfplotstableread{charts/decoding/streaming.tex}\streamingSize
\begin{tikzpicture}
\begin{axis}[
  width=\imageSize,
  ylabel={$Compression\ ratio$},
  xlabel={$PSNR$},
  ymode=log,
  yminorticks=false,
  legend style={at={(0.45,0.2)},anchor=east},
  legend entries={H.265, JPEG2000, ASTC}
]
\addplot[color=mainBlue] table[x=psnrVideo, y=compressionVideo]{\streamingSize};
\addplot[color=mainRed] table[x=psnrJP, y=compressionJP]{\streamingSize};
\addplot[only marks, mark=*, color=mainOrange] coordinates {(44.760845
,3.31797015455128)};
\end{axis}
\end{tikzpicture}
\caption{\label{fig:streamingSize}
The figure shows a compression ratio comparison for H.265 and JPEG2000 formats on \textit{Pavilion} dataset.
ASTC (Adaptive scalable texture compression) GPU accelerated texture compression format is added as a reference to show, how such formats relate to the other methods.
}
\end{figure}

\subsection{I-Frame Positioning}
It is expected that the user would be more interested in views near the center of the light field than at the borders.
The scene is usually captured so that the area of interest is in front of the central cameras, which are often used as reference views for further synthesis~\cite{trottnow2019potential, xue2016sampling, vaish}.
The central view was also previously proposed in other works due to the symmetric properties of the grid~\cite{khoury2019new, mehajabin2019efficient}.
Artifacts in light field, for example, in a window of a virtual scene, would not be easily visible from the extreme viewing position at the edges, and even widely used image-based methods~\cite{van2008} do not work well in such cases.
The quality of the decoded views should be the best around the middle because they are the most similar to the reference I-frame.
An experiment was conducted to compare the positioning of the I-frame in the proposed scheme. 
The \textit{Bunny} dataset was encoded with the proposed scheme, which has the reference frame in the middle of the sequence and at the beginning.
The result of the light field rendering quality using such decoded frames is shown in Fig.~\ref{fig:positionComparison}. 
As an alternative, saliency detection methods~\cite{fu2022light} might also be used to identify the area where the best quality is necessary.

\begin{figure}[!htbp]
\centering
\pgfplotstableread{charts/decoding/position.tex}\positionComparison
\begin{tikzpicture}
\begin{axis}[
width=\imageSize,
colormap={example}{
color(0)=(white)
color(25)=(mainOrange)
color(50)=(white)
},
colorbar horizontal,
colorbar style={ticks=none, no markers, opacity=0.8, height=0.1cm,
at={(0,0)},anchor=north west},
  ylabel={$SSIM$},
  xlabel={$Frame$},
  yminorticks=false,
  legend style={at={(0.6,0.16)},anchor=east},
  legend entries={middle, first, area of interest},
]

\addplot[color=mainBlue] table[x=frame, y=middle]{\positionComparison};
\addplot[color=mainRed] table[x=frame, y=first]{\positionComparison};
\addlegendimage{fill=mainOrange, no markers, draw=mainOrange, ultra thick}
\end{axis}

\end{tikzpicture}
\caption{\label{fig:positionComparison}
The positioning of the I-frame and its effect on the visual quality of the light field per-pixel rendering method is shown in the figure.
The I-frame is placed at the beginning and in the middle of the sequence.
The experiment was measured on \textit{Bunny} dataset.
}
\end{figure}

\subsection{Decoding Time}
The speedup of the proposed scheme could be even more significant when decoding four frames in the 2D light field, since only one I-frame plus four P-frames would be uploaded.
The I-frame would be placed in the middle of the grid, similarly to the proposed scheme.
In addition, any subsequent decodings would not require the decoding of the first I-frame, which would be kept as a reference.
The proposed scheme is more efficient than encoding all I-frames in the stream.
This was proved in an experiment, where the I-frame was uploaded and decoded only once; see Fig.~\ref{fig:keyVSproposed}.
The decoding time $t$ of a frame encoded at position $pos$ encoded with constant a $GOP$ size in the classic scheme can be roughly estimated by Eq.~\ref{eq:classicTime} assuming a simplified scenario with constant decoding time $t^I$ of I-frames and $t^P$ of P-frames and no B-frames. The function $\mathrm{mod}()$ performs the standard modulo operation.

\begin{equation}
    t = t^I + \textrm{mod}(pos-1, GOP) \cdot t^P
    \label{eq:classicTime}
\end{equation}

\begin{figure}[!htbp]
\centering
\pgfplotstableread{charts/decoding/keyVSproposed.tex}\keyVSproposed
\begin{tikzpicture}
\begin{axis}[
  width=\imageSize,
  ylabel={$Decoding\ speed\ [fps]$},
  xlabel={$QP$},
  legend style={at={(0.45,0.8)},anchor=east},
  legend entries={all-I-NV, proposed-NV, all-I-libx, proposed-libx}
]
\addplot[color=mainBlue] table[x=qp, y=keyGPU]{\keyVSproposed};
\addplot[color=mainBlue, dashed] table[x=qp, y=propGPU]{\keyVSproposed};
\addplot[color=mainRed] table[x=qp, y=keyCPU]{\keyVSproposed};
\addplot[color=mainRed, dashed] table[x=qp, y=propCPU]{\keyVSproposed};
\end{axis}
\end{tikzpicture}
\caption{\label{fig:keyVSproposed}
The speed of H.265 GPU-accelerated decoding was compared for the all-I-frame and proposed encoding scheme.
Two H.265 implementations were used for the encoding: libx265 and hevc\_nvenc. The proposed scheme outperforms the all-I in both cases. 
The measurement was conducted over the standardized \textit{quantization parameter} $QP$.}
\end{figure}

Fig.~\ref{fig:schemeTimes} shows the times necessary to decode one frame in the compared schemes.
Note that the times for comparison of the schemes were measured including the loading time of the data from the file, uploading time of the packet to GPU and actual decoding time.
Preloading in RAM would be used as an optimization in real-time applications, and the actual speed would achieve similar results as in Fig.~\ref{fig:keyVSproposed}, depending on the HW used. 
Having the encoded packets already in the GPU memory would also be possible due to the reduced size of the data, and the proposed scheme would still speed up the decoding.

\begin{figure}[!htbp]
\centering
\pgfplotstableread{charts/decoding/schemeTimes.tex}\schemeTimes
\begin{tikzpicture}
\begin{axis}[
  width=\imageSize,
  ylabel={$Decoding\ time\ [ms]$},
  xlabel={$Frame$},
  yminorticks=false,
  xmax=50,
  legend style={at={(0.4,0.8)},anchor=east},
  legend entries={proposed, one-GOP, GOP30, all-I},
]

\addplot[color=mainRed] table[x=frame, y=proposed]{\schemeTimes};
\addplot[color=mainBlue] table[x=frame, y=classic]{\schemeTimes};
\addplot[color=mainOrange] table[x=frame, y=classic30]{\schemeTimes};
\addplot[color=black] table[x=frame, y=allkey]{\schemeTimes};
\end{axis}

\end{tikzpicture}
\caption{\label{fig:schemeTimes}
The compared schemes and decoding times for each frame were measured on \textit{Bunny} dataset with CRF (Constant Rate Factor) set to 35. 
The proposed scheme has a longer decoding of the first frame due to the initial I-frame decoding. 
The I-frame is used directly when queried, so the decoding time in the middle of the sequence is almost zero. 
The jump in GOP30 marks the beginning of the new GOP. 
}
\end{figure}

\subsection{Encoding Schemes Comparison}
The packets are stored in one file, beginning with the I-frame.
A second file contains a list of pointers in the packet file, compressed by the Deflate algorithm.
The existing encoders and frameworks do not support the proposed scheme directly.
The encoding was implemented in pairs, where each frame was encoded in a two-frame video with the same first I-frame and the packets were then stored separately.
YouTube video platform recommends setting the GOP size of uploaded videos to half of the framerate\footnote{\href{https://support.google.com/youtube/answer/1722171}{support.google.com/youtube/answer/1722171}}.
The stream can be viewed as a 60 fps video, and the classic scheme was encoded with GOP size set to 30 frames.
All videos were encoded using the libx265 encoder and decoded with GPU-accelerated NVDEC\footnote{\href{https://developer.nvidia.com/video-codec-sdk}{developer.nvidia.com/video-codec-sdk}}. 
The encoder was used mainly with the default parameters. 
The main settings consist of: Main 4:4:4 profile, Level-5.1 (Main tier), Coding QT 64/8. 

The original frames are encoded using the schemes over CRF values from 0 to 49 jumping by 7 in an H.265 stream and decoded.
The decoded frames are compared with the original ones to discover how the schemes affect the compression quality. 
The decoded and original frames are also used to produce new synthetic views utilizing three light field rendering methods:

\begin{itemize}
    \item \textbf{LF per-pixel} is a simplified per-pixel light field focusing method~\cite{chlubna2021}. 
    This method is the fastest, but with the lowest quality, because it is initially designed for a 2D light field.
    \item \textbf{LF blend} is a mix of all input images that are shifted so that a certain part of the scene is sharp and the rest is defocused as proposed in the original light field rendering method~\cite{levoy1996light}. 
    \item \textbf{Deep} is a Pytorch implementation\footnote{\href{https://github.com/avinashpaliwal/Super-SloMo}{github.com/avinashpaliwal/Super-SloMo}} of deep-learning method Super SloMo~\cite{jiang2017super} for frame interpolation used in SAVFI framework~\cite{choi2020}. 
    This method offers the best quality but is the slowest one.
\end{itemize}

The results are compared to properly test the compression schemes in light field rendering use-case; see Fig.~\ref{fig:comparisonScheme}.

\begin{figure*}[!htbp]\centering
\includegraphics[width=\linewidth]{obrazky-figures/decoding/packetsCompare.pdf}
  \centering
   \caption{The scheme describes the evaluation.
   The original frames are encoded and decoded using different schemes. 
   The decoded views are compared directly with the original frames. 
   The decoded and original views are also used in light field rendering.}
 \label{fig:comparisonScheme}
\end{figure*}

The compression ratio of the proposed scheme is expected to be worse than that of the classic one due to the way video encoders are designed.
The main question is: \textit{How much would the visual quality decrease with the proposed scheme and would the packet size still be small enough to outperform the other schemes in terms of GPU decoding speed?}
The objective of the measurement is to find which scheme has the best decoding time over visual quality ratio.
The results presented in Fig.~\ref{fig:comparisonComp} show that the proposed scheme can be used in light field rendering methods to achieve real-time performance, faster and at the same quality levels as the alternate schemes.
Note that the measurement performed decoding of the frames one by one over the whole dataset.
When streaming continuously, the decoding times might be even shorter, similar to Fig.~\ref{fig:streaming}.

\begin{figure*}[!htbp]
\def \chartmetric{psnr}
\def \chartx{fps}
\def \chartlabels{2,3,5,7,10,15,20,30,40,60,80}

\pgfplotsset{width=0.5\linewidth}
\subfloat[50 frames from the dataset were compressed.]{
\pgfplotstableread{charts/decoding/50.tex}\chartData
\begin{tikzpicture}
\begin{axis}[
  ylabel={\textcolor{white}{$Decoding\ speed\ [fps]$}},
  ymode=log,
  ytick/.expand once=\chartlabels,
  yticklabels/.expand once=\chartlabels,
  legend style={at={(0.45,0.5)},anchor=east}
 ]

\addplot[forget plot, color=mainRed] table[x=proposed\chartmetric, y=proposed\chartx]{\chartData};
\addplot[forget plot, color=mainBlue] table[x=classic\chartmetric, y=classic\chartx]{\chartData};
\addplot[forget plot, color=mainOrange] table[x=classicGOP\chartmetric, y=classicGOP\chartx]{\chartData};
\addplot[forget plot, color=black] table[x=allkey\chartmetric, y=allkey\chartx]{\chartData};
\addplot[forget plot, color=mainBlue, dashed] table[x=classicblend\chartmetric, y=classic\chartx ]{\chartData};
\addplot[forget plot, color=mainOrange, dashed] table[x=classicGOPblend\chartmetric, y=classicGOP\chartx]{\chartData};
\addplot[forget plot, color=black, dashed] table[x=allkeyblend\chartmetric, y=allkey\chartx]{\chartData};
\addplot[forget plot, color=mainRed, dashed] table[x=proposedblend\chartmetric, y=proposed\chartx]{\chartData};

\addlegendimage{gray, mark=none}
\addlegendentry{decoded};
\addlegendimage{gray,dashed, mark=none}
\addlegendentry{LF blend};
\end{axis}
\end{tikzpicture}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{tikzpicture}
\begin{axis}[
  legend style={at={(0.9,0.35)},anchor=east},
  ymode=log,
  ytick/.expand once=\chartlabels,
  yticklabels/.expand once=\chartlabels
 ]

\addplot[color=mainRed] table[x=proposeddeep\chartmetric, y=proposed\chartx]{\chartData};
\addplot[color=mainBlue] table[x=classicdeep\chartmetric, y=classic\chartx]{\chartData};
\addplot[color=mainOrange] table[x=classicGOPdeep\chartmetric, y=classicGOP\chartx]{\chartData};
\addplot[color=black] table[x=allkeydeep\chartmetric, y=allkey\chartx]{\chartData};
\addplot[forget plot, color=mainBlue, dashed] table[x=classicinter\chartmetric, y=classic\chartx ]{\chartData};
\addplot[forget plot, color=mainOrange, dashed] table[x=classicGOPinter\chartmetric, y=classicGOP\chartx]{\chartData};
\addplot[forget plot, color=black, dashed] table[x=allkeyinter\chartmetric, y=allkey\chartx]{\chartData};
\addplot[forget plot, color=mainRed, dashed] table[x=proposedinter\chartmetric, y=proposed\chartx]{\chartData};

\addlegendentry{proposed};
\addlegendentry{one-GOP};  
\addlegendentry{GOP30};
\addlegendentry{all-I};
\addlegendimage{gray, mark=none}
\addlegendentry{deep};
\addlegendimage{gray,dashed, mark=none}
\addlegendentry{LF per-pixel};
\end{axis}
\end{tikzpicture}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subfloat[100 frames from the dataset were compressed.]{
\pgfplotstableread{charts/decoding/100.tex}\chartData
\begin{tikzpicture}
\begin{axis}[
  ylabel={$Decoding\ speed\ [fps]$},
  %ymode=log,
  %ytick/.expand once=\chartlabels,
  %yticklabels/.expand once=\chartlabels,
 ]

\addplot[forget plot, color=mainRed] table[x=proposed\chartmetric, y=proposed\chartx]{\chartData};
\addplot[forget plot, color=mainBlue] table[x=classic\chartmetric, y=classic\chartx]{\chartData};
\addplot[forget plot, color=mainOrange] table[x=classicGOP\chartmetric, y=classicGOP\chartx]{\chartData};
\addplot[forget plot, color=black] table[x=allkey\chartmetric, y=allkey\chartx]{\chartData};
\addplot[forget plot, color=mainBlue, dashed] table[x=classicblend\chartmetric, y=classic\chartx ]{\chartData};
\addplot[forget plot, color=mainOrange, dashed] table[x=classicGOPblend\chartmetric, y=classicGOP\chartx]{\chartData};
\addplot[forget plot, color=black, dashed] table[x=allkeyblend\chartmetric, y=allkey\chartx]{\chartData};
\addplot[forget plot, color=mainRed, dashed] table[x=proposedblend\chartmetric, y=proposed\chartx]{\chartData};
\end{axis}
\end{tikzpicture}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{tikzpicture}
\begin{axis}[
  ymode=log,
  ytick/.expand once=\chartlabels,
  yticklabels/.expand once=\chartlabels
 ]

\addplot[color=mainRed] table[x=proposeddeep\chartmetric, y=proposed\chartx]{\chartData};
\addplot[color=mainBlue] table[x=classicdeep\chartmetric, y=classic\chartx]{\chartData};
\addplot[color=mainOrange] table[x=classicGOPdeep\chartmetric, y=classicGOP\chartx]{\chartData};
\addplot[color=black] table[x=allkeydeep\chartmetric, y=allkey\chartx]{\chartData};
\addplot[forget plot, color=mainBlue, dashed] table[x=classicinter\chartmetric, y=classic\chartx ]{\chartData};
\addplot[forget plot, color=mainOrange, dashed] table[x=classicGOPinter\chartmetric, y=classicGOP\chartx]{\chartData};
\addplot[forget plot, color=black, dashed] table[x=allkeyinter\chartmetric, y=allkey\chartx]{\chartData};
\addplot[forget plot, color=mainRed, dashed] table[x=proposedinter\chartmetric, y=proposed\chartx]{\chartData};

\end{axis}
\end{tikzpicture}
}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subfloat[500 frames from the dataset were compressed.]{
\pgfplotstableread{charts/decoding/500.tex}\chartData
\begin{tikzpicture}
\begin{axis}[
  ylabel={\textcolor{white}{$Decoding\ speed\ [fps]$}},
  ymode=log,
  ytick/.expand once=\chartlabels,
  yticklabels/.expand once=\chartlabels,
 ]

\addplot[forget plot, color=mainRed] table[x=proposed\chartmetric, y=proposed\chartx]{\chartData};
\addplot[forget plot, color=mainBlue] table[x=classic\chartmetric, y=classic\chartx]{\chartData};
\addplot[forget plot, color=mainOrange] table[x=classicGOP\chartmetric, y=classicGOP\chartx]{\chartData};
\addplot[forget plot, color=black] table[x=allkey\chartmetric, y=allkey\chartx]{\chartData};
\addplot[forget plot, color=mainBlue, dashed] table[x=classicblend\chartmetric, y=classic\chartx ]{\chartData};
\addplot[forget plot, color=mainOrange, dashed] table[x=classicGOPblend\chartmetric, y=classicGOP\chartx]{\chartData};
\addplot[forget plot, color=black, dashed] table[x=allkeyblend\chartmetric, y=allkey\chartx]{\chartData};
\addplot[forget plot, color=mainRed, dashed] table[x=proposedblend\chartmetric, y=proposed\chartx]{\chartData};
\end{axis}
\end{tikzpicture}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{tikzpicture}
\begin{axis}[
  ymode=log,
  ytick/.expand once=\chartlabels,
  yticklabels/.expand once=\chartlabels
 ]

\addplot[color=mainRed] table[x=proposeddeep\chartmetric, y=proposed\chartx]{\chartData};
\addplot[color=mainBlue] table[x=classicdeep\chartmetric, y=classic\chartx]{\chartData};
\addplot[color=mainOrange] table[x=classicGOPdeep\chartmetric, y=classicGOP\chartx]{\chartData};
\addplot[color=black] table[x=allkeydeep\chartmetric, y=allkey\chartx]{\chartData};
\addplot[forget plot, color=mainBlue, dashed] table[x=classicinter\chartmetric, y=classic\chartx ]{\chartData};
\addplot[forget plot, color=mainOrange, dashed] table[x=classicGOPinter\chartmetric, y=classicGOP\chartx]{\chartData};
\addplot[forget plot, color=black, dashed] table[x=allkeyinter\chartmetric, y=allkey\chartx]{\chartData};
\addplot[forget plot, color=mainRed, dashed] table[x=proposedinter\chartmetric, y=proposed\chartx]{\chartData};

\end{axis}
\end{tikzpicture}
}

\begin{tikzpicture}
\node[text centered,text width = \textwidth]{\small{$PSNR\ [dB]\ \ \ $}};
\end{tikzpicture}

\caption{The video encoding schemes are compared in the charts.
The measurements were performed on the c) full dense dataset, b) the sparser one where each fifth frame was taken and a) even sparser where each tenth frame is taken.
Four comparisons were conducted: decoded frames as they were (left column solid lines), all frames blended together~\cite{levoy1996light} (left dashed), deep interpolation~\cite{jiang2017super, choi2020} (right solid) and per-pixel interpolation~\cite{chlubna2021} (right dashed) which is most error prone due to the compression artifacts.\label{fig:comparisonComp}}
\end{figure*}

The proposed scheme is not the best choice for storing light fields on the disk because of the poorer quality/size compression ratio.
However, the proposed scheme significantly outperforms other schemes in decoding speed and is the best in all measurements with the quality/speed ratio.
To further investigate the results, two additional metrics were measured.

The first $error$ metric in Eq.~\ref{eq:metric} to compare the average results takes into account errors in relation to the best values of the three measured quantities: $visual$ quality, decoding $time$ and the resultant stream $size$.
Visual quality error is usually small.
Therefore, it is amplified to be more significant.
The proposed scheme has the lowest error metric in $61\%$ of the measurements.

\begin{equation}
    error = 100err_{visual} + err_{time} + err_{size}
    \label{eq:metric}
\end{equation}

The second metric is based on a threshold of $5\%$ in visual quality error from the best result. 
This threshold was selected as a tolerable error that cannot be clearly distinguished by a human evaluator.
The proposed scheme reached this limit in $58\%$ of measurements.

Tab.~\ref{tab:overall} shows the average decoding time, size, and quality metrics of the results. 
Given the significant speedup of decoding, the loss of visual quality is not critical. 
The disparity-related artifacts in the proposed scheme are mitigated when used in the light field rendering, and the scheme seems to perform better with larger and dense data due to higher inter-frame similarity.

\begin{table*}[ht]\centering
\resizebox{\textwidth}{!}{
\begin{tabular}{*{16}c}
   &     & time  & \multicolumn{12}{c}{PSNR [dB], SSIM, VMAF} & size \\
 \cmidrule(l){4-15} 
  data & scheme & [ms]  & \multicolumn{3}{c}{decoded} & \multicolumn{3}{c}{deep} & \multicolumn{3}{c}{LF per-pixel} & \multicolumn{3}{c}{LF blend} & [kB]\\
 \cmidrule(l){1-1}  \cmidrule(l){2-2} \cmidrule(l){3-3} \cmidrule(l){4-6} \cmidrule(l){7-9} \cmidrule(l){10-12} \cmidrule(l){13-15} \cmidrule(l){16-16} 
 \input{tables/decodingOverall}
 \\ \\
\end{tabular}
}
\caption{\label{tab:overall} The proposed scheme is the fastest, has the best quality/speed ratio, and the quality loss is not significant.
The best absolute results are typed in \textbf{bold}.
The lowest results of the weighted sum of errors, prioritizing visual quality, are \underline{underlined}.
The loss in visual quality compared to the best result greater than $5\%$ is typed in \textit{italic}.}
\end{table*}
 
It even outperforms the other schemes in one case. 
Fig.~\ref{fig:visualComparison} compares the visual quality at the same decoding speed. 
The amount of artifacts in the other methods is much more visible than in the proposal. 

\begin{figure}[!htbp]\centering
\includegraphics[width=0.99\linewidth]{obrazky-figures/decoding/visualComparison.pdf}
  \centering
   \caption{The figure shows the difference in visual quality between the proposal and other schemes while reaching the same decoding speed at \textit{Pavilion} scene.
The quality was adjusted so that the decoding times would match.
The other methods had to use much more aggressive compression levels to reach the same decoding time as the proposal. 
The 10th frame was decoded from 100-frames dataset.
The position of the frame was chosen to ensure a similar decoding time but also reasonable position within the light field (not at the very beginning or end).
Frames further from I-frames in the other methods would not be able to compete with the proposal.
Decoding multiple frames would put JPEG2000 at a disadvantage.
}
 \label{fig:visualComparison}
\end{figure}

The following triplets of values are in the format of PSNR [dB], SSIM, VMAF.
The best visual results were obtained with the \textit{Bunny} scene, where the proposed methods yielded best results as 48.38, 0.9356, 96.59 and the worst as 32.23, 0.6655, 30.18.
The worst results were measured with \textit{Class} scene with the best 45.52, 0.9872, 96.16 and the worst 25.60, 0.7213, 17.54.
This shows that the complexity of the scene plays a role in the efficiency of the encoding, which is common for most existing compression methods.
The \textit{Bunny} scene contains only a simple object with constant background, whereas the \textit{Class} scene contains a lot of small details in all parts of the scene.

The proposed scheme is not capable of reaching the same quality as the other schemes at the almost lossless level.
The best results with \textit{Bunny} scene are for one-GOP 53.04, 0.9966, 97.046, GOP30 53.17, 0.9971, 97.048 and all-I 55.40, 0.9981, 97.29.
This reveals that the proposed method is not a good choice for archiving best-quality data.
However, these best-quality settings are not suitable for real-time rendering, where a certain amount of hardly detectable information loss is to be expected.

The average standard deviations over the quality range of the visual quality metrics are for the proposed method 2.41, 0.12, 5.35, one-GOP 2.37, 0.11, 3.94, GOP30 2.36, 0.10, 3.74 and all-I 2.67, 0.12, 5.23.
These values show that the methods are all similarly suitable for all datasets without a significant divergence in any case.

The standard deviations of decoding times in milliseconds at a default $CRF=28$ with \textit{Bunny} scene are for the proposed method 0.779, one-GOP 648, GOP30 86.7 and all-I 14.4. 
This shows that the proposed method is the most stable and all packets have the same decoding time, excluding the first reference, which is almost $16{\times}$ slower but is decoded only once during the initialization stage. 
The best and worst quality decoding speeds in frames per second are for the proposed method 28, 85, one-GOP 0.36, 13, GOP30 3, 14, and all-I 1, 31. 
The whole quality range of the proposed method can be considered suitable for real-time streaming. 

The best and worst quality data sizes [MB] are for the proposed method 764, 2.4, one-GOP 486, 0.33, GOP30 506, 0.5, and all-I 1 461, 4.7 on \textit{Sand} scene, which showed the highest variance of the sizes.
The proposed method is not capable of reaching the same data reduction as one-GOP due to the impossibility of reusing information from decoded P-frames.
Compared to all-I, which is second fastest after the proposed method, the possible data reduction is significantly higher.

The video encoding with one I-frame also helps to improve the quality of the light field rendering methods. 
The generated focus map in Fig.~\ref{fig:focusDecode}, from the per-pixel focusing method~\cite{chlubna2021}, shows how the inter-frame encoding aids the method. 
The focus map contains optimal displacement values between the images so that every pixel is focused at the correct depth. 
The map computation is based on color similarity between the displaced pixels of the views. 
The all-I-frame encoded frames are compressed separately and the compression artifacts that are different in each frame create bigger disparities even in the areas of the light field views that should be the same (the sky in the figure).
When the P-frames are encoded using one common I-frame, their similarity in the static areas is untouched, and the compression even serves as a denoising for such areas. 
The disparity detection algorithms based on the color difference between pixels or their blocks then identify such areas even more accurately than when using the uncompressed original frames. 
This hypothesis is also confirmed in Tab.~\ref{tab:overall}, where the all-I scheme does not always win in terms of visual quality during the interpolation process.

\begin{figure}[!htbp]
    \centering
    \subfloat[An image was used to compute a focus map.]{\includegraphics[width=0.24\linewidth]{obrazky-figures/decoding/focusColor.jpg} }
    \subfloat[The focus map was estimated from the original.]{\includegraphics[width=0.24\linewidth]{obrazky-figures/decoding/focusOriginal.jpg} }
    \subfloat[The focus map was estimated from the decompressed I-frame.]{\includegraphics[width=0.24\linewidth]{obrazky-figures/decoding/focusKey.jpg} }
    \subfloat[The focus map was estimated from the decompressed P-frame]{\includegraphics[width=0.24\linewidth]{obrazky-figures/decoding/focusClassic.jpg} }
    
    \caption{A part of a wall and the sky from \textit{Pavilion} dataset is shown in a). 
    The focus map produced by a light field rendering method of the original input in b) resembles a disparity or depth map, but light noise is visible in the static sky.
    All-I-frames in c) decoded and used in the method produce even more noise. 
    The P-frames similarly used in d) produce the best focus map due to the inter-frame compression and reduction of color differences in static areas.}
    \label{fig:focusDecode}
\end{figure}

The discovered results are expected to be valid even for higher-dimensional light fields, which will be further investigated and proved in future work. 
The random access feature would be even more convenient in 2D light field grids where at least 4 views are necessary to decode. 
One row-wide offset between the two pairs would also require additional decoding.
A test performed on 2D light field \textit{Bunny} scene from Stanford dataset~\cite{vaish} indicates similar improvements as in the 1D measurement; see Tab.~\ref{tab:2d}.

\begin{table}[ht]\centering
\begin{tabular}{ c c c c }
 scheme & grid & speedup of proposed \\
 \toprule
 \multirow{2}{*}{all-I}  & 17x17 & 2.066  \\ 
                         & 8x8   & 1.94 \\ 
\midrule 
\multirow{2}{*}{GOP30}   & 17x17 & 8.64 \\ 
                         & 8x8   & 3.19  \\ 
                             \\
\end{tabular}
\caption{\label{tab:2d} The table shows how many times is the proposed scheme faster when decoding 4 views in the center of the grid (coordinates $(3,3), (3,4), (4,3), (4,4)$ for $8{\times}8$ and $(8,8), (8,9), (9,8), (9,9)$ for $17{\times}17$ grid) of a 2D light field scene, compared to all-I and GOP30 schemes encoded with the same default quality settings ($CRF=28$).}
\end{table}

The proposed scheme theoretically ensures the best memory usage since only the initial I-frame is kept in the memory. 
The memory usage on GPU was measured experimentally, decoding a frame at the end of the 100 frames light field. 
Tab.~\ref{tab:memory} shows that the proposal needs the least amount of memory. 
The all-I-frame decoding needs a bit more, probably due to the decoder allocating more memory in advance for further sequential decoding.

\begin{table}[ht]\centering
\begin{tabular}{ l | c c c c }
 scheme & proposed & one-GOP & GOP30 & all-I \\
 used memory [MB] & 597 & 1019 & 1049 & 800
\end{tabular}
\caption{\label{tab:memory} The table shows a comparison of the schemes at the same quality level $CRF=35$ with 4K data.}
\end{table}

\subsection{Comparison to Existing Methods}
An optimal light field data transfer to GPU was previously published, but it is not directly comparable with this proposal~\cite{lee2022}.
The idea of video encoding and the principles of the transfer scheme might be fused in a novel method in the future.
The reordering of the views~\cite{Han2018} before encoding could also bring about a certain improvement that is inferior to the proposed scheme.
Previously published optimal subsampling of light field views~\cite{chen2022view} might be combined with the proposed video encoding.
Compression and reconstruction networks~\cite{liu2022, ravishankar2023data} or light field-friendly displacement intra prediction~\cite{Li2016} use H.265 video encoding for light field data, so this proposal could be used to improve their performance.
The proposed method is versatile and can be used as a part of other previously proposed methods.

Although deep central light field image warping~\cite{hedayati2021} outperforms video encoders, the approach would be problematic with wide spacing of the views since occlusions in the scene might cause warping artifacts.
This proposal supports newer and potentially faster decoding standards even with wide datasets.
A deep generative model using a randomly initialized network can be used without learning for light field compression~\cite{jiang2022}.
The proposed method does not depend on the training process.

GPU support is a crucial requirement as modern light field rendering methods and 3D simulations are computed mostly on GPU and the data upload to GPU is the performance bottleneck.
This method focuses on achieving the best quality/decoding time ratio.
Other approaches generally aim to achieve the best compression ratio~\cite{magnor2000data} even with partial random access support~\cite{mehajabin2019efficient}.

An apparent candidate for this task - Multiview Video Coding (MVC) extension of H.265 that offers an optimal coding of animated multi-view sequences~\cite{mesloul2018} is not fully supported by commonly used codecs and no plans for GPU HW support are expected.
According to the current state of this extension, it cannot be expected to be present in more efficient GPU-accelerated formats such as (AV1) AOMedia Video 1.
The proposed scheme does not depend on a specific format extension and uses the basic compression available in any video encoding format.
The scheme can be used in any future format. Similar proposals~\cite{avramelos2020random, wang2016} to this method exploit the MVC extension, but it is not clear how efficient this approach would be in real-life scenarios on GPU.
Compared to this method, the measured datasets were sparse, in a low resolution, the actual GPU decoding speed was not measured, and the decoded views were not used in any light field rendering method before comparison.
Other specific approaches such as HEVC-HR~\cite{monteiro2020} are most likely to be outperformed with the novel video formats.
Although other custom light field compression methods exist~\cite{brites2020lenslet}, the proposed scheme is universal, scalable, independent on underlying video encoding method, and guaranteed to be efficient even on future GPUs.

The encoding speed or compression ratios are not relevant for this study.
The real GPU-based decoding speed at the given quality plays the major role. Fig.~\ref{fig:streamingSize} already showed how GPU-accelerated image compressions are outperformed by the video approach.

The untrained neural compression paper~\cite{jiang2022} reported decoding of $81$ views of $432{\times}624$ resolution simultaneously, measured on GeForce RTX\textsuperscript{\texttrademark} 2080 Ti, in $262$ ms.
The same scene \textit{Danger de morts} of the pleno dataset~\cite{rerabek2016new} was measured using video compression with H.265 with one reference frame on GPU GeForce RTX\textsuperscript{\texttrademark} 2070.
$81$ views can be decoded in $273$ ms while achieving the same quality.
The performance is the same.
However, video decoders are improving and new coding standards are implemented, so the decoding speed might be even better soon.
The natively HW-accelerated decoding proposed in the research in this thesis can gain the same quality as a complex training-based method and might be a more robust solution.
The authors in the mentioned paper measured \textit{decoding time} and it is not clear if uploading of the data to GPU was also measured.
The upload time was included in the measurement of the method proposed in the experiments, as it plays an important role in real-time streaming.
They also mention that the network needs to be transmitted to the decoder for each light field, which brings an additional unmeasured delay.
It is not clear how fast the decoding would be when only certain views are necessary. 

Similar issues were found with neural video compression with diverse contexts~\cite{li2023neural} where decoding of $96$ images takes $765$ ms for $1920{\times}1080$ video on 2080 Ti GPU.
Video compression is capable of reaching $634$ ms for the same task according to the measurements conducted.
The same experiment with $96$ images was conducted with the published code of learning for video compression paper~\cite{yang2020Learning}. 
The same issues prevail, and the decoding took minutes.
The decoding of a frame took approximately $10$ s.

A dictionary learning approach paper~\cite{miandji2019unified} reports reaching $40$ fps for a $2048{\times}1088$ light field data, measured on NVIDIA\textsuperscript{\textregistered} Titan Xp.
The proposed decoding scheme reaches up to $80$ fps with almost $2{\times}$ larger images.
The mentioned method also supports random access.
The disparity-aware learning-based model~\cite{singh2021} outperforms other methods but does not provide a significant gain in terms of decoding speed.
This method reaches decoding times for low-resolution Lytro data in tens of seconds, which is not comparable to fast GPU decoders.
The methods mentioned above were designed for Lytro camera data, while the proposal aims at data with a wider spacing between the views.
Compared to Lytro data, where the maximal disparity reaches tens of pixels, the proposed method can process disparities of hundreds of pixels.

Dual discriminator generative adversarial network compression~\cite{bakir2020} reaches certain quality gain, but the decoding takes tens of seconds due to additional synthesis of the missing frames from VVC encoded reference frames.
Bi-level view compensation compression~\cite{Hou2019} is almost $1000{\times}$ slower than H.265 at decoding. Authors of adversarial network-based view synthesis paper~\cite{Jia2019} also admit that the method cannot be used in real-time.
Various JPEG extensions to 3D and 4D for multiview data were also beaten by video compression methods when used with light fields~\cite{barina2022, barina2019}.