3D Video Compression

This document presents a novel diffusion-based video compression technique. We leverage the inherent expressiveness, photorealism and 3D awareness of denoising diffusion generative AI models as a powerful general-purpose prior that only requires small complement of low-quality guidance data (or "hints") to produce video with high spatio-temporal perceptual quality and optional novel view synthesis. Our use of small, finetuned, low-rank adaptations (e.g., LoRA) efficiently compresses a batch of frames and, when combined with the general purpose base model, allows compression that can significantly exceed the performance of state-of-the-art methods such as H.264 and H.265 in terms of perceptual and technical quality to compression size ratios.

Depth map compression is important for efficient network transmission of 3D visual data in texture-plus-depth format, where the observer can synthesize an image of a freely chosen viewpoint via depth-image-based rendering (DIBR) using received neighboring texture and depth maps as anchors. Unlike texture maps, depth maps exhibit unique characteristics like smooth interior surfaces and sharp edges that can be exploited for coding gain. In this paper, we propose a multi-resolution approach to depth map compression using previously proposed graph-based transform (GBT). The key idea is to treat smooth surfaces and sharp edges of large code blocks separately and encode them in different resolutions: encode edges in original high resolution (HR) to preserve sharpness, and encode smooth surfaces in low-pass-filtered and down-sampled low resolution (LR) to save coding bits. Because GBT does not filter across edges, it produces small or zero high-frequency components when coding smooth-surface depth maps and leads to a compact representation in the transform domain. By encoding down-sampled surface regions in LR GBT, we achieve representation compactness for a large block without the high computation complexity associated with an adaptive large-block GBT. At the decoder, encoded LR surfaces are up-sampled and interpolated while preserving encoded HR edges. Experimental results show that our proposed multi-resolution approach using GBT reduced bitrate by 68% compared to native H.264 intra with DCT encoding original HR depth maps, and by 55% compared to single-resolution GBT encoding small blocks.

There has been increasing demand for multiview video transmission over band limited channel over past years and various techniques have been proposed to fulfil this need. In this paper, a High Efficiency Video Codec (HEVC) based spatial resolution scaling type of mixed resolution coding model, MRHEVC-MVC, for frame interleaved multiview videos is presented. However, enabling the HEVC to encode video with different frame resolutions is a challenge due to the coding tree partitioning used by the codec. This has been overcome by superimposing the low resolution replica of each full resolution frame on their respective decoded picture buffer and setting the remaining space of the frame buffer to zero. The codec's reference frames structure is designed to efficiently encode frame interleaved multiview videos using a HEVC based mixed resolution codec. The proposed MRHEVC-MVC codec has been tested against the standard multiview extension of high efficiency video codec (MV-HEVC) for BBalloon^, BNewspaper1^, BUndo_Dancer^, BKendo^and BBPoznan_Street^standard multiview video sequences. Results show that the proposed codec gives significantly higher coding performance to that of the MV-HEVC codec at low bitrate both subjectively and objectively.

In this paper, we propose a method to extract depth from motion, texture and intensity. We first analyze the depth map to extract a set of depth cues. Then, based on these depth cues, we process the colored reference video, using texture, motion, luminance and chrominance content, to extract the depth map. The processing of each channel in the Y C R C B −color space is conducted separately. We tested this approach on different video sequences with different monocular properties. The results of our simulations show that the extracted depth maps generate a 3D video with quality close to the video rendered using the ground truth depth map. We report objective results using 3VQM and subjective analysis via comparison of rendered images. Furthermore, we analyze the savings in bitrate as a consequence of eliminating the need for two video codecs, one for the reference color video and one for the depth map. In this case, only the depth cues are sent as a side information to the color video.

Light field imaging is characterized by capturing brightness, color, and directional information of light rays in a scene. This leads to image representations with huge amount of data that require efficient coding schemes. In this paper, lenslet images are rendered into sub-aperture images. These images are organized as a pseudo-sequence input for the HEVC video codec. To better exploit redundancy among the neighboring sub-aperture images and consequently decrease the distances between a sub-aperture image and its references used for prediction, sub-aperture images are divided into four smaller groups that are scanned in a serpentine order. The most central sub-aperture image, which has the highest similarity to all the other images, is used as the initial reference image for each of the four regions. Furthermore, a structure is defined that selects spatially adjacent sub-aperture images as prediction references with the highest similarity to the current image. In this way, encoding...

Light field data records the amount of light at multiple points in space, captured e.g. by an array of cameras or by a light-field camera that uses microlenses. Since the storage and transmission requirements for such data are tremendous, compression techniques for light fields are gaining momentum in recent years. Although plenty of efficient compression formats do exist for still and moving images, only a little research on the impact of these methods on light field imagery is performed. In this paper, we evaluate the impact of state-of-the-art image and video compression methods on quality of images rendered from light field data. The methods include recent video compression standards, especially AV1 and XVC finalised in 2018. To fully exploit the potential of common image compression methods on four-dimensional light field imagery, we have extended these methods into three and four dimensions. In this paper, we show that the four-dimensional light field data can be compressed much more than independent still images while maintaining the same visual quality of a perceived picture. We gradually compare the compression performance of all image and video compression methods, and eventually answer the question, "What is the best compression method for light field data?".

The aim of the Leeds Beckett Repository is to provide open access to our research, as required by funder policies and permitted by publishers and copyright law. The Leeds Beckett repository holds a wide range of publications, each of which has been checked for copyright and the relevant embargo period has been applied by the Research Services team. We operate on a standard take-down policy. If you are the author or publisher of an output and you would like it removed from the repository, please contact us and we will investigate on a case-by-case basis.

3D video coding includes the use of multiple color views and depth maps associated to each view. An adequate coding of depth maps should be adapted to the characteristics of depth maps: smooth regions and sharp edges. In this paper a segmentation-based technique is proposed for improving the depth map compression while preserving the main discontinuities that exploits the color-depth similarity of 3D video. An initial coarse depth map segmentation is used to locate the main discontinuities in depth. The resulting partition is improved by fusing a color partition. We assume that the color image is first encoded and available when the associated depth map is encoded, therefore the color partition can be segmented in the decoder without introducing any extra cost. A new segmentation criterion inspired by super-pixels techniques is proposed to obtain the color partition. Initial experimental results show similar compression efficiency to hevc with a big potential for further improvements.

Public transportation is one of the things that is often used by everyone. One type of public transportation that is often used in Indonesia is the Bus Rapid Transit (BRT) or commonly known as the busway. The city of Padang, West Sumatra has a BRT to serve the community called the Trans Padang Bus. However, in this bus service there are still several shortcomings that make the bus service ineffective. The gate system which is still manual on this bus makes service slow and inefficient. By using a flap barrier gate that has an image processing system for suit passengers, the service on this bus will be more effective. Utilizing the Raspberry Pi and You Only Look Once (YOLO), this system can be made as efficient as possible for the place and use. Passengers who will enter will be immediately detected by the PIR sensor and the system will start with a camera that will take pictures of passenger clothes. The image will be processed using YOLO and the results of the process in the form o...

The state-of-the-art High Efficiency Video Coding (HEVC) standard adopts a hierarchical coding structure to improve its coding efficiency. This allows for the Quantization Parameter Cascading (QPC) scheme that assigns Quantization Parameters (Qps) to different hierarchical layers in order to further improve the Rate-Distortion (RD) performance. However, only static QPC schemes have been suggested in HEVC test model (HM), which are unable to fully explore the potentials of QPC. In this paper, we propose an adaptive QPC scheme for HEVC hierarchical structure to code natural video sequences characterized by diversified textures, motions and encoder configurations. We formulate the adaptive QPC scheme as a non-linear programming problem and solve it in a scientifically sound way with a manageable low computational overhead. The proposed model addresses a generic Qp assignment problem of video coding. Therefore, it also applies to Group-Of-Picture (GOP)- level, frame-level and Coding Uni...

Public transportation is one of the things that is often used by everyone. One type of public transportation that is often used in Indonesia is the Bus Rapid Transit (BRT) or commonly known as the busway. The city of Padang, West Sumatra has a BRT to serve the community called the Trans Padang Bus. However, in this bus service there are still several shortcomings that make the bus service ineffective. The gate system which is still manual on this bus makes service slow and inefficient. By using a flap barrier gate that has an image processing system for suit passengers, the service on this bus will be more effective. Utilizing the Raspberry Pi and You Only Look Once (YOLO), this system can be made as efficient as possible for the place and use. Passengers who will enter will be immediately detected by the PIR sensor and the system will start with a camera that will take pictures of passenger clothes. The image will be processed using YOLO and the results of the process in the form o...

Light field imaging is characterized by capturing brightness, color, and directional information of light rays in a scene. This leads to image representations with huge amount of data that require efficient coding schemes. In this paper, lenslet images are rendered into sub-aperture images. These images are organized as a pseudo-sequence input for the HEVC video codec. To better exploit redundancy among the neighboring sub-aperture images and consequently decrease the distances between a sub-aperture image and its references used for prediction, sub-aperture images are divided into four smaller groups that are scanned in a serpentine order. The most central sub-aperture image, which has the highest similarity to all the other images, is used as the initial reference image for each of the four regions. Furthermore, a structure is defined that selects spatially adjacent sub-aperture images as prediction references with the highest similarity to the current image. In this way, encoding...

The acquisition of the spatial and angular information of a scene using light field (LF) technologies supplement a wide range of post-processing applications, such as scene reconstruction, refocusing, virtual view synthesis, and so forth. The additional angular information possessed by LF data increases the size of the overall data captured while offering the same spatial resolution. The main contributor to the size of captured data (i.e., angular information) contains a high correlation that is exploited by state-of-the-art video encoders by treating the LF as a pseudo video sequence (PVS). The interpretation of LF as a single PVS restricts the encoding scheme to only utilize a single-dimensional angular correlation present in the LF data. In this paper, we present an LF compression framework that efficiently exploits the spatial and angular correlation using a multiview extension of high-efficiency video coding (MV-HEVC). The input LF views are converted into multiple PVSs and are organized hierarchically. The rate-allocation scheme takes into account the assigned organization of frames and distributes quality/bits among them accordingly. Subsequently, the reference picture selection scheme prioritizes the reference frames based on the assigned quality. The proposed compression scheme is evaluated by following the common test conditions set by JPEG Pleno. The proposed scheme performs 0.75 dB better compared to state-of-the-art compression schemes and 2.5 dB better compared to the x265-based JPEG Pleno anchor scheme. Moreover, an optimized motionsearch scheme is proposed in the framework that reduces the computational complexity (in terms of the sum of absolute difference [SAD] computations) of motion estimation by up to 87% with a negligible loss in visual quality (approximately 0.05 dB). INDEX TERMS Compression, light field, MV-HEVC, plenoptic.

In a traditional multi-view image generation algorithm, partial image information might be lost at the pixel mapping step during the 3D image acquisition. A lower hardware cost and shorter operation time can be realized if an effective algorithm is employed. Therefore, a fast multi-view image rendering method based on reverse search is proposed in this paper, it uses pixel mapping information to derive a rendering image. Compared with the conventional depth-image-based rendering (DIBR) which using multiple images, the method only generates one image with reverse mapping in the image rendering step. First, the parallax range of the original image is calculated. Then, a warping image is generated through reverse mapping; this image has contained information for eight different viewpoints. Finally, the image modification can be made. Experimental results show that the proposed method effectively reduces rendering time and memory size by 58.82% and 83.78%, respectively. In addition, the mean structural similarity between the two images created by DIBR and our method is 1, which means that our method has the same effect as DIBR.

View synthesis prediction (VSP) is a coding mode that predicts video blocks from synthesised frames. It is particularly useful in a multi-camera setup with large inter-camera distances. Adding a VSP-based SKIP mode to a standard Multiview Video Coding (MVC) framework improves the rate-distortion (RD) performance but increases the time complexity of the encoder. This letter proposes an early mode decision technique for VSP SKIP-enhanced MVC. Our method uses the correlation between the RD costs of the VSP SKIP mode in neighbouring views and Bayesian decision theory to reduce the number of candidate coding modes for a given macroblock. Simulation results showed that our technique can save up to 36.20% of the encoding time without any significant loss in RD performance.

The standard HEVC codec and its extension for coding multiview videos, known as MV-HEVC, have proven to deliver improved visual quality compared to its predecessor, H.264/MPEG-4 AVC’s multiview extension, H.264-MVC, for the same frame resolution with up to 50% bitrate savings. MV-HEVC’s framework is similar to that of H.264-MVC, which uses a multi-layer coding approach. Hence, MV-HEVC would require all frames from other reference layers decoded prior to decoding a new layer. Thus, the multi-layer coding architecture would be a bottleneck when it comes to quicker frame streaming across different views. In this paper, an HEVC-based Frame Interleaved Stereo/Multiview Video Codec (HEVC-FISMVC) that uses a single layer encoding approach to encode stereo and multiview video sequences is presented. The frames of stereo or multiview video sequences are interleaved in such a way that encoding the resulting monoscopic video stream would maximize the exploitation of temporal, inter-view, and c...

Light Field (LF) offers unique advantages such as post-capture refocusing and depth estimation, but low-light conditions severely limit these capabilities. To restore low-light LFs we should harness the geometric cues present in different LF views, which is not possible using single-frame low-light enhancement techniques. We, therefore, propose a deep neural network architecture for Low-Light Light Field (L3F) restoration, which we refer to as L3Fnet. The proposed L3Fnet not only performs the necessary visual enhancement of each LF view but also preserves the epipolar geometry across views. We achieve this by adopting a two-stage architecture for L3Fnet. Stage-I looks at all the LF views to encode the LF geometry. This encoded information is then used in Stage-II to reconstruct each LF view. To facilitate learning-based techniques for low-light LF imaging, we collected a comprehensive LF dataset of various scenes. For each scene, we captured four LFs, one with near-optimal exposure and ISO settings and the others at different levels of low-light conditions varying from low to extreme low-light settings. The effectiveness of the proposed L3Fnet is supported by both visual and numerical comparisons on this dataset. To further analyze the performance of low-light reconstruction methods, we also propose an L3F-wild dataset that contains LF captured late at night with almost zero lux values. No ground truth is available in this dataset. To perform well on the L3F-wild dataset, any method must adapt to the light level of the captured scene. To do this we propose a novel pre-processing block that makes L3Fnet robust to various degrees of low-light conditions. Lastly, we show that L3Fnet can also be used for low-light enhancement of singleframe images, despite it being engineered for LF data. We do so by converting the single-frame DSLR image into a form suitable to L3Fnet, which we call as pseudo-LF.

Multi-view video coding technique has been widely applied to 3D stereoscopic display. This paper proposes a rate control algorithm for multi-view video codec which is a tradeoff method between the data compression ratio and picture quality of 3D video. This study utilizes the characteristics of visual perception for 3D video viewers to determine the interesting regions in all view videos. Interesting regions with different levels, which depend on the existence of macroblocks with high motion, objects with individual shape, or obvious texture, are determined in the proposed algorithm by considering the corresponding Just Noticeable Distortion function, luminance histogram bin variation and edge strength, respectively. Then the proposed algorithm assigns adequate quantization parameters to control the bit-rate for different levels of interesting regions such that the video quality of the interesting regions is preserved and the bit-rate in the non-interesting regions is reallocated to 3D video storage and transmission for stereoscopic display. The experiment results show that the proposed method successfully achieves high quality preservation and low bit-rate for 3D effects. Index Terms-Visual perception system (VPS), rate control, multi-view video coding.

Error resilient encoding in video communication is becoming increasingly important due to data transmission over unreliable channels. In this paper, we propose a new power-aware error resilient coding scheme based on network error probability and user expectation in video communication using mobile handheld devices. By considering both image content and network conditions, we can achieve a fast recoverable and energy-efficient error resilient coding scheme. More importantly, our approach allows system designers to evaluate various operating points in terms of error resilient level and energy consumption over a wide range of system operating conditions. We have implemented our scheme on an H.263 video codec algorithm, compared it with the previous AIR, GOP and PGOP coding schemes, and measured energy consumption and video quality on the IPAQ and Zaurus PDAs. Our experimental results show that our approach reduces energy consumption by 34%, 24% and 17% compared with AIR, GOP and PGOP schemes respectively, while incurring only a small fluctuation in the compressed frame size. In addition, our experimental results prove that our approach allows faster error recovery than the previous AIR, GOP and PGOP approaches. We believe our error resilient coding scheme is therefore eminently applicable for video communication on energy-constrained wireless mobile handheld devices. Recent advances in technology enable mobile handheld devices to be equipped with wireless interfaces and there will be growing demand for high quality mobile multimedia communications. However, wireless multimedia communications in the mobile handheld environment face several challenges, including high error rate, bandwidth variations, and limitations of the mobile devices such as battery lifetime constraints and the low CPU computation capability. To overcome the bandwidth limitation, there are several existing video coding techniques developed, for example, H.263 and MPEG, to compress raw video sequences to encoded bitstreams. These video encoding techniques exploit spatial and temporal correlation to achieve a high compression ratio, but they are usually unaware about the device status and the network conditions during the coding process. Therefore, multimedia data encoding requires a large amount of information, leading to high computation and communication energy consumption, and transmitting multimedia data over wireless networks can be very unreliable due to packet loss. This problem should be solved with the reasonable compression efficiency with high error resiliency considering resource constraints, which is a crucial factor for the real-time multimedia communication over error prone and lossy network using mobile handheld devices. Video communication over unreliable networking environments is challenging since data loss and corruption from several reasons such as traffic congestion and physical channel failure affect video quality severely unless a guaranteed quality of service (QoS) is available between the source and the destination. Also, the spatio-temporal prediction encoding and variable length coding (VLC) of the source coding cause error propagation. Since spatiotemporal prediction requires the previous frame to reconstruct the current frame, a single error can lead to consecutive errors in the following frames. Likewise, because of VLC, a single bit error causes the decoder to lose a synchronization point that makes the following bits useless. Therefore, a variety of techniques have been proposed to enhance the resilience of the video data encoding against the packet errors [1, 2]. The most well recognized method is to insert intra-coding 1 to mitigate the effect of error propagation in a predictive video compression algorithm [3, 4, 5, 6]. However, inserting intra-coding influences compression efficiency adversely since it tends to increase total length of the encoded bitstream. From this observation, the prior studies on error resilient video encoding mainly tried to find out a solution that maximizes bitstream robustness with low bit rate. Meanwhile, as mobile devices increasingly have video communication functionality, low power encoding has become important. Several encoding schemes have been proposed to reduce energy consumption for multimedia applications [9, 10, 11, 12]. However, these studies dealt with either error resilience or low power issues independently. We believe it is critical for both issues to be addressed together, especially in the context of energy-constrained mobile devices. In this paper, we propose a new energy-efficient, error-resilient encoding scheme. Especially, we note the dual role of intra-coding: not only does intra-coding improve error resilience, but it also contributes to reducing encoding energy consumption since it does not require motion estimation (which is the most power consuming operation in a predictive video compression algorithm). Indeed, the system designer will therefore need to evaluate the trade-off between the error resiliency level, compression efficiency, and power consumption. In this paper, we focus our attention on this tradeoff. Specifically, we (i) propose PBPAIR (Probability Based Power Aware Intra Refresh), a new energy-efficient and error-resilient encoding scheme, based on the network condition and the image content, (ii) implement our scheme as well as other existing error resilient encoding schemes on an H.263 codec, (iii) extensively compare with other error resilient encoding schemes in the context of error resiliency vs. encoding efficiency (both bit rate and energy consumption), and (iv) evaluate the trade-offs between the error resiliency level, compression efficiency, and energy consumption on top of real implementation platform. Our performance results indicate that PBPAIR saves as much as 17% to 34% energy compared with other error resilient techniques allowing faster error recovery than the previous approaches. This paper is organized as follows. In the next section, we briefly review previous work on error resilient

Holoscopic imaging is a prospective acquisition and display solution for providing true 3D content and fatigue-free 3D visualization. However, efficient coding schemes for this particular type of content are needed to enable proper storage and delivery of the large amount of data involved in these systems. Therefore, this paper proposes an alternative HEVC-based coding scheme for efficient representation of holoscopic images. In this scheme, some directional intra prediction modes of the HEVC are replaced by a more efficient prediction framework based on locally linear embedding techniques. Experimental results show the advantage of the proposed prediction for 3D holoscopic image coding, compared to the reference HEVC standard as well as previously presented approaches in this field.

This paper proposes a two-stage high order intra block prediction method for light field image coding. This method exploits the spatial redundancy in lenslet light field images by predicting each image block, through a geometric transformation applied to a region of the causal encoded area. Light field images comprise an array of micro-images that are related by complex geometric transformations that cannot be efficiently compensated by state-of-the-art image coding techniques, which are usually based on low order translational prediction models. The two-stage nature of the proposed method allows to choose the order of the prediction model most suitable for each block, ranging from pure translations to projective or bilinear transformations, optimized according to an appropriate rate-distortion criterion. The proposed higher order intra block prediction approach was integrated into an HEVC codec and evaluated for both unfocused and focused light field camera models, using different resolutions and microlens arrays. Experimental results show consistent bitrate savings, which can go up to 12.62%, when compared to a lower order intra block prediction solution and 49.82% when compared to HEVC still picture coding.

Light field imaging is a promising new technology that allows the user not only to change the focus and perspective after taking a picture, as well as to generate 3D content, among other applications. However, light field images are characterized by large amounts of data and there is a lack of coding tools to efficiently encode this type of content. Therefore, this paper proposes the addition of two new prediction tools to the HEVC framework, to improve its coding efficiency. The first tool is based on the local linear embedding-based prediction and the second one is based on the self-similarity compensated prediction. Experimental results show improvements over JPEG and HEVC in terms of average bitrate savings of-71.44% and-31.87%, and average PSNR gains of 4.73dB and 0.89dB, respectively.

During traditional road surveys, inspectors capture images of pavement surface using cameras that produce 2D images, which can then be automatically processed to get a road surface condition assessment. This paper proposes a novel crack detection system that uses a light field imaging sensor, notably the Lytro Illum camera, instead of a conventional 2D camera, to capture road surface light field images. Light field images capture the light rays originating from different directions, thus providing a richer representation of the observed scene. The proposed system explores the disparity information, which can be computed from the light field, to obtain information about cracks observable in the pavement images. A simple processing system is considered, to show the potential use of this type of sensors for crack detection. Encouraging experimental crack detection results are presented based on a set of road pavement light field images captured over different pavement surface textures. A performance comparison with a state-of-the-art 2D image crack detection system is included, confirming the potential of using this type of sensors.

DOI to the publisher's website. • The final author version and the galley proof are versions of the publication after peer review. • The final published version features the final layout of the paper including the volume, issue and page numbers. Link to publication General rights Copyright and moral rights for the publications made accessible in the public portal are retained by the authors and/or other copyright owners and it is a condition of accessing publications that users recognise and abide by the legal requirements associated with these rights. • Users may download and print one copy of any publication from the public portal for the purpose of private study or research. • You may not further distribute the material or use it for any profit-making activity or commercial gain • You may freely distribute the URL identifying the publication in the public portal. If the publication is distributed under the terms of Article 25fa of the Dutch Copyright Act, indicated by the "Taverne" license above, please follow below link for the End User Agreement:

Conventional depth video compression uses video codecs designed for color images. Given the performance of current encoding standards, this solution seems efficient. However, such an approach suffers from many issues stemming from discrepancies between depth and light perception. To exploit the inherent limitations of human depth perception, we propose a novel depth compression method that employs a disparity perception model. In contrast to previous methods, we account for disparity masking, and model a distinct relation between depth perception and contrast in luminance. Our solution is a natural extension to the H.264 codec and can easily be integrated into existing decoders. It significantly improves both the compression efficiency without sacrificing visual quality of depth of rendered content, and the output of depth-reconstruction algorithms or depth cameras.

This paper presents the optimal mode selection of disparity-compensated (DC) wavelet lifting in a multi-view image coding framework. The optimal mode is a combination of macroblock (MB) coding and block partitioning mode of DC wavelet lifting. There are seven coding and seven MB partitioning modes, totally 49 modes to be selected. Rate distortion optimization is used as a tool to determine the optimal mode of DC wavelet lifting. The residue error from DC wavelet lifting is compressed by SPIHT codec. The experimental results of the proposed scheme show the PSNR improvement up to 1.3 dB at the same compression ratio over 513 DC wavelet lifting and even more comparing to Haar DC wavelet lifting without applying optimal mode selection.

Today, stereoscopic 3D (S3D) cinema is already mainstream, and almost all new display devices for the home support S3D content. S3D distribution infrastructure to the home is partly already established in form of 3D Blu-ray discs, video on demand services, or television channels. However, the necessity to wear glasses is often considered as an obstacle, which hinders broader acceptance of this technology in the home. Multiview autostereoscopic displays enable a glasses free perception of S3D content for several observers simultaneously, and support head motion parallax in a limited range. In order to support multiview autostereoscopic displays in an already established S3D distribution infrastructure, a synthesis of new views from S3D video is needed. In this paper, a view synthesis method based on Image-domain-Warping (IDW) is presented which synthesizes new views directly from S3D video and functions completely automatically. IDW relies on an automatic and robust estimation of sparse disparities and image saliency information, and enforces target disparities in synthesized images using an image warping framework. Two configurations of the view synthesizer in the scope of a transmission and view synthesis framework are analyzed and evaluated. A transmission and view synthesis system that uses IDW was recently submitted to MPEG's call for proposals on 3D Video Technology, where it was ranked among the four best performing proposals.

In this paper we present two methods of depth extraction for 2D-to-3D video conversion. One for a scene captured with a static camera and other for the case of a moving camera, both using information from the motion present on the scene. In the first method, temporal difference, morphological operations and a region filling technique are used to segment the moving objects and define the foreground. Moreover, analysis of how the detected regions vary over nearby frames is applied to ensure the temporal consistency. For the regions corresponding to background, depth values are obtained merging information from linear perspective and texture characteristics. The second method is applied when a dynamic background is detected. It requires an input sequence encoded with H.264, so the motion information extracted from the compressed bitstream is used to assign depth values for the entire scene.

Motion estimation (ME) is one of the element keys in video compression that takes up to 60% in processing time. Block matching algorithm (BMA) is a technique that is used to reduce the computational complexity of ME algorithm due to its efficiency and good performance. Strategy of searching is one of the factors in developing motion estimation algorithm that has the potential to provide good performance. This study aims to implement several selected BMAs for achieving the least number of computations and to give better Peak Signal to Noise Ratio (PSNR) values using different video sequences. The proposed algorithms are modified based on the search strategy adapted from the standard algorithms approach. The results have proved that both modification algorithms (MDS and MARPS) have the potential in reducing the number of computations and achieved good PSNR values in all motion types as compared to DS and ARPS respectively. This work could be improved by using metaheuristic algorithms ...

Several real-time visual monitoring applications such as surveillance, mental state monitoring, driver drowsiness and patient care, require equipping high-quality cameras with wireless sensors to form visual sensors and this creates an enormous amount of data that has to be managed and transmitted at the sensor node. Moreover, as the sensor nodes are battery-operated, power utilization is one of the key concerns that must be considered. One solution to this issue is to reduce the amount of data that has to be transmitted using specific compression techniques. The conventional compression standards are based on complex encoders (which require high processing power) and simple decoders and thus are not pertinent for battery-operated applications, i.e., VSN (primitive hardware). In contrast, compressive sensing (CS) a distributive source coding mechanism, has transformed the standard coding mechanism and is based on the idea of a simple encoder (i.e., transmitting fewer data-low proces...

JPEG Pleno is an upcoming standard from the ISO/IEC JTC 1/SC 29/WG 1 (JPEG) Committee. It aims to provide a standard framework for coding new imaging modalities derived from representations inspired by the plenoptic function. The image modalities addressed by the current standardization activities are light field, holography, and point clouds, where these image modalities describe different sampled representations of the plenoptic function. The applications that may benefit from these emerging image modalities range from supporting varying capture platforms, interactive content viewing, cultural environments exploration and medical imaging to more immersive browsing with novel special effects and more realistic images. These use cases come with a set of requirements addressed by the JPEG Pleno standard. Main requirements envision high compression efficiency, random access, scalability, error-resilience, low complexity, and metadata support. This paper presents a synopsis of the stat...

This paper describes a light field compression scheme based on a novel homography-based low rank approximation method called HLRA. The HLRA method jointly searches for the set of homographies best aligning the light field views and for the low rank approximation matrices. The light field views are aligned using either one global homography or multiple homographies depending on how much the disparity across views varies from one depth plane to the other. The light field low-rank representation is then compressed using HEVC. The best pair of rank and QP parameters of the coding scheme, for a given target bit-rate, is predicted with a model defined as a function of light field disparity and texture features. The results are compared with those obtained by directly applying HEVC on the light field views restructured as a pseudo-video sequence. The experiments using different data sets show substantial PSNR-rate gain of our compression algorithm, as well as the accuracy of the proposed parameter prediction model, especially for real light fields. A scalable extension of the coding scheme is finally proposed.

This paper presents a predictive-coding algorithm for the compression of multiple depth-sequences obtained from a multi-camera acquisition setup. The proposed depth-prediction algorithm works by synthesizing a virtual depth-image that matches the depth-image (of the predicted camera). To generate this virtual depth-image, we use an image-rendering algorithm known as 3D image-warping. This newly proposed prediction technique is employed in a 3D coding system in order to compress multiview depth-sequences. For this purpose, we introduce an extended H.264 encoder that employs two prediction techniques: a blockbased motion prediction and the previously mentioned 3D image-warping prediction. This extended H.264 encoder adaptively selects the most efficient prediction scheme for each image-block using a rate-distortion criterion. We present experimental results for several multiview depthsequences, which show a quality improvement of about 2.5 dB as compared to H.264 inter-coded depth-images.

We investigate the coding of multiview images obtained from a set of multiple cameras. To exploit the interview correlation, two viewprediction tools have been implemented and used in parallel: a blockbased motion compensation scheme and a Depth Image Based Rendering technique (DIBR). Whereas DIBR relies on an accurate depth image, the block-based motion-compensation scheme can be performed without any geometry information. Our encoder adaptively selects the most appropriate prediction scheme using a rate-distortion criterion for an optimal prediction-mode selection. The attractiveness of the algorithm is that the compression algorithm is robust against inaccurately estimated depth images and requires only one single reference camera for fast random-access to different views. We present experimental results for several multiview sequences, that result in a quality improvement of up to 1.4 dB as compared to H.264 compression.

DOI to the publisher's website. • The final author version and the galley proof are versions of the publication after peer review. • The final published version features the final layout of the paper including the volume, issue and page numbers. Link to publication General rights Copyright and moral rights for the publications made accessible in the public portal are retained by the authors and/or other copyright owners and it is a condition of accessing publications that users recognise and abide by the legal requirements associated with these rights. • Users may download and print one copy of any publication from the public portal for the purpose of private study or research. • You may not further distribute the material or use it for any profit-making activity or commercial gain • You may freely distribute the URL identifying the publication in the public portal. If the publication is distributed under the terms of Article 25fa of the Dutch Copyright Act, indicated by the "Taverne" license above, please follow below link for the End User Agreement:

Light field imaging is characterized by capturing brightness, color, and directional information of light rays in a scene. This leads to image representations with huge amount of data that require efficient coding schemes. In this paper, lenslet images are rendered into sub-aperture images. These images are organized as a pseudo-sequence input for the HEVC video codec. To better exploit redundancy among the neighboring sub-aperture images and consequently decrease the distances between a sub-aperture image and its references used for prediction, sub-aperture images are divided into four smaller groups that are scanned in a serpentine order. The most central sub-aperture image, which has the highest similarity to all the other images, is used as the initial reference image for each of the four regions. Furthermore, a structure is defined that selects spatially adjacent sub-aperture images as prediction references with the highest similarity to the current image. In this way, encoding...

In recent years, light field imaging has attracted the attention of the academic and industrial communities thanks to its enhanced rendering capabilities that allow to visualise contents in a more immersive and interactive way. However, those enhanced capabilities come at the cost of a considerable increase in content size when compared to traditional image and video applications. Thus, advanced compression schemes are needed to efficiently reduce the volume of data for storage and delivery of light field content. In this paper, we introduce a novel method for compression of light field images. The proposed solution is based on a graph learning approach to estimate the disparity among the views composing the light field. The graph is then used to reconstruct the entire light field from an arbitrary subset of encoded views. Experimental results show that our method is a promising alternative to current compression algorithms for light field images, with notable gains across all bitrates with respect to the state of the art.

Pattern-based video coding (PVC) has already established its superiority over H.264 in low bit rate areas because of an extra pattern-mode to segment out the arbitrary shape of the moving region in macroblock. To determine the pattern-mode, the PVC however uses three thresholds which sometimes depend on the content of the video sequences. Thus, the best performance of the PVC may not be achieved using the pattern-mode. In this paper, a novel PVC scheme is proposed which does not use any thresholds to determine the pattern-mode and hence more efficient performance is achieved without knowing the content of the video sequences. In addition, efficient pattern similarity metric and new Lagrangian multipliers are also developed. The experimental results confirm that this new scheme significantly improves the quality compared to the existing PVC as well as H.264 video coding standard.

Pattern-based video coding representing moving regions in macroblock has very good potential for improved coding efficiency over existing standard H.264/AVC in the very low bit-rate range. However, the coding efficiency diminishes compared to H.264 with the bit rates because variable block sizes of H.264 for motion estimation and compensation are successfully selected for any bit rates using weighting between bit rate and image quality. The existing pattern-based video coding scheme could not handle this flexibility with bit rates due to its fixed size and lack of verification with other modes of H.264. In this paper we propose two large and one small pre-defined pattern sets and treat them as extra modes of H.264. We also investigate the weighting factor between bit rate and image quality compared to the existing one. The experimental results confirm the superiority of our approach compared to H.264 and other existing pattern-based video coding for not only the very low bit rate range but also wide range of bit rates.

The selection of an optimal regular-shaped pattern set for very low bit-rate video coding, focusing on moving regions has been the objective of much recent research in order to try and improve bit-rate eficieciency. Selecting the optimal pattern set however, is an NP hard problem. This paper presents a Generic Variable Pattern Selection (GYPS) algorithm. which introduces a pattern selection parameter that is able to control the performance in terms of computational complexity as well as bit-rate andpicture quality. While using a sub-optimal variable pattern set, GYPS oblains a coding perJormance comparable to near-optimal algorithms, such as the k-change neighbourhood solution, while being much less computationally intensive, so that it is able to process all types of video sequences in real-time, with minimal pre-processing overheads

Pattern-based video coding (PVC) has already established its superiority over H.264 in low bit rate areas because of an extra pattern-mode to segment out the arbitrary shape of the moving region in macroblock. To determine the pattern-mode, the PVC however uses three thresholds which sometimes depend on the content of the video sequences. Thus, the best performance of the PVC may not be achieved using the pattern-mode. In this paper, a novel PVC scheme is proposed which does not use any thresholds to determine the pattern-mode and hence more efficient performance is achieved without knowing the content of the video sequences. In addition, efficient pattern similarity metric and new Lagrangian multipliers are also developed. The experimental results confirm that this new scheme significantly improves the quality compared to the existing PVC as well as H.264 video coding standard.

Very low bit-rate video coding algorithms using predefined regular-shaped patterns to segment out moving objects at macroblock level have exhibited good potential for improved coding efficiency when embedded in the H.264 standard as an extra mode. Even the best-matched regular-shaped pattern from a predefined codebook cannot approximate the shape of the object well, and there is no guarantee that even a regular-shaped object will have a close match with one of the limited number of predefined patterns. Intuitively, improved coding performance can be achieved if patterns are dynamically extracted from the video content. This letter presents a content-based pattern generation (CPG) algorithm for a set of macroblocks, which is shown optimal when only one pattern is allowed to represent the entire set. Coupling CPG, generating a pattern codebook after clustering the macroblocks into several disjoint sets, with any pattern selection algorithm outperforms the existing regular-shaped pattern-based coding while both embedded in H.264.

Algorithms using content-based patterns to segment moving regions at the macroblock (MB) level have exhibited good potential for improved coding efficiency when embedded into the H.264 standard as an extra mode. The content-based pattern generation (CPG) algorithm provides local optimal result as only one pattern can be optimally generated from a given set of moving regions. But, it failed to provide optimal results for multiple patterns from entire sets. Obviously, a global optimal solution for clustering the set and then generation of multiple patterns enhances the performance farther. But a global optimal solution is not achievable due to the non-polynomial nature of the clustering problem. In this paper, we propose a near-optimal content-based pattern generation (OCPG) algorithm which outperforms the existing approach. Coupling OCPG, generating a set of patterns after clustering the MBs into several disjoint sets, with a direct pattern selection algorithm by allowing all the MBs in multiple pattern modes outperforms the existing pattern-based coding when embedded into the H.264.

In the context of very low hit-rate video coding, pre-defined fixed pattern representations of moving regions in blocked-based motion estimation and compensation has become increasingly attractive over H.264 as the former represents an MB by a smaller size moving region covered by the best available pattern that approximates the shape of the region more closely and hence, requiring no extra motion vector, which is not the case with the latter. But fixed set patterns sometimes fail to code efficiently for all video sequences. In this paper a novel idea of selecting a sub.^et of best-matched patterns through preferential selection technique is developed by presenting two algorithms. Variable Pattern Selection iVPS) and Extended VPS (EVPS)for an initial pattern codebook size of 32 using a new parametric macroblock classification definition and a new similarity metric. The complexity analysis confirmed that EVPS guaranteed to be nearly six times faster than VPS, with the peak performance providing an improvement factor of nine times. The overall performance of EVPS is identical to VPS for certain parameters but on average, 0.2dB and O.SdB better than the contemporary algorithm using fixed set patterns and Advanced Video coding standard (H.264) respectively, for the same number of bits per frame. ACM Classification: 1.4.2 (Computing methodologies-image processing and computer vi.sion-compression (coding)) 1. INTRODUaiON Reducing the transmission bit-rate whiie concomitantly retaining image quality continues to be a major challenge for efficient very low bit-rate video compression standards, such as H.26X

Motion estimation (ME) is one of the element keys in video compression that takes up to 60% in processing time. Block matching algorithm (BMA) is a technique that is used to reduce the computational complexity of ME algorithm due to its efficiency and good performance. Strategy of searching is one of the factors in developing motion estimation algorithm that has the potential to provide good performance. This study aims to implement several selected BMAs for achieving the least number of computations and to give better Peak Signal to Noise Ratio (PSNR) values using different video sequences. The proposed algorithms are modified based on the search strategy adapted from the standard algorithms approach. The results have proved that both modification algorithms (MDS and MARPS) have the potential in reducing the number of computations and achieved good PSNR values in all motion types as compared to DS and ARPS respectively. This work could be improved by using metaheuristic algorithms ...

Motion estimation (ME) is one of the element keys in video compression that takes up to 60% in processing time. Block matching algorithm (BMA) is a technique that is used to reduce the computational complexity of ME algorithm due to its efficiency and good performance. Strategy of searching is one of the factors in developing motion estimation algorithm that has the potential to provide good performance. This study aims to implement several selected BMAs for achieving the least number of computations and to give better Peak Signal to Noise Ratio (PSNR) values using different video sequences. The proposed algorithms are modified based on the search strategy adapted from the standard algorithms approach. The results have proved that both modification algorithms (MDS and MARPS) have the potential in reducing the number of computations and achieved good PSNR values in all motion types as compared to DS and ARPS respectively. This work could be improved by using metaheuristic algorithms ...

Light field data records the amount of light at multiple points in space, captured e.g. by an array of cameras or by a light-field camera that uses microlenses. Since the storage and transmission requirements for such data are tremendous, compression techniques for light fields are gaining momentum in recent years. Although plenty of efficient compression formats do exist for still and moving images, only a little research on the impact of these methods on light field imagery is performed. In this paper, we evaluate the impact of state-of-the-art image and video compression methods on quality of images rendered from light field data. The methods include recent video compression standards, especially AV1 and XVC finalised in 2018. To fully exploit the potential of common image compression methods on four-dimensional light field imagery, we have extended these methods into three and four dimensions. In this paper, we show that the four-dimensional light field data can be compressed mu...

Depth image based rendering techniques for multiview applications have been recently introduced for efficient view generation at arbitrary camera positions. Encoding rate control has thus to consider both texture and depth data. Due to different structures of depth and texture images and their different roles on the rendered views, distributing the available bit budget between them however requires a careful analysis. Information loss due to texture coding affects the value of pixels in synthesized views while errors in depth information lead to shift in objects or unexpected patterns at their boundaries. In this paper, we address the problem of efficient bit allocation between textures and depth data of multiview video sequences. We adopt a rate-distortion framework based on a simplified model of depth and texture images. Our model preserves the main features of depth and texture images. Unlike most recent solutions, our method permits to avoid rendering at encoding time for distortion estimation so that the encoding complexity is not augmented. In addition to this, our model is independent of the underlying in

In this paper, we present a modified interview prediction multiview video coding (MVC) scheme form the perspective of viewer interactivity. This latter requires a high transmission bite-rate, when a viewer requests some view(s). Conventional solutions to such a problem assume that the requested views are already encoded by a MVC scheme. In this paper, we modify the MVC-HBP standard scheme yielding reducing the average transmission bite-rate required to retrieve the requested views (i.e., interactivity). With real data sequences, clear improvements are shown using the proposed MVC scheme compared to competing MVC schemes in terms of rate-distortion (RD) and average transmission bite-rate.