Academia.eduAcademia.edu

Figure 14 – uploaded by Kari Pulli

See full PDFdownloadDownload figure
Figure 14: Extending the reconstruction pipeline to the receiver. We compare the JPEG (quality = 30), Adobe Photoshop deblocking, SA- DCT, and our reconstruction on the display side (PSNR in images). (Images courtesy of Wikipedia user kallerna, and A. Torralba & B. Russell / LabelMe dataset.) The improvement is quite modest, as the JPEG coefficients do not couple much with the rest of the pipeline. However, strong JPEG compression inevitably creates blocking and quantization artifacts, which this approach cannot address. To overcome such issues, we can extend the image pipeline all the way to decompression and solve Eq. 23 on the receiver side, in essence deblocking the JPEG image [Foi et al. 2007].

Figure 14 Extending the reconstruction pipeline to the receiver. We compare the JPEG (quality = 30), Adobe Photoshop deblocking, SA- DCT, and our reconstruction on the display side (PSNR in images). (Images courtesy of Wikipedia user kallerna, and A. Torralba & B. Russell / LabelMe dataset.) The improvement is quite modest, as the JPEG coefficients do not couple much with the rest of the pipeline. However, strong JPEG compression inevitably creates blocking and quantization artifacts, which this approach cannot address. To overcome such issues, we can extend the image pipeline all the way to decompression and solve Eq. 23 on the receiver side, in essence deblocking the JPEG image [Foi et al. 2007].

Related Figures (16)

FlexISP: A Flexible Camera Image Processing Framework Figure 1: Our end-to-end system reconstructs images and jointly accounts for demosaicking, denoising, deconvolution, and missing data reconstruction. Thanks to the separation of image model and formulation based on natural-image priors, we support both conventional and unconventional sensor designs. Examples (our results at lower right): (a) Demosaicking+denoising of a Bayer-sensor image (top: regular pipeline). (b) Demosaicking+denoising of a burst image stack (top: first frame shown). (c) Demosaicking+denoising+HDR from an interlaced exposure image (top: normalized exposure image). (d) Denoising+reconstruction of a color camera array image (top: naive reconstruction).
FlexISP: A Flexible Camera Image Processing Framework Figure 1: Our end-to-end system reconstructs images and jointly accounts for demosaicking, denoising, deconvolution, and missing data reconstruction. Thanks to the separation of image model and formulation based on natural-image priors, we support both conventional and unconventional sensor designs. Examples (our results at lower right): (a) Demosaicking+denoising of a Bayer-sensor image (top: regular pipeline). (b) Demosaicking+denoising of a burst image stack (top: first frame shown). (c) Demosaicking+denoising+HDR from an interlaced exposure image (top: normalized exposure image). (d) Denoising+reconstruction of a color camera array image (top: naive reconstruction).
Figure 2: The traditional camera processing pipeline consists of many cascaded modules, introducing cumulative errors. We propose to replace this pipeline with a unified and flexible camera processing system, leveraging natural-image priors and modern optimization techniques.
Figure 2: The traditional camera processing pipeline consists of many cascaded modules, introducing cumulative errors. We propose to replace this pipeline with a unified and flexible camera processing system, leveraging natural-image priors and modern optimization techniques.
Table 1: Table comparing previous work and ours. ML = machine learning, PD = primal dual, DM = demosaic, DN = denoise, DC = deconvolution, SR = superresolution.
Table 1: Table comparing previous work and ours. ML = machine learning, PD = primal dual, DM = demosaic, DN = denoise, DC = deconvolution, SR = superresolution.
Figure 4: Effects of different initial iterate x° on the convergence for a simple demosaicking example. We compare starting with x =0, averaging of neighbors, and the output of a classical demosaicking method. All cases converge to the same solution; with a good starting point, 5 iterations suffice. More details in the supplemental. Convergence Both ADMM and the primal-dual algorithm are guaranteed to converge to the global optimum for convex F and G, with the primal-dual algorithm having provably optimal convergence. However, we make use of non-convex regularization terms in F’, so theoretical convergence guarantees are not available. That said, we always combine both convex and non-convex terms in F' (a deliberate choice, see the following subsection), and therefore can control the degree of non-convexity simply by adjusting the weights for the individual terms. This weight selection depends on the ill-posedness of the data term G for a given camera design and image formation model A, but in our experiments we have not found a need to vary it based on the image content. In practice, we get convergence for all initial iterates we tried (even for x° = 0, see Fig. 4).
Figure 4: Effects of different initial iterate x° on the convergence for a simple demosaicking example. We compare starting with x =0, averaging of neighbors, and the output of a classical demosaicking method. All cases converge to the same solution; with a good starting point, 5 iterations suffice. More details in the supplemental. Convergence Both ADMM and the primal-dual algorithm are guaranteed to converge to the global optimum for convex F and G, with the primal-dual algorithm having provably optimal convergence. However, we make use of non-convex regularization terms in F’, so theoretical convergence guarantees are not available. That said, we always combine both convex and non-convex terms in F' (a deliberate choice, see the following subsection), and therefore can control the degree of non-convexity simply by adjusting the weights for the individual terms. This weight selection depends on the ill-posedness of the data term G for a given camera design and image formation model A, but in our experiments we have not found a need to vary it based on the image content. In practice, we get convergence for all initial iterates we tried (even for x° = 0, see Fig. 4).
Table 3: Different weighting between internal and external prior and the resulting change in SNR.
Table 3: Different weighting between internal and external prior and the resulting change in SNR.
Figure 5: Illustration of using both the TV and BM3D priors for reconstructing images from the interlaced HDR sensor (Sec. 4.3).
Figure 5: Illustration of using both the TV and BM3D priors for reconstructing images from the interlaced HDR sensor (Sec. 4.3).
Figure 6: The impact of selected natural-image priors on the recon- struction quality of various image models.
Figure 6: The impact of selected natural-image priors on the recon- struction quality of various image models.
Figure 9: Joint demosaicking and denoising results for the Kodak color image dataset. We show average PSNR over all images, color channels, and sigmas. Figure 8: Demosaicking results for the McMaster color image dataset. We show average PSNR over all images and color channels.
Figure 9: Joint demosaicking and denoising results for the Kodak color image dataset. We show average PSNR over all images, color channels, and sigmas. Figure 8: Demosaicking results for the McMaster color image dataset. We show average PSNR over all images and color channels.
Figure 7: Impact of the regularizer weights (relative to the datc term) on the solution. The weights are y; = YQ: Y is shown in the images, while 9; are ¢9 = 0.1 (TV), 6; = 1.0 (BM3D), and ¢2 = 0.1 (cross-channel). High y-values impair the reconstruction process b) enforcing too much sparsity in the output. On the other hand, a toc small y ignores the priors, leading to sharp but aliased results.
Figure 7: Impact of the regularizer weights (relative to the datc term) on the solution. The weights are y; = YQ: Y is shown in the images, while 9; are ¢9 = 0.1 (TV), 6; = 1.0 (BM3D), and ¢2 = 0.1 (cross-channel). High y-values impair the reconstruction process b) enforcing too much sparsity in the output. On the other hand, a toc small y ignores the priors, leading to sharp but aliased results.
Our method outperforms all competing methods by at least 0.68 dB. A detailed list of PSNR values can be found in the supplemental.
Our method outperforms all competing methods by at least 0.68 dB. A detailed list of PSNR values can be found in the supplemental.
Table 4: Reconstruction quality for five non-blind deconvolution scenarios as in Schuler et al. [2013].
Table 4: Reconstruction quality for five non-blind deconvolution scenarios as in Schuler et al. [2013].
Figure 10: We compare our demosaicking method with commonly used demosaicking tools (DCRaw and Adobe Camera Raw) and the best competing published method (LDI-NAT) on two patches of the McMaster dataset (left) and on two real-world examples (right). Note how our method produces results virtually indistinguishable from ground truth, whereas other methods contain visible artifacts (please zoom in to the pdf). More examples can be found in the supplemental.
Figure 10: We compare our demosaicking method with commonly used demosaicking tools (DCRaw and Adobe Camera Raw) and the best competing published method (LDI-NAT) on two patches of the McMaster dataset (left) and on two real-world examples (right). Note how our method produces results virtually indistinguishable from ground truth, whereas other methods contain visible artifacts (please zoom in to the pdf). More examples can be found in the supplemental.
Figure 11: Image reconstruction for interlaced HDR sensor data. (a) Upsampling-based reconstruction [Gu et al. 2010] improves the dynamic range, but fails to recover highly structured details. The Magic Lantern method provides higher quality, but still loses resolution and produces artifacts. Our method generates an artifact-free result and provides the best overall reconstruction quality. (b, c) In addition to extending the dynamic range (bottom row), we account for local scene motion and simultaneously denoise and reduce interlacing artifacts (see insets).
Figure 11: Image reconstruction for interlaced HDR sensor data. (a) Upsampling-based reconstruction [Gu et al. 2010] improves the dynamic range, but fails to recover highly structured details. The Magic Lantern method provides higher quality, but still loses resolution and produces artifacts. Our method generates an artifact-free result and provides the best overall reconstruction quality. (b, c) In addition to extending the dynamic range (bottom row), we account for local scene motion and simultaneously denoise and reduce interlacing artifacts (see insets).
Figure 12: Image reconstruction for color-array camera. (a) Raw input red, green, and blue images overlaid. (b) Naive registration of color channels via optical-flow produces color leaks. (c) Anaglyph-based reconstruction [Joulin and Kang 2013] partially reduces the color leakage. (d) Our method reconstructs the image without such artifacts.
Figure 12: Image reconstruction for color-array camera. (a) Raw input red, green, and blue images overlaid. (b) Naive registration of color channels via optical-flow produces color leaks. (c) Anaglyph-based reconstruction [Joulin and Kang 2013] partially reduces the color leakage. (d) Our method reconstructs the image without such artifacts.
Figure 13: Burst Image Comparison. The first row shows a real-world example taken in low light (8 images at ISO 12800 on a Canon EOS 650D; GT was taken at ISO 100). Each image in the simulated doll dataset (16 images, courtesy Flickr user susan402) has multiplicative white Gaussian noise with 6 = 0.1 and additive white Gaussian noise with o = 0.1. Parameters for each method were optimized for highest PSNR.
Figure 13: Burst Image Comparison. The first row shows a real-world example taken in low light (8 images at ISO 12800 on a Canon EOS 650D; GT was taken at ISO 100). Each image in the simulated doll dataset (16 images, courtesy Flickr user susan402) has multiplicative white Gaussian noise with 6 = 0.1 and additive white Gaussian noise with o = 0.1. Parameters for each method were optimized for highest PSNR.
Table 5: Running times for: desktop CPU (Core i7 2.4 Ghz), desktop GPU (NVIDIA GTX TITAN), and mobile GPU (NVIDIA Tegra K1). References
Table 5: Running times for: desktop CPU (Core i7 2.4 Ghz), desktop GPU (NVIDIA GTX TITAN), and mobile GPU (NVIDIA Tegra K1). References
Related topics:
Information SystemsComputer ScienceArtificial IntelligenceComputer Vision
580 California St., Suite 400
San Francisco, CA, 94104
© 2025 Academia. All rights reserved