Papers by Miguel Bordallo
Proceedings of SPIE - The International Society for Optical Engineering, 2008
The video applications on mobile communication devices have usually been designed for content cre... more The video applications on mobile communication devices have usually been designed for content creation, access, and playback. For instance, many recent mobile devices replicate the functionalities of portable video cameras and video recorders, and digital TV receivers. These are all demanding uses, but nothing new from the consumer point of view. However, many of the current devices have two cameras built in, one for capturing high resolution images, and the other for lower, typically VGA (640x480 pixels) resolution video telephony. We employ video to enable new applications and describe four actual solutions implemented on mobile communication devices. The first one is a real-time motion based user interface that can be used for browsing large images or documents such as maps on small screens. The motion information is extracted from the image sequence captured by the camera. The second solution is a real-time panorama builder, while the third one assembles document panoramas, both from individual video frames. The fourth solution is a real-time face and eye detector. It provides another type of foundation for motion based user interfaces as knowledge of presence and motion of a human faces in the view of the camera can be a powerful application enabler.
Super-resolution pipeline for fast adjudication in watchlist screening
2015 International Conference on Image Processing Theory, Tools and Applications (IPTA), 2015
A panoramic image is a wide-angle mosaic image done by combining several still images. Panoramas ... more A panoramic image is a wide-angle mosaic image done by combining several still images. Panoramas usually refer to single viewpoint images, created by rotating the camera around its optical center. Digital panoramas, on the other hand, are assembled from a series of perspective images through a computationally costly and a memory hungry multistage process. This thesis describes a real-time implementation of a video based image stitcher, designed to create high-quality image mosaics on a mobile phone, which can be integrated on the image pipeline. Handheld devices such as mobile phones have limited memory and processing capabilities. The slow performance or lack of floating point arithmetic units on mobile processors requires a careful selection of the panorama construction algorithms and their implementations. In the context of this thesis, several solutions have been developed to adapt the computationally costly application to a small footprint platform. These include fixed point implementations of the most expensive algorithms and the use of a carefully tailored memory management. The application runs in real time on multimedia phones, producing 360 degrees panorama pictures on the fly. In most practical cases the quality of the pictures for a human viewer is indistinguishable from optical panoramas.
Mobile devices have been identified as promising platforms for interactive vision-based applicati... more Mobile devices have been identified as promising platforms for interactive vision-based applications. However, this type of applications still pose significant challenges in terms of latency, throughput and energy-efficiency. In this context, the integration of reconfigurable architectures on mobile devices allows dynamic reconfiguration to match the computation and data flow of interactive applications, demonstrating significant performance benefits compared to general purpose architectures. In this context, this paper presents concepts laying on platform level adaptability , exploring the acceleration of vision-based interactive applications through the utilization of three reconfigurable architectures: A low-power EnCore processor with a Con-figurable Flow Accelerator co-processor, a hybrid recon-figurable SIMD/MIMD platform and Transport-Triggered Architecture-based processors. The architectures are evaluated and compared with current processors, analyzing their advantages and weaknesses in terms of performance and energy-efficiency when implementing highly interactive vision-based applications. The results show that the inclusion of reconfigurable platforms on mobile devices can enable the computation of several computationally heavy tasks with high performance and small energy consumption while providing enough flexibility.
—The Kinship Face in the Wild data sets, recently published in TPAMI, are currently used as a ben... more —The Kinship Face in the Wild data sets, recently published in TPAMI, are currently used as a benchmark for the evaluation of kinship verification algorithms. We recommend that these data sets are no longer used in kinship verification research unless there is a compelling reason that takes into account the nature of the images. We note that most of the image kinship pairs are cropped from the same photographs. Exploiting this cropping information, competitive but biased performance can be obtained using a simple scoring approach, taking only into account the nature of the image pairs rather than any features about kin information. To illustrate our motives, we provide classification results utilizing a simple scoring method based on the image similarity of both images of a kinship pair. Using simply the distance of the chrominance averages of the images in the Lab color space without any training or using any specific kin features, we achieve performance comparable to state-of-the-art methods. We provide the source code to prove the validity of our claims and ensure the repeatability of our experiments.
Proceedings of …, Jan 1, 2011
The future multi-modal user interfaces of battery-powered mobile devices are expected to require ... more The future multi-modal user interfaces of battery-powered mobile devices are expected to require computationally costly image analysis techniques. The use of Graphic Processing Units for computing is very well suited for parallel processing and the addition of programmable stages and high precision arithmetic provide for opportunities to implement energy-efficient complete algorithms. At the moment the first mobile graphics accelerators with programmable pipelines are available, enabling the GPGPU implementation of several image processing algorithms. In this context, we consider a face tracking approach that uses efficient gray-scale invariant texture features and boosting. The solution is based on the Local Binary Pattern (LBP) features and makes use of the GPU on the pre-processing and feature extraction phase. We have implemented a series of image processing techniques in the shader language of OpenGL ES 2.0, compiled them for a mobile graphics processing unit and performed tests on a mobile application processor platform (OMAP3530). In our contribution, we describe the challenges of designing on a mobile platform, present the performance achieved and provide measurement results for the actual power consumption in comparison to using the CPU (ARM) on the same platform.
Local binary pattern (LBP) is a texture operator that is used in several different computer visio... more Local binary pattern (LBP) is a texture operator that is used in several different computer vision applications requiring, in many cases, real-time operation in multiple computing platforms. The irruption of new video standards has increased the typical resolutions and frame rates, which need considerable computational performance. Since LBP is essentially a pixel operator that scales with image size, typical straightforward implementations are usually insufficient to meet these requirements. To identify the solutions that maximize the performance of the real-time LBP extraction, we compare a series of different implementations in terms of computational performance and energy efficiency, while analyzing the different optimizations that can be made to reach real-time performance on multiple platforms and their different available computing resources. Our contribution addresses the extensive survey of LBP implementations in different platforms that can be found in the literature. To provide for a more complete evaluation, we have implemented the LBP algorithms in several platforms, such as graphics processing units, mobile processors and a hybrid programming model image coprocessor. We have extended the evaluation of some of the solutions that can be found in previous work. In addition, we publish the source code of our implementations.
Proceedings of …, Jan 1, 2009
Modern mobile communication devices frequently contain built-in cameras allowing users to capture... more Modern mobile communication devices frequently contain built-in cameras allowing users to capture highresolution still images, but at the same time the imaging applications are facing both usability and throughput bottlenecks. The difficulties in taking ad hoc pictures of printed paper documents with multi-megapixel cellular phone cameras on a common business use case, illustrate these problems for anyone. The result can be examined only after several seconds, and is often blurry, so a new picture is needed, although the view-finder image had looked good. The process can be a frustrating one with waits and the user not being able to predict the quality beforehand. The problems can be traced to the processor speed and camera resolution mismatch, and application interactivity demands. In this context we analyze building mosaic images of printed documents from frames selected from VGA resolution (640x480 pixel) video. High interactivity is achieved by providing real-time feedback on the quality, while simultaneously guiding the user actions. The graphics processing unit of the mobile device can be used to speed up the reconstruction computations. To demonstrate the viability of the concept, we present an interactive document scanning application implemented on a Nokia N95 mobile phone.
Abstract Computer vision enables camera data to be utilized in user interfaces to analyze the 3-D... more Abstract Computer vision enables camera data to be utilized in user interfaces to analyze the 3-D context and automatically detect the user intentions. Using cameras as an input modality provides single-handed operations in which the users' actions are recognized without interactions with the screen or keypad.
ABSTRACT Stitching frames from a video sequence has become a good alternative to the traditional ... more ABSTRACT Stitching frames from a video sequence has become a good alternative to the traditional method with only a few photographs. The big overlapping between consecutive video-frames makes it possible to construct high-quality panoramas. Most stitching methods require a vast amount of computational resources that make them not suitable for mobile devices. We present a panorama stitching method that is able to create panorama images from a video sequence.
Conclusion We have demonstrated that proper video analysis and machine intelligence can be used t... more Conclusion We have demonstrated that proper video analysis and machine intelligence can be used to create new mobile device applications. All the described solutions have been implemented on Nokia Nseries mobile phones. More information about these computer vision developments is available on our Web site, including a demonstration video.
ABSTRACT Automatic surface inspection has been used in the industry to reliably detect all kinds ... more ABSTRACT Automatic surface inspection has been used in the industry to reliably detect all kinds of surface defects and to measure the overall quality of a produced piece. Structured light systems (SLS) are based on the reconstruction of the 3D information of a selected area by projecting several phase-shifted sinusoidal patterns onto a surface. Due to the high speed of production lines, surface inspection systems require extremely fast imaging methods and lots of computational power.
Proceedings of …, Jan 1, 2007
Image stitching is used to combine several images into one wide-angled mosaic image. Traditionall... more Image stitching is used to combine several images into one wide-angled mosaic image. Traditionally mosaic images have been constructed from a few separate photographs, but nowadays that video recording has become commonplace even on mobile phones, it is possible to consider also video sequences as a source for mosaic images. However, most stitching methods require vast amounts of computational resources that make them unusable on mobile devices.
Proceedings of SPIE- The …, Jan 1, 2011
The increased sensing and computing capabilities of mobile devices can provide for enhanced mobil... more The increased sensing and computing capabilities of mobile devices can provide for enhanced mobile user experience. Integrating the data from different sensors offers a way to improve application performance in camera-based applications. A key advantage of using cameras as an input modality is that it enables recognizing the context. Therefore, computer vision has been traditionally utilized in user interfaces to observe and automatically detect the user actions. The imaging applications can also make use of various sensors for improving the interactivity and the robustness of the system. In this context, two applications fusing the sensor data with the results obtained from video analysis have been implemented on a Nokia Nseries mobile device. The first solution is a real-time user interface that can be used for browsing large images. The solution enables the display to be controlled by the motion of the user's hand using the built-in sensors as complementary information. The second application is a real-time panorama builder that uses the device's accelerometers to improve the overall quality, providing also instructions during the capture. The experiments show that fusing the sensor data improves camera-based applications especially when the conditions are not optimal for approaches using camera data alone.
Society of Photo-Optical …, Jan 1, 2008
The video applications on mobile communication devices have usually been designed for content cre... more The video applications on mobile communication devices have usually been designed for content creation, access, and playback. For instance, many recent mobile devices replicate the functionalities of portable video cameras and video recorders, and digital TV receivers. These are all demanding uses, but nothing new from the consumer point of view. However, many of the current devices have two cameras built in, one for capturing high resolution images, and the other for lower, typically VGA (640x480 pixels) resolution video telephony. We employ video to enable new applications and describe four actual solutions implemented on mobile communication devices. The first one is a real-time motion based user interface that can be used for browsing large images or documents such as maps on small screens. The motion information is extracted from the image sequence captured by the camera. The second solution is a real-time panorama builder, while the third one assembles document panoramas, both from individual video frames. The fourth solution is a real-time face and eye detector. It provides another type of foundation for motion based user interfaces as knowledge of presence and motion of a human faces in the view of the camera can be a powerful application enabler.
Teaching Documents by Miguel Bordallo
Uploads
Papers by Miguel Bordallo
Teaching Documents by Miguel Bordallo