Colour Saliency on Video

Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering, 2012

We investigate the contribution of local spatio-temporal variation of image intensity to saliency. To measure different types of variation, we use the geometrical invariants of the structure tensor. With a video represented in spatial axes x and y and temporal axis t, the ndimensional structure tensor can be evaluated for different combinations of axes (2D and 3D) and also for the (degenerate) case of only one axis. The resulting features are evaluated on several spatio-temporal scales in terms of how well they can predict eye movements on complex videos. We find that a 3D structure tensor is optimal: the most predictive regions of a movie are those where intensity changes along all spatial and temporal directions. Among two-dimensional variations, the axis pair yt, which is sensitive to horizontal translation, outperforms xy and xt by a large margin, and is even superior in prediction to two baseline models of bottom-up saliency.

Vision Research, 2007

To what extent can a computational model of the bottom-up visual attention predict what an observer is looking at? What is the contribution of the low-level visual features in the attention deployment? To answer these questions, a new spatio-temporal computational model is proposed. This model incorporates several visual features; therefore, a fusion algorithm is required to combine the different saliency maps (achromatic, chromatic and temporal). To quantitatively assess the model performances, eye movements were recorded while naive observers viewed natural dynamic scenes. Four completing metrics have been used. In addition, predictions from the proposed model are compared to the predictions from a state of the art model [Itti's model (Itti, L., . A model of saliency-based visual attention for rapid scene analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence 20(11), 1254-1259)] and from three non-biologically plausible models (uniform, flicker and centered models). Regardless of the metric used, the proposed model shows significant improvement over the selected benchmarking models (except the centered model). Conclusions are drawn regarding both the influence of low-level visual features over time and the central bias in an eye tracking experiment.

SPIE Proceedings, 2004

The saliency-based or bottom-up model of visual attention presented in this paper deals with still color images. The model we built is based on numerous properties of the human visual system (HVS), thus providing a biologically plausible system. The computation of early visual features such as color and orientation is a key step for any bottom-up model and the way to extract these visual features easily permits to differentiate a model from an other. The novelty of the proposed approach lies on the fact that the computation of early visual features is fully based on a HVS model consisting in projecting the picture into an opponent-colors space, applying a perceptual decomposition, contrast sensitivity and masking functions. Moreover, a strategy essentially based on a center surround mechanism and on the perceptual grouping phenomena underscores conspicuous locations by combining visual feature maps. A saliency map which is defined as a 2D topographic representation of conspicuity is then deduced. The model is applied to a number of natural images. Our results are then compared with the results of a well-know bottom-up model.

Cognitive Computation, 2010

When looking at a scene, we frequently move our eyes to place consecutive interesting regions on the fovea, the retina centre. At each fixation, only this specific foveal region is analysed in detail by the visual system. The visual attention mechanisms control eye movements and depend on two types of factor: bottom-up and top-down factors. Bottom-up factors include different visual features such as colour, luminance, edges, and orientations. In this paper, we evaluate quantitatively the relative contribution of basic low-level features as candidate guiding factors to visual attention and hence to eye movements. We also study how these visual features can be combined in a bottom-up saliency model. Our work consists of three interactive parts: a functional saliency model, a statistical model and eye movement data recorded during free viewing of natural scenes. The functional saliency model, inspired by the primate visual system, decomposes a visual scene into different feature maps. The statistical model indicates which features best explain the recorded eye movements. We show an essential role of high frequency luminance and an important contribution of central fixation bias. The relative contribution of features, calculated by the statistical model, is then used to combine the different feature maps into a saliency map. Finally, the comparison between the saliency model and experimental data confirmed the influence of these contributions.

IEEE Access

In the first seconds of observation of an image, several visual attention processes are involved in the identification of the visual targets that pop-out from the scene to our eyes. Saliency is the quality that makes certain regions of an image stand out from the visual field and grab our attention. Saliency detection models, inspired by visual cortex mechanisms, employ both colour and luminance features. Furthermore, both locations of pixels and presence of objects influence the Visual Attention processes. In this paper, we propose a new saliency method based on the combination of the distribution of interest points in the image with multiscale analysis, a centre bias module and a machine learning approach. We use perceptually uniform colour spaces to study how colour impacts on the extraction of saliency. To investigate eye-movements and assess the performances of saliency methods over object-based images, we conduct experimental sessions on our dataset ETTO (Eye Tracking Through Objects). Experiments show our approach to be accurate in the detection of saliency concerning state-of-the-art methods and accessible eye-movement datasets. The performances over object-based images are excellent and consistent on generic pictures. Besides, our work reveals interesting findings on some relationships between saliency and perceptually uniform colour spaces. INDEX TERMS Eye-movements, interest points, saliency map, visual attention.

Computer Vision and Image Understanding, 2005

Visual attention is the ability of a vision system, be it biological or artificial, to rapidly detect potentially relevant parts of a visual scene, on which higher level vision tasks, such as object recognition, can focus. The saliency-based model of visual attention represents one of the main attempts to simulate this visual mechanism on computers. Though biologically inspired, this model has only been partially assessed in comparison with human behavior. Our methodology consists in comparing the computational saliency map with human eye movement patterns. This paper presents an in-depth analysis of the model by assessing the contribution of different cues to visual attention. It reports the results of a quantitative comparison of human visual attention derived from fixation patterns with visual attention as modeled by different versions of the computer model. More specifically, a one-cue gray-level model is compared to a two-cues color model. The experiments conducted with over forty images of different nature and involving twenty human subjects assess the quantitative contribution of chromatic features in visual attention.

Human Vision and Electronic Imaging IX, 2004

The saliency-based or bottom-up model of visual attention presented in this paper deals with still color images. The model we built is based on numerous properties of the human visual system (HVS), thus providing a biologically plausible system. The computation of early visual features such as color and orientation is a key step for any bottom-up model and the way to extract these visual features easily permits to differentiate a model from an other. The novelty of the proposed approach lies on the fact that the computation of early visual features is fully based on a HVS model consisting in projecting the picture into an opponent-colors space, applying a perceptual decomposition, contrast sensitivity and masking functions. Moreover, a strategy essentially based on a center surround mechanism and on the perceptual grouping phenomena underscores conspicuous locations by combining visual feature maps. A saliency map which is defined as a 2D topographic representation of conspicuity is then deduced. The model is applied to a number of natural images. Our results are then compared with the results of a well-know bottom-up model.

International Journal of Software Science and Computational Intelligence, 2018

A key factor in designing saliency detection algorithms for videos is to understand how different visual cues affect the human perceptual and visual system. To this end, this article investigated the bottom-up features including color, texture, and motion in video sequences for a one-by-one scenario to provide a ranking system stating the most dominant circumstances for each feature. In this work, it is considered the individual features and various visual saliency attributes investigated under conditions in which the authors had no cognitive bias. Human cognition refers to a systematic pattern of perceptual and rational judgments and decision-making actions. First, this paper modeled the test data as 2D videos in a virtual environment to avoid any cognitive bias. Then, this paper performed an experiment using human subjects to determine which colors, textures, motion directions, and motion speeds attract human attention more. The proposed benchmark ranking system of salient visual ...

ELCVIA Electronic Letters on Computer Vision and Image Analysis

Visual attention is the ability of the human vision system to detect salient parts of the scene, on which higher vision tasks, such as recognition, can focus. In human vision, it is believed that visual attention is intimately linked to the eye movements and that the fixation points correspond to the location of the salient scene parts. In computer vision, the paradigm of visual attention has been widely investigated and a saliencybased model of visual attention is now available that is commonly accepted and used in the field, despite the fact that its biological grounding has not been fully assessed. This work proposes a new method for quantitatively assessing the plausibility of this model by comparing its performance with human behavior. The basic idea is to compare the map of attention-the saliency map-produced by the computational model with a fixation density map derived from eye movement experiments. This human attention map can be constructed as an integral of single impulses located at the positions of the successive fixation points. The resulting map has the same format as the computer-generated map, and can easily be compared by qualitative and quantitative methods. Some illustrative examples using a set of natural and synthetic color images show the potential of the validation method to assess the plausibility of the attention model.

Frontiers in Human Neuroscience, 2023

Many visual attention models have been presented to obtain the saliency of a scene, i.e., the visually significant parts of a scene. However, some mechanisms are still not taken into account in these models, and the models do not fit the human data accurately. These mechanisms include which visual features are informative enough to be incorporated into the model, how the conspicuity of di erent features and scales of an image may integrate to obtain the saliency map of the image, and how the structure of an image a ects the strategy of our attention system. We integrate such mechanisms in the presented model more e ciently compared to previous models. First, besides low-level features commonly employed in state-of-the-art models, we also apply medium-level features as the combination of orientations and colors based on the visual system behavior. Second, we use a variable number of center-surround di erence maps instead of the fixed number used in the other models, suggesting that human visual attention operates di erently for diverse images with di erent structures. Third, we integrate the information of di erent scales and di erent features based on their weighted sum, defining the weights according to each component's contribution, and presenting both the local and global saliency of the image. To test the model's performance in fitting human data, we compared it to other models using the CAT dataset and the Area Under Curve (AUC) metric. Our results show that the model has high performance compared to the other models (AUC =. and sAUC = .) and suggest that the proposed mechanisms can be applied to the existing models to improve them.