Video segmentation by MAP labeling of watershed segments

Image Processing, IEEE International Conference, 2008

In this paper, we propose a Markov random field sequence segmentation and regions tracking model, which aims at combining color, texture, and motion features. First a motion-based segmentation is realized. Namely the global motion of the video sequence is estimated and compensated. From the remaining motion information, a rough motion segmentation is achieved. Then, we use a Markovian approach to

Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001

Video segmentation is different from segmentation of a single image. While several correct solutions may exist for segmenting a single image, there needs to be a consistency among segmentations of each frame for video segmentation. Previous approaches of video segmentation concentrate on motion, or combine motion and color information in a batch fashion. We propose a maximum a posteriori probability (MAP) framework that uses multiple cues, like spatial location, color and motion, for segmentation. We assign weights to color and motion terms, which are adjusted at every pixel, based on a confidence measure of each feature. We also discuss the appropriate modeling of pdfs of each feature of a region. The correct modeling of the spatial pdf imposes temporal consistency among segments in consecutive frames. This approach unifies the strengths of both color segmentation and motion segmentation in one framework, and shows good results on videos that are not suited for either of these approaches.

Signal Processing: Image …, 2003

2007

This paper presents an approach to multi-object image segmentation based on object motion using Markov Random Fields 1. To support the information gained from motion and to achieve robustness, several additional visual cues extracted from the image data are integrated. Depth information gained from stereo disparity is included to maintain segmentation in case an segmented object stops moving. Motion is estimated with a correspondence matching scheme. The approach differs from regular optical flow in the way that rich matching results are used for segmentation rather than only the best matches. The representation of segmented regions is realized implicitly as labeling on a 2D lattice. Motion segmentation is a key to many modern image processing applications. In video compression algorithms, the analysis of motion and regions with coherent motion helps to drastically reduce the amount of information that has to be stored and transmitted for each frame [11]. Motion segmentation and motion understanding, for example, plays an essential role in detecting and/or avoiding obstacles in vehicles or with a mobile robot. The rest of this paper is organized as follows: while Sect. 2 provides a short overview of the work done in the field of image segmentation and relations to our approach, Sect. 3 describes the architecture of the proposed system. Its evaluation is presented in Sect. 4. Finally, the paper concludes with Sect. 5. 2 State of the Art Recent research in the area of motion segmentation focuses on feature-based motion estimation, approaches using level set methods [4], and on multi-cue segmentation on Markov Random Fields [6]. Optimization on Markov Random Fields is an established approach to segmentation [9]. It permits to let motion estimation be part of the segmentation process in one single optimization framework. When Markov Random Fields are used in combination with optimization methods like iterated conditional modes (ICM) [2], the initialization of the field configuration and the label parameters has a strong influence on the time needed for convergence as well as on the quality of the result. In multi-frame applications, results from the last frame can be used for initialization [3]. Our approach uses motion estimates that fulfill certain quality criteria, as well as results from the previous frame.

This paper proposes an improvement over moving objects segmentation method for video sequence based on Bayesian network. The method integrates temporal and spatial features by Bayesian network through three fields, which are motion vector field, intensity segmentation field and object video segmentation field. Markov random field aims to push the spatial connectivity between regions. The improvement concentrates on the MAP estimation procedure in order to obtain the exact segmentation results. The Iterative MAP Estimation may cause much more error in estimation procedure and degrade the convergence of the algorithm. This paper proposes a noniterative Estimation as an improvement for this algorithm. The non-iterative MAP estimation does not need the previous segmentation result. Therefore, the inaccurate segmentation result of former stage does not have effect on the current segmentation stage. Additionally, the non-iterative MAP estimation was designed to adapt the original model so that it does not cause failure from the theory. Experiments show that the improvement is better than the original version and has good results in some benchmark video sequences.

2010

Multiple Hypothesis Video Segmentation (MHVS) is a method for the unsupervised photometric segmentation of video sequences. MHVS segments arbitrarily long video streams by considering only a few frames at a time, and handles the automatic creation, continuation and termination of labels with no user initialization or supervision. The process begins by generating several pre-segmentations per frame and enumerating multiple possible trajectories of pixel regions within a short time window. After assigning each trajectory a score, we let the trajectories compete with each other to segment the sequence. We determine the solution of this segmentation problem as the MAP labeling of a higher-order random field. This framework allows MHVS to achieve spatial and temporal long-range label consistency while operating in an on-line manner. We test MHVS on several videos of natural scenes with arbitrary camera and object motion.

2003

In this thesis, a new method for performing motion segmentation on image sequences is introduced. This method uses the popular background model / background subtraction paradigm used in other motion segmentation algorithms. However, our approach exploits the spatial and temporal dependencies that objects in motion impose on their images. This is achieved by the construction of a Markov random field of binary motion segmentation variates. Three variations of field structures were tested against two traditional threshold-based background subtraction algorithms. The segmentations produced by the MRF-based techniques were consistently more accurate and less prone to noise and background camouflaging effects than their traditional counterparts. While the speed of this new approach is slower than the two traditional algorithms, its time complexity remains linear. Thus, the difference in algorithm speed will become less significant as processors evolve, while the performance characteristic...

2007

While several methods of automatic video segmentation for the identification of shot transition have been proposed, they have not been systematically compared. In this paper, several segmentation techniques cross different type of video are examined. Each of these techniques defines a measure of dissimilarity between successive frames, which is then compared to a threshold .Dissimilarity values exceeding the threshold identify shot transition. The techniques are compared in terms of the percentage of correct and false identifications for various thresholds, their sensitivity to the threshold value their performance across different type of video, their ability to identify complicated transition effects, and their requirements for computational resources. Finally, the definition of a priori set of values for the threshold parameter is also examined. Most techniques can identify over 90% of the real shot transitions but have a high percentage of false positives. Reducing the false positives was a major challenge; this paper introduced a local filtering technique that was fairly effective.

—Extracting spatio-temporally consistent segments from a video sequence is a challenging problem due to the complexity of color, motion and occlusions. Most existing spatio-temporal segmentation approaches have inherent difficulties in handling large displacement with significant occlusions. This paper presents a novel framework for spatio-temporal segmentation. With the estimated depth data beforehand by a multi-view stereo technique, we project the pixels to other frames for collecting the boundary and segmentation statistics in a video, and incorporate them into the segmentation energy for spatio-temporal optimization. In order to effectively solve this problem, we introduce an iterative optimization scheme by first initializing segmentation maps for each frame independently, and then link the correspondences among different frames and iteratively refine them with the collected statistics, so that a set of spatio-temporally consistent volume segments are finally achieved. The effectiveness and usefulness of our automatic framework are demonstrated via its applications for 3D reconstruction, video editing and semantic segmentation on a variety of challenging video examples.

Pattern Recognition Letters, 2003

The video segmentation problem consists in the identification of the boundary between consecutive shots. The common approach to solve this problem is based on dissimilarity measures between frames. In this work, the video segmentation problem is transformed into a problem of pattern detection, where each video event is transformed into a different pattern on a 2D image, called visual rhythm, obtained by a specific transformation. In our analysis we use topological and morphological tools to detect cuts. Also, we use discrete line analysis and max tree analysis to detect fade transitions and flashes, respectively. We present a comparative analysis of our method for cut detection with respect to some other methods, which shows the better results of our method.