Papers by Margaret Pinson
Sigcomm, 2001
Over the past few years, there have been a number of proposals aimed at introducing different lev... more Over the past few years, there have been a number of proposals aimed at introducing different levels of service in the Internet. One of the more recent proposals is the Differentiated Services (Diff-Serv) architecture, and in this paper we explore how the policing actions and associated rate guarantees provided by the Expedited Forwarding (EF) translate into perceived benefits for applications that are the presumed users of such enhancements. Specifically, we focus on video streaming applications that arguably have relatively strong service quality requirements, and which should, therefore, stand to benefit from the availability of some form of enhanced service. Our goal is to gain a better understanding of the relation that exists between application level quality measures and the selection of the network level parameters that govern the delivery of the guarantees that an EF based service would provide. Our investigation, which is experimental in nature, relies on a number of standard streaming video servers and clients that have been modified and instrumented to allow quantification of the perceived quality of the received video stream. Quality assessments are performed using a Video Quality Measurement tool based on the ANSI objective quality standard. Measurements were made over both a local Diff-Serv testbed and across the QBone, a QoS enabled segment of the Internet2 infrastructure. The paper reports and analyzes the results of those measurements.
Subject bias: Introducing a theoretical user model
2014 Sixth International Workshop on Quality of Multimedia Experience (QoMEX), 2014
Video Scaling Estimation Technique
In-Service Video Quality Metric (IVQM) User's Manual
APPLICATION OF THE NTIA GENERAL VIDEO QUALITY METRIC (VQM) TO HDTV QUALITY MONITORING
This paper summarizes results from an experiment whose goal was to assess whether the NTIA Genera... more This paper summarizes results from an experiment whose goal was to assess whether the NTIA General Video Quality Metric (VQM) is an acceptable objective metric for measuring High Definition TV (HDTV) video quality. The HDTV subjective test that was performed to evaluate the NTIA General VQM contained 60 30-second video clips that were rated using the Single Stimulus Continuous Quality Evaluation (SSCQE) method. The 60 clips included twelve 1080i HDTV originals and 48 processed versions of these originals from 16 different video systems. The video systems included 5 different HDTV codecs running at bit rates from 2 to 19 Mbps and broadcast transmission errors (i.e., RF transmission with poor signal-to-noise-ratio). Excellent objective-to- subjective correlation results for this experiment demonstrate the potential application of the NTIA General VQM to HDTV quality monitoring.
IEEE Transactions on Broadcasting, 2014
We announce a new video quality model (VQM) that accounts for the perceptual impact of variable f... more We announce a new video quality model (VQM) that accounts for the perceptual impact of variable frame delays (VFD) in videos with demonstrated top performance on the laboratory for image and video engineering (LIVE) mobile video quality assessment (VQA) database. This model, called VQM_VFD, uses perceptual features extracted from spatialtemporal blocks spanning fixed angular extents and a long edge detection filter. VQM_VFD predicts video quality by measuring multiple frame delays using perception based parameters to track subjective quality over time. In the performance analysis of VQM_VFD, we evaluated its efficacy at predicting human opinions of visual quality. A detailed correlation analysis and statistical hypothesis testing show that VQM_VFD accurately predicts human subjective judgments and substantially outperforms top-performing image quality assessment and VQA models previously tested on the LIVE mobile VQA database. VQM_VFD achieved the best performance on the mobile and tablet studies of the LIVE mobile VQA database for simulated compression, wireless packet-loss, and rate adaptation, but not for temporal dynamics. These results validate the new model and warrant a hard release of the VQM_VFD algorithm. It is freely available for any purpose, commercial, or noncommercial at http://www.its.bldrdoc.gov/vqm/.
Video Quality Assessment: Subjective testing of entertainment scenes
IEEE Signal Processing Magazine, 2015
ABSTRACT This article describes how to perform a video quality subjective test. For companies, th... more ABSTRACT This article describes how to perform a video quality subjective test. For companies, these tests can greatly facilitate video product development; for universities, removing perceived barriers to conducting such tests allows expanded research opportunities. This tutorial assumes no prior knowledge and focuses on proven techniques. (Certain commercial equipment, materials, and/or programs are identified in this article to adequately specify the experimental procedure. In no case does such identification imply recommendation or endorsement by the National Telecommunications and Information Administration, nor does it imply that the program or equipment identified is necessarily the best available for this application.)
Objective quality assessment of digitally transmitted video
IEEE Pacific Rim Conference on Communications, Computers and Signal Processing (Cat. No.91CH2954-6), 1991
... A newly developed automated method and laboratory system for petforming objective video quali... more ... A newly developed automated method and laboratory system for petforming objective video quality measurements is presented. ... Application to Video Teleconferencing (VTC) This section describes one application of the objective video quality measurement system. ...
An objective video quality assessment system based on human perception
Proceedings of the SPIE - The International Society for Optical Engineering, 1993
The Institute for Telecommunication Sciences (ITS) has developed an objective video quality asses... more The Institute for Telecommunication Sciences (ITS) has developed an objective video quality assessment system that emulates human perception. The system returns results that agree closely with quality judgements made by a large panel of viewers. Such a system is valuable ...
Spatial-temporal distortion metric for in-service quality monitoring of any digital video system
Proceedings of the SPIE - The International Society for Optical Engineering, 1999
Many organizations have focused on developing digital video quality metrics which produce results... more Many organizations have focused on developing digital video quality metrics which produce results that accurately emulate subjective responses. However, to be widely applicable a metric must also work over a wide range of quality, and be useful for in-service quality monitoring. The Institute for Telecommunication Sciences (ITS) has developed spatial-temporal distortion metrics that meet all of these requirements. These objective metrics are described in detail and have a number of interesting properties, including utilization of (1) spatial activity filters which emphasize long edges on the order of 10 arc min while simultaneously performing large amounts of noise suppression, (2) the angular direction of the spatial gradient, (3) spatial-temporal compression factors of at least 384:1 (spatial compression of at least 64:1 and temporal compression of at least 6:1, and 4) simple perceptibility thresholds and spatial-temporal masking functions. Results are presented that compare the objective metric values with mean opinion scores from a wide range of subjective data bases spanning many different scenes, systems, bit-rates, and applications.
Proceedings of the SPIE - The International Society for Optical Engineering, 2003
International recommendations for subjective video quality assessment (e.g., ITU-R BT.500-11) inc... more International recommendations for subjective video quality assessment (e.g., ITU-R BT.500-11) include specifications for how to perform many different types of subjective tests. Some of these test methods are double stimulus where viewers rate the quality or change in quality between two video streams (reference and impaired). Others are single stimulus where viewers rate the quality of just one video stream (the impaired). Two examples of the former are the double stimulus continuous quality scale (DSCQS) and double stimulus comparison scale (DSCS). An example of the latter is single stimulus continuous quality evaluation (SSCQE). Each subjective test methodology has claimed advantages. For instance, the DSCQS method is claimed to be less sensitive to context (i.e., subjective ratings are less influenced by the severity and ordering of the impairments within the test session). The SSCQE method is claimed to yield more representative quality estimates for quality monitoring applications. This paper considers data from six different subjective video quality experiments, originally performed with SSCQE, DSCQS and DSCS methodologies. A subset of video clips from each of these six experiments were combined and rated in a secondary SSCQE subjective video quality test. We give a method for postprocessing the secondary SSCQE data to produce quality scores that are highly correlated to the original DSCQS and DSCS data. We also provide evidence that human memory effects for time-varying quality estimation seem to be limited to about 15 seconds. Keywords: single stimulus continuous quality evaluation (SSCQE), double stimulus continuous quality scale (DSCQS), double stimulus comparison scale (DSCS), correlation, video quality, image quality, subjective testing, picture quality. 1 Reference [1] presents a new form of double stimulus testing called the simultaneous double stimulus for continuous evaluation (SDSCE) that utilizes side-by-side presentation of the original and impaired clips rather than randomized time ordering. However, this method has the drawback that the viewer must shift attention between the right and left presentations.
An objective method for combining multiple subjective data sets
Proceedings of the SPIE - The International Society for Optical Engineering, 2003
International recommendations for subjective video quality assessment (eg, ITU-R BT.500-11) inclu... more International recommendations for subjective video quality assessment (eg, ITU-R BT.500-11) include specifications for how to perform many different types of subjective tests. In addition to displaying the video sequences in different ways, subjective tests also have ...
Video Performance Requirements for Tactical Video Applications
2007 IEEE Conference on Technologies for Homeland Security, 2007
ABSTRACT The Public Safety Statement of Requirements (PS-SoR) for Communications & Intero... more ABSTRACT The Public Safety Statement of Requirements (PS-SoR) for Communications & Interoperability focuses on the needs of first responders to communicate and share information as authorized, when it is needed, where it is needed, and in a mode or form that allows the practitioners to effectively use it. PS-SoR Volume I defined functional communication and interoperability requirements. Published in September, 2006, PS-SoR Volume II identifies quantitative performance metrics, including minimum video performance requirements for public safety's tactical video applications. The goal was not to identify what is achievable with current technology but rather, looking towards the future, to investigate the minimum level of performance that first responders need in order to effectively use their video equipment. On behalf of the SAFECOM Program and the Office of Law Enforcement Standards, the Institute for Telecommunication Sciences (ITS) conducted subjective video quality testing to estimate the level of video quality that first responders find acceptable for tactical video applications. This subjective testing utilized source video content that is typical of public safety operations in structured subjective viewing experiments with 35 first responders. The evaluations from these first responders, in viewing high quality video (original video) and purposefully degraded video (using video compression and transmission equipment), allowed determination of basic quality thresholds for public safety tactical video applications. These perceptual quality thresholds have been translated into technical parameters for use by video equipment designers, manufacturers, and customers. This paper summarizes those findings. Other testing to evaluate requirements for other public safety applications is underway.
The history of video quality model validation
2013 IEEE 15th International Workshop on Multimedia Signal Processing (MMSP), 2013
ABSTRACT This paper describes objective video quality validation efforts conducted in the past tw... more ABSTRACT This paper describes objective video quality validation efforts conducted in the past two decades. Validation efforts to be examined include a validation test performed by the T1A1 committee in the early 1990's; five rounds of validation testing performed by the Video Quality Experts Group; and validation tests performed by ITU-T Study Group 12. Useful products that resulted from those efforts will be identified, including standards, datasets, and model validation techniques.
Impact of mobile devices and usage location on perceived multimedia quality
2012 Fourth International Workshop on Quality of Multimedia Experience, 2012
ABSTRACT We explore the quality impact when audiovisual content is delivered to different mobile ... more ABSTRACT We explore the quality impact when audiovisual content is delivered to different mobile devices. Subjects were shown the same sequences on five different mobile devices and a broadcast quality television. Factors influencing quality ratings include video resolution, viewing distance, and monitor size. Analysis shows how subjects' perception of multimedia quality differs when content is viewed on different mobile devices. In addition, quality ratings from laboratory and simulated living room sessions were statistically equivalent.
Batch Video Quality Metric (BVQM) User’s Manual
Committee T1 Performance Standards Contribution
The Relationship Between Performance and Spatial-Temporal Region Size for Reduced-Reference, In-Service Video Quality Monitoring Systems
This paper presents objective-to-subjective correlation results for a reduced -reference, in-serv... more This paper presents objective-to-subjective correlation results for a reduced -reference, in-service, video quality monitoring system. This reduced-reference system utilizes quality parameters that are computed by comparing features extracted from spatial-temporal (S-T) regions of the input video stream with identical features extracted from the output video stream. The amount of reduced - reference information that is required to compute the quality parameters is inversely related to the size of the S- T region. Smaller amounts of reference information (i.e., larger S-T regions) are desired since less transmission or storage bandwidth is required for the reference information. However, objective-to-subjective correlation drops off if the S-T region size becomes too large. In this paper we examine the tradeoffs between objective-to- subjective correlation results and S-T region size. Correlation results for S-T region sizes from 8 vertical lines x 8 horizontal pixels x 2 video frame...
Validating objective measures of MPEG video quality
SMPTE …, 1998
In 1996, the American National Standards Institute (ANSI) adopted ANSI T1. 801.03, which presents... more In 1996, the American National Standards Institute (ANSI) adopted ANSI T1. 801.03, which presents a number of new objective video quality metrics for quantifying the effects of digital compression and transmission impairments. The measurements in ANSI ...
The Impact of Monitor Resolution and Type on Subjective Video Quality Testing
Uploads
Papers by Margaret Pinson