The Precision and Repeatability of Media Quality Comparisons: Measurements and New Statistical Methods

This paper calculates confidence intervals for 89 datasets that use the 5-level Absolute Category Rating (ACR) method to evaluate the quality of speech, video, images, and video with audio. This data allows us to compute the subjective test confidence interval <inline-formula> <tex-math not...

Full description

Saved in:
Bibliographic Details
Published inIEEE transactions on broadcasting Vol. 69; no. 2; pp. 1 - 18
Main Author Pinson, Margaret H.
Format Journal Article
LanguageEnglish
Published New York IEEE 01.06.2023
The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Subjects
Online AccessGet full text
ISSN0018-9316
1557-9611
1557-9611
DOI10.1109/TBC.2023.3236528

Cover

Abstract This paper calculates confidence intervals for 89 datasets that use the 5-level Absolute Category Rating (ACR) method to evaluate the quality of speech, video, images, and video with audio. This data allows us to compute the subjective test confidence interval <inline-formula> <tex-math notation="LaTeX">(\Delta </tex-math> </inline-formula>SCI) for 5-level ACR tests. We use a confusion matrix to compare conclusions reached by 88 lab-to-lab comparisons, 22 method-to-method comparisons, and 12 comparisons between expert and naïve subjects. We estimate the differences in conclusions reached by ad hoc evaluations, compared to subjective tests. We recommend using the disagree incidence rate to identify lab-to-lab differences (i.e., the likelihood that significantly different stimulus pairs receive opposing rank order from the two labs). Disagree incidence rates above 0.31% are unusual enough to warrant investigation and disagree incidence rates above 1.0% indicate differences in method, test environment, test implementation, or subject demographics. These incidence rates form the basis for a new statistical method that calculates the confidence interval of a metric (<inline-formula> <tex-math notation="LaTeX">\Delta </tex-math> </inline-formula>MCI). When <inline-formula> <tex-math notation="LaTeX">\Delta </tex-math> </inline-formula>MCI is used to make decisions, the equivalence to a video-quality test (EVQT) method determines whether a metric acts similarly to a subjective test. When <inline-formula> <tex-math notation="LaTeX">\Delta </tex-math> </inline-formula>MCI is not used, the metric is likened to a certain number of people in a video-quality test (PVQT). This information will help users make the better decisions when applying quality metrics. The algorithm code is made available for any purpose. Most of the ratings used in this paper come from open datasets.
AbstractList This paper calculates confidence intervals for 89 datasets that use the 5-level Absolute Category Rating (ACR) method to evaluate the quality of speech, video, images, and video with audio. This data allows us to compute the subjective test confidence interval ([Formula Omitted]) for 5-level ACR tests. We use a confusion matrix to compare conclusions reached by 88 lab-to-lab comparisons, 22 method-to-method comparisons, and 12 comparisons between expert and naïve subjects. We estimate the differences in conclusions reached by ad hoc evaluations, compared to subjective tests. We recommend using the disagree incidence rate to identify lab-to-lab differences (i.e., the likelihood that significantly different stimulus pairs receive opposing rank order from the two labs). Disagree incidence rates above 0.31% are unusual enough to warrant investigation and disagree incidence rates above 1.0% indicate differences in method, test environment, test implementation, or subject demographics. These incidence rates form the basis for a new statistical method that calculates the confidence interval of a metric ([Formula Omitted]). When [Formula Omitted] is used to make decisions, the equivalence to a video-quality test (EVQT) method determines whether a metric acts similarly to a subjective test. When [Formula Omitted] is not used, the metric is likened to a certain number of people in a video-quality test (PVQT). This information will help users make the better decisions when applying quality metrics. The algorithm code is made available for any purpose. Most of the ratings used in this paper come from open datasets.
This paper calculates confidence intervals for 89 datasets that use the 5-level Absolute Category Rating (ACR) method to evaluate the quality of speech, video, images, and video with audio. This data allows us to compute the subjective test confidence interval <inline-formula> <tex-math notation="LaTeX">(\Delta </tex-math> </inline-formula>SCI) for 5-level ACR tests. We use a confusion matrix to compare conclusions reached by 88 lab-to-lab comparisons, 22 method-to-method comparisons, and 12 comparisons between expert and naïve subjects. We estimate the differences in conclusions reached by ad hoc evaluations, compared to subjective tests. We recommend using the disagree incidence rate to identify lab-to-lab differences (i.e., the likelihood that significantly different stimulus pairs receive opposing rank order from the two labs). Disagree incidence rates above 0.31% are unusual enough to warrant investigation and disagree incidence rates above 1.0% indicate differences in method, test environment, test implementation, or subject demographics. These incidence rates form the basis for a new statistical method that calculates the confidence interval of a metric (<inline-formula> <tex-math notation="LaTeX">\Delta </tex-math> </inline-formula>MCI). When <inline-formula> <tex-math notation="LaTeX">\Delta </tex-math> </inline-formula>MCI is used to make decisions, the equivalence to a video-quality test (EVQT) method determines whether a metric acts similarly to a subjective test. When <inline-formula> <tex-math notation="LaTeX">\Delta </tex-math> </inline-formula>MCI is not used, the metric is likened to a certain number of people in a video-quality test (PVQT). This information will help users make the better decisions when applying quality metrics. The algorithm code is made available for any purpose. Most of the ratings used in this paper come from open datasets.
Author Pinson, Margaret H.
Author_xml – sequence: 1
  givenname: Margaret H.
  orcidid: 0000-0003-3369-4726
  surname: Pinson
  fullname: Pinson, Margaret H.
  organization: National Telecommunications and Information Administration, Institute for Telecommunication Sciences, Boulder, CO, USA
BookMark eNptkc9P2zAYhq0JpBW2-w47RNo55fOPxAm3rdoACQaMcrZc57MwSu3MdlT1v19KEULVTpbs53n1fa9PyJEPHgn5QmFOKbRnyx-LOQPG55zxumLNBzKjVSXLtqb0iMwAaFO2nNYfyUlKzwDAAdiM2OUTFncRjUsu-EL7rviDA-qsV653eVsEW9xg53RxP-qXi0VYDzq6FHw6n550GiOu0ef0Iv_GTfGQdXYpO6P7CchPoUufyLHVfcLPr-cpefz1c7m4LK9vL64W369Lw7nIpbDQ8Uoy3TVStK00aG3V8kaIBixIvrKVFFDVFippGUppJXSWT4zhVIDhp4Tuc0c_6O1G970aolvruFUU1K4olVdG7YpSr0VNzre9M8Twd8SU1XMYo5_GVKxhnDa1kHSi6j1lYkgpolXG7fYMPkft-rf46R8O4-FAPJzoP8rXveIQ8R0Ogk0l8H8gE5Mc
CODEN IETBAC
CitedBy_id crossref_primary_10_1016_j_image_2024_117176
crossref_primary_10_1109_ACCESS_2025_3544307
crossref_primary_10_1109_TSC_2024_3451215
Cites_doi 10.1109/JSTSP.2012.2215306
10.1109/MSP.2011.942470
10.1109/TBC.2010.2086750
10.1109/QoMEX48832.2020.9123115
10.1109/ICIP.2002.1038897
10.1145/3427931
10.1109/THS.2007.370025
10.2352/ISSN.2470-1173.2016.16.HVEI-124
10.1109/QoMEX.2016.7498936
10.1016/j.image.2015.05.001
10.1109/QoMEX.2011.6065690
10.1109/JSTSP.2016.2638681
10.1109/TMM.2015.2484963
10.1111/j.1467-8659.2012.03188.x
10.1109/ACCESS.2021.3127395
10.1109/QOMEX.2010.5517948
10.3390/jimaging6030007
10.1007/s41233-019-0026-4
10.1109/TBC.2009.2034511
10.1109/ICIP.2016.7532325
10.1109/BMSB.2016.7521936
ContentType Journal Article
Copyright Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2023
Copyright_xml – notice: Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2023
DBID 97E
ESBDL
RIA
RIE
AAYXX
CITATION
7SP
8FD
L7M
ADTOC
UNPAY
DOI 10.1109/TBC.2023.3236528
DatabaseName IEEE Xplore (IEEE)
IEEE Xplore Open Access Journals
IEEE All-Society Periodicals Package (ASPP) 1998–Present
IEEE Electronic Library (IEL)
CrossRef
Electronics & Communications Abstracts
Technology Research Database
Advanced Technologies Database with Aerospace
Unpaywall for CDI: Periodical Content
Unpaywall
DatabaseTitle CrossRef
Technology Research Database
Advanced Technologies Database with Aerospace
Electronics & Communications Abstracts
DatabaseTitleList Technology Research Database

Database_xml – sequence: 1
  dbid: RIE
  name: IEEE Electronic Library (IEL)
  url: https://proxy.k.utb.cz/login?url=https://ieeexplore.ieee.org/
  sourceTypes: Publisher
– sequence: 2
  dbid: UNPAY
  name: Unpaywall
  url: https://proxy.k.utb.cz/login?url=https://unpaywall.org/
  sourceTypes: Open Access Repository
DeliveryMethod fulltext_linktorsrc
Discipline Journalism & Communications
Engineering
Statistics
EISSN 1557-9611
EndPage 18
ExternalDocumentID 10.1109/tbc.2023.3236528
10_1109_TBC_2023_3236528
10042073
Genre orig-research
GrantInformation_xml – fundername: National Telecommunications and Information Administration (NTIA), Institute for Telecommunication Sciences (ITS)
GroupedDBID -~X
.DC
0R~
29I
4.4
5GY
6IK
97E
AAJGR
AARMG
AASAJ
AAWTH
ABAZT
ABQJQ
ABVLG
ACGFO
ACGFS
ACIWK
ACNCT
AENEX
AGQYO
AHBIQ
AKJIK
AKQYR
ALMA_UNASSIGNED_HOLDINGS
ASUFR
ATWAV
BEFXN
BFFAM
BGNUA
BKEBE
BPEOZ
CS3
DU5
EBS
ESBDL
HZ~
IFIPE
IPLJI
JAVBF
LAI
M43
MS~
O9-
OCL
P2P
RIA
RIE
RNS
TAE
TN5
5VS
AAYXX
AETIX
AGSQL
AI.
AIBXA
ALLEH
CITATION
EJD
H~9
IAAWW
IBMZZ
ICLAB
IDIHD
IFJZH
VH1
VJK
7SP
8FD
L7M
ADTOC
UNPAY
ID FETCH-LOGICAL-c334t-4f0d3572ad874997ceff59384480f073bf574056f057f2e77f70df3f59c3140c3
IEDL.DBID UNPAY
ISSN 0018-9316
1557-9611
IngestDate Wed Oct 01 16:20:30 EDT 2025
Mon Jun 30 08:42:23 EDT 2025
Wed Oct 01 03:26:46 EDT 2025
Thu Apr 24 23:00:01 EDT 2025
Wed Aug 27 02:55:38 EDT 2025
IsDoiOpenAccess true
IsOpenAccess true
IsPeerReviewed true
IsScholarly true
Issue 2
Language English
License https://creativecommons.org/licenses/by/4.0/legalcode
cc-by
LinkModel DirectLink
MergedId FETCHMERGED-LOGICAL-c334t-4f0d3572ad874997ceff59384480f073bf574056f057f2e77f70df3f59c3140c3
Notes ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ORCID 0000-0003-3369-4726
OpenAccessLink https://proxy.k.utb.cz/login?url=https://ieeexplore.ieee.org/ielx7/11/4359969/10042073.pdf
PQID 2823186471
PQPubID 85473
PageCount 18
ParticipantIDs proquest_journals_2823186471
crossref_citationtrail_10_1109_TBC_2023_3236528
crossref_primary_10_1109_TBC_2023_3236528
unpaywall_primary_10_1109_tbc_2023_3236528
ieee_primary_10042073
ProviderPackageCode CITATION
AAYXX
PublicationCentury 2000
PublicationDate 2023-06-01
PublicationDateYYYYMMDD 2023-06-01
PublicationDate_xml – month: 06
  year: 2023
  text: 2023-06-01
  day: 01
PublicationDecade 2020
PublicationPlace New York
PublicationPlace_xml – name: New York
PublicationTitle IEEE transactions on broadcasting
PublicationTitleAbbrev TBC
PublicationYear 2023
Publisher IEEE
The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Publisher_xml – name: IEEE
– name: The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
References ref13
ref35
ref12
ref34
(ref25) 1998
cermak (ref3) 1994
ref15
ref14
webster (ref4) 1994
ref31
webster (ref22) 2009
pinson (ref1) 2020
ref11
ref33
ref10
ref32
pinson (ref5) 2008
(ref2) 2020
ref17
pinson (ref39) 2019
ref16
(ref20) 2002
webster (ref26) 2008
rohaly (ref19) 2000
corriveau (ref18) 1999
lee (ref24) 2014
pinson (ref37) 2019
pinson (ref36) 2018
ref41
pinson (ref38) 2019
ref27
ref29
ref8
ref7
(ref30) 2022
ref9
ref6
webster (ref23) 2010
ref40
corriveau (ref21) 2003
pinson (ref28) 2011
References_xml – ident: ref10
  doi: 10.1109/JSTSP.2012.2215306
– ident: ref27
  doi: 10.1109/MSP.2011.942470
– year: 2000
  ident: ref19
  article-title: Final report from the video quality experts group on the validation of objective models of video quality assessment
– year: 2002
  ident: ref20
  publication-title: FR-TV Full-reference television phase II subjective test plan
– ident: ref8
  doi: 10.1109/TBC.2010.2086750
– ident: ref40
  doi: 10.1109/QoMEX48832.2020.9123115
– year: 1994
  ident: ref4
  publication-title: Two criteria for video test scene selection
– ident: ref15
  doi: 10.1109/ICIP.2002.1038897
– ident: ref14
  doi: 10.1145/3427931
– year: 2018
  ident: ref36
  publication-title: ITS4S A video quality dataset with four-second unrepeated scenes
– year: 1998
  ident: ref25
  publication-title: ITU-T coded-speech database
– year: 1999
  ident: ref18
  publication-title: VQEG subjective test plan full reference phase 1
– year: 2011
  ident: ref28
  publication-title: Preliminary investigation into the impact of audiovisual synchronization of impaired audiovisual sequences
– ident: ref29
  doi: 10.1109/THS.2007.370025
– ident: ref34
  doi: 10.2352/ISSN.2470-1173.2016.16.HVEI-124
– year: 2003
  ident: ref21
  article-title: Final report from the video quality experts group on the validation of objective models of video quality assessment, phase II
– ident: ref16
  doi: 10.1109/QoMEX.2016.7498936
– ident: ref32
  doi: 10.1016/j.image.2015.05.001
– ident: ref9
  doi: 10.1109/QoMEX.2011.6065690
– ident: ref12
  doi: 10.1109/JSTSP.2016.2638681
– year: 2008
  ident: ref5
  article-title: Techniques for evaluating objective video quality models using overlapping subjective datasets
– ident: ref41
  doi: 10.1109/TMM.2015.2484963
– ident: ref13
  doi: 10.1111/j.1467-8659.2012.03188.x
– ident: ref17
  doi: 10.1109/ACCESS.2021.3127395
– year: 2019
  ident: ref38
  publication-title: ITS4S3 A video quality dataset with unrepeated videos camera impairments and public safety scenarios
– year: 2019
  ident: ref39
  publication-title: ITS4S4 A video quality study of camera pans
– ident: ref7
  doi: 10.1109/QOMEX.2010.5517948
– ident: ref35
  doi: 10.3390/jimaging6030007
– ident: ref33
  doi: 10.1007/s41233-019-0026-4
– ident: ref6
  doi: 10.1109/TBC.2009.2034511
– ident: ref11
  doi: 10.1109/ICIP.2016.7532325
– year: 2022
  ident: ref30
  publication-title: Public safety #2 (PS2) video quality dataset
– year: 2019
  ident: ref37
  publication-title: ITS4S2 An image quality dataset with unrepeated images from consumer cameras
– year: 2020
  ident: ref1
  article-title: Confidence intervals for subjective tests and objective metrics that assess image, video, speech, or audiovisual quality
– year: 2010
  ident: ref23
  article-title: Report of the validation of video quality models for high definition video content
– year: 2009
  ident: ref22
  publication-title: Validation of Reduced-Reference and No-Reference Objective Models for Standard Definition Television Phase i
– year: 2020
  ident: ref2
  publication-title: NR metric framework
– year: 2008
  ident: ref26
  article-title: Final report from the video quality experts group on the validation of objective models of multimedia quality assessment, phase 1
– year: 1994
  ident: ref3
  publication-title: T1A1 5/94-148 Correlation of objective and subjective measures of video quality
– ident: ref31
  doi: 10.1109/BMSB.2016.7521936
– year: 2014
  ident: ref24
  article-title: Hybrid perceptual/bitstream validation test final report
SSID ssj0003002
Score 2.373185
Snippet This paper calculates confidence intervals for 89 datasets that use the 5-level Absolute Category Rating (ACR) method to evaluate the quality of speech, video,...
SourceID unpaywall
proquest
crossref
ieee
SourceType Open Access Repository
Aggregation Database
Enrichment Source
Index Database
Publisher
StartPage 1
SubjectTerms Algorithms
Analysis of variance
Audio data
Audiovisual quality
Confidence
confidence interval
Confidence intervals
confusion matrix
Correlation
Datasets
Decisions
false ranking
Image quality
Mathematical analysis
Measurement
Media
metric
MOS
precision
Statistical analysis
Statistical methods
statistics
subjective test
Telecommunications
video quality
SummonAdditionalLinks – databaseName: IEEE Electronic Library (IEL)
  dbid: RIE
  link: http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwjV3fS-QwEB7UF_Xh7lwV99QjDyIotJs226a5t3M5EUHxQcG3kuYHiGt3sV2Ovb_-Jk237CoevhU6aRpmksxkMt8HcJIOqdKC6yATmTutMiKQBTVBIWXKeFRkkXSFwje36dXD8PoxeWyL1ZtaGGNMc_nMhO6xyeXriZq5o7KBQzeL0SbXYZ1z4Yu1umWXUeqhwSOcwSzqcpJUDO4vRqGjCQ9ZzNLEEa8v7UENqcqKf7k5K6dy_keOx0tbzeVXuF38pL9h8hzO6iJUf9_gN356FN_gS-t0kl_eSnZgzZQ92F6CIuzBYSv0VL2QU7JSNlL1YMv5pB7SeRcsmha5e23JeYgsNUE3Htd0D_k9JxNLXP5HEg_QMSejjuyw-omvulPJqmmM6yzpvo-_edOQWld78HD5-350FbR0DYFibFgHQ0s1S3gsdcYxjuLKWJsIlmEASC2Ot7AJR_cwtegi2thwbjnVlqGMYhjmKbYPG-WkNAdAEpYIKgy3GHyix6SljiPHfBVFqaJayj4MFgrMVYtl7ig1xnkT01CRo8pzp_K8VXkfzroWU4_j8R_ZPae1JTmvsD4cLYwkb2d6lccuj5qluMf34bwznHd91IVa6eP7B30cwpYT85fRjmCjfp2ZY3R76uJHY-7_ADWe_Hw
  priority: 102
  providerName: IEEE
Title The Precision and Repeatability of Media Quality Comparisons: Measurements and New Statistical Methods
URI https://ieeexplore.ieee.org/document/10042073
https://www.proquest.com/docview/2823186471
https://ieeexplore.ieee.org/ielx7/11/4359969/10042073.pdf
UnpaywallVersion publishedVersion
Volume 69
hasFullText 1
inHoldings 1
isFullTextHit
isPrint
journalDatabaseRights – providerCode: PRVIEE
  databaseName: IEEE Electronic Library (IEL)
  customDbUrl:
  eissn: 1557-9611
  dateEnd: 99991231
  omitProxy: false
  ssIdentifier: ssj0003002
  issn: 1557-9611
  databaseCode: RIE
  dateStart: 19630101
  isFulltext: true
  titleUrlDefault: https://ieeexplore.ieee.org/
  providerName: IEEE
link http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwnV3fa9swED629GHbw350HcvaFT2MwQa2JSuWrL11YaUMWvrQQLcXI1sSlGZuqB3a7K_fSXaCs8Fge3PwGdu5k-47n_R9AO_EhFZGSRPlKvdfq6yKdEltVGotuGRlzrTfKHx6Jk5mk6-X2eVA6ssrnITFZzb2h6GXf2Xn9zJhLMHUjvBcJZ7lLMXYjBfGPYQd4XtLI9iZnZ0ffetmXhzEPMieYrqUkRKMrVuUVCVt6bkLUx7zlIvM67APUlLQWNmCm4-W9UKv7vR8Psg8x8_g-_qZuwUn1_GyLePq5290jv_1Us_haY9HyVEXQC_gga134cmApXAX9nujq-YHeU-2dpQ0L8FhoJHz216qh-jaEAT1OMN3BOArcuOI7wZp0tF1rMh0I33YfMJTm2-UTbgYZ13iEXAgkMYnOw0S180ezI6_XExPol68Iao4n7TRxFHDM5lqk0usqmRlncsUz7EcpA5fs3SZRLAoHAJGl1opnaTGcbSpOBZ9FX8Fo_qmtq-BZPgPUWWlw1IU8ZPRJmVeB4sxUVGj9RiStf-Kqmc29wIb8yJUOFQVF5-nhfd40Xt8DB82Vyw6Vo-_2O55pw3sOj-N4WAdI0U_7psi9V3VXGDGH8PHTdz8cQ-MwK17vPkX43147H9269UOYNTeLu1bREZteRi2Lx72I-EXErMF2g
linkProvider Unpaywall
linkToHtml http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwjV3dT9swED_x8QA8bKOA6GCbHxDSJiV14iZO9rZVoA5otYci8RY5_pDQuhSRVFP31-8cp1HLBOItUs5xrDvbdz7f7wdwFvepVClXXpIm9rRKp57IqfZyIWLGgzwJhC0UHo3j4W3_6i66a4rV61oYrXV9-Uz79rHO5auZnNujsp5FNwvRJjdhO8KwgrtyrXbhZZQ6cPAA5zAL2qwkTXuT7wPfEoX7LGRxZKnXV3ahmlZlzcPcmRcPYvFHTKcrm83lWxgvf9PdMfnlz6vcl3-fIDi-ehzv4E3jdpJvzk72YUMXHdhbASPswEkjdF_-JudkrXCk7MCu9UodqPMBGDQu8vOxoecholAEHXlc1R3o94LMDLEZIEEcRMeCDFq6w_IrvmrPJcu6Ma60pP0-_uaoprUuD-H28mIyGHoNYYMnGetXXt9QxSIeCpVwjKS41MZEKUtQV9TgeHMTcXQQY4NOogk154ZTZRjKSIaBnmRHsFXMCn0MJGJRSlPNDYaf6DMpocLAcl8FQSypEqILvaUCM9mgmVtSjWlWRzU0zVDlmVV51qi8C5_bFg8OyeMF2UOrtRU5p7AunC6NJGvmepmFNpOaxLjLd-FLazj_9VHlcq2P98_08Ql2hpPRTXbzY3x9Aru2ibuadgpb1eNcf0AnqMo_1qb_D50M_80
linkToUnpaywall http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwnV1Na9wwEB3SzaHtIU3TlGy-0KEUWrAtWWvJyi1dGkIgIYcspL0Y2ZIgdOMssZd2--szsr2Lt4FCe7PxGH_MSPPGI78H8EGMaGGUNEGqUv-1yqpA59QGudaCS5anTPsfhS-vxPlkdHGb3PakvrzCSbP4zIZ-s-nl39npLxkxFmFqR3iuIs9yFmNshjPjXsCm8L2lAWxOrq5Pv7UzLw5i3sieYrqUgRKMLVuUVEV17rkLYx7ymIvE67D3UlKjsbIGN1_Oy5le_NTTaS_znL2B78t7bhec_AjndR4Wv_-gc_yvh9qGrQ6PktM2gN7Chi134HWPpXAHDjqju-qefCRrf5RU78BhoJHrx06qh-jSEAT1OMO3BOAL8uCI7wZp0tJ1LMh4JX1YneCh1TfKqjkZZ13iEXBDII13dtlIXFe7MDn7ejM-DzrxhqDgfFQHI0cNT2SsTSqxqpKFdS5RPMVykDp8zNwlEsGicAgYXWyldJIax9Gm4Fj0Ffw9DMqH0u4BSfANUWWlw1IU8ZPRJmZeB4sxUVCj9RCipf-yomM29wIb06ypcKjKbr6MM-_xrPP4ED6tzpi1rB5_sd31TuvZtX4awuEyRrJu3FdZ7LuqqcCMP4TPq7h5dg2MwLVr7P-L8QG88rvterVDGNSPc3uEyKjOj7sx8AT1GQTZ
openUrl ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+Precision+and+Repeatability+of+Media+Quality+Comparisons%3A+Measurements+and+New+Statistical+Methods&rft.jtitle=IEEE+transactions+on+broadcasting&rft.au=Pinson%2C+Margaret+H.&rft.date=2023-06-01&rft.pub=IEEE&rft.issn=0018-9316&rft.spage=1&rft.epage=18&rft_id=info:doi/10.1109%2FTBC.2023.3236528&rft.externalDocID=10042073
thumbnail_l http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=0018-9316&client=summon
thumbnail_m http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=0018-9316&client=summon
thumbnail_s http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=0018-9316&client=summon