The Precision and Repeatability of Media Quality Comparisons: Measurements and New Statistical Methods
This paper calculates confidence intervals for 89 datasets that use the 5-level Absolute Category Rating (ACR) method to evaluate the quality of speech, video, images, and video with audio. This data allows us to compute the subjective test confidence interval <inline-formula> <tex-math not...
Saved in:
Published in | IEEE transactions on broadcasting Vol. 69; no. 2; pp. 1 - 18 |
---|---|
Main Author | |
Format | Journal Article |
Language | English |
Published |
New York
IEEE
01.06.2023
The Institute of Electrical and Electronics Engineers, Inc. (IEEE) |
Subjects | |
Online Access | Get full text |
ISSN | 0018-9316 1557-9611 1557-9611 |
DOI | 10.1109/TBC.2023.3236528 |
Cover
Abstract | This paper calculates confidence intervals for 89 datasets that use the 5-level Absolute Category Rating (ACR) method to evaluate the quality of speech, video, images, and video with audio. This data allows us to compute the subjective test confidence interval <inline-formula> <tex-math notation="LaTeX">(\Delta </tex-math> </inline-formula>SCI) for 5-level ACR tests. We use a confusion matrix to compare conclusions reached by 88 lab-to-lab comparisons, 22 method-to-method comparisons, and 12 comparisons between expert and naïve subjects. We estimate the differences in conclusions reached by ad hoc evaluations, compared to subjective tests. We recommend using the disagree incidence rate to identify lab-to-lab differences (i.e., the likelihood that significantly different stimulus pairs receive opposing rank order from the two labs). Disagree incidence rates above 0.31% are unusual enough to warrant investigation and disagree incidence rates above 1.0% indicate differences in method, test environment, test implementation, or subject demographics. These incidence rates form the basis for a new statistical method that calculates the confidence interval of a metric (<inline-formula> <tex-math notation="LaTeX">\Delta </tex-math> </inline-formula>MCI). When <inline-formula> <tex-math notation="LaTeX">\Delta </tex-math> </inline-formula>MCI is used to make decisions, the equivalence to a video-quality test (EVQT) method determines whether a metric acts similarly to a subjective test. When <inline-formula> <tex-math notation="LaTeX">\Delta </tex-math> </inline-formula>MCI is not used, the metric is likened to a certain number of people in a video-quality test (PVQT). This information will help users make the better decisions when applying quality metrics. The algorithm code is made available for any purpose. Most of the ratings used in this paper come from open datasets. |
---|---|
AbstractList | This paper calculates confidence intervals for 89 datasets that use the 5-level Absolute Category Rating (ACR) method to evaluate the quality of speech, video, images, and video with audio. This data allows us to compute the subjective test confidence interval ([Formula Omitted]) for 5-level ACR tests. We use a confusion matrix to compare conclusions reached by 88 lab-to-lab comparisons, 22 method-to-method comparisons, and 12 comparisons between expert and naïve subjects. We estimate the differences in conclusions reached by ad hoc evaluations, compared to subjective tests. We recommend using the disagree incidence rate to identify lab-to-lab differences (i.e., the likelihood that significantly different stimulus pairs receive opposing rank order from the two labs). Disagree incidence rates above 0.31% are unusual enough to warrant investigation and disagree incidence rates above 1.0% indicate differences in method, test environment, test implementation, or subject demographics. These incidence rates form the basis for a new statistical method that calculates the confidence interval of a metric ([Formula Omitted]). When [Formula Omitted] is used to make decisions, the equivalence to a video-quality test (EVQT) method determines whether a metric acts similarly to a subjective test. When [Formula Omitted] is not used, the metric is likened to a certain number of people in a video-quality test (PVQT). This information will help users make the better decisions when applying quality metrics. The algorithm code is made available for any purpose. Most of the ratings used in this paper come from open datasets. This paper calculates confidence intervals for 89 datasets that use the 5-level Absolute Category Rating (ACR) method to evaluate the quality of speech, video, images, and video with audio. This data allows us to compute the subjective test confidence interval <inline-formula> <tex-math notation="LaTeX">(\Delta </tex-math> </inline-formula>SCI) for 5-level ACR tests. We use a confusion matrix to compare conclusions reached by 88 lab-to-lab comparisons, 22 method-to-method comparisons, and 12 comparisons between expert and naïve subjects. We estimate the differences in conclusions reached by ad hoc evaluations, compared to subjective tests. We recommend using the disagree incidence rate to identify lab-to-lab differences (i.e., the likelihood that significantly different stimulus pairs receive opposing rank order from the two labs). Disagree incidence rates above 0.31% are unusual enough to warrant investigation and disagree incidence rates above 1.0% indicate differences in method, test environment, test implementation, or subject demographics. These incidence rates form the basis for a new statistical method that calculates the confidence interval of a metric (<inline-formula> <tex-math notation="LaTeX">\Delta </tex-math> </inline-formula>MCI). When <inline-formula> <tex-math notation="LaTeX">\Delta </tex-math> </inline-formula>MCI is used to make decisions, the equivalence to a video-quality test (EVQT) method determines whether a metric acts similarly to a subjective test. When <inline-formula> <tex-math notation="LaTeX">\Delta </tex-math> </inline-formula>MCI is not used, the metric is likened to a certain number of people in a video-quality test (PVQT). This information will help users make the better decisions when applying quality metrics. The algorithm code is made available for any purpose. Most of the ratings used in this paper come from open datasets. |
Author | Pinson, Margaret H. |
Author_xml | – sequence: 1 givenname: Margaret H. orcidid: 0000-0003-3369-4726 surname: Pinson fullname: Pinson, Margaret H. organization: National Telecommunications and Information Administration, Institute for Telecommunication Sciences, Boulder, CO, USA |
BookMark | eNptkc9P2zAYhq0JpBW2-w47RNo55fOPxAm3rdoACQaMcrZc57MwSu3MdlT1v19KEULVTpbs53n1fa9PyJEPHgn5QmFOKbRnyx-LOQPG55zxumLNBzKjVSXLtqb0iMwAaFO2nNYfyUlKzwDAAdiM2OUTFncRjUsu-EL7rviDA-qsV653eVsEW9xg53RxP-qXi0VYDzq6FHw6n550GiOu0ef0Iv_GTfGQdXYpO6P7CchPoUufyLHVfcLPr-cpefz1c7m4LK9vL64W369Lw7nIpbDQ8Uoy3TVStK00aG3V8kaIBixIvrKVFFDVFippGUppJXSWT4zhVIDhp4Tuc0c_6O1G970aolvruFUU1K4olVdG7YpSr0VNzre9M8Twd8SU1XMYo5_GVKxhnDa1kHSi6j1lYkgpolXG7fYMPkft-rf46R8O4-FAPJzoP8rXveIQ8R0Ogk0l8H8gE5Mc |
CODEN | IETBAC |
CitedBy_id | crossref_primary_10_1016_j_image_2024_117176 crossref_primary_10_1109_ACCESS_2025_3544307 crossref_primary_10_1109_TSC_2024_3451215 |
Cites_doi | 10.1109/JSTSP.2012.2215306 10.1109/MSP.2011.942470 10.1109/TBC.2010.2086750 10.1109/QoMEX48832.2020.9123115 10.1109/ICIP.2002.1038897 10.1145/3427931 10.1109/THS.2007.370025 10.2352/ISSN.2470-1173.2016.16.HVEI-124 10.1109/QoMEX.2016.7498936 10.1016/j.image.2015.05.001 10.1109/QoMEX.2011.6065690 10.1109/JSTSP.2016.2638681 10.1109/TMM.2015.2484963 10.1111/j.1467-8659.2012.03188.x 10.1109/ACCESS.2021.3127395 10.1109/QOMEX.2010.5517948 10.3390/jimaging6030007 10.1007/s41233-019-0026-4 10.1109/TBC.2009.2034511 10.1109/ICIP.2016.7532325 10.1109/BMSB.2016.7521936 |
ContentType | Journal Article |
Copyright | Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2023 |
Copyright_xml | – notice: Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2023 |
DBID | 97E ESBDL RIA RIE AAYXX CITATION 7SP 8FD L7M ADTOC UNPAY |
DOI | 10.1109/TBC.2023.3236528 |
DatabaseName | IEEE Xplore (IEEE) IEEE Xplore Open Access Journals IEEE All-Society Periodicals Package (ASPP) 1998–Present IEEE Electronic Library (IEL) CrossRef Electronics & Communications Abstracts Technology Research Database Advanced Technologies Database with Aerospace Unpaywall for CDI: Periodical Content Unpaywall |
DatabaseTitle | CrossRef Technology Research Database Advanced Technologies Database with Aerospace Electronics & Communications Abstracts |
DatabaseTitleList | Technology Research Database |
Database_xml | – sequence: 1 dbid: RIE name: IEEE Electronic Library (IEL) url: https://proxy.k.utb.cz/login?url=https://ieeexplore.ieee.org/ sourceTypes: Publisher – sequence: 2 dbid: UNPAY name: Unpaywall url: https://proxy.k.utb.cz/login?url=https://unpaywall.org/ sourceTypes: Open Access Repository |
DeliveryMethod | fulltext_linktorsrc |
Discipline | Journalism & Communications Engineering Statistics |
EISSN | 1557-9611 |
EndPage | 18 |
ExternalDocumentID | 10.1109/tbc.2023.3236528 10_1109_TBC_2023_3236528 10042073 |
Genre | orig-research |
GrantInformation_xml | – fundername: National Telecommunications and Information Administration (NTIA), Institute for Telecommunication Sciences (ITS) |
GroupedDBID | -~X .DC 0R~ 29I 4.4 5GY 6IK 97E AAJGR AARMG AASAJ AAWTH ABAZT ABQJQ ABVLG ACGFO ACGFS ACIWK ACNCT AENEX AGQYO AHBIQ AKJIK AKQYR ALMA_UNASSIGNED_HOLDINGS ASUFR ATWAV BEFXN BFFAM BGNUA BKEBE BPEOZ CS3 DU5 EBS ESBDL HZ~ IFIPE IPLJI JAVBF LAI M43 MS~ O9- OCL P2P RIA RIE RNS TAE TN5 5VS AAYXX AETIX AGSQL AI. AIBXA ALLEH CITATION EJD H~9 IAAWW IBMZZ ICLAB IDIHD IFJZH VH1 VJK 7SP 8FD L7M ADTOC UNPAY |
ID | FETCH-LOGICAL-c334t-4f0d3572ad874997ceff59384480f073bf574056f057f2e77f70df3f59c3140c3 |
IEDL.DBID | UNPAY |
ISSN | 0018-9316 1557-9611 |
IngestDate | Wed Oct 01 16:20:30 EDT 2025 Mon Jun 30 08:42:23 EDT 2025 Wed Oct 01 03:26:46 EDT 2025 Thu Apr 24 23:00:01 EDT 2025 Wed Aug 27 02:55:38 EDT 2025 |
IsDoiOpenAccess | true |
IsOpenAccess | true |
IsPeerReviewed | true |
IsScholarly | true |
Issue | 2 |
Language | English |
License | https://creativecommons.org/licenses/by/4.0/legalcode cc-by |
LinkModel | DirectLink |
MergedId | FETCHMERGED-LOGICAL-c334t-4f0d3572ad874997ceff59384480f073bf574056f057f2e77f70df3f59c3140c3 |
Notes | ObjectType-Article-1 SourceType-Scholarly Journals-1 ObjectType-Feature-2 content type line 14 |
ORCID | 0000-0003-3369-4726 |
OpenAccessLink | https://proxy.k.utb.cz/login?url=https://ieeexplore.ieee.org/ielx7/11/4359969/10042073.pdf |
PQID | 2823186471 |
PQPubID | 85473 |
PageCount | 18 |
ParticipantIDs | proquest_journals_2823186471 crossref_citationtrail_10_1109_TBC_2023_3236528 crossref_primary_10_1109_TBC_2023_3236528 unpaywall_primary_10_1109_tbc_2023_3236528 ieee_primary_10042073 |
ProviderPackageCode | CITATION AAYXX |
PublicationCentury | 2000 |
PublicationDate | 2023-06-01 |
PublicationDateYYYYMMDD | 2023-06-01 |
PublicationDate_xml | – month: 06 year: 2023 text: 2023-06-01 day: 01 |
PublicationDecade | 2020 |
PublicationPlace | New York |
PublicationPlace_xml | – name: New York |
PublicationTitle | IEEE transactions on broadcasting |
PublicationTitleAbbrev | TBC |
PublicationYear | 2023 |
Publisher | IEEE The Institute of Electrical and Electronics Engineers, Inc. (IEEE) |
Publisher_xml | – name: IEEE – name: The Institute of Electrical and Electronics Engineers, Inc. (IEEE) |
References | ref13 ref35 ref12 ref34 (ref25) 1998 cermak (ref3) 1994 ref15 ref14 webster (ref4) 1994 ref31 webster (ref22) 2009 pinson (ref1) 2020 ref11 ref33 ref10 ref32 pinson (ref5) 2008 (ref2) 2020 ref17 pinson (ref39) 2019 ref16 (ref20) 2002 webster (ref26) 2008 rohaly (ref19) 2000 corriveau (ref18) 1999 lee (ref24) 2014 pinson (ref37) 2019 pinson (ref36) 2018 ref41 pinson (ref38) 2019 ref27 ref29 ref8 ref7 (ref30) 2022 ref9 ref6 webster (ref23) 2010 ref40 corriveau (ref21) 2003 pinson (ref28) 2011 |
References_xml | – ident: ref10 doi: 10.1109/JSTSP.2012.2215306 – ident: ref27 doi: 10.1109/MSP.2011.942470 – year: 2000 ident: ref19 article-title: Final report from the video quality experts group on the validation of objective models of video quality assessment – year: 2002 ident: ref20 publication-title: FR-TV Full-reference television phase II subjective test plan – ident: ref8 doi: 10.1109/TBC.2010.2086750 – ident: ref40 doi: 10.1109/QoMEX48832.2020.9123115 – year: 1994 ident: ref4 publication-title: Two criteria for video test scene selection – ident: ref15 doi: 10.1109/ICIP.2002.1038897 – ident: ref14 doi: 10.1145/3427931 – year: 2018 ident: ref36 publication-title: ITS4S A video quality dataset with four-second unrepeated scenes – year: 1998 ident: ref25 publication-title: ITU-T coded-speech database – year: 1999 ident: ref18 publication-title: VQEG subjective test plan full reference phase 1 – year: 2011 ident: ref28 publication-title: Preliminary investigation into the impact of audiovisual synchronization of impaired audiovisual sequences – ident: ref29 doi: 10.1109/THS.2007.370025 – ident: ref34 doi: 10.2352/ISSN.2470-1173.2016.16.HVEI-124 – year: 2003 ident: ref21 article-title: Final report from the video quality experts group on the validation of objective models of video quality assessment, phase II – ident: ref16 doi: 10.1109/QoMEX.2016.7498936 – ident: ref32 doi: 10.1016/j.image.2015.05.001 – ident: ref9 doi: 10.1109/QoMEX.2011.6065690 – ident: ref12 doi: 10.1109/JSTSP.2016.2638681 – year: 2008 ident: ref5 article-title: Techniques for evaluating objective video quality models using overlapping subjective datasets – ident: ref41 doi: 10.1109/TMM.2015.2484963 – ident: ref13 doi: 10.1111/j.1467-8659.2012.03188.x – ident: ref17 doi: 10.1109/ACCESS.2021.3127395 – year: 2019 ident: ref38 publication-title: ITS4S3 A video quality dataset with unrepeated videos camera impairments and public safety scenarios – year: 2019 ident: ref39 publication-title: ITS4S4 A video quality study of camera pans – ident: ref7 doi: 10.1109/QOMEX.2010.5517948 – ident: ref35 doi: 10.3390/jimaging6030007 – ident: ref33 doi: 10.1007/s41233-019-0026-4 – ident: ref6 doi: 10.1109/TBC.2009.2034511 – ident: ref11 doi: 10.1109/ICIP.2016.7532325 – year: 2022 ident: ref30 publication-title: Public safety #2 (PS2) video quality dataset – year: 2019 ident: ref37 publication-title: ITS4S2 An image quality dataset with unrepeated images from consumer cameras – year: 2020 ident: ref1 article-title: Confidence intervals for subjective tests and objective metrics that assess image, video, speech, or audiovisual quality – year: 2010 ident: ref23 article-title: Report of the validation of video quality models for high definition video content – year: 2009 ident: ref22 publication-title: Validation of Reduced-Reference and No-Reference Objective Models for Standard Definition Television Phase i – year: 2020 ident: ref2 publication-title: NR metric framework – year: 2008 ident: ref26 article-title: Final report from the video quality experts group on the validation of objective models of multimedia quality assessment, phase 1 – year: 1994 ident: ref3 publication-title: T1A1 5/94-148 Correlation of objective and subjective measures of video quality – ident: ref31 doi: 10.1109/BMSB.2016.7521936 – year: 2014 ident: ref24 article-title: Hybrid perceptual/bitstream validation test final report |
SSID | ssj0003002 |
Score | 2.373185 |
Snippet | This paper calculates confidence intervals for 89 datasets that use the 5-level Absolute Category Rating (ACR) method to evaluate the quality of speech, video,... |
SourceID | unpaywall proquest crossref ieee |
SourceType | Open Access Repository Aggregation Database Enrichment Source Index Database Publisher |
StartPage | 1 |
SubjectTerms | Algorithms Analysis of variance Audio data Audiovisual quality Confidence confidence interval Confidence intervals confusion matrix Correlation Datasets Decisions false ranking Image quality Mathematical analysis Measurement Media metric MOS precision Statistical analysis Statistical methods statistics subjective test Telecommunications video quality |
SummonAdditionalLinks | – databaseName: IEEE Electronic Library (IEL) dbid: RIE link: http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwjV3fS-QwEB7UF_Xh7lwV99QjDyIotJs226a5t3M5EUHxQcG3kuYHiGt3sV2Ovb_-Jk237CoevhU6aRpmksxkMt8HcJIOqdKC6yATmTutMiKQBTVBIWXKeFRkkXSFwje36dXD8PoxeWyL1ZtaGGNMc_nMhO6xyeXriZq5o7KBQzeL0SbXYZ1z4Yu1umWXUeqhwSOcwSzqcpJUDO4vRqGjCQ9ZzNLEEa8v7UENqcqKf7k5K6dy_keOx0tbzeVXuF38pL9h8hzO6iJUf9_gN356FN_gS-t0kl_eSnZgzZQ92F6CIuzBYSv0VL2QU7JSNlL1YMv5pB7SeRcsmha5e23JeYgsNUE3Htd0D_k9JxNLXP5HEg_QMSejjuyw-omvulPJqmmM6yzpvo-_edOQWld78HD5-350FbR0DYFibFgHQ0s1S3gsdcYxjuLKWJsIlmEASC2Ot7AJR_cwtegi2thwbjnVlqGMYhjmKbYPG-WkNAdAEpYIKgy3GHyix6SljiPHfBVFqaJayj4MFgrMVYtl7ig1xnkT01CRo8pzp_K8VXkfzroWU4_j8R_ZPae1JTmvsD4cLYwkb2d6lccuj5qluMf34bwznHd91IVa6eP7B30cwpYT85fRjmCjfp2ZY3R76uJHY-7_ADWe_Hw priority: 102 providerName: IEEE |
Title | The Precision and Repeatability of Media Quality Comparisons: Measurements and New Statistical Methods |
URI | https://ieeexplore.ieee.org/document/10042073 https://www.proquest.com/docview/2823186471 https://ieeexplore.ieee.org/ielx7/11/4359969/10042073.pdf |
UnpaywallVersion | publishedVersion |
Volume | 69 |
hasFullText | 1 |
inHoldings | 1 |
isFullTextHit | |
isPrint | |
journalDatabaseRights | – providerCode: PRVIEE databaseName: IEEE Electronic Library (IEL) customDbUrl: eissn: 1557-9611 dateEnd: 99991231 omitProxy: false ssIdentifier: ssj0003002 issn: 1557-9611 databaseCode: RIE dateStart: 19630101 isFulltext: true titleUrlDefault: https://ieeexplore.ieee.org/ providerName: IEEE |
link | http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwnV3fa9swED629GHbw350HcvaFT2MwQa2JSuWrL11YaUMWvrQQLcXI1sSlGZuqB3a7K_fSXaCs8Fge3PwGdu5k-47n_R9AO_EhFZGSRPlKvdfq6yKdEltVGotuGRlzrTfKHx6Jk5mk6-X2eVA6ssrnITFZzb2h6GXf2Xn9zJhLMHUjvBcJZ7lLMXYjBfGPYQd4XtLI9iZnZ0ffetmXhzEPMieYrqUkRKMrVuUVCVt6bkLUx7zlIvM67APUlLQWNmCm4-W9UKv7vR8Psg8x8_g-_qZuwUn1_GyLePq5290jv_1Us_haY9HyVEXQC_gga134cmApXAX9nujq-YHeU-2dpQ0L8FhoJHz216qh-jaEAT1OMN3BOArcuOI7wZp0tF1rMh0I33YfMJTm2-UTbgYZ13iEXAgkMYnOw0S180ezI6_XExPol68Iao4n7TRxFHDM5lqk0usqmRlncsUz7EcpA5fs3SZRLAoHAJGl1opnaTGcbSpOBZ9FX8Fo_qmtq-BZPgPUWWlw1IU8ZPRJmVeB4sxUVGj9RiStf-Kqmc29wIb8yJUOFQVF5-nhfd40Xt8DB82Vyw6Vo-_2O55pw3sOj-N4WAdI0U_7psi9V3VXGDGH8PHTdz8cQ-MwK17vPkX43147H9269UOYNTeLu1bREZteRi2Lx72I-EXErMF2g |
linkProvider | Unpaywall |
linkToHtml | http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwjV3dT9swED_x8QA8bKOA6GCbHxDSJiV14iZO9rZVoA5otYci8RY5_pDQuhSRVFP31-8cp1HLBOItUs5xrDvbdz7f7wdwFvepVClXXpIm9rRKp57IqfZyIWLGgzwJhC0UHo3j4W3_6i66a4rV61oYrXV9-Uz79rHO5auZnNujsp5FNwvRJjdhO8KwgrtyrXbhZZQ6cPAA5zAL2qwkTXuT7wPfEoX7LGRxZKnXV3ahmlZlzcPcmRcPYvFHTKcrm83lWxgvf9PdMfnlz6vcl3-fIDi-ehzv4E3jdpJvzk72YUMXHdhbASPswEkjdF_-JudkrXCk7MCu9UodqPMBGDQu8vOxoecholAEHXlc1R3o94LMDLEZIEEcRMeCDFq6w_IrvmrPJcu6Ma60pP0-_uaoprUuD-H28mIyGHoNYYMnGetXXt9QxSIeCpVwjKS41MZEKUtQV9TgeHMTcXQQY4NOogk154ZTZRjKSIaBnmRHsFXMCn0MJGJRSlPNDYaf6DMpocLAcl8FQSypEqILvaUCM9mgmVtSjWlWRzU0zVDlmVV51qi8C5_bFg8OyeMF2UOrtRU5p7AunC6NJGvmepmFNpOaxLjLd-FLazj_9VHlcq2P98_08Ql2hpPRTXbzY3x9Aru2ibuadgpb1eNcf0AnqMo_1qb_D50M_80 |
linkToUnpaywall | http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwnV1Na9wwEB3SzaHtIU3TlGy-0KEUWrAtWWvJyi1dGkIgIYcspL0Y2ZIgdOMssZd2--szsr2Lt4FCe7PxGH_MSPPGI78H8EGMaGGUNEGqUv-1yqpA59QGudaCS5anTPsfhS-vxPlkdHGb3PakvrzCSbP4zIZ-s-nl39npLxkxFmFqR3iuIs9yFmNshjPjXsCm8L2lAWxOrq5Pv7UzLw5i3sieYrqUgRKMLVuUVEV17rkLYx7ymIvE67D3UlKjsbIGN1_Oy5le_NTTaS_znL2B78t7bhec_AjndR4Wv_-gc_yvh9qGrQ6PktM2gN7Chi134HWPpXAHDjqju-qefCRrf5RU78BhoJHrx06qh-jSEAT1OMO3BOAL8uCI7wZp0tJ1LMh4JX1YneCh1TfKqjkZZ13iEXBDII13dtlIXFe7MDn7ejM-DzrxhqDgfFQHI0cNT2SsTSqxqpKFdS5RPMVykDp8zNwlEsGicAgYXWyldJIax9Gm4Fj0Ffw9DMqH0u4BSfANUWWlw1IU8ZPRJmZeB4sxUVCj9RCipf-yomM29wIb06ypcKjKbr6MM-_xrPP4ED6tzpi1rB5_sd31TuvZtX4awuEyRrJu3FdZ7LuqqcCMP4TPq7h5dg2MwLVr7P-L8QG88rvterVDGNSPc3uEyKjOj7sx8AT1GQTZ |
openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+Precision+and+Repeatability+of+Media+Quality+Comparisons%3A+Measurements+and+New+Statistical+Methods&rft.jtitle=IEEE+transactions+on+broadcasting&rft.au=Pinson%2C+Margaret+H.&rft.date=2023-06-01&rft.pub=IEEE&rft.issn=0018-9316&rft.spage=1&rft.epage=18&rft_id=info:doi/10.1109%2FTBC.2023.3236528&rft.externalDocID=10042073 |
thumbnail_l | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=0018-9316&client=summon |
thumbnail_m | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=0018-9316&client=summon |
thumbnail_s | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=0018-9316&client=summon |