The assessment of fundus image quality labeling reliability among graders with different backgrounds
For the training of machine learning (ML) algorithms, correctly labeled ground truth data are inevitable. In this pilot study, we assessed the performance of graders with different backgrounds in the labeling of retinal fundus image quality. Color fundus photographs were labeled using a Python-based...
Saved in:
| Published in | PloS one Vol. 17; no. 7; p. e0271156 |
|---|---|
| Main Authors | , , , |
| Format | Journal Article |
| Language | English |
| Published |
United States
Public Library of Science
26.07.2022
Public Library of Science (PLoS) |
| Subjects | |
| Online Access | Get full text |
| ISSN | 1932-6203 1932-6203 |
| DOI | 10.1371/journal.pone.0271156 |
Cover
| Abstract | For the training of machine learning (ML) algorithms, correctly labeled ground truth data are inevitable. In this pilot study, we assessed the performance of graders with different backgrounds in the labeling of retinal fundus image quality.
Color fundus photographs were labeled using a Python-based tool using four image categories: excellent (E), good (G), adequate (A) and insufficient for grading (I). We enrolled 8 subjects (4 with and 4 without medical background, groups M and NM, respectively) to whom a tutorial was presented on image quality requirements. We randomly selected 200 images from a pool of 18,145 expert-labeled images (50/E, 50/G, 50/A, 50/I). The performance of the grading was timed and the agreement was assessed. An additional grading round was performed with 14 labels for a more objective analysis.
The median time (interquartile range) for the labeling task with 4 categories was 987.8 sec (418.6) for all graders and 872.9 sec (621.0) vs. 1019.8 sec (479.5) in the M vs. NM groups, respectively. Cohen's weighted kappa showed moderate agreement (0.564) when using four categories that increased to substantial (0.637) when using only three by merging the E and G groups. By the use of 14 labels, the weighted kappa values were 0.594 and 0.667 when assigning four or three categories, respectively.
Image grading with a Python-based tool seems to be a simple yet possibly efficient solution for the labeling of fundus images according to image quality that does not necessarily require medical background. Such grading can be subject to variability but could still effectively serve the robust identification of images with insufficient quality. This emphasizes the opportunity for the democratization of ML-applications among persons with both medical and non-medical background. However, simplicity of the grading system is key to successful categorization. |
|---|---|
| AbstractList | Purpose For the training of machine learning (ML) algorithms, correctly labeled ground truth data are inevitable. In this pilot study, we assessed the performance of graders with different backgrounds in the labeling of retinal fundus image quality. Methods Color fundus photographs were labeled using a Python-based tool using four image categories: excellent (E), good (G), adequate (A) and insufficient for grading (I). We enrolled 8 subjects (4 with and 4 without medical background, groups M and NM, respectively) to whom a tutorial was presented on image quality requirements. We randomly selected 200 images from a pool of 18,145 expert-labeled images (50/E, 50/G, 50/A, 50/I). The performance of the grading was timed and the agreement was assessed. An additional grading round was performed with 14 labels for a more objective analysis. Results The median time (interquartile range) for the labeling task with 4 categories was 987.8 sec (418.6) for all graders and 872.9 sec (621.0) vs. 1019.8 sec (479.5) in the M vs. NM groups, respectively. Cohen's weighted kappa showed moderate agreement (0.564) when using four categories that increased to substantial (0.637) when using only three by merging the E and G groups. By the use of 14 labels, the weighted kappa values were 0.594 and 0.667 when assigning four or three categories, respectively. Conclusion Image grading with a Python-based tool seems to be a simple yet possibly efficient solution for the labeling of fundus images according to image quality that does not necessarily require medical background. Such grading can be subject to variability but could still effectively serve the robust identification of images with insufficient quality. This emphasizes the opportunity for the democratization of ML-applications among persons with both medical and non-medical background. However, simplicity of the grading system is key to successful categorization. Purpose For the training of machine learning (ML) algorithms, correctly labeled ground truth data are inevitable. In this pilot study, we assessed the performance of graders with different backgrounds in the labeling of retinal fundus image quality. Methods Color fundus photographs were labeled using a Python-based tool using four image categories: excellent (E), good (G), adequate (A) and insufficient for grading (I). We enrolled 8 subjects (4 with and 4 without medical background, groups M and NM, respectively) to whom a tutorial was presented on image quality requirements. We randomly selected 200 images from a pool of 18,145 expert-labeled images (50/E, 50/G, 50/A, 50/I). The performance of the grading was timed and the agreement was assessed. An additional grading round was performed with 14 labels for a more objective analysis. Results The median time (interquartile range) for the labeling task with 4 categories was 987.8 sec (418.6) for all graders and 872.9 sec (621.0) vs. 1019.8 sec (479.5) in the M vs. NM groups, respectively. Cohen’s weighted kappa showed moderate agreement (0.564) when using four categories that increased to substantial (0.637) when using only three by merging the E and G groups. By the use of 14 labels, the weighted kappa values were 0.594 and 0.667 when assigning four or three categories, respectively. Conclusion Image grading with a Python-based tool seems to be a simple yet possibly efficient solution for the labeling of fundus images according to image quality that does not necessarily require medical background. Such grading can be subject to variability but could still effectively serve the robust identification of images with insufficient quality. This emphasizes the opportunity for the democratization of ML-applications among persons with both medical and non-medical background. However, simplicity of the grading system is key to successful categorization. For the training of machine learning (ML) algorithms, correctly labeled ground truth data are inevitable. In this pilot study, we assessed the performance of graders with different backgrounds in the labeling of retinal fundus image quality.PURPOSEFor the training of machine learning (ML) algorithms, correctly labeled ground truth data are inevitable. In this pilot study, we assessed the performance of graders with different backgrounds in the labeling of retinal fundus image quality.Color fundus photographs were labeled using a Python-based tool using four image categories: excellent (E), good (G), adequate (A) and insufficient for grading (I). We enrolled 8 subjects (4 with and 4 without medical background, groups M and NM, respectively) to whom a tutorial was presented on image quality requirements. We randomly selected 200 images from a pool of 18,145 expert-labeled images (50/E, 50/G, 50/A, 50/I). The performance of the grading was timed and the agreement was assessed. An additional grading round was performed with 14 labels for a more objective analysis.METHODSColor fundus photographs were labeled using a Python-based tool using four image categories: excellent (E), good (G), adequate (A) and insufficient for grading (I). We enrolled 8 subjects (4 with and 4 without medical background, groups M and NM, respectively) to whom a tutorial was presented on image quality requirements. We randomly selected 200 images from a pool of 18,145 expert-labeled images (50/E, 50/G, 50/A, 50/I). The performance of the grading was timed and the agreement was assessed. An additional grading round was performed with 14 labels for a more objective analysis.The median time (interquartile range) for the labeling task with 4 categories was 987.8 sec (418.6) for all graders and 872.9 sec (621.0) vs. 1019.8 sec (479.5) in the M vs. NM groups, respectively. Cohen's weighted kappa showed moderate agreement (0.564) when using four categories that increased to substantial (0.637) when using only three by merging the E and G groups. By the use of 14 labels, the weighted kappa values were 0.594 and 0.667 when assigning four or three categories, respectively.RESULTSThe median time (interquartile range) for the labeling task with 4 categories was 987.8 sec (418.6) for all graders and 872.9 sec (621.0) vs. 1019.8 sec (479.5) in the M vs. NM groups, respectively. Cohen's weighted kappa showed moderate agreement (0.564) when using four categories that increased to substantial (0.637) when using only three by merging the E and G groups. By the use of 14 labels, the weighted kappa values were 0.594 and 0.667 when assigning four or three categories, respectively.Image grading with a Python-based tool seems to be a simple yet possibly efficient solution for the labeling of fundus images according to image quality that does not necessarily require medical background. Such grading can be subject to variability but could still effectively serve the robust identification of images with insufficient quality. This emphasizes the opportunity for the democratization of ML-applications among persons with both medical and non-medical background. However, simplicity of the grading system is key to successful categorization.CONCLUSIONImage grading with a Python-based tool seems to be a simple yet possibly efficient solution for the labeling of fundus images according to image quality that does not necessarily require medical background. Such grading can be subject to variability but could still effectively serve the robust identification of images with insufficient quality. This emphasizes the opportunity for the democratization of ML-applications among persons with both medical and non-medical background. However, simplicity of the grading system is key to successful categorization. For the training of machine learning (ML) algorithms, correctly labeled ground truth data are inevitable. In this pilot study, we assessed the performance of graders with different backgrounds in the labeling of retinal fundus image quality. Color fundus photographs were labeled using a Python-based tool using four image categories: excellent (E), good (G), adequate (A) and insufficient for grading (I). We enrolled 8 subjects (4 with and 4 without medical background, groups M and NM, respectively) to whom a tutorial was presented on image quality requirements. We randomly selected 200 images from a pool of 18,145 expert-labeled images (50/E, 50/G, 50/A, 50/I). The performance of the grading was timed and the agreement was assessed. An additional grading round was performed with 14 labels for a more objective analysis. The median time (interquartile range) for the labeling task with 4 categories was 987.8 sec (418.6) for all graders and 872.9 sec (621.0) vs. 1019.8 sec (479.5) in the M vs. NM groups, respectively. Cohen's weighted kappa showed moderate agreement (0.564) when using four categories that increased to substantial (0.637) when using only three by merging the E and G groups. By the use of 14 labels, the weighted kappa values were 0.594 and 0.667 when assigning four or three categories, respectively. Image grading with a Python-based tool seems to be a simple yet possibly efficient solution for the labeling of fundus images according to image quality that does not necessarily require medical background. Such grading can be subject to variability but could still effectively serve the robust identification of images with insufficient quality. This emphasizes the opportunity for the democratization of ML-applications among persons with both medical and non-medical background. However, simplicity of the grading system is key to successful categorization. PurposeFor the training of machine learning (ML) algorithms, correctly labeled ground truth data are inevitable. In this pilot study, we assessed the performance of graders with different backgrounds in the labeling of retinal fundus image quality.MethodsColor fundus photographs were labeled using a Python-based tool using four image categories: excellent (E), good (G), adequate (A) and insufficient for grading (I). We enrolled 8 subjects (4 with and 4 without medical background, groups M and NM, respectively) to whom a tutorial was presented on image quality requirements. We randomly selected 200 images from a pool of 18,145 expert-labeled images (50/E, 50/G, 50/A, 50/I). The performance of the grading was timed and the agreement was assessed. An additional grading round was performed with 14 labels for a more objective analysis.ResultsThe median time (interquartile range) for the labeling task with 4 categories was 987.8 sec (418.6) for all graders and 872.9 sec (621.0) vs. 1019.8 sec (479.5) in the M vs. NM groups, respectively. Cohen's weighted kappa showed moderate agreement (0.564) when using four categories that increased to substantial (0.637) when using only three by merging the E and G groups. By the use of 14 labels, the weighted kappa values were 0.594 and 0.667 when assigning four or three categories, respectively.ConclusionImage grading with a Python-based tool seems to be a simple yet possibly efficient solution for the labeling of fundus images according to image quality that does not necessarily require medical background. Such grading can be subject to variability but could still effectively serve the robust identification of images with insufficient quality. This emphasizes the opportunity for the democratization of ML-applications among persons with both medical and non-medical background. However, simplicity of the grading system is key to successful categorization. For the training of machine learning (ML) algorithms, correctly labeled ground truth data are inevitable. In this pilot study, we assessed the performance of graders with different backgrounds in the labeling of retinal fundus image quality. Color fundus photographs were labeled using a Python-based tool using four image categories: excellent (E), good (G), adequate (A) and insufficient for grading (I). We enrolled 8 subjects (4 with and 4 without medical background, groups M and NM, respectively) to whom a tutorial was presented on image quality requirements. We randomly selected 200 images from a pool of 18,145 expert-labeled images (50/E, 50/G, 50/A, 50/I). The performance of the grading was timed and the agreement was assessed. An additional grading round was performed with 14 labels for a more objective analysis. The median time (interquartile range) for the labeling task with 4 categories was 987.8 sec (418.6) for all graders and 872.9 sec (621.0) vs. 1019.8 sec (479.5) in the M vs. NM groups, respectively. Cohen's weighted kappa showed moderate agreement (0.564) when using four categories that increased to substantial (0.637) when using only three by merging the E and G groups. By the use of 14 labels, the weighted kappa values were 0.594 and 0.667 when assigning four or three categories, respectively. Image grading with a Python-based tool seems to be a simple yet possibly efficient solution for the labeling of fundus images according to image quality that does not necessarily require medical background. Such grading can be subject to variability but could still effectively serve the robust identification of images with insufficient quality. This emphasizes the opportunity for the democratization of ML-applications among persons with both medical and non-medical background. However, simplicity of the grading system is key to successful categorization. |
| Audience | Academic |
| Author | Cabrera DeBuc, Delia Laurik-Feuerstein, Kornélia Lenke Sapahia, Rishav Somfai, Gábor Márk |
| AuthorAffiliation | 3 Department of Ophthalmology, Stadtspital Zürich, Zurich, Switzerland 5 Department of Ophthalmology, Semmelweis University, Budapest, Hungary 1 Department of Ophthalmology, Klinikum Ernst von Bergmann, Potsdam, Germany 4 Spross Research Institute, Zurich, Switzerland 2 Miller School of Medicine, Bascom Palmer Eye Institute, University of Miami, Miami, Florida, United States of America University of Warmia, POLAND |
| AuthorAffiliation_xml | – name: University of Warmia, POLAND – name: 3 Department of Ophthalmology, Stadtspital Zürich, Zurich, Switzerland – name: 2 Miller School of Medicine, Bascom Palmer Eye Institute, University of Miami, Miami, Florida, United States of America – name: 1 Department of Ophthalmology, Klinikum Ernst von Bergmann, Potsdam, Germany – name: 5 Department of Ophthalmology, Semmelweis University, Budapest, Hungary – name: 4 Spross Research Institute, Zurich, Switzerland |
| Author_xml | – sequence: 1 givenname: Kornélia Lenke surname: Laurik-Feuerstein fullname: Laurik-Feuerstein, Kornélia Lenke – sequence: 2 givenname: Rishav surname: Sapahia fullname: Sapahia, Rishav – sequence: 3 givenname: Delia surname: Cabrera DeBuc fullname: Cabrera DeBuc, Delia – sequence: 4 givenname: Gábor Márk orcidid: 0000-0001-6329-442X surname: Somfai fullname: Somfai, Gábor Márk |
| BackLink | https://www.ncbi.nlm.nih.gov/pubmed/35881576$$D View this record in MEDLINE/PubMed |
| BookMark | eNqNkltr3DAQhU1JaS7tPyitoVDah91aliVbfSiE0EsgEGjTvoqRPPZqq7U2kt10_33lrBOyIdDgB5nxN0dn5vgw2etch0nykmRzQkvyYekG34Gdr2N5nuUlIYw_SQ6IoPmM5xndu_O-nxyGsMwyRivOnyX7lFUVYSU_SOqLBaYQAoawwq5PXZM2Q1cPITUraDG9HMCafpNaUGhN16Y-HqDMdRFWLlZaDzX6kF6ZfpHWpmnQj0oK9O_WuygWnidPG7ABX0znUfLzy-eLk2-zs_OvpyfHZzPNRd7Pcp5DqVRZoMBSZ3WRaaWYEhWWIJAVoApSEoGkaJAoorUATaAqcsVF01BNj5LXW921dUFOCwoy56IQWc6piMTplqgdLOXaxyH9Rjow8rrgfCvB90ZblIo3rKrzsqYVKbAqoa5EjbQCzlhGBYtabKs1dGvYXIG1t4Ikk2NGNxbkmJGcMop9nyaXg1phreOyPNgdM7tfOrOQrfsjY5qkKGgUeDcJeHc5YOjlygSN1kKHbpjm5Zmgo8c399CHtzJRLcTBTde4eK8eReXx6JkVjI6-5w9Q8alxZXScsDGxvtPwfqchMj3-7VsYQpCnP74_nj3_tcu-vcMuEGy_CM4OvXFd2AVf3d307Ypvfv8IfNwC2rsQPDZSmx5GnTiasf_LsbjX_Kj4_wHHjy6R |
| CitedBy_id | crossref_primary_10_1007_s11357_024_01445_0 crossref_primary_10_1371_journal_pone_0289555 crossref_primary_10_1371_journal_pone_0292776 crossref_primary_10_1167_tvst_12_7_16 crossref_primary_10_1167_tvst_13_4_19 crossref_primary_10_1136_bmjophth_2023_001493 crossref_primary_10_2196_59914 |
| Cites_doi | 10.1001/jama.2016.17216 10.2147/OPTH.S235751 10.1167/iovs.12-10347 10.1167/iovs.05-1155 10.1016/S2589-7500(20)30240-5 10.1136/bjophthalmol-2018-313173 10.1016/j.ins.2019.06.011 10.1097/ICU.0000000000000678 10.1016/j.ophtha.2019.09.025 10.1016/j.ophtha.2019.12.015 10.1016/j.ophtha.2018.01.034 10.2196/jmir.3807 10.1016/j.preteyeres.2020.100900 10.1002/uog.5256 10.1177/193229680900300315 10.1038/s41591-018-0107-6 10.1186/s12886-016-0295-0 10.1371/journal.pone.0179310 |
| ContentType | Journal Article |
| Copyright | COPYRIGHT 2022 Public Library of Science 2022 Laurik-Feuerstein et al. This is an open access article distributed under the terms of the Creative Commons Attribution License: http://creativecommons.org/licenses/by/4.0/ (the “License”), which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License. 2022 Laurik-Feuerstein et al 2022 Laurik-Feuerstein et al |
| Copyright_xml | – notice: COPYRIGHT 2022 Public Library of Science – notice: 2022 Laurik-Feuerstein et al. This is an open access article distributed under the terms of the Creative Commons Attribution License: http://creativecommons.org/licenses/by/4.0/ (the “License”), which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License. – notice: 2022 Laurik-Feuerstein et al 2022 Laurik-Feuerstein et al |
| DBID | AAYXX CITATION CGR CUY CVF ECM EIF NPM IOV ISR 3V. 7QG 7QL 7QO 7RV 7SN 7SS 7T5 7TG 7TM 7U9 7X2 7X7 7XB 88E 8AO 8C1 8FD 8FE 8FG 8FH 8FI 8FJ 8FK ABJCF ABUWG AEUYN AFKRA ARAPS ATCPS AZQEC BBNVY BENPR BGLVJ BHPHI C1K CCPQU D1I DWQXO FR3 FYUFA GHDGH GNUQQ H94 HCIFZ K9. KB. KB0 KL. L6V LK8 M0K M0S M1P M7N M7P M7S NAPCQ P5Z P62 P64 PATMY PDBOC PHGZM PHGZT PIMPY PJZUB PKEHL PPXIY PQEST PQGLB PQQKQ PQUKI PRINS PTHSS PYCSY RC3 7X8 5PM ADTOC UNPAY DOA |
| DOI | 10.1371/journal.pone.0271156 |
| DatabaseName | CrossRef Medline MEDLINE MEDLINE (Ovid) MEDLINE MEDLINE PubMed Gale In Context: Opposing Viewpoints Gale In Context: Science ProQuest Central (Corporate) Animal Behavior Abstracts Bacteriology Abstracts (Microbiology B) Biotechnology Research Abstracts Nursing & Allied Health Database Ecology Abstracts Entomology Abstracts (Full archive) Immunology Abstracts Meteorological & Geoastrophysical Abstracts Nucleic Acids Abstracts Virology and AIDS Abstracts ProQuest Agricultural Science Health & Medical Collection ProQuest Central (purchase pre-March 2016) Medical Database (Alumni Edition) ProQuest Pharma Collection Public Health Database Technology Research Database ProQuest SciTech Collection ProQuest Technology Collection ProQuest Natural Science Journals Hospital Premium Collection Hospital Premium Collection (Alumni Edition) ProQuest Central (Alumni) (purchase pre-March 2016) Materials Science & Engineering Collection ProQuest Central (Alumni) ProQuest One Sustainability ProQuest Central UK/Ireland Advanced Technologies & Computer Science Collection Agricultural & Environmental Science Collection ProQuest Central Essentials Biological Science Collection ProQuest Central ProQuest Technology Collection Natural Science Collection Environmental Sciences and Pollution Management ProQuest One ProQuest Materials Science Collection ProQuest Central Engineering Research Database Health Research Premium Collection Health Research Premium Collection (Alumni) ProQuest Central Student AIDS and Cancer Research Abstracts SciTech Premium Collection ProQuest Health & Medical Complete (Alumni) Materials Science Database (Proquest) Nursing & Allied Health Database (Alumni Edition) Meteorological & Geoastrophysical Abstracts - Academic ProQuest Engineering Collection Biological Sciences Agricultural Science Database ProQuest Health & Medical Collection Medical Database Algology Mycology and Protozoology Abstracts (Microbiology C) Biological science database Engineering Database Nursing & Allied Health Premium Advanced Technologies & Aerospace Database ProQuest Advanced Technologies & Aerospace Collection Biotechnology and BioEngineering Abstracts Environmental Science Database Materials Science Collection Proquest Central Premium ProQuest One Academic Publicly Available Content Database ProQuest Health & Medical Research Collection ProQuest One Academic Middle East (New) ProQuest One Health & Nursing ProQuest One Academic Eastern Edition (DO NOT USE) ProQuest One Applied & Life Sciences ProQuest One Academic ProQuest One Academic UKI Edition ProQuest Central China Engineering Collection Environmental Science Collection Genetics Abstracts MEDLINE - Academic PubMed Central (Full Participant titles) Unpaywall for CDI: Periodical Content Unpaywall DOAJ Directory of Open Access Journals |
| DatabaseTitle | CrossRef MEDLINE Medline Complete MEDLINE with Full Text PubMed MEDLINE (Ovid) Agricultural Science Database Publicly Available Content Database ProQuest Central Student ProQuest Advanced Technologies & Aerospace Collection ProQuest Central Essentials Nucleic Acids Abstracts SciTech Premium Collection ProQuest Central China Environmental Sciences and Pollution Management ProQuest One Applied & Life Sciences ProQuest One Sustainability Health Research Premium Collection Meteorological & Geoastrophysical Abstracts Natural Science Collection Health & Medical Research Collection Biological Science Collection ProQuest Central (New) ProQuest Medical Library (Alumni) Engineering Collection Advanced Technologies & Aerospace Collection Engineering Database Virology and AIDS Abstracts ProQuest Biological Science Collection ProQuest One Academic Eastern Edition Agricultural Science Collection ProQuest Hospital Collection ProQuest Technology Collection Health Research Premium Collection (Alumni) Biological Science Database Ecology Abstracts ProQuest Hospital Collection (Alumni) Biotechnology and BioEngineering Abstracts Environmental Science Collection Entomology Abstracts Nursing & Allied Health Premium ProQuest Health & Medical Complete ProQuest One Academic UKI Edition Environmental Science Database ProQuest Nursing & Allied Health Source (Alumni) Engineering Research Database ProQuest One Academic Meteorological & Geoastrophysical Abstracts - Academic ProQuest One Academic (New) Technology Collection Technology Research Database ProQuest One Academic Middle East (New) Materials Science Collection ProQuest Health & Medical Complete (Alumni) ProQuest Central (Alumni Edition) ProQuest One Community College ProQuest One Health & Nursing ProQuest Natural Science Collection ProQuest Pharma Collection ProQuest Central ProQuest Health & Medical Research Collection Genetics Abstracts ProQuest Engineering Collection Biotechnology Research Abstracts Health and Medicine Complete (Alumni Edition) ProQuest Central Korea Bacteriology Abstracts (Microbiology B) Algology Mycology and Protozoology Abstracts (Microbiology C) Agricultural & Environmental Science Collection AIDS and Cancer Research Abstracts Materials Science Database ProQuest Materials Science Collection ProQuest Public Health ProQuest Nursing & Allied Health Source ProQuest SciTech Collection Advanced Technologies & Aerospace Database ProQuest Medical Library Animal Behavior Abstracts Materials Science & Engineering Collection Immunology Abstracts ProQuest Central (Alumni) MEDLINE - Academic |
| DatabaseTitleList | MEDLINE - Academic Agricultural Science Database MEDLINE |
| Database_xml | – sequence: 1 dbid: DOA name: DOAJ (selected full-text) url: https://www.doaj.org/ sourceTypes: Open Website – sequence: 2 dbid: NPM name: PubMed url: https://proxy.k.utb.cz/login?url=http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed sourceTypes: Index Database – sequence: 3 dbid: EIF name: MEDLINE url: https://proxy.k.utb.cz/login?url=https://www.webofscience.com/wos/medline/basic-search sourceTypes: Index Database – sequence: 4 dbid: UNPAY name: Unpaywall url: https://proxy.k.utb.cz/login?url=https://unpaywall.org/ sourceTypes: Open Access Repository – sequence: 5 dbid: 8FG name: ProQuest Technology Collection url: https://search.proquest.com/technologycollection1 sourceTypes: Aggregation Database |
| DeliveryMethod | fulltext_linktorsrc |
| Discipline | Sciences (General) |
| DocumentTitleAlternate | Image quality labeling among graders with different backgrounds |
| EISSN | 1932-6203 |
| ExternalDocumentID | 2694902639 oai_doaj_org_article_b6f58d27d3814e87ad89de38a6550395 10.1371/journal.pone.0271156 PMC9321443 A711554536 35881576 10_1371_journal_pone_0271156 |
| Genre | Research Support, Non-U.S. Gov't Journal Article Research Support, N.I.H., Extramural |
| GeographicLocations | United States |
| GeographicLocations_xml | – name: United States |
| GrantInformation_xml | – fundername: NEI NIH HHS grantid: P30 EY014801 – fundername: ; – fundername: ; grantid: P30-EY014801 |
| GroupedDBID | --- 123 29O 2WC 53G 5VS 7RV 7X2 7X7 7XC 88E 8AO 8C1 8CJ 8FE 8FG 8FH 8FI 8FJ A8Z AAFWJ AAUCC AAWOE AAYXX ABDBF ABIVO ABJCF ABUWG ACGFO ACIHN ACIWK ACPRK ACUHS ADBBV AEAQA AENEX AEUYN AFKRA AFPKN AFRAH AHMBA ALMA_UNASSIGNED_HOLDINGS AOIJS APEBS ARAPS ATCPS BAWUL BBNVY BCNDV BENPR BGLVJ BHPHI BKEYQ BPHCQ BVXVI BWKFM CCPQU CITATION CS3 D1I D1J D1K DIK DU5 E3Z EAP EAS EBD EMOBN ESTFP ESX EX3 F5P FPL FYUFA GROUPED_DOAJ GX1 HCIFZ HH5 HMCUK HYE IAO IEA IGS IHR IHW INH INR IOV IPY ISE ISR ITC K6- KB. KQ8 L6V LK5 LK8 M0K M1P M48 M7P M7R M7S M~E NAPCQ O5R O5S OK1 OVT P2P P62 PATMY PDBOC PHGZM PHGZT PIMPY PJZUB PPXIY PQGLB PQQKQ PROAC PSQYO PTHSS PUEGO PV9 PYCSY RNS RPM RZL SV3 TR2 UKHRP WOQ WOW ~02 ~KM ADRAZ ALIPV CGR CUY CVF ECM EIF IPNFZ NPM RIG BBORY 3V. 7QG 7QL 7QO 7SN 7SS 7T5 7TG 7TM 7U9 7XB 8FD 8FK AZQEC C1K DWQXO FR3 GNUQQ H94 K9. KL. M7N P64 PKEHL PQEST PQUKI PRINS RC3 7X8 5PM ADTOC UNPAY - 02 AAPBV ABPTK ADACO BBAFP KM |
| ID | FETCH-LOGICAL-c692t-262a7bb74e9e7c0d40cbb5b98e7a9e54ab41719e14fe1b1cc9ac1a842b69ff3c3 |
| IEDL.DBID | M48 |
| ISSN | 1932-6203 |
| IngestDate | Sun Jul 31 00:48:02 EDT 2022 Fri Oct 03 12:41:13 EDT 2025 Sun Oct 26 04:11:33 EDT 2025 Tue Sep 30 16:29:00 EDT 2025 Fri Sep 05 12:09:16 EDT 2025 Tue Oct 07 07:42:25 EDT 2025 Mon Oct 20 22:21:37 EDT 2025 Mon Oct 20 16:31:02 EDT 2025 Thu Oct 16 15:20:21 EDT 2025 Thu Oct 16 13:54:17 EDT 2025 Thu May 22 21:19:39 EDT 2025 Mon Jul 21 06:07:44 EDT 2025 Wed Oct 01 03:01:58 EDT 2025 Thu Apr 24 23:02:42 EDT 2025 |
| IsDoiOpenAccess | true |
| IsOpenAccess | true |
| IsPeerReviewed | true |
| IsScholarly | true |
| Issue | 7 |
| Language | English |
| License | This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. cc-by Creative Commons Attribution License |
| LinkModel | DirectLink |
| MergedId | FETCHMERGED-LOGICAL-c692t-262a7bb74e9e7c0d40cbb5b98e7a9e54ab41719e14fe1b1cc9ac1a842b69ff3c3 |
| Notes | ObjectType-Article-1 SourceType-Scholarly Journals-1 ObjectType-Feature-2 content type line 14 content type line 23 Competing Interests: The authors have declared that no competing interests exist. |
| ORCID | 0000-0001-6329-442X |
| OpenAccessLink | http://journals.scholarsportal.info/openUrl.xqy?doi=10.1371/journal.pone.0271156 |
| PMID | 35881576 |
| PQID | 2694902639 |
| PQPubID | 1436336 |
| PageCount | e0271156 |
| ParticipantIDs | plos_journals_2694902639 doaj_primary_oai_doaj_org_article_b6f58d27d3814e87ad89de38a6550395 unpaywall_primary_10_1371_journal_pone_0271156 pubmedcentral_primary_oai_pubmedcentral_nih_gov_9321443 proquest_miscellaneous_2694960935 proquest_journals_2694902639 gale_infotracmisc_A711554536 gale_infotracacademiconefile_A711554536 gale_incontextgauss_ISR_A711554536 gale_incontextgauss_IOV_A711554536 gale_healthsolutions_A711554536 pubmed_primary_35881576 crossref_citationtrail_10_1371_journal_pone_0271156 crossref_primary_10_1371_journal_pone_0271156 |
| ProviderPackageCode | CITATION AAYXX |
| PublicationCentury | 2000 |
| PublicationDate | 2022-07-26 |
| PublicationDateYYYYMMDD | 2022-07-26 |
| PublicationDate_xml | – month: 07 year: 2022 text: 2022-07-26 day: 26 |
| PublicationDecade | 2020 |
| PublicationPlace | United States |
| PublicationPlace_xml | – name: United States – name: San Francisco – name: San Francisco, CA USA |
| PublicationTitle | PloS one |
| PublicationTitleAlternate | PLoS One |
| PublicationYear | 2022 |
| Publisher | Public Library of Science Public Library of Science (PLoS) |
| Publisher_xml | – name: Public Library of Science – name: Public Library of Science (PLoS) |
| References | MA Zapata (pone.0271156.ref013) 2020; 14 A Kras (pone.0271156.ref003) 2020; 31 J Krause (pone.0271156.ref016) 2018; 125 R Thapa (pone.0271156.ref020) 2016; 16 J Cuadros (pone.0271156.ref012) 2009; 3 S Waite (pone.0271156.ref025) 2016 DSW Ting (pone.0271156.ref009) 2019; 103 J Loo (pone.0271156.ref010) 2020; 127 pone.0271156.ref006 pone.0271156.ref001 M McKenna (pone.0271156.ref022) 2018; 0 CJ Brady (pone.0271156.ref023) 2014; 16 DC DeBuc (pone.0271156.ref008) 2020; 12 E Trucco (pone.0271156.ref018) 2013; 54 CJ Flaxel (pone.0271156.ref007) 2020; 127 V Gulshan (pone.0271156.ref015) 2016; 316 AD Fleming (pone.0271156.ref014) 2006; 47 FMA Islam (pone.0271156.ref021) 2017; 12 SM Khan (pone.0271156.ref005) 2021; 3 J. W. Bartlett (pone.0271156.ref017) 2008; 31 T Panch (pone.0271156.ref004) 2020; 3 JPO Li (pone.0271156.ref019) 2021; 82 J De Fauw (pone.0271156.ref002) 2018; 24 T Li (pone.0271156.ref011) 2019; 501 CJ Brady (pone.0271156.ref024) 2021; 62 37796944 - PLoS One. 2023 Oct 5;18(10):e0292776 |
| References_xml | – ident: pone.0271156.ref001 – volume: 316 start-page: 2402 issue: 22 year: 2016 ident: pone.0271156.ref015 article-title: Development and Validation of a Deep Learning Algorithm for Detection of Diabetic Retinopathy in Retinal Fundus Photographs publication-title: JAMA doi: 10.1001/jama.2016.17216 – volume: 14 start-page: 419 year: 2020 ident: pone.0271156.ref013 article-title: Artificial Intelligence to Identify Retinal Fundus Images, Quality Validation, Laterality Evaluation, Macular Degeneration, and Suspected Glaucoma publication-title: Clin Ophthalmol doi: 10.2147/OPTH.S235751 – volume: 54 start-page: 3546 issue: 5 year: 2013 ident: pone.0271156.ref018 article-title: Validating retinal fundus image analysis algorithms: issues and a proposal publication-title: Invest Ophthalmol Vis Sci doi: 10.1167/iovs.12-10347 – volume: 47 start-page: 1120 issue: 3 year: 2006 ident: pone.0271156.ref014 article-title: Automated assessment of diabetic retinal image quality based on clarity and field definition publication-title: Invest Ophth Vis Sci doi: 10.1167/iovs.05-1155 – volume: 3 issue: 87 year: 2020 ident: pone.0271156.ref004 article-title: “Yes, but will it work for my patients?” Driving clinically relevant research with benchmark datasets publication-title: npj Digit Med – volume: 3 start-page: e51 issue: 1 year: 2021 ident: pone.0271156.ref005 article-title: A global review of publicly available datasets for ophthalmological imaging: barriers to access, usability, and generalisability publication-title: Lancet Digit Health doi: 10.1016/S2589-7500(20)30240-5 – year: 2016 ident: pone.0271156.ref025 article-title: Tired in the Reading Room: The Influence of Fatigue in Radiology publication-title: J Am Coll Radiol – volume: 0 start-page: 1 year: 2018 ident: pone.0271156.ref022 article-title: Accuracy of trained rural ophthalmologists versus non-medical image graders in the diagnosis of diabetic retinopathy in rural China publication-title: Br J Ophthalmol – volume: 12 start-page: 1 issue: 23 year: 2020 ident: pone.0271156.ref008 article-title: Artificial intelligence in the ophthalmic landscape publication-title: Nepal J Ophthalmol – volume: 103 start-page: 167 issue: 2 year: 2019 ident: pone.0271156.ref009 article-title: Artificial intelligence and deep learning in ophthalmology publication-title: Br J Ophthalmol doi: 10.1136/bjophthalmol-2018-313173 – volume: 501 start-page: 511 year: 2019 ident: pone.0271156.ref011 article-title: Diagnostic assessment of deep learning algorithms for diabetic retinopathy screening publication-title: Inform Sciences doi: 10.1016/j.ins.2019.06.011 – volume: 31 start-page: 337 issue: 5 year: 2020 ident: pone.0271156.ref003 article-title: Accelerating ophthalmic artificial intelligence research: the role of an open access data repository publication-title: Curr Opin Ophthalmol doi: 10.1097/ICU.0000000000000678 – volume: 127 start-page: 66 issue: 1 year: 2020 ident: pone.0271156.ref007 article-title: Diabetic retinopathy preferredpractice pattern publication-title: Ophthalmology doi: 10.1016/j.ophtha.2019.09.025 – volume: 62 start-page: 1788 year: 2021 ident: pone.0271156.ref024 article-title: Crowdsourcing Can Match Field Grading Validity for Follicular Trachoma publication-title: IOVS – ident: pone.0271156.ref006 – volume: 127 start-page: 793 issue: 6 year: 2020 ident: pone.0271156.ref010 article-title: Beyond Performance Metrics: Automatic Deep Learning Retinal OCT Analysis Reproduces Clinical Trial Outcome publication-title: Ophthalmology doi: 10.1016/j.ophtha.2019.12.015 – volume: 125 start-page: 1264 issue: 8 year: 2018 ident: pone.0271156.ref016 article-title: Grader Variability and the Importance of Reference Standards for Evaluating Machine Learning Models for Diabetic Retinopathy publication-title: Ophthalmology doi: 10.1016/j.ophtha.2018.01.034 – volume: 16 start-page: e233 issue: 10 year: 2014 ident: pone.0271156.ref023 article-title: Rapid grading of fundus photographs for diabetic retinopathy using crowdsourcing publication-title: J Med Internet Res doi: 10.2196/jmir.3807 – volume: 82 start-page: 100900 year: 2021 ident: pone.0271156.ref019 article-title: Digital technology, tele-medicine and artificial intelligence in ophthalmology: A global perspective publication-title: Prog Retin Eye Res doi: 10.1016/j.preteyeres.2020.100900 – volume: 31 start-page: 466 issue: 4 year: 2008 ident: pone.0271156.ref017 article-title: Reliability, Repeatability and Reproducibility: Analysis of Measurement Errors in Continuous Variables publication-title: Ultrasound in Obstetrics & Gynecology: The Official Journal of the International Society of Ultrasound in Obstetrics and Gynecology doi: 10.1002/uog.5256 – volume: 3 start-page: 509 issue: 3 year: 2009 ident: pone.0271156.ref012 article-title: EyePACS: an adaptable telemedicine system for diabetic retinopathy screening publication-title: J Diabetes Sci Technol doi: 10.1177/193229680900300315 – volume: 24 start-page: 1342 issue: 9 year: 2018 ident: pone.0271156.ref002 article-title: Clinically applicable deep learning for diagnosis and referral in retinal disease publication-title: Nat Med doi: 10.1038/s41591-018-0107-6 – volume: 16 start-page: 112 year: 2016 ident: pone.0271156.ref020 article-title: Intra- and inter-rater agreement between an ophthalmologist and mid-level ophthalmic personnel to diagnose retinal diseases based on fundus photographs at a primary eye center in Nepal: the Bhaktapur Retina Study publication-title: BMC Ophthalmol doi: 10.1186/s12886-016-0295-0 – volume: 12 start-page: e0179310 issue: 6 year: 2017 ident: pone.0271156.ref021 article-title: Accuracy and reliability of retinal photo grading for diabetic retinopathy: Remote graders from a developing country and standard retinal photo grader in Australia publication-title: PLoS One doi: 10.1371/journal.pone.0179310 – reference: 37796944 - PLoS One. 2023 Oct 5;18(10):e0292776 |
| SSID | ssj0053866 |
| Score | 2.4519827 |
| Snippet | For the training of machine learning (ML) algorithms, correctly labeled ground truth data are inevitable. In this pilot study, we assessed the performance of... Purpose For the training of machine learning (ML) algorithms, correctly labeled ground truth data are inevitable. In this pilot study, we assessed the... PurposeFor the training of machine learning (ML) algorithms, correctly labeled ground truth data are inevitable. In this pilot study, we assessed the... Purpose For the training of machine learning (ML) algorithms, correctly labeled ground truth data are inevitable. In this pilot study, we assessed the... |
| SourceID | plos doaj unpaywall pubmedcentral proquest gale pubmed crossref |
| SourceType | Open Website Open Access Repository Aggregation Database Index Database Enrichment Source |
| StartPage | e0271156 |
| SubjectTerms | Algorithms Artificial intelligence Automation Biology and Life Sciences Categories Color Color vision Computer and Information Sciences Datasets Diabetes Diabetic retinopathy Diagnostic Techniques, Ophthalmological Engineering and Technology Fundus Oculi Humans Image processing Image quality Labeling Labelling Labels Machine learning Medicine and Health Sciences Objective analysis Ophthalmology Performance assessment Photography - methods Pilot Projects Reliability analysis Reproducibility of Results Research and Analysis Methods Retina User interface |
| SummonAdditionalLinks | – databaseName: DOAJ Directory of Open Access Journals dbid: DOA link: http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwrV1Lb9QwELbQXuCCKK8GWjAICThkm8SOHR8LoiocQAKKerNsx2krlmS12VW1_57xI1EjKrUHTivF440yM55HMvMNQm90DV6FKJNmRImUaqNTbd17K17nmnILMbOv8v3Kjk_ol9Py9MqoL1cTFuCBA-MONGvKqi54Da6F2oqruhK1JZViEFsT4dFLs0oMyVSwwXCKGYuNcoTnB1Eu82XX2jkkYhAGsYkj8nj9o1WeLRddf13I-W_l5N1Nu1TbS7VYXHFLRw_Q_RhP4sPwHDvojm0fop14Ynv8LsJKv3-EatAIrEYgTtw1GJxavenxxR8wKji0V24xqIXvUccr-Ako3lvsZxLhs5Wve8bu5S0eRqussVbmt-sOaev-MTo5-vTz43EaZyykholinRasUFxrTq2w3GQ1zYzWpRaV5UrYkipNc54Lm9PG5jo3RiiTq4oWmommIYY8QbMWuLqLcMY5KWATEVrQsmRKKZMRQ1UjMtpkNkFkYLg0EYDczcFYSP9VjUMiEngmnZhkFFOC0nHXMgBw3ED_wclypHXw2f4CKJWMSiVvUqoEvXSaIEMv6mgE5KG7BcScBG7z2lM4CI3W1eicqU3fy8_fft2C6Mf3CdHbSNR0wA6jYl8EPJOD5ppQ7k0owRCYyfKu09uBK710TcoCcmwiYOegy9cvvxqX3Z-6urvWdptIw9zH8gQ9Dao_cpaUVZVDupogPjkUE9ZPV9qLc49gLtx4LEoSNB-Pz62E--x_CPc5ule4HpaMg_bvodl6tbH7EFmu9QtvRP4Co8F42w priority: 102 providerName: Directory of Open Access Journals – databaseName: ProQuest Central dbid: BENPR link: http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwjV3db9MwELdG9wAviPG1wgCDkICHdEnsxPEDQhvaNHgoaDC0t8hfKRMlKU0r1P-eu8QJREywp0rxuVHOd-c7--53hDzXFnYVpkwQMiUDro0OtMNzK2EjzYUDn7nJ8p2mJ2f8_XlyvkWmXS0MplV2NrEx1LYyeEa-jxWXEgIGJt8sfgTYNQpvV7sWGsq3VrCvG4ixa2Q7RmSsEdk-PJp-PO1sM2h3mvoCOiaifb9ek0VVugkEaOAepYMNqsHx7631aDGv6stc0b8zKq-vy4Xa_FTz-R_b1fEtctP7mfSgFYwdsuXK22THa3JNX3q46Vd3iAVJoaoH6KRVQWGzs-uaXnwHY0PbsssNBXFpatfpEn5adO8NbXoV0dmyyYemeKhLu5YrK6qV-YZVI6Wt75Kz46PPb08C33shMKmMV0GcxkpoLbiTTpjQ8tBonWiZOaGkS7jSPBKRdBEvXKQjY6Qykcp4rFNZFMywe2RUAld3CQ2FYDFMYlJLniSpUsqEzHBVyJAXoRsT1jE8Nx6YHPtjzPPmtk1AgNLyLMdlyv0yjUnQz1q0wBz_oT_EtexpEVa7eVAtZ7nX0lynRZLZWFjwY7jLhLKZtI5lKoVAjslkTJ6gJORtjWpvHPIDfAX4ogxe86yhQGiNEnN3Zmpd1_m7D1-uQPTpdED0whMVFbDDKF8vAd-EkF0Dyr0BJRgIMxjeRbntuFLnv1UJZnayfPnw034Y_xTz8UpXrT1NipfoY3K_Ff2esyzJsgjC2DERA6UYsH44Ul58bZDNJbbN4mxMJr36XGlxH_z7Ox6SGzFWrYQC5HqPjFbLtXsEvuRKP_YG4hetN3dg priority: 102 providerName: ProQuest – databaseName: Unpaywall dbid: UNPAY link: http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwtV3Nb9MwFLdGd4ALML4WGGAQ4uOQLImdOD4WxDQ4DAQUjcMU2Y4zppW0alqhcuBv573EiQgMMQ6cWtXPTvPz8_Oz_d7PhDzSBcwqTBk_ZEr6XBvta4v7VqKINBcWfOYmyvcg3Z_w14fJ4QY56nJhHIKwRpzO6uYkH7_MKrvrkNxFvqL29DSImIi6GsEchAJYZIGLkz5uGIdwZ2yJCUgXyGaagKs-IpuTg7fjT-1Jc-yncchcOt2fWhpMVw2rf2-7R_jPznJMf4-vvLiq5mr9VU2nP01ee1fI9-6125iV02C11IH59gsj5H_D5Sq57NxeOm5b2SIbtrpGtpxhqelTx3797DopQHGp6vlC6aykMPcWq5qefAHbR9ss0DUF7W1S6ekCPlqy8TVtrk6ix4smPJviHjPtboBZUq3MKSaxVEV9g0z2Xn54se-7qyB8k8p46cdprITWgltphQkLHhqtEy0zK5S0CVeaRyKSNuKljXRkjFQmUhmPdSrLkhl2k4wqAGKb0FAIFkMlJrXkSZIqpUzIDFelDHkZWo-wrsdz43jS8bqOad4c_glYL7WY5Yhs7pD1iN_Xmrc8IX-Rf47K1Msiy3fzA3Rt7ro012mZZEUsCnCruM2EKjJZWJapFNaVTCYeuY-qmLcps72tysf4CHCNGTzmYSOBTB8VhhIdq1Vd56_efDyH0Pt3A6EnTqicARxGufQNeCfUvIHkzkAS7JUZFG-j6nao1DnmUsswBlcZanaD6eziB30xNorhgZWdrZxMimf6HrnVjr0eWZZkWQSrao-IwagcQD8sqU4-N0TrEm_x4swjQT9-z9W5t_-1wh1yKca0mlCApu-Q0XKxsnfB2V3qe85k_QCh37B- priority: 102 providerName: Unpaywall |
| Title | The assessment of fundus image quality labeling reliability among graders with different backgrounds |
| URI | https://www.ncbi.nlm.nih.gov/pubmed/35881576 https://www.proquest.com/docview/2694902639 https://www.proquest.com/docview/2694960935 https://pubmed.ncbi.nlm.nih.gov/PMC9321443 https://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0271156&type=printable https://doaj.org/article/b6f58d27d3814e87ad89de38a6550395 http://dx.doi.org/10.1371/journal.pone.0271156 |
| UnpaywallVersion | publishedVersion |
| Volume | 17 |
| hasFullText | 1 |
| inHoldings | 1 |
| isFullTextHit | |
| isPrint | |
| journalDatabaseRights | – providerCode: PRVFSB databaseName: Free Full-Text Journals in Chemistry customDbUrl: eissn: 1932-6203 dateEnd: 99991231 omitProxy: true ssIdentifier: ssj0053866 issn: 1932-6203 databaseCode: HH5 dateStart: 20060101 isFulltext: true titleUrlDefault: http://abc-chemistry.org/ providerName: ABC ChemistRy – providerCode: PRVAFT databaseName: Open Access Digital Library customDbUrl: eissn: 1932-6203 dateEnd: 99991231 omitProxy: true ssIdentifier: ssj0053866 issn: 1932-6203 databaseCode: KQ8 dateStart: 20060101 isFulltext: true titleUrlDefault: http://grweb.coalliance.org/oadl/oadl.html providerName: Colorado Alliance of Research Libraries – providerCode: PRVAFT databaseName: Open Access Digital Library customDbUrl: eissn: 1932-6203 dateEnd: 99991231 omitProxy: true ssIdentifier: ssj0053866 issn: 1932-6203 databaseCode: KQ8 dateStart: 20061001 isFulltext: true titleUrlDefault: http://grweb.coalliance.org/oadl/oadl.html providerName: Colorado Alliance of Research Libraries – providerCode: PRVAON databaseName: DOAJ (selected full-text) customDbUrl: eissn: 1932-6203 dateEnd: 99991231 omitProxy: true ssIdentifier: ssj0053866 issn: 1932-6203 databaseCode: DOA dateStart: 20060101 isFulltext: true titleUrlDefault: https://www.doaj.org/ providerName: Directory of Open Access Journals – providerCode: PRVEBS databaseName: Academic Search Ultimate customDbUrl: https://search.ebscohost.com/login.aspx?authtype=ip,shib&custid=s3936755&profile=ehost&defaultdb=asn eissn: 1932-6203 dateEnd: 99991231 omitProxy: true ssIdentifier: ssj0053866 issn: 1932-6203 databaseCode: ABDBF dateStart: 20080101 isFulltext: true titleUrlDefault: https://search.ebscohost.com/direct.asp?db=asn providerName: EBSCOhost – providerCode: PRVEBS databaseName: EBSCOhost Food Science Source customDbUrl: eissn: 1932-6203 dateEnd: 99991231 omitProxy: false ssIdentifier: ssj0053866 issn: 1932-6203 databaseCode: A8Z dateStart: 20080101 isFulltext: true titleUrlDefault: https://search.ebscohost.com/login.aspx?authtype=ip,uid&profile=ehost&defaultdb=fsr providerName: EBSCOhost – providerCode: PRVBFR databaseName: Free Medical Journals customDbUrl: eissn: 1932-6203 dateEnd: 99991231 omitProxy: true ssIdentifier: ssj0053866 issn: 1932-6203 databaseCode: DIK dateStart: 20060101 isFulltext: true titleUrlDefault: http://www.freemedicaljournals.com providerName: Flying Publisher – providerCode: PRVFQY databaseName: Open access medical journals (GFMER) customDbUrl: eissn: 1932-6203 dateEnd: 99991231 omitProxy: true ssIdentifier: ssj0053866 issn: 1932-6203 databaseCode: GX1 dateStart: 20060101 isFulltext: true titleUrlDefault: http://www.gfmer.ch/Medical_journals/Free_medical.php providerName: Geneva Foundation for Medical Education and Research – providerCode: PRVHPJ databaseName: ROAD: Directory of Open Access Scholarly Resources (selected full-text only) customDbUrl: eissn: 1932-6203 dateEnd: 99991231 omitProxy: true ssIdentifier: ssj0053866 issn: 1932-6203 databaseCode: M~E dateStart: 20060101 isFulltext: true titleUrlDefault: https://road.issn.org providerName: ISSN International Centre – providerCode: PRVAQN databaseName: PubMed Central (Selected Fulltext) customDbUrl: eissn: 1932-6203 dateEnd: 99991231 omitProxy: true ssIdentifier: ssj0053866 issn: 1932-6203 databaseCode: RPM dateStart: 20060101 isFulltext: true titleUrlDefault: https://www.ncbi.nlm.nih.gov/pmc/ providerName: National Library of Medicine – providerCode: PRVPQU databaseName: Health & Medical Collection customDbUrl: eissn: 1932-6203 dateEnd: 99991231 omitProxy: true ssIdentifier: ssj0053866 issn: 1932-6203 databaseCode: 7X7 dateStart: 20061201 isFulltext: true titleUrlDefault: https://search.proquest.com/healthcomplete providerName: ProQuest – providerCode: PRVPQU databaseName: ProQuest Central customDbUrl: http://www.proquest.com/pqcentral?accountid=15518 eissn: 1932-6203 dateEnd: 99991231 omitProxy: true ssIdentifier: ssj0053866 issn: 1932-6203 databaseCode: BENPR dateStart: 20061201 isFulltext: true titleUrlDefault: https://www.proquest.com/central providerName: ProQuest – providerCode: PRVPQU databaseName: ProQuest Technology Collection customDbUrl: eissn: 1932-6203 dateEnd: 99991231 omitProxy: true ssIdentifier: ssj0053866 issn: 1932-6203 databaseCode: 8FG dateStart: 20061201 isFulltext: true titleUrlDefault: https://search.proquest.com/technologycollection1 providerName: ProQuest – providerCode: PRVPQU databaseName: Public Health Database customDbUrl: eissn: 1932-6203 dateEnd: 99991231 omitProxy: true ssIdentifier: ssj0053866 issn: 1932-6203 databaseCode: 8C1 dateStart: 20061201 isFulltext: true titleUrlDefault: https://search.proquest.com/publichealth providerName: ProQuest – providerCode: PRVFZP databaseName: Scholars Portal - Open Access customDbUrl: eissn: 1932-6203 dateEnd: 20250930 omitProxy: true ssIdentifier: ssj0053866 issn: 1932-6203 databaseCode: M48 dateStart: 20061201 isFulltext: true titleUrlDefault: http://journals.scholarsportal.info providerName: Scholars Portal |
| link | http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwjR1db9Mw0Nq6B3hBjK8FRjEICXhIlcROHD8g1E0rA4kyDYrGU2Q7TpkoSWlaQf89Z8eNiOjEXhIpvkuU8935zr4PhJ7LHFYVIpQfEMF9KpX0pTb7ViwPJWUabGYb5TtOTif0_UV8sYM2PVsdAeutrp3pJzVZzAa_f67fgMC_tl0bWLhBGsyrUg_AzQIjJ9lFe7BWcdPM4QNtzxVAupPEJdBdhdlZoGwd_1Zb9-azqt5miv4bUXljVc7F-peYzf5arka30S1nZ-Jhwxj7aEeXd9C-k-Qav3Tlpl_dRTlwChZtgU5cFRgWu3xV48sfoGxwk3a5xsAuNncdL-DWVPdeY9urCE8XNh4am01dvGm5ssRSqO8ma6TM63toMjr5fHzqu94Lvkp4tPSjJBJMSkY110wFOQ2UlLHkqWaC65gKSUMWch3SQocyVIoLFYqURjLhRUEUuY96JVD1AOGAMRIBEuGS0zhOhBAqIIqKgge0CLSHyIbgmXKFyU1_jFlmT9sYOCgNzTIzTZmbJg_5Lda8KczxH_gjM5ctrCmrbR9Ui2nmpDSTSRGnecRysGOoTpnIU55rkooEHDnCYw89MZyQNTmqrXLIhuYTYIsS-MwzC2FKa5QmdmcqVnWdvfv45RpAn847QC8cUFEBOZRw-RLwT6ZkVwfysAMJCkJ1hg8M326oUmcmeZmD7004YG54efvw03bYvNTE45W6WjmYxByie-hBw_otZUmcpiG4sR5iHaHokL47Ul5-s5XNuWmbRYmHBq34XGtyH179i4_QzchkrAQMePoQ9ZaLlX4MduRS9tEuu2BwTY9Dcx297aO9o5Px2Xnf7sz0reqAZ5Px2fDrH7Tce7A |
| linkProvider | Scholars Portal |
| linkToHtml | http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwtV1Lb9QwELbKcigXRHk1UKhBIOCQbRI7cXxAqDyqXVqKBG21t2A7zlKxJMtmo2r_FL-RceIEIiropaeV1uNEmRl_nrHngdATmcKuQoRyPSK4S6WSrtTm3IqlvqRMg81cR_keRqNj-n4STtbQzzYXxoRVtphYA3VaKHNGvmMyLjk4DIS_mv9wTdcoc7vattBo1GJfr87AZStfjt-CfJ8Gwd67ozcj13YVcFXEg6UbRIFgUjKquWbKS6mnpAwljzUTXIdUSOozn2ufZtqXvlJcKF_ENJARzzKiCDz3CrpKCWAJrB826Rw8wI4osul5hPk7VhuG8yLXQ3D_wPiKettf3SWg2wsG81lRnmfo_h2vuV7lc7E6E7PZH5vh3g103VqxeLdRuw20pvObaMPiRImf22LWL26hFPQQi678Jy4yDFtpWpX49DtAGW6SOlcYlLHOjMcL-Glqh69w3QkJTxd1tDU2R8a4beiyxFKobyYnJU_L2-j4UmRwBw1y4Oomwh5jJIBJhEtOwzASQiiPKCoy7tHM0w4iLcMTZcuem-4bs6S-y2Pg_jQ8S4yYEismB7ndrHlT9uM_9K-NLDtaU7S7_qNYTBOLAYmMsjBOA5aClUR1zEQa81STWETgJhIeOmjbaELSZMB20JPsmleApUvgNY9rClO4IzeRQVNRlWUy_nhyAaLPn3pEzyxRVgA7lLDZGPBNpiBYj3KrRwnwo3rDm0ZvW66Uye-FCjNbXT5_-FE3bB5qov1yXVSWJjJX9A6626h-x1kSxrEPTrKDWG9R9FjfH8lPv9Z107lpykWJg4bd8rmQcO_9-zu20fro6MNBcjA-3L-PrgUmP8ZjoONbaLBcVPoBWK1L-bCGCoy-XDY2_QJgeK9X |
| linkToPdf | http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwtV3db9MwELdGkYAXxPhaYTCDQMBDtiR24vgBocGYNoYGAob6FmzHKRMlKU2rqf8afx13iROImGAve6pUnxPl7ufznX0fhDzSGewqTBnPZ0p6XBvtaYvnViILNBcWbOY6yvcw3jvib0bRaIX8bHNhMKyy1Ym1os5Kg2fkW5hxKcFhYHIrd2ER73d2X0x_eNhBCm9a23YaDUQO7PIE3Lfq-f4OyPpxGO6-_vRqz3MdBjwTy3DuhXGohNaCW2mF8TPuG60jLRMrlLQRV5oHIpA24LkNdGCMVCZQCQ91LPOcGQbPvUAuCsYkhhOKUefsgR6JY5eqx0Sw5ZCxOS0LuwmuIBhicW8rrDsGdPvCYDopq9OM3r9jNy8viqlanqjJ5I-NcfcaueosWrrdQHCVrNjiOll1OqOiT11h62c3SAaYpKorBUrLnMK2mi0qevwd1BptEjyXFIBZZ8nTGfw0dcSXtO6KRMezOvKa4vExbZu7zKlW5hvmpxRZdZMcnYsMbpFBAVxdI9QXgoUwiUkteRTFSinjM8NVLn2e-3ZIWMvw1LgS6NiJY5LW93oCXKGGZymKKXViGhKvmzVtSoD8h_4lyrKjxQLe9R_lbJw6fZDqOI-SLBQZWEzcJkJlicwsS1QMLiOgaUg2EAlpkw3bqaF0G18BVi-D1zysKbCIR4HLYawWVZXuv_t8BqKPH3pETxxRXgI7jHKZGfBNWBysR7neowRVZHrDa4jblitV-nvRwswWy6cPP-iG8aEY-VfYcuFoYryuH5LbDfQ7zrIoSQJwmIdE9BZFj_X9keL4a11DXWKDLs6GZLNbPmcS7p1_f8cGuQRaKX27f3hwl1wJMVXGFwDxdTKYzxb2Hhiwc32_1hSUfDlv1fQLr-6zmg |
| linkToUnpaywall | http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwtV3Nb9MwFLdGd4ALML4WGGAQ4uOQLImdOD4WxDQ4DAQUjcMU2Y4zppW0alqhcuBv573EiQgMMQ6cWtXPTvPz8_Oz_d7PhDzSBcwqTBk_ZEr6XBvta4v7VqKINBcWfOYmyvcg3Z_w14fJ4QY56nJhHIKwRpzO6uYkH7_MKrvrkNxFvqL29DSImIi6GsEchAJYZIGLkz5uGIdwZ2yJCUgXyGaagKs-IpuTg7fjT-1Jc-yncchcOt2fWhpMVw2rf2-7R_jPznJMf4-vvLiq5mr9VU2nP01ee1fI9-6125iV02C11IH59gsj5H_D5Sq57NxeOm5b2SIbtrpGtpxhqelTx3797DopQHGp6vlC6aykMPcWq5qefAHbR9ss0DUF7W1S6ekCPlqy8TVtrk6ix4smPJviHjPtboBZUq3MKSaxVEV9g0z2Xn54se-7qyB8k8p46cdprITWgltphQkLHhqtEy0zK5S0CVeaRyKSNuKljXRkjFQmUhmPdSrLkhl2k4wqAGKb0FAIFkMlJrXkSZIqpUzIDFelDHkZWo-wrsdz43jS8bqOad4c_glYL7WY5Yhs7pD1iN_Xmrc8IX-Rf47K1Msiy3fzA3Rt7ro012mZZEUsCnCruM2EKjJZWJapFNaVTCYeuY-qmLcps72tysf4CHCNGTzmYSOBTB8VhhIdq1Vd56_efDyH0Pt3A6EnTqicARxGufQNeCfUvIHkzkAS7JUZFG-j6nao1DnmUsswBlcZanaD6eziB30xNorhgZWdrZxMimf6HrnVjr0eWZZkWQSrao-IwagcQD8sqU4-N0TrEm_x4swjQT9-z9W5t_-1wh1yKca0mlCApu-Q0XKxsnfB2V3qe85k_QCh37B- |
| openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+assessment+of+fundus+image+quality+labeling+reliability+among+graders+with+different+backgrounds&rft.jtitle=PloS+one&rft.au=Korn%C3%A9lia+Lenke+Laurik-Feuerstein&rft.au=Contributed+equally+to+this+work+with%3A+Korn%C3%A9lia+Lenke+Laurik-Feuerstein&rft.au=Rishav+Sapahia+Rishav+Sapahia&rft.au=Rishav+Sapahia+Delia+Cabrera+DeBuc&rft.date=2022-07-26&rft.pub=Public+Library+of+Science&rft.eissn=1932-6203&rft.volume=17&rft.issue=7&rft_id=info:doi/10.1371%2Fjournal.pone.0271156&rft.externalDocID=2694902639 |
| thumbnail_l | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=1932-6203&client=summon |
| thumbnail_m | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=1932-6203&client=summon |
| thumbnail_s | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=1932-6203&client=summon |