Annotating for Artificial Intelligence Applications in Digital Pathology: A Practical Guide for Pathologists and Researchers
Training machine learning models for artificial intelligence (AI) applications in pathology often requires extensive annotation by human experts, but there is little guidance on the subject. In this work, we aimed to describe our experience and provide a simple, useful, and practical guide addressin...
Saved in:
| Published in | Modern pathology Vol. 36; no. 4; p. 100086 |
|---|---|
| Main Authors | , , , , , , |
| Format | Journal Article |
| Language | English |
| Published |
United States
Elsevier Inc
01.04.2023
|
| Subjects | |
| Online Access | Get full text |
| ISSN | 0893-3952 1530-0285 1530-0285 |
| DOI | 10.1016/j.modpat.2022.100086 |
Cover
| Abstract | Training machine learning models for artificial intelligence (AI) applications in pathology often requires extensive annotation by human experts, but there is little guidance on the subject. In this work, we aimed to describe our experience and provide a simple, useful, and practical guide addressing annotation strategies for AI development in computational pathology. Annotation methodology will vary significantly depending on the specific study’s objectives, but common difficulties will be present across different settings. We summarize key aspects and issue guiding principles regarding team interaction, ground-truth quality assessment, different annotation types, and available software and hardware options and address common difficulties while annotating. This guide was specifically designed for pathology annotation, intending to help pathologists, other researchers, and AI developers with this process. |
|---|---|
| AbstractList | Training machine learning models for artificial intelligence (AI) applications in pathology often requires extensive annotation by human experts, but there is little guidance on the subject. In this work, we aimed to describe our experience and provide a simple, useful, and practical guide addressing annotation strategies for AI development in computational pathology. Annotation methodology will vary significantly depending on the specific study's objectives, but common difficulties will be present across different settings. We summarize key aspects and issue guiding principles regarding team interaction, ground-truth quality assessment, different annotation types, and available software and hardware options and address common difficulties while annotating. This guide was specifically designed for pathology annotation, intending to help pathologists, other researchers, and AI developers with this process. Training machine learning models for artificial intelligence (AI) applications in pathology often requires extensive annotation by human experts, but there is little guidance on the subject. In this work, we aimed to describe our experience and provide a simple, useful, and practical guide addressing annotation strategies for AI development in computational pathology. Annotation methodology will vary significantly depending on the specific study's objectives, but common difficulties will be present across different settings. We summarize key aspects and issue guiding principles regarding team interaction, ground-truth quality assessment, different annotation types, and available software and hardware options and address common difficulties while annotating. This guide was specifically designed for pathology annotation, intending to help pathologists, other researchers, and AI developers with this process.Training machine learning models for artificial intelligence (AI) applications in pathology often requires extensive annotation by human experts, but there is little guidance on the subject. In this work, we aimed to describe our experience and provide a simple, useful, and practical guide addressing annotation strategies for AI development in computational pathology. Annotation methodology will vary significantly depending on the specific study's objectives, but common difficulties will be present across different settings. We summarize key aspects and issue guiding principles regarding team interaction, ground-truth quality assessment, different annotation types, and available software and hardware options and address common difficulties while annotating. This guide was specifically designed for pathology annotation, intending to help pathologists, other researchers, and AI developers with this process. |
| ArticleNumber | 100086 |
| Author | Macedo-Pinto, Isabel Cardoso, Jaime S. Monteiro, Ana Oliveira, Domingos Neto, Pedro C. Montezuma, Diana Oliveira, Sara P. |
| Author_xml | – sequence: 1 givenname: Diana orcidid: 0000-0001-9551-4589 surname: Montezuma fullname: Montezuma, Diana email: diana.felizardo@impdiagnostics.com organization: IMP Diagnostics, Porto, Portugal – sequence: 2 givenname: Sara P. surname: Oliveira fullname: Oliveira, Sara P. organization: Telecommunications and Multimedia Unit, Institute for Systems and Computer Engineering, Technology and Science (INESC TEC), Porto, Portugal – sequence: 3 givenname: Pedro C. surname: Neto fullname: Neto, Pedro C. organization: Telecommunications and Multimedia Unit, Institute for Systems and Computer Engineering, Technology and Science (INESC TEC), Porto, Portugal – sequence: 4 givenname: Domingos surname: Oliveira fullname: Oliveira, Domingos organization: IMP Diagnostics, Porto, Portugal – sequence: 5 givenname: Ana surname: Monteiro fullname: Monteiro, Ana organization: IMP Diagnostics, Porto, Portugal – sequence: 6 givenname: Jaime S. surname: Cardoso fullname: Cardoso, Jaime S. organization: Telecommunications and Multimedia Unit, Institute for Systems and Computer Engineering, Technology and Science (INESC TEC), Porto, Portugal – sequence: 7 givenname: Isabel surname: Macedo-Pinto fullname: Macedo-Pinto, Isabel organization: IMP Diagnostics, Porto, Portugal |
| BackLink | https://www.ncbi.nlm.nih.gov/pubmed/36788085$$D View this record in MEDLINE/PubMed |
| BookMark | eNqFkUFv1DAQhS1URLeFf4CQj1yyTOyN4_SAFBUolSpRIThbju1sZ5W1g-1FqsSPx9u0HDjAaTSj7z1p3jsjJz54R8jrGtY11OLdbr0PdtZ5zYCxcgKQ4hlZ1Q2HCphsTsgKZMcr3jXslJyltAOoN41kL8gpF62UIJsV-dV7H7LO6Ld0DJH2MeOIBvVEr31204Rb542j_TxPaAoXfKLo6QfcYi7Qrc53YQrb-wva09uoTS7URK8OaN2D4ROAKSeqvaVfXXI6mjsX00vyfNRTcq8e5zn5_unjt8vP1c2Xq-vL_qYyXLBcGRiYFFbyrh1qbTg4LUfZDHocxMZacFxzAe1oBefMQNeZWshNWcbyJnSMn5O3i-8cw4-DS1ntMZnynPYuHJJibdtCDbxpCvrmET0Me2fVHHGv4716SqwAmwUwMaQU3fgHqUEdi1E7tRSjjsWopZgiu_hLZjA_xJmjxul_4veL2JWQfqKLKhk81mIxOpOVDfhvg9-Duqws |
| CitedBy_id | crossref_primary_10_1080_21681163_2024_2396595 crossref_primary_10_1016_j_pathol_2023_09_003 crossref_primary_10_1016_j_procs_2024_06_405 crossref_primary_10_1016_j_labinv_2024_102203 crossref_primary_10_1038_s41598_023_30497_z crossref_primary_10_1007_s10278_024_01248_x crossref_primary_10_1007_s40620_023_01775_w crossref_primary_10_1136_jcp_2024_210031 crossref_primary_10_1016_j_ijhcs_2024_103315 crossref_primary_10_1016_j_media_2024_103360 crossref_primary_10_1016_j_ajpath_2025_01_009 crossref_primary_10_1007_s10278_024_01043_8 crossref_primary_10_3390_jimaging10110292 |
| Cites_doi | 10.1038/s41746-021-00549-7 10.1109/TSMC.1979.4310076 10.1038/s41598-022-19278-2 10.4103/jpi.jpi_81_18 10.1007/s10278-020-00384-4 10.1038/s41416-020-01122-x 10.1158/0008-5472.CAN-17-0629 10.1016/j.media.2022.102466 10.1002/cjp2.229 10.1002/path.5921 10.1371/journal.pcbi.1007313 10.1002/cjp2.256 10.1186/s13000-021-01126-y 10.1038/s41598-017-17204-5 10.1016/j.ajpath.2019.05.007 10.1038/s41379-022-01147-y 10.1002/prca.201800057 10.1016/j.bbe.2021.04.012 10.1038/nmeth.1896 10.1158/0008-5472.CAN-17-0323 10.4103/jpi.jpi_83_20 10.1186/s13000-020-00995-z 10.1016/j.jneumeth.2021.109371 10.1002/path.5662 10.1186/s13000-020-00957-5 10.1007/s00330-019-6003-8 10.1016/j.ajpath.2021.05.023 10.1016/S2589-7500(21)00216-8 10.1038/s41591-019-0508-1 10.1002/path.5797 10.1038/s41467-021-21467-y 10.1109/RBME.2020.3013489 10.1038/s41598-021-93746-z 10.1016/j.patter.2021.100383 10.3390/cancers14122974 10.1038/s41591-021-01620-2 10.3390/cancers14102489 |
| ContentType | Journal Article |
| Copyright | 2022 The Authors Copyright © 2022 The Authors. Published by Elsevier Inc. All rights reserved. |
| Copyright_xml | – notice: 2022 The Authors – notice: Copyright © 2022 The Authors. Published by Elsevier Inc. All rights reserved. |
| DBID | 6I. AAFTH AAYXX CITATION CGR CUY CVF ECM EIF NPM 7X8 |
| DOI | 10.1016/j.modpat.2022.100086 |
| DatabaseName | ScienceDirect Open Access Titles Elsevier:ScienceDirect:Open Access CrossRef Medline MEDLINE MEDLINE (Ovid) MEDLINE MEDLINE PubMed MEDLINE - Academic |
| DatabaseTitle | CrossRef MEDLINE Medline Complete MEDLINE with Full Text PubMed MEDLINE (Ovid) MEDLINE - Academic |
| DatabaseTitleList | MEDLINE MEDLINE - Academic |
| Database_xml | – sequence: 1 dbid: NPM name: PubMed url: https://proxy.k.utb.cz/login?url=http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed sourceTypes: Index Database – sequence: 2 dbid: EIF name: MEDLINE url: https://proxy.k.utb.cz/login?url=https://www.webofscience.com/wos/medline/basic-search sourceTypes: Index Database |
| DeliveryMethod | fulltext_linktorsrc |
| Discipline | Medicine Biology |
| EISSN | 1530-0285 |
| ExternalDocumentID | 36788085 10_1016_j_modpat_2022_100086 S0893395222055260 |
| Genre | Research Support, Non-U.S. Gov't Journal Article |
| GroupedDBID | --- -Q- .GJ 0R~ 123 29M 2WC 36B 39C 3V. 4.4 53G 5RE 6I. 70F 7RV 7X7 88A 88E 8AO 8FE 8FH 8FI 8FJ 8R4 8R5 AADWK AAFTH AANZL AAQQT AASDW AAWBL AAWTL AAXUO AAYFA AAYJO AAZLF ABAWZ ABGIJ ABJNI ABLJU ABUWG ACBMV ACBRV ACBYP ACGFO ACGFS ACIGE ACKTT ACPRK ACRQY ACTTH ACVWB ACZOJ ADBBV ADFRT ADHDB ADMDM ADQMX ADYYL AEDAW AEFTE AEJRE AENEX AEXYK AFKRA AFOSN AFSHS AGAYW AGEZK AGGBP AGHAI AHGBK AHMBA AHSBF AILAN AJDOV AJRNO ALFFA ALMA_UNASSIGNED_HOLDINGS AMRAJ AMRJV AMYLF AXYYD BAWUL BBNVY BENPR BHPHI BKEYQ BKKNO BPHCQ BVXVI CAG CCPQU COF CS3 DIK DNIVK DU5 E3Z EBLON EBS EE. EIOEI EJD EX3 F5P FDB FDQFY FERAY FIZPM FSGXE FYUFA GX1 HCIFZ HMCUK HZ~ IWAJR JSO JZLTJ KQ8 LK8 M0L M1P M7P NAO NAPCQ NQJWS NYICJ O9- OK1 OWW P2P PQQKQ PROAC PSQYO Q2X RNS RNT RNTTT ROL SNX SNYQT SOHCF SRMVM SWTZT TAOOD TBHMF TDRGL TR2 TSG UKHRP WOW YFH ZA5 ZGI ZXP AALRI AAYWO AAYXX ACVFH ADCNI ADVLN AEUPX AFJKZ AFPUW AIGII AITUG AKBMS AKRWK AKYEP APXCP CITATION EFKBS PHGZM PHGZT PJZUB PPXIY PQGLB AGCQF ALIPV CGR CUY CVF ECM EIF NPM 7X8 |
| ID | FETCH-LOGICAL-c362t-c0b286d8397b1ac30ea8f85bafb64dd0e3a3607fd6332c099c1684633f6780923 |
| ISSN | 0893-3952 1530-0285 |
| IngestDate | Thu Oct 02 10:12:41 EDT 2025 Mon Jul 21 06:05:49 EDT 2025 Thu Oct 16 04:44:22 EDT 2025 Thu Apr 24 23:10:48 EDT 2025 Fri Feb 23 02:36:43 EST 2024 |
| IsDoiOpenAccess | true |
| IsOpenAccess | true |
| IsPeerReviewed | true |
| IsScholarly | true |
| Issue | 4 |
| Keywords | annotation computational pathology digital pathology artificial intelligence |
| Language | English |
| License | This is an open access article under the CC BY-NC-ND license. Copyright © 2022 The Authors. Published by Elsevier Inc. All rights reserved. |
| LinkModel | OpenURL |
| MergedId | FETCHMERGED-LOGICAL-c362t-c0b286d8397b1ac30ea8f85bafb64dd0e3a3607fd6332c099c1684633f6780923 |
| Notes | ObjectType-Article-1 SourceType-Scholarly Journals-1 ObjectType-Feature-2 content type line 23 |
| ORCID | 0000-0001-9551-4589 |
| OpenAccessLink | https://dx.doi.org/10.1016/j.modpat.2022.100086 |
| PMID | 36788085 |
| PQID | 2777010355 |
| PQPubID | 23479 |
| ParticipantIDs | proquest_miscellaneous_2777010355 pubmed_primary_36788085 crossref_primary_10_1016_j_modpat_2022_100086 crossref_citationtrail_10_1016_j_modpat_2022_100086 elsevier_sciencedirect_doi_10_1016_j_modpat_2022_100086 |
| PublicationCentury | 2000 |
| PublicationDate | April 2023 2023-04-00 20230401 |
| PublicationDateYYYYMMDD | 2023-04-01 |
| PublicationDate_xml | – month: 04 year: 2023 text: April 2023 |
| PublicationDecade | 2020 |
| PublicationPlace | United States |
| PublicationPlace_xml | – name: United States |
| PublicationTitle | Modern pathology |
| PublicationTitleAlternate | Mod Pathol |
| PublicationYear | 2023 |
| Publisher | Elsevier Inc |
| Publisher_xml | – name: Elsevier Inc |
| References | Bulten, Kartasalo, Chen (bib1) 2022; 28 Albuquerque, Moreira, Barros (bib12) 2022 Tran, Liu, Dusenberry (bib29) 2022 Möhle, Bascuñana, Brackhan, Pahnke (bib24) 2021; 364 Wahab, Miligy, Dodd (bib9) 2022; 8 Gutman, Khalilia, Lee (bib33) 2017; 77 bib32 da Silva, Pereira, Salles (bib44) 2021; 254 Campanella, Hanna, Geneslaw (bib13) 2019; 25 Korzynska, Roszkowiak, Zak, Siemion (bib37) 2021; 41 Mehta, Lu, Wu (bib2) 2022; 79 Oliveira, Neto, Fraga (bib10) 2021; 11 Neto, Oliveira, Montezuma (bib11) 2022; 14 Runz, Rusche, Schmidt, Weihrauch, Hesser, Weis (bib39) 2021; 16 Kocak, Ates, Durmaz, Ulusan, Kilickesmez (bib28) 2019; 29 Yakimovich, Beaugnon, Huang, Ozkirimli (bib15) 2021; 2 Shvetsov, Grønnesby, Pedersen (bib21) 2022; 14 Homeyer, Geißler, Schwen (bib23) 2022; 35 Rubens, Hoyoux, Vanosmael (bib31) 2019; 13 Mitchell, Cohen, Cohen (bib7) 2021; 191 Qayyum, Qadir, Bilal, Al-Fuqaha (bib8) 2021; 14 Miao, Toth, Zhou, Madabhushi, Janowczyk (bib25) 2021; 7 Wang, Yang, Rong, Zhan, Xiao (bib16) 2019; 189 Stadler, Lindvall, Lundström (bib6) 2021; 34 Bankhead (bib38) 2022; 257 Dudgeon, Wen, Hanna (bib26) 2021; 12 Allan, Burel, Moore (bib35) 2012; 9 Weigert, Schmidt (bib20) 2022 Bankhead, Loughrey, Fernández (bib30) 2017; 7 Hameed, Garcia-Zapirain, Aguirre, Isaza-Ruget (bib18) 2022; 12 Aubreville, Stathonikos, Bertram (bib19) 2022 Stritt, Stalder, Vezzali (bib34) 2020; 16 Lindman, Rose, Lindvall, Lundström, Treanor (bib22) 2019; 10 Feng, Deng, Yang (bib4) 2020; 15 bib43 Yang, Yu (bib17) 2021; 11 Martel, Hosseinzadeh, Senaras (bib36) 2017; 77 de Hond, Leeuwenberg, Hooft (bib41) 2022; 5 Otsu (bib42) 1979; 9 Chen, Mermel, Liu (bib27) 2021; 3 Pantanowitz, Hartman, Qi (bib3) 2020; 15 Boschman, Farahani, Darbandsari (bib40) 2022; 256 Echle, Rindtorff, Brinker, Luedde, Pearson, Kather (bib5) 2021; 124 Chen, Chen, Yu (bib14) 2021; 12 Lindman (10.1016/j.modpat.2022.100086_bib22) 2019; 10 Otsu (10.1016/j.modpat.2022.100086_bib42) 1979; 9 Yang (10.1016/j.modpat.2022.100086_bib17) 2021; 11 Gutman (10.1016/j.modpat.2022.100086_bib33) 2017; 77 Kocak (10.1016/j.modpat.2022.100086_bib28) 2019; 29 Bulten (10.1016/j.modpat.2022.100086_bib1) 2022; 28 da Silva (10.1016/j.modpat.2022.100086_bib44) 2021; 254 Weigert (10.1016/j.modpat.2022.100086_bib20) 2022 Mitchell (10.1016/j.modpat.2022.100086_bib7) 2021; 191 Bankhead (10.1016/j.modpat.2022.100086_bib38) 2022; 257 Runz (10.1016/j.modpat.2022.100086_bib39) 2021; 16 Wang (10.1016/j.modpat.2022.100086_bib16) 2019; 189 Bankhead (10.1016/j.modpat.2022.100086_bib30) 2017; 7 Feng (10.1016/j.modpat.2022.100086_bib4) 2020; 15 Aubreville (10.1016/j.modpat.2022.100086_bib19) 2022 Neto (10.1016/j.modpat.2022.100086_bib11) 2022; 14 Mehta (10.1016/j.modpat.2022.100086_bib2) 2022; 79 Pantanowitz (10.1016/j.modpat.2022.100086_bib3) 2020; 15 Hameed (10.1016/j.modpat.2022.100086_bib18) 2022; 12 Homeyer (10.1016/j.modpat.2022.100086_bib23) 2022; 35 Dudgeon (10.1016/j.modpat.2022.100086_bib26) 2021; 12 Miao (10.1016/j.modpat.2022.100086_bib25) 2021; 7 Oliveira (10.1016/j.modpat.2022.100086_bib10) 2021; 11 Qayyum (10.1016/j.modpat.2022.100086_bib8) 2021; 14 Rubens (10.1016/j.modpat.2022.100086_bib31) 2019; 13 Wahab (10.1016/j.modpat.2022.100086_bib9) 2022; 8 Albuquerque (10.1016/j.modpat.2022.100086_bib12) 2022 Boschman (10.1016/j.modpat.2022.100086_bib40) 2022; 256 Martel (10.1016/j.modpat.2022.100086_bib36) 2017; 77 Stritt (10.1016/j.modpat.2022.100086_bib34) 2020; 16 Allan (10.1016/j.modpat.2022.100086_bib35) 2012; 9 Campanella (10.1016/j.modpat.2022.100086_bib13) 2019; 25 Chen (10.1016/j.modpat.2022.100086_bib27) 2021; 3 Möhle (10.1016/j.modpat.2022.100086_bib24) 2021; 364 Shvetsov (10.1016/j.modpat.2022.100086_bib21) 2022; 14 de Hond (10.1016/j.modpat.2022.100086_bib41) 2022; 5 Chen (10.1016/j.modpat.2022.100086_bib14) 2021; 12 Korzynska (10.1016/j.modpat.2022.100086_bib37) 2021; 41 Tran (10.1016/j.modpat.2022.100086_bib29) 2022 Echle (10.1016/j.modpat.2022.100086_bib5) 2021; 124 Stadler (10.1016/j.modpat.2022.100086_bib6) 2021; 34 Yakimovich (10.1016/j.modpat.2022.100086_bib15) 2021; 2 |
| References_xml | – volume: 3 start-page: e693 year: 2021 end-page: e695 ident: bib27 article-title: Evaluation of artificial intelligence on a reference standard based on subjective interpretation publication-title: Lancet Digit Health – volume: 7 year: 2017 ident: bib30 article-title: QuPath: Open source software for digital pathology image analysis publication-title: Sci Rep – volume: 189 start-page: 1686 year: 2019 end-page: 1698 ident: bib16 article-title: Pathology image analysis using segmentation deep learning algorithms publication-title: Am J Pathol – volume: 9 start-page: 62 year: 1979 end-page: 66 ident: bib42 article-title: A threshold selection method from gray-level histograms publication-title: IEEE Trans Syst Man Cybern – volume: 9 start-page: 245 year: 2012 end-page: 253 ident: bib35 article-title: OMERO: flexible, model-driven data management for experimental biology publication-title: Nat Methods – volume: 28 start-page: 154 year: 2022 end-page: 163 ident: bib1 article-title: Artificial intelligence for diagnosis and Gleason grading of prostate cancer: the PANDA challenge publication-title: Nat Med – volume: 364 year: 2021 ident: bib24 article-title: Development of deep learning models for microglia analyses in brain tissue using DeePathology™ STUDIO publication-title: J Neurosci Methods – volume: 14 start-page: 2974 year: 2022 ident: bib21 article-title: A pragmatic machine learning approach to quantify tumor-infiltrating lymphocytes in whole slide images publication-title: Cancers (Basel) – volume: 34 start-page: 105 year: 2021 end-page: 115 ident: bib6 article-title: Proactive construction of an annotated imaging database for artificial intelligence training publication-title: J Digit Imaging – volume: 10 start-page: 22 year: 2019 ident: bib22 article-title: Annotations, ontologies, and whole slide images—development of an annotated ontology-driven whole slide image library of normal and abnormal human tissue publication-title: J Pathol Inform – volume: 29 start-page: 4765 year: 2019 end-page: 4775 ident: bib28 article-title: Influence of segmentation margin on machine learning-based high-dimensional quantitative CT texture analysis: a reproducibility study on renal clear cell carcinomas publication-title: Eur Radiol – volume: 25 start-page: 1301 year: 2019 end-page: 1309 ident: bib13 article-title: Clinical-grade computational pathology using weakly supervised deep learning on whole slide images publication-title: Nat Med – volume: 77 start-page: e75 year: 2017 end-page: e78 ident: bib33 article-title: The digital slide archive: a software platform for management, integration, and analysis of histology for cancer research publication-title: Cancer Res – volume: 257 start-page: 391 year: 2022 end-page: 402 ident: bib38 article-title: Developing image analysis methods for digital pathology publication-title: J Pathol – volume: 124 start-page: 686 year: 2021 end-page: 696 ident: bib5 article-title: Deep learning in cancer pathology: a new generation of clinical biomarkers publication-title: Br J Cancer – volume: 14 start-page: 2489 year: 2022 ident: bib11 article-title: iMIL4PATH: a semi-supervised interpretable approach for colorectal whole-slide images publication-title: Cancers (Basel) – volume: 15 start-page: 65 year: 2020 ident: bib4 article-title: Automated quantitative analysis of Ki-67 staining and HE images recognition and registration based on whole tissue sections in breast carcinoma publication-title: Diagn Pathol – volume: 12 year: 2022 ident: bib18 article-title: Multiclass classification of breast cancer histopathology images using multilevel features of deep convolutional neural network publication-title: Sci Rep – volume: 41 start-page: 1436 year: 2021 end-page: 1453 ident: bib37 article-title: A review of current systems for annotation of cell and tissue images in digital pathology publication-title: Biocybern Biomed Eng – volume: 191 start-page: 1709 year: 2021 end-page: 1716 ident: bib7 article-title: Dealing with multi-dimensional data and the burden of annotation: easing the burden of annotation publication-title: Am J Pathol – volume: 16 year: 2020 ident: bib34 article-title: Orbit image analysis: an open-source whole slide image analysis tool publication-title: PLoS Comput Biol – volume: 14 start-page: 156 year: 2021 end-page: 180 ident: bib8 article-title: Secure and robust machine learning for healthcare: a survey publication-title: IEEE Rev Biomed Eng – volume: 12 start-page: 45 year: 2021 ident: bib26 article-title: A pathologist-annotated dataset for validating artificial intelligence: a project description and pilot study publication-title: J Pathol Inform – volume: 77 start-page: e83 year: 2017 end-page: e86 ident: bib36 article-title: An image analysis resource for cancer research: PIIP-pathology image informatics platform for visualization, analysis, and management publication-title: Cancer Res – volume: 16 start-page: 71 year: 2021 ident: bib39 article-title: Normalization of HE-stained histological images using cycle-consistent generative adversarial networks publication-title: Diagn Pathol – volume: 5 start-page: 2 year: 2022 ident: bib41 article-title: Guidelines and quality criteria for artificial intelligence-based prediction models in healthcare: a scoping review publication-title: NPJ Digit Med – volume: 79 start-page: 102466 year: 2022 ident: bib2 article-title: End-to-End diagnosis of breast biopsy images with transformers publication-title: Med Image Anal – volume: 11 start-page: 14358 year: 2021 ident: bib10 article-title: CAD systems for colorectal cancer from WSI are still not ready for clinical acceptance publication-title: Sci Rep – year: 2022 ident: bib29 article-title: Plex: towards reliability using pretrained large model extensions publication-title: arXiv Preprint – ident: bib32 article-title: Radboud University Medical Center. ASAP – ident: bib43 article-title: Directorate-General for Research and Innovation. Turning FAIR into reality: final report and action plan from the European Commission expert group on FAIR data, Publications Office (2018) – volume: 254 start-page: 147 year: 2021 end-page: 158 ident: bib44 article-title: Independent real-world application of a clinical-grade automated prostate cancer detection system publication-title: J Pathol – year: 2022 ident: bib19 article-title: Mitosis domain generalization in histopathology images—the MIDOG challenge publication-title: arXiv Preprint – volume: 12 start-page: 1193 year: 2021 ident: bib14 article-title: An annotation-free whole-slide training approach to pathological classification of lung cancer types using deep learning publication-title: Nat Commun – volume: 7 start-page: 542 year: 2021 end-page: 547 ident: bib25 article-title: Quick Annotator: an open-source digital pathology-based rapid image annotation tool publication-title: J Pathol Clin Res – volume: 256 start-page: 15 year: 2022 end-page: 24 ident: bib40 article-title: The utility of color normalization for AI-based diagnosis of hematoxylin and eosin-stained pathology images publication-title: J Pathol – volume: 15 start-page: 80 year: 2020 ident: bib3 article-title: Accuracy and efficiency of an artificial intelligence tool when counting breast mitoses publication-title: Diagn Pathol – volume: 8 start-page: 116 year: 2022 end-page: 128 ident: bib9 article-title: Semantic annotation for computational pathology: multidisciplinary experience and best practice recommendations publication-title: J Pathol Clin Res – start-page: 1 year: 2022 end-page: 4 ident: bib20 article-title: Nuclei instance segmentation and classification in histopathology images with Stardist publication-title: 2022 IEEE International Symposium on Biomedical Imaging Challenges (ISBIC) – volume: 13 year: 2019 ident: bib31 article-title: Cytomine: toward an open and collaborative software platform for digital pathology bridged to molecular investigations publication-title: Proteomics Clin Appl – start-page: 588 year: 2022 end-page: 593 ident: bib12 article-title: Quality control in digital pathology: automatic fragment detection and counting publication-title: Annu Int Conf IEEE Eng Med Biol Soc – volume: 2 year: 2021 ident: bib15 article-title: Labels in a haystack: approaches beyond supervised learning in biomedical applications publication-title: Patterns – volume: 11 year: 2021 ident: bib17 article-title: Artificial convolutional neural network in object detection and semantic segmentation for medical imaging analysis publication-title: Front Oncol – volume: 35 start-page: 1759 year: 2022 end-page: 1769 ident: bib23 article-title: Recommendations on compiling test datasets for evaluating artificial intelligence solutions in pathology publication-title: Mod Pathol – volume: 5 start-page: 2 issue: 1 year: 2022 ident: 10.1016/j.modpat.2022.100086_bib41 article-title: Guidelines and quality criteria for artificial intelligence-based prediction models in healthcare: a scoping review publication-title: NPJ Digit Med doi: 10.1038/s41746-021-00549-7 – volume: 9 start-page: 62 year: 1979 ident: 10.1016/j.modpat.2022.100086_bib42 article-title: A threshold selection method from gray-level histograms publication-title: IEEE Trans Syst Man Cybern doi: 10.1109/TSMC.1979.4310076 – volume: 12 issue: 1 year: 2022 ident: 10.1016/j.modpat.2022.100086_bib18 article-title: Multiclass classification of breast cancer histopathology images using multilevel features of deep convolutional neural network publication-title: Sci Rep doi: 10.1038/s41598-022-19278-2 – start-page: 588 year: 2022 ident: 10.1016/j.modpat.2022.100086_bib12 article-title: Quality control in digital pathology: automatic fragment detection and counting publication-title: Annu Int Conf IEEE Eng Med Biol Soc – volume: 10 start-page: 22 year: 2019 ident: 10.1016/j.modpat.2022.100086_bib22 article-title: Annotations, ontologies, and whole slide images—development of an annotated ontology-driven whole slide image library of normal and abnormal human tissue publication-title: J Pathol Inform doi: 10.4103/jpi.jpi_81_18 – volume: 34 start-page: 105 issue: 1 year: 2021 ident: 10.1016/j.modpat.2022.100086_bib6 article-title: Proactive construction of an annotated imaging database for artificial intelligence training publication-title: J Digit Imaging doi: 10.1007/s10278-020-00384-4 – volume: 124 start-page: 686 issue: 4 year: 2021 ident: 10.1016/j.modpat.2022.100086_bib5 article-title: Deep learning in cancer pathology: a new generation of clinical biomarkers publication-title: Br J Cancer doi: 10.1038/s41416-020-01122-x – volume: 77 start-page: e75 issue: 21 year: 2017 ident: 10.1016/j.modpat.2022.100086_bib33 article-title: The digital slide archive: a software platform for management, integration, and analysis of histology for cancer research publication-title: Cancer Res doi: 10.1158/0008-5472.CAN-17-0629 – volume: 79 start-page: 102466 year: 2022 ident: 10.1016/j.modpat.2022.100086_bib2 article-title: End-to-End diagnosis of breast biopsy images with transformers publication-title: Med Image Anal doi: 10.1016/j.media.2022.102466 – year: 2022 ident: 10.1016/j.modpat.2022.100086_bib29 article-title: Plex: towards reliability using pretrained large model extensions publication-title: arXiv Preprint – volume: 7 start-page: 542 issue: 6 year: 2021 ident: 10.1016/j.modpat.2022.100086_bib25 article-title: Quick Annotator: an open-source digital pathology-based rapid image annotation tool publication-title: J Pathol Clin Res doi: 10.1002/cjp2.229 – volume: 257 start-page: 391 issue: 4 year: 2022 ident: 10.1016/j.modpat.2022.100086_bib38 article-title: Developing image analysis methods for digital pathology publication-title: J Pathol doi: 10.1002/path.5921 – volume: 16 issue: 2 year: 2020 ident: 10.1016/j.modpat.2022.100086_bib34 article-title: Orbit image analysis: an open-source whole slide image analysis tool publication-title: PLoS Comput Biol doi: 10.1371/journal.pcbi.1007313 – volume: 8 start-page: 116 issue: 2 year: 2022 ident: 10.1016/j.modpat.2022.100086_bib9 article-title: Semantic annotation for computational pathology: multidisciplinary experience and best practice recommendations publication-title: J Pathol Clin Res doi: 10.1002/cjp2.256 – volume: 16 start-page: 71 issue: 1 year: 2021 ident: 10.1016/j.modpat.2022.100086_bib39 article-title: Normalization of HE-stained histological images using cycle-consistent generative adversarial networks publication-title: Diagn Pathol doi: 10.1186/s13000-021-01126-y – volume: 7 issue: 1 year: 2017 ident: 10.1016/j.modpat.2022.100086_bib30 article-title: QuPath: Open source software for digital pathology image analysis publication-title: Sci Rep doi: 10.1038/s41598-017-17204-5 – volume: 189 start-page: 1686 issue: 9 year: 2019 ident: 10.1016/j.modpat.2022.100086_bib16 article-title: Pathology image analysis using segmentation deep learning algorithms publication-title: Am J Pathol doi: 10.1016/j.ajpath.2019.05.007 – start-page: 1 year: 2022 ident: 10.1016/j.modpat.2022.100086_bib20 article-title: Nuclei instance segmentation and classification in histopathology images with Stardist – volume: 35 start-page: 1759 issue: 12 year: 2022 ident: 10.1016/j.modpat.2022.100086_bib23 article-title: Recommendations on compiling test datasets for evaluating artificial intelligence solutions in pathology publication-title: Mod Pathol doi: 10.1038/s41379-022-01147-y – volume: 13 issue: 1 year: 2019 ident: 10.1016/j.modpat.2022.100086_bib31 article-title: Cytomine: toward an open and collaborative software platform for digital pathology bridged to molecular investigations publication-title: Proteomics Clin Appl doi: 10.1002/prca.201800057 – volume: 41 start-page: 1436 issue: 4 year: 2021 ident: 10.1016/j.modpat.2022.100086_bib37 article-title: A review of current systems for annotation of cell and tissue images in digital pathology publication-title: Biocybern Biomed Eng doi: 10.1016/j.bbe.2021.04.012 – volume: 9 start-page: 245 issue: 3 year: 2012 ident: 10.1016/j.modpat.2022.100086_bib35 article-title: OMERO: flexible, model-driven data management for experimental biology publication-title: Nat Methods doi: 10.1038/nmeth.1896 – volume: 77 start-page: e83 issue: 21 year: 2017 ident: 10.1016/j.modpat.2022.100086_bib36 article-title: An image analysis resource for cancer research: PIIP-pathology image informatics platform for visualization, analysis, and management publication-title: Cancer Res doi: 10.1158/0008-5472.CAN-17-0323 – year: 2022 ident: 10.1016/j.modpat.2022.100086_bib19 article-title: Mitosis domain generalization in histopathology images—the MIDOG challenge publication-title: arXiv Preprint – volume: 12 start-page: 45 year: 2021 ident: 10.1016/j.modpat.2022.100086_bib26 article-title: A pathologist-annotated dataset for validating artificial intelligence: a project description and pilot study publication-title: J Pathol Inform doi: 10.4103/jpi.jpi_83_20 – volume: 15 start-page: 80 issue: 1 year: 2020 ident: 10.1016/j.modpat.2022.100086_bib3 article-title: Accuracy and efficiency of an artificial intelligence tool when counting breast mitoses publication-title: Diagn Pathol doi: 10.1186/s13000-020-00995-z – volume: 364 year: 2021 ident: 10.1016/j.modpat.2022.100086_bib24 article-title: Development of deep learning models for microglia analyses in brain tissue using DeePathology™ STUDIO publication-title: J Neurosci Methods doi: 10.1016/j.jneumeth.2021.109371 – volume: 254 start-page: 147 issue: 2 year: 2021 ident: 10.1016/j.modpat.2022.100086_bib44 article-title: Independent real-world application of a clinical-grade automated prostate cancer detection system publication-title: J Pathol doi: 10.1002/path.5662 – volume: 15 start-page: 65 issue: 1 year: 2020 ident: 10.1016/j.modpat.2022.100086_bib4 article-title: Automated quantitative analysis of Ki-67 staining and HE images recognition and registration based on whole tissue sections in breast carcinoma publication-title: Diagn Pathol doi: 10.1186/s13000-020-00957-5 – volume: 29 start-page: 4765 issue: 9 year: 2019 ident: 10.1016/j.modpat.2022.100086_bib28 article-title: Influence of segmentation margin on machine learning-based high-dimensional quantitative CT texture analysis: a reproducibility study on renal clear cell carcinomas publication-title: Eur Radiol doi: 10.1007/s00330-019-6003-8 – volume: 191 start-page: 1709 issue: 10 year: 2021 ident: 10.1016/j.modpat.2022.100086_bib7 article-title: Dealing with multi-dimensional data and the burden of annotation: easing the burden of annotation publication-title: Am J Pathol doi: 10.1016/j.ajpath.2021.05.023 – volume: 3 start-page: e693 issue: 11 year: 2021 ident: 10.1016/j.modpat.2022.100086_bib27 article-title: Evaluation of artificial intelligence on a reference standard based on subjective interpretation publication-title: Lancet Digit Health doi: 10.1016/S2589-7500(21)00216-8 – volume: 25 start-page: 1301 issue: 8 year: 2019 ident: 10.1016/j.modpat.2022.100086_bib13 article-title: Clinical-grade computational pathology using weakly supervised deep learning on whole slide images publication-title: Nat Med doi: 10.1038/s41591-019-0508-1 – volume: 256 start-page: 15 issue: 1 year: 2022 ident: 10.1016/j.modpat.2022.100086_bib40 article-title: The utility of color normalization for AI-based diagnosis of hematoxylin and eosin-stained pathology images publication-title: J Pathol doi: 10.1002/path.5797 – volume: 12 start-page: 1193 issue: 1 year: 2021 ident: 10.1016/j.modpat.2022.100086_bib14 article-title: An annotation-free whole-slide training approach to pathological classification of lung cancer types using deep learning publication-title: Nat Commun doi: 10.1038/s41467-021-21467-y – volume: 11 year: 2021 ident: 10.1016/j.modpat.2022.100086_bib17 article-title: Artificial convolutional neural network in object detection and semantic segmentation for medical imaging analysis publication-title: Front Oncol – volume: 14 start-page: 156 year: 2021 ident: 10.1016/j.modpat.2022.100086_bib8 article-title: Secure and robust machine learning for healthcare: a survey publication-title: IEEE Rev Biomed Eng doi: 10.1109/RBME.2020.3013489 – volume: 11 start-page: 14358 issue: 1 year: 2021 ident: 10.1016/j.modpat.2022.100086_bib10 article-title: CAD systems for colorectal cancer from WSI are still not ready for clinical acceptance publication-title: Sci Rep doi: 10.1038/s41598-021-93746-z – volume: 2 issue: 12 year: 2021 ident: 10.1016/j.modpat.2022.100086_bib15 article-title: Labels in a haystack: approaches beyond supervised learning in biomedical applications publication-title: Patterns doi: 10.1016/j.patter.2021.100383 – volume: 14 start-page: 2974 issue: 12 year: 2022 ident: 10.1016/j.modpat.2022.100086_bib21 article-title: A pragmatic machine learning approach to quantify tumor-infiltrating lymphocytes in whole slide images publication-title: Cancers (Basel) doi: 10.3390/cancers14122974 – volume: 28 start-page: 154 issue: 1 year: 2022 ident: 10.1016/j.modpat.2022.100086_bib1 article-title: Artificial intelligence for diagnosis and Gleason grading of prostate cancer: the PANDA challenge publication-title: Nat Med doi: 10.1038/s41591-021-01620-2 – volume: 14 start-page: 2489 issue: 10 year: 2022 ident: 10.1016/j.modpat.2022.100086_bib11 article-title: iMIL4PATH: a semi-supervised interpretable approach for colorectal whole-slide images publication-title: Cancers (Basel) doi: 10.3390/cancers14102489 |
| SSID | ssj0014582 |
| Score | 2.492808 |
| Snippet | Training machine learning models for artificial intelligence (AI) applications in pathology often requires extensive annotation by human experts, but there is... |
| SourceID | proquest pubmed crossref elsevier |
| SourceType | Aggregation Database Index Database Enrichment Source Publisher |
| StartPage | 100086 |
| SubjectTerms | annotation Artificial Intelligence computational pathology digital pathology Humans Machine Learning Pathologists Software |
| Title | Annotating for Artificial Intelligence Applications in Digital Pathology: A Practical Guide for Pathologists and Researchers |
| URI | https://dx.doi.org/10.1016/j.modpat.2022.100086 https://www.ncbi.nlm.nih.gov/pubmed/36788085 https://www.proquest.com/docview/2777010355 |
| Volume | 36 |
| hasFullText | 1 |
| inHoldings | 1 |
| isFullTextHit | |
| isPrint | |
| journalDatabaseRights | – providerCode: PRVAFT databaseName: Open Access Digital Library customDbUrl: eissn: 1530-0285 dateEnd: 99991231 omitProxy: true ssIdentifier: ssj0014582 issn: 0893-3952 databaseCode: KQ8 dateStart: 20000101 isFulltext: true titleUrlDefault: http://grweb.coalliance.org/oadl/oadl.html providerName: Colorado Alliance of Research Libraries – providerCode: PRVBFR databaseName: Free Medical Journals customDbUrl: eissn: 1530-0285 dateEnd: 20241102 omitProxy: true ssIdentifier: ssj0014582 issn: 0893-3952 databaseCode: DIK dateStart: 20000101 isFulltext: true titleUrlDefault: http://www.freemedicaljournals.com providerName: Flying Publisher – providerCode: PRVFQY databaseName: GFMER Free Medical Journals customDbUrl: eissn: 1530-0285 dateEnd: 20241102 omitProxy: true ssIdentifier: ssj0014582 issn: 0893-3952 databaseCode: GX1 dateStart: 20000101 isFulltext: true titleUrlDefault: http://www.gfmer.ch/Medical_journals/Free_medical.php providerName: Geneva Foundation for Medical Education and Research – providerCode: PRVLSH databaseName: Elsevier Journals customDbUrl: mediaType: online eissn: 1530-0285 dateEnd: 99991231 omitProxy: true ssIdentifier: ssj0014582 issn: 0893-3952 databaseCode: AKRWK dateStart: 20000101 isFulltext: true providerName: Library Specific Holdings |
| link | http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwnV1Zb9QwELaWIlBfEJSjyyUj8bZKlMQ5HN6iHlRIhT600r5Fju3AVt1ktd08tOof4d8yPnIsbCnlJVpF9sjOfDueGc-B0EfKEhGFYOQwOP6dkMvSAS0_cRgVHo9SkJva4Xb8NT46C79Mo-lo9HMQtdSsCpdfb8wr-R-uwjvgq8qSvQdnO6LwAn4Df-EJHIbnP_E4q6paXaXbYMhsqQN_TPmMQaXNbLEeNb4_-65ahajy_D9MsopOTzelixTPPjczYYqBt0MADJc2l9HW_7C3QG0zKNNSbdFS7Pioil9dN3NzqwRQ7E6BbxcgZ2e6y5F2S09O3N4xrXs7TU6kWNaTPXfTlP16Dtuu17wWARkEu1jhlhKHpNGaJDalUCziwoFY9bXptVHiG-fDuTuvBewRDP4gcPvh6wW2fzv4unDENtLtPDdUckUlN1QeoIcBHBi6K8i0ix7y1XWjNlDsNtqsTB06-OdabtN6brNqtHZz-hQ9sWYJzgzGnqGRrHbQI9Oo9GoHPT62IRjP0U0POgwIwT3o8BB0eAg6PKuwBR3uQPcJZ7iDHNaQ0wSHkMMAOTyA3At0dnhwunfk2BYeDgfNaOVwrwhoLEALTwqfceJJRksaFaws4lAITxJGYi8pRUxIwMFa4X4MGjEhJShRHhgfL9FWVVdyF-HY42XISt-jPAxjv0hpkoJy6gtaFj4J5RiR9hPn3Na3V21WLvK_sXeMnG7WwtR3uWN80nIvtzqq0T1zAOQdMz-0zM5BhKt7OVbJurnMgyRJVLuVKBqjVwYF3VoIfAcKZtHre67zDdru_3Zv0dZq2ch3oD6vivcayL8AQLTGRw |
| linkProvider | Geneva Foundation for Medical Education and Research |
| openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Annotating+for+Artificial+Intelligence+Applications+in+Digital+Pathology%3A+A+Practical+Guide+for+Pathologists+and+Researchers&rft.jtitle=Modern+pathology&rft.au=Montezuma%2C+Diana&rft.au=Oliveira%2C+Sara+P.&rft.au=Neto%2C+Pedro+C.&rft.au=Oliveira%2C+Domingos&rft.date=2023-04-01&rft.issn=0893-3952&rft.volume=36&rft.issue=4&rft.spage=100086&rft_id=info:doi/10.1016%2Fj.modpat.2022.100086&rft.externalDBID=n%2Fa&rft.externalDocID=10_1016_j_modpat_2022_100086 |
| thumbnail_l | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=0893-3952&client=summon |
| thumbnail_m | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=0893-3952&client=summon |
| thumbnail_s | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=0893-3952&client=summon |