A Survey of Domain Knowledge Elicitation in Applied Machine Learning
Eliciting knowledge from domain experts can play an important role throughout the machine learning process, from correctly specifying the task to evaluating model results. However, knowledge elicitation is also fraught with challenges. In this work, we consider why and how machine learning researche...
Saved in:
| Published in | Multimodal technologies and interaction Vol. 5; no. 12; p. 73 |
|---|---|
| Main Authors | , , |
| Format | Journal Article |
| Language | English |
| Published |
Basel
MDPI AG
01.12.2021
|
| Subjects | |
| Online Access | Get full text |
| ISSN | 2414-4088 2414-4088 |
| DOI | 10.3390/mti5120073 |
Cover
| Abstract | Eliciting knowledge from domain experts can play an important role throughout the machine learning process, from correctly specifying the task to evaluating model results. However, knowledge elicitation is also fraught with challenges. In this work, we consider why and how machine learning researchers elicit knowledge from experts in the model development process. We develop a taxonomy to characterize elicitation approaches according to the elicitation goal, elicitation target, elicitation process, and use of elicited knowledge. We analyze the elicitation trends observed in 28 papers with this taxonomy and identify opportunities for adding rigor to these elicitation approaches. We suggest future directions for research in elicitation for machine learning by highlighting avenues for further exploration and drawing on what we can learn from elicitation research in other fields. |
|---|---|
| AbstractList | Eliciting knowledge from domain experts can play an important role throughout the machine learning process, from correctly specifying the task to evaluating model results. However, knowledge elicitation is also fraught with challenges. In this work, we consider why and how machine learning researchers elicit knowledge from experts in the model development process. We develop a taxonomy to characterize elicitation approaches according to the elicitation goal, elicitation target, elicitation process, and use of elicited knowledge. We analyze the elicitation trends observed in 28 papers with this taxonomy and identify opportunities for adding rigor to these elicitation approaches. We suggest future directions for research in elicitation for machine learning by highlighting avenues for further exploration and drawing on what we can learn from elicitation research in other fields. |
| Author | Kerrigan, Daniel Hullman, Jessica Bertini, Enrico |
| Author_xml | – sequence: 1 givenname: Daniel surname: Kerrigan fullname: Kerrigan, Daniel – sequence: 2 givenname: Jessica surname: Hullman fullname: Hullman, Jessica – sequence: 3 givenname: Enrico surname: Bertini fullname: Bertini, Enrico |
| BookMark | eNp9kE1PGzEQhi0EUink0l-wEreigL_W9h4joCUiiANwtmbt2eBoYy_eTVH-PUtDS1VVPXnkeebRzPuZ7McUkZAvjJ4JUdHz9RBKxinVYo8ccsnkVFJj9v-oP5FJ368opZxJqqk5JJez4n6Tf-C2SE1xmdYQYnET00uLfonFVRtcGGAIKRZjY9Z1bUBf3IJ7ChGLBUKOIS6PyUEDbY-T9_eIPH67eri4ni7uvs8vZoupE4oN04YKw8B4g4JDzbBBcNroklelLFVJvWpQK1l5QCmU0ViNZOVNpXxZ1wrEEZnvvD7BynY5rCFvbYJgf36kvLSQh-BatMrU3GvusHJeKtkYcEarxkghfQ3Mja7TnWsTO9i-QNv-FjJq3_K0H3mO9MmO7nJ63mA_2FXa5Dgea7li3HBDlR4puqNcTn2fsbG_0hsyhPbf4q9_jfxni1c0WpL- |
| CitedBy_id | crossref_primary_10_1103_PRXLife_3_013005 crossref_primary_10_1111_cgf_14823 crossref_primary_10_1002_sta4_70054 crossref_primary_10_3389_fped_2023_1005579 crossref_primary_10_1145_3580887 crossref_primary_10_1109_TVCG_2023_3326591 crossref_primary_10_3389_fped_2022_1008840 crossref_primary_10_1002_oby_24258 crossref_primary_10_3233_THC_230809 crossref_primary_10_3390_app142411612 crossref_primary_10_1016_j_cie_2022_108120 crossref_primary_10_1016_j_giq_2024_101976 |
| Cites_doi | 10.1017/CBO9780511816796.010 10.1198/016214505000000105 10.1097/NNA.0b013e3182942c3c 10.1109/TSMCB.2011.2148197 10.1016/j.media.2021.102062 10.1145/3351095.3372827 10.1016/j.cogpsych.2005.05.004 10.1145/3313831.3376219 10.1145/3359206 10.14778/3157794.3157797 10.1145/3290605.3300830 10.1037//0033-295X.104.2.367 10.1016/j.eswa.2017.01.028 10.1016/j.envsoft.2011.03.003 10.1145/3290605.3300912 10.1093/bioinformatics/bty257 10.1016/S0022-5371(83)90189-5 10.1145/1978942.1978966 10.1007/978-3-030-05297-3 10.1109/TVCG.2018.2864769 10.1016/j.apenergy.2018.10.107 10.1007/BF00994016 10.1109/TVCG.2011.185 10.1023/A:1008193214890 10.1017/S1930297500004940 10.2196/preprints.19612 10.1145/2702123.2702594 10.1080/03610929508831616 10.1016/j.ijar.2013.03.009 10.1109/TAMD.2010.2051030 10.1126/sciadv.aba9338 10.1145/3392878 10.1007/s10994-017-5651-7 10.1145/3290605.3300234 10.1016/S0169-2070(99)00018-7 10.1167/tvst.8.6.40 10.1016/j.artint.2007.09.009 10.1145/2157689.2157693 10.1016/0950-7051(96)01033-7 10.1002/0470033312 10.1001/jamapsychiatry.2017.0298 10.1111/1468-0394.00159 10.1145/3134664 10.31219/osf.io/uvjqh 10.1080/00031305.2018.1518265 10.1017/9781316480748 10.1145/3368555.3384452 10.4338/ACI-2015-11-RA-0161 10.1016/j.future.2022.05.014 10.24963/ijcai.2019/271 10.1007/978-3-319-65052-4 10.1109/TVCG.2017.2743898 10.1207/s15516709cog0502_2 10.1093/oso/9780195064650.001.0001 10.1145/3196709.3196729 10.1162/99608f92.3ab8a587 10.1145/3361118 10.1145/3172944.3172989 10.1145/1719970.1720006 10.1037/a0017201 |
| ContentType | Journal Article |
| Copyright | 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License. |
| Copyright_xml | – notice: 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License. |
| DBID | AAYXX CITATION 8FE 8FG ABUWG AFKRA ARAPS AZQEC BENPR BGLVJ CCPQU DWQXO HCIFZ P5Z P62 PHGZM PHGZT PIMPY PKEHL PQEST PQGLB PQQKQ PQUKI PRINS ADTOC UNPAY DOA |
| DOI | 10.3390/mti5120073 |
| DatabaseName | CrossRef ProQuest SciTech Collection ProQuest Technology Collection ProQuest Central (Alumni) ProQuest Central UK/Ireland Advanced Technologies & Computer Science Collection ProQuest Central Essentials ProQuest Central Technology collection ProQuest One Community College ProQuest Central SciTech Premium Collection Advanced Technologies & Aerospace Database ProQuest Advanced Technologies & Aerospace Collection ProQuest Central Premium ProQuest One Academic (New) Publicly Available Content Database ProQuest One Academic Middle East (New) ProQuest One Academic Eastern Edition (DO NOT USE) ProQuest One Applied & Life Sciences ProQuest One Academic ProQuest One Academic UKI Edition ProQuest Central China Unpaywall for CDI: Periodical Content Unpaywall DOAJ Open Access Full Text |
| DatabaseTitle | CrossRef Publicly Available Content Database Advanced Technologies & Aerospace Collection Technology Collection ProQuest One Academic Middle East (New) ProQuest Advanced Technologies & Aerospace Collection ProQuest Central Essentials ProQuest One Academic Eastern Edition ProQuest Central (Alumni Edition) SciTech Premium Collection ProQuest One Community College ProQuest Technology Collection ProQuest SciTech Collection ProQuest Central China ProQuest Central Advanced Technologies & Aerospace Database ProQuest One Applied & Life Sciences ProQuest One Academic UKI Edition ProQuest Central Korea ProQuest Central (New) ProQuest One Academic ProQuest One Academic (New) |
| DatabaseTitleList | Publicly Available Content Database CrossRef |
| Database_xml | – sequence: 1 dbid: DOA name: DOAJ Directory of Open Access Journals url: https://www.doaj.org/ sourceTypes: Open Website – sequence: 2 dbid: UNPAY name: Unpaywall url: https://proxy.k.utb.cz/login?url=https://unpaywall.org/ sourceTypes: Open Access Repository – sequence: 3 dbid: 8FG name: ProQuest Technology Collection url: https://search.proquest.com/technologycollection1 sourceTypes: Aggregation Database |
| DeliveryMethod | fulltext_linktorsrc |
| Discipline | Computer Science |
| EISSN | 2414-4088 |
| ExternalDocumentID | oai_doaj_org_article_68b2d72ce9cd464f8ac876f8434dba1c 10.3390/mti5120073 10_3390_mti5120073 |
| GroupedDBID | 8FE 8FG AADQD AAFWJ AAYXX ADBBV AFKRA AFPKN AFZYC ALMA_UNASSIGNED_HOLDINGS ARAPS BCNDV BENPR BGLVJ CCPQU CITATION GROUPED_DOAJ HCIFZ IAO MODMG M~E OK1 P62 PHGZM PHGZT PIMPY PQGLB PROAC ABUWG AZQEC DWQXO PKEHL PQEST PQQKQ PQUKI PRINS PUEGO ADTOC ITC UNPAY |
| ID | FETCH-LOGICAL-c361t-f0381a8d8e32ab1efeac787529545650d6fe7649dae43687e98e39d896d5bb6a3 |
| IEDL.DBID | BENPR |
| ISSN | 2414-4088 |
| IngestDate | Fri Oct 03 12:44:00 EDT 2025 Sun Oct 26 03:45:44 EDT 2025 Sun Sep 07 03:42:51 EDT 2025 Thu Oct 16 04:31:25 EDT 2025 Thu Apr 24 22:57:34 EDT 2025 |
| IsDoiOpenAccess | true |
| IsOpenAccess | true |
| IsPeerReviewed | true |
| IsScholarly | true |
| Issue | 12 |
| Language | English |
| License | cc-by |
| LinkModel | DirectLink |
| MergedId | FETCHMERGED-LOGICAL-c361t-f0381a8d8e32ab1efeac787529545650d6fe7649dae43687e98e39d896d5bb6a3 |
| Notes | ObjectType-Article-1 SourceType-Scholarly Journals-1 ObjectType-Feature-2 content type line 14 |
| OpenAccessLink | https://www.proquest.com/docview/2612828067?pq-origsite=%requestingapplication%&accountid=15518 |
| PQID | 2612828067 |
| PQPubID | 2059548 |
| ParticipantIDs | doaj_primary_oai_doaj_org_article_68b2d72ce9cd464f8ac876f8434dba1c unpaywall_primary_10_3390_mti5120073 proquest_journals_2612828067 crossref_citationtrail_10_3390_mti5120073 crossref_primary_10_3390_mti5120073 |
| PublicationCentury | 2000 |
| PublicationDate | 2021-12-01 |
| PublicationDateYYYYMMDD | 2021-12-01 |
| PublicationDate_xml | – month: 12 year: 2021 text: 2021-12-01 day: 01 |
| PublicationDecade | 2020 |
| PublicationPlace | Basel |
| PublicationPlace_xml | – name: Basel |
| PublicationTitle | Multimodal technologies and interaction |
| PublicationYear | 2021 |
| Publisher | MDPI AG |
| Publisher_xml | – name: MDPI AG |
| References | Cartwright (ref_3) 2017; 1 Cai (ref_39) 2019; 3 Mao (ref_37) 2019; 3 ref_57 ref_56 ref_11 ref_55 ref_54 ref_53 ref_52 ref_51 Sundin (ref_7) 2018; 34 Garthwaite (ref_43) 2005; 100 Thomaz (ref_50) 2008; 172 ref_18 ref_17 ref_16 ref_15 Martinelli (ref_29) 1999; 26 ref_61 ref_60 Amershi (ref_48) 2014; 35 Chi (ref_41) 1981; 5 Hu (ref_27) 2019; 235 ref_69 ref_68 ref_23 ref_21 ref_65 ref_62 Griffiths (ref_66) 2005; 51 Bostock (ref_70) 2011; 17 Goldstein (ref_44) 2014; 9 Bowles (ref_12) 2013; 43 (ref_5) 2019; 73 Masegosa (ref_19) 2013; 54 Wagner (ref_46) 2017; 76 Webb (ref_28) 1996; 9 Heckerman (ref_14) 1995; 20 Cakmak (ref_49) 2010; 2 Kehlbeck (ref_25) 2020; 26 Langseth (ref_22) 2003; 4 Griffiths (ref_67) 2009; 116 ref_36 ref_35 ref_34 ref_33 ref_32 Wagner (ref_45) 2001; 18 Ustun (ref_10) 2017; 74 Daee (ref_26) 2017; 106 Bowles (ref_13) 2016; 7 Clark (ref_58) 1983; 22 (ref_59) 2012; 36 ref_47 Sperrle (ref_24) 2019; 25 Hong (ref_38) 2020; 4 ref_42 Ustun (ref_31) 2019; 20 ref_40 ref_1 ref_2 Cano (ref_20) 2011; 41 ref_9 Rowe (ref_63) 1999; 15 Ratner (ref_30) 2017; 11 Hullman (ref_64) 2017; 24 ref_4 ref_6 Madigan (ref_8) 1995; 24 |
| References_xml | – ident: ref_1 doi: 10.1017/CBO9780511816796.010 – volume: 100 start-page: 680 year: 2005 ident: ref_43 article-title: Statistical methods for eliciting probability distributions publication-title: J. Am. Stat. Assoc. doi: 10.1198/016214505000000105 – volume: 43 start-page: 355 year: 2013 ident: ref_12 article-title: Conducting research using the electronic health record across multi-hospital systems: Semantic harmonization implications for administrators publication-title: J. Nurs. Adm. doi: 10.1097/NNA.0b013e3182942c3c – volume: 41 start-page: 1382 year: 2011 ident: ref_20 article-title: A Method for Integrating Expert Knowledge When Learning Bayesian Networks From Data publication-title: IEEE Trans. Syst. Man Cybern. Part B doi: 10.1109/TSMCB.2011.2148197 – ident: ref_53 doi: 10.1016/j.media.2021.102062 – ident: ref_11 doi: 10.1145/3351095.3372827 – volume: 51 start-page: 334 year: 2005 ident: ref_66 article-title: Structure and strength in causal induction publication-title: Cogn. Psychol. doi: 10.1016/j.cogpsych.2005.05.004 – ident: ref_35 doi: 10.1145/3313831.3376219 – volume: 3 start-page: 1 year: 2019 ident: ref_39 article-title: “Hello AI”: Uncovering the Onboarding Needs of Medical Practitioners for Human-AI Collaborative Decision-Making publication-title: Proc. ACM Hum.-Comput. Interact. doi: 10.1145/3359206 – volume: 11 start-page: 269 year: 2017 ident: ref_30 article-title: Snorkel: Rapid Training Data Creation with Weak Supervision publication-title: Proc. VLDB Endow. doi: 10.14778/3157794.3157797 – ident: ref_34 doi: 10.1145/3290605.3300830 – ident: ref_65 doi: 10.1037//0033-295X.104.2.367 – volume: 76 start-page: 85 year: 2017 ident: ref_46 article-title: Trends in expert system development: A longitudinal content analysis of over thirty years of expert system case studies publication-title: Expert Syst. Appl. doi: 10.1016/j.eswa.2017.01.028 – ident: ref_42 – volume: 36 start-page: 35 year: 2012 ident: ref_59 article-title: Probabilistic Uncertainty Specification: Overview, Elaboration Techniques and Their Application to a Mechanistic Model of Carbon Flux publication-title: Environ. Model. Softw. doi: 10.1016/j.envsoft.2011.03.003 – ident: ref_69 doi: 10.1145/3290605.3300912 – volume: 34 start-page: i395 year: 2018 ident: ref_7 article-title: Improving genomics-based predictions for precision medicine through active elicitation of expert knowledge publication-title: Bioinformatics doi: 10.1093/bioinformatics/bty257 – volume: 22 start-page: 245 year: 1983 ident: ref_58 article-title: Common ground at the understanding of demonstrative reference publication-title: J. Verbal Learn. Verbal Behav. doi: 10.1016/S0022-5371(83)90189-5 – ident: ref_32 doi: 10.1145/1978942.1978966 – ident: ref_17 doi: 10.1007/978-3-030-05297-3 – volume: 25 start-page: 374 year: 2019 ident: ref_24 article-title: Visual Analytics for Topic Model Optimization based on User-Steerable Speculative Execution publication-title: IEEE Trans. Vis. Comput. Graph. doi: 10.1109/TVCG.2018.2864769 – volume: 235 start-page: 117 year: 2019 ident: ref_27 article-title: Design of machine learning models with domain experts for automated sensor selection for energy fault detection publication-title: Appl. Energy doi: 10.1016/j.apenergy.2018.10.107 – ident: ref_56 – volume: 20 start-page: 197 year: 1995 ident: ref_14 article-title: Learning Bayesian Networks: The Combination of Knowledge and Statistical Data publication-title: Mach. Learn. doi: 10.1007/BF00994016 – volume: 17 start-page: 2301 year: 2011 ident: ref_70 article-title: D3 Data-Driven Documents publication-title: IEEE Trans. Vis. Comput. Graph. doi: 10.1109/TVCG.2011.185 – volume: 26 start-page: 325 year: 1999 ident: ref_29 article-title: Application of Machine Learning in Water Distribution Networks Assisted by Domain Experts publication-title: J. Intell. Robot. Syst. doi: 10.1023/A:1008193214890 – volume: 9 start-page: 1 year: 2014 ident: ref_44 article-title: Lay understanding of probability distributions publication-title: Judgm. Decis. Mak. doi: 10.1017/S1930297500004940 – ident: ref_47 doi: 10.2196/preprints.19612 – ident: ref_55 doi: 10.1145/2702123.2702594 – volume: 24 start-page: 2271 year: 1995 ident: ref_8 article-title: Eliciting prior information to enhance the predictive performance of bayesian graphical models publication-title: Commun. Stat.-Theory Methods doi: 10.1080/03610929508831616 – volume: 54 start-page: 1168 year: 2013 ident: ref_19 article-title: An interactive approach for Bayesian network learning using domain/expert knowledge publication-title: Int. J. Approx. Reason. doi: 10.1016/j.ijar.2013.03.009 – volume: 4 start-page: 339 year: 2003 ident: ref_22 article-title: Fusion of Domain Knowledge with Data for Structural Learning in Object Oriented Domains publication-title: J. Mach. Learn. Res. – volume: 26 start-page: 1001 year: 2020 ident: ref_25 article-title: Semantic Concept Spaces: Guided Topic Model Refinement using Word-Embedding Projections publication-title: IEEE Trans. Vis. Comput. Graph. – volume: 35 start-page: 105 year: 2014 ident: ref_48 article-title: Power to the people: The role of humans in interactive machine learning publication-title: AI Mag. – volume: 2 start-page: 108 year: 2010 ident: ref_49 article-title: Designing interactions for robot active learners publication-title: IEEE Trans. Auton. Ment. Dev. doi: 10.1109/TAMD.2010.2051030 – ident: ref_9 doi: 10.1126/sciadv.aba9338 – volume: 4 start-page: 1 year: 2020 ident: ref_38 article-title: Human Factors in Model Interpretability: Industry Practices, Challenges, and Needs publication-title: Proc. ACM Hum.-Comput. Interact. doi: 10.1145/3392878 – volume: 106 start-page: 1599 year: 2017 ident: ref_26 article-title: Knowledge elicitation via sequential probabilistic inference for high-dimensional prediction publication-title: Mach. Learn. doi: 10.1007/s10994-017-5651-7 – ident: ref_15 doi: 10.1145/3290605.3300234 – volume: 15 start-page: 353 year: 1999 ident: ref_63 article-title: The Delphi technique as a forecasting tool: Issues and analysis publication-title: Int. J. Forecast. doi: 10.1016/S0169-2070(99)00018-7 – ident: ref_18 doi: 10.1167/tvst.8.6.40 – volume: 172 start-page: 716 year: 2008 ident: ref_50 article-title: Teachable robots: Understanding human teaching behavior to build more effective robot learners publication-title: Artif. Intell. doi: 10.1016/j.artint.2007.09.009 – ident: ref_4 doi: 10.1145/2157689.2157693 – volume: 9 start-page: 253 year: 1996 ident: ref_28 article-title: Integrating machine learning with knowledge acquisition through direct interaction with domain experts publication-title: Knowl.-Based Syst. doi: 10.1016/0950-7051(96)01033-7 – ident: ref_2 doi: 10.1002/0470033312 – volume: 74 start-page: 520 year: 2017 ident: ref_10 article-title: The World Health Organization Adult Attention-Deficit/Hyperactivity Disorder Self-Report Screening Scale for DSM-5 publication-title: JAMA Psychiatry doi: 10.1001/jamapsychiatry.2017.0298 – ident: ref_21 – volume: 18 start-page: 76 year: 2001 ident: ref_45 article-title: Selection of knowledge acquisition techniques based upon the problem domain characteristics of production and operations management expert systems publication-title: Expert Syst. doi: 10.1111/1468-0394.00159 – volume: 1 start-page: 1 year: 2017 ident: ref_3 article-title: Seeing Sound: Investigating the Effects of Visualizations and Complexity on Crowdsourced Audio Annotations publication-title: Proc. ACM Hum.-Comput. Interact. doi: 10.1145/3134664 – ident: ref_36 doi: 10.31219/osf.io/uvjqh – volume: 73 start-page: 69 year: 2019 ident: ref_5 article-title: Expert Knowledge Elicitation: Subjective but Scientific publication-title: Am. Stat. doi: 10.1080/00031305.2018.1518265 – ident: ref_40 doi: 10.1017/9781316480748 – ident: ref_16 doi: 10.1145/3368555.3384452 – volume: 7 start-page: 368 year: 2016 ident: ref_13 article-title: Using Electronic Case Summaries to Elicit Multi-Disciplinary Expert Knowledge about Referrals to Post-Acute Care publication-title: Appl. Clin. Inform. doi: 10.4338/ACI-2015-11-RA-0161 – ident: ref_33 – ident: ref_54 doi: 10.1016/j.future.2022.05.014 – ident: ref_23 doi: 10.24963/ijcai.2019/271 – ident: ref_61 doi: 10.1007/978-3-319-65052-4 – volume: 24 start-page: 446 year: 2017 ident: ref_64 article-title: Imagining replications: Graphical prediction & discrete visualizations improve recall & estimation of effect uncertainty publication-title: IEEE Trans. Vis. Comput. Graph. doi: 10.1109/TVCG.2017.2743898 – volume: 5 start-page: 121 year: 1981 ident: ref_41 article-title: Categorization and representation of physics problems by experts and novices publication-title: Cogn. Sci. doi: 10.1207/s15516709cog0502_2 – ident: ref_62 doi: 10.1093/oso/9780195064650.001.0001 – ident: ref_6 doi: 10.1145/3196709.3196729 – ident: ref_68 doi: 10.1162/99608f92.3ab8a587 – volume: 3 start-page: 1 year: 2019 ident: ref_37 article-title: How Data ScientistsWork Together with Domain Experts in Scientific Collaborations: To Find the Right Answer or to Ask the Right Question? publication-title: Proc. ACM Hum.-Comput. Interact. doi: 10.1145/3361118 – ident: ref_52 doi: 10.1145/3172944.3172989 – ident: ref_60 – ident: ref_51 doi: 10.1145/1719970.1720006 – volume: 116 start-page: 661 year: 2009 ident: ref_67 article-title: Theory-based causal induction publication-title: Psychol. Rev. doi: 10.1037/a0017201 – ident: ref_57 – volume: 20 start-page: 1 year: 2019 ident: ref_31 article-title: Learning Optimized Risk Scores publication-title: J. Mach. Learn. Res. |
| SSID | ssj0002140708 |
| Score | 2.3087125 |
| Snippet | Eliciting knowledge from domain experts can play an important role throughout the machine learning process, from correctly specifying the task to evaluating... |
| SourceID | doaj unpaywall proquest crossref |
| SourceType | Open Website Open Access Repository Aggregation Database Enrichment Source Index Database |
| StartPage | 73 |
| SubjectTerms | Artificial intelligence Collaboration Computer science Crowdsourcing Decision making domain expert domain knowledge Domains elicitation expert knowledge Interviews Knowledge acquisition Machine learning Researchers Scientists Subject specialists Taxonomy |
| SummonAdditionalLinks | – databaseName: DOAJ Open Access Full Text dbid: DOA link: http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwrV1LS8NAEF6kF734FqtVFuzFQ2iym8fusdqWotSLFnoL-5RCmpaaKv337iabWkH04jWZwzCTmfkmM3wDQBv5CmFFtB39S9Og0MhjKBCeNlBbM4ElKxlvRk_xcBw-TKLJ1qkvuxNW0QNXhuvEhCOZIKGokGEcasKECWBNQhxKzgJhs69P6FYzZXMwMn1D4pOKjxSbvr4zK6amttnB1LcKVBL1f0OXu6t8wdYfLMu2Cs3gEOw7hAi7lWZHYEflx-Cgvr4AXTCegF4XPq-W72oN5xr25jPT4sPH-g8Z7GdT4ei3oXnhwCYclbuTCjpa1ddTMB70X-6HnruJ4AkcB4Wn7WSPEUkURowHSpvEaWKuHNcZbObLWKskDqlkynLLJ4oaSSoJjWXEeczwGWjk81ydA8iDKOFCcM0FDmmiCZYGCyGOeMJ5RFUT3NZ2SmuN7d2KLDWNg7Vp-mXTJrjZyC4qmowfpe6suTcSltq6fGAcnjqHp385vAlatbNSF29vqSVCI3ZInDRBe-PAX1S5-A9VLsEesisu5XZLCzSK5UpdGYxS8Ovyc_wEIl_k5A priority: 102 providerName: Directory of Open Access Journals – databaseName: Unpaywall dbid: UNPAY link: http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwnV1LaxRBEC50c9CLiS9ck0iDuXiYbKZ7ph8nWfMgKAmCLsTT0M-wuJlddmcT4q-3erZnY0REvE5XQw9V1fVVV_EVwB498JR5GWLp32GCospM09xmAaF20JY53TLenJ3z01Hx8aK8SL05i9RWian4uL2kMboUuB9ztHKQ04Fgg5kL76_TQxIiCYFgWSr5EDZ4iVC8Bxuj88_Db3GgXLd1RUnKMLUfXDVjDG-xNnUvCLVc_fcA5qNlPdO3N3oy-SXWnGyuBqouWorC2GLyfX_ZmH374zcCx__-jS14klAoGa7M5ik88PUz2OwmPJDk8M_haEi-LOfX_pZMAzmaXulxTT51r3DkeDK2ieKb4EICtOSs7c_0JFG3Xr6A0cnx18PTLM1dyCzjeZOFWD3U0knPqDa5D3g5o1-3JUHEfweOBy94oZz2kb9eeIWSyknFXWkM1-wl9Opp7V8BMXkpjLUmGMsKJYJkDvEWNdQIY0rl-_CuU0TVnTjOxphUmJxEpVV3SuvD27XsbEXF8UepD1Gfa4lIn91-mM4vq-SNFZeGOkGtV9YVvAhSW4wKQRascEbntg87nTVUyacXVSRbk7EQLfqwt7aQvxzl9b-JbcNjGhtl2h6ZHeg186XfRaTTmDfJnn8Cymb4lQ priority: 102 providerName: Unpaywall |
| Title | A Survey of Domain Knowledge Elicitation in Applied Machine Learning |
| URI | https://www.proquest.com/docview/2612828067 https://www.mdpi.com/2414-4088/5/12/73/pdf?version=1637768898 https://doaj.org/article/68b2d72ce9cd464f8ac876f8434dba1c |
| UnpaywallVersion | publishedVersion |
| Volume | 5 |
| hasFullText | 1 |
| inHoldings | 1 |
| isFullTextHit | |
| isPrint | |
| journalDatabaseRights | – providerCode: PRVAON databaseName: DOAJ Directory of Open Access Journals customDbUrl: eissn: 2414-4088 dateEnd: 99991231 omitProxy: true ssIdentifier: ssj0002140708 issn: 2414-4088 databaseCode: DOA dateStart: 20170101 isFulltext: true titleUrlDefault: https://www.doaj.org/ providerName: Directory of Open Access Journals – providerCode: PRVHPJ databaseName: ROAD: Directory of Open Access Scholarly Resources customDbUrl: eissn: 2414-4088 dateEnd: 99991231 omitProxy: true ssIdentifier: ssj0002140708 issn: 2414-4088 databaseCode: M~E dateStart: 20160101 isFulltext: true titleUrlDefault: https://road.issn.org providerName: ISSN International Centre – providerCode: PRVPQU databaseName: ProQuest Central - New (Subscription) customDbUrl: http://www.proquest.com/pqcentral?accountid=15518 eissn: 2414-4088 dateEnd: 99991231 omitProxy: true ssIdentifier: ssj0002140708 issn: 2414-4088 databaseCode: BENPR dateStart: 20170301 isFulltext: true titleUrlDefault: https://www.proquest.com/central providerName: ProQuest – providerCode: PRVPQU databaseName: ProQuest Technology Collection customDbUrl: eissn: 2414-4088 dateEnd: 99991231 omitProxy: true ssIdentifier: ssj0002140708 issn: 2414-4088 databaseCode: 8FG dateStart: 20170301 isFulltext: true titleUrlDefault: https://search.proquest.com/technologycollection1 providerName: ProQuest |
| link | http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwfV1LSwMxEB60HvTiW6zWEtCLh8Vuso_sQaTVVlEsohb0tOQpQu3W2ir-e5M0WxXE4-4OS5hkJt9kJt8AHOCGwkRRbVP_0gQoWRwwHIpAG6itmSCSOcab625y0YsuH-KHOeiWd2FsWWXpE52jloWwZ-RHluqK2jRgejJ8DWzXKJtdLVtoMN9aQR47irF5WMCWGasCC6129-Z2duqCTTyRNuiUp5SYeP_oZfxs9jybsPq1MzkC_1-oc3EyGLLPD9bv_9iAOquw7JEjak6neg3m1GAdVsquDMgb6QacNdHdZPSuPlGh0VnxYkJ_dFWenKF2_1l4Wm5kPngQiq5dTaVCnm71aRN6nfb96UXgeyUEgiThONA248eopIpgxkOljUM1tujSeAazNWSiVZpEmWTKcs6nKjOSmaRZImPOE0a2oDIoBmobEA_jlAvBNRckylJNiTQYCXPMU87jTFXhsNRTXo7Y9rPo5yagsDrNv3Vahf2Z7HBKn_GnVMuqeyZhKa_di2L0lHsLyhPKsUyxUJmQURJpyoTx5JpGJJKchaIKtXKycm-Hb_n3qqnCwWwC_xnKzv9_2YUlbItaXD1LDSrj0UTtGVQy5nWYp53zul9wdRfbm6de96b5-AWwI-V8 |
| linkProvider | ProQuest |
| linkToHtml | http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwtV1LTxsxEB5RONBLW_pQ09JiqfTQw4qsvQ_7gCpogkJDoqoFidvWTxQp7KYhKcqf62_reOMNICFuXHdHljUznodn_A3ALm1byix3vvRvMEERaSRprCOHobaTmhlZI94MhlnvLPl-np6vwb_mLYxvq2xsYm2oTaX9Hfmeh7rivgyYf538ifzUKF9dbUZoyDBawezXEGPhYUffLq4xhbvaP-6gvD9TetQ9_daLwpSBSLMsnkXO18okN9wyKlVsHZoi1OK6AIbRTttkzuZZIoy0Hq09twIpheEiM6lSmWS47hPYSFgiMPnbOOwOf_xc3fJQzF_yNl_iojIm2nuXsxH6WF8gu-MJ64EBd6LczXk5kYtrOR7fcnhHL-BZiFTJwVK1tmDNli_heTMFggSj8Ao6B-TXfPrXLkjlSKe6lKOS9JubOtIdj3SAASf4IwS9ZFD3cFoS4F0vXsPZo3DtDayXVWnfAlFxmiutlVPa89BxZjAmo4qqXKlU2BZ8afhUNDv28zPGBSYwnqfFDU9b8GlFO1nCddxLdejZvaLwENv1h2p6UYQTW2RcUZNTbYU2SZY4LjV6DsdR0EbJWLdguxFWEc79VXGjpS3YXQnwga28e3iVHdjsnQ5OipPjYf89PKW-oabupdmG9dl0bj9gRDRTH4PaEfj92Jr-HyFyHxY |
| linkToPdf | http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwtV1LbxMxEB6VIgEXylMEWrBEOXBYJWvvru0DQm3T0BJaIUGl3rZ-VpHS3ZAmVPlr_DrGG29KJdRbr7sjyxqP5-1vALZpz1HmhA-lf4sBiswTRVOTeHS1vTLMqgbx5ui4ODjJvp7mp2vwp30LE9oqW53YKGpbm5Aj7waoKxHKgLzrY1vE9_7g8-RXEiZIhUprO05jKSJDt7jC8O3y02Efz_oDpYP9n3sHSZwwkBhWpLPEhzqZElY4RpVOnUc1hBLcFL_Q0-nZwjteZNIqF5DauZNIKa2Qhc21LhTDde_BfR5Q3MMr9cGXVX6HYuTCe2KJiMqY7HUvZiO0rqE0dsMGNqMCbvi3D-fVRC2u1Hj8j6kbPIHH0UclO0uhegprrnoGG-38BxLVwXPo75Af8-lvtyC1J_36Qo0qMmxzdGR_PDIRAJzgj-jukqOme9ORCOx6_gJO7oRnL2G9qiv3CohOc66N0V4blknuBbPojVFNNdc6l64DH1s-le2Ow-SMcYmhS-Bpec3TDrxf0U6WQB3_pdoN7F5RBHDt5kM9PS_jXS0Loanl1DhpbFZkXiiDNsOLjGVWq9R0YLM9rDLe-MvyWj47sL06wFu28vr2Vd7BA5Tv8tvh8fANPKKhk6ZpotmE9dl07rbQFZrpt43METi7ayH_C00ZHLA |
| linkToUnpaywall | http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwnV1LaxRBEC50c9CLiS9ck0iDuXiYbKZ7ph8nWfMgKAmCLsTT0M-wuJlddmcT4q-3erZnY0REvE5XQw9V1fVVV_EVwB498JR5GWLp32GCospM09xmAaF20JY53TLenJ3z01Hx8aK8SL05i9RWian4uL2kMboUuB9ztHKQ04Fgg5kL76_TQxIiCYFgWSr5EDZ4iVC8Bxuj88_Db3GgXLd1RUnKMLUfXDVjDG-xNnUvCLVc_fcA5qNlPdO3N3oy-SXWnGyuBqouWorC2GLyfX_ZmH374zcCx__-jS14klAoGa7M5ik88PUz2OwmPJDk8M_haEi-LOfX_pZMAzmaXulxTT51r3DkeDK2ieKb4EICtOSs7c_0JFG3Xr6A0cnx18PTLM1dyCzjeZOFWD3U0knPqDa5D3g5o1-3JUHEfweOBy94oZz2kb9eeIWSyknFXWkM1-wl9Opp7V8BMXkpjLUmGMsKJYJkDvEWNdQIY0rl-_CuU0TVnTjOxphUmJxEpVV3SuvD27XsbEXF8UepD1Gfa4lIn91-mM4vq-SNFZeGOkGtV9YVvAhSW4wKQRascEbntg87nTVUyacXVSRbk7EQLfqwt7aQvxzl9b-JbcNjGhtl2h6ZHeg186XfRaTTmDfJnn8Cymb4lQ |
| openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+Survey+of+Domain+Knowledge+Elicitation+in+Applied+Machine+Learning&rft.jtitle=Multimodal+technologies+and+interaction&rft.au=Kerrigan%2C+Daniel&rft.au=Hullman%2C+Jessica&rft.au=Bertini%2C+Enrico&rft.date=2021-12-01&rft.pub=MDPI+AG&rft.eissn=2414-4088&rft.volume=5&rft.issue=12&rft.spage=73&rft_id=info:doi/10.3390%2Fmti5120073&rft.externalDBID=HAS_PDF_LINK |
| thumbnail_l | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=2414-4088&client=summon |
| thumbnail_m | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=2414-4088&client=summon |
| thumbnail_s | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=2414-4088&client=summon |