A Study of Oracle Approximations in Testing Deep Learning Libraries

Due to the increasing popularity of deep learning (DL) applications, testing DL libraries is becoming more and more important. Different from testing general software, for which output is often asserted definitely (e.g., an output is compared with an oracle for equality), testing deep learning libra...

Full description

Saved in:
Bibliographic Details
Published inIEEE/ACM International Conference on Automated Software Engineering : [proceedings] pp. 785 - 796
Main Authors Nejadgholi, Mahdi, Yang, Jinqiu
Format Conference Proceeding
LanguageEnglish
Published IEEE 01.11.2019
Subjects
Approximation algorithms
C++ languages
Deep learning
Libraries
Maintenance engineering
Python
Software Quality
Software Testing
Test Oracle
Testing
Testing Deep Learning Libraries
Online AccessGet full text
ISSN2643-1572
DOI10.1109/ASE.2019.00078

Cover

More Information
Summary:Due to the increasing popularity of deep learning (DL) applications, testing DL libraries is becoming more and more important. Different from testing general software, for which output is often asserted definitely (e.g., an output is compared with an oracle for equality), testing deep learning libraries often requires to perform oracle approximations, i.e., the output is allowed to be within a restricted range of the oracle. However, oracle approximation practices have not been studied in prior empirical work that focuses on traditional testing practices. The prevalence, common practices, maintenance and evolution challenges of oracle approximations remain unknown in literature. In this work, we study oracle approximation assertions implemented to test four popular DL libraries. Our study shows that there exists a non-negligible portion of assertions that leverage oracle approximation in the test cases of DL libraries. Also, we identify the common sources of oracles on which oracle approximations are being performed through a comprehensive manual study. Moreover, we find that developers frequently modify code related to oracle approximations, i.e., using a different approximation API, modifying the oracle or the output from the code under test, and using a different approximation threshold. Last, we performed an in-depth study to understand the reasons behind the evolution of oracle approximation assertions. Our findings reveal important maintenance challenges that developers may face when maintaining oracle approximation practices as code evolves in DL libraries.
ISSN:2643-1572
DOI:10.1109/ASE.2019.00078