Goal-based composition of scalable hybrid analytics for heterogeneous architectures

• Published in: Journal of parallel and distributed computing Vol. 108; pp. 59 - 73
• Main Authors: Coetzee, P., Jarvis, S.A.
• Format: Journal Article
• Language: English
• Published: Elsevier Inc 01.10.2017
• Subjects: Analytic planning; Data intensive computing; Data science; Hadoop; Heterogeneous compute; Hybrid analytics; Streaming analysis; Streaming analysis; Hybrid analytics; Heterogeneous compute; Hadoop; Data intensive computing; Data science; Analytic planning
• ISSN: 0743-7315; 1096-0848; 1096-0848
• DOI: 10.1016/j.jpdc.2016.11.009

Crafting scalable analytics in order to extract actionable business intelligence is a challenging endeavour, requiring multiple layers of expertise and experience. Often, this expertise is irreconcilably split between an organisation’s engineers and subject matter domain experts. Previous approaches to this problem have relied on technically adept users with tool-specific training. Such an approach has a number of challenges: Expertise — There are few data-analytic subject domain experts with in-depth technical knowledge of compute architectures; Performance — Analysts do not generally make full use of the performance and scalability capabilities of the underlying architectures; Heterogeneity — calculating the most performant and scalable mix of real-time (on-line) and batch (off-line) analytics in a problem domain is difficult; Tools — Supporting frameworks will often direct several tasks, including, composition, planning, code generation, validation, performance tuning and analysis, but do not typically provide end-to-end solutions embedding all of these activities. In this paper, we present a novel semi-automated approach to the composition, planning, code generation and performance tuning of scalable hybrid analytics, using a semantically rich type system which requires little programming expertise from the user. This approach is the first of its kind to permit domain experts with little or no technical expertise to assemble complex and scalable analytics, for hybrid on- and off-line analytic environments, with no additional requirement for low-level engineering support. This paper describes (i) an abstract model of analytic assembly and execution, (ii) goal-based planning and (iii) code generation for hybrid on- and off-line analytics. An implementation, through a system which we call Mendeleev, is used to (iv) demonstrate the applicability of this technique through a series of case studies, where a single interface is used to create analytics that can be run simultaneously over on- and off-line environments. Finally, we (v) analyse the performance of the planner, and (vi) show that the performance of Mendeleev’s generated code is comparable with that of hand-written analytics. •A new abstract model of assembly and execution for arbitrary analytics, centred around a semantically rich type system.•Goal-based planning of hybrid analytic applications using this abstract model, requiring little programming ability from the user.•Automatic code generation across scalable compute architectures, integrating heterogeneous on- and off-line runtime environments.•Validation of the planning approach through its application to four case studies in telecommunications and image analysis, including an exploration of the performance and scalability of the planning engine for each of these case studies.•A demonstration of comparable performance with equivalent hand-written alternatives in both on- and off-line runtime environments.