MAD: A Monitor System for Big Data Applications

A big data application usually needs to build a pipeline on the top of workflow engine which connects relevant periodic workflow jobs. It’s crucial to timely alert pipeline issues, provide an issue diagnosis subsystem to find out root cause from a variety of sources, and measure pipeline/service by...

Full description

Saved in:
Bibliographic Details
Published inIntelligence Science and Big Data Engineering. Big Data and Machine Learning Techniques Vol. 9243; pp. 308 - 315
Main Authors Shi, Mingruo, Yuan, Ruiping
Format Book Chapter
LanguageEnglish
Published Switzerland Springer International Publishing AG 01.01.2015
Springer International Publishing
SeriesLecture Notes in Computer Science
Subjects
Graphics programming
Hadoop
Image processing
MAD (Monitoring Alerting and Diagnosis)
Oozie
Pattern recognition
SLA
Online AccessGet full text
ISBN3319238612
9783319238616
ISSN0302-9743
1611-3349
DOI10.1007/978-3-319-23862-3_30

Cover

More Information
Summary:A big data application usually needs to build a pipeline on the top of workflow engine which connects relevant periodic workflow jobs. It’s crucial to timely alert pipeline issues, provide an issue diagnosis subsystem to find out root cause from a variety of sources, and measure pipeline/service by predefined metrics. In this paper, we identify three indispensable qualities monitor systems must fulfill namely timeliness, accuracy and flexibility. We find that the conventional monitoring tools lack at least one of three qualities, and introduce a general purpose MAD (Monitoring, Alerting and Diagnosis) system for big data applications to keep data freshness, collect measurement metrics to meet SLA.
Bibliography:This work is specially supported by the Science and Technology Plan General Program of Beijing Municipal Education Commission (KM201510037001), Chinese Mountaineering Association (CMA2014-B-A04) and Intelligence Logistics System Beijing Key Laboratory (NO:BZ0211)
ISBN:3319238612
9783319238616
ISSN:0302-9743
1611-3349
DOI:10.1007/978-3-319-23862-3_30