基于Spark平台的NetFlow流量分析系统
目前典型的NetFlow分析系统多为基于私有架构或平台的第三方系统,面临扩展性较低、开放性不足、扩容代价大、分析时延长等问题。大数据技术的快速发展尤其是内存式计算平台如Spark的出现为集中处理大规模NetFlow数据提供了可能,本文提出了基于Spark的NetFlow分析系统,验证了核心算法（如流量应用构成统计）在Spark平台的性能。实验表明,基于Spark的NetFlow分析系统具有很高的性能和很强的扩展能力,较之Hadoop MapReduce有显著的性能提升。...
Saved in:
| Published in | 电信科学 Vol. 30; no. 10; pp. 48 - 51 |
|---|---|
| Main Author | |
| Format | Journal Article |
| Language | Chinese |
| Published |
中国通信学会
01.10.2014
人民邮电出版社有限公司 中国电信股份有限公司广东研究院 广州510630 |
| Subjects | |
| Online Access | Get full text |
| ISSN | 1000-0801 |
| DOI | 10.3969/j.issn.1000-0801.2014.10.009 |
Cover
| Abstract | 目前典型的NetFlow分析系统多为基于私有架构或平台的第三方系统,面临扩展性较低、开放性不足、扩容代价大、分析时延长等问题。大数据技术的快速发展尤其是内存式计算平台如Spark的出现为集中处理大规模NetFlow数据提供了可能,本文提出了基于Spark的NetFlow分析系统,验证了核心算法（如流量应用构成统计）在Spark平台的性能。实验表明,基于Spark的NetFlow分析系统具有很高的性能和很强的扩展能力,较之Hadoop MapReduce有显著的性能提升。 |
|---|---|
| AbstractList | 目前典型的NetFlow分析系统多为基于私有架构或平台的第三方系统,面临扩展性较低、开放性不足、扩容代价大、分析时延长等问题.大数据技术的快速发展尤其是内存式计算平台如Spark的出现为集中处理大规模NetFlow数据提供了可能,本文提出了基于Spark的NetFlow分析系统,验证了核心算法(如流量应用构成统计)在Spark平台的性能.实验表明,基于Spark的NetFlow分析系统具有很高的性能和很强的扩展能力,较之Hadoop MapReduce有显著的性能提升. 目前典型的NetFlow分析系统多为基于私有架构或平台的第三方系统,面临扩展性较低、开放性不足、扩容代价大、分析时延长等问题。大数据技术的快速发展尤其是内存式计算平台如Spark的出现为集中处理大规模NetFlow数据提供了可能,本文提出了基于Spark的NetFlow分析系统,验证了核心算法（如流量应用构成统计）在Spark平台的性能。实验表明,基于Spark的NetFlow分析系统具有很高的性能和很强的扩展能力,较之Hadoop MapReduce有显著的性能提升。 |
| Author | 丁圣勇 闵世武 樊勇兵 |
| AuthorAffiliation | 中国电信股份有限公司广东研究院,广州510630 |
| AuthorAffiliation_xml | – name: 中国电信股份有限公司广东研究院 广州510630 |
| Author_FL | Ding Shengyong Fan Yongbing Min Shiwu |
| Author_FL_xml | – sequence: 1 fullname: Ding Shengyong – sequence: 2 fullname: Min Shiwu – sequence: 3 fullname: Fan Yongbing |
| Author_xml | – sequence: 1 fullname: 丁圣勇 闵世武 樊勇兵 |
| BookMark | eNpNj7tOAkEYhafARERewNJouevcd6Y0RNSEaCH9ZmYvuICzuKMBSxOjVmJtIa9gQiyE13HZ-BYugRirk_OfL-fP2QIVk5oIgD0EXSK5POi6ibXGRRBCBwqIXAwRLa0LoayA6t99E9StTTTEhFMOJaoCJ5_MvmcvFwOV9fKvaT7-KN4ezqKbZj8dLj7vf57G-fPj4v21mM6L-WQbbMSqb6P6Wmug3TxqN06c1vnxaeOw5QQel07IBJQiCrHgmvAQlaI1ETiINfVEwBSXEEkmGFFUacY8QjHRniRxwBDWlNTA_qp2qEysTMfvpreZKR_64ag3Wm5bLpIlt7PiAmWtb6wNfc4xp4RKrwx31-FlajrXSVkzyJIrld39g34Bm6pmSg |
| ContentType | Journal Article |
| Copyright | Copyright © Wanfang Data Co. Ltd. All Rights Reserved. |
| Copyright_xml | – notice: Copyright © Wanfang Data Co. Ltd. All Rights Reserved. |
| DBID | 2RA 92L CQIGP W92 ~WA NSCOK 2B. 4A8 92I 93N PSX TCJ |
| DOI | 10.3969/j.issn.1000-0801.2014.10.009 |
| DatabaseName | 维普期刊资源整合服务平台 中文科技期刊数据库-CALIS站点 中文科技期刊数据库-7.0平台 中文科技期刊数据库-工程技术 中文科技期刊数据库- 镜像站点 国家哲学社会科学文献中心 (National Center for Philosophy and Social Sciences Documentation) Wanfang Data Journals - Hong Kong WANFANG Data Centre Wanfang Data Journals 万方数据期刊 - 香港版 China Online Journals (COJ) China Online Journals (COJ) |
| DatabaseTitleList | |
| DeliveryMethod | fulltext_linktorsrc |
| DocumentTitleAlternate | A Large Scale NetFlow Analysis System Based on Spark |
| DocumentTitle_FL | A Large Scale NetFlow Analysis System Based on Spark |
| EndPage | 51 |
| ExternalDocumentID | dxkx201410009 662643497 |
| GroupedDBID | -0Y 2RA 5XA 5XJ 92L ALMA_UNASSIGNED_HOLDINGS CCEZO CQIGP CUBFJ GROUPED_DOAJ U1G U5S W92 ~WA NSCOK 2B. 4A8 92I 93N PSX TCJ |
| ID | FETCH-LOGICAL-c769-d58098ed286b36d186bbb382cfb478c5a690195853a4ab5573423b793fc512b43 |
| ISSN | 1000-0801 |
| IngestDate | Thu May 29 04:00:33 EDT 2025 Tue Jan 21 20:53:48 EST 2025 Wed Feb 14 10:35:16 EST 2024 |
| IsDoiOpenAccess | true |
| IsOpenAccess | true |
| IsPeerReviewed | false |
| IsScholarly | false |
| Issue | 10 |
| Keywords | NetFlow Spark 流量分析 traffic analysis |
| Language | Chinese |
| LinkModel | OpenURL |
| MergedId | FETCHMERGED-LOGICAL-c769-d58098ed286b36d186bbb382cfb478c5a690195853a4ab5573423b793fc512b43 |
| Notes | The existing systems usually adopt private distributed architectures, which face scalability, openness, costand latency problems. The development of big data technology such as Spark offers new opportunity for large scaleNetFlow processing systems. A new analysis system based on Spark platform was proposed and the effectiveness ofthe method was verified. The experimental results show its superior performance. NetFlow, Spark, traffic analysis 11-2103/TN Ding Shengyong, Min Shiwu, Fan Yongbing （Guangdong Research Institute of China Telecom Co., Ltd., Guangzhou 510630, China） |
| OpenAccessLink | http://dx.doi.org/10.3969/j.issn.1000-0801.2014.10.009 |
| PageCount | 4 |
| ParticipantIDs | wanfang_journals_dxkx201410009 cass_nssd_662643497 chongqing_primary_662643497 |
| PublicationCentury | 2000 |
| PublicationDate | 2014-10-01 |
| PublicationDateYYYYMMDD | 2014-10-01 |
| PublicationDate_xml | – month: 10 year: 2014 text: 2014-10-01 day: 01 |
| PublicationDecade | 2010 |
| PublicationTitle | 电信科学 |
| PublicationTitleAlternate | Telecommunications Science |
| PublicationTitle_FL | Telecommunications Science |
| PublicationYear | 2014 |
| Publisher | 中国通信学会 人民邮电出版社有限公司 中国电信股份有限公司广东研究院 广州510630 |
| Publisher_xml | – name: 人民邮电出版社有限公司 – name: 中国通信学会 – name: 中国电信股份有限公司广东研究院 广州510630 |
| SSID | ssib023646091 ssj0002912124 ssib001102832 ssib000459930 ssib051374496 ssib036437025 ssib017479463 ssib058759007 |
| Score | 1.6356599 |
| Snippet | 目前典型的NetFlow分析系统多为基于私有架构或平台的第三方系统,面临扩展性较低、开放性不足、扩容代价大、分析时延长等问题。大数据技术的快速发展尤其是内存式计算平台... 目前典型的NetFlow分析系统多为基于私有架构或平台的第三方系统,面临扩展性较低、开放性不足、扩容代价大、分析时延长等问题.大数据技术的快速发展尤其是内存式计算平台... |
| SourceID | wanfang cass chongqing |
| SourceType | Aggregation Database Publisher |
| StartPage | 48 |
| SubjectTerms | NetFlow Spark 流量分析 |
| Title | 基于Spark平台的NetFlow流量分析系统 |
| URI | http://lib.cqvip.com/qk/90580X/201410/662643497.html https://www.ncpssd.cn/Literature/articleinfo?id=662643497&type=journalArticle https://d.wanfangdata.com.cn/periodical/dxkx201410009 |
| Volume | 30 |
| hasFullText | 1 |
| inHoldings | 1 |
| isFullTextHit | |
| isPrint | |
| journalDatabaseRights | – providerCode: PRVAON databaseName: Directory of Open Access Journals (DOAJ) issn: 1000-0801 databaseCode: DOA dateStart: 20090101 customDbUrl: isFulltext: true dateEnd: 99991231 titleUrlDefault: https://www.doaj.org/ omitProxy: true ssIdentifier: ssj0002912124 providerName: Directory of Open Access Journals – providerCode: PRVHPJ databaseName: ROAD: Directory of Open Access Scholarly Resources issn: 1000-0801 databaseCode: M~E dateStart: 20120101 customDbUrl: isFulltext: true dateEnd: 99991231 titleUrlDefault: https://road.issn.org omitProxy: true ssIdentifier: ssib058759007 providerName: ISSN International Centre |
| link | http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwnV1Lb9QwEI5KkRAXxFOUwopDfUIpSew49jHZZlUhtRcWqbdVnEcrgbKF3RVVD0hICDhRzhzoX0CqOND-HbYr_gUzjjcJVcXrkh3ZzuexZ50Z2-OxZa04XlJQUES2oClMUGhW2MopEjv1g8SnuDUo8HDyxiZff8IebflbCxdoy2tpMlar6f6550r-R6qQBnLFU7L_INkaFBKABvnCEyQMz7-SMYl9InskCknM8Cnix7vo-wzJkSQRRUJAvkPigEjIZ5v5uPds-JLEnESQ55JYEhFgISwqiOCYJWMi9TsAEUWaiKCitiGrERmCYNU9ErqYEkKii1DhGgnrNT9dROjagN8uCSvGItIoQeRDBnM4QaTm4wwKJABKWL-sCWiq3166cFntBFf92QwiYGHtEYnWzuMdgD1NMGwrluHIqpCaM6n7Q9cWdue9Kub93Jsz3dV94CA4EpDFzeuh96BVGgjgIIYPFTfbRZVa0OfvheHd6I2mwKT2za20QBU71NgTVTzds5qKSi61pkL81RoffQ3ZqnY3lI2Grv0ms72ne552zdWHVi96Aed4bcfGq7hto0vZ3irVJmRtw-KFAdxpAiRR3LltbTX7LoxQ1sRg9GFCKx2zRoDmjSddMHe0t8ac60vWimnSw981CKyeFGaEGMVkZ1huPwfbTB-VK4uk3G5Zdf2r1hUzHbsfVmPrmrWwv3PDsqeHx9-PP-ixNP12ND34Mvv0xoyc06-vf7w7mL5_e_r54-zoZHZyeNPq9-J-d90214rYacClnfnCkSLPPMEV5ZkLP0pR4aWFYoFI_QSvaJMwi6YJS5TvBxgjU4EaK1IwjhWjt6zFcljmtzHeAd6PAAZCkipWeJngPFeBl8GUKSkYzZes69jgQTkaZQOQFHQ1k8GStVy3f7BbxZRp53ZMjwzMl2Y0-EXsd_5UYNm63Ay2u9bi-MUkvweW81h19IpTR_9ffgKpxoyJ |
| linkProvider | ISSN International Centre |
| openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=%E5%9F%BA%E4%BA%8ESpark%E5%B9%B3%E5%8F%B0%E7%9A%84NetFlow%E6%B5%81%E9%87%8F%E5%88%86%E6%9E%90%E7%B3%BB%E7%BB%9F&rft.jtitle=%E7%94%B5%E4%BF%A1%E7%A7%91%E5%AD%A6&rft.au=%E4%B8%81%E5%9C%A3%E5%8B%87&rft.au=%E9%97%B5%E4%B8%96%E6%AD%A6&rft.au=%E6%A8%8A%E5%8B%87%E5%85%B5&rft.date=2014-10-01&rft.pub=%E4%B8%AD%E5%9B%BD%E7%94%B5%E4%BF%A1%E8%82%A1%E4%BB%BD%E6%9C%89%E9%99%90%E5%85%AC%E5%8F%B8%E5%B9%BF%E4%B8%9C%E7%A0%94%E7%A9%B6%E9%99%A2+%E5%B9%BF%E5%B7%9E510630&rft.issn=1000-0801&rft.volume=30&rft.issue=10&rft.spage=48&rft.epage=51&rft_id=info:doi/10.3969%2Fj.issn.1000-0801.2014.10.009&rft.externalDocID=dxkx201410009 |
| thumbnail_s | http://utb.summon.serialssolutions.com/2.0.0/image/custom?url=http%3A%2F%2Fimage.cqvip.com%2Fvip1000%2Fqk%2F90580X%2F90580X.jpg http://utb.summon.serialssolutions.com/2.0.0/image/custom?url=http%3A%2F%2Fwww.wanfangdata.com.cn%2Fimages%2FPeriodicalImages%2Fdxkx%2Fdxkx.jpg |